Tuesday, October 22, 2019

Raspberry Pi Pi-hole Network-wide blocking of Ads, tracking, and popups

How-To: In two hours, with no previous experience, you can build a small "DNS Sink Hole" that can block ads, tracking cookies, popups, and email-trackers -- all by using a small $50 computer called a Raspberry Pi. 

But most importantly, questionable sites, such as ransomware and other scams, are blocked at the network layer, long before your browser has a chance to see them.

This works for all devices in your network, including all desktops, laptops, phones, and tablets.

You no longer need to install ad-blocking software.  All the benefits happen for all devices behind your router -- and you do not have to configure them to gain the benefits.


This replaces a previous Keyliner article:
Stopping Tracking Cookies with whack-a-mole - blocking DNS using Acrylic DNS.



The Raspberry Pi becomes a dedicated computer that handles all DNS (Domain Name Service) requests -- taking the function away from your existing routers.  When an address, such as "keyliner.com" is typed, a request goes to your Domain Name Service.  It translates the human-readable name into an IP address.  If the address is nefarious, or an ad-network, the packet is discarded, keeping the traffic from reaching the device.

As of this article, the device blocks 107,000 domains (illustrated), now 116,000 domains.  Here is a chart showing the normal traffic at my house, with blocked requests in blue.  Of the 13,000 requests, 2,500 were blocked:





Raspberry Pi, you say?


To make this work, build a small dedicated computer using a device called a Raspberry Pi.  Then install DNS software called "Pi-Hole" (an open-source, community-developed Domain Name Service supported by hundreds of volunteers).

"I don't know anything about that!" Neither did I!

And yet, with zero experience, I built the PC, installed the operating system, and configured everything -- all in about two hours.  The operating system and DNS software are free.

You can do this! 


What is a Raspberry Pi?

A Pi is a small computer, running Linux and costs about $50.  It has 4 USB ports, an HDMI Video port, an RJ45 wired network jack, Bluetooth, a wireless adapter, and a slot for an SD-card drive. You do not need to know Linux to build this project. 

I found this model on Amazon, which included a case, power-supply.

I also like this model (where you supply your own cell-phone 2.5A charger), or this model.  These are all similar, except for cosmetics, accoutrements, HDMI cables, and the like.  I noted Walmart.com carries the same products, with free shipping for the same cost.  I am not getting a kickback on these links or ad revenue.

I am using an older Raspberry Pi 2.0.  Version 3b is now available.  Either will work.  A Pi 2.0 can easily support up to 2M DNS transactions per hour -- well within the realm of a 50 workstation network.


You will also need the following:

HDMI cable to connect to your TV or monitor (temporary, just for setup).  The cable may be included in your kit.

Short .5 or 1 Meter (2 - 3 ft) Ethernet patch cable ($5)
Wired or wireless Keyboard (borrow from your PC)
Wired or wireless Mouse (borrow from your PC)

16GB (recommended) or 32GB Micro SD card, with adapter. 
This is often included in the purchased Raspberry Pi kit:


The SD card acts as the Pi's hard drive.  Note, this is the Micro SD card, which is much smaller than a postage stamp.  Buy these at any electronics or office store.  Shop around for the best price (expect about $10).  Buy the card with a standard-sized adapter so it can plug it into a laptop or desktop's SD slot.


Before staring the buildt, some research is required.


Important Prerequisites

A.  From your PC, discover your IP-address pool-range with these geeky but easy steps:

From a DOS / Command prompt (windows-R, "CMD"), type this command:

ipconfig (enter)


* Note your IPV4 address, illustrated.

Yours will probably read something like
192.168.0.10    (mine happens to be 192.168.100.10)

* Note the Default Gateway (mine is 192.168.100.1) - This is your main router.

B.  Decision

If your workstation's ipv4 address is below 10, such as 192.168.0.2  or 192.168.1.3,
write-down, and later use, this fixed IP address: 192.168.0.151

This will be the Pi's new internal IP address, where the first three octets will be the same as your workstation and "151" is probably beyond the largest value the home router will provide.  


If the displayed IPV4 address is something like 192.168.0.11 (or some number higher than 9), then consider using this IP Address for your soon-to-be-built Pi:  192.168.0.5 (-now in retrospect, it is probably safest to always use a .151 address because this address is beyond most auto-assigned addresses).

Technical notes for those who care:  Home routers assign automatic DHCP addresses to each workstation on the network using a range or pool of numbers.  This range varies by router manufacturer.  Some start at 2 - through 100, others start at 10 through 150.  The range does not matter, but the Raspberry Pi needs a number from outside that range.  With admin rights (see immediately below), login to the router's 100.1 address, and confirm the exact "DHCP" Address range.  But the steps above are a good-enough approximation, and .151 is likely safe in all normal cases. 


C.  You must be able to login to your main Router's admin screen to complete this project. 

If you have a secondary (Wireless) router, that router may be the one reported on the DOS screens above.  Use that address for all following steps.  If the secondary router has an IP Address of .2, ignore it and do all the work on the .1 address.  With two routers, this can be complicated.  Do the work on your workstation's .1 router.

Open a browser.  In the URL line, type your router's main IP address -- illustrated in the DOS screen above as the "Default Gateway."  The first three octets will be the same as your workstation's address.  The last octet will most likely be a dot-1. 

For example,

192.168.0.1  

(also typical are addresses like this:  192.168.1.1 .  Your network may be different.  The main router's last octet is almost always dot-one.). 


D.  Look on the side of the router for a printed label that shows the admin login ID and password, or you may have recorded the password when the network was first built.

Login with "admin"  (usually, lower-case-a)
Confirm you can login and get to the router's administrative screens.



You must be able to login to your router's admin screens before continuing.  If not, consider this keyliner article, and this one.  Your ISP or the person who setup your original network may be able to help. 






Raspberry Pi Hardware Setup


A new Raspberry Pi is a small circuit board.  Snap it into the kit's plastic case, and if the kit came with self-adhesive heat-sinks, apply them now. 

Next, download and install the Linux operating system with these steps (this step can be skipped if your raspberry kit came with a pre-installed NOOB operating system; if so, jump to step 4, but if you have the time, do these steps for the most recent versions.):


1.  From a PC, go to the Raspbian download site and download the "NOOBS Offline and Network Install.zip".  This download is slow and will take several hours (they have a slow network connection and it probably has to cross the Atlantic. 1.7Gig file.):

https://www.raspberrypi.org/downloads/noobs/

Save the .ZIP to a known location.

2.  Insert the MicroSD card into your PC's card-reader.

The card must be a 16GB or 32GB card (64GB is too large).
If prompted, format the card - format it like you would with any disk or USB thumb drive.


3.  On the PC, using File Explorer, open the .ZIP and copy all files and folders within the .zip to the SD card's root directory.

(Important:  Do not copy the .zip file -- copy the contents inside the zip. 
Use Copy-and-Paste -- not Cut-and-paste.  Do not click-and-drag
).

 
Details:  To copy, double-click .ZIP to open.  
               On detail side. click the first file/folder. 
               Shift-Click the last file/folder.
               Hover on the highlighted files, "other-mouse-click", choose "Copy" (not cut)

               Find the SD Card drive (On my PC, this showed as Drive G:"
               In the details pane, other-mouse-click and chose "Paste"

Once copied, eject the SD card.

The .zip is no longer needed and can be deleted.


4.  Remove the Micro-SD Card-insert from the SD-card adapter.
  • Insert into the Raspberry Pi's card-slot
  • The SD-card installs "up-side-down," into the board's slot
  • Push until it clicks in place

5.  Connect the HDMI cable to a TV or Monitor  (I used my TV).
  • Connect a USB Keyboard (borrow from your desktop; can be wireless)
  • Connect a USB Mouse (can be wireless) If you have a Cat-5 cable and can easily connect to the router, do so now.  Otherwise, you can use a wireless connection for the initial setup.
     
  • For the initial setup, use either a wired or wireless connection.  Wired is preferred. 

    For a Wired RJ45:  If near the main router, connect an RJ45 network cable to any open port on any router.  (Do not plug in to the router's "uplink" port (plug into one of the 4 or 8 port areas)).   Connect the other end to the Pi's RJ45 port. 

    If Wireless, continue with the USB power supply step.  Note: Later, you must switch to a wired connection.
  • Connect the USB 2.5a power supply to the Pi.  (Any 2.5a micro-USB cell charger will work.  Usually supplied in a the Pi-kit.)

The Raspberry Pi will boot; visible on TV.  You may need to switch the TV's INPUT to find the right HDMI port.



Raspberry Pi Operating System Install


6.  When the Pi first-time boots with the new SD-card installed, it will arrive automatically at the Raspian Operating System Installation screen.  Select the top-most Raspberry PI operating system,

[x] Raspian Full (Recommended)

Click the Install button on the ribbon bar. 
Install takes apx 45 minutes.  When done, it will boot to a desktop.

Black Screen:  I had troubles when partway through the install, the TV showed "signal not found."  The TV was routed through a stereo, and the stereo would go into power-save mode.  Rebooting the stereo returned the TV's Pi image.


When prompted:
  • Select Country:  (e.g. United States)
  • Language: (e.g. note American English, British English) 
  • TimeZone:  (oddly by City name)
  • Important:  If United States, you must click the [x] US Keyboard option
When prompted for the Admin password:
  • Change the admin password to a password of your choosing
  • Write the password on the checklist above

7.  Network Decision - Wired or Wireless:

If the Pi is currently connected to a Wired network, allow the Pi to auto-update and patch.

If using a wireless for the install, See the top-menus. 
(Most modern Pi's have Wireless built-in)

Right-click the right-side wireless-strength icon.  Configure the SSID on WLan0, etc., connecting the Pi to the wireless network, much like any other device.   The wireless connection is temporary and can only be used for initial setup.  Later, it must be changed to a wired connection.

7a.  Allow Pi to auto-update and patch. 
  • After patching, the Pi will reboot
  • After patching, re-enter the Location, Keyboard, and Admin password, when prompted

7b.  At the Raspberry-top-menu icon,
  • Click "Raspberry-pi Configuration", "Interfaces"
  • Enable SSH  (allows remote desktop control - handy for geeks; do it now, while convenient)


At this stage, you have a fully-installed, fully-usable copy of Linux.  Pat yourself on the back because you are good!


8.  Optional Cleanup Steps:


The Raspian operating system comes pre-installed with extra software that is not needed for this project.  The Raspberry Pi and Pi-hole software will run as-is, but if you are geek, and don't mind spending another hour, consider uninstalling the following programs.  This will make the Pi faster and leaves more space on the drive for logs and updates:

From the main Linux desktop, top-menu, open a Terminal Window.
Type these commands, pressing ENTER after each.  If software is not found, press the up-arrow and double-check the spelling, or move to the next command.

Answer with "Y" (capital Y), when prompted:

a.   sudo apt-get purge wolfram-engine
b.   sudo apt-get remove --purge libreoffice* 
c.   sudo apt-get purge sonic-pi
d.   sudo apt-get purge scratch
e.   sudo apt-get purge greenfoot
f.   sudo apt-get purge geany
  g.   sudo apt-get purge nuscratch
h.   sudo apt-get purge python-pygame
i.   sudo apt-get purge pygame
j.   sudo apt-get purge squeak-vm
k.   sudo apt-get purge dillo
 
l.   sudo apt-get purge minecraft-pi
m.   sudo apt-get purge penguinspuzzle
n.   sudo apt-get purge oracle-java8-jkk
o.   sudo apt-get purge oracle-java7-jdk
p.   sudo apt-get purge openjdk-8-jre

If any of the above, reboot.  Then follow with these two commands:

x.  sudo apt-get clean
y.  sudo apt-get autoremove --purge

Update the OS with this command:

z.  sudo apt-get update && sudo apt-get upgrade -y

Optional software is now de-installed.  Approximately 2G of disk space is freed.



The next step is to install the Pi-Hole DNS Server software.


Install Pi-Hole DNS

Once the operating system is installed and patched, install the Pi-Hole software:

9.  On the Raspberry Pi's top-menu, open the "Terminal Window" (command prompt)

  • Type this case-sensitive command. 
    Note the "-sSL" -- is very case-sensitive.  Note the split-vertical bar:
  • curl -sSL https://install.pi-hole.net | bash

10.  Answer these prompts:

"This installer will transform your device into a network-wide ad-blocker" 
tab to the OK button, press Enter

  • You may be prompted to: 
    Choose eth0 for the hard-wired port
    (This must be selected even if using wireless during the base install)
  • Accept Google (or OpenDNS) as the upstream DNS Provider
    I prefer Google,  knowing the Pi subscribes to the same lists as OpenDNS
  • Accept the default third-party list; tab for OK
  • Choose IPV4 (not IPV6) for the protocol

!!!  Important:  When prompted
!!!  "do you want to use your current network settings as a static address"
  • -- tab to "No"
  • Press Enter 
  • If this step is missed, press ESC and restart at the CURL step.
(Reason:  Set a static, hard-coded IP-address on the wired network.) 
  • For an IP Address, set a "Static" / fixed/hard-coded IP address, found and written down in the prerequisite steps:
     
  • Type the Raspberry Pi's IP-address, from the prerequisites (it was either xx.5, or xx.151).
    Type the full address, appending a trailing "/24" --  (slash /24 sets the subnet mask to 255.255.255.0)

    Examples from your prerequisite/decision:
192.168.0.151/24   or
192.168.0.5/24
192.168.1.151/24 etc.
  • Set the Default "Gateway" to the same address as your workstation's Gateway IP Address. 

    This is the main router's IP address; the same as your workstation's main router address.
    See the checklist, above:

    Typically:
192.168.0.1    (192.168.1.1 etc.)

  • Allow it to install the Web Admin Interface
  • Accept Log Queries, ON  (recommended)
  • (If the install goes "south," reboot the PI and restart the curl command.)


11.  At the "Installation Complete" screen

Step away from the keyboard and
carefully write down the insufferable "Administrative login/password"
and the set-installed IP address.

For example, my machine showed: 
192.168.100.5/admin   Password: xxxxxxx______________________


12.  Change the Pi's admin password.  Do this now, while it is easy to get to these screens:

From the main desktop, open a terminal window.

Type this command:
pihole -a -p

Follow the prompts to change the password.
* Record this final password in the checklist above.

The Pi-Hole is now fully configured and ready to use. A moment of self-congratulations is in order.




Pi Final (Production) Wiring Steps:

Using the top-Raspberry menu, shut-down the server.
Unplug the HDMI cable; The monitor is no longer needed.
Unplug the Keyboard and Mouse; these are no longer needed.


13.  Move the Raspberry Pi to a location near the main router. 

Using a short Cat-5 Network cable, plug in the Raspberry into any available port on your router. A hard-wired connection is required.

For example, my home network looks like this, where the Pi was connected to an 8-port switch.  It could have been easily connected to the DSL or Wireless router's open (yellow) port -- any open port can be used, where there are groups of 4 or 8 network jacks.  Do not plug it into the up-link port (a lonely port, usually a different color):

* If your network is run only on the optional (secondary) wireless router (with a .1 address), plug the Pi into that wireless router.


Plug in the power adapter. 

Give the Pi a minute to boot and get settled.
Note the activity lights on the Pi's RJ45 network port.

(See this keyliner article for a photo of my home setup)


14.  Initial Test: 

From your PC-workstation, open a DOS / Command Prompt and ping the Pi to see if it is on the network.  Type this command:

PING 192.168.0.151  (or 0.5, or 1.151, etc)

It should reply in xx milliseconds.


Router Setup

The final step is to configure your router(s) to point to this new Domain Service. 
These changes are required in order to activate the Pi. 
This is a one-time setup.

For most households, the main DSL or Cable-Modem router (the box with a .1 ip-address) is the one which needs to be changed, but some networks may use a secondary wireless router as the main router.  In any case, make these changes on the workstation's .1 router. 




A.  Login to the main router's 192.168.0.1  address (as tested in the per-requisites, above)

Typically, type this address in the URL address bar and press enter.
Your address may be different:

192.168.0.1


B.  Login with "admin" and the previously-recorded password.

The main setup screens vary by modem manufacturer.  Several examples are illustrated below:
  • Usually under an Advanced Configuration menu
  • Look for a DNS Setup section
    (or sometimes DHCP/DNS)
  • Look for
    "Dynamic DNS" (or "Auto-DNS", or "use these DNS Servers", depending on modem)
     
  • Change to:
    Static DNS or "Use these DNS Servers"...
  • At the Primary DNS, type the IP address of the Raspberry Pi.
    For example, on my network, 192.168.100.5  (or 192.168.0.151, from your prerequisites)
  • Optionally type a Secondary DNS (not necessarily recommended)
    8.8.8.8
    (Or use an Open DNS address, documented at the end of this article)

    I leave my secondary blank, as the Pi already defaults to your favorite secondary DNS as part of its initial install.  If the Pi fails, I want the network DNS to shut-down and not find an alternate path.  See the end of this article for more discussion about this. 

    Some routers require a secondary DNS (and one must be typed).  If so, use 8.8.8.8.  But, if you want to force all DNS traffic through the Pi, use a dummy secondary address of 127.0.0.1.  I personally like the 127.0.0.1 option.  Again, see the discussion near the end of this article for reasons. 
See the red-section, directly below for other modem examples.


C.  Important:  Save the changes by clicking this screen's SAVE or APPLY button.  Do this before moving to any other screen.  The router will reboot.


Example Modem Setup Screens:

My Zyxcel DSL router looked/s like this:

Click for larger view


* Some of newer models of routers require a secondary DNS
-- Used Google's  8.8.8.8 -- which is redundant because this is the go-to address used by the Pi, or better yet, use a dummy address of 127.0.0.1 to disable this feature:

Click for larger view



* A typical Linksys router looked like this, where in this case, the network was 192.168.1.150 (should have been 192.168.1.151):




* Another version of a linksys router looked like this, where the pi-hole's address of, 192.168.100.5 was added:



* A NetGear Genie AC1450 looked like this, where the Raspberry Pi was the primary and again, Google's DNS was set as a secondary:

Click to enlarge

                                           


D.  If you have a secondary, wireless router (rare for most households), typically at 192.168.100.2, look to see if it needs to be configured. 

Login to that device's admin screen by opening a browser and typing the wireless router's IP Address
typically:  192.168.0.2,   (but could be something like 192.168.1.1 See your prerequisites)

Login to the administrator's screen, again with a default password likely printed on a back label. Snoop-around the setup screens (Basic Setup, Advanced Setup), looking for a DNS Server. 

Usually these routers use the main router for DNS and likely, you will *not* find a DNS Server setting (don't confuse with DHCP -- which is probably disabled).  If DNS settings cannot be found, jump to the Testing steps. 

If a DNS entry is found, make similar DNS changes.

*Note:  If you can't login to the router's admin screen -- and often you can't while passing through a wired network, consider the following:

1.  Use a wireless device to reach the configuration screens.  Or,

2.  With a laptop or desktop,
     Run a temporary hard-wired RJ45 connection directly from the PC
     to any available yellow-port on the wireless router.

3. Reboot the PC to get a new IP address.  IPConfig to see your new IP Address.
4. Try logging into the dot-1, dot-2 IP Address again.


Raspberry Pi and Pi-hole configuration is complete!
I recommend the following tests and recommend logging into the Pi-hole's admin screens.  These topics are covered next.



TESTING

Ublock Origin, illustrated
To properly test, disable the workstation's locally-installed ad-blocking software.  Reason: Ad-blockers also block traffic.  The difference is they block the traffic *after* it has downloaded where-as the Pi keeps them from ever downloading. 

You may or may not have ad-blocking software installed.  Look in your browser's Tools, Add-Ins menu and look for "adblock-plus" or "uBlock Origin" (the two most commonly used blockers).

If installed, close the Add-in screens and look on your browser's upper-right menu bar, looking for a UBlock Origin or an Adblock-plus icon.  Click the icon and temporarily disable the ad-blocker.

Test 1: 

This test makes sure the network is functioning properly and you have the routers pointing to the right DNS-resolver (the Pi-hole).
  • From your normal workstation, browse to www.google.com.
  • If you arrive, the DNS is working correctly.
Test 2:
  • Browse to Yahoo.com
     
  • Note "holes" in the page -- blank spaces, illustrated below in orange.  There are being snuffed by the Pi-hole.
     
  • Be sure adblockers are disabled or this test will be distorted.
     
  • Note "holes" in displayed page. These are never transmitted; speeding up page-loads.  The drawback is content providers cannot monetize their content.  There are moral and ethical considerations; see the end of this article for a discussion.  On the other hand, they are often abusive and can (accidentally) provide malicious content.
Click for larger view

Test 3:
  • Attempt to browse to  http://tag.bounceexchange.com - a nefarious site
  • Note how pi-hole blocks the address.  It may look like this or this, depending on your browser:



    or this:

  • Browse to  didtheyreadit.com  (an email tracking service that uses one-pixel white images on emails to track if you opened the email).  As-of this article, you will likely succeed and arrive at the site.
     
  • Consider "Blacklisting" this and other such sites.  See the blacklist later in this article.
     
  • Note that *all* devices in your network benefit from the Pi.  And, more importantly, none of the devices need to be told about the setup -- it just works.  But if your device (cell phone, tablet, laptop) strays from the network, the Pi's benefits are lost.

Side-notes:  If the domain is on the naughty-list, the Pi dumps the DNS request into a dark hole, hence "pi-hole."  As of this article, over 116,000 domains are in the discard list.  If the address is on the good-boy list, it is handed off to (Google's) Domain Services.  Google resolves the address normally. 

Most home routers use your ISP's Domain Name Services, for example, CableOne, Century Link, Comcast, etc., and some ISPs have been known to slip-stream their own advertisements into your data-stream(!), replacing ads with their own.  With the Pi-hole (or Google's DNS, 8.8.8.8), all DNS calls are resolved with a trustworthy source.


Testing:  Simulate a pi-failure:

Unplug the Raspberry Pi's power and attempt to browse any site from any workstation. 
You will find no internet addresses resolve*.  In other words, the Pi is required to be online -- just like your router is required to be online.  Restart the Pi and give it a few minutes to boot and repeat the test, confirming the network returned to normal.

(* If you typed a secondary DNS in your router, traffic routes to the secondary address when the Pi is offline.  This is good and bad.  The secondary will resolve domain addresses, stopping a catastrophic failure, but you will not know the Pi is offline and will lose the benefits of nefarious-site-blocking.  If your router forces a secondary DNS, consider using a dummy  ip-address of 127.0.0.1.  This will force all traffic through the Pi -- making the Pi, once again, a critical component.)


What happens under the hood:

When a device tries to resolve a blocked domain name, the DNS service drops the request in the hole and discards it.  The target domain does not even know a call was attempted.  No graphics, scripts, or other code runs from that site.  Similarly, if a page has embedded code that reaches out to other (blocked) third party domains, those domains are dropped; the code will think no network was available.  This is a win-win for you.

Cell phone and table surfing, using the local networks, display s ugly "webpage not available" in the middle of the article -- this is likely an advertisement and likely that ad is recording your PC's IP address and other information.  The 'page not available' message is the Pi-hole at work, discarding the traffic.  Each application or browser decides how to handle the error in its own fashion.



Many applications and browsers show white-space where the ad lived -- with no obvious errors. 

The neat thing about this is the vendor never knew you attempted the connection because it is blocked before the traffic left the house.  You won't be tracked, monitored, or recorded as you read articles, and big advertising graphics won't download.

Note: Some ads are now being hosted directly in the target's internal pages.  If the main site can be reached, those types of embedded ads will be allowed through.  This is hard to trap.


As seen on the administrative screens, here is a snapshot of recent activity after a few random seconds of activity.  My TV is busy on the network, playing Pandora.  I caught a Nest Thermostat checking on the daily weather.  This traffic was allowed to pass. 

Click for larger view

But "settings-win.data.microsoft.com" was blocked.  This is Microsoft collecting diagnostic data for the Consumer Windows Experience program; see link Infoworld article.  The Pi-hole team decided this was intrusive, and added this address to the blocked domain list.  From the admin panel, it could be white-listed with a click.

This report is where I find ad and email tracking sites.


Pi Administrative Login:

Test the administrative login.

From any browser, type the Pi's IP address/admin:

192.168.0.5/admin                (press enter)
192.168.0.151/admin

On the left-nav, click Login, using the Pi's administrator password (changed and recorded in the setup steps).  The Dashboard displays.  There are two areas of particular interest:  White and Black lists.


White Lists:

For sites where you want to support advertising, such as the NewYork Times, allow them their ad-revenue by adding their domain to the Pi's white-list.

If you decide to keep your browser's ad-blocking software installed, you will also have to add the domain to that program's white-list.  With this said, I would de-install adblockers from your desktop clients -- but leave them installed on laptops that might travel outside of the pi-network.

Add these domains to the Pi's whitelist

nexus.officeapps.live.com    (Microsoft; used by Outlook; Media Player)
redire.metaservices.microsoft.com  (reported by Windows Media Player)
 



Black Lists:


Keyliner recommends manually adding the following to the Black List -- especially the Email tracking addresses. These are addresses I have discovered, that have not made it to the Pi's official lists. From the Pi-hole's administrative login screen, manually blacklisted these additional sites.   As of 2019.10, some of the sites are now on the Pi-block lists.  It does not hurt to still add them.

When black-listing; always add as a "WildCard":

123banners.com
l90.com
adforce.com
advertising.com
agkn.com
appnet.com
avenuea.com
babator.com
bananatag.com      #email tracking
bluekai.com
bluestreak.com
burstmedia.com
burstnet.com
cirrusinsight.com  #email tracking
clearslide.com     #email tracking
clipix.com
contactmonkey.com  #email tracking
demdex.net
deskun.com         #email tracking
didtheyreadit.com  #email tracking
doubleclick.com
doubleclick.net
dynamicyield.com
engage.com
exelator.com
extreme-dm.com
fastclick.net
filepicker.io
g2crowd.com       #email tracking iko system also velocify
getnotify.com     #email tracking
gigya.com
gmelius.com       #email tracking
gobankingrates.com
go.com
hubspot.com       #email tracking
icanbuy.com
imgis.com
imrworldwide.com
intelliverse.com  #email tracking
keywee.co         #Note the .co, not .com
livehive.com      #email tracking
mail-track.com    #email tracking
minute.ly
newtonmail.com    #email tracking
nr-data.net
optimizely.com
outbrain.com
outreach.com      #email tracking
pagefair.com
pixelsite.info    #email tracking
pubexchange.com
quantserve.com
remail.com        #email tracking
remail.io         #email tracking
rlcdn.com
rocketbolt.com    #email tracking
ru4.com
salesloft.com     #email tracking
sidekick.com      #email tracking, now hubspot
saleshandy.com    #email tracking
scorecardresearch.com
stats.net
streak.comp
sync.optimatic.com
taboola.com
teknosurf.com
tinypass.com
toutapp.com       #email tracking
tru.am
valueclick.com
velocify.com      #email tracking Velocity Pulse
voicefive.com
websidestory.com
w55c.net
yesware.com       #email tracking




De-Installing the Pi:

From the admin screen, temporarily disable the Pi for (5-minutes, 10-minutes) while testing.  When disabled, all requests pass through to (Google's) DNS service, and all Pi-protection is lost.  Note: This was specified in the Pi-installation screens -- and this is not your router's secondary DNS setting.

To permanently remove the Pi-hole from the network, re-edit the local .1 Gateway router(s), changing the Static DNS field

from (192.168.0.5  or 192.168.1.151, etc.)
to Google's DNS: 8.8.8.8

A worse choice would be to return the Routers to "Auto-DNS" -- this would put you at the ISP's mercy.

Click "Save".  The router will reboot.

Once changed, the Pi can be unplugged and removed from the network.  No workstations or other devices need to be told of this change.


Known Problems:

Some sites, especially those that show the "top 100 celebrity before and after photos" will be blocked.  Reason: These are trolling sites, with obtrusive ads and with possible fly-by installs.  These sites were deemed dangerous, and were blocked by the Pi-hole community.  Trust their decision.


Sadly, every other type of web failure will be blamed on the Pi. 

My experience is the Pi has not been wrong, but the family will blame the Pi for all network problems.

To test if the Pi is causing a problem with a site, use the admin screens to temporarily disable the Pi-hole.  Re-test the site or page in question (see side-illustration, directly above).

If the site still malfunctions, then the Pi is innocent.  The Pi does not interfere with non-blocked sites.  If the Pi blocks the site, it almost always has a good reason for doing so.  If you trust the blocked site, and insist on arriving (overriding thousands of volunteer's opinions), add the domain to the white list. 




Pi-hole and Ad-blocker Ethics

A word about publishers who need revenue to keep producing content.  Ad-blocking, and the Pi-hole, cut into these revenue streams. But the current model of using third-parties to display ads is broken.  With this, we might want to let the New York Times broadcast ads, but the ad-sites are being blocked as a third-parties -- and it is not possible to allow an exception without allowing the same ad-network across all sites. 

Many publishers now detect ad-blockers, such as Ublock-Origin, and refuse to display the content.  The Pi-hole can sometimes dance around that restriction.  In other words, disable the locally-installed ad-blocker, and often the Pi-hole can do the blocking undetected.

Ultimately, as the industry matures, publishers will be forced to host the ads on their own site and ad-blockers, and the pi-hole, will be less effective.  In other words, you can't block the New York Times completely, or none of the content will show.

The other side of this argument is obvious:  Publishers and Advertisers have abused ads.  Displaying annoying ads, ads that occupy most of the screen, non-dismissable ads, and small articles broken into dozens of pages, all to force ad-impressions.  Abuse is everywhere.  This is why Pi-hole exists.



Update:
2019.10 - Pi is still doing a fabulous job.  Completely unattended.  Every few months I review logs to see what is going on.  A few weeks ago, a visiting friend could not get to disneyworld.disney.go.com sites.  The Pi folks must have thought the site low-quality, with a lot of spam.  I whitelisted for him.  This is my first-ever required white-list.

 2019.01 - I completely rebuilt the Pi, with new OS and new versions of Pi, using these same instructions.  The new version looks and acts identically to the old.  And, as before, all is well.  Still very pleased with the device.  I added more bling to this article to make it easier to follow.  This is an admittedly a complex project.

2018.06 - Six months and the PI is still going strong!  Still a fun and recommended project.

2018.03 - My spouse was trying to login to a site to pay a bill.  The site turned out to be a phishing site from an email (never click links in email!).  The pi intercepted.  Spouse complained she could not login.  The Pi saved our checking account that day.



Related Keyliner Articles:
This is the way I used to do this -- manually blocking about 50 high-volume sites.  With this article, I now block 100,000 sites!
Stopping Tracking Cookies with whack-a-mole - blocking DNS using Acrylic DNS.

Learn more about the pi-hole project here:
https://pi-hole.net/2018/01/11/pi-hole-is-open-source-consume-contribute-or-both/#more-9734
and
https://pi-hole.net/2017/05/12/seven-things-you-may-not-know-about-pi-hole


Related Thoughts:
Some routers run Linux under the hood and can be re-programmed to run a pi-hole directly on the router.  After reading this article, https://www.ab-solution.info, my co-worker reviewed such a project.  The router seems to be the best place to run this type of process, but not all routers can be re-programmed, and this takes skills which are more simply done on a Pi-hole.  Ultimately, he reported back (unspecified) troubles and abandoned that solution, returning to a simpler Pi-hole.


Instead of Google's DNS 8.8.8.8, you can use openDNS's ip address. 
These are now options on the Pi-hole's installation screens:

208.67.222.222 or
208.67.220.220.

OpenDNS also has a "Home" service that blocks phishing sites, porn, acting much like the PI.  Use these addresses on your router, instead of this project, for a simpler solution.  But the Pi is a tad faster and it subscribes to the same OpenDSN list, so little is gained by making this the Pi's secondary address:
  • 208.67.222.123
  • 208.67.220.123
Your comments:

I would love to hear your comments on this project. If you like the Pi-hole project, donate a few dollars their way; they deserve the support.  See the admin-login screens.

Originally published: 2017.11
2019.01  Rewritten and updated.
2019.10  Improved grammar; an editor's work is never done.