2019-10-22

Raspberry Pi Pi-hole Network-wide blocking of Ads, tracking, and popups

How-To: In two hours, with no previous experience, you can build a "DNS Sink Hole" that can block ads, tracking cookies, popups, and email-trackers -- all by using a small $50 computer called a Raspberry Pi. 

But most importantly, questionable sites, such as ransomware and other scams, are blocked at the network layer, long before your browser sees them.

This works for all devices in your network, including all desktops, laptops, phones, and tablets.  You no longer need to install ad-blocking software.  Benefits happen for all devices behind the  router -- They do not have to be configured to gain the benefits.


This replaces a previous Keyliner article:
Stopping Tracking Cookies with whack-a-mole - blocking DNS using Acrylic DNS.

(This is a long article.  But you can really do this in a few hours.)


The Raspberry Pi becomes a dedicated computer that handles all DNS (Domain Name Service) requests -- taking the function away from your existing routers.  When an address, such as "keyliner.com" is typed, a request goes to your Domain Name Service.  It translates the human-readable name into an IP address.  If the address is nefarious, or an ad-network, the packet is discarded, keeping the traffic from reaching the device.

As of this article, the device blocks 170,000 domains (illustrated at 107,000).  Here is a chart showing the normal traffic at my house, with blocked requests in blue.  Of the 13,000 requests, 2,500 were blocked (average 27% of all DNS requests are dropped!):



Spot-check update: 2023.01  Last 24 hours:  24,000 total queries, 4,800 blocked (20%).


Raspberry Pi, you say?

To make this work, build a small dedicated computer using a device called a Raspberry Pi. 

"I don't know anything about that!" Neither did I!

And yet, with zero experience, I built the server, installed Linux, and configured everything -- all in about two hours.  The operating system and the soon-to-be-installed "pi-hole" software are free.  The only expense is a $50 computer.

You can do this! 

What is a Raspberry Pi?

A Pi is a small computer, the size of a deck of cards, running Linux, and costs about $50 to $60.  It has 4 USB ports, an HDMI Video port, an RJ45 wired network jack, Bluetooth, a wireless adapter, and a slot for an SD-card. 

Google-search to find a reseller  I bought mine from Walmart.com.  Any Pi version will work, where I am using an older version 3; version 4 is the newer (2023).


You will also need the following:

HDMI cable to connect to your TV or monitor (temporary, just for setup).
Short .5 or 1 Meter (2 - 3 ft) Ethernet patch cable ($5)
Wired or wireless Keyboard (borrow from your PC, temporary)
Wired or wireless Mouse (borrow from your PC, temporary)

16GB (recommended) or 32GB Micro SD card, with adapter. 
This is often included in the purchased Raspberry Pi kit:


The SD card acts as the Pi's hard drive.  This is the Micro SD card, which is smaller than a postage stamp.  Buy these at any electronics or office store.  Shop around for the best price (expect about $10).  Buy the card with a standard-sized adapter so it can plug it into a laptop or desktop's SD slot.  ***Do not buy a 64GB or larger drive; it is not needed, nor will it work.

Prerequisites:

You must be able to login to your router's admin screens before continuing.  If not, consider this keyliner article, and this one.  The person who setup your original network may be able to help with this password and some routers will offer to reset the password after too many failed attempts (e.g.: Type the router's serial number, etc.).

A.  Identify a static (fixed) IP address on your home network/internal network, where the Raspberry Pi can live.  From your local workstation, discover your network's IP-address range with these geeky but easy steps:

From a DOS / Command prompt (windows-R, "CMD"), type this command:
ipconfig (enter)


* Note your IPV4 address, illustrated.

Yours will probably read something like
192.168.100.10    or
192.168.1.17        etc.

* Note the default Gateway's dot-1 address:  (mine is 192.168.100.1) - This is your main router.  The ".100" is the subnet.  If the secondary (wireless) router has an IP Address of .2, ignore it and do all the work on the (.1) address.

You an guess at an address in the same subnet, such as 192.168.100.151, where that last octect is above (say 151 and below 254). 

For example, my home network's internal IP addresses are 192.168.100.x. and I picked ".151".
(Subnets like 192.168.200.x,  192.168.1.x, 10.10.1.x are common.).  For that last octect, aim for a higher number, below 254.

Test if this address is free:  
From your workstation, launch a DOS prompt.  Type this command:

PING 192.168.100.151

If no device answers, this address is safe.  Otherwise, try a different near-by-number (.152, .153,  250, etc.).  Write this address down.

Technical notes for those who care:  Home routers assign automatic DHCP addresses to each workstation using a range (or pool) of numbers.  This range varies by manufacturer.  Some start at 2 - through 100, others start at 10 through 150.  The range does not matter, but the Raspberry Pi needs a number outside that range, where I am choosing .151 

You can research this by logging into the router.  Type the "x.x.x.1" IP Address in a browser URL.  A login panel appears.  Login as "admin" and use the password the router was installed with (sometimes printed on the back of the device), or as set in your long-lost documentation.  Once logged in, find an advanced setup menu for "LAN Setup" and "Use the Router as a "DHCP Server" (menus will vary).  Confirm the exact DHCP Address range it uses, then pick a number outside of that range.  



Continue with these software downloads:

B.  On your local workstation, download a small program called Putty.exe (a remote desktop console).

link: https://www.putty.org
Download and install.  Program will be used later to help administer the Pi.
Place a tile on the start menu or desktop so the program is easy-to-find.
Install, but don't worry about how to use this program yet.

C.  Although not needed for this project, on your local workstation, consider downloading and installing a remote desktop viewer called "VNC".  This can also used for remote administration - A remote desktop.

link: https://www.realvnc.com/en/connect/download/viewer/

Select Standalone EXE x64 Installer
Install, but don't worry about how to use this program yet.  Put an icon on the start menu.


-----------------------------------------------------------
Raspberry Pi Hardware Setup

-----------------------------------------------------------

A new Raspberry Pi is a small circuit board.  Snap the board into the kit's plastic case, and if the kit came with self-adhesive heat-sinks, apply them now.  This is all obvious as you assemble the box.

Next, download and install the Linux operating system:

1.  On your local workstation, plug in your SD-card Reader/adapter; if Windows offers to format the drive, skip.

2.  From a PC, Google "Raspberry Pi OS" download or this link

https://www.raspberrypi.com/software

Click the "Windows" download button and download "Imager.exe" (2023: Version 1.7.5)


3.  Locate the downloaded imager.exe program,
Right-Mouse-Click and choose "Run as Administrator"

Select the top Operating System,  Raspberry Pi OS (32-bit)"
Choose storage:  (selecting the 16/32GB card)

Click the gear/settings icon

[x] Set HostName:    pihole.local     (the cosmetic name of your server)
[x] Enable SSH        *Use password authentication
[x] Username:           Change from "pi" to "admini";  Set a new password
[x] Set your location (United States, etc.)

Click "Write"

Note the recommended change from the default "pi" admin user ID to a different user ID.  This is wise for security. 
*Write the UserID and Password down!

Once copied, close the program and eject the SD card.

4.  Insert the Micro-SD card into the Raspberry Pi's card-slot.
  • Remove the Micro-SD card from the SD-adapter
  • >Insert into the Raspberry Pi's card-slot
  • The SD-card installs "up-side-down," into the board's slot
  • Push until it clicks in place

5.  Connect an HDMI cable to a TV or Monitor  (I used my TV).
  • Connect a USB Keyboard to the Pi's USB ports (borrow from your desktop; can be wireless)
  • Connect a USB Mouse (can be wireless) 
     
  • For the initial setup, use either a wired or wireless connection.  Wired is preferred. 

    For a Wired RJ45:  If near the main router, connect an RJ45 network cable to any open port on any router.  (Do not plug into the router's "uplink" port; instead plug into one of the 4 or 8 port areas).   Connect the other end to the Pi's RJ45 port. 

    If Wireless, continue with the USB power supply step; later connect to the wireless network. 
  • Connect the USB 2.5a power-supply to the Pi.  (Any 2.5a micro-USB cell charger will work.  Usually supplied in a the Pi-kit.)

The Raspberry Pi will boot (visible on TV or monitor).  You may need to switch the INPUT to find the right HDMI port.

Black Screen:  I had troubles, when partway through the install, the TV said "signal not found."  The TV was routed through a stereo, and the stereo would go into power-save mode.  Rebooting the stereo returned the TV's Pi image.

---------------------------------------------------------------------------
Raspberry Pi Operating System Install
---------------------------------------------------------------------------

When the Pi first-time boots with the new SD-card installed, it arrives at the Linux desktop.  

6.  Set the Pi to a static (fixed) IP address (see above, e.g. 192.168.100.151).  

a.  From the desktop's top bar, far-right side, look for an up-down arrow icon.
     Right-click the icon
     Select "Wired and Wireless Settings"
     Select "Interface" and in the second pull-down, choose "Eth0" (Ethernet zero)

b.  In the IPV4 address, type "192.168.100.151" (no quotes)



c.  Close the window

Note:  You can use wireless for the initial setup.  If so, also set the wireless IP to this same address, but once this box goes into production, it really needs to be on a wired connection. 
 
d.  From the Raspberry menu, top left, reboot the server


7.  Login again.  From the desktop, launch the chromium browser and confirm you can see the Internet (open Google.com or something).


8.  Patch

In the top, left-side menu, near the Raspberry icon, locate the "terminal" icon.  This opens to a DOS-like screen.  Update the Pi's OS with these case-sensitive commands:

sudo apt-get update
sudo apt-get upgrade -y


Testing:

9.  Test Putty

From your Windows workstation, launch "putty.exe"
In the Host Name or IP Address, type "192.168.100.151"  (or "pihole.local")

You will be prompted, "This is a new connection. Do you trust it?"
Login using "admini" and the password  (case-sensitive)

This confirms the program works.  Close the Putty window.

10.  Optionally: Test VNC, if installed:

VNC is a remote desktop (vs a remote Terminal).

a) To enable VNC, login to the server using Putty and "admini" (see step above).
At the command prompt, type

sudo raspi-config

b) From the menu, select "Interface Options"; enable "VNC"
Press Esc a few times to save the changes, then close the terminal window

c)  From your local workstation, launch "VNC" (Real VNC)
Select the small link, "Use RealVNC without signing in"

(You do not need to register a new account or login this company's website to use VNC.)

d) In VNC's search bar, type "192.168.100.151" (or "pihole")
Accept "VNC Server not recognized..."
Confirm you can see the desktop.
Close VNC
VNC has a bug with their Windows client (as of 2023.09).  Connecting a second time always hangs.  To make a second connection, delete the .151 icon and rebuild each time needed.  Unclear how to resolve this problem

See this keyliner article for other thoughts about this bug:


Congratulations.  The Raspberry Pi's Operating System is installed and ready for the Pihole software.

The next step installs the Pi-Hole DNS Server software.


--------------------------------------------------------
Install Pi-Hole DNS Software
---------------------------------------------------------

Once the operating system is installed and patched, install the Pi-Hole software:

12.  Using either Putty.exe or the Raspberry Pi's top-menu, open the "Terminal Window" (command prompt)
  • Type this case-sensitive command. 
    Note the "-sSL" -- is case-sensitive. 
    Note the split-vertical bar:

    curl -sSL https://install.pi-hole.net | bash

13.  Answer these prompts:

"This installer will transform your device into a network-wide ad-blocker" 
tab for OK, press Enter
  • You may be prompted to: 
    Choose eth0 for the hard-wired port
    (This must be selected even if using wireless during the base install)
  • Accept Google (or OpenDNS) as the upstream DNS Provider
    I prefer Google,  knowing the Pi subscribes to the same lists as OpenDNS
     
  • Accept the default third-party list; tab for OK
  • Choose IPV4 (not IPV6) for the protocol
!!!  Important:  If prompted for a static IP address, follow these steps.  If you set a static IP address above, it will skip this section and jump to step 14.
  • -- tab to "No"       You do not want to use your current address as a static address
  • Press Enter 
  • If this step is missed, press ESC and restart at the CURL step.
(Reason:  Set a static, hard-coded IP-address on the wired network. 
Do not accept the suggested static address, as it is within the public pool and can be stolen by other devices.) 
  • Use a "Static" / fixed/hard-coded IP address from outside of your DHCP pool:

    recommended:   192.168.100.151    or
                               192.168.1.151    etc., depending on your subnet

  • Type the Pi-Hole's static IP address, plus an appended "/24"
    e.g. 
          192.168.100.151/24    (Note the appended "/24")

Set the Default "Gateway" to the same address as your workstation's Gateway IP Address.  This is in the same subnet, here illustrated as "100":
  • Typically:  192.168.100.1 

14.  When prompted for these options, accept all recommended settings:
  • [x] Install the Web Admin Interface
    [x] Install the Web Interface PHP modules

  • [x] Log Queries, ON  (recommended)

  • Select "Privacy Mode for FTL" = 0 Show Everything
(If the install goes "south," reboot the PI and restart the curl command.)


15.  At the "Installation Complete" message  (wait for this prompt), Step away from the keyboard

Write down the insufferable auto-generated "Administrative login/password" 
This is a dashboards "admin" userID and password and is different than the Pi's default "admini" userID!  This dashboard/admin password will be changed in a moment.

Dashboard Password example: 
192.168.100.151/admin    Password: xxxxxxx______________________

Click "OK"  (but wait on restarting)


16.  ! Change the Pi's dashboard admin password. 

Do this now, while it is easy to get to these screens:
From the main desktop, open a terminal window.

Type this command:
pihole -a -p

Follow the prompts to change the password.  I'd make it the same as the Pi's SSH password.
(It is unnerving, but the terminal does not show characters as you type a new password)

* Record this final password in a checklist.

The Pi-Hole hardware is now ready to use. A moment of self-congratulations is in order.  Wiring work, and router-steps are still required.

Write this stuff down!

PiHole Server IP Addr:
__________________ 
(recommended:  xxx.xxx.xxx.151)

PiHole Server Name:
pihole.local      

PiHole SSH Admin User ID:
admini_           
(or "pi" if not changed)

PiHole SSH Password:
__________________

PiHole Dashboard (web) Password:
(login at xxx.xxx.xxx.151/admin)
__________________ (not admini)

Router Address:
__________________
(usually xxx.xxx.xxx.1)

Router Login:
__________________
(usually admin / pwd)



--------------------------------------------------------------------
Final Pi (Production) Wiring Steps
--------------------------------------------------------------------

Using the top-Raspberry menu, shut-down the server.
Unplug the HDMI cable; The monitor is no longer needed.
Unplug the Keyboard and Mouse; these are no longer needed.

A.  Move the Raspberry Pi to a location near the main router. 

1.  Using a short Cat-5 Network cable, plug in the Raspberry into any available port on your router or other network hub.  The Pi does not have to be directly plugged into the main router; it can use a downstream router if you have more than one.  Important:  A hard-wired connection is required.

 -- any open port can be used, where there are groups of 4 or 8 network jacks.  Do not plug it into the up-link port (a lonely port, usually a different color - yellow):

2.  Plug-in the USB power-adapter
3.  Give the Pi a minute to boot and get settled.

Note the activity lights on the Pi's RJ45 network port.

Of vague interest: Here is my nifty home wiring closet, showing the white Raspberry Pi Hole, near the bottom.  A second Raspberry Pi hosts a webserver for www.keyliner.com.  Velcro is a wonderful thing.

B.  Initial Test: 

From your PC-workstation, open a DOS / Command Prompt and ping the Pi to see if it is on the network.  Type this command:

PING 192.168.100.151

Expect a reply in xx milliseconds.

Pi Administrative Dashboard Login
---------------------------------------------------------------------

Test the administrative dashboard login.
From any browser, type the Pi's IP address/admin:

192.168.100.151/admin 

On the left-nav, click Login, using the Pi's administrator dashboard password (see the pre-requisite checklist from above -- you did record this, didn't you?). 

The Dashboard displays.
There will not be activity until the Routers are changed.

 
-------------------------
Router Setup
--------------------------

The final step is to configure the router(s) to point to this new Domain Service. 
These changes are required in order to activate the Pi. 
This is a one-time setup.

For most households, the main DSL or or wireless router (the box with a ".1" IP address) is the one which needs to be changed (I don't think it would ever be the Cable Modem).  Some networks may have a larger, secondary wireless router, where all devices and hubs are connected.  If so, use that device.  See your prerequisite checklist.  


1.  Login to the main router's (.1) address (as tested in the per-requisites, above)

From any browser, type the main router's address in the URL address bar. 

For example:  192.168.100.1
                       192.168.1.1
                       192.168.0.1   (again, see the pre-requisite checklist)

This is not the Raspberry Pi's IP address.


2.  Login to the router with the (Netgear's) administrative account:  Sometimes named "admin" or "administrator", and the previously-recorded router password (not the Pi's administrative password).

The main setup screens vary by modem manufacturer.  Several example routers are illustrated next:
  • Usually under an Advanced Configuration menu
  • Look for a DNS Setup section
    (or sometimes DHCP/DNS)
  • Look for
    "Dynamic DNS" (or "Auto-DNS", or "use these DNS Servers", depending on modem)
     
  • Change to:
    either "Static DNS" or  "Use these DNS Servers"...
  • At the Primary DNS, type the IP address of the Raspberry Pi.

    For example, on my network, 192.168.100.151 
    (This is the Pi's IP Address; must be in the same subnet (.100))
  • Optionally (and not particularly recommended) type a Secondary DNS.

    If possible, leave the secondary DNS blank, or populate in this order:
    a) 192.168.100.151  (duplicating the Pi-hole's address -- use this when field is required.

    or b) 127.0.0.1  (a dummy address, or if you can, leave blank)

    or c)  208.67.222.222  -- OpenDNS - a third-party pi-hole-like DNS provider.  The Pi-Hole subscribes to this same address for some of its blocking.  Using this as a backup is not a bad idea.  See the end of this article for more details about OpenDNS.  But as before, if the Pi failed, you would never know and if the Pi failed, you would loose control over reporting, white lists, and black lists.  I do not do this myself, but for my Mom, I'd consider it....

    or d), with reservations, if you never want a DNS failure, tell the router to use Google's DNS:
    8.8.8.8

    I leave my secondary blank, as the Pi already defaults to Google for those addresses it does not want to resolve (this was part of the Pi-Hole's original install).  Reason:  If the Pi fails or crashes (which it has never done), I want the network DNS to shut-down and not find any alternate path.  If the Pi was offline, an the router's knew about 8.8.8.8, DNS would resolve and you would never see the Pi as offline -- you would never suspect a problem.  I want the network to fail.  See the end of this article for more discussion about this. 
3.  Important:  Save the changes by clicking this screen's SAVE or APPLY button. 

Do this before moving to any other screen. 
The router will reboot.


Other Example Modem Screens:

 Zyxcel DSL router looked/s like this:



* Some of newer models of routers require a secondary DNS  

If using Google's  8.8.8.8 as a secondary DNS, this is redundant because this is the go-to address used by the Pi.  In this router, which requires a secondary, I would type 127.0.0.1



* As another example, a typical older Linksys router looked like this:

* Another version of a linksys router looked like this.  In this illustration the secondary DNS was left empty, at 0.0.0.0 - which is fine by me:

* A NetGear Genie AC1450 looked like this, where the Raspberry Pi was the primary and again, Google's DNS was set as a secondary (not recommended):



  (end of examples)                            

Other Devices

Most computers and devices on the network (desktops, laptops, tablets, phones) automatically connect using DHCP.  They get their address and domain services from the router.  No other action is needed.  They automatically pass through the Pi.  This includes cellphones, tablets, etc.

** If you have a device with a hard-coded IP address, typically a printer or perhaps a TV, then manually set that device's IP Address, Subnet, and DNS.  Practically speaking, for the DNS, you could use the Raspberry Pi's address (e.g. 192.168.100.151) -- but these devices do not surf the web, and dumb humans are not doing anything strange on them.  In these cases, you could set their DNS directly to Google's 8.8.8.8.  I did this on my TV's and Printers, bypassing the Pi.



Raspberry Pi and Pi-hole configuration is complete!

I recommend the following tests and recommend logging into the Pi-hole's admin screens.  These topics are covered next.


TESTING

To properly test the new pi-hole, disable the workstation's locally-installed ad-blocking software. 

Reason: Ad-blockers also block traffic.  The difference is they block the traffic *after* it has downloaded where-as the Pi keeps them from ever downloading. 

You may or may not have ad-blocking software installed.  Look in your browser's Tools, Add-Ins menu and look for "adblock-plus" or "uBlock Origin" (the two most commonly used blockers).

Example: Blocking extensions in Firefox (Extension Manager)



If installed, look on your browser's upper-right menu bar, looking for a UBlock Origin or an Adblock-plus icon.  Click the icon and temporarily disable the ad-blocker.

In your ad-block testing, you will really see this while reading news articles (say on Google News).

When using the local network, Cell phones and tablet will often see an ugly "webpage not available" in the middle of the article -- this is an advertisement and likely that ad is recording your PC's IP address and other information.  The 'page not available' message is the Pi-hole at work, discarding the traffic.  This is good.



Each application or browser decides how to handle the error in its own fashion.  Many show empty white-space where the ad lived -- with no obvious errors. 

The neat thing about this is the vendor never knew you attempted the connection because it is blocked before the traffic left the house.  You won't be tracked, monitored, or recorded as you read articles, and big advertising graphics won't download.

Test 1: 

This test makes sure the network is functioning properly and you have the routers pointing to the right DNS-resolver (the Pi-hole).
  • From your normal workstation, browse to www.google.com.
  • If you arrive, the pi-hole / DNS is working correctly.
Test 2:
  • Browse to Yahoo.com
    (Update:  Yahoo has cleaned up their page since this article was written and most ads are now within their own yahoo.com domain and are not blocked by the pi-hole because Yahoo got wise to this and other similar ad-blockers.)

  • Note "holes" in the page -- blank spaces, illustrated below in orange.  There are being snuffed by the Pi-hole.   (some browsers treat this differently, not showing the holes).  Later, from the Pi-hole admin screens, you can temporarily disable the Pi, and can see what the page looks like, before-and-after.
     
  • Be sure adblockers are disabled or this test will be distorted.
     
  • Note "holes" in displayed page. These are never transmitted; speeding up page-loads.  The drawback is content providers cannot monetize their content.  There are moral and ethical considerations; see the end of this article for a discussion.  On the other hand, they are often abusive and can (accidentally) provide malicious content.
Click for larger view

Test 3:
  • Attempt to browse to  http://tag.bounceexchange.com - a nefarious site
  • Note how pi-hole blocks the address.  It may look like this or this, depending on your browser:



    or this:
  • or if you have an ad-blocker installed, and it were still active:
  • Browse to  "didtheyreadit.com"  (an email tracking service).  As-of this article, you will likely succeed and arrive at the site.   Consider "Blacklisting" this and other such sites.  See the blacklist later in this article.
     
  • Note that *all* devices in your network benefit from the Pi.  And, more importantly, none of the devices need to be told about the setup -- it just works.  But if your device (cell phone, tablet, laptop) strays from the network, the Pi's benefits are lost.

Side-notes:  If the domain is on the naughty-list, the Pi dumps the DNS request into a dark hole, hence "pi-hole."  As of this article, over 125,000 domains are in the discard list.  If the address is on the good-boy list, it is handed off to to your default (Google's) Domain Services.  (Google) resolves the address normally. 

Most home routers use your ISP's Domain Name Services, for example,SparkLight (cableOne), Century Link, Comcast, etc., and some ISPs have been known to slip-stream their own advertisements into your data-stream(!), replacing other ads with their own.  With the Pi-hole (or Google's DNS, 8.8.8.8), all DNS calls are resolved with a more trustworthy source.


Testing:  Simulate a pi-failure:

Unplug the Raspberry Pi's power and attempt to browse any site from any workstation. 
Assuming the routers have no secondary DNS, you will find Internet addresses do not resolve*.  In other words, the Pi is required to be online -- just like the router is required to be online. 

Restart the Pi and give it a few minutes.  Repeat the test, confirming the network returned to normal.

(* If a secondary DNS is in the router, traffic routes to the secondary address when the Pi is offline.  This is good and bad. 

The secondary resolves domain addresses, stopping Pi failures from being a complete network stoppage -- but you will not know the Pi is offline, and will lose the benefits of nefarious-site-blocking.  For this reason, I do not use the router's secondary DNS setting, substituting a dummy 127.0.0.1.)

Testing:  PiHole Administrative screens:

On the administrative screen (login to 192.168.100.151/admin)
Select "Recent Queries" to see a log of recent activity can be reviewed. 

You will be amazed that after a few random minutes of surfing, from any device in the network, the blocked DNS traffic.  Illustrated, my "smart" TV is busy on the network, playing Pandora, and I caught a Nest Thermostat checking on the daily weather.  This traffic was allowed to pass. 

Click for larger view

But "settings-win.data.microsoft.com" was blocked.  This is Microsoft collecting diagnostic data for the Consumer Windows Experience program; see link Infoworld article.  The Pi-hole team decided this was intrusive, and added this address to the blocked domain list.  From the admin panel, it could be white-listed with a click.


What happens under the hood:

When a device tries to resolve a blocked domain name, the DNS service drops the request in the hole and discards it.  The page does not even know a call was attempted.  No graphics, scripts, or other code runs from the discarded site.  Similarly, if a page has embedded code that reaches out to other (blocked) third party domains, those domains are dropped; the code will think no network was available.  This is a win-win for you.

Note: Some ads are now being hosted directly in the target's internal pages.  If the main site can be reached, those types of embedded ads will be allowed through.  This cannot be trapped by the Pi or by traditional ad-blockers. 


There are two other areas of particular interest:  White and Black lists.


White Lists:

For sites where you want to support advertising, such as the NewYork Times, allow them their ad-revenue by adding their domain to the Pi's white-list. 

See the dashboard's left-nav, "Domains".
When adding a domain, always select [x] Add domain as Wildcard
Choose "Add to WhiteList"  (or "Add to Black list")

If you decide to keep your browser's ad-blocking software installed, you will have to add the site to both the Pi-Hole's white list and your local ad-blocker's white-list.  

Regardless, add these domains to the Pi's whitelist:
When adding, recommend [x] Add domain as wildcard

+ nexus.officeapps.live.com    (Outlook; Media Player)
+ redire.metaservices.microsoft.com  (Windows Media Player)
+ d2avsp0hxzxate.cloudfront.net (something Apple needed)
+ statcounter.com (if you track your own site's traffic)
+ trackcmp.net



Black Lists:

Blacklists allow you to add your own bad domains to the pi-hole, even if the pi-hole's adlists are not blocking the traffic.  I manually added the following to my Pi-hole Black List -- especially the Email tracking addresses.


These are addresses I have discovered and they were not on the Pi's official lists (as of 2019.10).  Some sites are now on the Pi-block lists.  It does not hurt to explicitly add them.

See the dashboard's left-nav, "Domains".
When adding a domain, always select [x] Add domain as Wildcard
Choose "Add to WhiteList"  (or "Add to Black list")

Type each address manually (without a "*" wild-card symbol):   
but do check [x] Add Domain as wild-card; which builds a regex expression

When black-listing; always add as a "WildCard":

123banners.com
l90.com
adblade.com        #clickbait
adforce.com
adsnative.com      #clickbait
advertising.com
agkn.com
appnet.com

assia-inc.com      #seems to be a tracking site
avenuea.com

babator.com
bananatag.com      #email tracking
bluekai.com
bluestreak.com
burstmedia.com
burstnet.com

cdn.nmcdn.us       #click bait
cirrusinsight.com  #email tracking
clearslide.com     #email tracking
clipix.com

cloudcheck.net    
contactmonkey.com  #email tracking
content-ad.net
content.ad
crownpeak         
demdex.net
deskun.com         #email tracking
didtheyreadit.com  #email tracking
doubleclick.com
doubleclick.net
dynamicyield.com

engage.com

evidon.com        
exelator.com
extreme-dm.com
fastclick.net
filepicker.io
g2crowd.com       #email tracking iko system also velocify
getnotify.com     #email tracking
gigya.com
gmelius.com       #email tracking
gobankingrates.com
go.com
hubspot.com       #email tracking
icanbuy.com
imgis.com
imrworldwide.com
intelliverse.com  #email tracking
keywee.co         #Note the .co, not .com

livehive.com      #email tracking
mail-track.com    #email tracking
minute.ly
newtonmail.com    #email tracking
nr-data.net
optimizely.com
outbrain.com
outreach.com      #email tracking
pagefair.com
pixelsite.info    #email tracking
production.kabutoservices.com   #Remote Control 2023.01
pubexchange.com
quantserve.com

remail.com        #email tracking
remail.io         #email tracking
remail.net
rlcdn.com
rocketbolt.com    #email tracking
ru4.com
salesloft.com     #email tracking
sidekick.com      #email tracking, now hubspot
saleshandy.com    #email tracking
scorecardresearch.com
stats.net
streak.comp
swallow.apple.com  #added 2023.01 - large traffic; svc not used
sync.optimatic.com

taboola.com
teknosurf.com
tinypass.com
toutapp.com       #email tracking
tru.am
valueclick.com
velocify.com      #email tracking Velocity Pulse
voicefive.com
websidestory.com
w55c.net
yesware.com       #email tracking



Subscription Additional Recommendations:

As of mid 2021, the default pi-hole installs with this subscription list, seen during the base-install:
60,000 items.  See Group Management, AdLists:

https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts

Keyliner also recommends adding other lists:

https://raw.githubusercontent.com/badmojr/1Hosts/master/mini/hosts.txt
https://raw.githubusercontent.com/Goooler/1024_hosts/master/hosts
https://blocklistproject.github.io/Lists/tracking.txt 


Click this link to view other community-supported lists. 

filterlists.com


Some of these lists are specific ("Click Bait sites in Japan").  Some are too short to bother subscribing and if interested, simply add the list manually to your own black list.   For compatible lists, look in the Software column for the Raspberry Pi icon (see illustration directly below). 

To add other lists:

a.  Browse filterlists.com;
b.  When a list is found of interest, click right-nav "View" to examine 
c.  Copy the URL of a list to the clipboard.
d.  Launch the Pi-Hole admin console (192.168.100.5/admin, etc.)
e.  Select Left-Nav "Group Management"
f.  Select AddLists
g.  Paste the URL; click Add

Once all lists of interest are added:

h.  In the pi-hole admin console, Tools, click "update Gravity"
     From the Dashboard, note the new number of Domains on Block List


De-Installing Pi-Hole:

To temporarily disable the pi-hole (for testing a blocked site):

- From the admin screen, left-nav, click "Disable"  

When disabled, all requests pass through to (Google's) DNS service, and all Pi-protection is lost.  Note: This is not your router's secondary DNS setting, this is the Pi-Hole's default DNS resolver.

To permanently remove the Pi-hole from the network:

- Login to your router's .1 address
- Change the Static DNS field

from (192.168.100.151)
to Google's DNS: 8.8.8.8

A worse choice would be to return the Routers to "Auto-DNS" -- this would put you at the ISP's mercy.

Click "Save".  The router will reboot.

Once changed, the Pi can be unplugged and removed from the network.  No workstations or other devices need to be told of this change.


Known Problems:

Some sites, especially those that show the "top 100 celebrity before and after photos" will be blocked.  Reason: These are trolling sites, with obtrusive ads and with possible fly-by installs.  These sites were deemed dangerous, and were blocked by the Pi-hole community.  Trust their decision.


Sadly, every other type of web failure will be blamed on the Pi. 

My experience is the Pi has not been wrong, but the family will blame the Pi for all network problems.

To test if the Pi is causing a problem with a site, use the admin screens to temporarily disable the Pi-hole  (see left-nav, Disable).  Re-test the site or page in question (see side-illustration, directly above).  View pi-hole logs to see which sites were blocked and which were allowed.  If the pi-hole inappropriately blocked a site, use the logs to find the address, then add that site to the pi-hole's white-list.

If the pi is temporarily disabled and the site still malfunctions, then the Pi is innocent.  The Pi does not interfere with non-blocked sites

But, if the Pi blocks the site, it has a good reason for doing so.  If you trust the blocked site, and insist on arriving (overriding thousands of volunteer's opinions), add the domain to the white list. 


Updating Pi-Hole Software
----------------------------------------------------
With an SSH Console login (setup during the initial install)

A.  Login with either Putty or DOS SSH
      With Putty, login to "pihole" (or 192.168.100.151)
       UserID:  "admini" (or userID "pi" if you did not change the default during install)

       With DOS, type "SSH.exe admini@192.168.100.151"   (no quotes)

B.  At the "admini@pihole:~ $" prompt, update with this command:
      type pihole -up 

      Once or twice a year, I recommend doing an update.


Alternatives to the Pi-Hole:

---------------------------------------------------------------------------
The Pi is an admittedly complicated project and it involves machinery.  

An alternative is to use OpenDNS's IP addresses as your router's primary and secondary DNS.   OpenDNS subscribes to some of the same black-lists the Pi-hole uses and blocks similar phishing, ad, and malware sites.  (Update:  OpenDNS was purchased by Cisco, and has since been renamed to "Cisco Umbrella".  According to Wikipedia, "Cisco intends to continue development".)

Instead of the pi-hole, use OpenDNS with these similar steps:
-  Login to your .1 routers, as described above.
- Change the primary and secondary DNS address from your ISP's address, or from the pi-hole's 100.151 address, to OpenDNS's addresses:  

Primary:  208.67.222.222 
Secondary 208.67.220.220

These addresses are also an option on the Pi-Hole installation screen, where you can pick OpenDNS, or Google's 8.8.8.8 address as the downstream DNS name resolver.  Doing this is somewhat redundant.  But the Pi is a tad faster, and has white-listing and blacklisting overrides, which OpenDNS does not provide without costs.  See the comments at the end of this article for more discussion about this.

They have a different version, called "FamilyShield", which uses these addresses to block all of the above, plus Porn, and Proxy Servers.
  • 208.67.222.123
  • 208.67.220.123
 

Pi-hole and Ad-blocker Ethics
------------------------------------------------------------------------
A word about publishers who use revenue to keep producing content.  Ad-blocking, and the Pi-hole, cut into these revenue streams. But the current model of using third-parties to display ads is broken.  We might want to let the New York Times broadcast ads, but the ad-sites are often third-parties, and are sometimes untrustworthy.  

Many publishers detect ad-blockers and refuse to display content.  The Pi-hole can sometimes dance around that restriction.  In other words, disable the locally-installed ad-blocker, and let the pi-hole sink the DNS requests undetected.

The industry has matured and many publishers now host the ads on their own sites, their own domains.  If the NewYorkTimes domain is blocked for the ads, it also blocks the content, and the pi-hole and all ad-blockers will be ineffective. 

The other side of this argument is obvious:  Publishers and Advertisers have abused ads by displaying full-screen, non-dismissable ads, and by breaking articles into dozens of pages for ad-impression rates.  These are why the pi-hole exists and these sites deserve to be blocked.



-----------------------------

Related Keyliner Articles:
This is the way I used to do this -- manually blocking about 50 high-volume sites.  With this article, I now block 125,000 sites!
Stopping Tracking Cookies with whack-a-mole - blocking DNS using Acrylic DNS.

Learn more about the pi-hole project here:
https://pi-hole.net/2018/01/11/pi-hole-is-open-source-consume-contribute-or-both/#more-9734
and
https://pi-hole.net/2017/05/12/seven-things-you-may-not-know-about-pi-hole


Related Thoughts:
Some routers run Linux under the hood and can be re-programmed to run a pi-hole directly on the router.  After reading this article, https://www.ab-solution.info, my co-worker tried this.  The router seems to be the best place to run this type of process, but not all routers can be re-programmed.  Doing this takes skills which are more simply done on a Pi-hole.  Ultimately, he reported back (unspecified) troubles, and abandoned the idea, returning to the Pi-hole.


Your comments:
I would like to hear your comments on this project.
If you like the Pi-hole, donate a few dollars their way; they deserve the support.  See the admin-login screens.


History:
2017.11  Initial Install
2018.03  Pi-hole intercepted an email phishing site. Saved the checkbook.

2019.01  Rewritten and updated this article. Donated to PiHole project.
2019.10  Improved grammar; an editor's work is never done.  
2020.02  More editing.  Dang, this article was hard!
2021.02  More editing.  Dang, this article is still hard.
2021.02  Donated to PiHole project.  Running 4 years, no issues.
2023.09  Six years. Zero problems.  Not even a hiccup. Amazing.  Noticed a heatsink fell off.

----------------------------
Now obsolete notes:
Some versions of the Raspberry Pi operating system have other pre-installed software; often these are called "Noob" installs.  Much of the software that comes with these 'distros' can be removed with these steps:

Optional Cleanup Steps:

The operating system comes pre-installed with extra software that is not needed for this project.  
 
The Raspberry Pi and Pi-hole software will run as-is, but if you are geek, and don't mind spending another half-hour, consider uninstalling the following programs.  This will make the Pi faster, and leaves more space on the drive for logs and updates:

From the main Linux desktop, top-menu, open a Terminal Window.
Type these commands, pressing ENTER after each.  
Answer with "Y" (capital Y), when prompted.

If software is not found, press the up-arrow and double-check the spelling, or move to the next command.  Note: In many of these packages are no longer installed in some versions of Linux.  If not found, don't panic and move the next one.

a.   sudo apt-get purge wolfram-engine
b.   sudo apt-get remove --purge libreoffice* 
c.   sudo apt-get purge sonic-pi
d.   sudo apt-get purge scratch

e.   sudo apt-get purge greenfoot
f.   sudo apt-get purge geany
g.   sudo apt-get purge nuscratch
h.   sudo apt-get purge python-pygame
i.   sudo apt-get purge pygame

j.   sudo apt-get purge squeak-vm
k.   sudo apt-get purge dillo
 
l.   sudo apt-get purge minecraft-pi
m.   sudo apt-get purge penguinspuzzle
n.   sudo apt-get purge oracle-java8-jkk
o.   sudo apt-get purge oracle-java7-jdk
p.   sudo apt-get purge openjdk-8-jre

a.  If one or more of the above were successfully de-installed:
- use the top-menu to reboot the Pi.
 
b.  After the reboot, open the Terminal Window.
     Then issue these commands:

sudo apt-get clean
sudo apt-get autoremove --purge


11.  Update the OS with this long command:
       (The command will inform you if it is up-to-date)

sudo apt-get update
sudo apt-get upgrade -y


Optional software is now de-installed.  
Approximately 2G of disk space is freed.

 

 

3 comments:

  1. Anonymous2/06/2018

    The author comments: A friend complained that a site, "The 10 best ways to...." would not load, being blocked by the Pi.

    I said hurray! This was a trolling site, whose sole purpose was to rack-up page-views by showing one sentence at a time, and making you click 'Next" 20 times. Blocked by design.

    ReplyDelete
  2. Reader wrote: Did I see OpenDNS offers the same basic services as the pi-Hole (ad-blocking, tracking, malware-site-blocking)? Why pick the pi-hole over that design?

    I replied:
    The Pi-hole subscribes to the same list as Open DNS, plus a few others. Using OpenDNS is an easier, less technical solution than the Pi-hole. It would be redundant to configure the Pi-hole to use OpenDNS as the final DNS resolution.

    In this case, the pi benefits are marginal and may or may not be important to you. First, the initial DNS traffic stays in your local network (but ultimately approved traffic is resolved by the upstream DNS. I use Google's 8.8.8.8. Reason, I do not want my ISP to resolve DNS.)

    The Pi's other remaining benefit, which I often use, is control of my own whitelist and blacklists. Although OpenDNS has over 100,000 blacklists, it is cautious about what addresses are listed. I find things, such as email trackers, that I like to block.

    Finally, I occasionally review pihole logs, finding all kinds of interesting things that are not visible with OpenDNS. It lets me know shenanigans with all kinds of vendor software, where I catch them linking to third-party entities.

    ReplyDelete
  3. I updated the article, discussing Alternatives to Pi-Hole.

    ReplyDelete

Comments are moderated and published upon review. (As an aside, not a single spam has been allowed through; why bother?)