WikiDevi.Wi-Cat.RU:Tomato/Clean, Lean and Mean Adblocking

From WikiDevi.Wi-Cat.RU
Jump to navigation Jump to search
Source
@ LinksysInfo.org
by haarp

I've been looking into blocking ads on Tomato firmwares. There are already scripts for this, but they did not suit my needs.

xcooling's script is outdated, uses very ugly and slow code and is on top of that buggy.
ALL-U-NEED Ad Blocking is very cumbersome through their decoder, bloated, unreadable and again, slow.

Both also don't employ the method I envisioned. So I wrote my own script, took ideas from xcooling's script and this DD-WRT script, improved it, combined it with my own sh expertise and optimized the heck out of it.

Features

  • Takes public blocklists for known ad hosts and redirects them via DNS poisoning
  • pixelserv (optional) through a second IP (Web GUI on port 80 still works!)

-> e.g. router on 192.168.0.1, but also responds as 192.168.0.254 with pixelserv
-> this serves transparent pixels instead of causing error messages on blocked ads

  • Does not interfere with normal dnsmasq operation

-> does not try to "optimize" it (that's what the "Custom configuration" box on the web GUI is for, people!)
-> does not break Tomato's ability to restart dnsmasq should it crash

  • Additional blocklist sources can easily be added
  • Easy blacklist and whitelist
  • Very optimized: Updates as quickly and with as little CPU/memory usage as possible
  • Small and lean: Only does what it needs to do, then gets out of the way
  • Readable code

Instructions

If you're using Adblock pre-v4.0, please remove it completely from your router and reboot it first! Note: If the paste procedure fails for you, try transferring the corresponding file manually using scp, WinSCP or something (the DD-WRT link up there has some WinSCP usage examples)

  • Verify that your Tomato supports custom dnsmasq configs (i.e. shows this line under Advanced->DHCP/DNS: "Note: The file /etc/dnsmasq.custom is also added to the end of Dnsmasq's configuration file if it exists.")
  • Set up some kind of non-volatile storage. This is up to you, options are JFFS, CIFS, SD card, USB and possibly more. Note the path.

-> The simplest would probably be JFFS. Check this link -> I recommend other storage methods however, as JFFS is very limited in size (depends on which filter lists you'll ultimately use of course)

  • Designate a directory on your storage for adblock, e.g. /jffs/adblock/ (as seen by the router). Avoid spaces! This is the PREFIX.
  • Install pixelserv v30 if desired (thread). Take the entire chunk of script in this link, adjust PREFIX at the top, paste it into the box on Tools->System and press Execute. This is also how you can update pixelserv in the future.

If this fails, extract the pixelserv binary from this link and manually transfer it to PREFIX/pixelserv

  • Install adblock.sh v4.1. Take the entire chunk of script in this link, adjust PREFIX at the top, paste it into the box on Tools->System and press Execute. This is also how you can update Adblock in the future.

If this fails, paste the script from this link into a file and manually transfer it to PREFIX/adblock.sh

  • Install the config file. It will become PREFIX/config. Take the script below, adjust the config to your tastes, paste it into the box on Tools->System and press Execute. That's also how you can change the configuration in the future.

Config File

PREFIX="/cifs1/adblock/" ## adjust this!

echo '
### Settings ###
PIXEL_IP="254" ## 0: disable pixelserv
## 1-254: last octet of IP to run pixelserv on (default=254)
PIXEL_OPTS="" ## additional options for pixelserv
BRIDGE="br0" ## bridge interface for pixelserv (default=br0)
RAMLIST="0" ## keep blocklist in RAM (e.g. for small JFFS) (default=0)
CONF="/etc/dnsmasq.custom" ## dnsmasq custom config (must be sourced by dnsmasq!)


### Sources (uncomment desired blocklists) [must be compatible to the hosts file format!] ###
## MVPS HOSTS (~600k) [default]:
SOURCES="$SOURCES http://winhelp2002.mvps.org/hosts.txt"
## pgl.yoyo.org (~70k) [default]:
SOURCES="$SOURCES http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&mimetype=plaintext"
## Hosts File Project (~3M!):
#SOURCES="$SOURCES http://hostsfile.mine.nu/Hosts"
## The Cameleon Project (~600k):
#SOURCES="$SOURCES http://sysctl.org/cameleon/hosts"
## AdAway mobile ads (~20k):
#SOURCES="$SOURCES http://adaway.sufficientlysecure.org/hosts.txt"
## hpHosts ad/tracking servers (~400k):
#SOURCES="$SOURCES http://hosts-file.net/ad_servers.asp"
## hpHosts ad/tracking/malicious servers (~6M! replaces hpHosts ad/tracking list):
#SOURCES="$SOURCES http://hosts-file.net/download/hosts.txt, http://hosts-file.net/hphosts-partial.asp"
## MalwareDomainList.com (~40k):
SOURCES="$SOURCES http://www.malwaredomainlist.com/hostslist/hosts.txt"


### Blacklist additional sites ###
## (add hostnames inside the quotes, space-separated, without http://) ##
BLACKLIST=""

### Whitelist sites from blocking ###
## (add hostnames inside the quotes, space-separated, without http://) ##
WHITELIST="de.ign.com followerscounter.com redirectingat.com"
' > "$PREFIX/config" && echo success
  • Ready for the first run! Paste this into the box on Tools->System, edit the PREFIX and press Execute: /YOUR/PREFIX/HERE/adblock.sh
  • The script also accepts a few command-line options:

-> /YOUR/PREFIX/HERE/adblock.sh - default, update and enable adblocker
-> /YOUR/PREFIX/HERE/adblock.sh force - force updating of filters, even if not outdated
-> /YOUR/PREFIX/HERE/adblock.sh stop - disable the adblocker
-> /YOUR/PREFIX/HERE/adblock.sh toggle - disable the adblocker if active, enable if inactive (perfect for the SES button on your WRT54G!)
-> /YOUR/PREFIX/HERE/adblock.sh restart - restart adblocker (e.g. for config changes, script updates)

  • If you want to enable Adblock automatically when the router boots, just put /YOUR/PREFIX/HERE/adblock.sh into the WAN Up section on Administration->Scripts
  • If you want to have Adblock automatically update its filters, just put /YOUR/PREFIX/HERE/adblock.sh into one of the Custom commands on Administration->Scheduler
  • Have fun! :)

Notes

  • This script was tested with TomatoUSB v1.28.8754 ND vpn3.6 on a WRT54GL.
  • Remember that the script, pixelserv and all data reside on PREFIX. If you have other means of accessing that storage, those will probably be more convenient than pasting into boxes on the web GUI.
  • The script will automatically block anything bound for the pixelserv IP that is not intended for pixelserv itself.
  • Subsequent updates will only fetch blocklists that have changed, this however only works when the source server runs http on port 80.
  • You will need 2x-3x as much non-volatile storage as all filters combined. If your storage is too small, set RAMLIST to 1. Obviously, you now need enough free RAM to hold the filters instead! Additionally, the script will now have to redownload the filters when the router reboots.
  • If you change settings or update the script, please run the script with the restart option afterwards!
  • I've you're having problems, check the router logs!

Changelog

  • 3.3 - Fixed small issue with IPv6 present
  • 3.4 - Added bridge interface selector and minor changes
  • 3.5 - Prevent multiple instances, block everything but pixelserv on pixel IP, add Malware source and minor changes
  • 3.6 - Now checks if blocklists are outdated
  • 3.6.1 - Added "force" option to ignore Last-Modified headers
  • 3.7 - Added "bigmem" mode
  • 4.0 - Fundamental changes, see this thread
  • 4.1 - Minor changes, more checks

Adblock v4.1

#!/bin/sh
## Clean, Lean and Mean Adblock v4.1 by haarp
##
## Options:
## 'force': force updating sources,
## 'stop': disable Adblock, 'toggle': quickly toggle Adblock on and off
## 'restart': restart Adblock (e.g. for config changes)

## TODO: 'clean' option

alias elog='logger -t ADBLOCK -s'
alias iptables='/usr/sbin/iptables'

pidfile=/var/run/adblock.pid
kill -0 $(cat $pidfile 2>/dev/null) &>/dev/null && {
	elog "Another instance found ($pidfile), exiting!"
	exit 1
}
echo $$ > $pidfile

pexit() {
	elog "Exiting"
	rm $pidfile
	exit $@
}

stop() {
	elog "Stopping"
	rm "$CONF" &>/dev/null

	iptables -D INPUT -i $BRIDGE -p tcp -d $redirip --dport 80 -j ACCEPT &>/dev/null
	killall pixelserv &>/dev/null
	ifconfig $BRIDGE:1 down &>/dev/null
	iptables -D INPUT -i $BRIDGE -p all -d $redirip -j DROP &>/dev/null

	elog "Done, restarting dnsmasq"
	service dnsmasq restart
}

grabsource() {
	local host=$(echo $1 | awk -F"/" '{print $3}')
	local path=$(echo $1 | awk -F"/" '{print substr($0, index($0,$4))}')
	local lastmod=$(echo -e "HEAD /$path HTTP/1.1\r\nHost: $host\r\n" | nc $host 80 | tr -d '\r' | grep "Last-Modified")

	local lmfile="$listprefix/lastmod-$(echo $1 | md5sum | cut -c 1-8)"
	local sourcefile="$listprefix/source-$(echo $1 | md5sum | cut -c 1-8)"

	[ "$force" != "1" -a -e "$sourcefile" -a -n "$lastmod" -a "$lastmod" == "$(cat "$lmfile" 2>/dev/null)" ] && {
		elog "Unchanged: $1 ($lastmod)"
		echo 2 >>"$listprefix/status"
		return 2
	}

	(
		if wget $1 -O -; then
			[ -n "$lastmod" ] && echo "$lastmod" > "$lmfile"
			echo 0 >>"$listprefix/status"
		else
			elog "Failed: $1"
			echo 1 >>"$listprefix/status"
		fi
	) | tr -d "\r" | sed -e '/^[[:alnum:]:]/!d' | awk '{print $2}' | sed -e '/^localhost$/d' > "$sourcefile"
}

confgen() {
	elog "Generating $listprefix/blocklist"
	rm "$CONF" &>/dev/null

	sort -u "$listprefix"/source-* > "$listprefix/blocklist"
	for b in $BLACKLIST; do
		echo "$b" >> "$listprefix/blocklist"
	done
	for w in $WHITELIST; do
		sed -i -e "/$w/d" "$listprefix/blocklist"
	done
	sed -i -e '/^$/d' -e "s:^:address=/:" -e "s:$:/$redirip:" "$listprefix/blocklist"

	elog "Config generated, $(wc -l < "$listprefix/blocklist") unique hosts to block"
}

prefix="$(cd "$(dirname "$0")" && pwd)"
[ -e "$prefix/config" ] || {
	elog "$prefix/config not found!"
	pexit 11
}
source "$prefix/config"

if [ "$RAMLIST" == "1" ]; then
	listprefix="/var/lib/adblock"
	[ -d "$listprefix" ] || mkdir "$listprefix"
else
	listprefix="$prefix"
fi
if [ "$PIXEL_IP" == "0" ]; then
	redirip="0.0.0.0"
else
	[ -x "$prefix/pixelserv" ] || {
		elog "$prefix/pixelserv not found/executable!"
		pexit 10
	}
	redirip=$(ifconfig $BRIDGE | awk '/inet addr/{print $3}' | awk -F":" '{print $2}' | sed -e "s/255/$PIXEL_IP/")
fi


case "$1" in
	restart) stop;;
	stop)	stop; pexit 0;;
	toggle)	[ -e "$CONF" ] && { stop; pexit 0; };;
	force)	force="1";;
	"")	:;;
	*)	elog "'$1' not understood!"; pexit 1;;
esac


elog "Download starting"
until ping -q -c1 google.com >/dev/null; do
	elog "Waiting for connectivity..."
	sleep 30
done

trap 'elog "Signal received, cancelling"; rm "$listprefix"/source-* &>/dev/null; rm "$listprefix"/lastmod-* &>/dev/null; pexit 130' SIGQUIT SIGINT SIGTERM SIGHUP

echo -n "" >"$listprefix/status"
for s in $SOURCES; do
	grabsource $s &
done
wait

while read ret; do
	case "$ret" in
		0)	downloaded=1;;
		1)	failed=1;;
		2)	unchanged=1;;
	esac
done < "$listprefix/status"
rm "$listprefix/status"

trap - SIGQUIT SIGINT SIGTERM SIGHUP

if [ "$downloaded" == "1" ]; then
	elog "Downloaded"
	confgen
elif [ "$unchanged" == "1" ]; then
	elog "Filters unchanged"
	if [ ! -e "$listprefix/blocklist" ]; then
		confgen
	elif [ ! -e "$CONF" ]; then #needlink
		:
	else pexit 2
	fi
else
	elog "Download failed"
	if [ -e "$listprefix/blocklist" -a ! -e "$CONF" ]; then #needlink
		:
	else pexit 3
	fi
fi

ln -s "$listprefix/blocklist" "$CONF" #dnsmasq ignores broken symlinks :)

if [ "$PIXEL_IP" != "0" ]; then
	if ps | grep -v grep | grep -q "$prefix/pixelserv $redirip"; then
		elog "pixelserv already running, skipping"
	else
		elog "Setting up pixelserv on $redirip"

		iptables -vL INPUT | grep -q "$BRIDGE.*$redirip *tcp dpt:www" || {
			iptables -I INPUT -i $BRIDGE -p all -d $redirip -j DROP
			iptables -I INPUT -i $BRIDGE -p tcp -d $redirip --dport 80 -j ACCEPT
		}
		ifconfig $BRIDGE:1 $redirip up
		"$prefix/pixelserv" $redirip $PIXEL_OPTS
	fi
fi

elog "Done, restarting dnsmasq"
service dnsmasq restart

pexit 0
Retrieved from "https://wikidevi.wi-cat.ru/index.php?title=WikiDevi.Wi-Cat.RU:Tomato/Clean,_Lean_and_Mean_Adblocking&oldid=347886"