How NOT to ban Googlebot

Google do not provide a list of IPs to identify their bots, so you can’t simply add that to fail2ban’s ‘ignoreip =’ line.

Instead, according to their answer per https://support.google.com/webmasters/answer/80553?hl=en you can only verify the bot’s provenance by checking the DNS for the bot’s IP. In fact, they ask you to run 2 queries (1 reverse and 1 forward lookup) to make sure that the IP is not spoofed.

My simple 1 reverse lookup script is:

#!/bin/bash
IP="$1"
HOSTRESULT="$(host -W ${IP})" # updated thanks to comment from Martin
REGEX='.*(googlebot\.com\.|google\.com\.)

And add that to /etc/fail2ban/jail.local

ignorecommand = /usr/local/bin/ignore_ip_check.sh 

This needs more testing, and I should add the second forward lookup, for for now it seems to do the trick

if [[ "$HOSTRESULT" =~ $REGEX ]]; then exit 0; else exit 1; fi

And add that to /etc/fail2ban/jail.local


This needs more testing, and I should add the second forward lookup, for for now it seems to do the trick

Similar Posts:




3 Comments

This is very helpful, I recommend to use

host -W 1 ${IP}

That limits the rDNS to about 4-6 seconds (strangely). As otherwise it might get stuck for 15 seconds trying to resolve some IP which doesn’t have rDNS.

Get error with that script

2019-11-03 18:27:17,655 fail2ban.action [27943]: ERROR /usr/local/bin/ignore_ip_check.sh 46.101.121.139 — stdout: b”
2019-11-03 18:27:17,655 fail2ban.action [27943]: ERROR /usr/local/bin/ignore_ip_check.sh 46.101.121.139 — stderr: b”
2019-11-03 18:27:17,657 fail2ban.action [27943]: ERROR /usr/local/bin/ignore_ip_check.sh 46.101.121.139 — returned 1
2019-11-03 18:27:17,868 fail2ban.action [27943]: ERROR /usr/local/bin/ignore_ip_check.sh 46.101.157.225 — stdout: b”
2019-11-03 18:27:17,869 fail2ban.action [27943]: ERROR /usr/local/bin/ignore_ip_check.sh 46.101.157.225 — stderr: b”
2019-11-03 18:27:17,870 fail2ban.action [27943]: ERROR /usr/local/bin/ignore_ip_check.sh 46.101.157.225 — returned 1

the script seems to work as expected if I use the command in terminal
Any idea ?
Ubuntu , fail2ban 0.9.3, postfix

Leave a Reply

Your email address will not be published. Required fields are marked *