Google do not provide a list of IPs to identify their bots, so you can’t simply add that to fail2ban’s ‘ignoreip =’ line.
Instead, according to their answer per https://support.google.com/webmasters/answer/80553?hl=en you can only verify the bot’s provenance by checking the DNS for the bot’s IP. In fact, they ask you to run 2 queries (1 reverse and 1 forward lookup) to make sure that the IP is not spoofed.
My simple 1 reverse lookup script is:
#!/bin/bash IP="$1" HOSTRESULT="$(host -W ${IP})" # updated thanks to comment from Martin REGEX='.*(googlebot\.com\.|google\.com\.)
And add that to /etc/fail2ban/jail.local
ignorecommand = /usr/local/bin/ignore_ip_check.sh
This needs more testing, and I should add the second forward lookup, for for now it seems to do the trick
if [[ "$HOSTRESULT" =~ $REGEX ]]; then exit 0; else exit 1; fi
And add that to /etc/fail2ban/jail.local
This needs more testing, and I should add the second forward lookup, for for now it seems to do the trick
5 Comments
This is very helpful, I recommend to use
host -W 1 ${IP}
That limits the rDNS to about 4-6 seconds (strangely). As otherwise it might get stuck for 15 seconds trying to resolve some IP which doesn’t have rDNS.
Get error with that script
2019-11-03 18:27:17,655 fail2ban.action [27943]: ERROR /usr/local/bin/ignore_ip_check.sh 46.101.121.139 — stdout: b”
2019-11-03 18:27:17,655 fail2ban.action [27943]: ERROR /usr/local/bin/ignore_ip_check.sh 46.101.121.139 — stderr: b”
2019-11-03 18:27:17,657 fail2ban.action [27943]: ERROR /usr/local/bin/ignore_ip_check.sh 46.101.121.139 — returned 1
2019-11-03 18:27:17,868 fail2ban.action [27943]: ERROR /usr/local/bin/ignore_ip_check.sh 46.101.157.225 — stdout: b”
2019-11-03 18:27:17,869 fail2ban.action [27943]: ERROR /usr/local/bin/ignore_ip_check.sh 46.101.157.225 — stderr: b”
2019-11-03 18:27:17,870 fail2ban.action [27943]: ERROR /usr/local/bin/ignore_ip_check.sh 46.101.157.225 — returned 1
the script seems to work as expected if I use the command in terminal
Any idea ?
Ubuntu , fail2ban 0.9.3, postfix
Check you have the latest version of fail2ban? Ubuntu doesn’t always have the latest in the LTS repos.
At the end of the “ignorecommand” the “” is missing
At the end of the “ignorecommand” the “ip” is missing between a less-than and greater-than sign.
BTW: WordPress swallows the characters.