Google do not provide a list of IPs to identify their bots, so you can’t simply add that to fail2ban’s ‘ignoreip =’ line.
Instead, according to their answer per https://support.google.com/webmasters/answer/80553?hl=en you can only verify the bot’s provenance by checking the DNS for the bot’s IP. In fact, they ask you to run 2 queries (1 reverse and 1 forward lookup) to make sure that the IP is not spoofed.
My simple 1 reverse lookup script is:
#!/bin/bash
IP="$1"
HOSTRESULT="$(host -W ${IP})" # updated thanks to comment from Martin
REGEX='.*(googlebot\.com\.|google\.com\.)
And add that to /etc/fail2ban/jail.local
ignorecommand = /usr/local/bin/ignore_ip_check.sh
This needs more testing, and I should add the second forward lookup, for for now it seems to do the trick
if [[ "$HOSTRESULT" =~ $REGEX ]]; then exit 0; else exit 1; fi
And add that to /etc/fail2ban/jail.local
This needs more testing, and I should add the second forward lookup, for for now it seems to do the trick