The Lost Notebook

Google do not provide a list of IPs to identify their bots, so you can’t simply add that to fail2ban’s ‘ignoreip =’ line.

Instead, according to their answer per https://support.google.com/webmasters/answer/80553?hl=en you can only verify the bot’s provenance by checking the DNS for the bot’s IP. In fact, they ask you to run 2 queries (1 reverse and 1 forward lookup) to make sure that the IP is not spoofed.

My simple 1 reverse lookup script is:

#!/bin/bash
IP="$1"
HOSTRESULT="$(host -W ${IP})" # updated thanks to comment from Martin
REGEX='.*(googlebot\.com\.|google\.com\.)

And add that to /etc/fail2ban/jail.local

ignorecommand = /usr/local/bin/ignore_ip_check.sh 

This needs more testing, and I should add the second forward lookup, for for now it seems to do the trick

if [[ "$HOSTRESULT" =~ $REGEX ]]; then exit 0; else exit 1; fi

And add that to /etc/fail2ban/jail.local

This needs more testing, and I should add the second forward lookup, for for now it seems to do the trick

Similar Posts:

• Match SSL Certificate to Key and CSR
• IP Location
• MFA from command line
• Update PIP on Ubuntu
• Dave’s Notepad