Things I tend to forget


Return a static page for specific users/IPs

Why do I want to do that?
Different reasons. One time, because I wanted a certain user not to access the dynamic WordPress site. Another time, I wanted to provide a bot that has been crawling the site with a “legitimate” page, without actually allowing it to go through the site. The main thing I was looking for is a way to do an internal redirect, so no 3xx code returned. I know there are probably better ways to achieve these goals (are there?). But hey, I learned some stuff about Nginx while doing this.

So here it goes, the first attempt:

  location ~* /some/path/with_numbers/\d+ {
    if ($remote_addr = 11.11.111.1) {
       return 200 "sample reply - should be empty";
    }
    # the next line is reached only when the above is not executed
    try_files $uri $uri/ /index.php$is_args$args;
  }

One problem with the above is that replacing the IP or adding more IPs is a bit problematic. So, we replace it with the following that relies on the Geo module:

geo $bad_ip {
  default 0;
  1.2.3.4/32 1;
  4.3.2.1/32 1;
}

server {
[...]
 location ~* /some/path/with_numbers/\d+ {
    if ($bad_ip) {
       return 200 "sample reply - should be empty";
    }
    # the next line is reached only when the above is not executed
    try_files $uri $uri/ /index.php$is_args$args;
  }

The other problem is that the text returned with the 200 code is a bit simplistic and I really wanted to send an HTML static page, not a stupid line. The fix uses error_page


[...]
 location ~* /some/path/with_numbers/\d+ {
    if ($bad_ip) { return 410; }
    error_page 410 =200 /my_static_page.html;
    # the next line is reached only when the above is not executed
    try_files $uri $uri/ /index.php$is_args$args;
  }

The result is a 200 (OK) code sent to the browser with a static HTML page that should load much faster than a PHP/RoR/etc alternative.

Of course, more can be done to identify the blocked entity, for example using UserAgent string, etc.
Leaving that for another day.

Similar Posts:




How NOT to ban Googlebot

Google do not provide a list of IPs to identify their bots, so you can’t simply add that to fail2ban’s ‘ignoreip =’ line.

Instead, according to their answer per https://support.google.com/webmasters/answer/80553?hl=en you can only verify the bot’s provenance by checking the DNS for the bot’s IP. In fact, they ask you to run 2 queries (1 reverse and 1 forward lookup) to make sure that the IP is not spoofed.

My simple 1 reverse lookup script is:

#!/bin/bash
IP="$1"
HOSTRESULT="$(host -W ${IP})" # updated thanks to comment from Martin
REGEX='.*(googlebot\.com\.|google\.com\.)

And add that to /etc/fail2ban/jail.local

ignorecommand = /usr/local/bin/ignore_ip_check.sh 

This needs more testing, and I should add the second forward lookup, for for now it seems to do the trick

if [[ "$HOSTRESULT" =~ $REGEX ]]; then exit 0; else exit 1; fi

And add that to /etc/fail2ban/jail.local


This needs more testing, and I should add the second forward lookup, for for now it seems to do the trick

Similar Posts:




MySQL Slow Query Log

Why, oh why, is this so complicated?!

Well, it’s not… just a bit confusing when you don’t know where to look.

TL,DR: (Ubuntu 16.04, otherwise YMMV, read below)

Add the following to your active configuration file for mysql

[mysqld]
slow_query_log
slow_query_log_file = /var/log/mysql/mysql-slow.log
long_query_time = 2
log-queries-not-using-indexes

What could go wrong?

Wrong configuration file:

on Ubuntu 16.04 the file can be in /etc/mysql/my.cnf or /etc/mysql/conf.d/mysql.cnf or even /etc/mysql/mysql.conf.d/mysqld.cnf

to find which one is “active”, run: mysqld --verbose --help |grep -A 1 "Default options"

the result is something like the following:
Default options are read from the following files in the given order:
/etc/my.cnf /etc/mysql/my.cnf ~/.my.cnf

if you open /etc/mysql/my.cnf, you will find at the bottom:
!includedir /etc/mysql/conf.d/

That’s where the actual files are stored.

Wrong variable names:

If you follow different tutorials on the Internet you will find configurations that mention:
log_slow_queries=/var/log/mysql/slow-query.log
DO NOT use that, it’s deprecated.
the new name of the variable is `slow_query_log_file` for the actual log file, and YOU SHOULD have `slow_query_log` as a boolean variable (ON, 1, or just mentioned in the file as per the code above).

Troubleshooting

Tail the /var/log/mysql/error.log in a separate terminal, see what’s failing. Example:

[Warning] option 'slow_query_log': boolean value '/var/log/mysql/mysql-slow.log' wasn't recognized. Set to OFF.
Obviously the variable is set wrongly to the filename. They are now 2 separate variables.

Similar Posts:




Corky!

wpid-wp-1425419051162.jpegthis is not a pipe

Similar Posts:

    None Found




Install same packages for different PHP version

This is one of the things I should have thought of a long time ago.
More times than I can count I needed to install the same (or similar) list of PHP packages for a different version (ie. installing 7.0 while keeping 5.6) So instead of manually copying over the list or guessing as to what will be installed via dependencies here’s a quick one liner. Again, it’s a “facepalm” moment here for me:

# this installs 7.1 packages from a 7.0 list
dpkg -l | grep php7 | awk '{gsub(/7\.0/,"7.1",$2); print $2}' | xargs apt install

Also on my mind: instead of a sit/stand desk, can I install a shower desk. This way I can work and take showers at the same time!

Similar Posts:




All of us cats

image

Similar Posts:

    None Found




How to rename terminal tab title in gnome-terminal

from http://unix.stackexchange.com/a/186167/13739

Create a function in ~/.bashrc:

function set-title() {
if [[ -z “$ORIG” ]]; then
ORIG=$PS1
fi
TITLE=”\[\e]2;$@\a\]”
PS1=${ORIG}${TITLE}
}
Then use your new command to set the terminal title. It works with spaces in the name too

set-title my new tab title

Similar Posts:




Using MySQL + xargs to restore a Mydumper backup

Split the restore, first the schema, then the data:


dump# ls my_database.* | grep schema | xargs -I % -n 1 bash -c "mysql anotherDB < %"
dump# ls my_database.* | grep -v schema | xargs -I % -n 1 bash -c "mysql anotherDB < %"

Skip restoring data for a few tables:


dump# ls my_database.* | egrep -v 'tbl_a|tbl_b|tbl_c' | grep -v schema | xargs -I % -n 1 bash -c "mysql anotherDB < %"

Reference: http://www.dctrwatson.com/2010/07/using-mydumper-to-parallel-dumpimport-fromto-mysql/

Similar Posts:




Update PIP on Ubuntu

From SO answer

Your pip may be outdated. Even in Ubuntu 14.04 LTS, the pip version it installed using apt-get install python-pip was 1.5.4. Try updating pip manually, and possibly the new packages again as well.

pip --version # 1.5.4
curl -O https://bootstrap.pypa.io/get-pip.py
sudo python get-pip.py
hash -r # reset bash cache
pip --version # 6.0.8

The hash -r line is essential!

reference: https://pip.pypa.io/en/latest/installing.html

Similar Posts:




Bind Solr 5.5 to localhost

Edit /opt/solr/bin/solr

Find the line with SOLR_START_OPTS and add the following after "-Djetty.port=$SOLR_PORT":
"-Djetty.host=localhost"

Mine looks like this:
SOLR_START_OPTS=('-server' "${JAVA_MEM_OPTS[@]}" "${GC_TUNE[@]}" "${GC_LOG_OPTS[@]}" \
"${REMOTE_JMX_OPTS[@]}" "${CLOUD_MODE_OPTS[@]}" \
"-Djetty.port=$SOLR_PORT" "-Djetty.host=localhost" "-DSTOP.PORT=$stop_port" "-DSTOP.KEY=$STOP_KEY" \
"${SOLR_HOST_ARG[@]}" "-Duser.timezone=$SOLR_TIMEZONE" \
"-Djetty.home=$SOLR_SERVER_DIR" "-Dsolr.solr.home=$SOLR_HOME" "-Dsolr.install.dir=$SOLR_TIP" \
"${LOG4J_CONFIG[@]}" "${SOLR_OPTS[@]}")

Someone also suggested adding it to the SOLR_HOST_ARG array earlier in the same file. You can try that:
SOLR_HOST_ARG+=("-Djetty.host=$SOLR_HOST")

I hope it helps!

Similar Posts: