linux (17)


Amazon SES Dashboard

At work, we wanted to switch from Mandrill/Mailchimp to Amazon SES for a long time. But that was not happening mainly because the tools SES offered to monitor sent mail were, how should I say, DIY.
So, after some deliberation and when I found some time to tackle it, I did it 🙂

The setup is not too complex? Well, it is. But once you understand it, it’s pretty basic.

Let’s start at the source: Amazon

You will see this notice under Notifications for each Email Address you create/verify in SES:

Amazon SES can send you detailed notifications about your bounces, complaints, and deliveries.
Bounce and complaint notifications are available by email or through Amazon Simple Notification Service (Amazon SNS).

Next step is to create the SNS Topic, it’s just a label really.

You will also need an Amazon SQS queue. A standard queue should be good. Once it’s there, copy the ARN as you will need that for the SNS subscription.

Let’s go back to the SNS Topic we created and click on the Create subscription button. Choose Amazon SQS for the Protocol and paste the ARN of the SQS queue you created earlier. You may need to confirm that too? Just click the button if it’s there.

That’s all on the Amazon side! See how easy that was?!

Next you need a Graylog setup.

Where do I start? Well, first choose where do you want to put that Graylog “machine”. For Amazon EC2 I would just go with their ready-made AMIs. Here’s the link/docs to follow: http://docs.graylog.org/en/latest/pages/installation/aws.html (but and I quote: The Graylog appliance is not created to provide a production ready solution)

Another way to get started quickly is an Ansible role you can pick/install from Ansible Galaxy. Check out the QuickStart in the README per https://galaxy.ansible.com/Graylog2/graylog-ansible-role/#readme

But since I like doing things the “easy” way, I went with the Ubuntu 16.04 package per http://docs.graylog.org/en/latest/pages/installation/operating_system_packages.html
Seriously, it’s much easier to use and maintain since I know where everything is. Maybe it’s just me …
Anyway, here’s my bash session:

apt update && apt upgrade
sudo apt-get install apt-transport-https openjdk-8-jre-headless uuid-runtime pwgen
apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 2930ADAE8CAF5059EE73BB4B58712A2291FA4AD5
echo "deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu xenial/mongodb-org/3.6 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-3.6.list
apt update && apt install -y mongodb-org
systemctl daemon-reload
systemctl enable mongod.service
systemctl restart mongod.service
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a /etc/apt
echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list
apt update && apt-get install elasticsearch
vi /etc/elasticsearch/elasticsearch.yml)
vi /etc/elasticsearch/elasticsearch.yml
systemctl daemon-reload
systemctl enable elasticsearch.service
systemctl restart elasticsearch.service
wget https://packages.graylog2.org/repo/packages/graylog-2.4-repository_latest.deb
dpkg -i graylog-2.4-repository_latest.deb
apt-get update && sudo apt-get install graylog-server
vi /etc/graylog/server/server.conf
systemctl daemon-reload
systemctl enable graylog-server.service
systemctl start graylog-server.service

I followed the instructions there, and installed Apache on top of that with the following configuration for the VirtualHost


ServerName example.com

# Letsencrypt it
SSLCertificateFile /etc/letsencrypt/live/example.com/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/example.com/privkey.pem
Include /etc/letsencrypt/options-ssl-apache.conf

# The needed parts start here
ProxyRequests Off
Order deny,allow
Allow from all

RequestHeader set X-Graylog-Server-URL "https://example.com/api/"
ProxyPass http://127.0.0.1:9000/
ProxyPassReverse http://127.0.0.1:9000/

This will leave you with a Graylog server ready to receive the logs. Now, how do we get the logs over to Graylog? Easy! Pull them from SQS.

Start by adding a GELF HTTP Input in Graylog (System > Inputs > Select Input: GELF HTTP > Launch new input)
Make sure to get the port there right, you will need to configure the script below.
Then download the script, make sure it’s executable. Do run it manually, that way it will tell you what’s missing (BOTO3)
Make sure to configure AWS credentials. The quickest way is:
* to install awscli: apt-get install awscli
* and run its configuration: aws configure

Edit the script with the right configuration vars, add it to cron to run as much as you feel necessary (I use it @hourly)

import boto3
import json
import requests
from datetime import datetime
import sys
HOST = 'MY.HOST.ADDRESS'
PORT = 12201 # change if you create graylog input with different port
queue_url = 'https://sqs.ZONE.amazonaws.com/ACCOUNT/QUEUENAME'
sqs = boto3.client('sqs')
response = sqs.get_queue_attributes(
QueueUrl=queue_url,
AttributeNames=['ApproximateNumberOfMessages']
)
number_of_messages = int(response['Attributes']['ApproximateNumberOfMessages'])
for i in range(1, number_of_messages + 1):
data = sqs.receive_message(QueueUrl=queue_url)
if 'Messages' in data:
body = json.loads(data['Messages'][0]['Body'])
receipt_handle = data['Messages'][0]['ReceiptHandle']
msg = json.loads(body['Message'])
version = "1.1"
host = "localhost"
short_message = "Type: {}; Source: {}; Destination: {}".format(msg['notificationType'], msg['mail']['source'],
msg['mail']['destination'][0])
full_message = msg
timestamp = datetime.strptime(msg['mail']['timestamp'].strip('Z'), '%Y-%m-%dT%H:%M:%S.%f').timestamp()
to_gelf = {
"version": version,
"host": "localhost",
"short_message": short_message,
"full_message": full_message,
"timestamp": timestamp,
"level": 1
}
r = requests.post('http://{}:{}/gelf'.format(HOST,PORT), json=to_gelf)
if r.ok:
sqs.delete_message(QueueUrl=queue_url, ReceiptHandle=receipt_handle)
sys.exit(0)

Enjoy the dashboard! Oh, there’s plenty to learn about Graylog if it’s your first time, but it’s pretty good once you get the hang of it.

Similar Posts:




MySQL Slow Query Log

Why, oh why, is this so complicated?!

Well, it’s not… just a bit confusing when you don’t know where to look.

TL,DR: (Ubuntu 16.04, otherwise YMMV, read below)

Add the following to your active configuration file for mysql

[mysqld]
slow_query_log
slow_query_log_file = /var/log/mysql/mysql-slow.log
long_query_time = 2
log-queries-not-using-indexes

What could go wrong?

Wrong configuration file:

on Ubuntu 16.04 the file can be in /etc/mysql/my.cnf or /etc/mysql/conf.d/mysql.cnf or even /etc/mysql/mysql.conf.d/mysqld.cnf

to find which one is “active”, run: mysqld --verbose --help |grep -A 1 "Default options"

the result is something like the following:
Default options are read from the following files in the given order:
/etc/my.cnf /etc/mysql/my.cnf ~/.my.cnf

if you open /etc/mysql/my.cnf, you will find at the bottom:
!includedir /etc/mysql/conf.d/

That’s where the actual files are stored.

Wrong variable names:

If you follow different tutorials on the Internet you will find configurations that mention:
log_slow_queries=/var/log/mysql/slow-query.log
DO NOT use that, it’s deprecated.
the new name of the variable is `slow_query_log_file` for the actual log file, and YOU SHOULD have `slow_query_log` as a boolean variable (ON, 1, or just mentioned in the file as per the code above).

Troubleshooting

Tail the /var/log/mysql/error.log in a separate terminal, see what’s failing. Example:

[Warning] option 'slow_query_log': boolean value '/var/log/mysql/mysql-slow.log' wasn't recognized. Set to OFF.
Obviously the variable is set wrongly to the filename. They are now 2 separate variables.

Similar Posts:




How to rename terminal tab title in gnome-terminal

from http://unix.stackexchange.com/a/186167/13739

Create a function in ~/.bashrc:

function set-title() {
if [[ -z “$ORIG” ]]; then
ORIG=$PS1
fi
TITLE=”\[\e]2;$@\a\]”
PS1=${ORIG}${TITLE}
}
Then use your new command to set the terminal title. It works with spaces in the name too

set-title my new tab title

Similar Posts:




Bind Solr 5.5 to localhost

Edit /opt/solr/bin/solr

Find the line with SOLR_START_OPTS and add the following after "-Djetty.port=$SOLR_PORT":
"-Djetty.host=localhost"

Mine looks like this:
SOLR_START_OPTS=('-server' "${JAVA_MEM_OPTS[@]}" "${GC_TUNE[@]}" "${GC_LOG_OPTS[@]}" \
"${REMOTE_JMX_OPTS[@]}" "${CLOUD_MODE_OPTS[@]}" \
"-Djetty.port=$SOLR_PORT" "-Djetty.host=localhost" "-DSTOP.PORT=$stop_port" "-DSTOP.KEY=$STOP_KEY" \
"${SOLR_HOST_ARG[@]}" "-Duser.timezone=$SOLR_TIMEZONE" \
"-Djetty.home=$SOLR_SERVER_DIR" "-Dsolr.solr.home=$SOLR_HOME" "-Dsolr.install.dir=$SOLR_TIP" \
"${LOG4J_CONFIG[@]}" "${SOLR_OPTS[@]}")

Someone also suggested adding it to the SOLR_HOST_ARG array earlier in the same file. You can try that:
SOLR_HOST_ARG+=("-Djetty.host=$SOLR_HOST")

I hope it helps!

Similar Posts:




header v3 rsa/sha1 signature, key id c105b9de: bad

error: rpmts_HdrFromFdno: Header V3 RSA/SHA1 Signature, key ID XXXXXXXX: BAD
error: rpmdbNextIterator: skipping h# XXXX Header V3 RSA/SHA1 Signature, key ID XXXXXXXX BAD

Solution:
wget http://mirror.centos.org/centos/6/updates/x86_64/Packages/nss-softokn-freebl-3.14.3-19.el6_6.x86_64.rpm
rpm2cpio nss-softokn-freebl-3.14.3-19.el6_6.x86_64.rpm | cpio -idmv
cp ./lib64/libfreeblpriv3.* /lib64
yum update

All good now, back to work everyone.

Similar Posts:




apt get install libapache2 mod evasive

apt-get install libapache2-mod-evasive

cat > /etc/apache2/mods-enabled/mod-evasive.conf <<EOF
<IfModule mod_evasive20.c>
 DOSHashTableSize 3097
 DOSPageCount 2
 DOSSiteCount 50
 DOSPageInterval 1
 DOSSiteInterval 1
 DOSBlockingPeriod 60
</IfModule>
EOF

Similar Posts:




Clearing the Queue

Talking about mail queues here. Especially when the queue gets filled with hundreds of thousands of spam emails.

Stop the MTA

The first thing you do is stop the MTA before it gets worse. On most Linux servers:
~: # /etc/init.d/exim stop # most WHM/cPanel based servers
~: # /etc/init.d/postfix stop # if you have postfix

Then check the queue

The commands to use are
~: # exim -bp
~: # mailq

I suggest keeping CTRL+C ready to stop the flow. If you have an infestation scrolling through the queue will take a very very long time.
But checking out the first few lines (pages) of the queue may reveal many things. For example, what user account is being exploited for sending out the spam. Sometimes that’s obvious, other times you may need to do some digging. Here’s a sample:

24h  1.1K 1VeKKb-00030t-42 <noreply@yahoo-inc.com>
        D alena@example.com
        D alenka@example.com
        D alepp@example.com
        D alerei@example.com

24h  1.1K 1VeKKb-00030u-5R <noreply@yahoo-inc.com>
        D alex@example.com
        D alex.b@example.com
        D alex.d@example.com
        D alex.f@example.com

24h  1.1K 1VeKKb-00030v-1R <noreply@yahoo-inc.com>
        D alex.t@example.com
[...]

The lines above are from an exim queue output. Obviously noreply@yahoo-inc.com is not an account on this server. Let’s find the username then clean up. A nice Exim utility is exigrep (basically it’s grep with exim ties!)
:~# exigrep 1VeKKb-00030t-42 /var/log/exim/mainlog
+++ 1VeKKb-00030t-42 has not completed +++
2013-11-08 21:18:14 1VeKKb-00030t-42 <= noreply@yahoo-inc.com H=(User) [XXX.XX.XX.XXX] P=esmtpa A=login:test@mydomain.com S=1193 T="Authenticate Your Email" from <noreply@yahoo-inc.com> for alena@example.com alenka@example.com alepp@example.com alerei@example.com [...]

2 things interest us here: the H=(User) [XXX.XX.XX.XXX] part tells us where that intruder is logging from (probably another exploited server, so it’s nice to alert the owners of that server as well. And A=login:test@mydomain.com tells us which user they are logged in as, so we can lock down that user, change their password, etc.

Cleaning up

Emptying the queue is usually an easy way out:
~: # postsuper -d ALL # for postfix
~: # exim -bp | exiqgrep -i | xargs exim -Mrm # for exim

A more subtle approach is needed when some of that mail in the queue is actually legit. If you checked out the queue earlier you might do something like the following:
~: # exiqgrep -i -f noreply@yahoo-inc.com | exim -Mrm # that will remove all mail sent by noreply@yahoo-inc.com

It’s a bit harder in postfix, here’s one recipe:
~: # mailq | tail +2 | awk 'BEGIN { RS = "" } { if ($7 == "noreply@yahoo-inc.com") print $1 }' | tr -d '*!' | postsuper -d -
remove the tail +2 | if that doesn’t work for you.

There are plenty of other things to check and do to fix your server when someone is abusing it to spam. Cleaning up is one way to start.

Ah, before I forget, here’s a link to a very nice cheatsheet for exim

Similar Posts:




Mbox to Maildir Recipe

apt-get install mb2md
cd /var/mail
for i in *; do echo $i; su - $i -c "/usr/bin/mb2md -s /var/mail/$i -d ~/Maildir"; done

Similar Posts:




Suspend Ubuntu Desktop

Simple command line: pm-suspend
An interesting alternative would be: pm-suspend-hybrid

Similar Posts:




Recovering from a failed RAID drive

I was trying to mount one of the disks from a failed RAID array. The disk was put in a USB enclosure and hooked to the machine. Here’s what I saw:

srv0:~# mount /dev/sdc3 /mnt/usb/
mount: unknown filesystem type 'mdraid'
srv0:~# mdadm --examine /dev/sdc
mdadm: No md superblock detected on /dev/sdc.

I panicked a little, then tried the following that worked:

srv0:~# mount /dev/sdc3 /mnt/usb/ -o ro -t ext3
srv0:~# ls /mnt/usb
bin   dev   etc   initrd.img  lib64	  media  opt   root  selinux  sys  usr	vmlinuz
boot  home  lib	      lost+found  mnt	 proc  sbin  srv      tmp  var

Similar Posts: