ubuntu (14)


Amazon SES Dashboard

At work, we wanted to switch from Mandrill/Mailchimp to Amazon SES for a long time. But that was not happening mainly because the tools SES offered to monitor sent mail were, how should I say, DIY.
So, after some deliberation and when I found some time to tackle it, I did it 🙂

The setup is not too complex? Well, it is. But once you understand it, it’s pretty basic.

Let’s start at the source: Amazon

You will see this notice under Notifications for each Email Address you create/verify in SES:

Amazon SES can send you detailed notifications about your bounces, complaints, and deliveries.
Bounce and complaint notifications are available by email or through Amazon Simple Notification Service (Amazon SNS).

Next step is to create the SNS Topic, it’s just a label really.

You will also need an Amazon SQS queue. A standard queue should be good. Once it’s there, copy the ARN as you will need that for the SNS subscription.

Let’s go back to the SNS Topic we created and click on the Create subscription button. Choose Amazon SQS for the Protocol and paste the ARN of the SQS queue you created earlier. You may need to confirm that too? Just click the button if it’s there.

That’s all on the Amazon side! See how easy that was?!

Next you need a Graylog setup.

Where do I start? Well, first choose where do you want to put that Graylog “machine”. For Amazon EC2 I would just go with their ready-made AMIs. Here’s the link/docs to follow: http://docs.graylog.org/en/latest/pages/installation/aws.html (but and I quote: The Graylog appliance is not created to provide a production ready solution)

Another way to get started quickly is an Ansible role you can pick/install from Ansible Galaxy. Check out the QuickStart in the README per https://galaxy.ansible.com/Graylog2/graylog-ansible-role/#readme

But since I like doing things the “easy” way, I went with the Ubuntu 16.04 package per http://docs.graylog.org/en/latest/pages/installation/operating_system_packages.html
Seriously, it’s much easier to use and maintain since I know where everything is. Maybe it’s just me …
Anyway, here’s my bash session:

apt update && apt upgrade
sudo apt-get install apt-transport-https openjdk-8-jre-headless uuid-runtime pwgen
apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 2930ADAE8CAF5059EE73BB4B58712A2291FA4AD5
echo "deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu xenial/mongodb-org/3.6 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-3.6.list
apt update && apt install -y mongodb-org
systemctl daemon-reload
systemctl enable mongod.service
systemctl restart mongod.service
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a /etc/apt
echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list
apt update && apt-get install elasticsearch
vi /etc/elasticsearch/elasticsearch.yml)
vi /etc/elasticsearch/elasticsearch.yml
systemctl daemon-reload
systemctl enable elasticsearch.service
systemctl restart elasticsearch.service
wget https://packages.graylog2.org/repo/packages/graylog-2.4-repository_latest.deb
dpkg -i graylog-2.4-repository_latest.deb
apt-get update && sudo apt-get install graylog-server
vi /etc/graylog/server/server.conf
systemctl daemon-reload
systemctl enable graylog-server.service
systemctl start graylog-server.service

I followed the instructions there, and installed Apache on top of that with the following configuration for the VirtualHost


ServerName example.com

# Letsencrypt it
SSLCertificateFile /etc/letsencrypt/live/example.com/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/example.com/privkey.pem
Include /etc/letsencrypt/options-ssl-apache.conf

# The needed parts start here
ProxyRequests Off
Order deny,allow
Allow from all

RequestHeader set X-Graylog-Server-URL "https://example.com/api/"
ProxyPass http://127.0.0.1:9000/
ProxyPassReverse http://127.0.0.1:9000/

This will leave you with a Graylog server ready to receive the logs. Now, how do we get the logs over to Graylog? Easy! Pull them from SQS.

Start by adding a GELF HTTP Input in Graylog (System > Inputs > Select Input: GELF HTTP > Launch new input)
Make sure to get the port there right, you will need to configure the script below.
Then download the script, make sure it’s executable. Do run it manually, that way it will tell you what’s missing (BOTO3)
Make sure to configure AWS credentials. The quickest way is:
* to install awscli: apt-get install awscli
* and run its configuration: aws configure

Edit the script with the right configuration vars, add it to cron to run as much as you feel necessary (I use it @hourly)

import boto3
import json
import requests
from datetime import datetime
import sys
HOST = 'MY.HOST.ADDRESS'
PORT = 12201 # change if you create graylog input with different port
queue_url = 'https://sqs.ZONE.amazonaws.com/ACCOUNT/QUEUENAME'
sqs = boto3.client('sqs')
response = sqs.get_queue_attributes(
QueueUrl=queue_url,
AttributeNames=['ApproximateNumberOfMessages']
)
number_of_messages = int(response['Attributes']['ApproximateNumberOfMessages'])
for i in range(1, number_of_messages + 1):
data = sqs.receive_message(QueueUrl=queue_url)
if 'Messages' in data:
body = json.loads(data['Messages'][0]['Body'])
receipt_handle = data['Messages'][0]['ReceiptHandle']
msg = json.loads(body['Message'])
version = "1.1"
host = "localhost"
short_message = "Type: {}; Source: {}; Destination: {}".format(msg['notificationType'], msg['mail']['source'],
msg['mail']['destination'][0])
full_message = msg
timestamp = datetime.strptime(msg['mail']['timestamp'].strip('Z'), '%Y-%m-%dT%H:%M:%S.%f').timestamp()
to_gelf = {
"version": version,
"host": "localhost",
"short_message": short_message,
"full_message": full_message,
"timestamp": timestamp,
"level": 1
}
r = requests.post('http://{}:{}/gelf'.format(HOST,PORT), json=to_gelf)
if r.ok:
sqs.delete_message(QueueUrl=queue_url, ReceiptHandle=receipt_handle)
sys.exit(0)

Enjoy the dashboard! Oh, there’s plenty to learn about Graylog if it’s your first time, but it’s pretty good once you get the hang of it.

Similar Posts:




MySQL Slow Query Log

Why, oh why, is this so complicated?!

Well, it’s not… just a bit confusing when you don’t know where to look.

TL,DR: (Ubuntu 16.04, otherwise YMMV, read below)

Add the following to your active configuration file for mysql

[mysqld]
slow_query_log
slow_query_log_file = /var/log/mysql/mysql-slow.log
long_query_time = 2
log-queries-not-using-indexes

What could go wrong?

Wrong configuration file:

on Ubuntu 16.04 the file can be in /etc/mysql/my.cnf or /etc/mysql/conf.d/mysql.cnf or even /etc/mysql/mysql.conf.d/mysqld.cnf

to find which one is “active”, run: mysqld --verbose --help |grep -A 1 "Default options"

the result is something like the following:
Default options are read from the following files in the given order:
/etc/my.cnf /etc/mysql/my.cnf ~/.my.cnf

if you open /etc/mysql/my.cnf, you will find at the bottom:
!includedir /etc/mysql/conf.d/

That’s where the actual files are stored.

Wrong variable names:

If you follow different tutorials on the Internet you will find configurations that mention:
log_slow_queries=/var/log/mysql/slow-query.log
DO NOT use that, it’s deprecated.
the new name of the variable is `slow_query_log_file` for the actual log file, and YOU SHOULD have `slow_query_log` as a boolean variable (ON, 1, or just mentioned in the file as per the code above).

Troubleshooting

Tail the /var/log/mysql/error.log in a separate terminal, see what’s failing. Example:

[Warning] option 'slow_query_log': boolean value '/var/log/mysql/mysql-slow.log' wasn't recognized. Set to OFF.
Obviously the variable is set wrongly to the filename. They are now 2 separate variables.

Similar Posts:




How to rename terminal tab title in gnome-terminal

from http://unix.stackexchange.com/a/186167/13739

Create a function in ~/.bashrc:

function set-title() {
if [[ -z “$ORIG” ]]; then
ORIG=$PS1
fi
TITLE=”\[\e]2;$@\a\]”
PS1=${ORIG}${TITLE}
}
Then use your new command to set the terminal title. It works with spaces in the name too

set-title my new tab title

Similar Posts:




Update PIP on Ubuntu

From SO answer

Your pip may be outdated. Even in Ubuntu 14.04 LTS, the pip version it installed using apt-get install python-pip was 1.5.4. Try updating pip manually, and possibly the new packages again as well.

pip --version # 1.5.4
curl -O https://bootstrap.pypa.io/get-pip.py
sudo python get-pip.py
hash -r # reset bash cache
pip --version # 6.0.8

The hash -r line is essential!

reference: https://pip.pypa.io/en/latest/installing.html

Similar Posts:




Php5{en,dis}mod mcrypt

Seriously. I’m doing too much php stuff!

Here are a few things I learned in the last couple of days:

apt-get install php5-mcrypt
dpkg -l php5-mcrypt
ii php5-mcrypt 5.4.6-0ubuntu6 amd64 MCrypt module for php5
php -m | grep -c mcrypt
0
php5enmod mcrypt
php -m | grep mcrypt
mcrypt

And to install xhgui+xhprof, follow this tutorial (README is lacking)

Similar Posts:




Codeception Testing – Part 2

I’m still setting up for BDD testing. All the basics are already there (see previous post).

I spoke quickly about setting up WPBrowser: WordPress specific set of extensions for Codeception. Here’s what to do (from the Readme):

  1. Require the package in the composer.json, then run “composer update” (details)
  2. Add the WPBrowser or WPWebDriver module in tests/acceptance.suite.yml (details)

The added bonus, if you’re using PhpStorm is that you will also get the related auto-complete package. And there’s plenty of functions that really speed things up.

Next for me was setting up MailCatcher

MailCatcher runs a super simple SMTP server which catches any message sent to it to display in a web interface

You need mailcatcher if you’re testing sending email out from the site. Here’s what I used:
\curl -sSL https://get.rvm.io | bash
source /etc/profile.d/rvm.sh
rvm install 2.2.2
rvm default@mailcatcher --create do gem install mailcatcher
rvm wrapper default@mailcatcher --no-prefix mailcatcher catchmail

You also need to add the following in /etc/php5/apache2/php.ini:

sendmail_path =  /usr/local/rvm/wrappers/default/catchmail -f some@from.address #


(and restart apache after that)

Next you need to install the codeception mailcatcher module using composer

"captbaritone/mailcatcher-codeception-module": "1.*"

And set the configuration inacceptance.suite.yml

You might get by with only that, but I had trouble with mail being routed through special plugins we use at work, so I had to install the ‘mailcatcher’ plugin

After that it was smooth testing …

 

Similar Posts:




Codeception Testing – Part 1

I started working on automating testing ( BDD style ) a few weeks ago, and it took me a while to go over the available tools and options. The obvious starting point was cucumber. Which then led me to Behat, which then took me to Codeception.

When someone tells you it works out of the box, you should always take that with a pinch of salt. Of course, if you follow the quickstart page, it will take you through the different steps that you would take to unwrap the shrink around the box, open it up, take the protective foam bar out, … You get my drift.

Turns out you would need a few more ingredients before you could start testing. Here’s what I needed:

  1. Download and run selenium
  2. Download and install chromedriver
  3. Setup PhpStorm with Codeception

Of course if you’re using the PhpBrowser for testing (no Javascript with that sorry),  you could get started faster. So no nagging please.

And, if you’re like me, going to test WordPress sites themes and plugins, you might find the WPBrowser very useful.

Here’s a list of the steps I took locally on an Ubuntu 15.10 machine (that already had Google Chrome installed)

If you didn’t have Chrome (or installing on a server):

wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
dpkg -i google-chrome-stable_current_amd64.deb
apt-get install -f # dirty shortcut to get apt to install dependencies for you!

wget http://goo.gl/rQhaxb -O /opt/selenium-server-standalone.jar
sudo apt-get install xvfb
alias run_selenium="DISPLAY=:1 xvfb-run java -jar /opt/selenium-server-standalone.jar"

wget -N http://chromedriver.storage.googleapis.com/2.20/chromedriver_linux64.zip
unzip chromedriver_linux64.zip
chmod +x chromedriver

sudo mv -f chromedriver /usr/local/share/chromedriver
sudo ln -s /usr/local/share/chromedriver /usr/local/bin/chromedriver
sudo ln -s /usr/local/share/chromedriver /usr/bin/chromedriver

A couple of parting notes:

  1. WebDriver and PhpBrowser do not work together, one of them must be commented out when you’re testing. Pretty obvious when you think about it!
  2. When  you change browser to ‘chrome’, you should also add ‘http_proxy: direct’ in the acceptance.suite.yml

Similar Posts:




Multiple IP addresses (aliases) for wifi connection on Ubuntu

From: http://askubuntu.com/a/217363/63517

 

I’m assuming you’re running NetworkManager here, you’ve already set up your wireless connection using DHCP and you’re talking about IPv4 here.

While you can’t configure the static addresses in NetworkManager GUI, there’s a hack possible.

  1. Find the connection UUID of the connection configured
    $ nmcli con
    
  2. Add a script in /etc/NetworkManager/dispatcher.d/, containing this starting point:
    #!/bin/bash
    
    WLAN_DEV=wlan0
    MYCON_UUID=31c48409-e77a-46e0-8cdc-f4c04b978901
    
    if [ "$CONNECTION_UUID" == "$MYCON_UUID" ]; then
        # add alias for Network 1: 192.168.0.123/24
        ifconfig $WLAN_DEV:0 192.168.0.123 netmask 255.255.255.0 up
        # add alias for Network 2: 192.168.1.123/24
        ifconfig $WLAN_DEV:1 192.168.1.123 netmask 255.255.255.0 up
    fi
    
  3. Make sure it has the right permissions (chmod +x /path/to/script.sh) and restart NetworkManager:
    $ sudo service network-manager restart
    

Now when you connect to your wireless connection, it should add the two aliases (check with ifconfig.

Similar Posts:




NVidia Driver on Ubuntu 13.04

A quick recipe that worked like a charm
Download driver from http://www.nvidia.com/object/linux-display-amd64-313.30-driver.html
then run:

sudo service lightdm stop
sudo chmod a+x NVIDIA-Linux-x86_64-313.30.run
sudo ./NVIDIA-Linux-x86_64-313.30.run

You may need to run the last command twice (including a reboot) if you are already running Nouveau drivers. Follow the on-screen instructions.

Similar Posts:




Customize the Ubuntu/Gnome Launchers

Google Chrome has a special “New Incognito Window” in the Gnome launcher. So how hard would it be to edit the launcher buttons to add some customized commands? Apparently not so hard. Here’s what I got:

Similar Posts: