Coding (29)


AWS Lambda Function Code

Quick snippet to get the function code

wget -O function.zip $(aws lambda get-function --function-name MyFunctionName --query 'Code.Location' --output text)

And another to update lambda with the latest

cd package 
zip -r9 ../function.zip .
cd ..
zip -g function.zip function.py
aws lambda update-function-code --function-name MyFunctionName --zip-file fileb://function.zip

Similar Posts:




Handling AWS Simple Email Service notifications

Apparently it’s not enough to gather information on your email sending, bounces and complaints. You also need to act when there’s a problem otherwise Amazon will put your account under probation and might even shut it completely if you don’t fix the problem.

And it’s pretty easy to implement per their sample code and the explanation in the aws blog

Here are the steps I use to set it up:

  • Create an SNS topic. Let’s call it SES-BOUNCES. Or create 2 SNS topics (1 for bounces, another for complaints). I will go with just one.
  • Create a service on your site to handle the bounces. I used some simple PHP code (pasted below) for that.
  • Create a subscription in your SNS topic (created first)

The service should handle 2 things:

  • the subscription (that will happen once!): call the SubscribeURL and you’re done.
  • the bounces/complaints: Remove the email addresses form your mailing list, or blacklist them, or … The main idea is to stop sending emails to those who complain. And stop trying to send emails to addresses that bounce.

Here’s the code for the PHP service, it’s pretty straight forward

if ( 'POST' !== $_SERVER['REQUEST_METHOD'] ) {
	http_response_code( 405 );
	die;
}

require 'vendor/autoload.php';

use Aws\Sns\Message;
use Aws\Sns\MessageValidator;

function parse_bounce( $message ) {
	$recipients = array();
	foreach ( $message['bounce']['bouncedRecipients'] as $recipient ) {
		$recipients[] = $recipient['emailAddress'];
	}
	switch ( $message['bounce']['bounceType'] ) {
		case 'Transient':
			error_log( "BouncesController - Soft bounces occurred for " . join( ', ', $recipients ) );

			return array();
			break;
		case 'Permanent':
			error_log( "BouncesController - Hard bounces occurred for " . join( ', ', $recipients ) );

			return $recipients;
			break;
	}
}

function parse_complaint( $message ) {
	$recipients = array();
	foreach ( $message['complaint']['complainedRecipients'] as $recipient ) {
		$recipients[] = $recipient['emailAddress'];
	}

	return $recipients;
}

function blacklist_recipients( $recipients ) {
	foreach ( $recipients as $email_address ) {
		// blacklist those emails
	}
}

try {
	$sns_message = Message::fromRawPostData();

	$validator = new MessageValidator();
	$validator->validate( $sns_message );

	if ( $validator->isValid( $sns_message ) ) {
		if ( in_array( $sns_message['Type'], [ 'SubscriptionConfirmation', 'UnsubscribeConfirmation' ] ) ) {
			file_get_contents( $sns_message['SubscribeURL'] );
			error_log( 'Subscribed to ' . $sns_message['SubscribeURL'] );
		}

		if ( $sns_message['Type'] == 'Notification' ) {
			$message           = $sns_message['Message'];
			$notification_type = $message['notificationType'];
			if ( $notification_type == 'Bounce' ) {
				blacklist_recipients( parse_bounce( $message ) );
			}
			if ( $notification_type == 'Complaint' ) {
				blacklist_recipients( parse_complaint( $message ) );
			}
		}
	}


} catch ( Exception $e ) {
	error_log( 'Error: ' . $e->getMessage() );
	http_response_code( 404 );
	die;
}

Don’t forget to run composer to get those dependencies:

{
  "require": {
    "aws/aws-sdk-php": "3.*",
    "aws/aws-php-sns-message-validator": "1.4.0"
  }
}

And make sure you don’t send unsolicited emails! it’s just not nice.

Similar Posts:




Amazon SES Dashboard

At work, we wanted to switch from Mandrill/Mailchimp to Amazon SES for a long time. But that was not happening mainly because the tools SES offered to monitor sent mail were, how should I say, DIY.
So, after some deliberation and when I found some time to tackle it, I did it 🙂

The setup is not too complex? Well, it is. But once you understand it, it’s pretty basic.

Let’s start at the source: Amazon

You will see this notice under Notifications for each Email Address you create/verify in SES:

Amazon SES can send you detailed notifications about your bounces, complaints, and deliveries.
Bounce and complaint notifications are available by email or through Amazon Simple Notification Service (Amazon SNS).

Next step is to create the SNS Topic, it’s just a label really.

You will also need an Amazon SQS queue. A standard queue should be good. Once it’s there, copy the ARN as you will need that for the SNS subscription.

Let’s go back to the SNS Topic we created and click on the Create subscription button. Choose Amazon SQS for the Protocol and paste the ARN of the SQS queue you created earlier. You may need to confirm that too? Just click the button if it’s there.

That’s all on the Amazon side! See how easy that was?!

Next you need a Graylog setup.

Where do I start? Well, first choose where do you want to put that Graylog “machine”. For Amazon EC2 I would just go with their ready-made AMIs. Here’s the link/docs to follow: http://docs.graylog.org/en/latest/pages/installation/aws.html (but and I quote: The Graylog appliance is not created to provide a production ready solution)

Another way to get started quickly is an Ansible role you can pick/install from Ansible Galaxy. Check out the QuickStart in the README per https://galaxy.ansible.com/Graylog2/graylog-ansible-role/#readme

But since I like doing things the “easy” way, I went with the Ubuntu 16.04 package per http://docs.graylog.org/en/latest/pages/installation/operating_system_packages.html
Seriously, it’s much easier to use and maintain since I know where everything is. Maybe it’s just me …
Anyway, here’s my bash session:

apt update && apt upgrade
sudo apt-get install apt-transport-https openjdk-8-jre-headless uuid-runtime pwgen
apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 2930ADAE8CAF5059EE73BB4B58712A2291FA4AD5
echo "deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu xenial/mongodb-org/3.6 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-3.6.list
apt update && apt install -y mongodb-org
systemctl daemon-reload
systemctl enable mongod.service
systemctl restart mongod.service
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a /etc/apt
echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list
apt update && apt-get install elasticsearch
vi /etc/elasticsearch/elasticsearch.yml)
vi /etc/elasticsearch/elasticsearch.yml
systemctl daemon-reload
systemctl enable elasticsearch.service
systemctl restart elasticsearch.service
wget https://packages.graylog2.org/repo/packages/graylog-2.4-repository_latest.deb
dpkg -i graylog-2.4-repository_latest.deb
apt-get update && sudo apt-get install graylog-server
vi /etc/graylog/server/server.conf
systemctl daemon-reload
systemctl enable graylog-server.service
systemctl start graylog-server.service

I followed the instructions there, and installed Apache on top of that with the following configuration for the VirtualHost


ServerName example.com

# Letsencrypt it
SSLCertificateFile /etc/letsencrypt/live/example.com/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/example.com/privkey.pem
Include /etc/letsencrypt/options-ssl-apache.conf

# The needed parts start here
ProxyRequests Off
Order deny,allow
Allow from all

RequestHeader set X-Graylog-Server-URL "https://example.com/api/"
ProxyPass http://127.0.0.1:9000/
ProxyPassReverse http://127.0.0.1:9000/

This will leave you with a Graylog server ready to receive the logs. Now, how do we get the logs over to Graylog? Easy! Pull them from SQS.

Start by adding a GELF HTTP Input in Graylog (System > Inputs > Select Input: GELF HTTP > Launch new input)
Make sure to get the port there right, you will need to configure the script below.
Then download the script, make sure it’s executable. Do run it manually, that way it will tell you what’s missing (BOTO3)
Make sure to configure AWS credentials. The quickest way is:
* to install awscli: apt-get install awscli
* and run its configuration: aws configure

Edit the script with the right configuration vars, add it to cron to run as much as you feel necessary (I use it @hourly)

import boto3
import json
import requests
from datetime import datetime
import sys
HOST = 'MY.HOST.ADDRESS'
PORT = 12201 # change if you create graylog input with different port
queue_url = 'https://sqs.ZONE.amazonaws.com/ACCOUNT/QUEUENAME'
sqs = boto3.client('sqs')
response = sqs.get_queue_attributes(
QueueUrl=queue_url,
AttributeNames=['ApproximateNumberOfMessages']
)
number_of_messages = int(response['Attributes']['ApproximateNumberOfMessages'])
for i in range(1, number_of_messages + 1):
data = sqs.receive_message(QueueUrl=queue_url)
if 'Messages' in data:
body = json.loads(data['Messages'][0]['Body'])
receipt_handle = data['Messages'][0]['ReceiptHandle']
msg = json.loads(body['Message'])
version = "1.1"
host = "localhost"
short_message = "Type: {}; Source: {}; Destination: {}".format(msg['notificationType'], msg['mail']['source'],
msg['mail']['destination'][0])
full_message = msg
timestamp = datetime.strptime(msg['mail']['timestamp'].strip('Z'), '%Y-%m-%dT%H:%M:%S.%f').timestamp()
to_gelf = {
"version": version,
"host": "localhost",
"short_message": short_message,
"full_message": full_message,
"timestamp": timestamp,
"level": 1
}
r = requests.post('http://{}:{}/gelf'.format(HOST,PORT), json=to_gelf)
if r.ok:
sqs.delete_message(QueueUrl=queue_url, ReceiptHandle=receipt_handle)
sys.exit(0)

Enjoy the dashboard! Oh, there’s plenty to learn about Graylog if it’s your first time, but it’s pretty good once you get the hang of it.

Similar Posts:




WordPress Ajax Skeleton

Similar Posts:




WPMorris Plugin First Test

Donut

[morris type=”Donut” element=”donut-div” xkey=”” ykey=”” labels=””] {label: “Download Sales”, value: 12}, {label: “In-Store Sales”, value: 30}, {label: “Mail-Order Sales”, value: 20} [/morris]


Bar

[morris type=”Bar” element=”bar-div” xkey=”y” ykeys=”‘a’, ‘b'” labels=”‘Series A’, ‘Series B'”]{ y: ‘2006’, a: 100, b: 90 }, { y: ‘2007’, a: 75, b: 65 }, { y: ‘2008’, a: 50, b: 40 }, { y: ‘2009’, a: 75, b: 65 }, { y: ‘2010’, a: 50, b: 40 }, { y: ‘2011’, a: 75, b: 65 }, { y: ‘2012’, a: 100, b: 90 }[/morris]


Line

[morris type=”Line” element=”line-div” xkey=”y” ykeys=”‘a’, ‘b'” labels=”‘Series A’, ‘Series B'”]{ y: ‘2006’, a: 100, b: 90 }, { y: ‘2007’, a: 75, b: 65 }, { y: ‘2008’, a: 50, b: 40 }, { y: ‘2009’, a: 75, b: 65 }, { y: ‘2010’, a: 50, b: 40 }, { y: ‘2011’, a: 75, b: 65 }, { y: ‘2012’, a: 100, b: 90 }[/morris]


Area

[morris type=”Area” element=”area-div” xkey=”y” ykeys=”‘a’, ‘b'” labels=”‘Series A’, ‘Series B'”]{ y: ‘2006’, a: 100, b: 90 }, { y: ‘2007’, a: 75, b: 65 }, { y: ‘2008’, a: 50, b: 40 }, { y: ‘2009’, a: 75, b: 65 }, { y: ‘2010’, a: 50, b: 40 }, { y: ‘2011’, a: 75, b: 65 }, { y: ‘2012’, a: 100, b: 90 }[/morris]

Similar Posts:




Quickly Start a Django Project

There are many things I like to do when starting a django project. I started to compile those in a script I’m using. The gist is linked below.

New Django Project.sh

#!/bin/bash
# 
# Author: Abdallah Deeb 
# Requirements: python, pip, virtualenv, virutalenvwrapper, git
#
# Edit the following 2 lines
PROJECTNAME=proj
APPNAME=myapp
if [ -n "$2" ]
then 
  APPNAME="$2"
fi
if [ -n "$1" ]
then 
  PROJECTNAME=$1
fi

source `which virtualenvwrapper.sh` 

echo "Starting a new Django project: $PROJECTNAME"
# Make a virtualenv
mkvirtualenv $PROJECTNAME

# Install latest django and start the project/app
pip install django
django-admin.py startproject $PROJECTNAME
cd $PROJECTNAME
django-admin.py startapp $APPNAME
mv $PROJECTNAME conf

# conf is much nicer than projname
replace $PROJECTNAME conf -- manage.py conf/settings.py conf/wsgi.py conf/urls.py
chmod +x manage.py
workon $PROJECTNAME

# Initialize and use git
git init
git add .
git commit -a -m 'initial commit'

and the raw gist download

Similar Posts:




Django on Codio

Codio is awesome. Coolio would have been a better name, but I guess that’s taken!

It’s free, but don’t let that fool you. What you get is a beautifully simple yet powerful IDE in your browser. It works great even on my tablet, so I can edit/update/deploy even on the road (or on the couch half asleep, and here’s where git comes in handy!). It supports github out of the box. And has a cool terminal with SSH and everything. Yay!

Now if that was not awesome enough, these guys went out of their way and added an even cooler feature:

Every project gets its own Box: an instantly available server-side development environment with full terminal access.

So you get to do it all without leaving their site. I wanted to try playing with a pet project I’m working on using django. It was pretty easy to get it all setup (especially that the code was on github, but you could as simply upload the code).

Here’s a step by step tutorial to get django up and running on codio.

Open a terminal:
Tools > Terminal

I’m using MySQL, so:

parts install mysql
parts mysql start
mysqladmin -u root password StrongPassword
mysql -u root -pStrongPassword
mysq> create database dbname;
mysql> grant all privileges on dbname.* to 'dbuser'@'localhost' identified by 'dbpasswd';
mysql> flush privileges;
mysql> \q

Also make sure to install python using the ‘parts’ command:

parts install python2

(see the related issue on github.com: https://github.com/codio/boxparts/issues/80)

Create the requirements.txt file if you don’t have that already. Hint: run ‘pip freeze > requirements.txt’ to save those. So in requirements.txt:

Django
MySQL-python
#distribute
#wsgiref

Run the following in the codio.com terminal:
parts install pip
pip install -r requirements.txt

Upload the django project, or import it from GitHub then:
python manage.py syncdb
python manage.py runserver 0.0.0.0:8009

I still need to get it running through apache. I’ll get on that later tonight.

parts install apache2_mod_wsgi # also installs apache2 and apr_util
============ apache2 ============                  
To start the Apache server:
  $ parts start apache2 
To stop the Apache server:
  $ parts stop apache2    
Apache config is located at:
  $ /home/codio/.parts/etc/apache2/httpd.conf                                               
Default document root is located at:  
  $ /home/codio/workspace 
     
============ apache2_mod_wsgi ============                
If Apache2 httpd is already running, you will need to restart it: 
  $ parts restart apache2
Default configuration for wsgi is:  
  WSGIScriptAlias / /home/codio/workspace                                                  
  You can change default in /home/codio/.parts/etc/apache2/config/wsgi.conf file

So I changed the file in /home/codio/.parts/etc/apache2/config/wsgi.conf to:

LoadModule wsgi_module  /home/codio/.parts/packages/apache2_mod_wsgi/3.4/mod_wsgi.so
WSGIScriptAlias / /home/codio/workspace/conf/wsgi.py # <-- this is where I put my wsgi.py file               
WSGIPythonPath /home/codio/workspace                         
                          
                                             
Order deny,allow                                           
Require all granted                                       
                                                  

Then start apache, and visit your site at http://UNIQUE-NAME.codio.io:3000/:
parts start apache2

You will probably see a ‘500 Internal Server Error’ message. So tail the apache ErrorLog file in the terminal to see what went wrong:
tail -f /home/codio/.parts/var/apache2/log/error_log

Here’s what I see (and I haven’t figured the fix yet):

[] mod_wsgi (pid=912): Target WSGI script '/home/codio/workspace/conf/wsgi.py' cannot be loaded as Python module.                                                                  
[] mod_wsgi (pid=912): Exception occurred processing WSGI script '/home/codio/workspace/conf/wsgi.py'.                                                                             
[] Traceback (most recent call last):                                                  
[]   File "/home/codio/workspace/conf/wsgi.py", line 5, in                     
[]     from django.core.wsgi import get_wsgi_application                               
[]   File "/home/codio/.parts/packages/python2/2.7.6/lib/python2.7/site-packages/django/core/wsgi.py", line 1, in                                                          
[]     from django.core.handlers.wsgi import WSGIHandler                               
[]   File "/home/codio/.parts/packages/python2/2.7.6/lib/python2.7/site-packages/django/core/handlers/wsgi.py", line 6, in                                                 
[]     from io import BytesIO                                                          
[]   File "/home/codio/.parts/packages/python2/2.7.6/lib/python2.7/io.py", line 51, in                                                                                     
[]     import _io                                                                      
[] ImportError: /home/codio/.parts/packages/python2/2.7.6/lib/python2.7/lib-dynload/_io.so: undefined symbol: PyUnicodeUCS2_Replace 

to be continued…
… issue fixed by installing python using the ‘parts’ command above

Similar Posts:




Email Tracking

Most email clients can request a return receipt. One way to set this up in Thunderbird for example is to go to “Preferences > Advanced > General > Return Receipts …” and check the “When sending messages, always request a return receipt”

The problem with this approach is that when the person you’re emailing has set his email client to reject that request (or does that manually every time), you will not get a receipt back. And honestly speaking there is no way to force that. Now there are some services out there that promise to track your mail when that’s read, and those rely on embedding a transparent gif that will call home when the message is opened. I tried a few of these services: the free ones did not deliver (or seemed too shady for my taste) and the paid ones did not look too good either. So I cooked up a quick solution that I can use when needed (like when I’m tracking my brother on his honeymoon trip *evil grin*)

The code is below, pretty self explanatory. You will need a transparent gif/png. That’s easy too 😉

Edit: Looks like some services are not so bad. Checkout bananatag and YesWare Email Tracking

Similar Posts:




RimuDNS python module

I wrote previously about building dns zone files mostly by querying DNS and saving the results to a file.

The method is not perfect, and it will not be possible to get all the records. In fact, you could get all the records incorporated in the result file if you already have a list and feed it to your code.

I wrote a python module to manage DNS records using the RimuHosting DNS API. The code is hosted at GitHub.

I also added a guessing feature.

Similar Posts:




Django-TimelineJS Pluggable App

I like VeriteCo’s TimelineJS. It’s elegant, easy to use and very useful. I don’t like editing Google a google spreadsheet every time I want to update the timeline although it might be easier for some people. So, I wrote a small pluggable app for django to embed a timeline in django templates.

The code is loosely based on the code for the WordPress plugin. And I’ll try to copy over more code and fixes from that soon.

For now, it’s a working app with an admin backend to enter the timeline information and events, and to edit the options. It’s also a custom template tag that you can use to embed the timeline in your own templates.

Should be easy enough to use it.

pip install django-timelinejs

then include it in your INSTALLED_APPS and ./manage.py syncdb

Drop me a note if you find a bug or want some help. You can use the issues in GitHub for that.

Similar Posts: