At work, we wanted to switch from Mandrill/Mailchimp to Amazon SES for a long time. But that was not happening mainly because the tools SES offered to monitor sent mail were, how should I say, DIY.
So, after some deliberation and when I found some time to tackle it, I did it 🙂
The setup is not too complex? Well, it is. But once you understand it, it’s pretty basic.
Let’s start at the source: Amazon
You will see this notice under Notifications for each Email Address you create/verify in SES:
Amazon SES can send you detailed notifications about your bounces, complaints, and deliveries.
Bounce and complaint notifications are available by email or through Amazon Simple Notification Service (Amazon SNS).
Next step is to create the SNS Topic, it’s just a label really.
You will also need an Amazon SQS queue. A standard queue should be good. Once it’s there, copy the ARN as you will need that for the SNS subscription.
Let’s go back to the SNS Topic we created and click on the Create subscription button. Choose Amazon SQS for the Protocol and paste the ARN of the SQS queue you created earlier. You may need to confirm that too? Just click the button if it’s there.
That’s all on the Amazon side! See how easy that was?!
Next you need a Graylog setup.
Where do I start? Well, first choose where do you want to put that Graylog “machine”. For Amazon EC2 I would just go with their ready-made AMIs. Here’s the link/docs to follow: http://docs.graylog.org/en/latest/pages/installation/aws.html (but and I quote: The Graylog appliance is not created to provide a production ready solution)
Another way to get started quickly is an Ansible role you can pick/install from Ansible Galaxy. Check out the QuickStart in the README per https://galaxy.ansible.com/Graylog2/graylog-ansible-role/#readme
But since I like doing things the “easy” way, I went with the Ubuntu 16.04 package per http://docs.graylog.org/en/latest/pages/installation/operating_system_packages.html
Seriously, it’s much easier to use and maintain since I know where everything is. Maybe it’s just me …
Anyway, here’s my bash session:
apt update && apt upgrade sudo apt-get install apt-transport-https openjdk-8-jre-headless uuid-runtime pwgen apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 2930ADAE8CAF5059EE73BB4B58712A2291FA4AD5 echo "deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu xenial/mongodb-org/3.6 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-3.6.list apt update && apt install -y mongodb-org systemctl daemon-reload systemctl enable mongod.service systemctl restart mongod.service wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add - echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a /etc/apt echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list apt update && apt-get install elasticsearch vi /etc/elasticsearch/elasticsearch.yml) vi /etc/elasticsearch/elasticsearch.yml systemctl daemon-reload systemctl enable elasticsearch.service systemctl restart elasticsearch.service wget https://packages.graylog2.org/repo/packages/graylog-2.4-repository_latest.deb dpkg -i graylog-2.4-repository_latest.deb apt-get update && sudo apt-get install graylog-server vi /etc/graylog/server/server.conf systemctl daemon-reload systemctl enable graylog-server.service systemctl start graylog-server.service
I followed the instructions there, and installed Apache on top of that with the following configuration for the VirtualHost
ServerName example.com # Letsencrypt it SSLCertificateFile /etc/letsencrypt/live/example.com/fullchain.pem SSLCertificateKeyFile /etc/letsencrypt/live/example.com/privkey.pem Include /etc/letsencrypt/options-ssl-apache.conf # The needed parts start here ProxyRequests Off Order deny,allow Allow from all RequestHeader set X-Graylog-Server-URL "https://example.com/api/" ProxyPass http://127.0.0.1:9000/ ProxyPassReverse http://127.0.0.1:9000/
This will leave you with a Graylog server ready to receive the logs. Now, how do we get the logs over to Graylog? Easy! Pull them from SQS.
Start by adding a GELF HTTP Input in Graylog (System > Inputs > Select Input: GELF HTTP > Launch new input)
Make sure to get the port there right, you will need to configure the script below.
Then download the script, make sure it’s executable. Do run it manually, that way it will tell you what’s missing (BOTO3)
Make sure to configure AWS credentials. The quickest way is:
* to install awscli: apt-get install awscli
* and run its configuration: aws configure
Edit the script with the right configuration vars, add it to cron to run as much as you feel necessary (I use it @hourly)
| import boto3 | |
| import json | |
| import requests | |
| from datetime import datetime | |
| import sys | |
| HOST = 'MY.HOST.ADDRESS' | |
| PORT = 12201 # change if you create graylog input with different port | |
| queue_url = 'https://sqs.ZONE.amazonaws.com/ACCOUNT/QUEUENAME' | |
| sqs = boto3.client('sqs') | |
| response = sqs.get_queue_attributes( | |
| QueueUrl=queue_url, | |
| AttributeNames=['ApproximateNumberOfMessages'] | |
| ) | |
| number_of_messages = int(response['Attributes']['ApproximateNumberOfMessages']) | |
| for i in range(1, number_of_messages + 1): | |
| data = sqs.receive_message(QueueUrl=queue_url) | |
| if 'Messages' in data: | |
| body = json.loads(data['Messages'][0]['Body']) | |
| receipt_handle = data['Messages'][0]['ReceiptHandle'] | |
| msg = json.loads(body['Message']) | |
| version = "1.1" | |
| host = "localhost" | |
| short_message = "Type: {}; Source: {}; Destination: {}".format(msg['notificationType'], msg['mail']['source'], | |
| msg['mail']['destination'][0]) | |
| full_message = msg | |
| timestamp = datetime.strptime(msg['mail']['timestamp'].strip('Z'), '%Y-%m-%dT%H:%M:%S.%f').timestamp() | |
| to_gelf = { | |
| "version": version, | |
| "host": "localhost", | |
| "short_message": short_message, | |
| "full_message": full_message, | |
| "timestamp": timestamp, | |
| "level": 1 | |
| } | |
| r = requests.post('http://{}:{}/gelf'.format(HOST,PORT), json=to_gelf) | |
| if r.ok: | |
| sqs.delete_message(QueueUrl=queue_url, ReceiptHandle=receipt_handle) | |
| sys.exit(0) |
Enjoy the dashboard! Oh, there’s plenty to learn about Graylog if it’s your first time, but it’s pretty good once you get the hang of it.