The Lost Notebook

configuration (11)


My Ansible AWS EC2 Dynamic Inventory

Start with the the Ansible configuration. This can be set in /etc/ansible/ansible.cfg or ~/.ansible.cfg (in the home directory) or ansible.cfg (in the current directory)

My suggestion is use one of the first 2 (ie. /etc/ or ~/.ansible.cfg if you’re going to be managing instances from your machine. Update the configuration as needed.

[defaults]
inventory = ./ansible_plugins
enable_plugins = aws_ec2
host_key_checking = False
pipelining = True
log_path = /var/log/ansible

You may need other plugins, this one is for aws_ec2. In the /etc/ansible/ansible_plugins directory, create the *_aws_ec2.yml configuration file for your inventory

# /etc/ansible/ansible_plugins/testing_aws_ec2.yml
---
plugin: aws_ec2
aws_profile: testing
regions:
  - us-east-1
  - us-east-2
filters:
  tag:Team: testing
  instance-state-name : running
hostnames:
  - instance-id
  - dns-name
keyed_groups:
  - prefix: team
    key: tags['Team']

I’m filtering using a tag:Team == testing and showing only running instances.

I’m also using the instance-id and dns-name attributes as hostname

And I’m using the tag['Team']as a grouping.

So now, I can do the following from any directory (since my configuration is global in /etc/ansible)

$ ansible-inventory --list --yaml

all:
  children:
    aws_ec2:
      hosts:
        i-xxxxxxxxxxxxxxx:
          ami_launch_index: 0
          architecture: x86_64
          block_device_mappings:
          - device_name: /dev/sda1
            ebs:
              attach_time: 2020-08-10 15:20:58+00:00
              delete_on_termination: true
              status: attached
              volume_id: vol-xxxxxxxxxxxxxx
...
    team_testing:
      hosts:
        i-xyxyxyxyxyyxyxyy: {}
        i-xyxyxy2321yxyxyy: {}
        i-xyxyxyxyxy89yxyy: {}
        i-xyxy1210xyyxyxyy: {}
        i-xyxy999999yxyxyy: {}
        i-xyxyxy44xyyxyxyy: {}
        i-xyx2323yxyyxyxyy: {}
        i-xyxyxyxyxy9977yy: {}
    ungrouped: {}

I can also use the team_testing or the individual instance_id in my Ansible hostscalls.

Similar Posts:



Amazon SES Dashboard

At work, we wanted to switch from Mandrill/Mailchimp to Amazon SES for a long time. But that was not happening mainly because the tools SES offered to monitor sent mail were, how should I say, DIY.
So, after some deliberation and when I found some time to tackle it, I did it 🙂

The setup is not too complex? Well, it is. But once you understand it, it’s pretty basic.

Let’s start at the source: Amazon

You will see this notice under Notifications for each Email Address you create/verify in SES:

Amazon SES can send you detailed notifications about your bounces, complaints, and deliveries.
Bounce and complaint notifications are available by email or through Amazon Simple Notification Service (Amazon SNS).

Next step is to create the SNS Topic, it’s just a label really.

You will also need an Amazon SQS queue. A standard queue should be good. Once it’s there, copy the ARN as you will need that for the SNS subscription.

Let’s go back to the SNS Topic we created and click on the Create subscription button. Choose Amazon SQS for the Protocol and paste the ARN of the SQS queue you created earlier. You may need to confirm that too? Just click the button if it’s there.

That’s all on the Amazon side! See how easy that was?!

Next you need a Graylog setup.

Where do I start? Well, first choose where do you want to put that Graylog “machine”. For Amazon EC2 I would just go with their ready-made AMIs. Here’s the link/docs to follow: http://docs.graylog.org/en/latest/pages/installation/aws.html (but and I quote: The Graylog appliance is not created to provide a production ready solution)

Another way to get started quickly is an Ansible role you can pick/install from Ansible Galaxy. Check out the QuickStart in the README per https://galaxy.ansible.com/Graylog2/graylog-ansible-role/#readme

But since I like doing things the “easy” way, I went with the Ubuntu 16.04 package per http://docs.graylog.org/en/latest/pages/installation/operating_system_packages.html
Seriously, it’s much easier to use and maintain since I know where everything is. Maybe it’s just me …
Anyway, here’s my bash session:

apt update && apt upgrade
sudo apt-get install apt-transport-https openjdk-8-jre-headless uuid-runtime pwgen
apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 2930ADAE8CAF5059EE73BB4B58712A2291FA4AD5
echo "deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu xenial/mongodb-org/3.6 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-3.6.list
apt update && apt install -y mongodb-org
systemctl daemon-reload
systemctl enable mongod.service
systemctl restart mongod.service
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a /etc/apt
echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list
apt update && apt-get install elasticsearch
vi /etc/elasticsearch/elasticsearch.yml)
vi /etc/elasticsearch/elasticsearch.yml
systemctl daemon-reload
systemctl enable elasticsearch.service
systemctl restart elasticsearch.service
wget https://packages.graylog2.org/repo/packages/graylog-2.4-repository_latest.deb
dpkg -i graylog-2.4-repository_latest.deb
apt-get update && sudo apt-get install graylog-server
vi /etc/graylog/server/server.conf
systemctl daemon-reload
systemctl enable graylog-server.service
systemctl start graylog-server.service

I followed the instructions there, and installed Apache on top of that with the following configuration for the VirtualHost


ServerName example.com

# Letsencrypt it
SSLCertificateFile /etc/letsencrypt/live/example.com/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/example.com/privkey.pem
Include /etc/letsencrypt/options-ssl-apache.conf

# The needed parts start here
ProxyRequests Off
Order deny,allow
Allow from all

RequestHeader set X-Graylog-Server-URL "https://example.com/api/"
ProxyPass http://127.0.0.1:9000/
ProxyPassReverse http://127.0.0.1:9000/

This will leave you with a Graylog server ready to receive the logs. Now, how do we get the logs over to Graylog? Easy! Pull them from SQS.

Start by adding a GELF HTTP Input in Graylog (System > Inputs > Select Input: GELF HTTP > Launch new input)
Make sure to get the port there right, you will need to configure the script below.
Then download the script, make sure it’s executable. Do run it manually, that way it will tell you what’s missing (BOTO3)
Make sure to configure AWS credentials. The quickest way is:
* to install awscli: apt-get install awscli
* and run its configuration: aws configure

Edit the script with the right configuration vars, add it to cron to run as much as you feel necessary (I use it @hourly)

import boto3
import json
import requests
from datetime import datetime
import sys
HOST = 'MY.HOST.ADDRESS'
PORT = 12201 # change if you create graylog input with different port
queue_url = 'https://sqs.ZONE.amazonaws.com/ACCOUNT/QUEUENAME'
sqs = boto3.client('sqs')
response = sqs.get_queue_attributes(
QueueUrl=queue_url,
AttributeNames=['ApproximateNumberOfMessages']
)
number_of_messages = int(response['Attributes']['ApproximateNumberOfMessages'])
for i in range(1, number_of_messages + 1):
data = sqs.receive_message(QueueUrl=queue_url)
if 'Messages' in data:
body = json.loads(data['Messages'][0]['Body'])
receipt_handle = data['Messages'][0]['ReceiptHandle']
msg = json.loads(body['Message'])
version = "1.1"
host = "localhost"
short_message = "Type: {}; Source: {}; Destination: {}".format(msg['notificationType'], msg['mail']['source'],
msg['mail']['destination'][0])
full_message = msg
timestamp = datetime.strptime(msg['mail']['timestamp'].strip('Z'), '%Y-%m-%dT%H:%M:%S.%f').timestamp()
to_gelf = {
"version": version,
"host": "localhost",
"short_message": short_message,
"full_message": full_message,
"timestamp": timestamp,
"level": 1
}
r = requests.post('http://{}:{}/gelf'.format(HOST,PORT), json=to_gelf)
if r.ok:
sqs.delete_message(QueueUrl=queue_url, ReceiptHandle=receipt_handle)
sys.exit(0)
view raw ses-sns-sqs-to-graylog.py hosted with ❤ by GitHub

Enjoy the dashboard! Oh, there’s plenty to learn about Graylog if it’s your first time, but it’s pretty good once you get the hang of it.

Similar Posts:



Return a static page for specific users/IPs

Why do I want to do that?
Different reasons. One time, because I wanted a certain user not to access the dynamic WordPress site. Another time, I wanted to provide a bot that has been crawling the site with a “legitimate” page, without actually allowing it to go through the site. The main thing I was looking for is a way to do an internal redirect, so no 3xx code returned. I know there are probably better ways to achieve these goals (are there?). But hey, I learned some stuff about Nginx while doing this.

So here it goes, the first attempt: 

  location ~* /some/path/with_numbers/\d+ {
    if ($remote_addr = 11.11.111.1) {
       return 200 "sample reply - should be empty";
    }
    # the next line is reached only when the above is not executed
    try_files $uri $uri/ /index.php$is_args$args;
  }

One problem with the above is that replacing the IP or adding more IPs is a bit problematic. So, we replace it with the following that relies on the Geo module: 

geo $bad_ip {
  default 0;
  1.2.3.4/32 1;
  4.3.2.1/32 1;
}

server {
[...]
 location ~* /some/path/with_numbers/\d+ {
    if ($bad_ip) {
       return 200 "sample reply - should be empty";
    }
    # the next line is reached only when the above is not executed
    try_files $uri $uri/ /index.php$is_args$args;
  }

The other problem is that the text returned with the 200 code is a bit simplistic and I really wanted to send an HTML static page, not a stupid line. The fix uses error_page


[...]
 location ~* /some/path/with_numbers/\d+ {
    if ($bad_ip) { return 410; }
    error_page 410 =200 /my_static_page.html;
    # the next line is reached only when the above is not executed
    try_files $uri $uri/ /index.php$is_args$args;
  }

The result is a 200 (OK) code sent to the browser with a static HTML page that should load much faster than a PHP/RoR/etc alternative.

Of course, more can be done to identify the blocked entity, for example using UserAgent string, etc.
Leaving that for another day.

Similar Posts:



Bind Solr 5.5 to localhost

Edit /opt/solr/bin/solr

Find the line with SOLR_START_OPTS and add the following after "-Djetty.port=$SOLR_PORT":
"-Djetty.host=localhost"

Mine looks like this:
SOLR_START_OPTS=('-server' "${JAVA_MEM_OPTS[@]}" "${GC_TUNE[@]}" "${GC_LOG_OPTS[@]}" \
"${REMOTE_JMX_OPTS[@]}" "${CLOUD_MODE_OPTS[@]}" \
"-Djetty.port=$SOLR_PORT" "-Djetty.host=localhost" "-DSTOP.PORT=$stop_port" "-DSTOP.KEY=$STOP_KEY" \
"${SOLR_HOST_ARG[@]}" "-Duser.timezone=$SOLR_TIMEZONE" \
"-Djetty.home=$SOLR_SERVER_DIR" "-Dsolr.solr.home=$SOLR_HOME" "-Dsolr.install.dir=$SOLR_TIP" \
"${LOG4J_CONFIG[@]}" "${SOLR_OPTS[@]}")

Someone also suggested adding it to the SOLR_HOST_ARG array earlier in the same file. You can try that:
SOLR_HOST_ARG+=("-Djetty.host=$SOLR_HOST")

I hope it helps!

Similar Posts:



Webmin/VirtualMin with Let’s Encrypt

Virtualmin team said the next version of Virtualmin/Webmin will automate most of the letsencrypt setup. Meanwhile there’s an ongoing conversation about it in the forums

My setup:
./letsencrypt-auto certonly --webroot --webroot-path /usr/share/nginx/html -d my.vmin.server

Then in Webmin > Webmin Configuration > SSL Encryption set:

  • Private key file to /etc/letsencrypt/live/my.vmin.server/privkey.pem
  • Certificate file to /etc/letsencrypt/live/my.vmin.server/fullchain.pem

Add a monthly crontab job to renew the certificate:
/usr/local/letsencrypt/letsencrypt-auto certonly \
--webroot --webroot-path /usr/share/nginx/html -d my.vmin.server \
--renew-by-default \
--agree-tos

Similar Posts:



Multiple IP addresses (aliases) for wifi connection on Ubuntu

From: http://askubuntu.com/a/217363/63517

 

I’m assuming you’re running NetworkManager here, you’ve already set up your wireless connection using DHCP and you’re talking about IPv4 here.

While you can’t configure the static addresses in NetworkManager GUI, there’s a hack possible.

  1. Find the connection UUID of the connection configured 
    $ nmcli con
  2. Add a script in /etc/NetworkManager/dispatcher.d/, containing this starting point: 
    #!/bin/bash

WLAN_DEV=wlan0
MYCON_UUID=31c48409-e77a-46e0-8cdc-f4c04b978901

if [ "$CONNECTION_UUID" == "$MYCON_UUID" ]; then
    # add alias for Network 1: 192.168.0.123/24
    ifconfig $WLAN_DEV:0 192.168.0.123 netmask 255.255.255.0 up
    # add alias for Network 2: 192.168.1.123/24
    ifconfig $WLAN_DEV:1 192.168.1.123 netmask 255.255.255.0 up
fi
  3. Make sure it has the right permissions (chmod +x /path/to/script.sh) and restart NetworkManager: 
    $ sudo service network-manager restart

Now when you connect to your wireless connection, it should add the two aliases (check with ifconfig.

Similar Posts:



fix of the day Drupal stores extra php…

fix of the day:
Drupal stores extra php configuration (like memory_limit) in sites/default/settings.php
Drush is your friend!

Similar Posts:



apt get install libapache2 mod evasive

apt-get install libapache2-mod-evasive

cat > /etc/apache2/mods-enabled/mod-evasive.conf <<EOF
<IfModule mod_evasive20.c>
 DOSHashTableSize 3097
 DOSPageCount 2
 DOSSiteCount 50
 DOSPageInterval 1
 DOSSiteInterval 1
 DOSBlockingPeriod 60
</IfModule>
EOF

Similar Posts:



Tweaking GitLab Setup

What?

GitLab is your own GitHub and more (or less). They have pretty good introduction on the home page, so I won’t repeat that here.

The recommended installation method for GitLab is using the Omnibus package. Head to the downloads page and follow the instructions. You should have a GitLab setup in no time, who needs GitHub! oh well, many many people…

Now to the tweaks.

Why?

If you’re like me trying to hide the ports on your server from the bots and prying eyes, they you would have SSH on a different port and your other services all bound to localhost and facing the Internet bravely from behind a proxy server. I use Apache on my personal server, it’s pretty robust and gets the job done.

So let’s say SSH is on port 2022, and apache is taking firm hold on ports 80 and 443. So GitLab’s NGINX should take port 8088.

And the domain you’re using for gitlab is not the machine’s hostname, so hostname is ‘host4339.moodeef.com’ and gitlab’s URL is ‘gitlab.deeb.me’

How?

Edit the “/etc/gitlab/gitlab.rb” file with the following changes/additions:

gitlab_rails['gitlab_host'] = 'gitlab.deeb.me'
gitlab_rails['gitlab_ssh_host'] = 'gitlab.deeb.me'
gitlab_rails['gitlab_port'] = 8088
gitlab_rails['gitlab_email_from'] = 'git-no-reply@deeb.me'
gitlab_rails['gitlab_support_email'] = 'git-no-reply@deeb.me'
gitlab_rails['gitlab_shell_ssh_port'] = 2022
external_url = 'https://gitlab.deeb.me'

Then run gitlab-ctl reconfigure and see how it goes from there.

If things seem to be too complicated, you can always get a subscription option with full support from the GitLab folks. Or hire me to fix it for you!

Similar Posts:



Images not displayed unless site is refreshed

I faced a bit of puzzle today with Tomcat/Apache setup.
Tomcat is running in the background with Apache as frontend via mod_proxy_ajp. The site loads ok except for static files that return a 404 (File Not Found) on first load, then show up normally on refresh!

The apache configuration looks like the following: 

Alias /static /var/www/static
ProxyPass /static !
ProxyPass / ajp://127.0.0.1:8009/
ProxyPassReverse / ajp://127.0.0.1:8009/

An example failing URL: http://example.com/static/images/email.png;jsessionid=3892BC4B4C26073338268AF98ECA73D6
And in the error log I see the following:
[Fri Sep 06 10:18:14 2013] [error] [client 00.00.0.000] File does not exist: /var/www/static/images/email.png;jsessionid=3892BC4B4C26073338268AF98ECA73D6, referer[…]

Then it dawned on me that apache wouldn’t know about the jsessionid if it was not sent over to tomcat for processing. Since apache was handling static files the session id needed to go.

Solution: I added the following rewrite rule


        RewriteEngine On
        RewriteRule static/(.*);jsessionid=.* /static/$1 [R,L]

related searches I came across while googling:
page not found when including jsessionid in URL
;jsessionid and 404 File Not Found
Apache getting confused by encoded jsessionid’s (404 Not Found)

Similar Posts:




The Lost Notebook, 2023. Theme Material for Coders