Things I tend to forget


MFA from command line

MFA, 2FA, 2 step validation, etc. are everywhere these days. And it’s a good thing.

Problem with using the phone to get the authentication code is that you need to have it handy at all times (when you want to login at least) and that you have to read the code then type it in (too many steps)

One possible alternative is to use the command line oathtool

Here’s my snippet, I added the following line in my .bashrc

Some preparation work:

If you want to use the same account with both a phone based MFA generator and the shell, set them up at the same time. Simply use the generated string for setting up the account in the Google Authenticator (as an example) and the add it to the ~/.mfa/account_name.mfa

The use of xclip automatically copies the 6 digit authentication code to the clipboard. You can go ahead and paste it.

The above setup works in Ubuntu. Didn’t try it on other systems.

Similar Posts:

    None Found



VaultPress SSH/SFTP access for WordPress site behind AWS load balancer

I recently worked on migrating a large WordPress site from a dedicated server to AWS following the reference architecture as outlined in https://github.com/aws-samples/aws-refarch-wordpress

One of the upsides of this architecture is that the site is entirely behind the load balancer and the instances running the web server are never accessible from the internet (not even via ssh or http). So any updates to the site need to be deployed using Systems Manager. I will post about that later … However due to this restriction, VaultPress is only able to access the site (for backups) over https, which causes extra load on the web instances.

To fix that, we need to allow VaultPress SSH access directly to the web instances (at least to 1 of them). And the only way in via SSH is through a bastion instance.

On this bastion instance, I installed Nginx with the following configuration in /etc/nginx/nginx.conf

where 10.0.1.12 is the internal IP of one web instances behind the load balancer.

At this stage (after reloading Nginx), I added the vaultpress user to the web instances and set their SSH public key in the .ssh/authorized_keys for that user.

Testing the connection works, all green on the VaultPress site. The SSH and the SFTP connection should work properly. To test manually, I added my own public key and connected to:

Since the IPs may change in case there’s an auto scaling event, I added a cron job that runs every 5 minutes on the bastion and updates the nginx configuration to point to one of the web instances internal IPs. A simplistic version looks like this:

You can also use CloudWatch Events to add a rule, that runs when AutoScaling happens. Pointing that to a Lambda function that calls Systems Manager to change the IP in Nginx. Here’s a reference for that https://docs.aws.amazon.com/autoscaling/ec2/userguide/cloud-watch-events.html#create-lambda-function

Oh, and don’t forget to lock down the bastion instance. In my case, I set the Security group in the EC2 console to allow only the following:

Similar Posts:




List IPs from CloudTrail events

A quick command to list the IPs from AWS CloudTrail events.

This of course can be extended to include more information, for example:

Similar Posts:




Change profiles automatically in Tilix when connecting to SSH hosts

I use Tilix as my main terminal app on Ubuntu. I like it mainly because it’s easy to use and shows multiple sessions in tiles and tabs, so it’s easy to switch between multiple servers without too much fuss

Tilix is also pretty customizable. I just needed to have the “profile” automatically switch according to the server/machine I switch to. This can be done according to the documentation per https://gnunn1.github.io/tilix-web/manual/profileswitch/ but I wanted something configured locally, not sent to the remote system.

So I just added a function that wraps my ssh command. Here’s how it looks in my .bashrc

This sets up the hostname of machine you’re logging into as the title. That’s your trigger. What remains is to create a profile and assign that profile to automatically switch when the trigger (hostname:directory, user can also be added) is set.

Go to Profiles > Edit Profile > Advanced (tab) and under Match add the hostname.


That’s about it. I’m going to add a new profile with a RED background now for my production machines! too much?

Similar Posts:




packet_write_wait: Connection to xxx port 22: Broken pipe

SSH connections started dropping left and right a few days ago. I thought at first it was a problem with my connection, our DSL connection in Beirut, Lebanon is getting a lot better this last year. But it still had its quirks now and then and I blamed the wires and bad weather.

But it happened again the second day and the third, and it was becoming really annoying with jobs killed in the middle of execution when I forget to start a screen.

Long story short, it seems that some settings were removed or reset after an upgrade in Ubuntu on my home machine. This is what I added in /etc/ssh/ssh_config (not sshd_config!)

And that’s it!

If you don’t want to edit the system-wide configuration, you can always edit ~/.ssh/config with the same for similar effect.

Similar Posts:




Check PHP-FPM Status from CLI

It’s good to have the status page, especially if you need to troubleshoot issues that are not showing up in the regular logs, such as high load or memory consumption.

However, looking at that page and refreshing it manually is not always useful. Sometimes you need to log that data, or have a way to pinpoint a single PID causing the load.

First make sure you have the status page accessible. Here’s a tutorial I like: https://easyengine.io/tutorials/php/fpm-status-page/

The create this script on the server. Make sure to change the connect part to your PHP-FPM pool’s correct port or socket

One way I use it is run top and check for the suspect process PID, then run fpm_status.sh <PID>

Similar Posts:




Find the owner of an AWS Access Key

This is something you will have to deal with at one time or another after managing AWS IAM users for a while. Basically, it’s straight forward with the following code:

  • Get a list of users
    • for each user get their access key IDs
    • (optional) pipe to grep for checking a specific ID

And here’s the aws-cli code

 

Similar Posts:




Handling AWS Simple Email Service notifications

Apparently it’s not enough to gather information on your email sending, bounces and complaints. You also need to act when there’s a problem otherwise Amazon will put your account under probation and might even shut it completely if you don’t fix the problem.

And it’s pretty easy to implement per their sample code and the explanation in the aws blog

Here are the steps I use to set it up:

  • Create an SNS topic. Let’s call it SES-BOUNCES. Or create 2 SNS topics (1 for bounces, another for complaints). I will go with just one.
  • Create a service on your site to handle the bounces. I used some simple PHP code (pasted below) for that.
  • Create a subscription in your SNS topic (created first)

The service should handle 2 things:

  • the subscription (that will happen once!): call the SubscribeURL and you’re done.
  • the bounces/complaints: Remove the email addresses form your mailing list, or blacklist them, or … The main idea is to stop sending emails to those who complain. And stop trying to send emails to addresses that bounce.

Here’s the code for the PHP service, it’s pretty straight forward

Don’t forget to run composer to get those dependencies:

And make sure you don’t send unsolicited emails! it’s just not nice.

Similar Posts:




Slash command for Mattermost

Following up on the code that set the nickname and status via bash function, I wanted to do the same using a slash command

Here’s the code in PHP

seems to work for me.

You’ll need to follow the instructions in the documentation to create the command on the server. Make sure to save the tokens in a safe place as usual.

Similar Posts:




Change Nickname in Mattermost

A colleague asked me today for a quick way to set the nickname in Mattermost. He needed to do that to provide more information about his status than what the actual “Status” in shows, which is limited to “Online”, “Away”, “Do Not Disturn” and “Offline”

So if you want to tell people you’re away for a couple of hours, or sick, walking the dog, etc. then you need to go IRC style and put the additional information in your nickname. Not too bad actually, just inconvenient.

I checked the Mattermost API docs and wrote a small bash script to get things going

A couple of things to watch out there:

  • You need to save the TOKEN, not the TOKEN ID. Once created and saved the actual TOKEN is no longer showing in the UI. So save that somewhere safe and use it in the script
  • The user needs to be able to create their own token. Follow the procedure per the docs here to allow them to do that. Yes, you need to do all that 🙂
  • The Channel ID can be copied from the channel drop-down menu > View info. In the bottom left, in grey you will see: ID: xxxxxxxxxx that’s the one you need!

 

For convenience, I added a few aliases in my bashrc:

I know it’s better to add a slash-command for that. Something like ‘/nick …’ or ‘/status …’. I’ll check out those docs later.

Similar Posts: