How we deployed Duo 2FA for SSH to 600 Ubiquiti servers

HostiFi Update

I started HostiFi, a Ubiquiti cloud hosting service, in May 2018. In January 2019 I went full-time on it, and since then I’ve been sharing monthly updates on website traffic, revenue, and what I’ve been working on at https://rchase.com.

My last blog post about HostiFi was in May 2019, when I wrote “Why I’m completely rewriting my $5,735 MRR SaaS” which was about some major changes that I’m building, including a website rewrite to move off of WordPress to a Laravel Spark platform.

I estimated that rewrite would take 2–3 months, and it’s been 5 months now, but it is finally getting close to completion!

Here is a preview of the new homepage.

Here’s a quick update on progress since my last blog post before I write about the Duo deployment.

• The business has doubled in size in several areas in the last 4 months— monthly recurring revenue increased from $5,735 to $11,403, Ubiquiti devices connected increased from 12,000 to 28,000, subscribers increased from 266 to 538, and servers increased from 309 to 600
• Employee headcount also doubled as I hired Safwan Shaikh to help with customer support. Safwan brings several years of experience from his previous work as a Technical Support Engineer at Ubiquiti and has been doing a great job helping our customers
• Migrated self-hosted email to GSuite
• Replaced Hubspot live chat with Intercom
• Will be using Intercom instead of Drip for automated email onboarding campaigns
• Using Intercom as our KB instead of Freshdesk https://support.hostifi.net
• Implemented an affiliate program in July with https://getrewardful.com
• Added GDPR compliance to the website with the help of https://termly.io, thus the pop up in the bottom of the homepage screenshots
• Purchased hostifi.com, still in the process of migrating everything
• We’ve added a ton of new internal software for our remote team of 2, probably should make a separate entire blog post about all the awesome tools we are using

Deploying Duo 2FA for SSH to 600 Ubiquiti servers

Since hiring Safwan to help with support, I’ve been thinking a ton about security as a team instead of just security as an individual. First off, we started using 1Password as a team. Previously I was using KeePass. 1Password manages storing passwords securely, reminds us to rotate them, and keeps separate credentials for every account to protect against password reuse.

In addition, we are 2FA’ing every account we can.

As I went about creating accounts in our various systems for Safwan, I’ve also followed the principle of least privilege by setting the account permissions so that he only has the access he needs to do his job and nothing extra.

One of the big steps in this process of securing things has been figuring out how to deploy Duo for 2FA SSH on all 600 of our Ubiquiti servers.

I’ve already been using 2FA in a way, by using public key authentication only, an attacker would need to have both the private key and also know the password to decrypt the key before being able to gain access to a server, but Duo adds the phone verification to the mix as well so it is more like 3FA instead of 2FA.

I was happy to find out that Duo is free for under 10 users.

But I quickly upgraded to a paid plan when I realized the free plan does not allow you to create policies — for example, I wanted to enforce using the Duo app, and disable SMS as a 2FA option because it’s not secure. I upgraded to a paid plan which requires a minimum of 10 users at $30/month. No problem with the pricing though, this is a huge security benefit for us.

After purchasing, I was able to change the allowed 2FA methods.

I enrolled some users, connecting the Duo app with each account.

And created a UNIX Application for SSH 2FA.

Next, I spun up a few servers on Vultr and began testing out the 2FA installation and integration process.

There were 2 methods available to integrate Duo, with documentation on each.

login_duo method

pam_duo method

I used the login_duo method because it was easier to set up multiple Duo users sharing a single root account, logging in with public keys instead of passwords, which is what we needed.

Once I had perfected the list of commands needed for the integration from testing manually on fresh servers, it was time to write a Python script to deploy all the commands to our actual 600 Ubiquiti servers.

I open-sourced that script here.

Here’s what the script does.

• Pulls a list of customer server names from our database

• Compares that list to a file “duo_already_done.txt” to see which servers don’t have Duo implemented yet

• For each server that needs Duo implemented, it logs into it via SSH

• It counts the number of SSH keys that already exist in /root/.ssh/authorized_keys to see if there are more than 2. If there are more than 2, it saves the server name to “duo_needs_fixed.txt” which we will later write another script to deal with. Basically, some customers currently have root access, and we are going to remove their root access now and instead give them a limited privilege user so they can still change Ubiquiti files but not server files.

• The rest of the script executes the actual Duo install and integration commands

Closing thoughts

2FA, when implemented properly (no SMS!), is one of the best things we can do to add security to accounts. We also recently published a guide on how you can implement 2FA on your UniFi server here.

https://support.hostifi.net/en/articles/3385881-how-to-setup-2fa-for-unifi

UNMS 2FA can be easily enabled as well under Settings > Users.

If you use UniFi or UNMS, check out our cloud hosting solution at https://hostifi.net! We provide updates, SSL install, monitoring, and support.