Security Best Practices for Deploying Rails 8 on Linux with Kamal
Michael Lubas, 2024-11-05
The best advice I can give as an IT security professional on running your own server: just don’t. - How To Protect Your Linux Server From Hackers!, LiveOverflow
It’s more fun to be competent! - Rails World 2024 Keynote, David Heinemeier Hansson
Kamal is an excellent way to deploy web applications, yet it can be intimidating if you do not have any background securing Linux servers professionally. This post will examine the most relevant risks when deploying an internet facing web application via Docker with Kamal, how to avoid those risks, and provide commentary on how to think about security.
This article assumes:
- You want to deploy a Rails (or Phoenix, Django, etc) application with Kamal on your own server.
- You are not a security expert, but would like to learn more about server security and how to not get hacked.
- Minimal prior knowledge of Linux and Docker. You should be able to open and close a port on a server and run a docker image. If you feel completely lost because you don't have this background, read:
The Goal: Deploy a Rails 8 application with Kamal, ensure the underlying server does not get hacked.
The top three actions that are most likely to lead to a security incident are:
- Exposing a sensitive service (Docker, MySQL, etc) to the public internet.
- Why? Bots are constantly scanning the public internet, if you expose the wrong service, one of them will find it and hack into your server.
- Pushing secrets into Docker Hub.
- Why? If you are using Docker Hub, and do not realize your image is public, you may accidentally upload private keys for SSH, AWS, etc. Bots scan Docker Hub every day looking for exposed credentials.
- Using a weak SSH password.
- Why? Once again, automated bots are constantly scanning the internet, looking for SSH servers with a weak or default password.
This seems like a short list. Where is encryption? Or keeping your software up to date? These topics will be covered, and they are important. My issue with most security advice is that for a list of top 10 risks, they are all treated with the same severity, when there should be bright red text on an entry midway down that says DOUBLE CHECK THIS ONE, YOU WILL GET HACKED IF THIS IS WRONG. The top three risks above all fit that description.
This article is going to exclude vulnerabilities in your Ruby on Rails (or Phoenix, Django, etc) code, for example if you write a controller that takes user input and passes it to the eval function, an attacker can use that vulnerability to hack into the server.
Case Study: Opening Port 2375 and Exposing the Docker Daemon
Recently a post by @rameerez reached over 60k views with the headline “Last night my Kamal server was hacked, and I almost got my Hetzner account blocked as a result”. Kamal is not the root cause of the vulnerability here, however the overall tone of the post is “be careful if you don’t know Docker very well, you might get hacked”. This should not scare you away from self hosting, it is an excellent teaching example and I’m glad rameerez shared it publicly. First lets go through why his server was hacked. I recommend reading the post before continuing.
Why is this not really Kamal’s fault? The key points are:
- The author decided to use a remote server to build the Docker images.
- He used Claude to generate a setup script for this server, which misconfigured Docker to listen on port 2375, facing the public internet. There is a popular GitHub Gist titled "Enable TCP port 2375 for external connection to Docker", which some in the comments speculated is the input that led to Claud recommending this configuration.
- A bot scanning the internet found the server and installed malware to mine Monero and join a botnet. Hetzner disabled the server due to abuse complaints.
So is it Claud’s fault? Docker? The bad guy? A law of physics for the internet: anything you put on the public internet is going to be attacked. Exposing a management interface (like Docker’s port 2375) without authentication is a bad idea, because this is what happens.
Just for fun, I was able to reproduce the vulnerable script with the following prompt:
It’s worth noting that a lot of people leave this Docker management interface open to the public internet:
Attackers deploying new tactics in campaign targeting exposed Docker APIs - June 13, 2024
TeamTNT’s Docker Gatling Gun Campaign - October 25, 2024
Publicly exposed instances on Shodan right now:
The Docker documentation has a page for this:
https://docs.docker.com/engine/daemon/remote-access/
Judging by the above articles, it is a common enough misconfiguration that botnet operators actively use it as a technique.
I searched for “Docker security tips” and these are the first two results:
https://docs.docker.com/engine/security/
- Official Docker documentation.
- Does not sufficiently highlight the risk.
- Should have a giant red warning label DO NOT EXPOSE THE DAEMON ON PORT 2375https://cheatsheetseries.owasp.org/cheatsheets/Docker_Security_Cheat_Sheet.html
- The daemon socket is Rule #1 (technically the second since it's indexed at 0)My issue with the official Docker documentation is that the main security article does not contain actionable information for someone who needs to secure their Docker server. Kernel namespaces, control groups, GRSEC, etc are all important concepts, but the official docs should highlight that the most important item is: DO NOT EXPOSE THE DOCKER DAEMON TO THE PUBLIC INTERNET.
The good news is a proper firewall configuration in your cloud provider means that even if you accidentally expose this port to the public internet on your Linux server, traffic will not be able to reach it.
Risk 1: Exposing a sensitive service (Docker, MySQL, etc) to the public internet
Solution: Use your cloud provider’s firewall service to only open port 80, 443, and 22.
In general the only ports you need open to the public internet are 80 and 443 for HTTP traffic. For port 22, restrict traffic to a list of IP addresses you connect from (your home, office, a VPN, etc). There are several benefits to doing this:
- The number of people in the world who can attempt to SSH into your server goes from millions to very few.
- No spam in your logs from people attempting to break into your server via SSH.
- Even if there is a vulnerability that allows people to break in via SSH, the probability of it being exploited is now far lower.
The reason I prefer using the cloud provider’s firewall over the Linux settings is that even if you screw up and accidentally expose your Docker daemon, nobody can connect to it. This is by far the best action you can take to reduce the risk of being hacked. The name of this feature varies by your cloud vendor:
Kamal is famously related to 37Signal’s Cloud Exit, and David’s post mentions using Deft to rack the machines in bring them online. This filtering can also be done on data center hardware, but that’s outside the scope of this post.
If you saw the RailsWorld keynote, you may be wondering about ufw from this slide:
The Uncomplicated Firewall (ufw) is also effective for closing ports. Just beware that if you override it, and you’ve remove the safety bumpers of the cloud layer firewall, you will be vulnerable. You saw this earlier with the script Claude generated to open ports. The setup video currently on the Kamal website warns about accidentally exposing MySQL port 3306 to the public internet in your deploy.yml file (13:25):
When writing this post I generated a new Rails 8.0.0.rc2 app, and the value was set to the safe version, "127.0.0.1:3306:3306". So the new deploy.yml is secure by default, but you can see how setting your cloud firewall to keep this port closed provides a nice extra layer of security.
What if you need to access your database though? You should not be exposing 3306 to the public internet in most cases, rather you want to use SSH tunneling.
Risk 2: Pushing secrets into Docker Hub
Solution: Do not push a codebase that contains hardcoded secret values to Docker Hub (or any public container registry).
Today, Kamal requires a container registry, which defaults to Docker Hub.
We’re working very hard, by the way, to get rid of this step. - Rails World 2024 Keynote (55:24)
Most developers today are aware that you should not push secrets to a public GitHub repository. However, consider the following situation:
- You have a private GitHub repository with some sensitive data in it
- You decide to use Kamal
- You don't realize that Docker Registry defaults to public for your image
Rob Zolkos on X ran into this exact situation, and posted a good warning:
If you’re curious how long it takes before a credential leaked on Docker Hub is accessed, the answer is about 7 days:
If you are thinking “No problem, I’ll just make sure to take the repo down before day 6”, please do not do that. As soon as a credential is leaked publicly, you must treat it is compromised and rotate it.
Risk 3: Using a weak SSH Password
Solution: Disable SSH password login, use a public/private key pair.
DHH mentions this during the Rails World keynote at 35:40.
The LiveOverflow video mentions this topic at 1:25.
I’m inclined to agree with David here. The tone of LiveOverflow on this topic is roughly “this does not magically make your server more secure compared with a password login”. And while you technically can have a secure SSH setup with a custom password, it introduces unnecessary risk.
By setting PasswordAuthentication no, you completely eliminate the risk of a weak default password. Combined with a restricted list of IPs that are able to use SSH, this massively reduces the risk of an incident.
After disabling password logins and restarting SSH, double check the setting worked by attempting to login via a password. You may need to perform some additional configuration.
Do bad guys scan the internet every day, checking for servers with weak SSH passwords? Yes.
Fail2Ban?
Fail2Ban scans log files like /var/log/auth.log and bans IP addresses conducting too many failed login attempts.
Do you need Fail2Ban? If only a few IP addresses in the world can login via SSH, no. If you must keep your SSH port open to the public internet, it’s useful to keep your authentication log files free of spam. If you left password logins on (not recommended!) and made your password something like “myserver”, it might stop that attack, but you should not be doing this. PasswordAuthentication no is the best way to do this by far.
Now that we have covered the top three most important risks, and proper mitigations, the rest of this post will give light treatment to some common best practices.
Logging and Slack Alerts
Uploading the SSH access logs, and getting notified on new login events, is a great practice. While it doesn’t necessarily prevent a hack, it’s incredibly useful for detecting and stopping one, like monitoring your bank statement for unauthorized transactions.
The SSH log is usually found in:
- /var/log/auth.log (Debian, Ubuntu)
- /var/log/secure (Red Hat, CentOS, Fedora)
And can be uploaded to your preferred log management tool:
App Signal - https://docs.appsignal.com/logging.html
New Relic - https://docs.newrelic.com/docs/logs/get-started/get-started-log-management/
Sending a notification to Slack or Discord when someone logs in is also recommended.
Keeping Software Up to Date
You should be using the most recent software for your server, because old versions of Linux and the related libraries do contain security problems. For example, the reason you don’t want to run a version of OpenSSL (included with Linux) is that it is most likely vulnerable to Heartbleed.
This is a big topic, so to be brief, you should regularly update your Linux server and container image. Running very old versions of any software facing the public internet is a bad idea.
Ensure your application is using HTTPS
This is an article on Kamal security, so where is the discussion of TLS and Lets Encrypt? The reality is data breaches are rarely caused by misconfigured HTTPS settings. Of course you should not allow login over HTTP to your application, but most developers already know this, so it’s rare to find an application missing HTTPS today. If you would like to test that you have HTTPS setup correctly, use the SSL Server Test Tool.
To close out this article, lets review the video “How To Protect Your Linux Server From Hackers!” by LiveOverflow.
Disable SSH password login
Covered above, you should disable password login and use keys instead.
Disable Direct root SSH login
I agree with the video, disabling root login will not make much of a difference. Your goal should be to prevent the bad guy from getting SSH access, not worrying about the permission settings on a compromised box.
Change default SSH port
Agree with the video, this is not useful and gives a false sense of security.
Disable IPv6 for SSH
Agree with the video, disabling IPv6 is not a real security control.
Setup a basic firewall
At 13:55 it’s a very similar screen to the Rails World keynote:
The point of the video that if your server is only listening on 80 and 443, adding a firewall does not technically improve security. He calls it a snake oil recommendation, which I think is a mistake. His point is subtle, that as long as you know what you’re doing when opening ports, the firewall is redundant. The reality is the firewall protects you against a configuration mistake, like a guard rail on the side of a road preventing cars from falling off a cliff.
For example, what if you make a mistake configuring your server and open port 3306? Even with ufw there is a chance that Kamal would override that setting if you used the wrong value in deploy.yml. Remember the Docker daemon on port 2375? Adding a firewall at the cloud provider layer (recommended earlier in the article) is what 99% of people hosting web applications should be doing, and would have prevented the hack that happened.
Auto Updates
I agree with the video, auto-updates sound nice in theory, in reality they will break things. You need to test system updates to ensure your application actually runs on the new versions.
Third Party Help
Security is complicated, and the majority of businesses look to external firms to validate their own systems. Paraxial.io offers both offensive (hire someone to hack your network) and defensive (hire someone to help secure it) consulting services.
Thank you for reading, I hope this article encourages you to host your next project with Kamal and learn more about security.
Author Contact: michael@paraxial.io
Paraxial.io stops data breaches by helping developers ship secure applications. Get a demo or start for free.