Securing Elixir: The Why and How of Paraxial.io

Michael Lubas, 2024-02-07

When a web application is deployed on the internet, it is immediately going to be attacked. Businesses and organizations that rely on these apps have to safeguard the customer information and financial transactions flowing through them every day. Data breaches are a financial and legal nightmare that can be prevented.

This is the problem Paraxial.io solves, preventing data breaches by securing Elixir and Phoenix web applications. Security work should be automated and built into the development process, with tools that communicate risks and remediation paths in clear language. Using Paraxial.io does not require you to be a security expert, the target audience is developers and executives seeking to prevent security incidents.

This article is a long form description of Paraxial.io. If your business is considering using Paraxial.io, and you would like to discuss your own situation and needs, please feel free to schedule a demo today.


Business Risk - The Why

In 2017 Equifax, a credit ratings agency in the United States, suffered a data breach where the private records of over 147 million people were stolen. This led to a $425 million settlement. The root of this incident was a vulnerable web server running Apache Struts, which Equifax was unable to find and upgrade before an attacker broke in. What is apache Struts? The Elixir programming language has the Phoenix framework for creating websites, just as Ruby has Rails, and Java has Struts.

In October 2023 the genetics company 23andMe suffered a credential stuffing attack that led to a data breach. In December 2023, the company stated that approximately 6.9 million users were affected by the hack. Credential stuffing is a type of attack where the bad guy obtains lists of username/password pairs from past breaches. These credentials are automatically entered into the login form of the victim company via a bot, which is able to access users that re-used their password across sites. This is extremely common, most account level security incidents can be traced back to password re-use and credential stuffing.

These two incidents are prefect examples of the types of attacks businesses must deal with every day. In both cases:

  1. The business is negatively impacted (fines, lawsuits)
  2. An internet-facing web application was attacked
  3. The incident could have been prevented

Paraxial.io exists to stop these incidents.


Actionable Security Milestones - The How

Paraxial.io has two sets of features: Application Secure and Bot Defense. Application Secure prevents the web application from being exploited via an OWASP-style attack (SQL injection, remote code execution), and is able to prevent an Equifax type breach. Bot defense is able to ban malicious clients sending automated traffic, that stops a 23andMe type incident.

There are seven application security milestones that can be implemented quickly and correctly with Paraxial.io:

  1. Asset inventory and management
  2. Automated code scans and dependency security
  3. Runtime exploit protection
  4. Traffic rate limiting
  5. Blocking known malicious IPs
  6. Detecting bot IPs with a honeypot
  7. Email reports and Slack alerts

Below is a detailed description of how Paraxial.io is used for each milestone. Speaking from my personal experience, most businesses, including those spending millions per year on security, do not have all these completed. Using Paraxial.io will massively reduce the risk of an incident because these milestones cover the most likely real world attacks you have to deal with.


1. Asset inventory and management

Consider the follow questions:

  1. How many Elixir/Phoenix apps do you currently have deployed?
  2. How many store personally identifiable information (PII)?
  3. How many are exposed to the public internet?
  4. For each deployed app, what version of Elixir, Phoenix, Plug, etc are running?
  5. When was the last time each app was scanned for security issues?

Manually editing a spreadsheet with this information is not a good use of engineering time, the very nature of this work means automation is better than doing it manually.

The Paraxial.io agent is installed in your Elixir project as a normal hex dependency, adding both run-time and compile-time features. When your application starts, the complete list of dependencies loaded at runtime is uploaded to the Paraxial.io backend. This is better than checking a lockfile, because the lockfile checked into Git may not match what is running in production.

The Apache Struts exploit from earlier in this article was available in March 2017. The Equifax hack started in May 2017, and a major cause of the incident was the lack of a comprehensive asset inventory. When a major exploit for the software stack of a business is posted online, the number one priority of that security team suddenly becomes hunting down every relevant server and patching it. With Paraxial.io, you automatically get a full inventory of every dependency running in production:

If these is a remote code execution vulnerability in cowboy v2.10.0 tomorrow, you will know exactly which applications are affected and need to be upgraded. A summary of this information is available on the site overview page:

This page summarizes several Paraxial.io features, including App Audit.


2. Automated code scans and dependency security

For each new code change in your web application, there is the risk of a critical vulnerability being introduced. The Equifax/Struts issue is known as remote code execution (RCE), and was located in the Struts web framework. The Elixir equivalent would be if Phoenix 1.6.4 was vulnerable to RCE. Framework level code is not the only place this issue can be introduced. It can also come from:

  1. Application level source code, the "business logic" that makes up most web applications
  2. Dependency level source code, in Elixir's case installed via the Hex package manager

Paraxial.io is able to scan for both these use cases using open source tooling (Sobelow and HexAudit), and provide additional context for developers. Here is a Sobelow finding for RCE:

Source: Sobelow
Confidence: high_confidence
File: lib/havana_web/controllers/page_controller.ex
Line: 15
Type: Misc.BinToTerm: Unsafe `binary_to_term`
Variable: b

Here is the finding with Paraxial.io context added:

The scans for each site are tracked in the Paraxial.io backend, as evidence that can be provided during compliance audits. SOC 2 Type II requires evidence of code scanning:

Logging into the web interface of a code security tool is most likely not part of an engineer’s normal workflow. Paraxial.io also includes a GitHub App, which runs in CI/CD and provides this feedback right in the pull request:

In some cases, the Paraxial.io bot can even suggest a fix:

Checking each code change for security issues and providing developers with actionable guidance is a requirement for every organization protecting customer data. Paraxial.io automates this process.


3. Runtime exploit protection

The previous milestones cover an automated asset inventory of Elixir apps, and checking all code changes for security issues. Consider the worst case, that even with these checks in place, somehow your production application is currently vulnerable to remote code execution. Worse, an attacker is actively trying to exploit it, either manually or though an automated scanner.

Paraxial.io is able to monitor your application at runtime, detect exploitation attempts, and block the attack. This feature is known as Exploit Guard, and the announcement blog post goes into the technical details. To summarize, a developer connecting to iex or ssh-ing into production will never trigger this alert, because that action does not hit binary_to_term. This will only trigger on an exceptional condition that signals an attack is happening.

If this alert triggers, you are on the edge of an Equifax-style breach, because your application is vulnerable and someone is attacking it. The Paraxial.io Slack App can also send you an alert:

This feature is similar to a fire extinguisher: ideally you never need it, but you want to have it available.

Now that the Application Secure milestones are complete, you have a robust defense against application level hacking attempts. The next milestones are related to Bot Defense.


4. Traffic rate limiting

If your web application receives 100 login attempts in 10 seconds from one IP address, that is obviously malicious and should be banned. Note that this is different from one IP attempting to log into the same account 100 times. Generally what bad guys do is send login attempts for different accounts each time, looking for users who re-used their username and password pair across sites.

Paraxial.io allows you to define rules for how many HTTP requests one IP can send your application in a short period of time. This may seem basic, IP rate limiting has a long history, yet it is not used as frequently as you would expect. When one defensive layer fails, the natural question that comes up after the attack is Why was rate limiting not enabled? While you can do this with open source Elixir libraries, the major benefit of Paraxial.io is context about attacks. Here is an IP that was banned for sending too many requests:

The login history for this IP is only failures, meaning this is a real attack. Now consider a major customer for your business, where thousands of users in the same office share one IP address. During a launch event everyone goes to login to their account, triggering the rate limit. Without Paraxial.io, you will find out about this when the business unit starts pinging engineering asking why things are broken for a major client.

With Paraxial.io, you can get a slack notification about the ban, see successful logins from the client, and add the IP to the allow list.


5. Blocking known malicious IPs

Rate limiting on individual IPs is an excellent defense, and you should have it setup. A well known technique for bypassing IP rate limiting is abusing a cloud provider’s service to send each HTTP request from a different IP. I’ve personally seen this attack work against a company that was paying for an expensive WAF, leading to customer financial accounts being compromised. For a write-up on the technical details, see How attackers bypass IP based rate limiting.

Paraxial.io is able to detect and block this attack when sourced from the following cloud providers:

  1. AWS
  2. Azure
  3. Google Cloud Platform
  4. Digital Ocean
  5. Oracle Cloud

You may want to allow good bots from these IPs, for example an endpoint such as /api/data_export expects bot traffic, compared with /users/log_in, which should always reject bot submissions. Paraxial.io ships with the plug:

Paraxial.BlockCloudIP - https://hexdocs.pm/paraxial/cloud_ips.html

Place it in the Elixir router file matching /users/log_in, and bot traffic will only be blocked on that route. The lookup on each request is extremely fast, your users will not notice any impact. Under the hood it uses a radix trie and persistent_term to get this excellent performance, see Classifying Data Center IP Addresses in Phoenix Web Applications with Radix Trees for the full details.


6. Detecting bot IPs with a honeypot

Paraxial.io is able to detect when traffic is coming from a bot through the use of a honeypot HTML form. These forms cannot be viewed or submitted by normal users, only bots that crawl your website and submit junk data will be detected and banned. Each ban is tracked by Paraxial.io, and you can even receive slack notifications about this event. Below are real bots that were submitting HTML forms on Paraxial.io sites:


7. Email reports and Slack alerts

The site overview page provides a summary of the features discussed here:

Once Paraxial.io is installed and configured, you may not login to the web interface for several days or weeks, because the necessary work is happening automatically. Staying in the loop on this activity is helpful. Did someone disable Paraxial.io scans for a site last week? With email reports, you can receive a weekly summary right in your inbox:

Paraxial.io also has a Slack app, examples of which have been showcased in this article. It currently supports alerting on:

  1. Exploit Guard triggers
  2. IP rate limiting rules
  3. Honeypot ban events

These features allow you to receive security updates and metrics as part of your daily workflow.


Conclusion

Application security work should be automated where possible. Maintaining an asset inventory, records of scans, and how often apps are being checked should not be done manually. Developers should be receiving actionable security guidance when a code change introduces a vulnerability. Attackers that are sending malicious traffic should be banned immediately. The best way to accomplish all this is through a language-native integration, which is why the Paraxial.io agent is written in Elixir, for maximum compatibility and performance.

If you want to secure your web applications, talking to vendors can be a frustrating experience. Do they even support Elixir? Why do they focus on one narrow problem, when there are multiple vectors for attack? Do they really understand your needs? Paraxial.io was created out of a desire to give Elixir teams the best tool available for the job. Thank you for reading.

Paraxial.io stops data breaches by securing your Elixir and Phoenix apps. Detect and fix critical security issues today.

Subscribe to stay up to date on new posts.