Throttling and Blocking Bad Requests in Phoenix Web Applications with PlugAttack

Web applications that accept username and password pairs for authentication may experience credential stuffing by malicious clients. We use the term “credential stuffing” to refer to the act of using credentials, taken from a website’s public data breach, to preform many authentication attempts against victim accounts on a different website. This tutorial will demonstrate how to mitigate credential stuffing against a Phoenix Framework application, using PlugAttack.

1. Setup victim application, Orru

The victim web application for this tutorial is named orru, created with Elixir 1.12 and Phoenix 1.6. You must have both Elixir and the Phoenix Framework installed locally to run orru. See the appropriate guide for your operating system. PostgreSQL is always required, see the official wiki for installation instructions.

To check that your local environment is setup correctly, open the “Up and Running” page from the official Phoenix documentation, and follow the instructions to run the hello application locally. Running mix phx.server from the hello directory, then navigating to http://localhost:4000 in your web browser, should produce the “Welcome to Phoenix!” page. This confirms your local environment is configured for this tutorial.

Open a new terminal, change to the directory where you want orru to be located, then use git to get a copy of the orru source code:

git clone https://github.com/paraxialio/orru.git

Run the following commands to setup orru:

cd orru
mix deps.get
mix ecto.setup

Start the server:

mix phx.server

Open http://localhost:4000 in your web browser, and you should see a page that is similar to the hello application you setup earlier. The difference is the “Register” and “Log in” text in the upper right hand corner, indicating that orru has login functionality. The authentication code for orru was created with the mix phx.gen.auth mix task.

Create a user to ensure the authentication system is working correctly, with the credentials:

Email    - crow@example.com
Password - crowSecret2022 

After completing registration, you should be taken to http://localhost:4000/ and see the message “User created successfully.” In the upper right hand corner of the page, “crow@example.com” should be visible, indicating you are now logged in. You have now completed the setup of orru, the victim web application for this tutorial.

2. Introducing the credential stuffing script, Envy

The credential stuffing program you use in this tutorial, envy, takes a list of username/password pairs as input, and attempts to log into each pair. Sending too many requests in a short period of time may overwhelm the victim server, so we introduce a rate limit argument to the program, which is used to determine how many login requests to send in a period of 1 minute.

Open a new terminal, change to the directory where envy will be located, then use git to download envy:

git clone https://github.com/paraxialio/envy.git

Run the following commands to setup envy:

cd envy
mix deps.get

Run the do_ato.exs script with no arguments:

envy % mix run do_ato.exs 
Account Takeover Tool v1.0
Example usage, default rate limit (60 requests per minute):
mix run do_ato.exs credentials_10.txt
Example usage, custom rate limit (100 requests per minute): 
mix run do_ato.exs credentials_10.txt 100

The do_ato.exs script takes two arguments. The first is a file containing username/password pairs, for use in credential stuffing. The second argument is an integer that represents the maximum number of HTTP requests per minute envy will send.

The command:

mix run do_ato.exs credentials.txt 60

means, “Use the username/password pairs in credentials.txt to send login requests at a maximum rate of 60 per minute”.

envy sends login requests to http://localhost:4000, and performs the necessary steps to send a POST request with a CSRF token, so the request will succeed. Now that you have envy and orru installed and running locally, let’s continue.

3. How to run Envy

Now you will be able to see an account takeover attack from both sides, as the person performing it and as the web application owner.

Ensure orru is running in one terminal:

orru % mix phx.server

Now switch to a different terminal, change to the envy directory, and run:

mix run do_ato.exs credentials_10.txt

If everything is setup correctly, the output will be:

envy % mix run do_ato.exs credentials_10.txt
2022-01-31 21:11:57 crow0@example.com/crowSecret99  Login POST status 200
2022-01-31 21:11:58 crow1@example.com/crowSecret98  Login POST status 200
2022-01-31 21:11:59 crow2@example.com/crowSecret97  Login POST status 200
2022-01-31 21:12:00 crow@example.com/crowSecret2022  Login POST status 302
2022-01-31 21:12:01 crow3@example.com/crowSecret96  Login POST status 200
2022-01-31 21:12:02 crow4@example.com/crowSecret95  Login POST status 200
2022-01-31 21:12:03 crow5@example.com/crowSecret94  Login POST status 200
2022-01-31 21:12:04 crow6@example.com/crowSecret93  Login POST status 200
2022-01-31 21:12:05 crow7@example.com/crowSecret92  Login POST status 200
2022-01-31 21:12:06 crow8@example.com/crowSecret91  Login POST status 200

Only one pair of credentials resulted in a successful login. The server responds with a 200 status code on a failed login because the HTTP POST request itself is successful, but the HTML response from the server indicates the login failed. When the login request is successful, the server replies with a 302.

The credentials_10.txt file only contains 10 pairs. Let’s increase the maximum number of requests per minute to 500, and use the credentials_100.txt file.

envy % mix run do_ato.exs credentials_100.txt 500
2022-01-31 21:12:33 crow0@example.com/crowSecret0  Login POST status 200
2022-01-31 21:12:33 crow1@example.com/crowSecret1  Login POST status 200
...
2022-01-31 21:13:26 crow98@example.com/crowSecret98  Login POST status 200
2022-01-31 21:13:26 crow@example.com/crowSecret2022  Login POST status 302

One client attempting hundreds of logins per minute is a clear sign of malicious intent. We will now add the PlugAttack project to orru, and examine different strategies for dealing with bad clients.

4. Adding PlugAttack to Orru

PlugAttack is a, “toolkit for blocking and throttling abusive requests”. To use it effectively, you should be familiar with the Plug project, the conn struct, and how Phoenix projects use plugs. Open the Github repo README and official documentation, then read through the examples. This should give you an idea of how the project is intended to be used. Now let’s add it to the orru project.

To add PlugAttack as a dependency, open orru/mix.exs and add one line:

  defp deps do
    [
      {:bcrypt_elixir, "~> 2.0"},
      {:phoenix, "~> 1.6.0"},
	  ...
      {:plug_cowboy, "~> 2.5"},
      {:plug_attack, "~> 0.4.3"} # <- Add this line
    ]
  end

Now add storage to the supervision tree with a child specification. Open orru/lib/orru/application.ex and add:

  def start(_type, _args) do
    children = [
      # Add this line
      {PlugAttack.Storage.Ets, name: Orru.PlugAttack.Storage, clean_period: 60_000},
      # Start the Ecto repository
      Orru.Repo,

Run mix deps.get, then run orru to check everything is working correctly.

5. Throttle login requests per IP address

Now define the PlugAttack plug, with a rule that limits the number of login requests one IP address can make to 10 per minute.

Create a new file, orru/lib/orru/plug_attack.ex, and enter the following:

defmodule Orru.PlugAttack do
  use PlugAttack
  rule "throttle login requests", conn do
    if conn.method == "POST" and conn.path_info == ["users", "log_in"] do
      throttle conn.remote_ip,
        period: 60_000, limit: 10,
        storage: {PlugAttack.Storage.Ets, Orru.PlugAttack.Storage}
    end
  end
end

The throttle rule is defined, but incoming requests need a way to reach it. Open orru/lib/orru_web/router.ex, define a new pipeline called :plug_attack, and add Orru.PlugAttack to it. Finally, add :plug_attack to the Authentication routes in the router.

defmodule OrruWeb.Router do
  use OrruWeb, :router
  import OrruWeb.UserAuth
  pipeline :browser do
    ...
  end
  
  # Add this pipeline to the router
  pipeline :plug_attack do
    plug Orru.PlugAttack
  end
  pipeline :api do
    plug :accepts, ["json"]
  end
  ...
  ## Authentication routes
  scope "/", OrruWeb do
    # Edit this line to include plug_attack
    pipe_through [:browser, :redirect_if_user_is_authenticated, :plug_attack] 
    get "/users/register", UserRegistrationController, :new
	...

Use the envy program to test if this rule works. Run orru with mix phx.server, then open envy in a different terminal and run these commands:

envy % mix run do_ato.exs credentials_10.txt 500
2022-01-31 21:23:05 crow0@example.com/crowSecret99  Login POST status 200
2022-01-31 21:23:05 crow1@example.com/crowSecret98  Login POST status 200
2022-01-31 21:23:05 crow2@example.com/crowSecret97  Login POST status 200
2022-01-31 21:23:05 crow@example.com/crowSecret2022  Login POST status 302
2022-01-31 21:23:05 crow3@example.com/crowSecret96  Login POST status 200
2022-01-31 21:23:05 crow4@example.com/crowSecret95  Login POST status 200
2022-01-31 21:23:05 crow5@example.com/crowSecret94  Login POST status 200
2022-01-31 21:23:05 crow6@example.com/crowSecret93  Login POST status 200
2022-01-31 21:23:06 crow7@example.com/crowSecret92  Login POST status 200
2022-01-31 21:23:06 crow8@example.com/crowSecret91  Login POST status 200
envy % mix run do_ato.exs credentials_10.txt 500
2022-01-31 21:23:09 crow0@example.com/crowSecret99  Login POST status 403
2022-01-31 21:23:09 crow1@example.com/crowSecret98  Login POST status 403
2022-01-31 21:23:09 crow2@example.com/crowSecret97  Login POST status 403
2022-01-31 21:23:09 crow@example.com/crowSecret2022  Login POST status 403
2022-01-31 21:23:09 crow3@example.com/crowSecret96  Login POST status 403
2022-01-31 21:23:09 crow4@example.com/crowSecret95  Login POST status 403
2022-01-31 21:23:09 crow5@example.com/crowSecret94  Login POST status 403
2022-01-31 21:23:09 crow6@example.com/crowSecret93  Login POST status 403
2022-01-31 21:23:10 crow7@example.com/crowSecret92  Login POST status 403
2022-01-31 21:23:10 crow8@example.com/crowSecret91  Login POST status 403

You have successfully limited the number of requests one IP address can send to 10 per minute.

6. How PlugAttack uses ETS (Erlang Term Storage)

Earlier you configured storage for PlugAttack, with the name Orru.PlugAttack.Storage. When an incoming request matches a rule you defined, it is placed in Erlang Term Storage, or ETS.

Run orru with iex -S mix phx.server. Then run:

iex(2)> :ets.tab2list(Orru.PlugAttack.Storage)
[]

The output should be empty, because you did not send any login requests. Now make one failed login request in orru, through your web browser, then run tab2list again:

iex(5)> :ets.tab2list(Orru.PlugAttack.Storage)
[{{:throttle, {127, 0, 0, 1}, 27391530}, 1, 1643491860000}]

PlugAttack stores details about requests that match the rules you define. Now send 100 login requests with envy:

envy % mix run do_ato.exs credentials_100.txt 500
...

Then check ETS:

iex(8)> :ets.tab2list(Orru.PlugAttack.Storage)
[{{:throttle, {127, 0, 0, 1}, 27391533}, 101, 1643492040000}]

This is how PlugAttack keeps track of requests that match the rule you defined. Your output may be different if the 60 second period ended while the requests were running.

7. Multiple rules and blocking IP addresses

Now that you have completed a basic example of throttling login requests, consider a more complex requirement.

1. Limit login requests from one IP address to 10 per minute 
2. If one IP does 50 login requests in a minute, ban it for seven days (one week)

Your first idea may be to add the following to plug_attack.ex, after the “throttle login requests” rule:

# This is wrong
rule "block login requests if over 50 in 60 seconds", conn do
  if conn.method == "POST" and conn.path_info == ["users", "log_in"] do
    fail2ban conn.remote_ip,
      period: 60_000, limit: 50, ban_for: 60_000 * 60 * 24 * 7,
      storage: {PlugAttack.Storage.Ets, Orru.PlugAttack.Storage}
  end
end

To demonstrate why this rule is wrong, you may add it in plug_attack.ex, then run orru and send 100 login requests with:

envy % mix run do_ato.exs credentials_100.txt 500

Check ETS:

iex(2)> :ets.tab2list(Orru.PlugAttack.Storage)
[
  {{:throttle, {127, 0, 0, 1}, 27391548}, 27, 1643492940000},
  {{:throttle, {127, 0, 0, 1}, 27391547}, 73, 1643492880000}
]

The reason for 27 and 73 is because the script was running as the clock changed minutes. This demonstrates the fail2ban rule was never matched, only the throttle rule. If you move the fail2ban rule above the throttle rule, only fail2ban will match, meaning an attacker can send 49 requests per minute without being throttled.

A new plug for the fail2ban rule will fix this problem. Create orru/lib/orru/plug_attack_fail2ban.ex and enter:

defmodule Orru.PlugAttackFail2Ban do
  use PlugAttack
  @week 60_000 * 60 * 24 * 7 
  rule "fail2ban on login", conn do
    if conn.method == "POST" and conn.path_info == ["users", "log_in"] do
      fail2ban conn.remote_ip,
        period: 60_000, limit: 50, ban_for: @week,
        storage: {PlugAttack.Storage.Ets, Orru.PlugAttack.Storage}
    end
  end
end

Now we need to add the PlugAttackFail2Ban plug to our router. The order here is important. Remember we want to have two rules in place - only allow 10 login requests per minute and if one IP does 50 requests in a minute, ban for a week. Let’s consider the wrong order first:

pipeline :plug_attack do
 # This order is wrong
 plug Orru.PlugAttack
 plug Orru.PlugAttackFail2Ban
end

The problem is that after 10 requests in a 60 second period, the conn of all future requests will be marked as throttled, and not count toward the fail2ban rule. The client will be throttled, but never banned for a week.

With orru running, and the plug order wrong, do:

envy % mix run do_ato.exs credentials_100.txt 500

Then check ETS:

iex(5)> :ets.tab2list(Orru.PlugAttack.Storage)
[
  {{{:fail2ban, {127, 0, 0, 1}}, 1643495107200}, 0, 1643495167200},
  {{{:fail2ban, {127, 0, 0, 1}}, 1643495107325}, 0, 1643495167325},
  {{{:fail2ban, {127, 0, 0, 1}}, 1643495106755}, 0, 1643495166755},
  {{{:fail2ban, {127, 0, 0, 1}}, 1643495106838}, 0, 1643495166838},
  {{:throttle, {127, 0, 0, 1}, 27391585}, 100, 1643495160000},
  {{{:fail2ban, {127, 0, 0, 1}}, 1643495106735}, 0, 1643495166735},
  {{{:fail2ban, {127, 0, 0, 1}}, 1643495107081}, 0, 1643495167081},
  {{{:fail2ban, {127, 0, 0, 1}}, 1643495107696}, 0, 1643495167696},
  {{{:fail2ban, {127, 0, 0, 1}}, 1643495107446}, 0, 1643495167446},
  {{{:fail2ban, {127, 0, 0, 1}}, 1643495107569}, 0, 1643495167569},
  {{{:fail2ban, {127, 0, 0, 1}}, 1643495106967}, 0, 1643495166967}
]

10 requests matched :fail2ban, then 100 matched :throttle.

Change orru/lib/orru_web/router.ex so the plugs are in the correct order:

# Correct order
pipeline :plug_attack do
  plug Orru.PlugAttackFail2Ban
  plug Orru.PlugAttack
end

Then send 100 login requests:

envy % mix run do_ato.exs credentials_100.txt 500

Check ETS:

iex(4)> :ets.tab2list(Orru.PlugAttack.Storage)
[
  {{{:fail2ban, {127, 0, 0, 1}}, 1643495266511}, 0, 1643495326511},
  ...
  {{:fail2ban_banned, {127, 0, 0, 1}}, true, 1644100066630},
  {{:throttle, {127, 0, 0, 1}, 27391587}, 50, 1643495280000},
  ...
]

This fulfills our original requirements:

1. Limit login requests from one IP address to 10 per minute 
2. If one IP does 50 login requests in a minute, ban it for seven days (one week)

Careful readers of PlugAttack’s documentation may have noticed we used the same storage for both our rules. The documentation says, “Be careful not to use the same key for different rules that use the same storage”.

To demonstrate how reuse of storage can cause problems, let’s add another throttle rule in orru/lib/orru/plug_attack.ex

defmodule Orru.PlugAttack do
  use PlugAttack
  rule "throttle login requests", conn do
    if conn.method == "POST" and conn.path_info == ["users", "log_in"] do
      throttle conn.remote_ip,
        period: 60_000, limit: 10,
        storage: {PlugAttack.Storage.Ets, Orru.PlugAttack.Storage}
    end
  end
  
  # Should not be using this storage
  rule "throttle register page GETs", conn do
    if conn.method == "GET" and conn.path_info == ["users", "register"] do
      throttle conn.remote_ip,
        period: 60_000, limit: 20,
        storage: {PlugAttack.Storage.Ets, Orru.PlugAttack.Storage}
    end
  end
end

In your web browser, go to http://localhost:4000/users/register, then refresh the page 9 times. Then attempt to login. Your attempt will fail, because any throttle rule for conn.remote_ip will increment the same :throttle tuple in ETS.

iex(8)> :ets.tab2list(Orru.PlugAttack.Storage)
[
  {{:throttle, {127, 0, 0, 1}, 27391617}, 11, 1643497080000}
]

When PlugAttack checks Orru.PlugAttack.Storage, :throttle has been incremented to 11, so any throttle rule for the key conn.remote_ip will have count. Defining an additional storage in application.ex will allow you to avoid this problem.

8. Conclusion and Future Work

This article does not cover deployment of a Phoenix application. One important note for anyone using PlugAttack is that conn.remote_ip will likely not reflect the real client IP. Use the remote_ip library to fix this.

ETS is reset when you deploy your application, so the seven day ban example from this article may not work if you do frequent deploys. Writing login history to Postgres, then using that data to make throttling decisions may be a better solution. You may also wish to throttle on failed login requests only, instead of on all logins.

PlugAttack supports callbacks that are triggered when a request is allowed or blocked. This is a powerful feature, for example if you want to record statistics about how many requests are being blocked, block_action/3 could be used to trigger writes to the database.

Thank you to Michał Muskała for creating PlugAttack.

---

Paraxial.io stops data breaches by securing your Elixir and Phoenix apps. Detect and fix critical security issues today.