has_many :codes

Protecting a Rails app from small scripted attacks

Published  

I don’t know about you, but I find it really annoying when I see lots and lots of entries in the logs of my app for failed login attempts against Wordpress or things like that. Before switching to Kubernetes, I would usually configure Fail2ban to stop these scripted attacks before they’d reach my apps and ban the offending IPs for an extended period of time, but with Kubernetes I found this is a bit trickier. One option is to add some config snippets to the ingress controller so to block these requests at the ingress level. That would work, and perhaps it would be more efficient than handling this at the app level, but by letting the app manage these small scripted attacks I can have more control on the configuration of the response to each type of attack, as well as receive notifications for example via email whenever an attack is stopped. One great option to handle all of this at the application level with Rails is the rack-attack gem by Kickstarter.

rack-attack is a Rack middleware for blocking & throttling abusive requests, and works pretty well. Because it’s a middleware, it’s pretty lightweight and doesn’t impact on performance noticeably, because it stops attacks before the full Rails framework is loaded (at least this is my understanding). Using the gem is very easy, and since I have just added it to an app I’m working on, here’s how I configured it and also how I added email notifications for when attacks are stopped.

For starters, you need to add the rack-attack gem to your Gemfile, as well as the dalli gem which is required to use memcached as the cache store. In fact, rack-attack can keep track of the requests by IP or other parameter using the Rails cache. By default it would use the in-memory cache store, but in production you definitely want to use a distributed cache so each instance of your application has access to the same rack-attack tracking.

gem 'rack-attack'
gem "dalli"

Once you have run bundle install, you need to add an initializer e.g. in config/initializers/rack_attack.rb with the configuration for rack-attack. I’ll paste here my current configuration, and then I’ll go through what it does:

require 'ipaddr'

class Rack::Attack
  class Request < ::Rack::Request
    def remote_ip
      @remote_ip ||= (env['HTTP_CF_CONNECTING_IP'] ||
                      env['action_dispatch.remote_ip'] ||
                      ip).to_s
    end

    def allowed_ip?
      allowed_ips = ["127.0.0.1", "::1"]
      allowed_ips.include?(remote_ip)
    end
  end

  safelist('allow from localhost') do |req|
    req.allowed_ip?
  end

  blocklist("fail2ban") do |req|
    Rack::Attack::Fail2Ban.filter("fail2ban-#{req.remote_ip}", maxretry: 1, findtime: 1.day, bantime: 1.day) do
      CGI.unescape(req.query_string) =~ %r{/etc/passwd} ||
        req.path.include?("/etc/passwd") ||
        req.path.include?("wp-admin") ||
        req.path.include?("wp-login") ||
        /\S+\.php/.match?(req.path)
    end
  end

  throttle("limit logins per email", limit: 5, period: 20.seconds) do |req|
    if req.path == "/users/sign_in" && req.post?
      if (req.params["user"].to_s.size > 0) and (req.params["user"]["email"].to_s.size > 0)
        req.params["user"]["email"]
      end
    end
  end

  throttle("limit signups", limit: 5, period: 1.minute) do |req|
    req.remote_ip if req.path == "/users" && req.post?
  end

  # Exponential backoff for all requests to "/" path
  #
  # Allows 240 requests/IP in ~8 minutes
  #        480 requests/IP in ~1 hour
  #        960 requests/IP in ~8 hours (~2,880 requests/day)
  (3..5).each do |level|
    throttle("req/ip/#{level}",
               limit: (30 * (2 ** level)),
               period: (0.9 * (8 ** level)).to_i.seconds) do |req|
      req.remote_ip # unless req.path.starts_with?('/assets')
    end
  end
end

ActiveSupport::Notifications.subscribe(/rack_attack/) do |name, start, finish, request_id, payload|
  req = payload[:request]

  request_headers = { "CF-RAY" => req.env["HTTP_CF_RAY"] }

  Rails.logger.info "[Rack::Attack][Blocked] remote_ip: #{req.remote_ip}, path: #{req.path}, headers: #{request_headers.inspect}"

  AdminMailer.rack_attack_notification(name, start, finish, request_id, req.remote_ip, req.path, request_headers).deliver_later
end

The first thing I do is configure how rack-attack must determine the IP address of the client. By default you’d use request.ip, but the request object in the context of the rack-attack middleware is not as complete as the request object you have access to in the Rails app normally. This means that if your app is behing Cloudflare or other proxy or load balancer, the IP address “seen” by rack-attack may actually be the IP address of the proxy/load balancer, not the IP of the client, so this would basically not work. In my case I’m using Cloudflare as proxy, so I need to tell rack-attack to look at the HTTP_CF_CONNECTING_IP header for the real IP of the client, if that header is present, or try to get the IP from action_dispatch. Otherwise it falls back to the default method used to determine the IP. I also add a method to define some IPs to whitelist, such as localhost’s IPs.

Next, I configure some rules that tell rack-attack how to behave when it processes requests. First I use the allowed_ips? method to whitelist those IPs:

Then I use a filter modeled on Fail2ban which will ban any IP that makes requests for /etc/password, or login attempts against Wordpress, or any request to PHP pages, which are common targets of scripted attacks. Of course this is a Rails app, so we can just ban for an extended period of time any offending IPs. Here I am instructing rack-attack to ban for one day any IP that makes a single such request within one day. It may sound strict, but my app is not expecting this kind of requests so they can only have malicious purposes.

The next two rules are for Devise, in order to limit sign in attempts as well as sign ups. Again, the number of max requests and the period of time within rack-attack should look for failures are configurable. I borrowed the final configuration from somewhere (I didn’t take a note of where, sorry); it enables exponential backoff so that requests are throttled for an increasing amount of time. This is to make the throttling more effective.

As you can see in the final block of code of the initializer, I am using ActiveSupport::Notifications both to log any failure to the Rails log, and to send myself an email notification each time. I like this to keep an eye on what’s happening and also to make sure rack-attack is working as expected.

The mailer (app/mailers/admin_mailer.rb for example) is simple:

class AdminMailer < ApplicationMailer
  default from: "...", to: "..."

  def rack_attack_notification(name, start, finish, request_id, remote_ip, path, request_headers)
    @name = name
    @start = start
    @finish = finish
    @request_id = request_id
    @remote_ip = remote_ip
    @path = path
    @request_headers = request_headers.inspect

    mail(subject: "[Rack::Attack][Blocked] remote_ip: #{remote_ip}")
  end
end

The email template (app/views/admin_mailer/rack_attack_notification.html.erb) is even simpler:

<h1>[Rack::Attack][Blocked] </h1>
<p>Name: <%= @name %></p>
<p>IP: <%= @remote_ip %></p>
<p>Path: <%= @path %></p>
<p>Start: <%= @start %></p>
<p>Finish: <%= @finish %></p>
<p>Request ID: <%= @request_id %></p>
<p>Headers: <%= @request_headers %></p>

One final piece of configuration, if you want to use memcached as the cache store used by rack-attack, is this line in production.rb:

  config.cache_store = :mem_cache_store, "memcache-host.example.com"

That’s it! rack-attack should now block attacks or throttle requests according to the rules you’ve defined, and you should receive an email notification when this happens. Very easy, right? Well, you need to keep in mind that this is to stop simple, scripted attacks. To stop any sophisticated attacks that might be targeting your app with purpose, it’s always a good idea to have the app behind a service like Cloudflare, which can help mitigate real DDoS attacks and alike. Nevertheless, blocking some annoying small attacks doesn’t hurt anyway. Hope this helps!

© Vito Botta