Recently we discovered that a customer’s website was being attacked in what is best described as a “slow DoS”. The attacker was running a script that scraped each page of the site to find possible PDF files to download, then was initiating many downloads of each file.

Because the site was fronted by a Content Delivery Network (CDN), the site itself was fine and experienced no increase in load or service disruption, but it did cause a large spike in bandwidth usage between the CDN and the clients. The increase in bandwidth was significant enough to increase the monthly charge from around NZ$1,500 to over NZ$5,000. Every time the customer banned the IP address that was sending all the requests, a new IP would appear to replace it. It seems the point of the attack was to waste bandwidth and cost our customer money — and it was succeeding.

The site itself was hosted in AWS on an EC2 instance, however the CDN service the site was using was a third party — Fastly. After some investigation, it seemed that Fastly didn’t have any automated mitigation features that would stop this attack. Knowing that AWS Web Application Firewall (WAF) has built in rate-based rules we decided to investigate whether we could migrate the CDN to CloudFront and make use of these rules.

All we needed to do was create a CloudFront distribution with the same behaviour as the Fastly one, then point the DNS records to CloudFront — easy right? Fastly has a neat feature that allows you to redirect at the edge which was being used to redirect the apex domain to the www subdomain — if we were to replicate this behaviour in CloudFront we would need some extra help, but first we needed to make sure we could make the required DNS changes.

To point a domain at CloudFront that is managed by Route 53 is easy, you can just set an ALIAS record on the apex domain and a CNAME on the www subdomain. However, this customers DNS was managed by a third-party provider who they were committed to sticking with (this is a blog post for another day). The third-party provider did not support ALIAS or ANAME records and insisted that apex domains could only have A records — that meant we could only use IP addresses!

Because CloudFront has so many edge locations (108 at the time of writing), it wasn’t practical to get a list of all of them and set 108 A records — plus this would require activating the “static IP” feature of CloudFront which gives you a dedicated IP for each edge location, which costs around NZ$1,000 a month.

And to top all that off, whatever solution we decided to use would only be in place for 2 months as the site was being migrated to a fully managed service. We needed a solution that would be quick and easy to implement — AWS to the rescue!

So, we had three choices:

1. Stay with Fastly and figure out how to ban the bad actors
2. Move to CloudFront and figure out the redirect (bearing in mind we only had A records to work with)
3. Do nothing and incur the NZ$5,000 cost each month — high risk if the move to a managed service ended up being delayed. We decided this wasn’t really an option.

We considered spinning up a reverse proxy and pointing the apex domain at it to redirect to the www subdomain (remember, we couldn’t use an S3 bucket because we could only set A records) but decided against this approach because we’d need to make the reverse proxy scalable given we’d be introducing it in front of the CDN during an ongoing DoS attack. Even though the current attack was slow, it could have easily been changed into something more serious.

We decided to stay with Fastly and figure out how to automatically ban IP addresses that were doing too many requests. Aside from the DNS limitation, one of the main drivers for this decision was inspecting the current rate of the DoS — it was so slow that it was below the minimum rate-based rule configuration that the AWS WAF allows (2,000 requests in 5 minutes). We needed to write our own rate-based rules anyway, so using CloudFront and WAF didn’t solve our problems straight away.

Thankfully, Fastly had an API that we could hit with a list of bad IPs — so all we needed to figure out was:

1. Get access to the Fastly logs,
2. Parse the logs and count the number of requests,
3. Auto-ban the bad IPs.

Because Fastly allows log shipping to S3 buckets, we configured it to ship to our AWS account in a log format that could be easily consumed by Athena, and wrote a couple of AWS Lambda functions that:

1. Queried the Fastly logs S3 bucket using Athena,
2. Inspected the logs and banned bad actors by hitting the Fastly API, maintaining state in DynamoDB,
3. Built a report of bad IPs and ISPs and generated a complaint email.

The deployed solution looked something like this:

By leveraging S3, Athena, Lambda and DynamoDB we were able to deploy a fully serverless rate-based auto-banner for bad actors with a very short turnaround. The customer was happy with this solution as it avoided having to incur the $5000 NZD / month cost, avoided needing to change the existing brittle DNS setup and also provided some valuable exposure into how powerful serverless technology on AWS is.

It’s implementing solutions like this that helps set Consegna apart from other cloud consultancies — we are a true technology partner and care deeply about getting outcomes for customers that align with their business goals, not just looking after our bottom line.