How Ahrefs Saved US$400M in 3 Years by NOT Going to the Cloud
Clouds for IT infrastructure are so popular lately that moving into the cloud has become a trend. Infrastructure as a service (IaaS) cloud provides multiple advantages: flexibility, low time for deployment, easy scaling, instant presence in many regions of the world, and many others.
Cloud service providers have become knowledgeable outsourcers of IT services, providing convenient and often easy-to-use services. With excellent marketing, conferences, certifications, and cherry-picked use case scenarios, it’s easy for one to think that clouds are the only reasonable destination for the IT of a modern business.
But the costs of these outsourced cloud benefits are sometimes high. They’re so high we were concerned about whether our business could exist if our infrastructure was 100% in the cloud. This inspired us to make an actual comparison based on facts. And here they are:
Ahrefs’ own hardware overview
Ahrefs has a rented colocation data center in Singapore with a rather homogenous infrastructure. So we decided to consolidate all the costs spent for this data center, divide them by the number of servers, and compare them to the cost of a similar installation in the Amazon Web Services (AWS) cloud (we use this as the leading IaaS example).
Our hardware is relatively new. The colocation contract started in the middle of 2020, during the peak of the COVID-19 lockdowns. All the installed equipment had been bought since then. Our servers in this data center have similar configurations. The only difference is in the CPUs of two generations with the same number of cores. We use high core-count CPUs, 2TB RAM, and 2x 100Gbps per server. On average, our servers have about 16x 15TB drives.
For the monthly cost calculation, we assumed that all the hardware will be used for only five years. We decided to consider anything beyond five years a “freebie.” Based on this, the monthly part of the initial equipment cost is its original price divided by 5*12 = 60 months.
All the current costs, such as rent and electricity, are as of October 2022. Although inflation affects us too, trying to account for it makes these calculations much harder, so we’ve ignored it for now.
Our colocation payments consist of two major parts: rent and de facto consumed metered electricity. The electricity price has significantly increased since the beginning of 2022. But we calculated our monthly charges based on the latest, higher electricity charges, not on the average for the whole rental period, which would be lower. Therefore, this should give AWS an advantage.
Also, we pay for IP Transit and dark fiber between the data center and our point of presence.
The following table shows our monthly spending per server. Server hardware takes up two-thirds of the overall monthly expenditure, while the data center rent with electricity (DC), internet service providers (ISP) with IP Transit, dark fiber (DF), and internal network hardware (Network HW) make up the remaining one-third.
AWS cost structure
Since our colocation in review is in Singapore, we used AWS prices for their Asia Pacific (Singapore) region for comparison.
The cost structure for AWS is different from the colocation. Unfortunately, AWS doesn’t provide an EC2 instance with the number of cores we have. So we found an EC2 configuration with half the cores and 1TB RAM. We then compared one Ahrefs server cost to the cost of two such EC2 instances.
Also, we considered long-term EC2 usage. Thus, we compared the lowest price of a three-year reserved EC2 instance to five years of on-premises servers.
Apart from EC2 instances, we added Elastic Block Storage (EBS). It is not an accurate replacement for directly attached storage, as we use big and fast NVMe drives in the servers. To make things simple, we chose cheaper gp3 EBS (much slower than ours, though). Its cost consists of two parts: storage size and charges for IOPS.
We keep two copies of a data chunk on our servers. But we only order usable space in EBS that takes care of the replication for us. So we should consider the price of the gp3 storage size equal to the size of our drives divided by 2: (1*1TB + 16*15TB)/2 ≈ 120TB per server.
We didn’t add the cost of higher IOPS. We also discarded various limitations EBS gp3 has; for example, gp3 Max Throughput/Instance is 10GB/s. At the same time, the performance of a single PCIe Gen 4 NVMe drive is 6–7GB/s, and we have 16 of those working in parallel. Thus, it’s not an apples-to-apples comparison. But this approach should significantly reduce the cost of storage in AWS, giving it a further advantage.
Unlike in the colocation, AWS charges per GB downloaded, not for bandwidth. So we roughly estimated what our average download per server is and used the figure for the AWS network calculation.
Bringing all three components together, we have the following cost distribution for AWS.
Own hardware vs. AWS
Upon consolidating the above tables, we see that the imaginary AWS spending is much, much higher.
A replacement EC2 instance with similar usable SSD space in AWS would cost us roughly the same amount of money for 11.3 servers in a colocation data center. Accordingly, that means our 20-server rack would transform into just ~2 servers.
So let’s say we run our 850 servers during the actual 2½ years of our data center usage. Once we compute the numbers, we can see a dramatic difference.
We would need to pay AWS a whopping ~USD 400 million more to keep our infrastructure running in Singapore since 2020.
One may think, “Perhaps Ahrefs can afford this?”
It’s true that Ahrefs is a profitable and sustainable company, so let’s take a look at its revenue and use it in our calculations. That being said, we are a private company and aren’t obliged to publish our financial data. But one can find some information about Ahrefs’ revenue in The Straits Times articles on Singapore’s fastest-growing companies in 2022 and 2023. These articles provide Ahrefs’ revenue figures for the years 2020 and 2021.
We can linearly extrapolate revenue for 2022 too. It is a rough estimate. But it is enough for us to draw some conclusions.
From the table above, we can see that Ahrefs’ total revenue for the last three years was ~USD 257 million. But we also calculated that the AWS costs would be ~USD 448 million for such a data center replacement. Thus, the company revenue wouldn’t even be close to covering the 2½-year AWS usage costs.
What a stunning result!
But where would all our profit go?
As per this 20-year-old report by Dr. LJ Hart-Smith for Boeing: “If the prime manufacturer, or systems integrator, cannot make his fortune by giving all his work away, who does benefit? The subcontractors, of course.”
It is good to remember that we already provided several advantages to AWS by using higher-than-average electricity costs for our colocation data center, calculating only the storage part of EBS, and ignoring how slow the EBS storage is. Also, this data center is not the only cost center for us. We spend money on other data centers, servers, services, people, offices, marketing activities, etc.
Therefore, Ahrefs would hardly survive if a cloud was our primary infrastructure destination.
Other considerations
This article doesn’t take into account other aspects that would make the comparison even more complicated. These include people skills, financial controls, cash flow, capacity planning depending on the load type, etc.
Conclusion
Ahrefs has saved ~USD 400 million by ensuring its infrastructure is NOT 100% in the IaaS cloud during the last 2½ years. This figure is growing bigger, as currently we are starting another massive colocation data center with new hardware.
While Ahrefs uses AWS benefits for hosting our frontend in different parts of the world, the vast majority of Ahrefs’ infrastructure iceberg hides in the colocation with owned hardware. Ahrefs wouldn’t be profitable, or even exist, if our products were 100% on AWS.
If we used a cloud-only approach, our infrastructure costs would be 10+ times higher. But because we don’t do that, we can spend the money we saved on actual product improvements and developments. Consequently, having faster and better results means that our servers are much faster than what a cloud can provide (given its limitations). Our reports are also generated faster and are more comprehensive, as the time needed per report is shorter.
On that note, I recommend CFOs, CEOs, and business owners interested in sustainable growth to consider and periodically re-evaluate cloud benefits vs. actual costs. While clouds are the natural go-to choice for an early-stage startup, staying 100% in clouds with substantial infrastructure may sink a company as it and its infrastructure grow.
And here comes the dilemma.
It is complicated to leave a cloud once you are there. A cloud is convenient and locked in. Also, abandoning a cloud infrastructure due to higher costs may not be what the engineering team wants. They may rightly view a cloud as a much easier and more flexible environment compared to old-fashioned brick-and-mortar data centers with physical servers.
For a more mature company, migrating from the cloud to its own infrastructure is difficult. It will also be challenging to keep the company alive during the migration. But such a painful move may be what saves the company, as it can avoid paying an increasing cut of its revenue to cloud service providers.
Big companies, FAANG in particular, vacuumed the job market for many years. They had been hiring engineers to run their enormous data centers and infrastructure, leaving only a few for smaller companies. But with the mass layoffs in Big Tech in recent months, this may be an opportunity to re-evaluate the approach to the cloud, consider a reverse migration from the cloud, and hire seasoned professionals of the data center world.
And if you start a new company, consider buying a rack and server and putting them in your basement. That might improve your sustainability from day one.
