[Last Week in AWS Extras]: Reader Mailbag: AWS Billing

 

Let's completely avoid the smoldering fire that is the TikTok / Oracle deal and instead turn our eyes to one of my personal favorite things: AWS bills. This week, we delve into a bunch of reader questions around AWS billing.

 

If you have questions you'd like to see answered in future Mailbag issues, hit reply. What've you got to lose besides your sanity once you understand the madness of AWS?

 

 

Does this sound like you? Your company is running an ELK Stack and you’re growing weary of constantly adding extra ES data nodes, or repeatedly re-indexing and allocating shards, or dealing with data field issues that force re-indexing, which slows down the entire cluster-f? Good news! There’s a much better way… ChaosSearch! Listen to what Jason Standiford, VP of Engineering at Revinate, said: “When we laid out all of the potential solutions, ChaosSearch was an easy choice. All of our users were already familiar with the Kibana UI, implementation was a breeze, and ChaosSearch has been a drop-in solution from day one.” So take it from me, or take it from Jason, and request a demo or start a free trial of ChaosSearch - the revolutionary, fully managed log analytics platform that turns your Amazon S3 into an UltraHot™ data store! Sponsored

 

 

At the start of These Unprecedented Times, my business partner and I hosted two Q&A sessions, fielding all sorts of questions from the audience about the world of AWS. Here are the questions that are filed under the general billing category, with a bit more nuance in the responses.

1. Why is my bill so big?!

In the world of cloud, you are not billed for what you use. You’re billed for what you forget to shut off.

 

When we start working with customers, they tend to assume we’re going to show up on day one, reach into our bag of tricks, and make the bill shrink with arcane magic.

 

In reality, here’s step one: Let’s take a look at the bill. Measure twice, cut once.

 

Many people love talking about the AWS bill in the same way it’s presented to them: alphabetically. But if someone else asks me about Alexa for Business one more time, I’m going to snap.

 

Look at the big numbers first and work your way down. There aren’t any secrets here. The bill is in the only single source of truth.

 

Unfortunately, AWS doesn’t have an inventory service—other than the bill itself. Welcome to hell.

2. What tips do you have for lowering the cost of AWS that apply very broadly?

Let’s jump right in:

 

Figure out where the money is going. You don’t want to spend time optimizing something that isn’t a meaningful portion of spend. Do the easy thing before doing the hard thing.

 

Focus on easy opportunities for winning. The patterns tend to be consistent across customers. EC2 is almost always the biggest spend category. And it’s followed by data transfer, Elastic Block Store, RDS, and S3, in no particular order. Start there and you’ll be in a good spot.

 

Check out managed NAT gateway. This is a handy service, but it has a way of being phenomenally expensive by adding a 4.5-cent per gigabyte data processing charge on top of any data transfer charges. If you’re storing data in S3 through one, use a private endpoint. They’re free.

 

Don’t buy the entire thing at once. If you’re hemming and hawing about making a Reserved Instance or Savings Plan purchase because you’re unsure of the future, buy some portion of it. Cut the proposed purchase by 20%--and then 20% again, and so on until you feel comfortable with it. You can always make another buy afterwards.

 

Get rid of data you don’t need. The biggest pattern I see about big data projects is they claim to be able to find anything except a business model. Do you really need all of those transaction logs from 2012? If you’re storing 4 petabytes of them, I’d posit maybe not. Worst case, you can always transition old data to Glacier Deep Archive (which is cheap as dirt), and you can retrieve it within a day’s time if you ever need it.

 

So there you have it. If you have a problem that I can solve in a tweet, that’s what I’ll do. Sending you a contract would be weird anyway. Truth be told, people gave me favors when I was starting out, and I believe in doing the same.

 

If you’re a small shop worried about your AWS bill, get in touch. I’ll be waiting!

3. What’s the biggest surprise you’ve had on your AWS bill?

The biggest surprise Mike once experienced way back when we were first getting started working together in an AWS environment was that the CloudWatch bill was larger than the DataDog bill that was hitting the CloudWatch API. That nonsense gets expensive.

 

My takeaway was: Holy crap, they charge for that? Today I’m less surprised. Of COURSE they charge for that! They charge for everything!

 

Take a look at what you’re actually monitoring. EBS volumes are a great example of this. You can get hypervisor metrics from CloudWatch—but not internal metrics. In almost every case, you don’t actually care about those hypervisor metrics for any of your EBS stuff. What happens is if you take a look at the various metrics you’re pulling, it doesn’t really answer the real question you have: Is the disk about to fill up?

 

Pro tip: Use gp2—not io1 or now io2—for almost everything. You’ll save a boatload of money. Provisioned IOPS are expensive.

 

To get the information you actually care about, there needs to be an agent inside the guest operating system. If you’re running something like DataDog, just query that about the data you care about in your volumes and turn off the things that are charging you per request for data you don’t actually need.

 

And if for some strange reason you ever need it in the future, you can access it directly within CloudWatch. You just won’t be getting charged for requests until then.

4. How do large enterprises pay their bills?

The way you or I end up paying our AWS bill in any case is here’s a credit card. And then when that card’s invariably maxed out, we move on to here’s another credit card. If you use the Amazon Prime card, despite what it says to the contrary in the terms, you can get 5% back on your AWS bill. You’re welcome. Thanks for reading!

 

Once you get into enterprise territory—which is somewhere in the $1 million/year range for cloud spend—both Amazon and companies that are paying Amazon switch over to invoice payments, which are paid via check, wire, or ACH. A customer might be able to negotiate 2% off of their bill while Amazon avoids getting hit with higher credit card processing fees.

 

It’s the same reason why you usually can’t buy a car on a credit card. The dealer doesn’t want to eat the interchange fees.

5. What are your suggestions for dealing with costs that don’t get attributed to resources or tags in the CUR (e.g., data transfer bytes out from Lambda)?

You’re never going to get full visibility into the spend of everything. At some point, you end up giving up. We’ve gotten to 80 or 90% coverage for tags, and the rest is going to slush. You want to be directionally correct. But, for almost every shop, you don't want to drive yourself mad by spending thousands of dollars to trace down pennies.

 

So the answer is this: It probably doesn’t matter.

 

For most architectures, the Lambda cost and auxiliary costs incurred by Lambda are miniscule. Any time we see thousands of dollars in Lambdas we see millions of dollars in EC2s.

 

Make sure this is a problem worth solving. Unless there’s a strategic reason to go down that rabbit hole, it probably isn’t worth it. Time that your team spends playing slap-and-tickle with the AWS bill is time they’re not spending working on their next feature.

6. What’s the first thing you would do to reduce EC2 networking costs?

I’d figure out what the [networking costs look like](https://www.duckbillgroup.com/resources/. Is it between different Availability Zones? Is it out to the internet? Is it something else? What region is it in? What workloads do I have hanging out there? What’s likely to be causing these costs?

 

You can get fairly granular after a few iterative cycles in Cost Explorer. But the other side of it is going down the rabbit hole of looking into what’s happening in those subnets.

 

I’ve found that it’s easier to solve this problem by talking directly to people and finding out what’s what. Maybe I’ll get a response like this: Oh, that’s the thing where we just replicate data around in a circle because we don’t understand what a Storage Area Network is!

 

Figuring out what it is that driving the cost usually starts the conversations. You can also go deeper with VPC Flow Logs, of course. But they are a bear to wind up doing meaningful analysis on.

7. Any advice on how to break down costs into a per-user metric? There seems to be a huge disconnect between bulk resource costs and a user doing a few API calls.

It’s a spectrum. You start off with the most naive approach that everyone does. You take the AWS bill for the last month, you look at the user metric you’re trying to get to (e.g., monthly active or daily active), and you do simple division.

 

That starts to break down because—no matter how many users you have or don’t have—you’re going to spend the same amount of money for the JIRA server, for example. And this is where conversations with finance begin to become valuable and why those tags are important.

 

When you’re trying to answer this question, you’re likely going to break things down into a model where you have a single user and you have to spend x dollars to get everything up and running for the infrastructure tooling around it. And then on top of that, there’s a marginal cost for every additional user.

 

For most workloads, that’s going to be more aligned to the cost per thousand users. Because unless you have a few very large customers, you’re not going to see a number that’s meaningful to humans. Every user we have costs .00003 cents. That number doesn’t make any sense to anyone. Focus instead on getting a number that is meaningful for planning and discussion purposes.

 

The challenge is people tend to optimize what they measure. And in some cases, driving costs down on metrics like this is the wrong answer. You’re going to want to optimize for other things, like user happiness. Otherwise, you can drop the cost per-user to zero by turning everything off. But that’s usually untenable—despite the fact that many companies should probably do that. Like Facebook!

 

If you have any other questions about AWS billing you’d like me to answer, please reach out.

 
 
 
Corey Quinn headshot

I’m Corey Quinn

I help companies address their horrifying AWS bills by both reducing the dollars spent and helping them understanding what they’re paying for.

 
 
Screaming in the Cloud Icon

Screaming in the Cloud & AWS Morning Brief

In addition to this newsletter, I host two podcasts: Screaming in the Cloud, about the business of cloud computing, featuring me talking to folks who are good at things; and AWS Morning Brief, a show about exclusively AWS with my snark at full-tilt.

 
 
Sponsorship Icon

Sponsor an Issue

Reach over 21,000 discerning engineers, managers, and enthusiasts who actually care about the state of Amazon's cloud ecosystems.

 



Want to skip these Last Week in AWS Extras? Click here and you won't receive these Wednesday dispatches anymore.

To make sure you keep getting these emails, please add corey@lastweekinaws.com to your address book or otherwise mark me as a permitted sender.

Want out of the loop completely? Click here to tell me to leave you alone.

 

Duckbill Group

1728 Ocean Ave #307, San Francisco, CA 94112

 
                                                           

Older messages

[Last Week in AWS] Issue #180: EC2 gets t4gging support

Monday, September 21, 2020

Good Morning! Another week come and gone, along with a veritable mountain of AWS announcements. Fortunately most were of the kind I discard ("Service you don't use is now available in a region

[Last Week in AWS Extras]: Is the AWS Free Tier really free?

Wednesday, September 16, 2020

This week is turning out squirrelly, so let's talk about AWS instead. What IS the AWS Free Tier? What will end up charging you? Let's delve into it. As always, hit reply and let me know if this

[Last Week in AWS] Issue #179: Going Flat Out Like a Koala In Season

Monday, September 14, 2020

Good Morning! Next week I'm giving a keynote at the of-course-it's-online Cloud Native Revolution conference, presented by Comcast. Unlike most things provided by Comcast, it's free. You

[Last Week in AWS Extras]: Dipping My Toes into the DigitalOcean

Wednesday, September 9, 2020

In February (before these Unprecedented Times), I decided it was time to play Explore That Cloud with DigitalOcean. They'd be sponsoring some of my nonsense in the near future, but this is why I

[Last Week in AWS] Issue #178: Amazon Repeatedly Stomps on Own Schmeckel

Monday, September 7, 2020

Good Morning! Welcome to Last Week in AWS issue 176. Over at the Duckbill Group, we've got a few new (and exciting!) services for your perusal. Run, don't walk, and see how we can help fix your

You Might Also Like

Daily Coding Problem: Problem #1647 [Medium]

Tuesday, December 24, 2024

Daily Coding Problem Good morning! Here's your coding interview problem for today. This problem was asked by Square. In front of you is a row of N coins, with values v 1 , v 1 , ..., v n . You are

Sentiment Analysis, Topological Sort, Web Security, and More

Tuesday, December 24, 2024

Exploring Modern Sentiment Analysis Approaches in Python #661 – DECEMBER 24, 2024 VIEW IN BROWSER The PyCoder's Weekly Logo Exploring Modern Sentiment Analysis Approaches in Python What are the

🤫 Do Not Disturb Mode Is My Secret to Sanity — 8 Gadgets I Want To See Nintendo Make

Tuesday, December 24, 2024

Also: The Best Christmas Movies to Watch on Netflix, and More! How-To Geek Logo December 24, 2024 Did You Know Their association with the Christmas season might make you think poinsettias hail from a

😱 AzureEdge.net DNS Retiring Jan. 2025, 🚀 Microsoft Phi-4 AI Outperforms, 🔒 Microsoft Secure Future Initiative

Tuesday, December 24, 2024

Blog | Advertise | View Online Your trusted source for Cloud, AI and DevOps guidance with industry expert Chris Pietschmann! Phi-4: Microsoft's New Small Language Model Outperforms Giants in AI

Mapped | The Top Health Insurance Companies by State 🏥

Tuesday, December 24, 2024

In 13 US states, a single company dominates the health insurance market, holding at least half of the total market share. View Online | Subscribe | Download Our App Presented by: Global X ETFs Power

The Stanford Grad Who Forgot How To Think

Tuesday, December 24, 2024

Top Tech Content sent at Noon! Boost Your Article on HackerNoon for $159.99! Read this email in your browser How are you, @newsletterest1? 🪐 What's happening in tech today, December 24, 2024? The

The next big HDMI leap is coming

Tuesday, December 24, 2024

Sora side hustles; Casio's tiny watch comes to the US -- ZDNET ZDNET Tech Today - US December 24, 2024 Ecovacs Deebot T30S Combo robot vacuum and mop The next big HDMI leap is coming next month -

⚙️ Robo-suits

Tuesday, December 24, 2024

Plus: The data center energy surge ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌

Apache Tomcat Vulnerability CVE-2024-56337 Exposes Servers to RCE Attacks

Tuesday, December 24, 2024

THN Daily Updates Newsletter cover The Data Science Handbook, 2nd Edition ($60.00 Value) FREE for a Limited Time Practical, accessible guide to becoming a data scientist, updated to include the latest

Edge 459: Quantization Plus Distillation

Tuesday, December 24, 2024

Some insights into quantized distillation ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏