Talk is cheap, show me the numbers! Benchmarking and beyond!
Talk is cheap, show me the numbers! Benchmarking and beyond!On benchmarking foundations for performance issues analysisBoy, it's the fourth episode of the refreshed format, time flies! We started by learning the importance of connection pooling, discussing how to implement it, and then plot twisting to explain why what we learned so far is not yet production-ready and what does it actually mean to be production-ready! Lots of talking, especially considering that those topics were more to trigger discussions on different deployment strategies (e.g., serverless or not), how and when to scale, and why queuing is essential. Why so much talking? Because it's cheap! It's cheap to discuss and analyse tradeoffs before we make final decisions. Yet, we can't fall into analysis paralysis constantly going back and forth; there's a time when we'll hear: Talk is cheap; show me the numbers! If we don't, we'll see them. And I intend to show you the real numbers today. But before I do, I need to tell you a secret. I'm writing those articles not only for you but also for me, maybe even primarily for me, as I'm writing them not to forget. Forget about what? About what I've learned. But I hope you'll forgive me for this selfishness, as there's an additional benefit for you: they're real. They're showing my journey and are the outcome of my real architecture considerations. Currently, I'm building two Open Source tools: Emmett and Pongo. Both of them are Node.js storage tools. Emmett intends to bring Event Sourcing to the JS/TS community and their applications back to the future. Pongo is like a Mongo, but it is on PostgreSQL. So joining Mongo experience with PostgreSQL consistency and capabilities. So far, the feedback is positive. But it's an intriguing journey touching multiple aspects. Not only technical but also product-building, as I want to make it sustainable for me and useful for users. The hidden trap of building something useful is that people will use it, which can cause some trouble. I'm a big fan of the slogan "Make It Work, Make It Right, Make It Fast". Why? From the C2 Wiki:
So, I think what's there in Emmett in Pongo works. Based on the feedback, it sounds kind of right. Is it fast? Yes and no. The issueThat's how we got to the troubles users can cause. They might want to use your tool, deploy it, and verify whether it works and how fast it works. And they may find out that it doesn't work as they expected. Jokes aside, having early adopters and getting a feedback loop is crucial. I'm lucky to have people like Fernando who have contacted me and said they observed something weird. He said that locally, hosting PostgreSQL on a Docker container, the application was running instantly and super fast. Still, he never got a request processing time below one second when deployed to the end environment. That could not have been an issue a few years back, but it definitely sounds unacceptable nowadays. I told you that I'm lucky, and I do. Fernando is building a serverless-first product, which is a common case for startups or those in the early phases of development. With this model, you pay only for what you use. That means each request will (or can) spin up a new stateless environment. We cannot have a shared state between the calls, e.g., we cannot share a connection pool. That's a challenge but also a valid use case that I want to support. One of the reasons why I selected the Node.js environment to build my tools is that it helps me deliver faster and more sustainably. We discussed already in Mastering Database Connection Pooling that there are "Connection Pools as a Service" like: They can keep the shared connection, but they can cause issues if you're using an application-level proxy like pg pool. I used connection pooling as a default design choice for both Emmett and Pongo. They're sharing a common dependency called Dumbo, which is responsible for connection management and running database queries. Nevertheless, I added the option to turn off pooling to support the "Connection Pool as a Service" scenario. Knowing his use case, I also recommended that option earlier to Fernando, and that's also one of the reasons for the performance issue. We're getting there. The Initial AnalysisEmmett and Pongo are logically simple tools; they're:
I'm outsourcing the database stuff to the PostgreSQL driver, so I won't do sneaky things that could complicate this simple process. That's why the first step was to cut off external and the most obvious issues. What could be they? Deployment specifics? Serverless, in general, is vulnerable to the cold start issue. Cold start happens when the function is run after not being called for some time. Then, the whole environment needs to be started. The next request can reuse this environment without additional delay. After some time of inactivity, the environment will be again "put to sleep" so you don't use resources and don't pay for them. Still, that wasn't an issue here, as we verified that it's happening constantly, not only on the first request. Some deployments are different. For example, Cloudflare Workers use custom, slimmed Node.js; even AWS introduced recently their own Node.js runtime. But it wasn't that; it also happened when running locally against the actual database. The distance between the database and application hosting? If they were hosting in different regions (e.g. database in the USA and application in Europe), that could add significant enough latency. Wasn't that? Both deployments were in a similar region. So maybe that's the issue with the database? It could have been underprovisioned (so had a shortage of needed resources). If you're using innovative hostings like NeonDB and Supabase, which are adding their sprinkles to make PostgreSQL serverless and cheap, then maybe those sprinkles are the reason. Nah, running against the regular PostgreSQL hosted on Azure RDS. Gave the same subpar results. Then maybe something in the application is causing delays. Maybe an unusually big payload is causing delays. That could increase the latency between the application and the database and add to the serialisation time. Nope, the payloads were even smaller than they typically are. So we're back to the initial point, and the thesis is that it can be a bug in Emmett or Pongo. Show me the numbers!If that sounds like guessing, then you're right; calling that analysis is a bit of an exaggeration. But it's a necessary step to avoid dumb mistakes and understand the deployment specifics, data usage, and scenarios. This is needed to reproduce that and set benchmarks. Yes, benchmarks, because we finally got to the numbers. I reached the moment when I knew the primary issue. I wanted to know more about the reasons, so I had to choose tooling. I was correct, I was right, but I wasn't fast, and to fix that, benchmarks were needed. I chose Benchmark.js as it's the most popular microbenchmark tool in Node.js. I didn't need load tests yet, as I wanted to pinpoint the specific issue to make the initial improvement. Each development has its own benchmarking tools, I'll show you the syntax, but it's not as critical as understanding what we're doing. The first step was reproducing the intended usage and confirming that we were getting similar numbers.
Having that, I could set the benchmark code: The next part of the article is for paid users. If you’re not such yet, till the end of August, you can use a free month's trial: https://www.architecture-weekly.com/b3b7d64d. You can check it out and decide if you like it and want to stay. I hope that you will!... Unlock this post for free, courtesy of Oskar Dudycz.A subscription gets you:
|
Older messages
Architecture Weekly #191 - Is It Production Ready?
Tuesday, August 20, 2024
Why is connection pooling important? How can queuing, backpressure, and single-writer patterns help? In previous posts, we did a learning by-doing. We've built a simple connection pool, added
Architecture Weekly #190 - Queuing, Backpressure, Single Writer and other useful patterns for managing concurrency
Monday, August 12, 2024
Welcome to the new week! ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏
Architecture Weekly #189 - Mastering Database Connection Pooling
Monday, August 5, 2024
Boom! This is the first Architecture Weekly in the new format. It's a deep dive to database connection pooling. Check to learn how to set it up starting from simple solution and learning how the
Architecture Weekly #188 - 29th July 2024
Monday, July 29, 2024
This time in an unusual edition, we focused on the underestimated aspect of Software Design: resting. We discussed the typical mistakes we do and why repeating them will not help you in making right
Architecture Weekly #187 - 8th July 2024
Monday, July 8, 2024
This week, we discussed the challenges of finding the right architecture strategy and pivoting it when needed. We brought and discussed a few insightful articles on how/when/why to move to serverless,
You Might Also Like
Polymarket Explained: How Blockchain Prediction Markets Are Shaping the Future of Forecasting
Saturday, November 9, 2024
Top Tech Content sent at Noon! How the world collects web data Read this email in your browser How are you, @newsletterest1? 🪐 What's happening in tech today, November 9, 2024? The HackerNoon
🐍 New Python tutorials on Real Python
Saturday, November 9, 2024
Hey there, There's always something going on over at Real Python as far as Python tutorials go. Here's what you may have missed this past week: Variables in Python: Usage and Best Practices In
Palo Alto Advises Securing PAN-OS Interface Amid Potential RCE Threat Concerns
Saturday, November 9, 2024
THN Daily Updates Newsletter cover Introduction to Algorithms: A Comprehensive Guide for Beginners: Unlocking Computational Thinking ($34.99 Value) FREE for a Limited Time Begin your journey into the
Tired of spam calls? My top pick for privacy protection...
Saturday, November 9, 2024
You may have seen my previous emails about Incogni — my top recommendation to protect against spam and identity theft. Today, I want to share with you their early Black Friday deal exclusively for
📧 Functional Programming in C#: The Practical Parts
Saturday, November 9, 2024
Functional Programming in C#: The Practical Parts Read on: my website / Read time: 6 minutes The .NET Weekly is brought to you by: You know that feeling when you have an idea in your head but can
📞 Stock Android Doesn't Exist Anymore — How I Stay Safe Using Facebook Marketplace
Friday, November 8, 2024
Also: Why You Should Use Apple Numbers, and More! How-To Geek Logo November 8, 2024 Did You Know The "Zoo Hypothesis" is a theory that posits that alien civilizations know that Earth exists,
iOS Dev Weekly - Issue 686
Friday, November 8, 2024
CoPilot vs Xcode Predictive Code Completion: Is the playing field level now? 🤖 View on the Web Archives ISSUE 686 November 8th 2024 Comment If you didn't catch GitHub Universe, you might have
Daily Coding Problem: Problem #1602 [Medium]
Friday, November 8, 2024
Daily Coding Problem Good morning! Here's your coding interview problem for today. This problem was asked by LinkedIn. You are given a string consisting of the letters x and y , such as xyxxxyxyy .
UX for product people
Friday, November 8, 2024
UX for Product Managers This week, we're digging into the powerful blend of UX and product management, bringing you resources and insights to take your product experience to the next level. For
SWLW #624: Strategy as a series of beliefs, How to identify your own Superpowers, and more.
Friday, November 8, 2024
Weekly articles & videos about people, culture and leadership: everything you need to design the org that makes the product. A weekly newsletter by Oren Ellenbogen with the best content I found