Architecture Weekly - Building your own Ledger Database
Building your own database sounds like moronic move. Yet, that’s the advice I gave recently on the Architecture Weekly Discord channel. Why did I do it? Jokes aside, Amazon states that:
And suggests migrating it to Amazon Aurora PostgreSQL. Which is an option, but I suggested something else. Before we go to my suggestion, let’s discuss what are actually Ledger Databases. We talk about them in the previous editions, so #169 - 4th March 2024 and #181 - 27th May 2024. What Are Ledger Databases?Ledger databases are specialized data stores designed to provide an immutable and verifiable record of transactions over time. They ensure that every change to the data is recorded and cannot be tampered with, making them ideal for applications where data integrity and audibility are critical. Many industries have such legal requirements. Unlike traditional databases that allow data to be overwritten or deleted, ledger databases use an append-only model. Every transaction is added as a new entry, and previous entries remain untouched. This creates a complete history of all data changes, which is crucial for compliance, auditing, and tracing. Under the hood, ledger databases combine familiar database technologies with cryptographic techniques to guarantee data integrity. Each transaction is often cryptographically hashed and linked to the previous one, forming a chain that makes tampering evident. Use cases for Ledger DatabasesSo, where do ledger databases fit into our architectural toolbox? In business domains where you have to prove that no one even has a slight option to mutate the operations log. In some of them, user access management is not enough. You’d better use databases natively made write-ahead, not allowing you to mutate or delete anything. For instance:
Immutable log of data, ummmm. Do I smell Event Sourcing? Ledger Databases vs. Event Stores and Event SourcingThere's often confusion between ledger databases and event stores used in event sourcing. While both deal with immutable data and event histories, they serve different purposes. Event Sourcing is a design pattern in which outcomes of the business logic are stored as events. Then, they’re used for making the next decisions. They’re stored in sequences called streams. That can sound a bit scary, but in other words, streams represent records in classical databases. Events are the state; there’s no other state. Thus, the name Event Sourcing. In Event Sourcing flow looks as follows:
So, Event Stores are essentially Key-Value databases: The key is the record id, and the value is the sequence of events. And they’re used for transactional operations focused on business logic. Read more in my articles: Ledger databases, on the other hand, are specialized databases optimized for storing transactions, often with financial data semantics. They focus on accurately recording transactions, with an emphasis on data integrity and compliance, and often include cryptographic verification. So, they’re more like audit logs with much different data granularity than event stores. Your main driver is to record and keep immutable data as side effects of your business logic, not your decision model. Event stores are also general-purpose databases, while ledger databases are tailored for specific domains like finance. For example, financial ledger databases accurately record monetary transactions and balances and ensure consistency in financial operations. Going down the niche allows for domain-specific optimizations and limits the scope of usage. Custom Ledger DatabasesThose optimisation needs drove Uber and Stripe to build their own Ledger Databases; I’ve gone through them in linked editions. Stripe LedgerStripe built its own ledger system focused on storing and tracking financial data. Stripe operates at a massive scale, processing billions of events daily. They had to standardize the representation of money movement. With various systems and partners involved, they required a unified way to represent transactions despite the complexity and imperfections of real-world financial data. The ledger had to be an immutable source of truth that could be relied upon for internal operations and external audits. So, they also had to ensure data integrity and trustworthiness. The real world is imperfect, and they had to embrace it. Banking and network partners can provide malformed reports or errors, and they needed custom system to keep these imperfections manageable. Stripe's Ledger is an immutable log of events that models internal data-producing systems with common patterns like:
As mentioned, they operate on an extreme scale. Ledger processes five billion events per day. To handle this scale, they optimized their systems for high throughput and low latency. By using double-entry bookkeeping and immutable logs, they ensured that every transaction is accounted for and that the system can detect discrepancies. Read more: Uber's LedgerStoreSimilarly to Stripe, Uber developed its own ledger system, LedgerStore. To better meet its needs, Uber moved from using DynamoDB to its custom solution. At Uber's scale, DynamoDB became expensive. Managing petabytes of data and trillions of indexes was not cost-effective. DynamoDB's limitations led to issues like hot partitions and write throttling, requiring complex workarounds that increased system complexity. Uber needed a system that ensured data integrity, handled massive data volumes efficiently, and provided better performance. Similarly, they needed to ensure the data integrity and correctness guarantees, ensuring that individual records are immutable and corrections are trackable. To do that, they came up with their own ledger database. They designed an indexing system capable of handling trillions of indexes, supporting various access patterns via different types of indexes (strongly consistent, eventually consistent, time-range indexes). Uber migrated over a trillion ledger entries from DynamoDB to LedgerStore. They used Apache Spark for incremental backfill, processing data in chunks to avoid system overloads. To ensure accuracy, Uber used shadow validation, comparing outputs from the old and new systems to detect discrepancies, achieving a 99.99% accuracy rate. They encountered performance issues due to poorly distributed index data and inefficient indexing methods. To address this, they optimized partition keys and revised their indexing approach to reduce unnecessary scans. What was the Discord member use case?👋 Before I move on, I’d like to note that besides the paid content, we have an exclusive community on Discord! I know how being a tech lead or architect can be lonely. Sometimes, we don’t have other people to challenge our ideas or discuss our challenges. That’s why the community was built. This article is an example of discussions we have there. Become a paid subscriber and join us! The person who asked the question faced the challenge of migrating over 100 ledgers before Amazon QLDB was deprecated. The company operates in a regulated environment, providing imaging solutions for healthcare and pharmaceutical clients. ... Continue reading this post for free in the Substack app |
Older messages
Tech Debt doesn't exist, but trade-offs do
Monday, November 4, 2024
Tech debt is deader than dead, shock is all in your head. At least I'm shocked that after 32 years we're still using this term. I discussed today why I consider Tech Debt metaphore harmful, why
Frontent Architecture, Backend Architecture or just Architecture? With Tomasz Ducin
Monday, October 28, 2024
What's more important Frontend or Backend? What is Frontend Architecture? Is it even a thing? Where to draw the line, what are the common challenges in Frontend world? How do we shape our teams:
Don't Oversell Ideas: Trunk-Based Development Edition
Monday, October 21, 2024
We're living in the kiss-kiss-bang-bang era. Answers have to be quick, solutions simple, takes hot. One of the common leitmotifs that I see in my bubble is "just do trunk-based development
Why to measure and make our system observable? How to reason on chaotic world
Sunday, October 20, 2024
The world is messy and chaotic, who knew? Embracing that hard fact can bring relief, and be a first step to understanding how to handle known knowns, unknown unknowns and all that jazz. Today I
Webinar #23 - Gojko Adzic on designing product development experiments with Lizard Optimization
Monday, October 7, 2024
"My favorite conspiracy theory is that the stuff we make in software actually has any sense." As you see, we started strong in this week's episode. That's a quote from Gojko Adzic,
You Might Also Like
Apache Tomcat Vulnerability CVE-2024-56337 Exposes Servers to RCE Attacks
Tuesday, December 24, 2024
THN Daily Updates Newsletter cover The Data Science Handbook, 2nd Edition ($60.00 Value) FREE for a Limited Time Practical, accessible guide to becoming a data scientist, updated to include the latest
Edge 459: Quantization Plus Distillation
Tuesday, December 24, 2024
Some insights into quantized distillation ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏
Prepare for a Lifetime of Adventure with Rosetta Stone
Tuesday, December 24, 2024
The Perfect Gift For Every Traveler on Your List Rosetta Stone makes it easy to connect with the world in a whole new way. With a Lifetime Unlimited plan, users can access 25 languages to prepare for
Tuesday Triage #232
Tuesday, December 24, 2024
Your weekly crème de la crème of the Internet is here! The 232nd edition featuring fish traps, little Mussolinis, and volvelles. ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏
Elastic Community Newsletter
Tuesday, December 24, 2024
Check out the latest from the Elastic Community ㅤㅤㅤㅤㅤㅤㅤㅤㅤㅤㅤㅤㅤㅤㅤㅤㅤㅤㅤㅤㅤㅤㅤㅤㅤㅤ ㅤㅤㅤㅤㅤㅤㅤㅤㅤㅤㅤㅤㅤㅤㅤㅤㅤㅤㅤㅤㅤㅤㅤㅤㅤㅤ ㅤㅤㅤㅤㅤㅤㅤㅤㅤㅤㅤㅤㅤㅤㅤㅤㅤㅤㅤㅤㅤㅤㅤㅤㅤㅤ elastic | Search. Observe. Protect community-newsletter-header-img.png
Daily Coding Problem: Problem #1646 [Medium]
Monday, December 23, 2024
Daily Coding Problem Good morning! Here's your coding interview problem for today. This problem was asked by Facebook. Write a function that rotates a list by k elements. For example, [1, 2, 3, 4,
GCP Newsletter #430
Monday, December 23, 2024
Welcome to issue #430 December 23rd, 2024 News Event Official Blog Calling all devs: Code the future of baseball with Google Cloud and MLB - Google Cloud and MLB are hosting a hackathon where
⏯️ Make a Holiday Guest Profile for Your Streaming Services — What Is Linux Mint?
Monday, December 23, 2024
Also: I Played the Worst Mobile Games So You Don't Have To, and More! How-To Geek Logo December 23, 2024 Did You Know The giant splashes of color that make poinsettias a popular holiday decoration
Ranked | The Most Satisfying vs. Most Reliable Car Brands in 2024 🚙
Monday, December 23, 2024
The most reliable car brands are rarely the most satisfying to own, according to recent Consumer Reports survey data. View Online | Subscribe | Download Our App Presented by: Find the megatrends
Bitcoin Enthusiasts Are Letting Altcoins Pass by
Monday, December 23, 2024
Top Tech Content sent at Noon! Boost Your Article on HackerNoon for $159.99! Read this email in your browser How are you, @newsletterest1? 🪐 What's happening in tech today, December 23, 2024? The