Welcome to the new week!

When you’re reading this article I’m currently at Domain Driven Design Europe conference doing 3-day workshop on Production Grade Event Sourcing. I’ve spent ton of time preparing for it, I hope that it’ll pay off for the attendees, and they’ll expand their knowledge on how to run Event Sourcing system efficiently.

If you’re also at the conference and you see me walking around, please come and say hi, I’ll be happy to talk with you!

In the 2nd edition of Papers We Love, we tackled the famous article Mel Conway’s article where he introduced his law.

We’ve read the whole paper to check how up-to-date it is nowadays. Not surprisingly, it has stood the test of time. It’s a basis for properly modelling our system design and organizational structure. It impacted all current socio-technical approaches, such as Team Topologies, both agile and more rigid approaches.

Watch (or listen as a podcast) the episode and drop me a note with your thoughts and experience facing the Mel Conway law. I’m sure you have plenty of them!

• Papers We Love #2 - How do committees invent? (Melvin E. Conway)

Check also additional reading:

• the engineering club - Inverse-Conway-Maneuver with Nick Tune

• Jonny LeRoy - Dealing with creaky legacy platforms

• Bruno Costa - You probably don’t need microservices

"Just do left fold" or "Just apply event one after another" is common advice on handling messages and event-driven processing. They're fine from the logical perspective, but in production, you really need to do better. If you want to get proper performance and throughput, you'll eventually need to incorporate batching.

AsyncEnumerable in .NET is a nice abstraction that allows you to handle both pull-based and push-based communication in a unified, straightforward way. Still, by default, it follows the "just do left fold" advice.

In the latest article, I showed how to do better and handle batching in AsyncEnumerable without losing its clarity.

• Why you should batch message processing and how to do it with .NET AsyncEnumerable

What are your thoughts? To batch or not to batch?

Staying on the messaging and event-driven design topic, let’s do a follow up from the two weeks ago. Adam Bellemare did a follow up on his article on fixing errors in event.

• Adam Bellemare - Preventing and Fixing Bad Data in Event Streams — Part 2

One nitpick from me, I wouldn’t say that Event Sourcing is about deltas, I think that’s oversimplification. It’s about capturing what has happened, this may look close to delta, but it’s not. Those are all informations about business fact. Still, even with that the advice on the article are correct, even for Event Sourcing, so good job!

Another follow up, this time from last week. We discussed wiping out by Google the whole customer Cloud account. They of course said that’s a rare experience. If you think that indeed such big issues cannot happen too often, Google will say “hold my beer”. Check the Register coverage:

• The Register - Google Cloud shows it can break things for lots of customers – not just one at a time

Google did “maintenance automation intended to shutdown an unused network control component in a single location.’ And this time they didn’t wipe out customer account, but they 33 services in 40 regions for two hours and forty-eight minutes weren’t working correctly. I don’t even know how to comment it. Reading that, I’m happy that I didn’t have yet chance to work on Google Cloud.

If you like networking post mortems check also the bug retrospective explained by Antithesis. Interesting is that’s a tool for finding bug. So it’s about finding a bug in a tool to find a bug. Bugception.

• Antithesis - The worst bug we faced at Antithesis

Another follow up, Architecture Weekly doesn’t forget! 3 months ago we discussed Stripe Ledger, their own database build and optimised for storing financial data. It seems that’s fashionable, as Uber did the same.

• Uber - How LedgerStore Supports Trillions of Indexes at Uber

What’s more, they now moved the storage from DynamoDB into it and provided they case study.

• Uber - Migrating a Trillion Entries of Uber’s Ledger Data from DynamoDB to LedgerStore

What’s Uber LedgerStore?

LedgerStore was introduced in 2018 to support Gulfstream, Uber’s fifth-generation collection and disbursement payment platform, in order to provide the following data integrity guarantees: 

1. Individual records are immutable

2. Corrections are trackable

3. Unauthorized data changes and inconsistencies must be detected

4. Queries are reproducible a bounded time after write

Sounds close to event stores, but it’s not the same. Let me quote my explanation (yeah, I know…) from discussing Stripe version:

There’s an overlap between that and event stores. They both are focused on storing immutable data and its quality. Event Sourcing works well for processes that can be modelled as state machines, and Stripe’s Ledger also makes it a foundational piece. It also allows events notifications, auditability and enhanced tracing (which is crucial for fraud detection etc.).

However, this sophistication comes at a cost. The system's complexity might pose challenges for understanding and troubleshooting, especially given the unpredictable nature of financial services and the potential for errors from external banking partners. While Ledger significantly enhances Stripe's data integrity and operational efficiency, its intricate architecture requires a deep understanding of the technology and the financial domain to manage effectively.

Same apply here, still, let’s move on to explain how they did it.

Uber's migration to LedgerStore involved transferring over a trillion ledger entries, which caused significant challenges around data integrity and system performance. To ensure accuracy, Uber used shadow validation, comparing outputs from the old and new systems to detect discrepancies and achieved a 99.99% accuracy rate. They tackled the large data volume by using Apache Spark for incremental backfill, processing data in chunks to avoid system overloads.

Performance issues arose from poorly distributed index data and inefficient indexing methods that led to excessive full-record scans. To fix these, Uber optimized partition keys for better data distribution and revised their indexing approach to reduce unnecessary scans. So as always, indexes. Whatever database type you choose, that’s always a PITA.

Continuing the databases, but more general usage.

Mike Stonebraker and Matei Zaharia, so people behind the Postgres and Databricks decided to build an Operating System dedicated for data processing. Now they build a Cloud serverless data platform.

• Mike Stonebraker - Introducing DBOS Cloud: Transactional Serverless Computing on a Cloud-Native OS

Of course, you’ve heard that song, but having those people behind it make it worth checking out. They’re saying that it’s “a transactional serverless platform built on DBOS, targeting stateful Typescript applications”. They offer reliable execution and time travel. All of that on top of SQL. I’m not sure how far they’ll be able to go with it, but it’s an intriguing case, backed by the research on MIT and serious people.

In the same way, you should check Mojo language, a Python-compliant Rust killer designed for AI solutions. Yeah, I felt like too much bs when I saw that, but then I listened the Developer Voices podcast episode with its creator:

• 📺 Developer Voices - Mojo Lang - Tomorrow's High Performance Python? (with Chris Lattner)

And then I realised that it’s the same guy that was behind Swift language and LLVM. Check this episode, Developer Voices is a nice geek-out podcast, that I recommend, and Chris is nicely explains the assumptions behind Mojo, and that kinda clicks!

Speaking about languages, Kotlin 2.0 was released. Check more in the great coverage by Artur Skowroński, and if you’re into JVM, subscribe to his JVM Weekly. It’s great:

• Artur Skowronski - Kotlin 2.0 & opinionated Wrap-up of KotlinConf 2024 – JVM Weekly vol. 84

• JetBrains - Celebrating Kotlin 2.0: Fast, Smart, and Multiplatform

• Kotlin - What's new in Kotlin 2.0.0

Ah, and React got compiler. Yes, R e a c t g o t c o m p i l e r.

• Jack Herrington - React Compiler With React 18

• React - React Compiler

Times!

And Angular 18 was released with a big set of improvements. I’m especially happy about Zone.js removal. I hate every minute of when I had to debug error routing me into Zone.js issue.

• Minko Gechev - Angular v18 is now available!

Check also other links!

Cheers

Oskar

p.s. I invite you to join the paid version of Architecture Weekly. It already contains the exclusive Discord channel for subscribers (and my GitHub sponsors), monthly webinars, etc. It is a vibrant space for knowledge sharing. Don’t wait to be a part of it!

p.s.2. Ukraine is still under brutal Russian invasion. A lot of Ukrainian people are hurt, without shelter and need help. You can help in various ways, for instance, directly helping refugees, spreading awareness, and putting pressure on your local government or companies. You can also support Ukraine by donating, e.g. to the Ukraine humanitarian organisation, Ambulances for Ukraine or Red Cross.

Architecture

• Uber - Migrating a Trillion Entries of Uber’s Ledger Data from DynamoDB to LedgerStore

• The Register - Google Cloud shows it can break things for lots of customers – not just one at a time

• Pinterest Engineering - How a one line change decreased our clone times by 99%

• the engineering club - Inverse-Conway-Maneuver with Nick Tune

• Jonny LeRoy - Dealing with creaky legacy platforms

• Bruno Costa - You probably don’t need microservices

• Adam Bellemare - Preventing and Fixing Bad Data in Event Streams — Part 2

• 📺 Developer Voices - Mojo Lang - Tomorrow's High Performance Python? (with Chris Lattner)

• Maaret Pyhäjärvi - Making Releases Routine

• Antithesis - The worst bug we faced at Antithesis

• J. Liu, S. Poulsen, E. Goodwin, H. Chen, G. Williams, Y. Gertner, D. Franklin - Teaching Algorithm Design: A Literature Review

• David Boike - What does idempotent mean in software systems?

DevOps

• Joseph Lynch - How Netflix Ensures Highly-Reliable Online Stateful Systems

Databases

• Uber - How LedgerStore Supports Trillions of Indexes at Uber

• Mike Stonebraker - Introducing DBOS Cloud: Transactional Serverless Computing on a Cloud-Native OS

• Alexander Belanger - An unfair advantage: multi-tenant queues in Postgres

• Jonathan Katz - Scalar and binary quantization for pgvector vector search and storage

Frontend

• Minko Gechev - Angular v18 is now available!

• Jack Herrington - React Compiler With React 18

• React - React Compiler

Java

• Aleksey Stukalov, Catherine Edelveis - Java Runtimes: Insights From the Spring Boot Point of View

Kotlin

• Artur Skowronski - Kotlin 2.0 & opinionated Wrap-up of KotlinConf 2024 – JVM Weekly vol. 84

• JetBrains - Celebrating Kotlin 2.0: Fast, Smart, and Multiplatform

• Kotlin - What's new in Kotlin 2.0.0

.NET

• Oskar Dudycz - Why you should batch message processing and how to do it with .NET AsyncEnumerable

• Microsoft - General Availability of .NET Aspire: Simplifying .NET Cloud-Native Development

• Microsoft - .NET Announcements and Updates from Microsoft Build 2024

• Andrew Lock - Thoughts about primary constructors: 3 pros and 5 cons

• Chris Simon - TDD, DDD, and C# from the Ground Up

Tools

• Gorilla CLI - LLMs for your CLI . Powers your command-line interactions with a user-centric tool

Management

• Aleix Morgadas - Awesome Engineering Strategy