MNW #026: Outbox Pattern For Reliable Microservices Messaging

Read time: 5 minutes

A big thank you to this week's sponsors who help keep this newsletter free to the reader:

Localizely is a translation management platform that helps you translate texts in your app for targeting multilingual markets.

Working with Microservices, or any distributed system for that matter, is difficult. In a distributed system many things can go wrong, and there are even research papers about this. If you want to explore this topic further, I suggest that you read about the fallacies of distributed computing.

Reducing the surface area for things to go wrong should be one of your goals, as an engineer. In this week's newsletter, we'll try to achieve exactly that using the Outbox pattern.

How can you implement reliable communication between components in a distributed system?

The Outbox pattern is an elegant solution to this problem, allowing you to achieve transactional guarantees in a single service and at-least-once message delivery to external systems.

Let's see how the Outbox pattern solves this, and how can we implement it.

What Problem Does The Outbox Pattern Solve?

To understand what problem the Outbox pattern solves, first we need a problem, of course.

Here's an example of a user registration flow. There are a few things going on here:

Saving the User to the database
Sending a welcome email to the User
Publishing a UserRegisteredEvent to a message bus

In the happy path, all of the operations complete without any issues and all is well.

But what happens if any one of these operations fail?

The database is unavailable, and saving the User fails
The email service is down and sending an email crashes
Publishing an event to the service bus doesn't succeed

Also, imagine a situation where you manage to save a User to the database, send him a welcome email, but fail to publish the UserRegisteredEvent to notify other services. How are you going to recover from this scenario?

The Outbox pattern allows you to atommically update the database and send messages to the message bus.

Implementing The Outbox Pattern

The first step is to introduce a table in your database to represent the Outbox. We can call this table OutboxMessages, and it's intended to store all messages that need to be delivered. Now instead of directly making requests to external services, we simply store a message as a new row in the Outbox table. The messages are usually stored as JSON in the database.

The second step is to introduce a background process that will periodically poll the OutboxMessages table. If the worker process finds a row with an unprocessed message, it's going to publish that message and mark it as sent. If publishing the message fails for some reason, the work process can retry in the next execution.

Notice that with retries, you now have at-least-once message delivery implemented. The message will be published exactly once for the happy path, and more than one time in case or retries.

We can rewrite the RegisterUserAsync method from the previous example, now using an Outbox:

The Outbox is part of the same transaction as our unit of work, so we can atomically save the User to the database and also persist the OutboxMessage. If saving to the database fails, the entire transaction is rolled back and no messages are sent to the message bus.

And since we now moved the publishing of the UserRegisteredEvent to the worker process, we need to add a handler so that we can send the welcome email to the user. Here's an example of that in the SendWelcomeEmailHandler class:

Architecture Diagram With Outbox

Here's a high level overview of the system architecture with the Outbox introduced to the system. You can see the Outbox table in the database. What changes now is that you store messages to the Outbox table in the same transaction along with your entities.

Architecture Diagram With Outbox

After reading this newsletter you should have a pretty good understanding of what the Outbox pattern is and what problems it solves. If you need to implement reliable messaging in a distributed system, it's a great solution for your problem.

What's missing is more details about how to implement the Outbox pattern, so here are a few videos you can watch:

Thanks for reading, and have an amazing Saturday.

Whenever you're ready, there are 2 ways I can help you 🔥

If you want to see more practical & in-depth coding examples you can do that by subscribing to my YouTube channel
If you want access to the source code that I use in my YouTube videos and to be able to ask me any questions you have, you can do that by supporting me on Patreon

Special Offers 📰

Ultimate ASP.NET Core Web API Second Edition - Premium Package
20% off with discount code: milanjovanovic20

YouTube Highlights 📢

How Structured Logging With Serilog Can Make Your Life Easier

Publish Subscribe Messaging In .NET With Redis Channels

Thank you so much for reading my newsletter!

If you enjoyed reading, share it with your friends.

Until next week, stay awesome!

You received this email because you subscribed to our list. You can unsubscribe at any time.

Dragise Cvetkovica 2, Niš, 18000, Serbia

MNW #026: Outbox Pattern For Reliable Microservices Messaging 📦

MNW #026: Outbox Pattern For Reliable Microservices Messaging

What Problem Does The Outbox Pattern Solve?

Implementing The Outbox Pattern

Architecture Diagram With Outbox

Architecture Diagram With Outbox

YouTube Highlights 📢

Older messages

Milan's .NET Weekly Newsletter And You 🎁

MNW #024: Messaging Made Easy With Azure Service Bus

Invitation - Modular Monoliths: How To Build One & Lessons Learned 🔥

MNW #025: Structured Logging In ASP.NET Core With Serilog

You Might Also Like

Unpacking “Craft” in the Software Interface & The Five Pillars of Creative Flow

What Investors Want From AI Startups in 2025

GCP Newsletter #426

⏳ 36 Hours Left: Help Get "The Art of Data" Across the Finish Line 🏁

DeveloPassion's Newsletter #180 - Black Friday Week

Meet HackerNoon's Latest Features: Boost Stories with Translations, Speech-to-Text & More

The ultimate holiday gadget gift

Deduplication in Distributed Systems: Myths, Realities, and Practical Solutions

How to know if your data has been exposed

⚙️ Amazon and Anthropic