MNW #034: Idempotent Consumer - Handling Duplicate Messages

Read time: 3 minutes

A big thank you to our sponsors who keep this newsletter free to the reader:

Today's issue is sponsored by Hasura, an open-source engine that gives instant GraphQL and REST APIs on new or existing SQL Server, enabling teams to ship apps faster.

And by Rebus Pro. Rebus is a free .NET "service bus", and Rebus Pro is the perfect one-up for serious Rebus users. Use Fleet Manager to get Slack alerts when something fails and retry dead-lettered messages with a click of the mouse.

What happens when a message is retried in an event-driven system?

It happens more often than you think.

The worst case scenario is that the message is processed twice, and the side effects can also be applied more than once.

Do you want your bank account to be double charged?

I'll assume the answer is no, of course.

You can use the Idempotent Consumer pattern to solve this problem.

In this week's issue I'll show you:

How the Idempotent Consumer pattern works
How to implement an Idempotent Consumer
The tradeoffs you need to consider

Let's see why the Idempotent Consumer pattern is valuable.

How The Idempotent Consumer Pattern Works

What's the idea behind the Idempotent Consumer pattern?

"An idempotent operation is one that has no additional effect if it is called more than once with the same input parameters."

We want to avoid handling the same message more than once.

This would require Exactly-once message delivery guarantees from our messaging system. And this is a really hard problem to solve in distributed systems.

A looser delivery guarantee is At-least-once, where we are aware that retries can happen and we can receive the same message more than once.

The Idempotent Consumer pattern works well with At-least-once message delivery, and solves the problem of duplicate messages.

Here's what the algorithm looks like from the moment we receive a message:

Was the message already processed?
If yes, it's a duplicate and there's nothing to do
If not, we need to handle the message
We also need to store the message identifier

We need a unique identifier for every message we receive, and a table in the database to store processed messages.

However, it's interesting how we choose the implement the message handling and storing of the processed message identifier.

You can implement the idempotent consumer as a decorator around a regular message handler.

I'll show you two implementations:

Lazy idempotent consumer
Eager idempotent consumer

Lazy Idempotent Consumer

The lazy idempotent consumer matches the flow shown in the algorithm above.

Lazy refers to how we store the message identifer to mark it as processed.

In the happy path, we handle the message and store the message identifier.

If the message handler throws an exception, we never store the message identifier and the consumer can be executed again.

Here's what the implementation looks like:

Eager Idempotent Consumer

The eager idempotent consumer is slightly different from the lazy implementation, but the end result is the same.

In this version, we eagerly store the message identifier in the database and then continue to handle the message.

If the handler throws an exception, we need to perform cleanup in the database and remove the eagerly stored message identifier.

Otherwise, we risk leaving the system in an inconsistent state since the message was never handled correctly.

Here's what the implementation looks like:

In Summary

Idempotency is an interesting problem to solve in a software system.

Some operations are naturally idempotent, and we don't need the overhead of the Idempotent Consumer pattern.

However, for those operations that aren't naturally idempotent, the Idempotent Consumer is a great solution.

The high-level algorithm is simple, and you can take two approaches in the implementation:

Lazy storing of message identifiers
Eager storing of message identifiers

I prefer to use the lazy approach, and only store the message identifier in the database when the handler completes successfully.

It's easier to reason about and there is one less call to the database.

Thanks for reading.

Hope that was helpful.

Whenever you're ready, there are 3 ways I can help you 🎁

To access all of the source code that I use in my YouTube videos support me on Patreon
For more practical & in-depth coding examples subscribe to my YouTube channel
Promote yourself to 15.000+ subscribers by sponsoring this newsletter (booked out 2 months)

Do you want a FREE conference ticket? 🎟️

I'm giving out 20 free tickets for the ABP Conference.

The rules are simple:

First 20 of you that email me saying `ABP Conference` win a ticket

Good luck! 🍀

You received this email because you subscribed to our list. You can unsubscribe at any time.

Dragise Cvetkovica 2, Niš, 18000, Serbia

MNW #034: Idempotent Consumer - Handling Duplicate Messages

MNW #034: Idempotent Consumer - Handling Duplicate Messages

How The Idempotent Consumer Pattern Works

Lazy Idempotent Consumer

Eager Idempotent Consumer

In Summary

Older messages

MNW #033: EF Core Raw SQL Queries

MNW #032: How To Use Rate Limiting In ASP .NET Core

MNW #031: Implementing The Saga Pattern With Rebus And RabbitMQ

MNW #030: How To Publish MediatR Notifications In Parallel

MNW #029: Creating Data-Driven Tests With xUnit

You Might Also Like

The ultimate holiday gadget gift

Deduplication in Distributed Systems: Myths, Realities, and Practical Solutions

How to know if your data has been exposed

⚙️ Amazon and Anthropic

⚡ THN Recap: Top Cybersecurity Threats, Tools & Tips (Nov 18-24)

Researchers Uncover Malware Using BYOVD to Bypass Antivirus Protections

Post from Syncfusion Blogs on 11/25/2024

Is there more to your iPhone?

🎉 Black Friday Early Access: 50% OFF

Open Pull Request #59