Sending Reliable Event Notifications with Transactional Outbox Pattern

Friends don’t let friends do dual writes!

Although the Microservices maintain their state private to them, they hardly operate in isolation. Some business use cases require them to change their state first, then notify that change to a broader audience.

In this post, I discuss why dual writes in distributed systems are bad, how you can fix that with the outbox pattern, and some inner workings of the pattern. This post doesn’t stand up as an implementation guide but instead brings in the fundamentals you should use when implementing in production.

Imagine there are two microservices; the Order service and the Shipping service. The business logic states the Order service should notify the Shipping service about a new order reception to prepare a shipment.

What are the options we have here?

Synchronous approach: The Order service synchronously invokes an API method of the Shipping service to do the notification. The drawback is that it introduces a coupling between two services. The Order service must depend on the Shipping service’s availability, dealing with retries, rate limits, etc. Considering the scalability, we can rule this approach out.

The asynchronous approach appears to be a fitting solution in terms of scalability. But what could be the possible odds?

A failure to update two systems atomically leaves the entire system in an inconsistent state.

As per the above sequence diagram, if the event publishing fails due to a broker outage, we will have an order in the system but without a shipment. Also, if the new order insertion fails due to a database error, the event anyway gets published. That creates a shipment without a corresponding order.

This problem related to dual writes demands us to be on the lookout for a solution to make the database update and event publishing atomic.

The Transaction Outbox pattern solves this problem by writing to two database tables, the aggregate table and an OUTBOX table, within the same transaction scope and then use the content written to the OUTBOX table to drive the event publishing process.

The pattern comprises two components.

The pattern introduces a supplementary table, called OUTBOX, to the service’s database. This table stores the event notifications that are supposed to send from the service to the message broker. When service writes to the aggregate table, it also writes a record to the OUTBOX table as a part of the same transaction.

The record written to the OUTBOX table describes a change event that happened in the service. For example, it could be a new customer registration or a customer changing the email address.

The message relay component asynchronously monitors the OUTBOX table for new entries. If any, they will be transformed into events and published to the message broker.

Once published, the message will be deleted from the OUTBOX table to prevent reprocessing and the table growth.

The OUTBOX table should have the following structure at a minimum.

The Outbox pattern enables achieving atomicity when writing to the database and publishing events to the broker. We can leverage the power of local transactions to do both actions or nothing.

By writing a record to the OUTBOX table, we benefit from the at-least-once delivery guarantee for the change event. In case of a broker outage, the message relay can retry reliably after reading OUTBOX messages. The broker may not be reachable for hours or days. But we have a persistent record of the message to be sent when the broker is back online. That way, we can guarantee that the change event will reach the broker at least once.

Furthermore, we can benefit from a local ACID transaction when writing to both tables at the same time. If writing to either of the tables fails, we will have a clean rollback. That prevents fake messages from getting published to the broker. Fake message as in, the aggregate table update could’ve failed, but an event has been published to the broker.

After publishing an event, the message relay deletes the corresponding record in the OUTBOX table to prevent reprocessing. The logic would look like the following:

But the message relay may fail to do so if it crashes during the attempt to delete the record. When it restarts, it sees the same record, thus publishes it to the broker for the second time.

That is one challenge often associated with the Outbox pattern. We can fix it by making the downstream event consumer idempotent.

A failure in the message relay to delete the OUTBOX record, a message broker restart, or an unknown error may cause the event consumer to receive duplicate events. Achieving exactly once semantics in a distributed system is a challenge we all have to face.

Shipping service in our example might receive the same Order twice so that it would prepare the same shipment twice, which is not acceptable. The consumer can prevent this by checking whether the event with the given UUID has been processed before. If so, any further calls for that same event will be ignored.

Similar to the OUTBOX table, we can maintain an INBOX table inside the consumer service’s database. It simply keeps track of what events were processed by recording their UUIDs.

After processing an event for the first time, the consumer marks the event as processed in the INBOX table. That should be transactional — making it possible to trap any rollbacks at the consumer level so that it can retry receiving the event.

I would say implementing this pattern is straightforward. All you need is a programming language that supports transactional writes to a database.

A simple Java code sample for sending events to the OUTBOX should look like the following:

The challenge comes after. That is, implementing the message relay. There are two strategies you can consider.

Here, you have a transaction log mining component that tails the database transaction log to capture the changes made to the OUTBOX table. The transaction log records all transactions committed against the database in a strongly ordered manner. The miner reads the transaction log and publishes each change as a message to the message broker.

In general, this pattern can be used to propagate state changes among Microservices reliably. Some use cases are as follows.

Related posts:

Paper Wallet. A paper wallet defines the storage of private information (public keys, private keys, and seed phrases) on, as the name implies, paper. This works because any public and private key…