Error handling strategies on a service bus

In my article on legacy migrations, I described how easy it is to get things wrong when moving away from a legacy platform. One of the tools that consistently helps with scalability is a service bus.

A data bus is an architectural piece that allows you to:

• Decouple your services responsibilities.
• Offload requests without losing your data.
• Avoid capacity bottlenecks.

Today, service buses are more relevant than ever. Platforms like RabbitMQ, Apache Kafka, Google Pub/Sub or Azure Service Bus have become staples in modern cloud-native, event-driven architectures. Whether queue-based or streaming-oriented, all of them enforce asynchronous communication and a pub/sub model that enables real scalability.

Why error handling matters

One of the most important aspects when implementing an event-driven architecture is to learn how to leverage the service bus to manage errors. Think, for instance, of the following scenarios:

1. Your service receives a message and tries to store it in the database, but the DB is down.
2. Your service forwards data to a third-party API — but it fails with a 500 error.
3. Your service gets a corrupted message missing an ID — impossible to process.

At first glance, they may look similar (“the message failed”), but the nature of the errors is fundamentally different:

• Recoverable errors under your control → e.g., your database being temporarily unavailable.
• Recoverable errors outside your control → e.g., a third-party service being down.
• Non-recoverable errors → e.g., malformed data that can’t be processed at all.

Thus, we can observe the errors 1 and 2 could be resolved (provided the operational dependencies start working properly again), whereas scenario 3 is hopeless. Each one of these problems requires a different strategy to manage the situation.

Retry strategy

When the error comes from infrastructure under your control — for example, your own database — a retry strategy is the most effective option. Once the anomaly is detected, the message goes back to the bus for reprocessing. If the DB is healthy the next time, the message succeeds and the system heals itself.

To avoid infinite loops, retries are configured with:

• A fixed delay (e.g., one minute between attempts), and
• A maximum number of retries.

Once the limit is reached, the message is sent to a dead letter queue (DLQ). DLQs act as a safety net: they isolate failed events, prevent the system from clogging up, and provide a clear signal for operators to step in.

The trade-off here is speed vs. resilience. You don’t want retries to delay the system too much, but you also need to give your own services a chance to recover. In practice, finding the sweet spot often requires observing real workloads in production and adjusting retry policies accordingly.

Retry with exponential backoff

When the failing dependency is outside your control — for instance, a third-party payments API — retries become trickier. Blindly retrying with a fixed delay could actually exacerbate the situation: if hundreds of services continue to hammer the same failing API, the downtime only gets longer.

That’s where exponential backoff comes in. The idea is simple: wait longer between each retry, giving the external service “air to breathe”:

• first retry after 1 min,
• then 2,
• then 4,
• then 8…

This spreading of attempts reduces pressure on the failing service and increases the chance of recovery. AWS explains this pattern well.

The logic still includes a cutoff point: after a certain number of retries, the message is sent to the DLQ for inspection. But compared to fixed retries, exponential backoff is far less likely to cause thundering herd problems, where every client retries at once and overloads the dependency even more.

This strategy is particularly valuable when integrating with SaaS APIs, payment providers, or banking systems — environments where uptime is out of your hands but your system still needs to stay resilient.

Dead letter queue strategy

Then there are the errors that will never succeed, no matter how often you retry. A corrupted message missing its ID, or an event with invalid schema, will always fail. Retrying them only wastes resources and adds noise to your system.

The right move is to route these messages straight into the DLQ. By doing this:

• You avoid unnecessary system load.
• You detect anomalies faster.
• You keep your pipeline flowing smoothly.

Once in the DLQ, operators (or automated processes) can inspect the messages, identify root causes, and decide whether they need to be fixed, replayed, or simply discarded.

In modern setups, DLQs are also a valuable observability tool. With real-time logging and alerting — e.g. via Better Stack or Datadog — teams can spot patterns early, such as a sudden spike in malformed events after a new release, and act before they affect production customers.

Conclusion

Resilient systems treat errors differently depending on their nature. Some can be retried, some require patience, and some are dead ends. Regardless of whether you use Kafka, RabbitMQ, Pub/Sub or another bus, these three patterns are universal.

At Orbitant, we put these principles into practice when building platforms for our clients. A well-designed error handling strategy isn’t just about keeping services alive — it’s about protecting data integrity, building trust, and enabling real business scalability.