How to Design Reliable and Scalable Webhooks with RabbitMQ
By Jim Liao
Feb 22, 2017
Webhooks are user-defined HTTP callbacks that are used to invoke behavior on another site when an event occurs. Webhooks are great for creating loosely coupled architectures for integrating multiple heterogenous systems. Unfortunately implementing webhooks requires you to handle multiple scenarios related to external system failures. In this video, I walk you through how to design scalable and reliable webhooks with RabbitMQ.
The challenge with implementing webhooks is that the external systems that you are calling can either be offline or unresponsive. In all failure cases, it is the responsibility of the webhook implementation to keep the event message and retry calling the destination system at a later time. By using RabbitMQ, you gain reliability by using message queues. By not removing messages from the message queue until it has been successfully processed by the external systems, you can guarantee that the event messages will not be dropped. By running multiple processes as consumers against the same message queue, you can gain scalability.
When you are considering implementing your own webhooks, consider using RabbitMQ or another software that provides message queues as a way to make your webhooks reliable.