Building best practices webhook listeners, part 2: Synchronous and Asynchronous processing

By Larry Kluger and Joey Peng

Because a Connect notification message is delivered as an HTTPS POST request, it is relatively easy to build a Connect listener to receive and process messages. In this series of blog posts, we’ll discuss best practices, practical techniques, and examples for building robust webhook listeners that will work well and scale as needed.

This blog post is the second installment in a series that describes best practices for building webhook listeners. If you missed it you can read part 1, which discussed the advantages of using webhooks and recommended Connect settings

Synchronous processing

From the client web browser’s point of view, most web servers operate synchronously: when the web browser (the client) sends a request, the web server processes the request and does not respond to the browser until the request has been completed.

Many Connect listeners use synchronous processing:

1. DocuSign Connect (the client) sends an HTTPS POST request to the listener (the server).
2. The listener receives the request’s notification message.
3. The listener verifies the message’s HMAC.
4. The listener processes the notification message (by updating internal databases, performing internal processes, downloading the envelope’s documents, or performing any other required steps.)
5. The listener responds to the POST request with a 200 status, acknowledging receipt of the message.

This architecture can work well for small volumes of messages but it does not scale well because the amount of time taken to process the notification message (step 4 above) can vary depending on the contents of the notification message, the volume of messages, and other loads on the shared databases or other systems.

Connect will wait a maximum of 100 seconds for your listener to acknowledge the HTTPS POST request, but the recommended best practice is for your listener to never take more than a couple of seconds to process each request. If you can’t guarantee a response time of 5 seconds or less, you should switch to use asynchronous processing.

Asynchronous processing

Due to the wide availability of low-cost (or no-cost), easy to use queuing systems, DocuSign recommends using asynchronous processing of notification messages for all Connect listeners.

To build a reliable Connect listener, we recommend using a durable queuing system. In this context, durable means that once the data’s storage in the queue is acknowledged, the data will not be lost even if the queue server is rebooted (or crashes) in the middle of the process.

Many durable queuing systems are available including IBM MQ, Microsoft Message Queuing, AWS SQS, Azure Service Bus queues, and smaller scale solutions including Bee-Queue, built on Redis.

Most durable queuing systems provide an at least once guarantee, meaning that once a message is properly enqueued, the queuing system will ensure that the message is delivered at least once. Many libraries also offer a First In First Out (FIFO) service. Note that, because a message can be delivered more than once, it is important that your application can properly handle duplicate messages or messages that are delivered out of order. For many Connect use-cases, this is not hard to implement.

Example workflow for asynchronous processing:

1. DocuSign Connect sends an HTTPS POST request to the listener server.
2. The listener receives the request’s notification message.
3. The listener verifies the message’s HMAC.
4. The listener adds the message to a durable queuing system.
5. The listener responds to the POST request with a 200 status, acknowledging receipt of the message.

In addition to the listener’s steps listed above, one or more worker processes are also (in parallel with the enqueuing processing):

1. Either awaiting notification from the queuing system that a new message is available, or polling the queuing system, looking for a new message. The choice of notification or polling depends on the queuing system being used.
2. Processing the notification message (same as step 4 for synchronous processing).
3. Acknowledging to the queuing system that the worker has completed processing of the message.

Advantage: Asynchronous queuing

An asynchronous architecture provides many advantages:

• Modern: This type of microservice architecture is recommended for building fault-tolerant, loosely-coupled, scalable systems.
• Scalable: It is relatively easy to add additional worker processes as the volume of transactions increases.
• Fast: Modern queuing libraries process messages quickly, providing all the advantages of an asynchronous architecture with little delay.
• Separation of concerns: The listener is focused on receiving and enqueuing the Connect messages. One or more workers process the messages as they become available.

Next time: Behind the firewall

An asynchronous architecture can also help with firewall issues. We’ll discuss this subject in the next part of this series.