From the Trenches: Troubleshooting Docusign Connect

One of the primary tasks that comes up in the design and expansion phases of any integration is always going to be efficiency. When it comes to Docusign, we do have certain rules and limits put into place to safeguard our systems from excessive demand from, among other things, envelope polling. If you’re not already familiar, our polling rules state that, while making GET requests for envelope resources is an acceptable practice, we do require that these requests be placed more than 15 minutes apart.

Some customers haven’t quite met the envelope velocity required to need a quicker refresh than that. For customers of ours that do require a quicker refresh, our primary system for accomplishing this is Docusign Connect.

First: The basics

Docusign Connect is a webhook system that enables you to receive real-time updates to your envelopes as they occur. The obvious benefit to this is that your integration will be able to keep track of an envelope’s status within seconds instead of making requests for envelope status changes or polling for individual envelope statuses. Docusign Connect can be configured to write envelope data back to your system based on a number of events, such as: Envelope Sent, Envelope Completed, Recipient Declined, Recipient Failed Authentication, and more.

In addition to providing writebacks on many events, we also have four different Connect messaging models available as of April 2022.

Different models? What does that mean exactly? 

That’s a good question! Docusign Connect can follow two different writeback models: Aggregate and Send Individual Message (SIM). The primary difference between the two models is the aggregation component: if an envelope has a rapid set of Connect triggers that would send writebacks several times within a few seconds, those messages can be merged together, or in some cases the previous message might sometimes be duplicated or dropped off. The solution we have for this is to use our SIM model, which sends out each message individually without aggregation.

The other consideration is whether you’re going to want your writeback to come in an XML or JSON format. Docusign’s XML format is our older system and relies on having Docusign’s XML schema incorporated into your listener. One of the major downsides to this is that this schema is based on our SOAP API WSDL, which has not been updated in several years. This format will not support some of the newest Docusign technologies, so using a JSON model is recommended if your system requires writeback detail from one of these newer systems.

Additionally, the new JSON SIM model supports events related to templates and Docusign Click, which the other three Connect models do not.

That’s interesting, how do we go about securing this? 

Also a good question! We have a number of mechanisms in place for you as a customer to secure your Connect traffic.

Docusign Connect requires that you host a listener on an HTTPS port, meaning 443, 2443, 3443, 4443, and so on. Your web server must use a server certificate to enable HTTPS. The server’s certificate must chain up to a certificate in the Microsoft list of trusted root certificate authorities. 

In addition to this, we also have support for basic authentication headers, HMAC, the DocuSignConnectUpdate method available for SOAP, and Mutual TLS.

Am I going to need to open up my firewall? 

Not necessarily. Presently we have two general recommendations for how your firewall should be involved in this process.

Connect’s writebacks are sent from a number of internal devices based on certain internal factors. The writeback you see from one envelope vs. another can come from a completely different device with a different IP address. The most common way for a customer to set up their listener is to obtain the necessary digital certificates and IP ranges from our Trust Site, and allowlist all of them.

If you’re uncomfortable with this, or if this practice is disallowed by your IT group, what we would recommend is hosting your listener on an online system such as Azure or AWS. The intent is that your listener can be housed on a remote server where data is processed and stored as it comes in; then, when your integration is ready to check for updates or status changes, it can reach out to the listener when, and only when, you so choose.

What resources are available if we need help? 

Docusign does provide assistance with Connect troubleshooting, though be aware that troubleshooting a webhook system is a very technical process. To help us assist you with an issue like this, please make sure that you have a technical, preferably network-savvy, representative from your team ready to help. Access to the listener’s code base, the inner and outer firewall logs, and administrative access to the Docusign account we’re troubleshooting are also extremely helpful.

In addition, when you’re configuring your Connect object within Docusign Admin, you should enable two flags to assist with troubleshooting: Enable Logging and Require Acknowledgement.

• Enable Logging turns on the Docusign Connect logs that you as an account Administrator can access under Settings > Connect > Logs, where the last 100 logs are available. 

• RequireAcknowledgement enables a 100-second timeout period following which, if we do not see a response from your system, an error will be registered. With RequireAcknowledgement, Connect’s retry mechanisms are also enabled, which will attempt to redeliver a failed payload a number of times over the next few days. 

Lastly, within Settings > Connect > Publish, envelopes that have had recently registered writeback events show up on a publish list from which you, as an admin, have the ability to manually republish envelopes that have previously failed. This is useful in the event that you did not have RequireAcknowledgement previously enabled, or if you’re trying to sync a large number of envelopes via Connect instead of downloading them manually or via a set of API calls.

While we understand this is asking a lot, having these steps taken care of before we begin our support engagement will save a significant amount of cycles for everyone involved.

Now that the caveats are out of the way, let’s look at some real-world scenarios.

1. My envelopes seem to be updating 15-20 minutes after they’re completed // Initial Envelope publishing fails but it succeeds on retry… why?

   Connect’s writebacks come from a range of devices that have their own unique IP addresses. The writeback that occurs from one envelope event compared to the next attempt or the next event may come from a completely different IP address. If your envelopes don’t appear to update ~50% of the time, there’s a very real chance that you have some addresses within our publishing IP ranges that are presently being blocked. You should have your network team check the entirety of our IP ranges on both your inner and outer firewalls to ensure traffic isn’t being stopped there.

2. I’ve actually checked this already and everything looks good. There don’t appear to be any blockers at all.

   If the allowlisted IP ranges look good, the next step would be to check your Connect logs or request that Customer Support have a look in the back end to see if a specific error message is being displayed. Pending the error message, we may need to approach this from a different angle.

   Responses:

   200 or 201: This indicates the connection to your listener was successful. If you’re not seeing updates come through even with a 200 or 201, you’ll need to check how the connection is being routed internally. Under some circumstances it’s possible to send back a 200 or 201 when the listener establishes a connection, but the listening application does not process the message correctly.

   401 [Unauthorized]: This indicates that the payload being sent to your listener has not been properly authenticated. If you’re using a Basic Authentication header, ensure that the credentials being supplied are still current. Double-check any authentication components set up on the Connect side of things to ensure that they’re known good credentials.

   403 [Forbidden]: This indicates that Docusign's attempt to access the listener has been blocked. This can be due to local restrictions or, in some cases, a firewall issue. You can use tools like Connect’s envelope publish to retry previous Connect writeback attempts; if you can track where the requests are going within your internal network, you should be able to isolate where the connection is being blocked.

   500 [Internal Server Error]: A 500 error can mean a number of things. If you’re using an XML format, make sure that you’re using the current WSDL located on Docusign Developer Center. This can also happen if the listener encounters headers or data it wasn’t prepared for; when you have a file size limitation in regards to how large of a document the listener can support and said limit has been passed; the listener could simply be overloaded; or it could be a generic application crash. Unfortunately, a 500 error being returned does indicate that the software on the receiving side crashed, which is not something that Docusign can help you effectively troubleshoot.

   If all else fails,Customer Support would request a Wireshark log at this point. Wireshark has the ability to capture virtually all of a machine’s internet activity. We can examine the Wireshark log to see if there’s anything that stands out.

3. My Connect writebacks don’t appear to be sending out at all.

   First, make sure that the Enable Logging and RequireAcknowledgement flags for your Connector are turned on. Once you’ve done that, check the Connect logs under Settings > Connect > Logs to see what data is appearing, if any.

   If no data is appearing, but Connect is active: Check your Associated Users within Connect and make sure that the users sending and processing envelopes are enabled for Connect writeback.

   If the user is correctly associated, double-check the port that your Listener is operating on. Docusign Connect requires that specific ports be used, meaning 443, 2443, 3443, 4443, 5443, and so on continuing with an X443 naming convention. Using a different port can cause either the Connect payload not to go out, or it may register as an attempted send that has outright failed.

   If you’re seeing data but it ends with an SSL/Handshake error, double-check to make sure that your listener is operating with at least TLS 1.2 and has the proper digital certificates installed to facilitate the SSL handshake.

   If you’re seeing the proper TLS versions registered and have the correct certificates installed, Customer Support would then request a Wireshark log in order to have a closer look at the exchanges to see if anything stands out.

4. I’m told we’re using Webhooks, but I don’t see a Connect option in my account?

   That’s entirely possible. Connect is Docusign’s premium webhook system, meaning that you’ll need a certain plan for entitlement to use the system, or purchase it separately. If that isn’t an option for you, there is a separate way of invoking all four of these webhook systems strictly within an envelope definition. The specific object to be used for this task is the eventNotification object.

   When it comes to XML writeback, the determining factor is whether or not your account has Connect - Send Individual Messages active. This can be configured under Settings > Updates. You’ll also need to specify a writeback URL along with a set of envelope triggers, as well as whether you want Document information written back. For XML writeback it would look similar to this:

   Copy to clipboard

   "eventNotification": {
               "url": “{ListenerUrl}",
               "requireAcknowledgment": "true",
               "loggingEnabled": "true",
       	"includeDocuments":"true",
               "envelopeEvents": [
                   {"envelopeEventStatusCode": "Sent"},
                   {"envelopeEventStatusCode": "Delivered"},
                   {"envelopeEventStatusCode": "Completed"},
                   {"envelopeEventStatusCode": "Declined"},
                   {"envelopeEventStatusCode": "Voided"}
               ],
               "recipientEvents": [
                   {"recipientEventStatusCode": "Sent"},
                   {"recipientEventStatusCode": "Delivered"},
                   {"recipientEventStatusCode": "Completed"},
                   {"recipientEventStatusCode": "Declined"},
                   {"recipientEventStatusCode": "AuthenticationFailed"},
                   {"recipientEventStatusCode": "AutoResponded"}
               ]
           },
       
   

   Show more

   If instead you’re planning on using the JSON structure, you’ll still want to use the envelopeNotification object, but it needs to be formatted a bit differently. Note that with the XML configuration the includeDocuments flag needs to be included at the top level of the eventNotification object, which is not true when working with the JSON comparison. Additionally, and unlike XML, the determination between SIM and aggregate is declared within the eventNotification object:

   Copy to clipboard

   "eventNotification": {
               "url": "https://example.com/listener",
               "requireAcknowledgment": "true",
               "loggingEnabled": "true",
               "envelopeEvents": [
                   {"envelopeEventStatusCode": "Sent"},
                   {"envelopeEventStatusCode": "Delivered"},
                   {"envelopeEventStatusCode": "Completed"},
                   {"envelopeEventStatusCode": "Declined"},
                   {"envelopeEventStatusCode": "Voided"}
               ],
               "recipientEvents": [
                   {"recipientEventStatusCode": "Sent"},
                   {"recipientEventStatusCode": "Delivered"},
                   {"recipientEventStatusCode": "Completed"},
                   {"recipientEventStatusCode": "Declined"},
                   {"recipientEventStatusCode": "AuthenticationFailed"},
                   {"recipientEventStatusCode": "AutoResponded"}
               ],
               "eventData": {
                   "version": "restv2.1",
                   "format":  "json",
                   "includeData": ["custom_fields", "extensions", "folders",
                   	"recipients", "powerform", "tabs", "payment_tabs","documents"]
               }
           },
       
   

   Show more

   Note that the primary difference between the two is that the JSON format requires the use of the eventData object. Supplying an eventData object is what specifically tells our system that you want your data to be sent back in a JSON format. Unlike the XML definitions, the included data is also supplied under the includeData array. The other primary difference is that the inclusion of documentBase64 or envelope data is also supplied via the includedData array. If you want to specify SIM over aggregate when requesting JSON writebacks, simply add a deliveryMode flag set to true at the top level of the eventNotification object. This will invoke the SIM functionality along with the improved writebacks created for JSON SIM.

   You’ll need to ensure that the eventNotification object you’re sending is valid and has the proper parameters including requireAcknowledgement and loggingEnabled. Otherwise, error messages you receive will not only be invisible to you; they’ll be invisible to Customer Support as well. If you need access to the troubleshooting and configuration UIs, you need to have a conversation with your Account Manager regarding your account’s current Connect entitlement and what it’s going to take for you to be granted access to the product.

5. I’m using HMAC but my payload isn’t validating...why?

   When we do see this issue, and it’s been determined this is not due to user or process error, the most likely cause will be text entered by your recipients onto the envelope itself. Connect sends back individual values for each one of tha tabs associated with an envelope; however, sometimes with Text tabs a user enters line breaks via a copy/paste operation without necessarily realizing they’ve done it. When the customer-side listener goes to validate these values, if the encrypted payload is being read and processed as a string instead of a byte array the process will actually fail validation. The fix for this is to interpret the payload as a byte array instead.

   In rarer cases, the individual responsible for configuring the listener may not be aware that a new hash is sent with every Connect message. When they attempt to reuse a previous hash the values also don’t line up and will fail validation as a result.

In conclusion…

Connect is an incredibly useful feature that allows you to receive updates within moments of an envelope being successfully completed; however, with any complex system there will be required troubleshooting. Verification that the Connect system is sending out information can be done by Customer Support; however, once the Connect payloads are verified as being sent out there is very little visibility we have, as these connections have already left our network and are now reliant on what your system sends back to us. Please note that in most cases we cannot troubleshoot these kinds of connections for you, as they’re often related to a myriad of potential internal and network issues that need dedicated and trained resources on hand to help navigate. If your organization does not have access to these types of resources, I would recommend you reach out to our Professional Services group to have a listener built and navigated for you. If you’re running into issues not covered in this blog post, feel free to reach out to Customer Support so we can have a closer look, keeping in mind what I noted above in terms of what resources you should have on hand to begin troubleshooting this.

Docusign does provide assistance with Connect troubleshooting, though be aware that troubleshooting a webhook system is a very technical process. To help us assist you with an issue like this, please make sure that you have a technical, preferably network-savvy, representative from your team ready to help assist.

Additional resources

• Docusign Developer Center

• @DocuSignAPI on Twitter

• Docusign for Developers on LinkedIn

• Docusign for Developers on YouTube

• Docusign Developer Newsletter