Increased Merge Queue Times and Errors Upon Opening Templates in Documents for Salesforce
Incident Report for Documents
Resolved
As of approximately 4:50 AM EST, both the queue backlog and merge delay times are completely within normal range. Thank you for your patience and understanding during this incident.
Posted Feb 17, 2022 - 13:05 UTC
Update
Our monitors indicate that both the merge queue times and the queue backlog are decreasing at a healthy rate. We have made some adjustments which will facilitate faster processing of the existing backlog. Currently the backlog, and by extension the delay, are quite substantial. If the current processing rate continues, we are optimistic that most, if not all of the backlog will be cleared by tomorrow morning. Thank you for your continued patience and understanding as we work to address this incident!
Posted Feb 16, 2022 - 23:58 UTC
Update
We are beginning to see a decrease in merge queue volume as well as delay times. We are making some adjustments that we believe will further expedite the processing of backlogged requests and further reduce the delays.
Posted Feb 16, 2022 - 21:14 UTC
Update
A quick update: We are seeing some encouraging signs as we continue to monitor the queue backlog and delays. Specifically, we are seeing that both the delays and the backlog are beginning to level out. We're not out of the woods yet, but we are cautiously optimistic that this trend will continue. We will update again once there is a sustained downward trend, or in the event of a significant development.
Posted Feb 16, 2022 - 18:59 UTC
Update
We are seeing some encouraging signs. The queue is actively processing jobs at a steady rate, however delays are still quite high due to the volume of backlogged requests. We will continue to monitor things on our end and hope to update you soon to announce declining delay times.
Posted Feb 16, 2022 - 15:33 UTC
Update
Good morning! We thought we'd share a quick update: after much caffeination and deliberation, we have formulated a game plan that will hopefully see us return to a semblance of normalcy sooner rather than later. Some users are still seeing what we would at this point call "indefinite delays" in merge requests. If this is you, we apologize deeply for the inconvenience this has caused. We will continue to work feverishly to bring things back to normal. Updates to follow.
Posted Feb 16, 2022 - 13:16 UTC
Update
Quick Update: We are continuing to experience unusual delays in processing merge requests. Affected merge requests should still be processed in the order in which they were received, however at a much slower rate than normal. We will update again once we have more significant information to share.
Posted Feb 16, 2022 - 00:43 UTC
Update
We continue to see escalated merge queue times. We are working feverishly to address the issue as quickly as possible without disrupting stability. At the moment, merge queue times are around 6 hours. We hope to share an update soon regarding decreasing queue times.
Posted Feb 15, 2022 - 23:53 UTC
Update
Just a quick update: we are still seeing merge delays increasing rather than leveling off or decreasing. We are still processing requests, just at a much slower rate than normal. Right now, delays are approximately 2 hours and 15 minutes.
Posted Feb 15, 2022 - 19:55 UTC
Monitoring
The deploy has finished and looks to be successful. At the moment, we are seeing merge delays of approximately 80 minutes. We will continue to monitor and will update once there has been a sustained decrease in merge times.
Posted Feb 15, 2022 - 18:52 UTC
Identified
We have been made aware of an issue where users are experiencing an error when attempting to open certain templates in Documents for Salesforce. We have already identified the cause and are in the process of implementing a fix. During the deployment window, users of Documents may experience some delays in merge processing. We expect the deployment to be complete in approximately an hour. We will update you once the deployment is completed.
Posted Feb 15, 2022 - 18:05 UTC
This incident affected: API & Document Generation and Documents For Salesforce.