Between 2023-11-13 at 23:05 UTC and 2023-11-14 at 00:01, for a total of 56 minutes, the Eleos Platform failed to process approximately 45% of incoming requests. At 23:14 UTC, our on-call engineers were paged due to the elevated error rates, and we started taking efforts to stabilize the errors by scaling up the processing capabilities for the system. This temporarily reduced the error rate, as the errors started to decline around 23:36 UTC, but then started to increase again at 23:49 UTC. Our engineers continued to take efforts to scale up the system and at around 2023-11-13 23:57 UTC, the failure rate started to decline, and by 2023-11-14 00:01 the error rate had returned to normal. During the incident window, drivers using the mobile app would have experienced intermittent issues with logging in, fetching their loads and sending messages, for example, and the app would have essentially functioned as if it was in offline mode or experiencing marginal network conditions. This also impacted requests to our public-facing APIs, such as retrieving documents from the document API and sending messages to drivers.
The underlying issue was due to an unanticipated interaction between the subsystem responsible for recording web service integration errors and overall request processing. The subsystem responsible for dealing with client API errors got behind on processing errors, which held up other subsystems, resulting in widespread request failure. The responding engineers worked to rectify this problem and saw the error rate drop dramatically.
While the responding engineers worked to relieve the above issues, less processing power was being used by the Platform servers since normal request processing was reduced due to the above issue. This caused the automatic scaling process to scale the number of running servers down, which further exacerbated the problem. The engineers immediately intervened and forced the automatic scaling process to scale up instead of down.
To prevent this from happening again, Eleos has identified and confirmed the underlying issue through logging, application tracing, recorded metrics, and new testing methods designed to identify unexpected interactions like this one. We are working to implement and release a complete fix. Until this work has been completed, mitigations are in place to prevent a recurrence.