As of 15:11 UTC, all queued notifications, including iOS push notifications delayed as a side effect, had either been successfully sent or were marked as permanently failed.
As of 16:48 UTC, our internal monitoring indicated error rates for sending push notifications to Android devices via Google's services had stabilized at normal levels.
At 17:05 UTC, Google confirmed their corresponding incident had been resolved as of 16:51 UTC.
Given both observed and confirmed upstream resolution, we're marking this incident as resolved.
Posted Apr 04, 2023 - 17:21 UTC
Update
We are continuing to experience a slight elevation in error rates when attempting deliver push notifications to Android users due to slow responses with Google's notification service although failed notifications are eventually sent to users due to our attempts to retry notifications that fail to send.
Google has indicated that their mitigation to drop the errors rates has made an improvement, but are still working to bring the notification service response times down to normal operation.
We are continuing to monitor our services, and will post an update when new information becomes available.
Posted Apr 04, 2023 - 16:13 UTC
Update
Google has indicated that their error rates for sending push notifications have decreased, and we have observed that we are successfully delivering push notifications to users.
We are going to continue to provide updates here every 30 minutes, and will not resolve this incident until Google indicates they have resolved their service outage.
Posted Apr 04, 2023 - 15:43 UTC
Update
We are continuing to monitor errors with delivering notifications to Android users. Google has indicated that they are putting mitigation in place on their notification services.
Posted Apr 04, 2023 - 15:08 UTC
Update
Google continue to have an incident with delivery push notifications to android app users.
Due to a mitigation put in place iOS users should begin to see notification delivered to their devices.
Posted Apr 04, 2023 - 14:33 UTC
Identified
The cause of the backlog of notifications failing to be sent to app users is caused by a Google outage of their notification services.
We are monitoring our system, and investigating ways to mitigate overall impact of the outage to users.
Posted Apr 04, 2023 - 14:14 UTC
Update
We have a put a mitigation in place, and have caught up on the backlog of messages that began being queued to be integrated around 13:04.
At this time, we also believe that notifications to app users are delayed. We are investigating on the cause of this to resolve notifications.
Posted Apr 04, 2023 - 14:00 UTC
Update
We are continuing to investigate the issue around delivering messages to messaging web services. We believe we have found a root cause and are working to put a mitigation in place to get messages flowing to integrations again.
Posted Apr 04, 2023 - 13:47 UTC
Investigating
We are currently experiencing a delay integrating messages sent by drivers to message integration services. Messages sent since 12:55 UTC are affected by this delay in integration.
We will post updates here every 20 minute during the incident, or when a new update is available over the course of our investigation.
Posted Apr 04, 2023 - 13:26 UTC
This incident affected: Eleos Platform (Mobile Apps).