[Incident Report] Summary: Chat full outage When: 9:51 - 10:06a October 17 PDT (1:51 - 2:06a October 18 KST; 4:51 - 5:06p October 17 GMT) Impact: Chat full unavailability - SDK and Platform API calls failed with 5xx errors or time outs. Cause: An underlying AWS Redis network issue led to a burst of reconnection attempts which then led to a CPU spike. System resources stabilized as connectivity was restored and retries ceased. Remediation: Services successfully recovered after connectivity restored and instances relaunched as part of on-going maintenance. Long-term, additional backoff logic to manage reconnection attempts is necessary to avoid utilizing system resources.
Posted Oct 17, 2022 - 13:06 EDT
Investigating
We're experiencing an elevated level of API errors and are currently looking into the issue.
Posted Oct 17, 2022 - 12:51 EDT
This incident affected: Sendbird N Virginia 2 server.