Application for Agents
Incident Report for LiveChat
Postmortem

Yesterday at 10:26 UTC we identified a problem with main database cluster. The dedicated team that handled this incident was making sure that it was solved as soon as possible. No estimation was made at that point due to the impact of the incident - we needed all developers' hands on deck. We managed to find the source of the problem and implement the fix as soon as possible. A direct implication of the issue was a separated case where our webhooks queue was overloaded. It caused a degradation of service for the next couple of hours. The whole situation was stabilised at 4:30 PM UTC.

The issue affected all agents that were trying to log into the application and new website visitors that tried to start the chat.

As a form of protection from these kinds of problems, we implemented additional double-checking procedures connected to all databases deployment and added additional metrics development to improve infrastructure monitoring. Additional safety measures and procedures were added to our roadmap as well.

We understand that the issue had a huge impact on your businesses and we are very sorry that this came from us.

LiveChat Team

Posted Oct 25, 2018 - 18:33 CEST

Resolved
The problem has been resolved, we will publish a post-mortem tomorrow, after admins sum up the incident.
Posted Oct 24, 2018 - 20:15 CEST
Monitoring
A fix has been implemented, servers are stabilised, and webhooks-based integrations are working properly. We are monitoring the situation.
Posted Oct 24, 2018 - 18:43 CEST
Identified
We are investigating the issue with agent app connections, our team is working that and we'll update you shortly.
Posted Oct 24, 2018 - 16:55 CEST
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Oct 24, 2018 - 15:01 CEST
Update
The system is getting more stable now. We are seeing more agents logging in to the system and the chat window coming back up. We continue monitoring the situation.
Posted Oct 24, 2018 - 13:56 CEST
Update
We are continuing to investigate the issue. Additional components were affected: chat widget, API, and support chat with us.
Posted Oct 24, 2018 - 13:10 CEST
Investigating
We are currently experiencing issue that can prevent agents from logging into the LiveChat app for agents. Our administrators are already investigating the problem, and we will do our best to resolve it as soon as possible.
Posted Oct 24, 2018 - 12:38 CEST
This incident affected: Chat widget, API, Agent apps, Integrations, and Support chat on www.livechat.com.