Major service outage

On February 10, 2025, our service experienced a 16-minute outage followed by 39 minutes of degraded functionality due to an issue with a system configuration.

What happened?
While improving our disaster recovery process, a misconfiguration in the system was introduced unintentionally. Initially, this did not cause issues, but when we attempted to roll back the change, it led to unexpected complications. As a result, our service platform became temporarily unavailable, requiring the reconstruction of certain system components to restore full functionality.

Impact

• 12:30 – 12:46 UTC: Service outage.
• 12:46 – 13:25 UTC: Service degradation—APP and API were operational. Some services remained unavailable (OTA, Workflows, Connectors, Review Center).

What we are doing to prevent this in the future
We recognize the importance of this event, and we have taken steps to ensure it does not happen again. Our key actions include:

• Improving validation and monitoring processes to identify configuration issues before deployment.
• Enhancing our tools and automations for faster services restoration.
• Implementing blue-green deployment techniques, or equivalent, for seamless system upgrades.

We sincerely apologize for the disruption this caused and appreciate your patience as we work to make our systems more resilient. If you have any questions, please reach out to support@lokalise.com.