Summary
On July 30, 2025, Lokalise became unavailable and unstable at 11:55 UTC due to a new feature release. The new feature caused an unpredictable usage pattern with excessive memory usage and resulted in an ingress crash-looping. This was rectified and by 12:10 UTC, and Lokalise became fully operational again.
What happened?
During the update to one of our services, a configuration mismatch was introduced. The change, intended to increase the maximum allowed size, was not fully propagated across our infrastructure. This discrepancy between the service and the infrastructure layer led to excessive memory consumption, ultimately causing the service to become unstable and fully unavailable.
Our team identified the problem quickly, the expected configuration was correctly propagated and by 12:10 UTC the problem was resolved.
Impact
11:55 UTC: Lokalise's performance starts to degrade.
11:58 UTC: Lokalise is not available and an investigation is started.
12:05 UTC: Problem is identified and solution is applied.
12:10 UTC: Lokalise's performance is restored.
12:10 - 12:30 UTC: Performance is monitored and eventually the problem incident is resolved.
We sincerely apologize for the disruption this caused. Thank you for your patience and continued trust in Lokalise. If you have any questions, feel free to contact us at support@lokalise.com.