Secondary database cluster issues
Incident Report for Labelbox
Postmortem

At 5:13pm PST a configuration change was initiated on a secondary database cluster which eventually failed, but in the process the cluster was incorrectly handling requests and our application was seeing degraded performance. In particular, metrics and labels were having problems being loaded correctly. When we canceled the configuration change and rolled back, the cluster came fully back online and the application performance was restored. We’re working on making sure that our configuration change process and rollback process is fully ironed out, and that we have appropriate failovers should an issue with our secondary database cluster occurs again.

Posted Nov 07, 2019 - 13:46 UTC

Resolved
The incident is now resolved and we will be following up with an update.
Posted Nov 06, 2019 - 01:32 UTC
Investigating
We are currently investigating this issue.
Posted Nov 06, 2019 - 01:19 UTC
This incident affected: Platform (common services).