Summary

An issue in our automated deployment tooling caused Labelbox services to become unavailable. The malfunction did not affect any data storage, database, network, security, or other infrastructure. A change to the automated deployment tooling was reverted and services were automatically restored. 

Details

A configuration change was made Friday, October 20th at approximately 15:45 PT (22:45 UTC) that was believed to be benign. However, when this change propagated through the configuration system it had unintended consequences. This resulted in Labelbox services becoming unavailable for approximately one hour.

The malfunction did not affect any data storage, database, network, security, or other infrastructure, and no data was lost or affected by the outage.

The change to the automated deployment tooling was reverted and services were automatically restored. The restoration of the services took some time to stabilize. The incident was deemed fully resolved at 16:56 PT (23:56 UTC)

‌

Labelbox has already completed several changes to prevent this type of issue from being able to occur again.