Service degraded for Labeling

Dear Customers,

We wanted to bring to your attention a recent labeling operation service degradation on the Labelbox platform. We will provide details on the cause and remediation as well as assurance on what actions we’re taking to prevent such incidents happening in the future.

Summary: On April 25, 2024, around 9:25 AM UTC, we experienced service degradation with labeling operations. This manifested mostly as general slowness in the product. An investigation was launched and the root cause was determined. Subsequently a fix was implemented and deployed within a few hours by 2:59 PM UTC.

Root Cause:

• Task queues were overwhelmed around the same time data rows were deleted. Reconciling these deletions for data consistency resulted in a database deadlock.
• Corrupt data was introduced into our global data processing pipeline causing it to halt and impact all services at Labelbox.

Resolution:

Upon identifying the root cause, fixes were implemented.

• Data row deletion reconciliation was decoupled from task consumer. Size of messages batch was also reduced to simplify and reduce the size of database query thus greatly lowering the probability of any deadlock.
• The corrupt data was identified and eliminated. We have implemented a code-patch to prevent similar corrupt data from impacting our global data processing pipeline.

We apologize for this inconvenience. We are here to make Labelbox the most reliable and responsive data-centric AI platform, and will provide support in any way we can for your continued use.

‌

Sincerely,

The Labelbox team