Uber Fights Bounding Box Errors

Training custom machine learning models for specific business needs demands high-quality data, often sourced from human annotations. However, these annotations, particularly for video, are prone to errors. Uber Engineering has developed an ML-based system to tackle these bounding box annotation errors, aiming to ensure data integrity before it feeds into model training.

The challenge lies in video annotation, where long footage is split into segments for operators, creating opportunities for mistakes during the rejoining process. Traditional human review workflows are costly and inconsistent. Uber's solution, integrated into their in-house tool uLabel, offers real-time, automated validation.

The Problem with Manual Review

Human annotators can make mistakes. A second pair of eyes helps, but it doubles cost and time. This sequential process is inefficient for large-scale projects.

Uber's ML-Powered Solution

Uber's system automatically detects critical annotation errors like ID swaps (a tracker mistakenly following the wrong object) and position jumps (unexplained shifts in coordinates). These are the most common and impactful failures, according to the Uber Engineering blog.

Why It's Tricky

Detecting these errors isn't straightforward. What looks like an error in one context might be normal in another. Object size, motion, camera movement, scene complexity, and even frame rate all influence what constitutes an anomaly. A 10-pixel shift is negligible for a car but significant for a distant pedestrian.

Fixed rules like "flag any jump greater than X pixels" are insufficient because they can't adapt to varying conditions.

Architecture for Accuracy

The validation pipeline uses an 11-frame sliding window to analyze features across visual, motion, and coordinate data. An XGBoost classifier then scores each frame for error probability.

This approach processes raw video and annotations, extracts features, classifies potential errors, and clusters them into actionable groups for human review.

Synthetic Data for Robustness

Since real-world errors are rare, Uber generates synthetic data by introducing perturbations that mimic human mistakes. This includes simulating ID swaps and position jumps across various magnitudes and distances.

This synthetic dataset, derived from six open-source datasets, ensures the system generalizes across diverse scenarios, from autonomous driving to crowded scenes. This focus on improving the quality of machine learning data labeling is critical.

In-Tool Validation and Future Plans

The system flags issues directly in uLabel, allowing operators to correct them or dismiss the suggestion. Uber is already deploying this solution across its bounding box annotation projects and plans to expand it to cover more error types, further enhancing video annotation quality.

This automated validation significantly improves data quality and streamlines workflows, contributing to more robust machine learning and robotics systems.