Spinning the Data Flywheel: From Curation to Fleet

Most autonomous driving systems today improve through the same loop. Each continuous recording session of one fleet vehicle is a drive. Drives are processed, labeled, and curated into training and evaluation sets. A new model trains, passes evaluation, and ships, and the fleet's new drives and failures feed the next round. The industry calls this loop the data flywheel. How well each stage works decides both how fast the flywheel turns and how much each turn improves the model.

At Applied Intuition, this loop runs over hundreds of petabytes of multimodal data, growing by more than a petabyte a day, from fleets in multiple verticals, such as automotive, trucking, mining, and robotics. Before curation, ingestion writes each drive's raw logs to a log store built for reading, frame generation cuts the drive into 30-second segments aligned onto shared frames, and enrichment labels them. This post picks up from there with a high-level tour of the stages that turn labeled frames into a model, all the way back to the fleet: curation, scenarios, data loading, training, and evaluation. For each one, we cover what it does, how it works, the challenges it raises, and what has worked for us at this scale.

Curation: Choosing Segments to Feed the Model

Curation selects the segments that make up each training and evaluation dataset. It runs against a curation table whose columns hold metrics derived from sensor and vehicle signals, such as speed, lighting, or the gap to a cutting-in vehicle, and record which labels exist and which labeler version produced them. The table is hundreds of columns wide and widens as the platform grows. Every metric registered for production gets a column, and so does every label version cleared for production, while retired versions drop out.

A dataset's selection criteria combine metric filters, drawn from a shared library of hundreds, such as nighttime driving with a cut-in under two seconds ahead, with label requirements, such as labeled dynamic actors within a given distance of the vehicle. The criteria run as a query over the curation table, and a segment qualifies only if every filter passes.

We keep refining metric definitions and backfilling them over past drives, retraining labelers, and changing selection criteria. Some scenarios aren't described by any metric or label yet. To find them, a video-text embedding model maps driving clips and queries into one shared space, with the embeddings stored in a Lance table. Lance's native vector search answers a text, image, or example-clip query fast enough to refine it interactively or programmatically, so an idea for a rare scenario becomes a candidate dataset without first waiting for a new metric or labeler. The criteria are versioned and run on a schedule to pick up matching drives as they arrive. Each selected segment also gets a label set, one version of each label it needs, chosen together because labels depend on each other. Dynamic actor labels, for example, depend on a localization run, which depends on a calibration version.

The challenge is the latency and accuracy of auto-labeling, which curation depends on. To cut latency, we queue only segments that pass quality filters for auto-labeling. Priority tiers, set by vehicle, drive, or date range, run first. To improve accuracy, we regularly compare dynamic-actor auto-labels with held-out human labels. Embedding search then finds scenes in unlabeled drives that resemble those where the auto-labeler fails most, and those segments go to human labelers, whose labels train the next version of the auto-labeler.

Scenarios: Assembling Data for Training

Training samples scenarios, windows of up to 300 aligned frames, each stored as one row with its labels, map, route, and pointers to the sensor bytes. We assemble scenarios from curated segments, joining each segment's frames with its label versions and splitting them into windows, one scenario per window. Each scheduled curation run then writes to the scenario table: it inserts new scenarios, updates scenarios whose labels or processing changed, and deletes scenarios that no longer meet the criteria.

We store scenarios in Lance tables, with each scenario's payload (frame metadata, labels, map, and route) in a blob column. Lance's blob v2 API makes that blob data easy to work with: it is written and versioned with the rest of the row, and read only when needed. Lance's data evolution adds a new tag or derived field as a column and backfills existing rows without rewriting the data files. Fast random access to individual rows lets training sample and shuffle scenarios across a large table.

The challenges are write latency and read performance. Writers cut latency by spreading skewed input evenly and fanning out across more workers. More parallel workers, though, write many small fragments to the Lance table, which slow reads. Compaction jobs run on the table regularly to keep reads fast, merging small fragments into larger ones and dropping deleted rows.

Data Loading: Turning Scenarios into Model Inputs

Data loading turns scenario tables into batches of model inputs for training. Each dataset is read at a pinned version; targeted datasets, such as hard cases or a new vehicle platform, feed model fine-tuning the same way. Each worker reads its own shard of the training split in shuffled batches. The loader then takes the sampled rows, hydrates them with the sensor bytes they point to, and transforms frames into model inputs such as camera views, lidar, ego history, map rasters, and routes.

The scenario tables are Lance tables, where a row ID resolves straight to its fragment and offset. Because Lance files have no row groups like Parquet's and their pages are laid out for partial reads, a take, Lance's read of rows by ID, fetches the requested rows and columns with a small, predictable number of I/O operations. Training samples rows non-sequentially in small, scattered batches, and each batch of sampled IDs is a single take. Because rows hold pointers rather than sensor bytes, the take stays small, and the sensor bytes are fetched from the log store in bulk, many consecutive frames per request rather than one request per frame.

Data loading runs mostly on CPUs and I/O. Its first challenge is throughput, the number of training samples per second the loader delivers for each GPU at a CPU and memory cost the training nodes can afford. Each stage can cap it in turn: object-store latency during hydration, CPU-side transforms such as decoding images and rasterizing map layers, memory for frames held in flight, and the hand-off between processes. Its second challenge is resuming cleanly. Long runs restart, so a checkpoint stores how many records the sampler has read, and a restarted job resumes from there instead of replaying.

Training: Imitation, Then Reinforcement Learning

Training runs in two stages. Imitation learning comes first. Each scenario is replayed in open loop from the log, and the model learns to plan by matching the path the human driver took, while the next frame comes from the recording whatever the model does. Map and route labels become inputs, and the tracks of other road users are targets for the model's detection and prediction heads. The resulting checkpoint drives like the fleet's drivers and is the starting point for reinforcement learning, which continues in closed loop. A scenario seeds a driving simulator, the policy drives the ego vehicle itself and is scored at each step on how well it drives, and a collision ends the episode. Many simulators run at once, and policy-gradient updates improve the policy from the rewards.

In imitation, the recording produces the next observation; in reinforcement learning, the simulator produces it in response to the policy. That lets reinforcement learning teach the model from the consequences of its own actions, including situations the recordings never show, while imitation teaches it to drive like the expert. The output is a candidate model for evaluation.

Imitation learns only what the logs cover, so coverage of rare scenarios matters more than raw volume, and a model that drifts into situations the expert never met compounds its own errors. Reinforcement learning learns whatever the simulator rewards, so sensor realism, reactive traffic, and well-shaped rewards decide whether its skills carry over to the road. For both, training throughput sets how long a run takes to converge, and with it how many experiments fit into a day. Removing per-minibatch GPU-to-CPU synchronizations and queuing the next update asynchronously have kept the GPUs busy, and skipping frozen parts of the model has cut the work itself.

Evaluation and Deployment: Closing the Loop

Each candidate runs through evaluation suites. Open-loop suites replay curated scenarios, score the model's plans against the logged path, and run behavior checks, such as whether a lane change avoids a collision. Closed-loop suites let it drive reconstructed scenes in simulation. A release gate compares the candidate with the baseline, the current release, on the same scenarios, and blocks on a statistically significant regression, a broken hard threshold on a suite's metrics, or a failing check on hand-picked sentinel scenarios. Open-loop and closed-loop scores can disagree (Dauner et al., 2023), so neither one alone is enough. An engineer reviews the results and signs off.

A signed-off model is compiled for the vehicle's onboard computer, verified on vehicle hardware, and road-tested before it reaches the wider fleet. The fleet's new drives come back as raw logs and start the next turn of the flywheel, and people curate the model's failures, such as disengagements, into evaluation sets for the next candidate and hard-case datasets for fine-tuning.

The first challenge is coverage. Each suite targets specific behaviors, such as lane keeping or traffic lights, so an improvement on one can hide a regression on another, and coverage has to grow as new failure modes appear. The second is deployment. Vehicle platforms differ in sensor layout, which changes what the model sees, and in dynamics and actuation, which change how the planned trajectory is tracked, so the model has to be verified on each.

Keeping the Flywheel Spinning

These stages make one turn of the flywheel. Ingestion, frame generation, and enrichment turn each new drive into aligned, labeled frames. Curation selects segments for each dataset, those segments become scenarios, and data loading feeds them to the model. Training produces a candidate, evaluation decides whether it's better, and the fleet takes it back onto the road, where its new drives start the next turn.

The stages are coupled, so both quality and efficiency carry from one stage into the next. A label error passes through curation into every dataset built from it. An evaluation failure should update the curation criteria, and embedding search can show which scenarios the datasets lack. In the infrastructure, tuned writers and regular compaction let new data land quickly without slowing training's reads, and how fast the data loader delivers samples and how efficiently the GPUs process them together set training throughput. The gains ahead come from working on both data quality and the efficiency of data processing and training. That way, the flywheel turns faster and gets better at choosing the right data with every turn.