Torc Robotics: self-play reinforcement learning lifts camera-based autonomous driving policies
Torc Robotics and academic partners say self-play reinforcement learning, not imitation of human driving logs, lets a camera-based end-to-end driving policy keep improving with scale.
- About 50,000 agent steps per second on a single NVIDIA A100 GPU
- About 1,000 times faster rendering
- 25 billion agent steps across 335,000 nuPlan scenarios
- 2.7 times higher for the imitation-trained model than the self-play policy
What Happened
At 100 km/h a truck covers close to 30 meters every second, and most of those seconds are uneventful. The moments that decide whether a driver is good — a stalled car in the lane ahead, a cut-in at speed, a merge that closes faster than expected — are rare on the road and therefore rare in recorded data too. Most end-to-end driving models today are trained with imitation learning, also called behavior cloning: the model is shown recorded human driving and copies what the human did, never driving on its own during training and never getting feedback on the outcome of its own choices. That means it learns the common case very well and the rare case barely at all, and it never practices recovering from its own mistakes.
- Gigapixel: a simulator that draws vehicles as plain boxes, lanes as thin strips and traffic lights as small spheres, rendering thousands of worlds at once on a GPU — robust driving does not need photorealism.
- Reinforcement learning trains a compact teacher of about 2.7 million parameters with the PPO algorithm in self-play for 25 billion agent steps across 335,000 nuPlan driving scenarios, with reward weights for collisions, comfort and progress randomized per vehicle so one model covers styles from cautious to assertive.
- Self-play DAgger distills that teacher into a transformer-based end-to-end planner (adapted from the DrivoR paper) that sees four rendered camera views and outputs a 4-second trajectory; at each situation the student reaches, the simulator is copied and the teacher drives on for 4 seconds to produce the student's target.
- Sim-to-real adaptation: the trained student is duplicated, one copy sees the rendering and the other the matching real camera frame, and only the real-image copy's perception backbone is updated while the planning head stays frozen — without the feature-matching term, the HUGSIM score drops from 33.2 to 18.5.
About 50,000agent steps per second
On a single NVIDIA A100 GPU — roughly 1,000 times faster than a photorealistic reconstruction simulator. The camera-based student trained on 150 million agent steps, which one GPU renders in under an hour at that rate but would take more than a month at the photorealistic simulator's speed.
- Imitation learning copies recorded human driving; self-play has one policy drive every vehicle in the scene and learn from all of them at once
- An imitation-trained end-to-end model stopped improving at around 100 million agent steps, while the self-play policy kept improving with scale
- Self-play DAgger reached the same level of driving in roughly 3,000 times fewer agent steps than training the camera model with RL directly
“Imitation learning is limited by the sparsity of useful long-tail driving scenarios, so it tends to plateau in performance. Self-play reinforcement learning greatly increases the density of these high-value scenarios to avoid this plateau, showing that it can enable much more robust, safe and performant planning that scales.”
Torc evaluated the approach on the HUGSIM benchmark, which reconstructs real driving scenes in a photorealistic simulator and lets a policy drive them in closed loop with surrounding traffic that becomes more adversarial across four tiers, from Easy to Extreme. The self-play policy scored best among the methods evaluated in the paper without using any human trajectories, with one exception: in the Extreme tier, where surrounding vehicles actively try to cause collisions, the imitation-trained version of the same model scored higher because it drove fast enough to outpace the aggressive traffic, while the self-play policy yielded, often got stuck and completed fewer routes. Across the full evaluation set, the imitation-trained model's average speed at collision was 2.7 times higher than the self-play policy's. Torc notes that a single aggregate benchmark number can't fully capture good driving behavior and can even reward the wrong behavior, which is why it evaluates policies from many perspectives, with many metrics and observers, in simulation and in real-world truck tests.
Torc is clear about what a box world can't teach: boxes and lane strips can't represent debris, unusual obstacles, weather or lighting, the student must infer from pixels what the teacher was simply given, and the adaptation step needs paired real and rendered views. The stated next steps are adding human driving data back into training so policies stay human-like, generating rarer and more informative starting scenarios, and eventually training camera-based policies with RL directly without a teacher. The work targets highway trucking, where popular urban datasets concentrate on objects within 100 m and speeds under 50 km/h while long-range highway data such as the TruckDrive dataset extends labels to 400 m, speeds above 100 km/h and trajectories to 800 m — and where the interactions that matter are the rarest in any log. Torc says the teacher in the paper is the same kind of structured self-play reinforcement learning policy it uses in production work, that this research was evaluated on passenger-car benchmarks, and that adapting it to heavy, articulated vehicles is part of the work ahead; the paper Scaling Self-Play for End-to-End Driving, written with Mila, NYU's EMERGE Lab and partner universities, will be presented as a poster at CoRL 2026 in Austin, Texas, November 9–12, and Gigapixel extends PufferDrive, with code coming soon.
Previously from Torc Robotics, Inc.
Torc has been consolidating the data and simulation work behind its autonomous trucks: in late September 2026 we reported that it merged data ingestion, processing, simulation and machine learning operations into a single DataLoop division inside its AI organization, with each truck mission producing about 9TB of data per hour. In early October 2026 we covered Torc's account of the three-pillar safety case behind its driverless deployment, which it describes as a structured, evidence-based argument built into development from day one.
- Torc Robotics Forms DL+S Division to Unify Data, Simulation and ML for Autonomous Trucks
- Torc Robotics Details Three-Pillar Safety Case Behind Driverless Deployment
Background drawn from MotorClaw's earlier coverage of Torc Robotics, Inc.'s official releases.
Why this matters
Most end-to-end driving models learn by copying recorded human driving, and those logs are thinnest exactly at the rare, high-speed moments that decide whether a drive is safe — a stalled car, a cut-in, a closing merge. Torc's research shows generated experience can fill that gap and keep improving with scale, which matters for automated highway trucking, where those rare interactions carry the most risk.
Terms in This Story
- Imitation learning (behavior cloning)
- A training method where a model is shown recorded human driving and learns to copy what the human did in each situation, without driving on its own or getting feedback on its own choices.
- Reinforcement learning
- A training method where a model drives in a simulator, tries actions and learns from a reward that scores what happened, such as progress made, collisions avoided and ride smoothness.
- Self-play
- A form of reinforcement learning in which one policy drives every vehicle in a simulated scene and learns from all of them at once, so the traffic keeps getting harder.
- DAgger
- A technique where a teacher shows a student what to do from the situations the student actually reaches, so the student learns to recover from its own mistakes.
Summarised from the linked release; details can be imperfect — always verify against the original source.