Jason Mancuso of Modal said async agent RL can cut weight-sync traffic 1,000-fold.
Mancuso told AI Engineer that swapping base models loses on cost and reliability. He argued production harnesses plus evals already form an RL environment. SFT breaks on branching agent traces, he said, while RL explores those branches.
His fix starts from GRPO, the method DeepSeek introduced for R1, Mancuso said. He adds masked importance sampling for token mismatch, score centering for drift, and top-P sampling with mask replay for rare tokens. Tests on Qwen 3.6 35B MoE showed steadier training, he said. Scores held on Stable Bench Pro and its 51-task hard subset, Mancuso said, adding "you should own your own agent in RL."
Sparse updates let Modal move inference off the training cluster.
Agent RL needs about two inference GPUs per training GPU, Mancuso said. Modal shards rollouts across B200s, H100s, A100s and mixed H200 pools at BF16 and FP8, he said. Cheap rollout hardware cut run cost 15 percent without destabilizing reward, he said.
Modal already bills Sandboxes as isolated environments for coding agents, RL rollouts and untrusted code, per its Snorkel work. The RL stack builds on Miles and SGLang, Mancuso said. Full code is slated for release very soon.
