Weights & Biases made its agent improve itself

Zubin Aysola of Weights & Biases demoed Arya, the company's self-improving research agent, live on AI Engineer after its general availability on Monday.

He followed a main-stage talk by his peer Tim the previous day with a deep dive that reused slides from his NeurIPS presentation.

The core claim is a tight flywheel between production traces and an offline simulation that run bite-wise identical agent code.

On one side Weights & Biases Weave captures every production trace in the same format as offline runs.

On the other side a research sandbox hydrates YAML configs, loads full training logs, and can even simulate GPU executions.

A four-hour sync job mirrors production code into research so hill climbing never drifts.

That loop let Aysola ask Arya to take a live production trace, log it as a task, and spin up training jobs against its own codebase artifact.

The demo prompt instructed the agent to launch evaluations, review traces, and write a new variant of itself.

He also showed a sanitized production view covering seven weeks of nightly CI jobs that benchmark candidate variants nightly.

Those jobs hover around 66 percent on the tracked task sets, and one break last night required Arya to fix itself before the talk.

Scoring is where he spends most of his time, with two modes: normative pass or fail and relativistic style comparisons.

The task library now holds 886 tasks categorized by levels and exposed to product teams for relevance checks.

Tasks range from single instructions to multi-turn personas that simulate users interacting with Arya in sequence.

The agent harness itself is deliberately simple and model agnostic, tested against CoreWeave inference and foundation model providers.

Context handling, compaction, and UI payload assembly are abstracted so YAML can spawn many parallel variants cheaply.

In the live run Arya converted a real production error, a missed weave.log SDK call in the sandbox, into a WBAF regression task and re-ran prod versus candidate.

He has not written code directly in about eight months, directing Claude to implement changes while he focuses on guardrails.

The limitation remains operational: simulation is expensive, evaluations take time to run, and the gap between offline scores and live behavior still needs manual review.

Outside Weights & Biases, others chase the same loop, with Nous Research describing its hermes-agent as a self-improving AI agent that grows with you.

For now Arya also trains models on H200s on CoreWeave infrastructure and runs auto-research on Karpathy's nano chat without leaving the Weave platform.