AI Agents Build Better AI

Artificial intelligence is no longer just a tool for end-user products; it's now a critical component in building more sophisticated AI itself. This shift is evident in how AI is optimizing infrastructure, training workflows, and the very systems used for AI development. LinkedIn Engineering began exploring this in August 2025, using agent loops to refine LLM post-training runs. The initial success was not just in task automation but in creating a structural loop of proposing, testing, measuring, and improving.

This realization spurred an internal project in January 2026 with a clear goal: leverage AI to enhance AI systems, necessitating platforms designed for a central role for agents. This led to a strategy focused on unifying three pillars for scaled experimentation: agents for distributed training code, comprehensive evaluation systems, and efficient GPU microscheduling. This framework enables agents to parallelize model trials with minimal human oversight.

Within this setup, agents optimize for both model quality and training efficiency in an inner loop. Once an optimal architecture is found, it's scaled through distributed training in an outer loop. This approach was first applied to migrating LinkedIn's large fleet of TensorFlow models to PyTorch, resulting in Autopilot for Torch. This specialized agent doesn't just convert; it iteratively refines generations based on LLM reasoning and verifier feedback.

The pattern quickly expanded to other use cases like kernel generation and auto-tuning, where agents autonomously search, evaluate, and enhance system performance. The core loop is a cycle of generate → verify → refine.

The Iterative Refinement Loop

Autopilot for Torch operates on a continuous generate → score → hint → regenerate loop until target metrics are met. Each iteration undergoes rigorous quality gates, with the verifier providing specific, actionable fixes rather than just a pass/fail signal. Once targets are achieved, the PyTorch implementation is validated on GPU pods and deployed via Flyte workflows.

This autopilot system is now applied to engineering problems where the output is AI infrastructure, models, or performance-critical code. This includes framework migration, model code generation directly from datasets, autoresearch for architecture optimization, and kernel generation for low-level GPU optimization. The common thread is not just code generation, but verifiable AI system building, where clear checks ensure iterative agent loops refine outputs effectively.

Defining Success: The Scoreboard

The system operates on a 'trust, but verify' principle. The scoreboard isn't an afterthought; it defines what 'good' means for the agent loop. Reward design is crucial, as agents optimize based on what they are rewarded for. Shallow rewards lead to shallow fixes.

The evaluation hierarchy prioritizes functional correctness. If a system cannot run, learn, or stabilize, other scores are irrelevant. Functional validity is a hard gate. Behavioral parity ensures expected outputs on representative inputs. Structural checks confirm component integration. Quality checks align with target stack conventions for maintainability. Finally, task-level metrics measure real-world performance.

Verification progresses in difficulty, starting with cheap structural and style checks and advancing to trainability, IO parity, numerical stability, and task-level metrics. This phased approach optimizes loop efficiency and builds confidence.

For model code evaluation, the rubric includes trainability (stability, convergence), IO parity (behavioral consistency), structural fidelity (architecture preservation), code style, and task metric parity (downstream quality). The loop leverages both failures and successes for structured feedback, preventing redundant work and accelerating improvement.

Reinforcement Through Structured Feedback

Reinforcement within the loop comes from the verifier, which provides structured natural-language feedback. This feedback acts as a coach, detailing weaknesses and suggesting fixes. Each piece of feedback is typed (e.g., NO_GRADIENT, NUMERICAL_INSTABILITY), prioritized (P1 for critical, P4 for minor), and actionable, including metrics, targets, and suggested directions.

Precise feedback drives systematic improvement, focusing on high-value changes first, like fixing trainability blockers before stylistic refinements. The verifier transforms evaluation into guidance, translating rubric failures into targeted actions.

Complementing this client-side strategy is a server-side tracking tool, the Autopilot Tracking Console. It offers a centralized view of active and completed conversions, training jobs, and Flyte executions, including status, metrics, and artifact links. This is vital for monitoring long-running jobs and reviewing historical runs.

This approach is driving higher productivity, with more models undergoing agentic migration and auto-tuning with significantly less manual effort. Early results show strong performance across benchmarks and the ability to match offline metrics for internal workloads. This system's success hinges on core design decisions: scoring-based iterative loops, natural language reasoning for feedback, rapid failure detection, modular and deterministic scoring, and bounded iterations to prevent infinite loops.

Comprehensive evaluations, including N-day replays against production traffic, build confidence. GPU microscheduling ensures cost-optimized compute consumption for massive experiments. This is an exciting area of development, with more details to come.