Hugging Face's Ben Burtenshaw on AI System Engineering

Ben Burtenshaw from Hugging Face recently presented on the potential of AI agents in system engineering, arguing that coding agents should be leveraged for these complex tasks. In his talk, Burtenshaw highlighted how AI agents are becoming increasingly capable, moving beyond simple code generation to more sophisticated system-level engineering.

The Role of AI Agents in System Engineering

Burtenshaw emphasized that AI agents are no longer just tools for writing snippets of code; they are evolving into sophisticated collaborators capable of tackling intricate engineering challenges. He pointed to the increasing acceptance of coding agents, citing examples from Andrej Karpathy and DHH who have been using them for years. This acceptance is growing as agents demonstrate their ability to perform tasks like discovering APIs, connecting systems, and even managing home automation devices.

Custom Kernels and Performance Optimization

A significant portion of Burtenshaw's presentation focused on the creation and optimization of custom compute kernels, particularly for AI workloads. He explained the fundamental components of a kernel, a function compiled to run on a GPU and executed from Python, and highlighted the importance of optimizing these for efficiency. Burtenshaw showcased how custom kernels, like the popular Flash Attention, can significantly increase arithmetic density, reduce time spent communicating tensors, and ultimately keep GPUs running at optimal performance.

He also introduced Hugging Face's 'kernels' library, a platform designed to facilitate the building of compute kernels. This library aims to enforce a unified and predictable structure, ensure reproducibility, offer native PyTorch compatibility, and foster community sharing. Burtenshaw demonstrated how developers can publish their own kernels to the Hub, making them accessible to others.

Benchmarking and Agent Performance

To illustrate the effectiveness of agents in this domain, Burtenshaw presented benchmarking results. He shared how agents were used to generate CUDA kernels, which were then benchmarked and optimized. A specific example highlighted an average speedup of 1.94x on an H100 GPU for a Qwen3-8B model when using agents to generate and optimize kernels. This demonstrates the tangible performance gains achievable through agent-assisted engineering.

The Power of Multi-Agent Autoresearch Labs

Burtenshaw also delved into the concept of multi-agent autoresearch labs, outlining a system composed of specialized agents working collaboratively. This system includes:

• Researcher: Scouts Hugging Face papers for ideas and defines research directions.
• Planner: Acts as a central coordinator, owning the experiment queue and proposing hypotheses.
• Worker Agents: Execute experiments, fetching code, and testing hypotheses.
• Reporter: Monitors the progress of jobs, synchronizes status, and provides an overview of active jobs and anomalies.

This multi-agent approach allows for a systematic and automated exploration of hyperparameters and model architectures, leading to more efficient and effective research cycles. The use of tools like Trackio for monitoring and visualizing these experiments provides crucial insights into the research process.

Key Takeaways

Burtenshaw concluded with several key takeaways:

• Agents work best with primitives and exposed, well-defined interfaces, rather than overly abstract ones.
• The Hugging Face Hub is a robust platform ready to support AI workloads with core infrastructure for storage, compute, and versioning.
• Multi-agent systems can be effectively structured with specialized roles to automate and accelerate AI research.

The presentation underscored the growing capabilities of AI agents in system engineering, highlighting their potential to drive efficiency and innovation in the field.