HeyGen: HTML is the Key for AI Video Agents

James Russo, co-creator and tech lead of HyperFrames at HeyGen, presented a compelling case for HTML as the universal language for AI agents tasked with video creation. At the AI Engineer World's Fair, Russo argued that while coding agents have democratized software development, the process of launching and promoting products through video remains a significant hurdle. HeyGen's mission is to bridge this gap by providing a robust platform for AI-driven video communication.

StartupHub data

HeyGen

HeyGen is an AI-powered video platform that enables users to create high-quality videos with realistic AI avatars and multilingual voice cloning, transforming...

Founded

2020

Location

San Francisco, United States

HeyGen's Vision for AI-Powered Video

Russo explained that HeyGen's core offering starts with high-quality AI avatars, often referred to as the 'A-roll.' However, he emphasized that truly effective videos require more than just a narrator; they need B-roll, animations, captions, and music to create a polished final product. HeyGen aims to provide AI agents with the necessary canvas to compose these elements seamlessly.

HTML as the Native Language of LLMs

The central thesis of Russo's presentation is that HTML, CSS, and JavaScript are the native languages of Large Language Models (LLMs). He posited that most LLM training data is derived from web content, making these web technologies the most intuitive and efficient communication channel for AI agents. Attempting to force LLMs to use custom domain-specific languages (DSLs) or JSON structures is akin to asking a master of one language to write poetry in another; it limits their potential.

Russo cited prominent figures in the AI community, such as Andrej Karpathy and Tarik, who have also highlighted HTML's utility as a new form of 'markdown' for LLM outputs, enabling visual representations. He shared his own conclusion that aligning AI agents with their native languages leads to superior results.

Evaluating Video Creation Tools

Russo presented a quadrant chart comparing various video creation tools based on 'Quality of Results' and 'Agent-Friendly' metrics. He placed professional tools like After Effects and Premiere Pro at the high-quality end but noted their lack of agent-friendliness, often serving as co-pilots rather than autonomous creators. Tools like Lottie and Rive, while using code-based formats, were deemed less agent-friendly and controllable due to not being native LLM languages.

Remotion was acknowledged as a strong contender, but Russo pointed out that it still required training agents on a specific framework. This led HeyGen to explore HTML directly. The breakthrough came with the advent of models like Gemini 3 Flash, which demonstrated a significant improvement in generating workable HTML, CSS, and JavaScript code with simple prompts and examples.

HyperFrames: HTML to Video

HeyGen's solution, HyperFrames, is an open-source framework that translates an agent's HTML output into video. The core challenge was ensuring deterministic rendering, as browsers are inherently asynchronous. To address this, HyperFrames freezes the browser's clock and seeks frame-by-frame, ensuring that all assets are loaded and ready before capturing each frame. This process guarantees that the video output precisely matches the browser's preview.

The power of this approach lies in its ability to leverage anything renderable in a browser, including 3.js, charts, SVGs, shaders, WebGL, WebGPU, and Lottie animations. This opens up a vast array of creative possibilities for AI-generated videos.

Skills Layer and Creative Output

Russo elaborated on the 'skills layer' that complements the HyperFrames framework. These skills are designed to teach agents about 'taste' and video-specific aspects, rather than the fundamental languages of HTML, CSS, and JavaScript. By reverse-engineering great videos and providing these as examples, agents can learn to produce high-quality output. He emphasized that achieving truly great results still requires craft, similar to traditional software engineering, involving breaking down problems and iterative refinement.

He showcased a demo of a single-shot prompt utilizing the 'website-to-video' skill, illustrating how an agent can take a website's URL, retrieve assets, and generate a video with appropriate design and branding. For power users, HyperFrames offers granular control through its studio, allowing for manual adjustments and keyframe editing, akin to professional motion design tools like After Effects.

HyperFrames at Scale and Future Directions

Russo highlighted the real-world adoption of HyperFrames, with over 1.3 million videos rendered by open-source users in the last 90 days, 267,000 creators trying the framework, and 15,000 videos rendered daily. The project has also garnered significant community interest with 32,000 GitHub stars.

Acknowledging that LLMs are still developing in creative work, Russo announced HeyGen's initiative to create a 'code-to-video benchmark' in collaboration with LLM labs and video agent creators. The goal is to collectively raise the standard of AI-generated videos.

In conclusion, Russo reiterated his core message: "HTML is all your agents need in order to make great videos and launch your product into the world." He invited attendees to explore the open-source repository and engage with the HyperFrames project.