Ace: Why Voice AI Needs Smarter Scaffolding, Not Bigger Models

Ornella Bahidika and Joel Allou, the creators of Ace, a live AI voice tutor, argue that the common instinct to use the largest, most powerful AI models for voice applications is a misconception. They presented their approach, which prioritizes speed and responsiveness by utilizing smaller, faster models and offloading complex reasoning to an external system they call 'scaffolding'. This method, they claim, is key to creating natural-feeling voice interactions where latency is critical.

The Latency Imperative in Voice AI

Bahidika emphasized that in voice-based interactions, silence is a significant problem. A pause of even a second can make a voice agent feel unresponsive or 'broken'. The critical metric for success in these applications is not necessarily the model's raw intelligence (IQ) but its speed, measured in milliseconds. They aim for the AI model to start speaking within approximately 950 milliseconds. A frontier model that takes a full second to process and respond, regardless of the quality of its answer, risks losing the user's engagement.

Offloading Complexity: The Power of Scaffolding

The core of their strategy involves taking the 'hard jobs' away from the AI model itself. These responsibilities include deciding the flow of the lesson, tracking what the student knows, and planning the next steps. Instead, Ace employs a system of external logic, including a state machine and an intelligent layer for mastery tracking. This system manages the lesson's progression and provides the core AI model with a summarized brief at each turn. The model's sole responsibility is then reduced to 'talking', generating speech based on the provided context.

Joel Allou elaborated on this, comparing it to using a powerful but slow model like Claude 4.7 for complex reasoning. While capable, the inherent latency in such models is detrimental for real-time voice applications. By extracting the thinking process into a state machine that coordinates lesson steps and derives student mastery, the system ensures that the large language model (LLM) only handles the final output generation. This allows for the use of smaller, more cost-effective, and faster models.

Demonstrating the Difference

The presentation included a comparison between a voice agent using a frontier model without this scaffolding and one that incorporates their approach. The former showed noticeable delays in response time, while the latter, using a smaller model like Haiku 4.5, achieved responses within the target 900-millisecond window, appearing almost instantaneous.

The Scaffolding Trade-off

Allou acknowledged that this approach is not without its costs. Smaller models, without the necessary structural guidance, can 'drift' over longer interactions and require strict rules to maintain coherence. The 'scaffolding', the state machine, reasoning process, and scenario management, is the price paid for this efficiency. However, he stressed that this is a one-time cost implemented in code, rather than an ongoing per-turn computational expense.

The Guiding Principle

The fundamental advice offered is to "Pick the fastest model your latency budget allows. Spend the rest on scaffolding." This principle applies broadly to voice applications, real-time systems, and any high-volume service where the LLM is just one component of a larger, more complex system. By building robust external logic, developers can optimize for speed and cost-effectiveness without sacrificing the quality of user experience.