# Ace: Why Voice AI Needs Smarter Scaffolding, Not Bigger Models _Ace creators Ornella Bahidika and Joel Allou explain why smaller AI models coupled with smart 'scaffolding' are superior for responsive voice applications._ **Published:** 2026-07-20 **Source:** https://www.startuphub.ai/ai-news/artificial-intelligence/2026/ace-why-voice-ai-needs-smarter-scaffolding-not-bigger-models --- Ornella Bahidika and Joel Allou, the creators of Ace, a live AI voice tutor, argue that the common instinct to use the largest, most powerful AI models for voice applications is a misconception. They presented their approach, which prioritizes speed and responsiveness by utilizing smaller, faster models and offloading complex reasoning to an external system they call 'scaffolding'. This method, they claim, is key to creating natural-feeling voice interactions where latency is critical. Voice AI LatencyDriver even a second's pause makes voice agents feel unresponsive or brokenFrom the article 7 mentionsThis method, they claim, is key to creating natural-feeling voice interactions where latency is critical.exacerbated byBig Models SlowDriverfrontier models take a full second to process and respond, regardless of IQFrom the articleJoel Allou elaborated on this, comparing it to using a powerful but slow model like Claude 4.7 for complex reasoning.avoided byAce's ScaffoldingCoreoffloading complex reasoning to an external system for faster responsesFrom the article 6 mentionsOrnella Bahidika and Joel Allou, the creators of Ace, a live AI voice tutor, argue that the common instinct to use the largest, most powerful AI models for voice applications is a misconception.Smaller AI ModelsCoreutilizing faster, more responsive models for core voice interactionFrom the article 9+ mentionsThey presented their approach, which prioritizes speed and responsiveness by utilizing smaller, faster models and offloading complex reasoning to an external system they call 'scaffolding'.Scaffolding Trade-offContextbalancing complexity offload with potential for increased system overheadFrom the article 4 mentionsThe presentation included a comparison between a voice agent using a frontier model without this scaffolding and one that incorporates their approach.enablesFaster Response TimeEffectaiming for AI model to start speaking within approximately 950 millisecondsFrom the articleThe former showed noticeable delays in response time, while the latter, using a smaller model like Haiku 4.5, achieved responses within the target 900-millisecond window, appearing almost instantaneous.results inNatural Voice InteractionOutcomeFrom the articleThis method, they claim, is key to creating natural-feeling voice interactions where latency is critical. ## The Latency Imperative in Voice AI Bahidika emphasized that in voice-based interactions, silence is a significant problem. A pause of even a second can make a voice agent feel unresponsive or 'broken'. The critical metric for success in these applications is not necessarily the model's raw intelligence (IQ) but its speed, measured in milliseconds. They aim for the AI model to start speaking within approximately 950 milliseconds. A frontier model that takes a full second to process and respond, regardless of the quality of its answer, risks losing the user's engagement. ## Offloading Complexity: The Power of Scaffolding The core of their strategy involves taking the 'hard jobs' away from the AI model itself. These responsibilities include deciding the flow of the lesson, tracking what the student knows, and planning the next steps. Instead, Ace employs a system of external logic, including a state machine and an intelligent layer for mastery tracking. This system manages the lesson's progression and provides the core AI model with a summarized brief at each turn. The model's sole responsibility is then reduced to 'talking', generating speech based on the provided context. Joel Allou elaborated on this, comparing it to using a powerful but slow model like Claude 4.7 for complex reasoning. While capable, the inherent latency in such models is detrimental for real-time voice applications. By extracting the thinking process into a state machine that coordinates lesson steps and derives student mastery, the system ensures that the large language model (LLM) only handles the final output generation. This allows for the use of smaller, more cost-effective, and faster models. ## Demonstrating the Difference The presentation included a comparison between a voice agent using a frontier model without this scaffolding and one that incorporates their approach. The former showed noticeable delays in response time, while the latter, using a smaller model like Haiku 4.5, achieved responses within the target 900-millisecond window, appearing almost instantaneous. ## The Scaffolding Trade-off Allou acknowledged that this approach is not without its costs. Smaller models, without the necessary structural guidance, can 'drift' over longer interactions and require strict rules to maintain coherence. The 'scaffolding', the state machine, reasoning process, and scenario management, is the price paid for this efficiency. However, he stressed that this is a one-time cost implemented in code, rather than an ongoing per-turn computational expense. ## The Guiding Principle The fundamental advice offered is to **"Pick the fastest model your latency budget allows. Spend the rest on scaffolding."** This principle applies broadly to voice applications, real-time systems, and any high-volume service where the LLM is just one component of a larger, more complex system. By building robust external logic, developers can optimize for speed and cost-effectiveness without sacrificing the quality of user experience. --- Original analysis from [startuphub.ai](https://www.startuphub.ai), the #1 AI startup directory.