Rachel Nabors: Local AI Models for Frontier Results

In a recent presentation, Rachel Nabors, known for her work with AI and UI standards, discussed the practical advantages of leveraging smaller, localized AI models over large, frontier models. Nabors, who has previously contributed to standards at Mozilla and the W3C, and worked with the React team, highlighted how companies can achieve significant cost savings and performance gains by opting for smaller, more specialized AI solutions, particularly when running models on device.

The Cost of One-Size-Fits-All Inference

Nabors began by addressing the inherent costs associated with using large frontier models, especially for tasks that don't require their full capabilities. She emphasized that every API call made to a large language model (LLM) incurs costs for both the user and the business. Furthermore, relying on cloud-based models introduces risks related to data exposure and potential outages, as demonstrated by a hypothetical scenario where a model fails to connect to the web.

She presented a compelling argument for considering smaller language models (SLMs) or task-specific models. These models, containing millions to a few billion parameters compared to the hundreds of billions or trillions in LLMs, are significantly more efficient in terms of computational resources, energy consumption, and memory footprint. This efficiency makes them ideal for on-device deployment, offering benefits like lower latency and enhanced privacy as data processing occurs locally.

The Benefits of Small and Local AI

Nabors outlined several key benefits of adopting smaller, local AI models:

• More Secure: Processing data on-device reduces the risk of data breaches associated with transmitting sensitive information to remote servers.
• Works Offline: Local models are not dependent on network connectivity, ensuring consistent performance even without internet access.
• No Fees: Eliminating API call costs can lead to substantial savings, especially for applications with high usage.
• More Efficient: Smaller models require less computational power and energy, leading to a smaller carbon footprint.
• Lower Latency: Processing data locally drastically reduces response times, improving the user experience.

She showcased a table comparing various smaller models, detailing their parameter counts and physical sizes, illustrating their suitability for deployment on devices with limited resources.

Right-Sizing AI in 4 Steps

Nabors proposed a four-step framework for effectively right-sizing AI models:

1. Prove it's possible: Start by testing the task with the largest, most capable model available to confirm feasibility.
2. Set success criteria: Define clear input-output pairs and performance metrics to evaluate models objectively.
3. Test from small to large: Experiment with smaller models, gradually increasing size until the criteria are met.
4. Select the smallest capable model: Choose the model that meets the defined success criteria with the lowest resource requirements.

She demonstrated this process by evaluating several models against a golden dataset, highlighting how prompt engineering and careful selection can lead to significant improvements.

Prompt Engineering for Better Results

The presentation also touched upon the importance of prompt engineering, showing how different prompt formulations can influence model performance. She showcased experiments comparing a baseline prompt with variations like numbered input, few-shot prompting, strict rules, and chain-of-thought prompting. The results indicated that few-shot prompting, which includes worked examples, significantly improved accuracy and reduced latency compared to the baseline.

Conclusion: Embrace Smaller, Localized AI

Nabors concluded by emphasizing that while frontier models have their place, smaller, localized AI models are often sufficient and more practical for many applications. By following a structured evaluation process and focusing on prompt engineering, developers can effectively leverage these efficient models to build powerful, cost-effective, and user-friendly AI experiences.