AI Research

Google's Cormac Brick on Tiny LLMs for On-Device Agents

Google's Cormac Brick discusses the fine-tuning of Tiny LLMs for on-device agents, highlighting the benefits of LiteRT-LM and Gemma 4 for edge AI applications.

StartupHub.ai Staff

By StartupHub.ai Staff, reviewed by Daniel Singer, Editor

3 min read

Presentation slide showing the title 'TLMs: Tiny LLMs and Agents on Edge Devices' with a network diagram.
Image credit: AI Engineer EuropeCredit: AI Engineer
Contents(5)

Cormac Brick, a Principal Engineer at Google AI Edge, recently presented insights into the advancements and practical applications of fine-tuning Tiny LLMs (TLMs) for on-device agents. The presentation, titled "From 46% to 90%: Fine-Tuning Tiny LLMs for On-Device Agents," detailed how these smaller, more efficient language models are being integrated into devices to provide powerful, responsive AI experiences. Brick highlighted the growing importance of running AI directly on devices, citing benefits such as reduced latency, enhanced privacy, offline usability, and cost savings compared to cloud-based solutions.

Google's Cormac Brick on Tiny LLMs for On-Device Agents - AI Engineer
Google's Cormac Brick on Tiny LLMs for On-Device Agents, AI Engineer

The Google AI Edge Stack and TLMs

Brick outlined the Google AI Edge stack, which includes components like MediaPipe and LiteRT-LM, a cross-platform runtime designed to run powerful LLMs on Android, Chrome, and iOS. He explained that TLMs, defined as models with fewer than a billion parameters, are small enough to be integrated directly into applications, offering greater customization and reach. The presentation also touched upon the wider applicability of these models beyond Android, extending to iOS, macOS, Windows, web, and IoT devices.

Benefits of On-Device AI

The core advantages of running AI on the edge were emphasized, including improved user experience through lower latency and the elimination of network dependency. Privacy is another significant benefit, as sensitive data can be processed directly on the device without being sent to the cloud. Furthermore, the ability to function offline ensures continuous availability of AI features, regardless of network connectivity. These factors contribute to significant savings by reducing the reliance on data center resources.

Gemma 4 and Agent Skills

A significant portion of the presentation focused on Gemma 4, a recent LLM from Google, and its application in creating on-device agents. Brick demonstrated how Gemma 4, when integrated with AI Core and the LiteRT-LM runtime, can power sophisticated agent skills. He showcased examples of the Google AI Edge Gallery app, which features various on-device AI functionalities powered by Gemma 4, such as text summarization, interactive chat, and image analysis. The presentation also highlighted the potential for creating custom agent skills, providing instructions and examples on GitHub for developers to build their own unique functionalities.

System vs. In-App Generative AI

Brick differentiated between system-level and in-app generative AI. System GenAI, which is pre-loaded onto devices, offers highly optimized and powerful features, with models typically ranging from 2 to 4 billion parameters. In contrast, in-app GenAI, utilizing smaller models (100 million to 1 billion parameters), offers greater customization for specific tasks and is integrated directly within individual applications. He noted that while system-level AI provides broad capabilities, in-app AI allows for tailored experiences.

TLMs in Production and Future Directions

The presentation concluded with a look at TLMs in production, emphasizing the efficiency and performance gains achieved through fine-tuning and optimization. Brick cited the use of LiteRT-LM to deploy models like Gemma 4 across a wide range of hardware, from mobile devices to embedded systems. He also pointed to the availability of pre-built TLMs on Hugging Face, encouraging developers to explore and experiment with these models. The ongoing development of LLMs for edge devices, including new API releases and expanded platform support, signals a significant shift towards more intelligent and capable on-device AI experiences.

Get the free daily brief every morning

Every morning the day's AI news, new AI startups and new funding rounds. Free and takes a few minutes to read, unsubscribe at any time.

Join 64,000 other subscribers.

TopicsCormac BrickGoogleGoogle DeepMindLiteRT-LMGemma 4LLMTiny LLMOn-Device AI

StartupHub.ai Staff

Written by

StartupHub.ai Staff

Editorial team

The staff writers of StartupHub.ai, ranging from investment analysts to avid AI tool users, early adopters and critical enthusiasts. Backgrounds span engineering, business and the arts. We hold every piece to rigorous standards of research and review.

Daniel Singer

Written by

Daniel Singer

Editor, reviewed this article

Daniel Singer is an AI expert who has been writing about, covering and building AI technology since 2018.

Test your Agent Readiness now

Paste your website. How well ChatGPT, Claude and the other agents can read and use a site, in about a minute.

Watch

The videos behind the news, one flip at a time

The feed mixes the videos from our articles with startup demos and you flip through them like TV channels for free with no signup.