Google AI Edge: Tiny LMs Powering Robotics & Devices

Cormac Brick, Principal Software Engineer on the AI Edge team at Google, recently spoke at the AI Engineer World's Fair about the growing importance of "tiny" AI models for edge devices and robotics. He highlighted the shift from solely relying on cloud-based AI to enabling intelligence directly on a vast array of devices, from smartphones to IoT gadgets.

The Case for Edge AI

Brick emphasized the core benefits of edge AI, which include reduced latency for faster, more consistent user experiences, enhanced privacy as data remains on the device, reliable offline functionality, and significant cost savings by reducing cloud compute and token usage, especially at scale. These advantages are critical for a wide range of applications.

Navigating AI Deployment Challenges

Deploying AI on edge devices, however, presents several challenges. Brick identified DRAM cost as a primary constraint, noting that even basic hardware like Raspberry Pis have seen significant price increases. This necessitates a strong focus on model optimization techniques like quantization and the use of the smallest possible models for specific tasks. Furthermore, the wide variety of target devices and the tendency for LLM research to focus on larger models mean that the lower end of the LLM spectrum is less explored.

Small vs. Tiny Models

Brick differentiated between "small" models, typically ranging from 1 to 4 billion parameters, and "tiny" models, which can be as small as 50 million parameters. Small models, requiring around 4-8GB of RAM for robotics and IoT applications, are already integrated into high-end smartphones and are suitable for devices like laptops. However, they may still be out of reach for lower-tier devices. The playbook for small models often involves prompting techniques like zero-shot or LoRA adapters, proving effective for tasks like function calling.

Tiny models, on the other hand, are designed for even greater accessibility, requiring less than 2GB of RAM and capable of running on a much wider array of consumer devices. These models are ideal when AI is a secondary feature or when system health and responsiveness are paramount. The approach for tiny models often involves leveraging off-the-shelf models for fixed tasks or fine-tuning models for specific outcomes.

The Power of Fine-Tuning and Community Models

Brick showcased how fine-tuning can significantly improve model performance and reduce resource requirements. He highlighted the "Mobile Actions" model, which, after fine-tuning with synthetically generated data, can reliably convert arbitrary text input into function calls for common mobile tasks. He also pointed to the availability of various community models on platforms like Hugging Face that are optimized for specific tasks such as Automatic Speech Recognition (ASR), vision, and embeddings.

The presentation also featured examples of Google's AI Edge tools, including LiteRT-LM and MediaPipe, which simplify the deployment of AI models across edge devices. Brick demonstrated the AI Edge Gallery app, available on iOS and Android, as a practical way for developers to experiment with small models directly on their phones.

Key Takeaways for Developers

Brick concluded with key takeaways for developers focusing on consumer devices and entry-level robotics:

• Small LLMs are user-friendly and perform well on NPUs.
• Tiny models expand the reach of AI to a broader range of devices, often requiring fine-tuning.
• Visual intelligence can be effectively incorporated using tiny VLMs.
• Robust voice-to-function calling is achievable by investing in appropriate synthetic datasets for fine-tuning.

Ultimately, the trend towards smaller, more efficient AI models is democratizing the deployment of intelligent capabilities across a vast spectrum of devices.