Gemini's Audio Stack: From Transcription to Music Generation

Thor Schaeff, a Developer Relations Engineer at Google DeepMind, recently provided an in-depth look at Gemini's audio stack, showcasing the evolving capabilities of AI in handling and generating sound. The presentation, titled "From Transcription to Live Music: Gemini's Audio Stack," offered a comprehensive overview of how Google DeepMind is pushing the boundaries in AI audio processing.

Understanding Gemini's Audio Capabilities

Schaeff began by highlighting Gemini's core strengths in audio processing, emphasizing its ability to not only transcribe speech with high accuracy but also to understand nuances like emotion, language variations, and even multiple speakers in a conversation. The Gemini API is designed to process a broad spectrum of audio inputs, aiming for seamless handling of diverse languages, dialects, and accents. This sophisticated understanding extends to identifying and labeling emotions within speech, adding a layer of depth to the AI's comprehension.

Multimodal Realtime Interactions

A significant focus was placed on Gemini's multimodal capabilities, particularly its real-time interaction features. Schaeff demonstrated how the Gemini API can process text, audio, and video inputs simultaneously, enabling more dynamic and interactive AI agents. This is facilitated through a WebSocket connection, allowing for low-latency communication between applications and the AI model. The presentation touched upon the AI Studio, a platform where developers can experiment with these models, including selecting different voices and adjusting parameters like media resolution and thinking level, to create custom AI experiences.

AI-Generated Music with Lyria 3

The presentation also introduced Lyria 3, Google's AI model for music generation. This powerful tool can create music based on textual descriptions, allowing users to specify genre, mood, and instrumentation. Lyria 3 comes in two variants: Lyria 3 Clip for short audio clips and loops, and Lyria 3 Pro for longer, more complex musical compositions with verses and choruses. This advancement signifies a significant step towards AI's creative potential in the music industry, enabling rapid prototyping and novel musical exploration.

Benchmarking Gemini's Audio Performance

Schaeff presented benchmark data illustrating Gemini's performance in audio tasks. In the "ComplexFuncBench audio" benchmark, Gemini 3.5 Flash Live achieved a leading 90.8% accuracy in function calling, surpassing other models like Gemini 2.5 Flash Native Audio and Gemini 2.0 Flash Native Audio. Similarly, in the "Big Bench Audio" benchmark for speech reasoning, Gemini 3.5 Flash Live also demonstrated superior performance with 95.9% accuracy. These benchmarks highlight the effectiveness and advancement of Google's audio AI stack.

The Future of AI Audio

The session concluded with a look at the future potential of AI in audio, emphasizing the ongoing development and integration of these technologies. Schaeff pointed to resources like the Gemini API examples on GitHub and the AI Studio's live playground as avenues for developers to explore and build upon these capabilities. The demonstration of the "Live Jukebox" further illustrated the practical application of these models, showcasing real-time music generation based on user prompts.