AI Guitar Sings: Todd Fisher's Tech Journey

Todd Fisher, a musician and engineer, shared his journey in a presentation titled "While my guitar gently speaks" at the AI Engineer World's Fair. His project focused on making a guitar 'speak' and 'sing' through the integration of artificial intelligence, showcasing a blend of music and technology.

Inspiration from Live Performances

Fisher began by reminiscing about impactful live performances that inspired him, from a Slipknot concert with a levitating drum set to the elaborate stage effects in "Stranger Things: The First Shadow" on Broadway. These experiences, he noted, highlighted the potential of technology in creative live settings and sparked his own project ideas.

The Evolution of Guitar Sound

He traced the history of guitar sound modification, starting from the invention of pickups and amplifiers to the proliferation of effects pedals and software emulation. Fisher pointed to the talk box, famously used by Peter Frampton, as an early example of modulating guitar sound with the human voice. Looking forward, he posed the question of how AI could further evolve the guitar's capabilities.

From Halloween Lights to AI-Generated Speech

Fisher shared a personal anecdote about his Halloween tradition of playing guitar while handing out candy. Three years ago, he dressed as Eddie Munson from "Stranger Things" and developed a small app that projected the show's iconic alphabet lights onto his garage door, triggered by his guitar playing. This project led him to the more ambitious goal of making his guitar speak.

Technical Implementation: Making the Guitar Speak

To achieve this, Fisher explored various tools and frameworks. He identified JUCE as a key framework for audio software development and mentioned common plugin formats like VST3 and AAX. For text-to-speech (TTS), he utilized Piper (a local neural model) and built-in Apple TTS, combined with digital signal processing (DSP) techniques like a 24-band channel vocoder and YIN pitch detection.

His initial approach involved converting raw text into an audio clip and playing it back whenever a guitar note was played. He demonstrated this in Logic Pro, where his custom plugin would trigger the audio. He then refined this by segmenting the audio per word, addressing challenges in automatically slicing speech using methods like energy gap segmentation and sonority peak syllabification. While acknowledging the difficulty in perfectly segmenting speech, he noted that manual editing of segments was also an option.

Bringing the Guitar to Sing

The next evolution of the project was to make the guitar 'sing'. This involved pitch detection algorithms, specifically the YIN algorithm, to translate guitar notes into synthesized sounds. By creating a synthesized note with ADSR (Attack, Decay, Sustain, Release) parameters and feeding it through a vocoder along with the voice clip, Fisher aimed to create a singing guitar effect, akin to a talk box but with synthesized tones.

Conversational AI with Guitar Interaction

Fisher also demonstrated how to integrate conversational AI, using speech-to-text (Whisper) and a local Large Language Model (LLM) to enable the guitar to respond to spoken questions. He showcased a demo where the guitar, powered by the LLM, provided an existential response to the question, "What is reality?"

Future Directions and Inspiration

Looking ahead, Fisher expressed his interest in further developing the AI's ability to truly 'sing' by utilizing vocal samples from projects like VocalSet. He also touched upon the computational intensity of some processes, like the WORLD pitch-shifting algorithm, which requires pre-baking. Fisher concluded by encouraging the audience to pursue their own passion projects, emphasizing that AI tools now make it more accessible than ever to bring innovative ideas to life.