How AI is Being Used to Optimize Your Wellbeing, With Music

The art of music permeates the globe as a universal expression, but as of late, advances in AI and IoT are elevating it from simply an art form to a whole new echelon rooted in science. What if listening to music was more than just a leisure pastime, and a song was more than just a choice of preference.

Rubato.life, a new LA-based startup, released their beta application SDK last week that enables wearable devices and music content providers to recommend music according to a listeners’ physiological and psychological state and ultimately enrich their well-being.

“We are changing the way music is currently consumed, from simply listening, to channeling a smart use of music in order to manage stress, improve sleep quality, and optimize your well-being,” explained Rubato CEO, Amit Sternberg, who founded the startup in 2019 with co-founder and CTO, Noam Guy. An educated musician, Sternberg, along with growing scientific studies, assert music’s categorical influence on the human brain and body. Considering a projected 526.8 million wearable device shipments in 2024, the biometric consumer data deluge coupled with advances in AI techniques have given biofeedback and predictive power in music new meaning.

Yet, the application of AI for music dates back around the same time the discipline was coined, mainly for the task of algorithmic composition and composing music based on previously fed songs. Over the last decade, countless projects have demonstrated the increasing acceptance of these techniques in practice. Examples include Eurovision's submission last year, Sony's Beatles-esque song, Pierre Barreau’s AIVA-based songs, or OpenAI’s new GPT-based Jukebox library. A handful of startups share the same ambition, like Amper Music's entire album and Landr's music creation tool, which combined have attracted a host of artists to adopt into their workflow. Despite these advances, meager progress has been made on the consumer’s experience front. Music recommendation systems of the top content providers, such as Spotify’s among many others, are based on usage history, audio data, demographic and social data, all devoid of indicators on the real-time state of the user.

“Naturally, everyone’s different and their reaction to music is unique,” explained Sternberg. “Some listener’s feel a dopamine rush in a song’s specific melody, while the resolve from an opera stimulates creativity for others.” According to Sternberg, these moments are no longer intangible, rather, they’re measurable, and predictable.

Rubato developed an AI engine to recommend music according to listeners' biometric data. Based on deep neural networks, their algorithms cluster different musical attributes (like tempo, key, modality, and structural patterns) with the user's heart-rate trends and variability vectors and even brain signals in the future, to quantify the effect of music on humans. Depending on the wearable biometric sensors, Rubato extracts biomarkers of the user’s state (like stress, level of relaxation, dopamine release, or heart and respiratory rate) and identifies clinical matches of songs, with which it outputs recommendations with the highest likelihood of matching the user’s desired or current biomarker state.

“Our current AI trained model, not only looks for correlations but also analyzes which attributes generate the highest effect on the pertinent biomarker,” explained Guy. “This method supports improved personalization and better prediction. We assess the model’s predictive accuracy by measuring the HR and HRV parameter of our smartwatch trial participants while listening to music pieces explicitly suggested by the previously trained model.”

The startup’s engaged in a pilot study with USC University, refining their algorithms and research. “We found some initial findings that indicate correlations between the patterns of stress and relief in music, to brain waves and heart rates variability, and we plan to make use of these findings in our next product versions,” shared Sternberg. 

Their companion app is currently available for beta users on the Play Store, working with Garmin smartwatches and Oura rings, as well as Spotify (free and pro). 

Rubato currently has the potential to accurately stimulate desired states in real-time. “Eventually, we believe we’ll be able to prescribe music to be able to manage your stress, anxiety and other ailments,” added Sternberg, giving rise to the possibility that we’re still scratching the surface of music and its integration in people’s lives. As chronic stress is linked to the six leading causes of death, and more than 75 percent of all physician office visits are for stress-related ailments and complaints, according to the American Psychological Association, the impact of Rubato is deeper than ostensibly making listeners more fervently bop their heads.

Materializing the words of Yuval Noah Harari, “In the not too distant future, [AI] could analyze the biometric data streaming from sensors on and inside your body… and calculate the emotional impact that a particular song, even a particular musical key, is likely to have on you,” 21 lessons for the 21^st century.