Harmony Robot Hits the Right Notes at CHIA 2024 Conference

A Groundbreaking Achievement in Human-Robot Music Collaboration

A musical robot capable of playing the piano alongside a human has taken the spotlight by winning the Best Demo Award at the Center for Human-Inspired Artificial Intelligence (CHIA) Conference 2024. This innovative achievement showcases the fusion of robotics and music, pushing the boundaries of human-robot collaboration.

Huijiang Wang, a Ph.D. student in Engineering at Cambridge University, developed the Harmony Robot at the Bio-Inspired Robotics Laboratory (BIRL). This unique robot features a custom anthropomorphic hand mounted on a robotic arm that can harmonically accompany human piano playing in real time.

The Harmony Robot’s creators, including Wang, Dr. Xiaoping Zhang, and Professor Fumiya Iida, contributed to a groundbreaking research paper published in the journal IEEE Transactions on Robotics.

Titled “Human-robot cooperative piano playing with learning-based real-time music accompaniment,” the paper outlines how the robot utilizes machine learning algorithms to predict suitable chord progressions that correspond with a human performer’s melody.

The robot showed an impressive 93% accuracy rate in its experiments and had been trained on chords inspired by popular music.

To ensure seamless collaboration, the robot features a behavior-adaptive controller that allows it to synchronize its timing with the human player, creating harmonious duets. This innovative design enables both the human musician and the robot to perform together fluidly.

The robotic apparatus uses vertical movement to strike the piano keys and horizontal actions for switching chords, thereby enhancing the expressive potential of the robot as it interacts musically.

“We have developed a musical robot that can engage in real-time piano performances alongside a human,” said Wang. “The robot learns from the melody played by its human counterpart and uses machine learning to predict the subsequent chords, maintaining harmonic integrity throughout the piece.”

He also emphasized the importance of a special control system that ensures the robot matches the timing of the human player, thus achieving synchronized performances. “Additionally, we have implemented a system that evaluates the quality of the collaboration between the robot and the human,” Wang added.

Looking ahead, the team aims to diversify the robot’s musical repertoire by integrating more chord types and improving the dexterity and adaptability of its hand, making it capable of playing a wider range of chords.

Professor Iida noted, “Our objective is to stretch the capabilities of robotics into emotionally responsive applications, particularly in artistic settings, where the robot must interpret and react to the subtleties of human emotions during performances.”

“Enhancing the robot’s improvisation ability while managing computational demands is essential for future developments in this area,” he remarked.

“The progress made with the Harmony Robot marks a significant step towards incorporating soft robotics in creative and interactive roles,” Wang concluded. “By employing advanced machine learning techniques, we aspire to refine human-robot cooperation to gracefully respond to real-world dynamic scenarios.”

Provided by University of Cambridge