Learning Piano is Great for Humans – and Robots!

I’m constantly advocating piano lessons for all sorts of people, young and old, experienced and novice.

But I had a thought today: piano lessons could be helpful for not just humans, but also robots.

Teaching humanoid robots to play piano could give them the dexterity they need for washing our dishes, cleaning our houses, and helping us humans in all sorts of incredible ways.

Here are a few reasons why piano is not only a fantastic instrument for humans to learn, but how it could be an incredible tool for AI researchers:

Starting Simple

One of the great things about piano is that you can start simple. VERY simple. Here’s the typical progression in complexity:

1. You typically start with one finger, playing one note at a time, choosing from only a few keys on the keyboard.
2. Then, you place your hand in the “Five Finger Position” (Thumb on C, Index Finger on D, Middle Finger on E, Ring Finger on F and Pinky on G), and learn to play simple melodies and scales using your five fingers, keeping your overall hand position static.
3. Then you repeat steps 1 & 2 with your left hand.
4. Then, you learn play both hands together. It’s coming together!
5. You’ll probably start seeing some variation in the rhythm – seeing whole, half, quarter, and maybe eighth notes. You might some some dotted notes, maybe some slurred notes.
6. Soon, you’re starting to reach for more notes and learning how to move your hands around the keyboard. Sheet music will have instructions for which fingers play which notes.
7. Then you’ll work on more musical dynamics – faster or slower, louder or quieter, different emphasizes for certain notes, and more overall emotive playing.
8. To facilitate this, you’ll want to learn how to use the right-hand Damper Pedal. Don’t worry, it’s not too tricky!

And that’ll typically get you to a pretty decent level of playing piano. There’s always more to learn and this article is not going to be an exhaustive exploration of the entire field of piano pedagogy, but personally, I find this basic progression of complexity very encouraging.

Any piano student (or AI researcher) can take something as complex as a playing a piano and break it down into very manageable step-by-step phases that facilitate significant improvements in dexterity and skill.

Piano is unique in that it doesn’t require special breathing, bowing, strumming, plucking, or other more complex techniques. You don’t need to grip drumsticks, hold a bow, hold your tongue just the right way, or anything terribly complicated. Essentially, you just need fingers to press keys up and down, and a foot for pressing a pedal (or two) up and down.

Incredible Library of Training Data / Sheet Music

Sheet music is a visual set of very specific instructions for the player to execute. There are many millions of pieces of piano sheet music available, if the robot has an adequate visual processing system.

Sheet music can often be converted to MIDI and from MIDI back to sheet music. I often explain that MIDI is like digital sheet music, although it is also a LOT more than that. But at a bare minimum, it includes information about which notes to press, when to press them, for how long, how hard, etc.

There are many excellent commercially available apps and games for teaching students how to play piano. These include Simply Piano, Yousician, Piano Marvel, Playground Sessions, Flowkey, and Skoove. These systems all have extensive libraries of music, in graduated levels of difficulty and complexity.

Piano, as the original foundation for Western music theory, arguably has the most amount of sheet music available for it, across arguably the widest swathes of genres.

Precise Feedback

Right now, many robots tests involve picking up apples or other objects that don’t have lots of precise sensors built-in.

A good digital piano has 3 different sensors for each and every key. That sensor data can send several different types of MIDI velocity data back into the AI system, so it instantly gets very detailed feedback about the notes that it triggers. The instrument’s velocity sensors can be compared with the desired velocity (did it play as softly as it was trying to?), and then the system can adjust the amount of force it exerts next time.

Current commercial player piano systems like PianoDisc and QRS have solenoids with over 1,000 levels of sensitivity for acoustic piano playback, with available systems designed to accurately playback very refined playing.

All of the piano-teaching apps have sophisticated systems for giving the user feedback for their success or lack thereof. And because these systems can give feedback automatically, they can run 24/7, reducing the need for human feedback.

Speaking of human feedback – there are plenty of highly-qualified piano teachers who are well-versed in giving students feedback on proper use of the fingers, wrists, elbows, and even the entire arm! Human students greatly benefit from working with private music teachers; these teachers could be invaluable partners for helping engineers find more efficient ways to direct the robots.

Possible Downsides

Training a robot to play piano won’t be quick or easy and will require extensive resources. But I do think it’s one of the more efficient possible routes to increasing robot dexterity. And if you’re trying to develop dexterous humanoid robots, you’ve probably got a pretty sweet research lab humming away anyway.

Will a robot be able to transfer the dexterity learned from playing piano to other skills? My guess is that not 100%. I’m personally pretty decent at playing piano, but horrible at drawing. But I do think that playing piano could generally improve dexterity to the point that it would reduce task-specific fine-tuning.

Could pianists lose their jobs over this? Personally, I doubt it. Player piano systems have been around for over 100 years, starting with paper-based systems, all the way to the state-of-the-art player systems. Sophisticated systems like PianoDisc, QRS, & Yamaha’s Disklavier have been around for over 30 years, with tons of player pianos already in homes, hotels, and other venues.

It’s worth noting that while I’m sure there has been some impact on how many gigs are available for professional pianists, there’s a time and place for both robotic piano player systems AND real human musicians. (Which is how I expect many AI advancements to affect jobs, but that’s a different, larger topic…)

Of course, there are wider questions of “will robots murder us in our sleep?” and “Will robots take all of our jobs, leaving us to scurry around in some underground dystopian hellscape?” As someone who has been geeking out about AI for a while, I’m pretty sure neither of those scenarios will happen, but it’s important for AI researchers to continue to think about AI safety and broader societal impact at each and every step of their progress. Personally, I think that enabling robots that will interact with the public to do beautiful things like play piano would make our world a better, richer place.

The piano was first invented over 300 years ago, but has seen significant innovation ever since. Perhaps I’m being naive and/or massively underestimating the complexity of it all, but I think it would be amazing if this instrument of nearly limitless invention could be key for helping us unlock the possibilities that science fiction writers have been dreaming about for decades.

If you’re a robotics researcher or even just a regular ol’ human curious about any of the aforementioned technologies, feel free to drop me a line at [email protected]. We’d love to hear from you! Also, it’s worth noting that in this Age of ChatGPT, I wrote this article entirely from my experience of running a piano store for 13+ years, not with the aid of any Large Language Model).