How Artificial Intelligence Can Keep Classical Music in Business

David Serkin Ludwig
The Juilliard School

Many fear that AI spells the end of human music and musicians. David Serkin Ludwig explains why new technologies represent an extraordinary opportunity, particularly for classical musicians, to expand their audience and share the art they love.

Long before I was appointed dean and director of music at The Juilliard School I gave a course in 20th century music history, and for just one class, I would slip in as a listening example an excerpt from Igor Stravinsky’s early ballet The Firebird – except instead of being a recording of human musicians, it was of high-quality computer synthesized instruments. I would see if any of the students noticed that what they just heard was not an actual orchestra, but a sophisticated ensemble of digitally rendered performers making sounds that came from zeros and ones rather than 10,000 hours of practice.

Inevitably, most of the students did not realize it was a computer playing, and those few that knew something was off explained it only by noting that the recording did not sound “right.” This exercise pre-dated many of the newer applications of artificial intelligence in realizing musical scores, but already illustrated how computers can be programmed to perform and ultimately create some very human sounding music.

We have been surrounded by computer-realized music for decades − in supermarkets, movies, and in our vehicles. The cheesy synth sounds of the 60s, 70s, and 80s have given way to sophisticated and convincing instrumental replicants. My composition students are writing music on their laptops using software that would fool the vast majority of people into thinking that what they were listening to was recorded in a live sound studio by living musicians. This realistic playback is assisted by AI algorithms to recreate the acoustics of concert halls and the idiosyncrasies of human performers with astonishing verisimilitude.

But rather than replacing creators and performers, as many fear, AI will be a tool to help musicians write, play, and teach more effectively and engage audiences on a far greater scale. Some of these opportunities have been around for decades but remain largely unrealized, while others are just arriving now. All have implications not only for classical music but for markets where consumers are looking for authentic, high-quality deliverables.

But first, let me clarify the term ‘classical music,’ which is a terrible descriptor. The so-called ‘classical’ era in music is also the shortest, from around 1750 to 1820, and two of its most prominent composers were Mozart and Haydn. Today, when we use the term ‘classical music,’ we are referring to a huge body of work going back 1,200 years, from the first written Medieval chants to pieces written by a living composer yesterday. I prefer the term ‘concert music’ to suggest music best heard live and best realized in concert rather than in a recording studio. Many musical observers also include jazz as concert music.

When we look into trends in audience engagement with classical music we find much ambiguity. There are studies showing that classical music is primarily and always has been consumed by older generations, but others suggest that millennials and GenZs are increasingly tuning in. One discouraging report says classical music represents only 1 percent of overall music sales, whereas a hopeful one says 30 percent of Americans have some connection to the art.¹ Live concert attendance numbers are demonstrably down across the board, but this is true for many cultural events. This ongoing ambiguity has caused classical music institutions to struggle to assert their relevance and value to society for decades.

Threats, opportunities, and robot overlords

With the developments in AI-generated music in mind, the next shareholder-friendly step for commercial record labels would be to cut out the middleman and train computers to make Top 40 music by themselves. This step would greatly lower costs and recording time, but of course would put human creators out of work. While programmable imitations of legendary composers do not usually fool trained musicians, for most listeners a computer simulacrum of Bach sounds pretty much like Bach.

And in popular music, where songs are often created by committees of writers relying on familiar and predictable patterns to make hits, AI can effectively emulate the bands and solo pop artists we hear all around us.² Amongst a near limitless number of examples of this on YouTube is a song written seven years ago by AI in the style of the Beatles called “Daddy’s Car” (to me, the vocals say middle-tolate period Fab Four).³ When I have played this track in public lectures and demonstrations, people are astonished to realize that a computer wrote not only the music, but also the words, and performed every note, too. Since “Daddy’s Car,” there have been huge advances such as Google’s MusicLM, which allow a user to simply enter descriptors (text prompts) or even an image, and the AI will compose a new work based on these inputs.

The opportunities for using AI-generated music in marketing and advertisement are considerable, as are the questions raised by the practice.

The opportunities for using AI-generated music in marketing and advertisement are considerable, as are the questions raised by the practice. Compensation for their intellectual property is a major concern for musicians because royalties represent an important portion of their income. Who gets paid for AI-composed music that effectively replicates the sound of a band or composer? The developer? The program itself, somehow? These questions come up every time AI is used for a widespread application in social media. At the time of this writing the controversy is around an app that creates fantasy genre user portraits for Instagram and appears to draw its templates from living human artists who are, yet again, uncompensated.

What rights are left

Legend has it that the composer Victor Herbert founded the performing rights organization ASCAP (American Society of Composers, Authors, and Publishers) after hearing a pianist playing his music all night to entertain patrons of a hotel. Observing that the establishment was making money off his hard work, Herbert started ASCAP to protect his own intellectual property and that of his fellow composers. It is not hard to imagine what he would think of music composed by a computer in his inimitable style to advertise cars, candy, and cat toys, all without a cent going to him.

No one in the arts is immune to the pressing concern that every job is about to be taken away by Skynet,⁴ and this concern is both warranted and understandable. We have seen the impact of computer-realized scores in the Hollywood film business, where the number of studio-session musicians in LA has substantially diminished over the past thirty years.

Visuals of human musicians are being simulated as well. There is a cottage industry – already a decade old – of recreating pop stars from the past as projected images so that beloved performing artists can take the stage long after they are gone. You can see convincing holographic replicas of Tupac, Amy Winehouse, Elvis, and Liberace perform in a stadium with a professional band of human musicians backing them up.

Holographic performance is not limited to actual people, either. Hatsune Miku is an anime character who dances and sings in massively attended concerts (her voice is an AI-synthesized amalgam of human voices). It was estimated that the brand had generated a revenue of around ¥10 billion in 2012, just five years after the character was ‘born.’ Miku has since topped charts in Asia, the U.S., and Europe, and was most recently slated to appear at the 2020 Coachella Festival before it was canceled due to COVID (ironic, given that the artist herself is vulnerable only to computer viruses).

Pandora’s Bachs

When technology closes one door it usually opens another, and artists are using the Internet and AI to cultivate audiences and market their work with great ingenuity. Since the new millennium, the Internet has widened our engagement into global cohorts, and this shift has changed music industry marketing in ways you might not expect. While music consumers used to be fed what they heard on the radio or listen to new recordings suggested by friends, many people today stumble upon artists new to them through algorithmic recommendations.

YouTube and Instagram are the most important social media platform for classical musicians, and both allow practitioners to not only offer snippets of their performances, but to share their bona fides with the world in the form of personal pictures and their most recent accomplishments: “I am humbled to receive this honor….” Classical music has not yet translated to TikTok as successfully as more popular forms of music, but it is only a matter of time.

One way that classical musicians are engaging audiences online is by posting practice videos that compel social media’s AI algorithms to pay attention. The iconic violinist Hilary Hahn is a friend from my student days. Along with her extraordinary musical abilities, she has an enviable savoir faire about engaging her followers as both a performer and teacher.

Hilary invented the “One Hundred Days of Practice” initiative and hashtag to accompany the eponymous goal with daily practice videos. This birthed a movement in which musicians from all walks to set work goals and accomplish them in front of their online followers. There is an authenticity in “One Hundred Days of Practice” that would take significant effort to replicate with AI – and little market incentive to do so.

AI and VR can help convert followers into audience members.

Best (virtual) seat in the (digital) house

A strong social media presence does not necessarily translate into ticket sales, but AI and virtual reality (VR) can help convert followers into audience members. For years, audiences have been able to tune in to concerts via livestreams from schools and presenting organizations. This was once a revolutionary idea, and many feared its effect on ‘butts in seats.’ We have since learned that greater engagement of any kind seems to improve attendance for live events, and livestreamed concerts are no longer a rare novelty but ubiquitous content in a saturated market of low production videos.

There is an exciting future in creating virtual concert spaces which are far more engaging than watching a video on your iPad. Imagine your VR headset taking you to a legendary concert hall far away to hear your favorite musicians perform in a visually believable environment. You are immersed in the sound of a fully recreated space modeled on the acoustics of the hall itself, specifically responsive to the shape of your ears, and with the sound source adjusting to your every movement.

Add some virtual friends to sit with you and you have the beginnings of an unforgettable but remote concert experience, with no one for you to disturb when it is time for a bathroom break. The whole event will be tailored by AI according to your feedback, and if you do not hear or see well, the algorithms and hardware can address that, too.

Perhaps you do not want a seat in the audience for the performance. Perhaps you would like to see what the principal clarinetist is seeing from the orchestra as the conductor elicits that lyrical solo in Rachmaninoff’s second symphony. Or maybe, (spoiler alert), while watching Puccini’s La Bohème, you would like to drop in from the point of view of Rodolfo calling out to his beloved Mimi on her death bed.

AI will help bring you to the space, note your preferences, and provide you with a custom virtual concert experience, though presumably with advertisements for the opera company, suggested recordings, and whatever snacks you ordered on Amazon that week.

I recall a heavy metal concert in World of Warcraft, the popular massively multiplayer online role-playing game (MMORPG), a decade ago. You could send your avatar, be it a warrior, sorcerer, or shaman, to the town square to hear a concert in which the performers transformed into characters themselves. The video game company created a way to engage its 12 million subscribers with a live concert happening in its fantasy universe of castles and dragons. The virtual concert idea was there, but still existed in a cartoon world.

The technologies of the fully virtual concert hall, with spatial audio, a visually convincing environment, and movable perspective are more or less available now, but no one has yet put in the time and resources to combine everything into the total experience. If someone does build a realistic virtual space for classical music or jazz, it is more or less guaranteed that there is a market audience of eager listeners who will pay to be there.

Compose yourself

My composition students are writing music with the help of AI algorithms, whether they know it or not, by using notation programs and synthesized sound modules for playback. When I was a student twenty years ago, I would bring to my teacher handwritten music on staff paper to be heard in our inner ear or realized on the piano.

Now student composers often write directly into a computer with access to immediate playback with AI instruments through a notation program or digital audio workstation like Ableton or Logic. And you do not have to be a conservatory student to use AI in your composition. For some years the music notation program Sibelius has had an inline ideas panel that suggests variations on new composers’ melodies, just as Microsoft’s ‘Clippy’ took us to the thesaurus when we needed a new word.

There is a future where composers work with dedicated AI tools to create unique acoustic worlds for public and personal enjoyment. Just as a chef knows how to mix ingredients, a composer learns to mix sounds. And if you don’t want a custom sound environment made for you, you can do it yourself, just like cooking at home.

Imagine putting on some tunes after work, but instead of your favorite Pandora playlist of commercial recordings, you used a simple composition app to create your own music, tailored specifically for those difficult Mondays. Maybe your musical concoction will be in the style of your favorite composer or band, or perhaps you will prefer a more ‘spatial’ sound environment like a musical analogue to a Rothko painting.

There is a future where composers work with dedicated AI tools to create unique acoustic worlds for public and personal enjoyment.

Either way, AI-assisted tools can give the consumer agency in making music, and that kind of access to the creative process inspires users to take greater interest in professionally presented music as well. There are currently 74,800 people in the United States who describe themselves as composers of classical music, including film- and videogame-scores.⁵ AI composing tools could expand that number by a hundredfold.

I, Teacher

Many, if not most, musicians teach in part to supplement their income. Around the world, young and old alike have a strong interest in learning to play musical instruments. In the United Kingdom alone, over a million adults say they took up an instrument during the COVID crisis. Seventy-four percent of those say that music is important to their quality of life and nearly 40 percent say that this importance increased during lockdown.⁶ Many of those new and returning instrumentalists are studying classical music.

The skills required to play classical music professionally are varied and deep, and the ability to teach those skills is a valuable asset, worthy of cultivation. Musicians who make a living by teaching privately have a limited number of hours a week to devote to students, But what if an AI teaching partner could expand that capacity? An application could be programmed to observe a student’s progress and offer feedback, mapping body movements and embouchures.

Much as a nurse prepares patient information for a doctor for more efficient diagnosis, an AI assistant could convey the metrics of a practice session to a music instructor. A programmable teaching partner could also double as a practice buddy, keeping the student company during the tedious work of playing scale exercises, not just notifying the student when they are sharp or flat, but using the pre-programmed guidance of a human instructor to tell them how to fix it.

This virtual teaching partner could function far beyond the meters and gauges we use now to show when we are playing in tune. We could communicate directly with an AI anime character or computer-generated human who would help us to play better. And we could use virtual students to train teachers, seeing how our pedagogical methods work in order to instill good instruction habits.

Not just for technical instruction, AI can also act as a collaborative performing partner. Classical musicians have tried, mostly unsuccessfully, to play together over an Internet limited by bandwidth and equipment. Meanwhile conservatories scrambled to find performing opportunities for students trapped at home during the pandemic.

We live in a world where an aspiring musician can develop their performance chops with a robot duettist who lives in their phone.

Computer accompaniment technologies that have existed for decades have evolved to be more responsive and expressive companions. There are many examples of AI-accompaniment software. Among them is Juho Pohjonen’s MyPianist app that listens to a musician’s phrasing, volume, and tempo and responds musically by changing its own tempo or volume just as a human collaborator would. We live in a world where an aspiring musician can develop their performance chops with a robot duettist who lives in their phone.

Ahead of the beat

The innovations of living musicians are essential to performance, composition, and instruction, but can musical creativity itself be modeled? That we even have to ask suggests that the answer is yes. But the intangible combination of intuition and order that goes into human expressive genius will not be replaced any time soon. Maybe we can simulate Beethoven well enough to convince most of the people most of the time, but some will always seek authentic performances and have the acumen to recognize the real thing when they hear it. These sensitive listeners and the pleasure they feel in experiencing live concert music is the brass ring of 21st century careers for classical and jazz musicians.

Classical music has no choice but to embrace technology and make the most out of it. In an era of instant gratification, this vast body of work, which requires dedicated study to play professionally and at least some exposure to connect meaningfully with, could fade away as a literature, leaving only the well-known hits behind. Eventually we would be left with nothing but the opening movement of ‘Eine Kleine Nachtmusik.’⁷

Classical music is often attacked, perhaps most often by classical musicians, as being too elitist. But what happens when what is considered elitist in art is made accessible to everybody? The greatest hope for this music’s future is for its practitioners to learn to wield AI to engage audiences and create inclusive ways for a wide variety of listeners to enjoy their work. Where some fear the ultimate death of art itself in the growth of artificial intelligence, I see an extraordinary opportunity to share the art we are so passionate about with a global audience. And a desire to share what we love is, after all, only human.

Author Bio

Endnotes

1. https://medium.com/@AmericanPublicU/popular-classical-music-how-popular-isclassical-music-part-ii-4040456752db
2. https://www.theguardian.com/music/2012/jul/27/pop-music-sounds-same-surveyreveals and https://www.wpr.org/popmusic-getting-more-repetitive-computerscientist-says
3. https://www.youtube.com/watch?v=LSHZ_b05W7o
4. https://en.wikipedia.org/wiki/Skynet_(Terminator)
5. https://www.careerexplorer.com/careers/composer/job-market/
6. https://www.classical-music.uk/news/article/one-million-adults-took-up-aninstrument-during-lockdown-uk-musicsurvey-reveals
7. https://www.youtube.com/watch?v=jLcmwY4hjIw