Future Opera for Robots and People Too

Tod Machover

MIT Media Lab

We all love music deeply; opera adds every other conceivable form to augment and unify the senses around music. Music and opera are capable of entertaining, stimulating, moving and transforming us as few other activities are. In fact, there is increasing research on the how and why of music's power, some of it—like the much-hyped 'Mozart Effect'—suggesting that merely listening to music on your iPod while driving, reading, sleeping or perhaps even playing music to your baby in the womb is enough to let music work its full magic.

Unfortunately, that isn't quite true. Music exerts its power when we are actively engaged, not when we listen subliminally. For this reason, I have been working with my group at the MIT Media Lab to create musical tools—often with specially designed technologies—that enable everyone to participate directly in music-making regardless of background.

This field has undergone a revolution in the past several years through the huge public success of the Guitar Hero and Rock Band videogames. Alex Rigopulos and Eran Egozy, students of mine from the MIT Media Lab, developed them based on ideas that we were working on in the early 1990s. The good news about Guitar Hero and Rock Band is that they clearly demonstrate the public's willingness to dive in and immerse themselves in music-making, given the right environment. The bad news is that neither platform is truly musical, nor do they encourage learning, expression or creativity.

What would happen if we could combine the excitement and 'stickiness' (bordering on addiction) of Guitar Hero with a more sustaining, personal and open-ended musical experience? How could we embed such a new type of activity in a more integrated musical ecology, where the current exaggerated distinctions between celebrities and amateurs would be diminished and the level of musical sophistication, excellence and, hence, enjoyment would be raised for all?

The research projects that I direct at the MIT Media Lab and many of the musical projects I undertake are attempting to nudge the field in these directions. We started by developing Hyperinstruments for some of the world's greatest performers, including Yo-Yo Ma and Prince, as well as orchestras, chamber music ensembles and rock bands. All kinds of sensors are built into the Hyperinstrument so it knows how it is being played. By changing the interpretation and feeling during performance, a cello—for instance—can be morphed into a voice or a whole orchestra or something that nobody has heard before.

As these virtuosic Hyperinstruments developed, we started imagining that we could use similar technologies and interpretation strategies to produce instruments and interfaces for music-lovers who weren't highly trained virtuosi. We have designed a series of such instruments and interfaces. One of the largest collections was the Brain Opera that was launched at the first Lincoln Center Festival in New York in 1996, toured the world and is now permanently installed at the Haus der Musik in Vienna.

We created a rather large orchestra of especially designed fantasy instruments (including Rhythm Trees, Harmonic Driving, Gesture Walls and Melody Easels) so that anybody could play them using natural skill. You can play a video game, drive through a piece of music, use gestures to control huge masses of sound, touch a special surface to make melodies and use your voice to make a whole aura. We designed the Brain Opera for adult concert-goers, but found that everywhere we went it was most easily understood and most creatively manipulated by the youngest (under 8) and oldest (over 70) visitors. This was perhaps due to lack of inhibition and desire for social play and creativity among those ages. We therefore decided to concentrate on activities that might engage those groups more directly.

This led to our Toy Symphony project (2002–2005), which attempts to reconsider how to introduce children to music in the most immersive, creative and enjoyable way possible. The goal is to have kids fall in love with making music first and then demand to learn more because of that love. We designed a set of new Music Toys, including the soft, squeezable Music Shapers that manipulate intensity and tone colour; Beatbugs, which capture rhythms that can be manipulated and shared with friends; and a software-composing environment called Hyperscore that lets anyone compose original music by shaping lines and colours. Another goal of Toy Symphony was to develop a project model—learning musical skills, creating new music and then rehearsing and performing a concert—that would bring children and orchestras together.

Our results with Toy Symphony were encouraging enough to make us decide to bring this model to other populations where there might be clear impediments to personal expression and creativity, where music, made accessible through new musical tools, might be an ideal medium. In 2004, we began concentrating on providing musical experiences and tools—based on ongoing research by colleagues at the MIT Media Lab and increasingly around the world—to help improve health, diagnose illness and provide a medium of expression and communication that would otherwise be lacking. This new area of Music, Mind and Health has led to research in using music for early detection of Alzheimer's disease, for social and emotional adaptation for autistics, for aiding physical and mental rehabilitation and for a growing number of other areas. With my student Adam Boulanger, I started this work at Tewksbury Hospital near Boston, where we were invited to work with a group of long-term residents with a wide range of severe physical and mental disabilities. We organized composing workshops with Hyperscore that resulted in a series of public concerts featuring music by patients. This process has become so successful that it has been replicated at many sites, resulting in marked and unexpected improvement in a wide range of conditions, and inspiring a number of patients to themselves mentor others in the uses of new tools and environments for creative musical expression.

A performance system designed for and with Dan Ellsey is an example of a new category of interfaces that we call Personal Instruments. Even an instrument as sophisticated as the Hypercello we designed for Yo-Yo Ma is a generalized instrument. In other words, anyone familiar with cello technique can play it, drawing more from it according to one's mastery and understanding. But Dan's instrument was designed for him and him only: it takes account of his particular style and detail of moving and the way that translates into musical expression, and compensates for his particular physical limitations. Future instruments and interfaces can and must be adaptable and tunable to each of our skills and limitations. For us, Dan's performance system represents the first step in this direction.

How did I come to undertake such unusual work? Through a desire to compose music, the activity that I love to do most. It is what best combines my various skills and interests—imagination, reflection, organization and the desire to communicate my thoughts and emotions to anyone who will listen. I also love solitude: I do my creative work in an 18th-century barn on our farm near Boston, where I can pursue my ideas without the need to explain or translate until all is ripe and ready. So it may seem like a paradox that another large chunk of my life is spent in one of the world's most futuristic, collaborative and intensive centers of technological invention—the MIT Media Lab. But the attractions and complexities of merging these worlds are central to how and why I work, and grow from seeds planted when I was very young. My mother is a Juilliard-trained pianist and a remarkable pedagogue and my father is one of the pioneers of computer graphics, but it actually took me a while to start combining these fields. I grew up as a cellist, first playing solo Bach, then chamber music (I never particularly enjoyed playing in orchestras), and then, by high school, original composed or improvised music using a wired and transformed rock cello that I created by placing large headphones around the cello for amplification, then sending the sound through tape recorder loops and analog transformation processes.

The appearance of the Beatles' Sgt. Pepper's Lonely Hearts Club Band had changed my life: it suggested a music that ideally balanced complexity and directness. There was a downside, though: as a product of the recording studio, most of the Beatles' music after 1967 couldn't actually be played live. That's when I started imagining a performance mode that would combine the physicality and intimacy of solo cello and the unhinged creativity of the recording studio. I was driven by the urge to bring this strange, enticing and intricate music filling my head out through my arms and fingers and into the world.

This desire compelled me not only to compose the music I was imagining, but also to invent new instruments and new modes of playing them, something that I never thought as a kid that I'd end up doing. So along with my colleagues and students at the MIT Media Lab I designed the projects outlined above.

Inventions like these have been part of a trend that has yielded amazing developments over the past 10 years. Technology has democratized music in ways that are surprising even to me, revolutionizing access to any music anytime with iPod and iTunes, opening interactive music-making to amateurs with Guitar Hero and Rock Band, providing digital production and recording facilities on any laptop that surpass what the Beatles used at Abbey Road, and redefining the performance ensemble with initiatives like the Stanford University iPhone Orchestra and YouTube Symphony.

In fact, near the end of 2010 one wonders whether there is any more music technology to invent, or whether our musical imaginations and artistic cultures simply need to catch up. The answer is both, and then some.

For the first time in my career, I feel as if there are enough tools on my laptop, enough brilliant and inventive playing chops amongst the younger generation of performers, enough ooomph in the iPhone, and increasing openness and entrepreneurship in musical organizations both large and small to stimulate my imagination and allow for the production and dissemination of my somewhat unusual creations.

But even though these evolving music technologies are already very powerful and increasingly ubiquitous, we can also see their current limitations and potential risks. Guitar Hero is rhythmically exciting but not yet expressive or creative enough—a "sticky" but not "open-ended" experience that does not obviously lead to better musicality, listening or ensemble awareness. The iPhone is a remarkable little chameleon but lacks the touch and sensitivity of even the simplest traditional instrument, better for selecting and switching than for subtly shaping. Amplified sound is loudly present and "surrounds" us ever more, but still emphasizes the boom box aspect rather than the "still small voice." And there isn't yet a performance measurement system that could come close to interpreting the exuberance, range and immediacy of someone like conductor Gustavo Dudamel or truly enhancing the experience of an "unplugged" symphony orchestra.

As a composer, I find that each new piece I undertake suggests exciting but daunting technological challenges; my imagination just seems to be wired that way. My current project, the opera Death and the Powers, is one example.

I had been invited to imagine a new (and unusual) opera by the Opera of Monte Carlo, and two fundamental impressions came to mind early on. The first came from thoughts about mortality and how difficult it is to sum up one's life in a way that can be shared and transmitted to loved ones through generations, and how music has a particularly powerful capacity for collecting and concentrating multiple experiences, then burning them indelibly into our memories. And I started imagining that this web of musical memories—the embodiment of an entire life—needed to transcend traditional notes and instruments, jump off the stage and physically envelope the listener, both aurally and visually. This turned into a mental impression of floating, undulating, palpable 3-D sounds represented visually through slowly moving, morphing objects filling a stage—like Fantasia become physical (but with my music and without dancing elephants). I felt the need to go beyond the flatness and harshness of usual multimedia tools to create something that was at the same time transcendent and magical but also completely human and down-to-earth.

I then sought out collaborators—the poet Robert Pinsky and the playwright Randy Weiner—to turn these initial impressions into an opera, a form that has long attracted me for its use of word and image to ground music's abstract qualities in concrete human experience. Together we crafted a story about a man who longs to leave the world in order to pass to a higher level of existence, but wants everything about himself—his memories, his ability to influence others, his contact with those he loves, his legacy—to remain behind.

This story evolved into a full opera libretto in which the main character, named Simon Powers, switches on The System at the end of Scene 1: he becomes embodied more and more in his surroundings, forcing those left behind to decide how to communicate with him or it, whether to follow, and what part of his legacy to retain or reject. The stage itself becomes the main character in the opera, taking over from—and extending—the physical presence of the singer. Realizing this vision was a daunting challenge, but happily, with the collaboration of the director Diane Paulus, the designer Alex McDowell, the choreographer Karole Armitage and my group at the MIT Media Lab, we designed sighing walls, morphing furniture, gliding robots and even a resonating chandelier to create The System on stage—and to make it "sing."

In helping to tell this story and to sonify the score, all aspects of this physical set translate and amplify Simon Powers' human presence, using our new technique of Disembodied Performance, challenging the current limits of our ability to measure and interpret all the subtleties of a great performance. The techniques we developed yield surprising results, turning elegantly refined gestures, barely perceptible touch, and the gentlest breath into sounds, shapes and movements that convey personality and feeling without looking or sounding exactly like a human being, although we end up feeling extremely close to Simon. This whole infrastructure is a new kind of instrument, and we indeed learned how to "play" it in time for the world premiere performances of Death and the Powers in Monaco in September 2010. All of the unusual elements of this massive project—music, story, words, visuals, movement, robotics and more—came together and appeared to be more than the sum of their parts. An uneasy relationship was established—as desired—that invited audiences to question the boundary between humans and machines, and often established an emotional connection with the chorus of OperaBots.

These OperaBots frame the opera by accepting to perform this inherited story, left to them in a future time when there are no more humans on Earth. Once the opera starts, the OperaBots are almost always on stage, reacting to live performers, commenting on the action, being the playthings or "pets" of Nicholas, Simon Powers' assistant who built them and being sort of intermediaries between the humans and The System. They are not exactly individual characters, but they do have individualized choreographies and behaviors, gliding and twisting about, flashing and modulating light and—indeed—singing from time to time. The OperaBots have "character"—they are fun, interesting, engaged, energetic—but they do not understand the kinds of questions that give meaning and texture to human lives: relationships, time, touch, sacrifice. They care about the actions of the human characters, but they do not have the kind of motivations that underlie Simon Powers' final confrontation with his daughter Miranda, where he pleads with her to enter The System with him, and she must decide what she would gain or lose by doing so.

Works of art do not have one single point or message. But two underlying inspirations behind this project were, first, how to allow technology to enhance human presence and communication on stage, as opposed to the huge distancing that happens more and more in mega-spectacle rock concerts where ugly, loud sound is pushed from the stage and performers look like ants against giant TV screens.

And I wanted to explore the possibility and poignancy of what is easy and what is hard to communicate between any two people—and especially across generations. I also wanted to create a journey where these questions and feelings would come alive through memorable melodies, unusual sonic textures, and pulsating rhythms…with the help of a few robots.

I believe that Death and the Powers is innovative in quite a few ways. An underlying goal of this opera has been to create a form of live performance which goes well beyond the typical multimedia practice of contemporary performance as seen constantly in—for example—arena rock shows. The norm these days is to create painfully overpowering sound to fill huge arenas, and gigantic video displays that dwarf human performers (even in U2's most recent tours, for example) rather than enhancing human presence. That is why we have instead created a stage filled with animated physical objects, from musical robots to animatronic walls to the Musical Chandelier. All translate sophisticated technology into physical form (including one of the most sophisticated sound systems ever used for live performance) with the goal of connecting audiences to the human beings on stage.

In addition, we have pushed the boundaries of robotic performance, not just with our autonomous OperaBots, moving walls of The System, and Chandelier, but with furniture robots that imbue inanimate objects with believable human characteristics. The opera might definitely induce people to think much more broadly about the potential of robots, and the productive relationship between robots and humans.

In order to control the unprecedented complexity of robots, visuals and sounds, we have developed special software to allow intermedia, integrated design of all aspects of the show, sophisticated enough for the most demanding AI programmer to fine-tune, but intuitive enough for a non-tech director or choreographer to feel comfortable with. This same software allows for real-time performance control of every aspect of the show, with perfect event synchronization and subtle inter-reaction between elements, so that everything on stage becomes part of a single, integrated "system."

And we have developed new sensing and interpretation technologies so that a combination of the singer's conscious (voice and hand gestures) and unconscious (breathing, heartbeat, muscle tension, etc.) behavior can be translated to control all of the interconnected stage elements, so that the stage and set itself truly feel as if they are alive, creating an uncannily believable representation of the human being who is no longer present. We believe that this technology could have significant impact on the future of telepresence and expressive communication and collaboration over distance.

The goal of any live performance, of course, is to lead the audience to concentrate on the experience itself, on the ideas and feelings, i.e. to make the "making" of the performance—and in this case all the crazy technology—look simple and inevitable. This is one of the most complex stage shows ever mounted, with numerous individual elements that must work with precision, delicacy, force and beauty, and—even more—function truly as a "system" of interacting machines that must work together in the most unbelievable ways. It is a testament to the opera's production team that for those who don't know, it does look easy. For those of us behind the scenes, it is quite another story of course!

Hopefully, these innovations will lead to new musical possibilities down the line that I can't predict right now, just as software and hardware designed to measure Yo-Yo Ma's bowing led—in a slightly zigzag way—to Guitar Hero. I would not be surprised, for example, if the sophisticated infrastructure that Simon Powers uses to construct and communicate his legacy were eventually to morph into a platform for everyone to create and share musical stories—a kind of Personal Opera—on your mobile phone, something on which we are already working with several partners, including the Royal Opera House at Covent Garden (London). Just as Simon Powers builds his legacy through the interconnected elements onstage, Personal Opera might provide a new form for preserving and communicating memories, telling stories, and establishing continuity across generations. And we are designing this environment to encourage the spontaneous accumulation of impressions and memories from one's personal databases, shaped by natural input such as vocalizing or gesturing, rather than favoring the Sargent-Pepper-like studio production that inspired me so many years ago to find another route. We believe that the result will be both liberating and surprising, and will bring together generations in story telling and listening, and experts and amateurs in a beneficial mentoring environment.

I think that it is precisely this kind of surprising freshness that technology can allow—through what can be precisely customized for each project and through the unexpected new discoveries that each project seems to require or reveal—that remains one of its continuing attractions for me.

But we can't take such freshness for granted. Musical technology is so ever-present in our culture, and we are all so very aware of it, that techno-clichés and techno-banalities are never far away and have become ever more difficult to identify and root out. It is deceptively challenging these days to apply technology to music in ways that explode our imaginations, deepen our personal insights, shake us out of boring routine and accepted belief, and pull us ever closer to one another.

That's what makes this kind of work worthwhile and inspires me. But it also leads to a paradox that I experience every single day: that the desire to shape the future is not perfectly compatible with the knowledge that musical experience—and its power to excite and transform us—is fleeting, here and now, at this very moment. And that we'd be extremely fortunate indeed to create new sounds and instruments and technologies that approach the compact, powerful perfection of playing, listening to or imagining Bach emanating from a solo cello.

What new technology can add to this mix is the potential for establishing a new model for the interrelationship between experts and amateurs in musical listening, performance and creation. Some of the boundaries to active engagement in music have eroded, but there is still much to be done to create a truly vibrant musical culture.

In my view, a prime example of the kind of new musical 'ecology' that we should seek is found in our culture's relationship with cuisine. We all enjoy eating at three-star restaurants and admire the achievements of the world's greatest chefs. At the same time, we do not hesitate to dive in ourselves to prepare special meals of high quality on special occasions. We also put together daily meals for ourselves, improvising content that reflects our personal styles. We enjoy eating and even studying the most 'expert' cuisine we can find, but are not scared to make and invent our own. In turn, the fact that we constantly prepare food ourselves makes us better understand and appreciate other food that we encounter.

Music—and most of the arts—have come very far from such a 'healthy' ecology, and it is this that we need to reinvent. Technology can help, as it can act as a bridge to each of us depending on our background and experience, taking advantage of our skills and compensating for our limitations. Even more importantly, we need to establish a fundamentally new partnership between all of the potential participants in our musical culture, including individual artists, all parts of the music business, technology, lifestyle, health and social organizations, music presenting and broadcasting entities, research institutions, artists-as-mentors and—last but not least—the music-loving public. Only in this way can we establish a culture that will allow music to reach its full potential in shaping and transforming our experience. Doing so will allow music to exert its most powerful possible influence on society at large. Surely we can imagine a world where music—and opera—is at least as nourishing as a three-star meal?

Contenidos

- Future Opera for Robots and People Too