Why I use a digital orchestra

People sometimes ask me, “Paul, if you think acoustic orchestras are so much better than the digital instruments that imitate them, why don’t you conduct a real orchestra?”

Good question. The problem, though, is that there are not enough orchestras for even the few very qualified and talented conductors. Nor is there enough money for orchestras to thrive, let alone advance artistically. And conductors who are lucky enough to find themselves leading an orchestra spend only about twenty percent of their time on music. The rest is spent on the necessary distractions of fund-raising and labor relations. Some spend even less on making music, choosing to add self-promotion to the list of distractions.

So, I began wondering if technology was good enough to be able to make music on a high level … or at a level I’m satisfied with. My initial work convinced me that this is possible, and that it’s just a matter of putting in the time and resources to make it work.

In 1981 I studied with Leonard Bernstein at Tanglewood. One of the lasting impressions from that experience was witnessing the trenchant, rude recalcitrance of the Boston Symphony Orchestra. There were rehearsals where almost nothing was accomplished because many orchestra members began to talk the instant Bernstein stopped to say anything.

As Bernstein himself once exasperatedly asked in one of those tense rehearsals, “what the hell is everyone talking about?!?” A voice in the orchestra shot back, “sex!”

Why would an orchestra choose to undercut its own success? It didn’t make any sense to me as a kid, and it makes even less sense to me now.

So I began to realize that the odds were against me if I were to try to go down that path as an un-known. Anyone with an ounce of sense would’ve concluded, as I did, that the likelihood of failure was nearly certain. If the mighty Leonard Bernstein couldn’t even get one of the best orchestras in the world to listen to him, what would my prospects be?

Conductors need middlemen … right?

Conductors are in a curious position. They don’t make any sound. They focus on an artistic vision and try to cajole, convince, and inspire musicians to try to get there. Musicians aren’t twice-removed like that. They can make their own sound and work to fit their “product” with their vision.

Back in 1988 I thought, wouldn’t it be great if there were some way for a conductor to directly produce sound, sculpting and molding it to his own artistic vision? So, I went to the MIT Media Lab to find out if that could work. There they had an early digital orchestra system: a lot of hardware-based sound samples, MIDI keyboard and a Macintosh computer.

After a year of trying, I concluded that the process was so tedious, the results so bad, and the instruments so inflexible and limited that I’d probably have a better shot at musical success by becoming a business man, making millions of dollars, and then spending that on my own orchestra. Really, it was that bad. (NOTE: I actually did try the latter approach, too.)

It seemed at the time that it would be beyond my lifetime before digital instruments would be viable for performing orchestral music at a high level. Fifteen years later I checked in on the field to see what progress had been made. I was shocked.

Never underestimate the pace of change

No longer were the machines so expensive and limited. No longer were the concerns of musical performance off the radar of the developers. Now, with cheap disk space and fast computers, a new “brute force” approach was starting to show promise. What would’ve cost $250,000 in 1989 could be done for $5,000. This brute force approach consisted of trying to sample all possible orchestral sounds, including the sounds between sounds, and make them instantly available at any second. This approach probably won’t be the one that prevails as digital instruments evolve, but at that time, around 2003, it had demonstrated its unequivocal superiority.

The proof of the pudding is in the eating, as they say. And the results of this sample-based, brute force approach were as musical as I had ever heard coming from digital orchestra instruments. But more important than those results was the recognition of the potential these instruments had. When I looked into how the instruments actually worked, I knew that I would be able to use them for musical ends eventually…within my lifetime.

Which brings us to now. The digital instruments are still limited, but they’ve gotten much better than they were even in 2003, and they’re still improving. My five-year investment in learning how to play them, how to master them, is paying off. And within the next ten years there is no question that I will be able to follow my musical imagination anywhere it leads with more suppleness, expression and ease than any orchestra I would have been able to get myself in front of. On top of that, I can spend 80 percent of my time on music, making the day-to-day aspect of this particular musical career far more rewarding than that of an orchestra conductor.

Now I’m adding the ability to play these instruments live in concert halls, unlocking the potential to create a musical experience on a par with that provided by an acoustic orchestra.

So, instead of seeing digital instruments with all their current limitations and saying “they don’t work,” I see them with all their potential and say “let’s make them work!” … And if I want to stop to try some new articulation, tempo or balance, my “orchestra” doesn’t ignore me and start blathering away like a bunch of nincompoops.

I always knew that having many expressive possibilities and an easy way to control of those possibilities would be the most important aspects and the hardest to work with. Still, I am a bit surprised at how easy it has been to use off-the-shelf hardware and software to do things like conduct the orchestra in real time in a concert hall. I had imagined the need for specialized equipment and software to do that. Now, I just use Apple’s Logic and a couple of Nintendo Wii controllers. Problem solved!

I am, however, disappointed in the utter lack of progress in the software used for playing music. Twenty years ago I used software that represented sound as piano-roll-like marks on a screen. One could change the properties of these sounds with crude, mouse-based tools or MIDI controllers.

That’s still how the software works today.

In music we still lack some of the basic features people take for granted in word processing programs. Features they’ve had for decades. You can’t, for example, search for all d-minor chords and make them d-major. You can’t apply a transformation (a crescendo, for example) to all instances of a melodic fragment. You can’t filter your view of a musical score to show you only certain melodies, regardless of what instrument they appear in.

These simple data manipulation tools are not available in music applications. We have only a crude model of music represented in our software that merely construes the music as “tracks,” ignoring the formidable computing power on our desks that would enable us to understand and manipulate what is in those tracks. The lack of excellent software for musical work was a big problem in 1989 and remains just as much a problem now.

So, there is a lot of work to do. But the potential is definitely there for digital instruments to be as expressive as acoustic instruments when performing orchestral music.