People sometimes ask me, “Paul, you think real orchestras are so much better than the digital instruments that imitate them, yet you don’t conduct a real orchestra. Why not?”

Good question. The problem, though, is simply that there are not enough orchestras for even the few very qualified and talented conductors. Nor is there enough money for orchestras to thrive, let alone advance artistically. And conductors who are lucky enough to find themselves leading an orchestra spend only about twenty percent of their time on music. The rest is spent on the necessary distractions of fund-raising and labor relations. Some spend even less on making music, choosing to add self-promotion to the list of distractions.

So, twenty years ago I began wondering if technology was good enough to be able to make music on a high level … or at a level I’m satisfied with. My initial work convinced me that this is possible, and that it’s just a matter of putting in the time and resources to make it work.

In 1981 I studied with Leonard Bernstein at Tanglewood. One lasting and shocking impression from that experience was witnessing the trenchant, rude recalcitrance of the Boston Symphony Orchestra. There were rehearsals where almost nothing was accomplished because many orchestra members began to talk the instant Bernstein stopped conducting. He couldn’t get a word in at all.

Leonard Bernstein conducing at Blossom Music Festival

Bernstein once exasperatedly asked in one of those tense rehearsals, “what the hell is everyone talking about?!” A voice in the back of the orchestra yelled out, “sex!”

As a teenager witnessing professional music making at the highest level for the first time, I thought Why would an orchestra choose to undercut its own success with this kind of behavior? It didn’t make any sense to me as then, and it makes even less sense to me now.

Slowly I began to realize that the odds were against me if I were to try to go down that path as an un-known conductor. Anyone with an ounce of sense would’ve concluded, as I did, that the likelihood of failure was nearly certain. If the mighty Leonard Bernstein couldn’t even get one of the best orchestras in the world to listen to him, what would my prospects be?

Conductors need middlemen … right?

Conductors are in a curious position. They don’t make any sound. They focus on an artistic vision and try to cajole, convince, and inspire musicians to try to get there. Musicians, on the other hand, aren’t twice-removed like that. They can make their own sound and work to fit their “product” to their vision, without having to convince others to enable it to happen.

So, back in 1988 I thought Wouldn’t it be great if there were some way for a conductor to directly produce sound, sculpting and molding it to his own artistic vision, just like any other musician can do with their instrument?

I went to the MIT Media Lab to find out if that could work. There they had an early digital orchestra system: a lot of hardware-based sound samples, MIDI keyboard and a couple of computers.

After a months of grappling with digital music making, I concluded that the process was so tedious, the results so bad, and the instruments so inflexible and limited that I’d probably have a better shot at musical success by becoming a business man, making millions of dollars, and then spending that on building an orchestra. Really, it was that bad. (NOTE: I actually did try the latter approach, but that’s another story.)

It seemed at the time that my lifetime would be long gone before digital instruments would be viable for performing orchestral music at a high level. Fifteen years later I checked in on the field to see what progress had been made. I was shocked.

The future came!

No longer were the machines so expensive and limited. No longer were the concerns of musical performance uninteresting to the software developers. Now, with cheap disk space and fast computers, a new “brute force” approach was starting to show promise. What would’ve cost $250,000 in 1989 could be done for $5,000. This brute force approach consisted of trying to sample all possible orchestral sounds, including the sounds between sounds, and make them instantly available at any second. This approach won’t be the one that prevails as digital instruments evolve, but at that time, around 2003, it had demonstrated its unequivocal superiority.

The proof of the pudding is in the eating, as they say. And the results of this sample-based, brute force approach were as musical as I had ever heard coming from digital orchestra instruments. But more important than those results was the recognition of the potential these instruments had. When I looked into how the instruments actually worked, I knew that I would be able to use them for musical ends eventually…within my lifetime, even.

Which brings us to now. The digital instruments are still limited, but they’ve gotten much better than they were even in 2003, and they’re still improving. They are improving faster than acoustic instruments. My five-year investment in learning how to play them, how to master them, is paying off. And within the next ten years there is no question that I will be able to follow my musical imagination anywhere it leads with more suppleness, expression and ease than the current generation of digital musical instruments allows.

Now I’m adding the ability to play these instruments live in concert halls, unlocking the potential to create a musical experience on a par with that provided by an acoustic orchestra.

So, instead of seeing digital instruments with all their current limitations and saying “they don’t work,” I see them with all their potential and say “let’s make them work as musically as possible!” … And if I want to stop to try some new articulation, tempo or balance, my “orchestra” doesn’t ignore me and start blathering away like a bunch of arrogant, passive-aggressive brats.

I always knew that having many expressive possibilities and an easy way to control of those possibilities would be the most important aspects and the hardest to work with. Still, I am a bit surprised at how easy it has been to use off-the-shelf hardware and software to do things like conduct the orchestra in real time in a concert hall. I had imagined the need for specialized equipment and software to do that. Now, I just use Apple’s Logic and a couple of Nintendo Wii controllers. Problem solved!

I am, however, disappointed in the utter lack of progress in the software used for playing music. Twenty years ago I used software that represented sound as piano-roll-like marks on a screen. One could change the properties of these sounds with crude, mouse-based tools or MIDI controllers.

That’s still how the software works today. It’s very limiting to musical imaginations. Let’s say, like Debussy, we want a note to morph from a trumpet to an oboe sound. In an orchestra, sensitive players know that’s the goal and the two players can make a seamless transition. And that’s great. But in a digital orchestra there is no need to have “trumpet” and “oboe” as separate entities. After all, they are not separate physical systems on the computer. How much more expressive and delightful it would be to play a note and perhaps by just moving your hand change a note from “oboe-y” to “trumpet-y.” Well that sort of thing has existed for years in music synthesis, but is still not yet a part of sample-based instruments. (Although some people are assiduously working on new instrumental models that will enable these sorts of expressions. And with a bit of tweaking, the sample-based instruments actually can be made to perform in this way.)

But in music technology we still lack some of the basic features people take for granted in word processing programs. Features they’ve had for decades. We can’t, for example, search for all d-minor chords and make them d-major. We can’t apply a transformation (a crescendo, for example) to all instances of a melodic fragment. We can’t filter our view of a musical score to show only certain melodies, regardless of what instrument they appear in. We are still too steeped in “tracks” and “instruments” to make progress in these areas.

These simple data manipulation tools are not available in music applications. We have only a crude model of music represented in our software that merely construes the music as “tracks,” ignoring the formidable computing power on our desks that would enable us to understand and manipulate what is in those tracks. The lack of excellent software for musical work was a big problem in 1989 and remains just as much a problem now.

So, there is a lot of work to do. But the potential is definitely there for digital instruments to be as expressive as acoustic instruments when performing orchestral music.

A few weeks ago I gave it a try in a public concert. I played Beethoven’s first symphony, among other works. And I think it came out sounding surprisingly good.

Have a listen to this, the world’s first performance of a Beethoven symphony live, using only a digital orchestra: Beethoven – Symphony No. 1 (first movement) Recorded May 20, 2009, at Holy Name Church, West Roxbury, Massachusetts.