This interview was conducted in 2009 with Tobias Fischer of Beat Magazine.

When did you first start getting interested in the idea of a digital orchestra?

When I graduated from conservatory in 1987 I was already demoralized. Even though I had been given the secret keys to making musical magic by Sergiu Celibidache [former conductor of the Munich Philharmonic], in America there was no way even such a great musician as he could find support for the intense (and lengthy) process needed to make musical performances really great. If Celibidache could not succeed here, with his fame, forceful personality and widely admired musical results, I thought, how could a guy like me, with few connections and no history, convince anyone to provide the considerable resources needed to make orchestral music on the highest level?

I saw a world where the priorities were upside down. People thought, for example, that there was nothing wrong with spending eight hours filming a three-second shot of a woman’s leg coming out of a car for a Hanes pantyhose commercial, while at the same time reacting with shock that a conductor might want to spend that same amount of time perfecting twenty minutes of music by Brahms. Even musicians were ready to undercut themselves in the name of a professional machismo: they didn’t need to spend that much time because they were professionals. They were full of indignant puffery and assumed that anyone who wanted to rehearse Brahms for more than 45 minutes was an inefficient “talker” and a waste of time. Hanes or Brahms? Which one was more important? Which deserved more attention and care to bring to life? Which one could deliver a magical, transcendent experience?

Sadly, not much has changed in thirty years … musicians still think spending less time on music is a sign of professionalism, and yet no one bats an eye if a movie studio spends 80 million dollars and hundreds of hours on thirty seconds of special effects. “Good enough” is making orchestral musical performance less and less relevant in the meantime.

And so the problem remains now as it was then: to achieve musical results on the highest level requires more time and effort than anyone seems willing to support. Could technology possibly provide a solution?

I was studying in Boston at the time and decided to find out if the newly-opened Media Lab at MIT (Massachusetts Institute of Technology) could help. I started working on a computer-controlled piano: a Bösendorfer 290 that was connected to a Sun computer. The power available was incredible. Each note could have 8,000 different levels of velocity (or loudness). Compare that with MIDI, which provides only 127 levels.

I worked on Mozart and Brahms and made some great music there. However, the process was tedious and lacked any sort of interface for the spontaneous, real-time control over musical parameters. It turns out to be very hard to plan changes in loudness and tempo off-line and have it work musically. It would be much better to be able to change those elements as the sounds are unfolding.

It was at the MIT Media Lab that I was introduced to a very early digital orchestra system. It was very limited, unwieldy, expensive and a jealously guarded resource. I tried it, though. It seemed to have potential, but I thought it would not be ready for serious musical use for fifty years. Luckily I was wrong … it was 15 years later that the digital orchestra became a viable, expressive instrument.

How would you describe reactions to your ideas in the early stages?

People were amazed at the piano music. And, even the orchestral work, which I thought was not very good, people were amazed about that, too. But that amazement had more to do with knowing that a computer was “playing” the music. I wanted people to be amazed at the musical experience without caring whether it came from a computer or Yo-Yo Ma’s cello.

My own reaction was negative. I could see that digital musical instruments would one day work well enough to make music at the highest level. But, I thought it would be fifty years before that would occur. On top of that, I could not afford the tools at MIT. They cost hundreds of thousands of dollars at that time. Ironically, they would remain largely un-touched in the lab while technology passed them by in the real world over the ensuing two decades.

Who supported the project financially and in other ways over the years?

Michael Hawley at MIT was a great supporter early on. He gave me access to the tools, wrote the software, and allowed me to follow my own musical interests. More recently, Herb Tucmandl and the people at the Vienna Symphonic Library have been very supportive of my work, providing software and support. Bang & Olufsen have helped make the live performance of the digital orchestra possible by providing top-of-the-line BeoLab 5 speakers. And many musicians, technicians, developers and composers who see the potential of the digital orchestra have been helping out all along.

What, to you, were significant milestones on the road to the digital orchestra of today?

The field of electronic music has been around for at least sixty years and a lot of the progress in that field as a whole has led to the digital orchestra possibilities we now have before us. Significant milestones include the idea of sampling, which was demonstrated as a successful approach to orchestral music with the Synclavier in the 1980s. Without MIDI there would be little commercial incentive for much of the advances in all digital music over the last twenty years. But perhaps the greatest boon to this field has been low cost disk space and processing power. Without these last two, digital orchestra instruments would still be ideas on the drawing board.

So, we are still far away from having the digital instruments we need to make music on the highest level. But I am now convinced that we’re not going to need to wait fifty years. It’s going to be more like twenty years to get to that point. That’s extremely exciting.

Your approach is based on very particular choices: A huge pool of predefind samples and performance techniques and an organic way of combining them into music. Were there alternative concepts which you considered in the beginning?

Yes, there were and there are always alternatives. It’s important to always be aware of developments and alternatives because no one can predict what will be invented and which approach might suddenly work better.

The sample-based approach is a brute-force approach to the problem of musical expression. It basically says, “let’s record every possible expression we may need.” In the long run this approach is absurd. It requires ever more complex organization, and runs the risk of obsolescence as musical ideas and language develop. But it’s the most successful approach available now.

The main alternative is to use computational models of how musical expression works. This doesn’t work very well yet, but I think computational modeling will be the ultimate direction we go in, leaving samples behind. Think about this: if I want to make a crescendo in the violins, with my sample library I can find that someone has recorded a violin crescendo of three seconds. What if I need 3.5 seconds? What if I’m playing in real-time and have no way to tell the computer whether I need a shorter or longer crescendo? And what about how much crescendo? How loud should it get, and what “shape” should the crescendo’s curve be? A purely sample-based approach would require a library of thousands of crescendos just for these violins. And then, there would need to be powerful software to manage access to those choices using fast processors to pluck the right file and pump it out to audio in a few milliseconds.

The computational modeling approach is more likely to succeed. This approach says, “let’s make a mathematical model of how a violin can make crescendos. Let’s let the musician play whatever shape, loudness and length of crescendo he wants and simply apply the model in real time to the sound that’s coming out.” Synful Orchestra is making amazing strides in this approach and it’s just a matter of time before it will reach a tipping point of usability. When that happens, sample libraries will be dinosaurs.

Of course, it pays to remember that people thought unlimited expression would be available with digital sound synthesis twenty years ago. The difference, however, is that the current computational approach is based on models of musical instrument sound, while earlier synthetic approaches were based on the abstract fundamental building blocks of sound. It sought to create complex sound from the bottom up by adding simple waveforms together. So, although it technically had the potential to create any imaginable sound, it was not practical for playing music! A mathematical model of how a violin-type sound changes during a crescendo is a much more powerful thing than a recording of a crescendo. This modeling approach is already successful in artificial reverb. Convolution reverbs simply apply a mathematical model of an actual acoustic space to a sound signal, and they work extremely well.

Was there a big discrepancy between what you thought would be the most difficult aspects of creating a digital orchestra and what really turned out to be most challenging parts?

Not really. I always knew that having many expressive possibilities and an easy way to control of those possibilities would be the most important aspects and the hardest to work with. Still, I am a bit surprised at how easy it has been to use off-the-shelf hardware and software to do things like conduct the orchestra in real time in a concert hall. I had imagined the need for specialized equipment and software to do that. Now, I just use Apple’s Logic and a couple of Nintendo Wii controllers. Problem solved!

I am disappointed in the utter lack of progress in the software used for playing music. Twenty years ago I used software that represented sound as piano-roll-like marks on a screen. One could change the properties of these sounds with crude, mouse-based tools or MIDI controllers. That’s still how the software works today. In music we still lack some of the basic features people now take for granted in word processing programs. You can’t, for example, search for all d-minor chords and make them d-major. You can’t apply a transformation (a crescendo, for example) to all instances of a melodic fragment. You can’t filter your view of a musical score to show you only certain melodies, regardless of what instrument they appear in. These simple data manipulation tools are not available in music applications. We have only a crude model of “tracks” stacked up without using the formidable computing power on our desks to enable us to understand and manipulate what is IN those tracks. The lack of excellent software for musical work was a big problem in 1989 and remains just as much a problem now.

Were there „external“ (meaning: not planned by yourself) technological developments, also from non-musical areas, which were beneficial to the project over the years?

Cheap disk space and CPU power. The falling cost of access to computational power has been the greatest boon to digital music making I’ve seen in decades. Products like the Vienna Instruments would have been insane in the days when 1 Gigabyte of disk space cost $4,000 (which was not long ago). Now it’s pocket change to have a Terabyte of data on my desk. It’s not just five times less expensive, it’s literally 500 times cheaper than only ten years ago.

The lesson is that falling costs are going to continue to open up greater opportunities … even ones that might now seem insane (like recording thousands of violin crescendo samples)!

How do you go about performing and recording a piece in practise? Even though the Wii controller seems an easy to use accessory, there’s the question of how to quickly access all the different options at your disposal…

My approach is to limit what I need to control in real time to the absolute minimum for musical expression. For example, I could theoretically use controllers to change notes, rhythms, balance, tempo and loudness. The fact is, however, that I don’t need to change most of those things. In fact, even for tempo and loudness I can create a good “ballpark” estimate and leave minor adjustments to work with the Wii controllers in real time.

This approach is exactly the same as playing an acoustic instrument. We don’t expect to change any possible musical element in a performance at any possible moment. No, instead we practice the notes, the dynamics and the tempo changes knowing that we may spontaneously adjust some of those elements at some times in response to the acoustics of the room, the musical demands of the moment, or whatever else may motivate a musician to “deviate” from what was rehearsed.

I’ve tried to make the Wii controllers as narrowly focused as acoustic instruments, while maintainng their accuracy and power in those limited areas.

How do you achieve the depth of an orchestral sound in terms of acoustics?

Good question! I’m always looking for shortcuts. So, for me, the shortest path to a great orchestral sound is to get as many excellent loudspeakers as I can and assign them to their appropriate instruments (I avoid using sub-woofers for violin sounds, for example). Then, bring all these into a concert hall and let the acoustics of the hall help build the sound.

For classical orchestra, I have just the right instrument in the Bang & Olufsen BeoLab 5 speakers. I pump a relatively dry signal through them, out into the concert hall, and the rest just happens “naturally.”

Now, with recording, this is not the way I work. There are no speakers involved … the acoustics are molded in software using EQ, stereo positioning and reverb. Altiverb is absolutely the best tool for getting to that sound. It’s a reverb plug-in.

But this question goes beyond technology, too. You have to know how an orchestra itself would work acoustically. And what a composer’s score is asking us to do. For example, you have to know that you will never hear a low flute sound when trumpets, trombones and horns are all playing as loud as possible at the same time. Seems obvious, but with the computer you can make the flute sound be heard in that context. And while this is an extreme example, perhaps, the idea is that you have to know how the sounds a composer wants would and should actually be combined and balanced in an acoustic orchestra setting before you can get even close to making a good digital orchestral performance.

One of the most striking features about your digital orchestra, to me, is that is seems to contradict most people’s idea of how a regular orchestra works, which is that playing emotionally gripping music (whatever that may mean) is not just a „mechanical“(I’m using the term provokingly of course) combination of notes played in a particular way. Why is it possible to make your recordings sound this organic and natural after all? Or to put it differently: Why will every performance of the digital orchestra still sound different from previous ones?

My attempts still fall short of what I really want to attain, but it’s good to know that someone hears something worthwhile in them, so far. The most important thing is to know how to play music. What’s written in a musical score is everything but what is essential to making music. When a passage is marked “loud” in the score, it does not tell you which instruments should be louder than others; it doesn’t say how loud it should be; it doesn’t tell you if the loudness should grow in intensity or fade away. There are many, many details that are not indicated in the score. These details are crucial to creating compelling performances.

If you take a score and simply tell the computer to play back what is written, it doesn’t matter how excellent your sample library is, it will sound lifeless and flat. That’s because the score does not contain enough information … and the software does not yet exist for “understanding” what a score implies to a musician.

That knowledge, what a composer is suggesting with a score, and understanding how to harmonize the elements that are not specified with those that are, are the key to being able to create a great musical performance. This comes from musical training and experience. And from being open to listening. And it’s completely independent of whether one is using acoustic or digital instruments. What is making the sound does not matter.

And so, yes, you are quite right that digital orchestra performances of the same piece of music will sound different. That’s true with acoustic orchestras, too, and for the same reasons. Even my own performances of the same piece … using the same files and speakers … differ from one to the next. That’s to be expected. Nothing is ever the same twice in your experience. That’s the way life is. Music has to be the same way.

The digital orchestra places a lot of musical power in the hands of the conductor. Do you, thus, see the axis between composer and conductor as essential to performances of the digital orchestra for the future (or maybe even of those two artistic disciplines as converging)?

Yes, of course. Most composers and orchestral musicians have a very bad relationship until after the composer dies, and then the musicians love him. Conductors often have to fight to convince an orchestra to play a composer’s work. But the digital orchestra doesn’t resist anything! This is another example of orchestras getting in the way of their own success. For, without continual nurturing and airing of new orchestral ideas, the orchestra itself becomes an irrelevant cultural enterprise.

This is one of the most exciting and important contributions digital orchestras can make in our musical life. They enable composers to get their music played without the cost and resistance of an acoustic orchestra. My hope is that this new-found avenue for orchestral composition will result in a flowering of activity supporting innovation and refinement in orchestral music, necessary to keep this mode of expression alive and thriving for both acoustic and digital orchestras. In my view, then, acoustic orchestras ultimately benefit from the emergence of very good digital orchestras. Just as interest in acoustic guitar music has been immeasurably increased by the emergence of the electric guitar.

Most of your current projects deal with symphonic-scale orchestras if I’m correctly informed. Would the digital orchestra work just as well on a smaller scale – i.e. for chamber musical settings?

Yes, technically digital orchestra technology can work just as well with a single instrument. However, there is much less economic incentive to use a digital instrument to play solo piano or violin or to play duets or quartets. It’s still cheaper and better to play such music with acoustic instruments. In other words, orchestral music is the area where the technology can have the greatest impact. And it is already doing that. In film music, for example, a full orchestra score can be realized using digital orchestra at a fraction of the cost of an acoustic orchestra. This is happening every day. At the same time, certain parts are still played with acoustic instruments where digital orchestras are lacking.

Not all instruments are equally expressive, however. A solo digital piano is a lot closer to an acoustic piano than a solo violin is to an acoustic one. So, digital piano music is likely to be much more successful, musically speaking, than solo digital violin music. But, the capabilities of these instruments are always improving, and their cost is always dropping. But, I know that the technology itself is never going to replace musicians, as some mistakenly fear. The fact is, as I’ve tried to explain above, that if someone with no musical training or ability plays a digital violin, the result is just as bad as if that person were playing a Stradivarius.

If four string players wanted to get together and play Beethoven string quartets with digital instruments, they would get much better results than four non-musicians. This is true in other media, too. The ubiquity and low cost of digital video tools poses no threat to the Spielbergs or Fassbinders of the world, as five minutes on YouTube will assure you!