You’ll be able to see four versions of footage from Peter Jackson’s King Kong. Version 1 has no music; Version 2 has digital orchestra only (and this step is now a regular occurrence in the process of making film scores); Version 3 is acoustic orchestra only; and Version 4 is the final version, which is, perhaps surprisingly, a combination of the digital and acoustic versions.

If you are interested in video game music, you may want to skip to the final 15 minutes to hear a discussion between audience member Filippo Beck Peccoz, composer and founder of the Berklee video game music club, and the panelists. It’s a vivid demonstration of what used to be called the generation gap!

This article is about a new technology in the motion picture industry for creating “digital doubles.” These are animated version of actors that can pass as real under many normal viewing situations.

Scroll to the bottom of the article to see the video (which is all digital double until the very end).

This goes a long way beyond having to edit film so that stunt doubles appearing on screen never (or only briefly) show their faces. Now an actor’s performance can be grafted onto any other filmed or digital “rig” as they call it.

So this is related to music made with musical samples. But much more computationally advanced and intensive.

From the article:

Instead of conducting an orchestra from a concert hall, IU music informatics professor Chris Raphael will bring the orchestra to a computer screen.  Raphael received a three-year, $450,000 grant from the National Science Foundation for his proposal, “Real-Time Planning of a Conductable Orchestra,” said Lisa Herrmann, manager of communications for the School of Informatics. 

The culmination of the project will be a computer program that is able to understand the gestural language of musical conducting through video

Wow, $450,000?

I guess the NSF didn’t realize that this sort of thing has already been done and already works on generic computer technology. I have been doing this for a year now using a Mac and a Wii controller along with off-the shelf applications that convert the Wii’s signal to control the applications. All for less than $20,000. (See http://www.fauxharmonic.com )

And, of course, other researchers have been working in this area for years. Notably Teresa Marin Nakra, Jan Borchers and others. Hopefully the grantees will do a careful, comprehensive review of the state-of-the-art before forging an already-trod path. That would be a very helpful contribution to the field of digital orchestra research.

But, still, why should it cost so much? Maybe because the symbolic gestural “language” of conducting is not and has never been so clear cut that a digital signal processing system could “interpret” these gestures, and that, therefore, perhaps hundreds of thousands of dollars might begin to crack that nut?

The model of conducting that construes it as communicating instructions to the orchestra is only, at best, partial. A great deal of the gestural language of conducting is dance-like, in that it reflects the character of the music in visual form … for the audience as well as for the musicians. So conducting is no more “pretty well defined” than dance in terms of being able to be “understood” by a computer program.

And this large collection of gesture, body language, facial expression and conducting tradition is highly unlikely to be cracked by the IU researchers. More likely they will have to severely restrict what gestures their system will recognize as conducting. Conducting students will have to conform to the system, artificially cutting themselves off from a vast repertoire of subtle aesthetic expression in order to get the technology to play faster, slower, louder or softer.

If that’s how it turns out, it will be a much poorer tool for training actual conductors than, say, conducting a student ensemble (which $450k could support for the next 20 years at IU).

And what’s with that souped-up Boesendorfer piano that has nothing to do with orchestra (digital or otherwise), and costs well over $100,000?  Wouldn’t a Yamaha Disklavier do just as well? Or, even more a propos the subject: an installation of the Vienna Symphonic Library?

Well, it will be fascinating to see what results come of this research. Hopefully Raphael will avoid the dead-ends of earlier work in this area over least 30 years and come up with something that offers truly responsive musical expression, and not just a fancier way to twiddle with the volume and speed knobs on a recording.

Why is this important?  What difference can space make?  Why do recording engineers spend years learning how best to recreate it artificially?

A live space introduces reverberation.  If performers are in that space and hear that, they can change the way they play (and they will, if they’re good musicians).  Perhaps the music will be slower, with more space between notes.  Maybe the time between the end of one phrase and the beginning of the next will be longer.  Maybe the slope of the crescendo curve will be flatter. And, beyond issues of tempo and dynamics, the demands of the space might induce the performers to change the tone color, and shorten or lengthen notes.

A good musician knows about, hears and responds to these demands of the room. When it’s done well, it can mean the difference between an ordinary and extraordinary performance.  

But what happens when music is routinely played in one space, decoupled from it, and then listened to in another? The performers have no chance to use any of their ability to respond to the acoustic space in which the listener is listening.  

So, what?  A good recording reproduces the original performance space so well that we “suspend disbelief” and imagine we are in a different space altogether.  Well, at least that’s the ideal.  But, unless you’ve spent tens of thousands on speakers, installation, calibration, sound damping and isolation, you’re not going to find yourself in such a listening space.  Still, can’t we imagine we are “there” no matter whether we’ve got 50 db of road noise, traffic, a small room or whatever other acoustic interference might be in our environment.  Yes, apparently we do that all that time … willingly.  And the reason we do so is because we intuitively understand the need to dampen the dissonance between the actual space we are listening in and the illusory space presented via recorded music.

However, that imagination and willingness to suspend disbelief is not enough to overcome the fact (and it is a fact) that the performer can not re-calibrate the sounds in response to the room the listener is in.  And so, even with the listener’s imagination and the engineer’s prowess, the full potential of the performer is diminished.  

Is performing and listening in the same acoustic space really all that important?  

If the performer is not one who “plays the room” in the first place, clearly it won’t matter if the effects of the original acoustic space on the performance are changed or eliminated.  The performance will have the same tempo, inflection, color, dynamics, etc., no matter what space it’s played in.  Anyone who performs along with pre-recorded material falls into this category.  

But, if the performer is one who plays the room, they can create conditions for musical experiences that take listeners well beyond the ordinary, the here and now.  Those fabled “transporting” experiences that seem, unfortunately, to happen so rarely, are far, far more likely to occur when listeners are in the same room as performers who know how to play the room.  

The room.  Awareness of it is what allows the performer to present the sounds with the clarity, balance and function that is absolutely essential to having great aesthetic experiences.  

To make the point another way, an analogy (although analogies seem frequently to be unintelligible to some who can only manage to see how the analogy diverges instead of clarifies – I’ll take a risk and present one anyway):

Suppose a sculptor creates a work that must be seen from a certain spatial perspective to be grasped, understood, or, even in some fundamental sense, to be seen at all?  And what if a viewer looks at the sculptor from another orientation?

In this work sculptor Shigeo Fukuda used over 800 pieces of cutlery.  But, unless you look at the sculpture’s shadow, you won’t see “it.”  

Seeing the shadow of the motorcycle is essential to this artist’s work and vision.  Now imagine you buy this work, put it in your living room, and light it from the side.  A viewer might admire the sculpture and have an aesthetic experience, but she won’t be having the experience the artist envisioned.  ”I like it,” she might say. And there is nothing wrong with that.  If you’re a cultural relativist, you just say, “to each his own” and move on.  

But if, like me, you think that artists who have been universally lauded for their ability to transport us and lead us to experience their vision are worth going through the delicate and difficult steps to try to get to their vision, then you realize that sometimes what may seem minor, personal preferences can actually be hindrances.  

The fact is we already strive to eliminate many of these hindrances without batting an eye.  Playing out of tune, for example, is not acceptable and almost every musician has to deal with this issue.  Everyone knows playing out of tune hinders the ability to get to the experience the artist (composer) envisioned. 

So, playing the room – responding to the room – is crucial to getting a chance to experience what a composer has envisioned.  Listening to recorded music even in the best audio environment cuts away that entire realm of musical technique from being brought into play.  Unlike playing out of tune, for some reason, that’s a hindrance that we seem to think is okay to live with. 

Playing music so that a great experience could occur is one of the most difficult things anyone can do.  That’s why it’s so rare to have these great experiences.  If your goal is to have those experiences (whether you’re a listener or a performer), it doesn’t make sense to accept such a big handicap right out of the gate.  Cutting off the possibility of responding to sounds as they unfold in a room virtually guarantees failing to reach the composer’s vision.