To Really Judge Good Music, Turn Off the Sound

In a study published in the Proceedings of the National Academy of Sciences, Chia-Jung Tsay showed musicians clips from classical music competitions. She asked them to guess the winners. Different musicians were given different kinds of clips: audio recordings… videos with sound… and videos with no sound!

They did best when they saw videos with no sound.

It’s not that the winners were good-looking. It’s that they moved in expressive ways.

This reminds me of how people are willing to pay more for wines whose names are hard to pronounce… or how you can predict who will win an election by watching videos of the candidates—with the sound off.

I think there’s something important about these results. They’re a bit depressing: if our cognitive apparatus is so deeply flawed, maybe we’re doomed. But maybe it’s not so bad. Maybe we just tend to define the point of various activities too narrowly!

We go to a concert, not just to listen to music, but to watch the performer. We drink a wine, not just to taste it in our mouth, but to bask in the sense of being sophisticated. We judge a candidate, not just by what they say, but by how they say it.

You can see the paper here:

• Chia-Jung Tsay, Sight over sound in the judgment of music performance, Proceedings of the National Academy of Sciences, 19 August 2013.

You can hear (or see) a nice summary and discussion here:

• Shankar Vedantam, How to win that music competition? Send a video, National Public Radio, 20 August 2013.

33 Responses to To Really Judge Good Music, Turn Off the Sound

  1. einzztein says:

    Isn’t the inference the other way around – to judge music really well, turn off the visual element and pay attention to the music?

  2. Mark Meckes says:

    There was another similar study, made famous by a Malcolm Gladwell article, in which students accurately predicted a professor’s course evaluations by watching an eight-second clip of a lecture with the sound off.

  3. Peter Morgan says:

    Not sure what your “sarcasm” comment means?
    We need at least one other experiment —for which one proposal would be prizes awarded by judges sight unseen, with otherwise the same experimental conditions. It may be that the judges would get the same results without the visual data stream, because they perceive details of phrasing in the audio data stream that most people do not perceive, but which are correlated with characteristics of the visual data stream that many people can perceive.

    • Mark Meckes says:

      This suggestion brings to mind the fact that the tremendous historical gender gap in professional orchestras began to decrease dramatically once the orchestras started having blind auditions. So I wouldn’t count on such a difference between the judges and most people.

      • Richard says:

        This suggestion brings to mind the fact that the tremendous historical gender gap in professional orchestras …

        This is indeed exactly what is going on. And it is a very very good thing that this is not what is going on with blind auditions in orchestras these days, at least with some orchestras.

        A female friend and a long-ago violin student at Juilliard relates the story of how one of the teachers was notorious among students (and presumably others) for favouring the willow blonde women. My friend and her cohort were, naturally, exercised about this state of affairs, but with the passing of time her view has become: “There are so many so very good musicians with so little technical or aesthetic difference to rank them, and so few jobs and so very few careers and so vanishingly few superstar slots, and there is so much chance involved in every step, that he might as well slightly prefer the blonds — it’s not so different from flipping a coin.” The Music of Chance.

        Along the same lines, since all of the entrants in any of the marquee musical competitions are of such spectacular ability — any of them with the technical skill to play, routinely, what used to be virtuosic finger-breakers that were the province of only the most celebrated of historical performers — then what have we left to decide who is The Winner? Well, since the job (should a solo touring concert career eventually follow from a win at a competition, which itself is bad odds) in the end is to deliver paying middle aged enraptured musical amateurs into concert hall seats, some sort of stage presence, flamboyance, theatricality, etc, can be a real-world edge, and one that competition juries aren’t ignoring, and perhaps ought not to ignore.

        I find Lang Lang (pictured) irritating to watch and not aesthetically interesting to hear, but there is absolutely no doubt that he is good for the (dismal) classical music biz, and that he sells seats and that he reaches vast audiences that another equally virtuosic pianist would not. Looks and some showboating do count!

        By the way if you think the job market for physics PhDs is bad, consider conservatory students. And no, there isn’t the “slumming as a quant on Wall Street” option when you wash out.

    • John Baez says:

      Peter wrote:

      Not sure what your “sarcasm” comment means?

      I meant the title of this article of mine is somewhat sarcastic.

  4. Cristi says:

    What if there is a correlation between how good a performer is, and how expressively she moves? I mean, what if she really lives music at all levels, including the body?

    • Mark Meckes says:

      Partly in connection with this point, I wonder how broad the pool of the musicians in the videos was. Was it a cross-section of all the musicians who entered the competitions (potentially already a quite select group)? Was it only finalists, whose musical abilities may be uniformly high enough for even experts to have difficulty to distinguish meaningfully?

    • jessemckeown says:

      This depends on what you mean by “good performer”, to which John alludes in remarking on the point of going to a concert.

      But if you mean: correlation between audible output and visible expression; then there should be a similar correlation in the judgments based on only-audio with judgements based on only-video.

      Out of curiosity, do we know what happens if videos are given the wrong audio? Can a thrilling dance redeem a flat musical delivery?

      • John Baez says:

        Jesse wrote:

        But if you mean: correlation between audible output and visible expression; then there should be a similar correlation in the judgments based on only-audio with judgements based on only-video.

        As Peli points out, participants in the study performed significantly worse than chance in guessing the winner based on audio alone. This makes me guess that like the participants in the study, the judges are relying heavily on visual information.

        Out of curiosity, do we know what happens if videos are given the wrong audio? Can a thrilling dance redeem a flat musical delivery?

        I don’t think anyone has tried this yet. It might be a bit hard, because I think a tolerably good musician could often notice that the video and audio were out of synch, unless you managed to find two performances that maintained exactly the same tempo. When it comes to audio aspects, there are far fewer degrees of freedom that make the difference between a good and bad piano performance than, say, violin—and one is tempo.

        Of course maybe I’m wrong and musicians wouldn’t notice the sound was out of synch. Someone should try it.

        In a related comment over on G+, Carsten Führmann thought of another trick we could try:

        Interesting. I guess the significance of the result hinges on whether the sound-only abilities of the competitors where in a narrow corridor. If that were so, differences in appearances might easily override the (small) differences in musical ability. It would be cool to inject a comparatively bad competitor and see how far they can compensate by appearance.

        I replied:

        That would be interesting. I suspect that past a certain level, sounding good is usually highly correlated to looking good—not in the sense of personal appearance, but in the sense of making gestures and motions that indicate you really understand the music in a deep way. But you might be able to train a musician with some acting talent to look better than they sound.

        After reading Peli’s comment, though, I’ve changed my guess. I now claim that the level of these piano competition finalists, looking good is not strongly correlated to sounding good. The judges mainly base their decisions on whether the contestants look good (e.g., move well).

        • Cristi says:

          Thank you for ruining the idea that music flows through the performer, animating her body… :-)

        • jessemckeown says:

          Cristi, that’s really not what I meant! It’s a fine thing to really enjoy music one is making, and sometimes that’s a big part of what an audience connects with as well. We just have to learn to be precise about what we mean by music and performance; musical skill and performance skill, etc.

          As for whether there should be a correlation: let me put it this way, (and as a modest violinist I know something of this first-hand): it is much harder to dance and control an instrument at the same time than it is to play with a relaxed, static posture. To move expressively and play with control, one’s arms and hands have to cancel out the extra torques and accelerations of moving about in a changeable manner. A pianist has less scope for movement than a violinist, but still, if he is moving about, then he has to move his arms differently to reach the same notes and hit the keys with the same accelerations. This is not to say it can’t be done, nor that the overall effect is less artistic; but it may at the same time be an exciting thing to watch and also less acoustically refined.

  5. The experimenter had people observe clips of musicians in competitions, then guess how well the musicians placed. Subjects guessed better when given video-only clips as compared to audio clips or audio+video clips. Conclusion: people care about looks far more than they think or admit they do.

    But I think we can’t jump to such a conclusion based on this paper for a few reasons.

    First, the clips were taken from the top three places at prestigious international competitions. These people are already the very best; there was probably very little variation between them. If we rate the auditory quality of the music they played out of 100, maybe they’re at 94, 95, and 96, or something. It’s not surprising that experts didn’t accurately judge who would win based on sound.

    […]

  6. The clips used in the study were only six seconds long. Also, they were only using the top three performers at prestigious international competitions. Thus, they were all of extremely-high quality and we had very short samples. It’s not surprising people couldn’t pick them apart well. That doesn’t mean the way we judge music is primarily based on visual input.

    longer version of this: http://arcsecond.wordpress.com/2013/08/21/people-hearing-without-listening/

  7. arch1 says:

    I suspect the full explanation involves a number of factors.

    Confidence is probably one of them. A performer who believes herself better (or perceived as better by judges) is likely to have a certain swagger (replete with flourishes such as we see above) which lends authority to her performance and influences ratings favorably. This is of course a) also true if the performer can muster the swagger absent the belief, b) applicable to much more than music competitions.

  8. John Baez says:

    Unlike me, Peli Grietzer read the actual paper I referred to. Over on G+ he said some very interesting things about it:

    The exact results of the study are incredibly bizarre—participants performed significantly worse than chance when listening to audio alone, performed at chance when getting audio+video, and performed significantly better than chance when getting video alone. So the study implies that ordinary musicians only barely use visual information for evaluation when audio is available, unlike the judges of the competitions (who are presumably some of the world’s top musicians, since these are the world’s top competitions). It also implies that there’s a consistent negative correlation between visual ability and musical ability at high levels. None of this makes very smooth sense. (They theorize that maybe cognitive load increases reliance on visual information, but the top three are judged at a separate event from the mass of the participants, and they’re dealing with picking the winner out of the top three.)

    An alternative explanation, though I don’t know how likely, would be that musicians at the very highest international level have musical agendas and evaluation criteria that are opaque to the average musician but the visual correlates of mastery are easy to read. Or maybe watching someone being expressive live overwhelms the auditory sense in a way that a mere video cannot? Really weird results. 

  9. Cristi says:

    The most expensive art seems to be the abstract art. I always considered Picasso overrated, given that, in my opinion, just in one painting (http://en.wikipedia.org/wiki/Family_of_Saltimbanques, “the masterpiece of Picasso’s Rose Period”) he severely broke several laws of perspective, physics, and saltimbanques’s bones (and it was before his cubist period). I was not very surprised that monkeys could be almost as good at abstract art like art students (http://arxiv.org/ftp/arxiv/papers/1112/1112.1669.pdf).

    Similarly, hearing that when the wine tasting is blind, California wines can beat the French wines (http://en.wikipedia.org/wiki/Judgment_of_Paris_%28wine%29), confirmed to me that I was right all the time to prefer Pepsi.

    Now (unless this music judgment study was made by a father angry because his daughter failed to be admitted at Juilliard School) music judges turn out to be more sensitive to the package in which music is delivered, rather than to the music. Hmm, this seems important enough for this study to be reproduced by others, and all possible loopholes identified and removed.

    On the same line, what would we expect from a study concerning the judgment of grant applications and science prizes?

    • John Baez says:

      Cristi wrote:

      Similarly, hearing that when the wine tasting is blind, California wines can beat the French wines (http://en.wikipedia.org/wiki/Judgment_of_Paris_%28wine%29), confirmed to me that I was right all the time to prefer Pepsi.

      Since the University of California at Davis has for decades run a program to make good wine using modern scientific techniques, I wouldn’t be surprised if the best wines from California actually are better than French wines—and perhaps even better than Pepsi.

      On the same line, what would we expect from a study concerning the judgment of grant applications and science prizes?

      The humanities use ‘double-blind refereeing’, where the referees don’t get to see the author’s name. The sciences do not, and it seems quite clear to me (from experience) that papers are largely judged based on the perceived prestige of the author. Unfortunately, it’s very hard to effectively hide the identity of the author from the referees (even in the humanities), so it’s hard to see how to correct for this effect.

    • Cristi says:

      Here is what I meant in my previous comment that Picasso is overrated: http://www.unitaryflow.com/2013/08/picasso-is-so-overrated.html

      • nad says:

        Here is what I meant in my previous comment that Picasso is overrated: http://www.unitaryflow.com/2013/08/picasso-is-so-overrated.html

        Yes and he even couldn’t draw faces properly.
        The escape from reality may be a great freedom.

        • Cristi says:

          Nad wrote:

          The escape from reality may be a great freedom.

          That painting was from Picasso’s Rose Period, not from the cubist one, as was the one you linked. It’s easy to escape from reality, when you already lost contact with it. Or maybe I fail to see the sublime message he transmitted by not drawing that guy’s leg, or the other elementary mistakes in that picture, and you really see something great in these. On the other hand, at others like Dali, such escapes from reality are done at purpose and suggest something, and you can’t mistake them for childish mistakes.

        • nad says:

          Cristi you seem to have a very good knowledge about Picassos paintings. I do not quite agree with you, but I am currently discussing some of your interpretations with my husband.

  10. jessemckeown says:

    To add some noise to the comments, Glen Gould reported that one of his favourite times to practise, when he was younger, was while somone in the house was vaccuuming the carpets, that he could hear the music better when the sound of his performance didn’t get in the way.

  11. Todd Trimble says:

    To add yet more noise: I confess that I really enjoy watching musicians, in addition to listening. It doesn’t have to be the triumphant flourish (as in the photo at the top of the post, from the looks of it the crowning moment of the performance) or otherwise anything overtly emotional, but sometimes one can read in the lines of the face (I have Michelangeli in my mind’s eye) the intense absorption and concentration of an artist, and I really dig that.

  12. David Lyon says:

    So the substitution of a cute lip syncing girl for a homely singer during the 2008 Olmypics opening ceremony really did improve the performance! The Chinese government was five years ahead of its critics.

  13. Paolo Carlini says:

    Admittedly, I have yet to read the article, but something that annoys me a lot these days is that many studies pretending to have as a topic “music” or “art”, actually are reporting findings about “a-specific-kind-of-music” or “a-specific-kind-of-art”. For example, I like a lot acousmatic music (http://en.wikipedia.org/wiki/Acousmatic_music) and, well, I can’t turn off the sound ;)

You can use Markdown or HTML in your comments. You can also use LaTeX, like this: $latex E = m c^2 $. The word 'latex' comes right after the first dollar sign, with a space after it.

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s