All the rest is sound and fury - Part II

Last January I revealed the results of a recent French NCSR double-blind listening test of violins, which placed authentic Stradivariuses and Amati violins in stark competition with less-famous old violins and some new, competently-made violins. The results indicated that a panel of "discriminating" listeners were not only unable to pick out the hallowed, legendary old Italian greats but many of the expert respondents even chose the newer instruments as their favorites from the bunch. This happened after only being able to listen--and not see--the instruments as they were played.

     I'm taking the opportunity here to print two responses that I was privileged to receive, to the revelation from highly-regarded scientist-friends: the first from Tim White, my old acoustician/ guitarmaking partner from the late Journal of Guitar Acoustics and then a response to his response from a past student of mine, the neuroscientist/guitar maker from Netherlands, Pieter Voorn.

Tim responds to the revealing double-blind violin listening test by remarking:

What would put a wooden stake through the heart of this thing would be to make the study really scientific by increasing the population of testers from 17 to a statistically significant size, say at least 50 from my practical experience (no math whiz but having to sound like it from time to time).
       In scientific measuring, there is no such thing as 100% certainty, with the scale of truthiness in evaluating measured results ranging from bullshit to dead-nuts being described in terms of so many "sigmas". For example, in particle physics, a "5-sigma" level of certainty is required to qualify as an actual "discovery" (Ref. section 5.3 of this document) I am mathematically clueless as to how to calculate the sigma level of certainty achievable with 17 perhaps not so random testers, but it can be done. If someone wanted to make real scientific waves in this area, I think it would require first obtaining results that met an appropriate scientific standard of truthiness, expressed as a "p-value" or sigma-level of certainty, then seeking to have the results published in a refereed scientific journal such as JASA (Journal of the Acoustical Society of America). Tom Rossing is still kicking around at Stanford U, if the effort needed a properly qualified mentor.
     I admit to being a bit burned out on the whole instrument tone thing, but the last guitar I built with Michael [Millard of Froggy Bottom Guitars] while still having apprentice mind in mid 1975, before getting lost in acoustics, has a tone that raises goosebumps on those who play it, including me, giving me faith that there is something to this discussion of instrument tone. I gave it to my daughter Hattie, who has a lot of musician friends passing through, and they all have the same reaction. So there is some magic there, and it had nothing to do with studying acoustics. I have never matched it since, and I find the mystery somehow comforting.

Pieter Voorn responded to me:

It came as no surprise to see your blog comments on the Fritz paper about old and new violins. I loved it, your Sound and Fury blog. However, I think that the owners of >$200,000 Martins or million dollar Stradivarii aren’t shaken up in the least by this study. Rational thinking is in no way involved in this process of appreciation. As any self-appreciating marketing expert will tell you: we are selling an emotion, not a product.

I guess the research paper did not do much for settling the issue. Some of the comments on the NPR website are hilarious, but they also prove that belief is way more important than ratio.

I don’t quite agree with Tim White’s comment on the Fritz paper not being scientific because I cannot find anything wrong with the science of the study. True, the sample size is not impressive (21 subjects), but the authors are aware of this and they are careful not to over-interpret their findings. For instance, they focus in the discussion of their data on something called confidence limits. If you would repeatedly draw samples (i.e. groups of violin players) of this size (21) from the population of professional violin players and calculate the confidence limits for each sample, you would find that 95% of the intervals between the limits would contain the true (!) mean. In this case, the limits were 18% - 62% for choosing to take home an old violin after playing. That is a pretty broad range around the observed 38% of subjects inclined to take home an old violin. The authors consider the upper limit of 62% “not high”: that is, even if the true mean, the 100% certainty that Tim White mentions, would be 62%, this, in the authors’ point of view, would be disappointingly low for a population of professional players (and old violin owners!). One may agree or disagree with this conclusion but there is nothing wrong with the science. How high a percentage of musician friends of Tim’s daughter Hattie wanting to take home the White/Millard guitar does it take to call it a magical instrument? Tim says the outcome is 100% - “they all have the same reaction”- and Fritz et al. would agree. Now how about 62% of 21 friends? 

Other than that, I (and judging from their paper Fritz et al. too) agree with Tim White that for a real discovery one would need a much bigger sample size and dramatically improve the confidence interval. But, hey, this is the age of publish or perish and the paper was published in an excellent refereed journal ...