Thread: Brilliant Art Dudley's article

No doubt many people here have very strong feelings and opinions on the usefulness and application of blind listening testing when reviewing audio components. Let me say this first. If you enjoy doing double blind tests, sit back and knock yourself out. (does anyone know anybody who has purchased their system based on blind testing?) But be aware of and research the underlying science rather than relying on someone else’s opinions in a summary article. Blind testing has always been a paradoxical situation, however, with backers calling it “science- or evidence-based ;” but truthfully and at times willfully, however, proponents of blind testing ignore the real science, biology, physiology, neuroscience, testing science, etc. that’s been in the literature for decades. In short, blind testing is simply telling us what we already know: short term memory (auditory) is highly unreliable (just ask the police at a crime site or do that old test of passing a message from the front to back of a line.) Now let’s look at the real science:

1) Inter-aural hearing differences: Researchers have for years shown that inter-aural hearing differences among the population swamps out any effort to statistically validate any listening test (a very good review piece on the subject appeared several years ago in Scientific American Mind.)

2) Perception and the inverse U curve: Perception goes down as the magnitude of the test goes up. In other words, a musician wants to be at the far left of the inverse U curve being able to perceive as much information as possible while performing; on the other hand, a weightlifter with only one task needs minimal perceptive properties to concentrate on one thing. Or take blitzing that NFL QB. One is cutting down the time available for decision making making the athlete more prone to making a mistake. Simply, you are reducing his perceptual abilities. And the simple act of taking a “test” reduces one perceptual ability. Perhaps, a blind test could have more validity if 1) the tester asked his test group to concentrate on one particular attribute or 2) chose specific (rather than familiar or random) types of music to test the DUT.

3) Adaptation: Long known about and originally written about by the late and renowned Canadian physiologist Hans Selye. In short, every stressor, be it weight training, alcohol, heat, cold, etc. has a specific and long term effect on the body (eg. a genetic change). In Selye’s model, there is an initial alarm phase followed by roughly a six week period of adaption (or in the case of say weightlifting, getting stronger). The same is true for that new component; it takes time for the brain/nervous system to adapt to this new stressor/music.

4) Short term memory and information processing: Short term memory is notoriously unreliable because it has both limited “hard disc space” and involves serial rather than parallel processing (not to mention we still don’t understand to this day how and/or why short term memories (or unconscious activity) are converted into long term memories). This is always a good test for effects of short term memory and its clogging points. Watch a basketball player make an initial fake and then follow that up within 100 ms. with a second move. If executed properly, the defender’s brain will “freeze” because the offensive player has overwhelmed the short term memory disc space and processing ability.

5) Neuroscientists constantly refer to the dual nature of the human brain and in particular the primitive, limbic system vs the newer, evolutionarily wise, frontal cortex. (See the book written by Jonathan Haider entitled The Happiness Hypothesis. The bottom line: the older limbic system controls our very powerful flight or fight response. Ergo, what our brain searches for initially when we hear music played through new gear is more related to the clues our brain needs for deciphering if something dangerous is present eg. detail at certain frequencies. It’s my experience in fact, that those components—or even parts like capacitors-- that initially sound good, don’t hold up with long term listening. And conversely, I often find those components that produce a ho-hum response with early listening often with extended listening sound much more like music.


I’m going to leave treatment of stats and methodology choices to another time.

Interesting Myles. Thanks for showing some interesting arguments against the benefits of blind testing.

I enjoy it for fun, and as you said, would likely never trust it solely for buying a piece of gear. Too many factors. But the tests are fun.


Sent from my iPhone using Tapatalk

Originally Posted by Mike

Interesting Myles. Thanks for showing some interesting arguments against the benefits of blind testing.

I enjoy it for fun, and as you said, would likely never trust it solely for buying a piece of gear. Too many factors. But the tests are fun.


Sent from my iPhone using Tapatalk

One can obviously make arguments against each form of testing. I think, however, that the benefits of careful, long term, well thought out listening sessions outweighs any arguments against prejudice.

I don't know if you read Art Dudley's article, Myles, but none ​of the effects and observations you posted were mentioned. And as I posted earlier, there were more problems with the article than just his comments on blind testing.

Originally Posted by rbbert

I don't know if you read Art Dudley's article, Myles, but none ​of the effects and observations you posted were mentioned. And as I posted earlier, there were more problems with the article than just his comments on blind testing.

Yes the above was simply my viewpoint on the subject from a scientific/audiophile perspective. I think that sometimes one has to reconcile my hobby with my background. (I hate conflict.) Or call it a brain dump from the rationalizing part of the brain. Whatever, hope it is food for thought on the matter.

Originally Posted by Myles B. Astor

Yes the above was simply my viewpoint on the subject from a scientific/audiophile perspective. I think that sometimes one has to reconcile my hobby with my background. (I hate conflict.) Or call it a brain dump from the rationalizing part of the brain. Whatever, hope it is food for thought on the matter.

And here's the thing; you put forth some reasonable and well-thought out arguments against rapid-switching short term blind testing. Art merely wrote some facile and superficial comments about blind testing before going further out into the objectivist vs. subjectivist debate, somewhat mischaracterizing both sides of that as well. And that's my problem with this month's column and AD's columns in general for the past year or so. Where once (back in the '90's, say) Art wrote well about audio topics to which he had apparently given significant thought, today he is writing about audio topics to which he has apparently given little thought. Either that or he is expressing himself poorly, in which case it's not appropriate for him to still be writing professionally.

More specifically addressing your points, though, I don't think you have presented good arguments about the potential usefulness of longer term blind listening. Logistically a much more difficult problem, certainly, but not insurmountable at least for some component types.

Originally Posted by AJ Soundfield

Hi Jerome,

As has been pointed out previously, this is completely incorrect. There is no time limit for a blind test whatsoever. It can last two years if you wish. The only thing "blind" means, is that you don't know what's playing. That's it.


I don't quite follow you here. If that is true (and it is), how are adding even more variables/biases in (sight, volume changes, reputation, etc, etc.) on top of mood/perception changes, going to help or make it less biased?


That can be done blind. Unfortunately, it will only tell you one thing. How it sounds to your ears.
As I noted previously, there is far more to component choice/enjoyment/ownership than that.
That applies to us all, hence the huge variety of choices we make.

cheers,

AJ

So you could do blind tests over periods of weeks ? You could listen to A during 2 weeks and then switch to B for two more weeks, without knowing at any time what are A and B ??? How come ?
With a curtain in front of the system and someone always here to turn it on ?

Of course we are talking about best sound to one's ears. There is no absolute in audio. Only personal preferences.


Envoyé de mon iPhone à l'aide de Tapatalk

Thanks for your detailed comments, Myles.

I would agree with you that things that make the most difference right away are often the components I sell the first. Go figure! I'm now very suspicious of such components and give them a week or two before making any conclusions. But I also don't think that 600 hours is required to hear a component in its glory- at that point your mind has very much "adapted" to the component as you say.

Originally Posted by Jerome W

So you could do blind tests over periods of weeks ? You could listen to A during 2 weeks and then switch to B for two more weeks, without knowing at any time what are A and B?

Sure. Or longer. As long as you don't know which is which throughout the listening, it's blind.

Originally Posted by Jerome W

How come ? With a curtain in front of the system and someone always here to turn it on ?

That's one possibility, but not always necessary. "Blind" doesn't necessarily mean you can't see, it means know. Could be as simple as hooking both to a high quality pre with Input 1 and Input2. You could have the whole thing in full view. Just no peeking at the cabling to see which goes to which, or any "pop" sound or flashing lights depending on which was switched to, etc, i.e. no "tells" to identify which was playing.

Originally Posted by Jerome W

Of course we are talking about best sound to one's ears.

Yes, precisely. That is by definition, what a blind audio test is. Sound>ears. No looks, no price, not what you read someone else say, etc, etc. Just sound>ears. That's all you have to judge with. Can be quite an eye opener! Or a can opener.

Originally Posted by Jerome W

There is no absolute in audio. Only personal preferences.

Bingo. That's why we all own different stuff. I don't listen for pleasure at home, blind. I allow all my biases and senses to be the final arbiter of what pleases me. Be that price, looks, status, reviews, etc, etc., along with the sound>ears.
I suspect you do too.

cheers,

AJ

I have an inherent bias in favor of DBT to detect differences, although not preferences. Despite that, it's just too difficult to design and perform a good (read: adequate) DBT for most types of audio equipment, so although I have that bias intellectually in practical terms it's almost never achievable. I might just point out here that this difficulty (designing and performing an adequate DBT) is a BIG problem today in all branches of science, unfortunately. It's the main reason you might read about "breakthroughs" one day that are either flat out contradicted or more often appropriately minmized by practical experience and later better designed scientific tests.

As far as short-term vs. long-term memory goes, there's some facinating research that suggests that we "re-write" our long-term memory each time we use it. This helps explain why some memories seem clear and sharp but are wrong!

That is in fact conventional wisdom re: long-term memory, but there's a fair amount of recent research that suggests just what I posted. However, that doesn't seem to always happen, and no one (AFAIK) knows why. I haven't tried to find it online but that's probably doable.

"By conducting carefully controlled blind speaker tests, then measuring all the speakers to see what sonic characteristics appealed to the listeners in the tests, we were able to figure out what matters and what doesn't." - Paul Barton, Founder and Chief Designer



Sent from my iPad using Tapatalk HD

Originally Posted by Mike

"By conducting carefully controlled blind speaker tests, then measuring all the speakers to see what sonic characteristics appealed to the listeners in the tests, we were able to figure out what matters and what doesn't." - Paul Barton, Founder and Chief Designer

More feasible for a manufacturer, but even so I bet they are only listening to (testing) one speaker at a time, not a stereo pair, which does qualify their findings somewhat. As has been discussed elsewhere, dipoles are at a significant disadvantage in this setting.

I think blind tests are more entertainment and fun than pure science, but Paul Barton has tremendous respect in the industry and I found his use of blind tests interesting.


Sent from my iPad using Tapatalk HD

Welcome Tom!


Sent from my iPad using Tapatalk HD

My favorite audio blind test
More
and auditions
Those seem to work just fine

Originally Posted by rbbert

That is in fact conventional wisdom re: long-term memory, but there's a fair amount of recent research that suggests just what I posted. However, that doesn't seem to always happen, and no one (AFAIK) knows why. I haven't tried to find it online but that's probably doable.

The only thing you can say about neuroscience is that there's a new paper published every minute. What was dogma ten years ago is discarded history today.

Rewritten is but one concern. Processing and even the use of primary vs. secondary neuronal circuitry is worthy of consideration. The Brain That Changes Itself is an interesting read and shows how the idea of select areas of the brain carrying out specific tasks is a myth. As is the old right vs. left brain stuff that continues to be promulgated even today.

Originally Posted by AJ Soundfield

Sure. Or longer. As long as you don't know which is which throughout the listening, it's blind.


That's one possibility, but not always necessary. "Blind" doesn't necessarily mean you can't see, it means know. Could be as simple as hooking both to a high quality pre with Input 1 and Input2. You could have the whole thing in full view. Just no peeking at the cabling to see which goes to which, or any "pop" sound or flashing lights depending on which was switched to, etc, i.e. no "tells" to identify which was playing.


Yes, precisely. That is by definition, what a blind audio test is. Sound>ears. No looks, no price, not what you read someone else say, etc, etc. Just sound>ears. That's all you have to judge with. Can be quite an eye opener! Or a can opener.


Bingo. That's why we all own different stuff. I don't listen for pleasure at home, blind. I allow all my biases and senses to be the final arbiter of what pleases me. Be that price, looks, status, reviews, etc, etc., along with the sound>ears.
I suspect you do too.

cheers,

AJ

Sorry AJ.
You could do that for Dacs but not for tube amps. They should not be on while not connected to a load and you won't let them on all the time.
Really not practical.


Envoyé de mon iPhone à l'aide de Tapatalk

Just read the article. Basically, all he is saying is if you do not think cables, or whatever, make a difference then fine. Just stop your screaming that people who think they make a difference are mentally weak, and are being fooled by charlatans.

Also, John Atkinson's negative review of the $33k Nola Metro Grand Reference speaker should put a stake in the heart of those whiners who say reviews are bogus, biased, and always positive.

+1

I can weed out a bunch of equipment after listening to the first song, but the true test for me takes a few weeks.

Originally Posted by BruceLet

Speaking strictly for myself, I find short term analysis of a piece of equipment or a system is almost useless to me….whether I like it or not and what is better, because of certain qualities that get my attention, does not give me the info I need. Unfortunately, I don't have faith in my ability to demo something and know what it will offer to my long term listening enjoyment. Many of the things that jump out in audiophile terms are impressive enough, but don't necessarily provide enjoyment later on. In fact, I find some of them become tiresome and distracting, demanding more attention than the music. The test for me often happens too late. A couple of weeks later, sitting in my listening room….if I am not turning down the volume or leaving the room after an hour or so, or playing the same fifteen albums over and over and over again, then I know I have a keeper.

Originally Posted by BruceLet

Speaking strictly for myself, I find short term analysis of a piece of equipment or a system is almost useless to me….whether I like it or not and what is better, because of certain qualities that get my attention, does not give me the info I need. Unfortunately, I don't have faith in my ability to demo something and know what it will offer to my long term listening enjoyment. Many of the things that jump out in audiophile terms are impressive enough, but don't necessarily provide enjoyment later on. In fact, I find some of them become tiresome and distracting, demanding more attention than the music. The test for me often happens too late. A couple of weeks later, sitting in my listening room….if I am not turning down the volume or leaving the room after an hour or so, or playing the same fifteen albums over and over and over again, then I know I have a keeper.

Says the man whose speakers were designed with the help (significant help) of blind testing

Originally Posted by Jerome W

Sorry AJ.
You could do that for Dacs but not for tube amps. They should not be on while not connected to a load and you won't let them on all the time.
Really not practical.


Envoyé de mon iPhone à l'aide de Tapatalk

A load resistor really isn't that difficult to hook up interim.

Originally Posted by rbbert

Says the man whose speakers were designed with the help (significant help) of blind testing

+1.

Originally Posted by mep

So true. Harman is one of the few places where actual blind testing takes place after the participants have been sheep-dipped by Harman in what to listen for in their speakers vice everyone else's speakers.

I'm not quite sure that's fair to say, although I have never participated. Have you?

Originally Posted by mep

So true. Harman is one of the few places where actual blind testing takes place after the participants have been sheep-dipped by Harman in what to listen for in their speakers vice everyone else's speakers.

Another great MEP perspective, in other words , he doesn't have a clue.

Originally Posted by mep

Nope. Just read about it. I think you would have to be extremely naive to think that when Harman is 'training' their listeners that they aren't training people on what to listen for with regards to how Harman designs speakers vs. how other manufacturers are designing their speakers.

Mark I don't think you've thought this through. Harman is looking to design speakers that will sell; to that end their research is much more valuable to it if it doesn't involve any value judgements or prejudices on their part. Listeners just telling Harmon what it "wants to hear" doesn't sell any product.

I

Originally Posted by puroagave

are you suggesting blind listening tests are useless the way harman does it? kevin voecks has been head of speaker engineering for Revel since day 1 and works closely with dr floyd toole, and has been since his days with Snell. they used the facilities at the NRC way before the competition were.

kevin is interviewed here, most of it is speaker design 101 but interesting nonetheless.

https://www.youtube.com/watch?v=g4lA0cO87OE

Yes have you actually read the testing methodology? 2% science, 98% ad copy. In fact, it's a joke.

Originally Posted by puroagave

whats the link to the paper you're referring to?

It was all posted and talked about on WBF. I don't remember the threads exactly but try Sean Olive's forum.

But seriously, no one has read how they [Harman] carry out their speaker tests or seen or even tried the downloadable software to carry out their testing but are criticizing Mark?

Originally Posted by rbbert

Mark I don't think you've thought this through. Harman is looking to design speakers that will sell; to that end their research is much more valuable to it if it doesn't involve any value judgements or prejudices on their part. Listeners just telling Harmon what it "wants to hear" doesn't sell any product.

Except when they use their testing methodology to trash every other speaker but theirs. Again, their testing of the Martin Logan speaker is funnier than most of the skits on SNL for the last twenty years.

Originally Posted by MDP

Another great MEP perspective, in other words , he doesn't have a clue.

So you've read how they do the testing Mark? Tell all of us how Harman does their training and what parameter their use to evaluate all the speakers in their test. While you're at it Mark, tell us how you scored on their downloadable training test. You do realize it's basically one parameter. And one of something else.

Originally Posted by mep

Nope. Just read about it. I think you would have to be extremely naive to think that when Harman is 'training' their listeners that they aren't training people on what to listen for with regards to how Harman designs speakers vs. how other manufacturers are designing their speakers.

Exactly.

Almost everything Harman applies to it's speaker design can be traced back to Toole/Olives work at the NRC before they got to Harman. Smooth on and off axis, free of resonances, especially low Q ones. That's why you see similar principles in most of the NRC offshoots like Paradigm, PSB and (now defunct) Mirage, Energy, etc.
But it's not just the NRC inspired manufacturers. Here is an example from Soundstage Networks NRC tests, for the Magico S5:


Clearly there are smaller manufacturers (myself included) that adhere to those principles....and listeners who like the results.
However, none of this is an indictment of blind/controlled testing, which are the defacto standard of science, whether it be audio/perceptual related science, or the statistics of particle physics.
Perhaps the biggest irony of the Dudley article, is that the founder of Stereophile, J Gordon Holt, had this to say about blind tests

As far as the real world is concerned, high-end audio lost its credibility during the 1980s, when it flatly refused to submit to the kind of basic honesty controls (double-blind testing, for example) that had legitimized every other serious scientific endeavor since Pascal. [This refusal] is a source of endless derisive amusement among rational people and of perpetual embarrassment for me, because I am associated by so many people with the mess my disciples made of spreading my gospel. For the record: I never, ever claimed that measurements don't matter. What I said (and very often, at that) was, they don't always tell the whole story. Not quite the same thing. Remember those loudspeaker shoot-outs we used to have during our annual writer gatherings in Santa Fe? The frequent occasions when various reviewers would repeatedly choose the same loudspeaker as their favorite (or least-favorite) model? That was all the proof needed that [blind] testing does work, aside from the fact that it's (still) the only honest kind. It also suggested that simple ear training, with DBT confirmation, could have built the kind of listening confidence among talented reviewers that might have made a world of difference in the outcome of high-end audio.

Obviously the direction of the magazine has changed since his departure, perhaps to better suit it's readers who seem to loath the defacto standard of perceptual science - blind tests. Which is fine, no one is forcing anyone to change their high end beliefs. No blind test police..yet.
But outside the high end and in the rest of the world, including as I noted, even selecting orchestra performers, blind tests are used. If you want to remove biases.
If not, enjoy anyway, it's all preferences.

cheers,

AJ

Great post AJ !

Originally Posted by Myles B. Astor

It was all posted and talked about on WBF. I don't remember the threads exactly but try Sean Olive's forum.
But seriously, no one has read how they [Harman] carry out their speaker tests or seen or even tried the downloadable software to carry out their testing but are criticizing Mark?

I believe the founder of WBF Amir M, has personally participated in the Harman speaker tests and is a proponent of their methods.

cheers,

AJ

Originally Posted by puroagave

i'm not after Mark, to the contrary, if someone demonstrates superior knowledge on a subject i'm all ears. I didn't see Sean's thread on WBF i'm referring to the snippet i read here: Revel Salon loudspeaker A Visit, Continued 2 | Stereophile.com

I'm a fan of Kevin's work, if pseudo science is in their ad copy so be it, and if he works at a co. that once bashed Martin logan whats new? they all bash each other brazenly or subtle jabs here and there. ML's own promo videos will have you think all moving coil speakers are inherently defective.

Rob, you weren't the one bashing Mark in the thread.

What we are talking about has nothing to do with Kevin's designs (he and I have been friends since his Snell days) but more to do with the arrogance of one company, the owners business plan and their true motivations.

And sorry but I do not agree with your statement about high-end companies commonly bashing
each other. Yes there are a handful of individuals but I've never heard 99% bash anyone. I've never heard ARC, cj, Magnepan, VPI, Lyra, Rockport, VTL, ML, D'Agostino, Spiral Groove, Rowland, Transparent Audio, Magico, etc, in 35 years ever knock a competitor. No, they are quite happy to let their products rest on their own laurels and let the chips rest where they may. And I do not consider saying that you believe your design is better is bad mouthing a competitor.

Originally Posted by AJ Soundfield

I believe the founder of WBF Amir M, has personally participated in the Harman speaker tests and is a proponent of their methods.

cheers,

AJ

Yes AJ I assume Amir brought Sean to WBF and he is also a Harman dealer. I don't remember if Amir directly or indirectly participated.

Originally Posted by AJ Soundfield

Almost everything Harman applies to it's speaker design can be traced back to Toole/Olives work at the NRC before they got to Harman. Smooth on and off axis, free of resonances, especially low Q ones. That's why you see similar principles in most of the NRC offshoots like Paradigm, PSB and (now defunct) Mirage, Energy, etc.
But it's not just the NRC inspired manufacturers. Here is an example from Soundstage Networks NRC tests, for the Magico S5:


Clearly there are smaller manufacturers (myself included) that adhere to those principles....and listeners who like the results.
However, none of this is an indictment of blind/controlled testing, which are the defacto standard of science, whether it be audio/perceptual related science, or the statistics of particle physics.
Perhaps the biggest irony of the Dudley article, is that the founder of Stereophile, J Gordon Holt, had this to say about blind tests

Obviously the direction of the magazine has changed since his departure, perhaps to better suit it's readers who seem to loath the defacto standard of perceptual science - blind tests. Which is fine, no one is forcing anyone to change their high end beliefs. No blind test police..yet.
But outside the high end and in the rest of the world, including as I noted, even selecting orchestra performers, blind tests are used. If you want to remove biases.
If not, enjoy anyway, it's all preferences.

cheers,

AJ

I would aver that Harman, like many companies back then, went to people doing basic research and told them to make money off their basic research. Yes basic research has changed since I began in 1976; otherwise, why do you think journals make authors sign releases about commercial interests before publication?

You can also read more about JGH's thoughts on the matter here:


http://www.herronaudio.com/images/Measurements.pdf

mep Mark, can you explain to me how having listeners (in their listening tests) prefer Harmon designs either advances Harmon's research or leads to increased sales? Because I can see no advantage whatsoever to Harmon by just having a few listeners say they prefer Harmon designs to others.

Originally Posted by mep

True, but my father-in-law drank instant coffee and swore it tasted just as good as coffee made from fresh ground high-quality beans.

I didn't anticipate some people getting so lathered up over Harman, but everybody needs to take a step back and breathe and relax a little. Harman designs their speakers to measure a certain way which they obviously think is superior to some other brands they want to use in their demos. Harman trains people on what to listen for and then conducts their tests with speakers behind the great Oz curtain moving into place via an automated platform if memory serves me correctly. After people have received the Harman training which helps people understand how Harman designs speakers and what the trainees need to listen for, Harman speakers always rank high in the blind test results. Does this really surprise anyone?? And now, people are taking pot shots at me for stating the obvious?

The ironic thing is that this thread was started to take pot shots at double blind testing and not by me. I earlier stated the obvious that the people who claim they love them some blind testing really never participate in it and don't make any of their purchasing decisions based on it. It's just a fact. That doesn't stop the proponents of double blind testing from beating everyone over the head with their cherished ideals even when they never practice what they preach. I also mentioned that the only real double blind testing in audio that I know about is conducted by Harman.

So, if you are a firm believer in double blind testing and you love how Harman conducts their speaker tests, shouldn't all of you own Harman speakers? You should all really leave your choice of speakers up to the trained listening experts and buy what they say is best because they figured that out for you using rigorous scientific methods. And if you do buy a Harman speaker and you decide you really don't like the way it sounds, you can take solace in the fact that you are not a Harman trained listener and you just don't know how to really listen and you should be enjoying the speaker.

Paragraph 2 essentially sums up Art's point and the subject of this thread.