Blind Test

Kingrex

Active member
Joined
Dec 17, 2017
Messages
1,187
Location
Seattle Washington USA
Many times we here people touting the line perform a blind test or A / B test. I think it's time we have a discussion of what this really means. More so I believe we need some insight into what constitutes an accurate test. Not just accurate but answering the base question. My understanding is a test can be used as a method to deceive not seek an answer. Not necessarily by dishonest intention, just a false process implemented by an unknowing party. How about we get some contribution to help clarify how to conduct a true blind test. It could be usefully to understand when trying to decide on a cable or component and how it may stack up to the competition.
 
Great thread.

I conducted a blind test with several people about 4 years ago. The process was that each would select one song. They wore sleeping masks which I bought from Walmart. This was an amp shootout. The order of amps was only known to me, but it was written on a piece of paper and taped to the back of the speakers.

It wasn’t meant to be scientific, but more fun. We had a few guys who swore they could tell the solid state amps from the tube amps (they were dead wrong by the way).

The funniest part of the whole test was the guy who brought his amp and ended up picking it dead last. When he had the blindfold on, he called his own amp “like nails on a chalkboard.” It was hard for me not to laugh out loud.
 
So all you were trying to do was have people pick what sounded best. I guess just listening is a way. Was the music familiar to everyone. What was the front end such as preamp and source.
 
My understanding is a test can be used as a method to deceive not seek an answer. Not necessarily by dishonest intention, just a false process implemented by an unknowing party.
Tests can be done for lots of reasons. What is to be tested should be clearly spelled out. Of course they can be done for deceptive reasons etc. Or they can be done to answer legitimate questions.
You wanted to know how I did mine. First, you need to be aware that there are many people who will avoid trust ears tests at all costs. On top of that, even if they do a test, they will vehemently deny the results and also start to spin the wheel of excuses. One of the excuses on the wheel is "stress". That they were temporarily deafened by the "stress" of listening to there favorite music in stereo.
One way to address the "stress" excuse, is to make them unaware they are being blind tested. If they are unaware, there is no way "stress" can be an excuse.
One would also not want to reveal how such a test is done if it were to have any future viability.:)

How about we get some contribution to help clarify how to conduct a true blind test. It could be usefully to understand when trying to decide on a cable or component and how it may stack up to the competition.
First, is to understand that "blind" does not mean wearing blindfolds like in Mikes test.:)
Blind, means "not knowing" https://en.wikipedia.org/wiki/Blinded_experiment certain information that can affect, in this case, perception of the DUT (device under test).
One can do a "blind" test totally sighted. Mine were. There are also levels of "blindness", such as single and double, where the experimenter is also "blind".
One critical aspect of a blind audio test, is levels must be fairly precisely matched. Easy to do with electronics, cables, etc. A royal PITA with loudspeakers, turntables, etc.
Then there is what you're trying to test. If it is sound>ears, then all of the above. There are fun casual ways to do it and dead serious stuff like AES, drug research, etc.
But as I explained in my previous post to you, if the sole intent is to find what pleases you most, at home, in full view of your system (sometimes a large audiophile shrine of shiny equipment :P), listening purely for pleasure.....then no blind test is required whatsoever.
Simply pick what looks, sounds, feels, pleases, etc, etc."best"...to you.

Mikes example above was the fun variety, I've done plenty of those, all participants knowing and enjoying. A quick follow up - the gent who picked his own amp as worst, using sound>ears only, still uses the exactly type of amp in his system at home and swears it is still better sounding than the ones he picked sound>ears. It had zero effect on his beliefs.
The only thing that should matter, is what pleases all your senses most. that will give the most satisfaction, even if short term.

cheers,

AJ
 
I think that most of your audiophile misconceptions were created by people that had a vested interest in blind tests not working well.

* * * * * * * *
If it's not ears only, then it's not a listening test.

* * * * * * * *
Blindfolds arn't needed (except for loudspeaker tests). Just block the equipment from view.
 
There was a blind test conducted about a month ago with wall outlets. I was not able to attend because I was traveling. But my friend Bart had rigged up a gizmo to test all the wall outlets from all major manufacturers. Nobody wore a blindfold, but nobody could see what outlets were being used. What was really interesting were the results. I’ll try to find his report and post it.


Sent from my iPhone using Tapatalk
 
I think that most of your audiophile misconceptions were created by people that had a vested interest in blind tests not working well.

* * * * * * * *
If it's not ears only, then it's not a listening test.

* * * * * * * *
Blindfolds arn't needed (except for loudspeaker tests). Just block the equipment from view.

Blindfolds are often needed. When we were doing the 6 amp shootout, there was just no way for them not to know what amp was on, connected, meters moving, etc. If someone even gets a sense of what’s playing, the test is ruined.

Yes, the guy that picked his amp last still has them. He’s blinded by love. [emoji6]

I have done several blind Aurender N100H Vs N10 tests with people in the store for folks who couldn’t believe they would hear a difference. They always did and they always chose the N10.

Oh, I did have a customer and his wife wear blindfolds early last year while I swapped out DAC’s using the same amp/preamp/Speakers they had at home. That was fun.


Sent from my iPhone using Tapatalk
 
Btw, another interesting tidbit was very clear differences were heard in Mikes blind amp test. Everyone didn't suddenly go deaf and the blinding didn't mysteriously "mask" the sound differences. Food for thought.
As someone not afraid of and having participated in many blind tests, including for training purposes, I can say that I find it easier to tell differences in a blind controlled test.
I was developing a pair of monitors as a replacement for an existing discontinued model. Swore I had got them sounding just like before, listening to both pairs casually, in stereo. Then I did a much maligned Harman style mono test with the speakers in the same spot, level matched as closely as possible and rapid A/B switch. Holy %**^. At first I though something was broken, spent the next several hours rechecking wiring, measuring, rechecking. Obviously something had gone amiss...except it hadn't.
That type of test was simply ruthlessly revealing. Back to the drawing board lol.
Btw, Harmans loudspeaker tests are NOT ABX, which is used mainly for JND or Just Noticeable Difference, i.e, finding if there is any difference whatsoever. Their tests are A/B preference tests. I.e., there are clear differences and they want to find what folks prefer...not whether there is a difference or not.
 
As store owners I get what your doing. Set up a system and hope people like the voicing. I have a local stores that put up the Wilson/Dagostino setup. It sounds horrible to me. One day they had the AR Ref on Mag 20.7. Pure magic to me. Then they put the Dagostino on the Mag. I walked out of the room. Blind test are not always needed.

Lets say you carry a couple lines of amps. A customer has been a tube guy for a while and thinking of SS now. Class D or A ????? How can an A /B be done for the customer. Sure you can set one up one day and take a listen, then a week later play the other. What do you like? If they are trying to understand the subtle differences this does not expose them. It may just reinforce preconceptions.

How about cables in your home. How do you test a demo product.
 
What was it you didn’t like about the D’Agostino as that seems to be the common denominator in your post above.

When A/B’ing amps, I always try to do it on the same day for the customer. Spreading it over several days or weeks is not ideal.


Sent from my iPhone using Tapatalk
 
Blind Test for AC Outlets:

5 AC Wall Outlets Compared:

Dedicated Line:
*Furutech GTX DR NCF Rhodium Plated MSRP: $280.00
*Maestro cruzeFirst MSRP $105.00
*MIT Z- Duplex Conditioning Outlet MSRP $149.00
*Hubble Under $10.00

Non-Dedicated Line:
*Standard builder grade

The System:

Powered by the receptacles:

EMM Labs P-2 Preamp
JenLabs modified Marantz SA7s1 (Sound characteristics very similar to Esoteric K01)

Balance of system:

Wireworld Silver 7 power cords & IC’s to amp
David Berning OTL Amplifier
Synergistic Research Resolution Reference X2 Speaker Cables
Avantgarde Trio Loudspeakers w/2 pair 225 subwoofers and one pair Velodyne DD15 subwoofers
Fully Treated Resolution Acoustics room

Methodology:

The four premium receptacles were wired in parallel and mounted to a board and wired into a dedicated line with Romex. This was located adjacent to the standard household receptacle for easy and rapid transfer of the plugs to the test receptacles. This was all aloud to cook for several days. All receptacles had burn in time prior to being mounted to the assembly.

The listening test would be a single blind test. Only I would know which receptacle would be tested. I had not conducted and comparative listening beforehand so I would not prejudice the tests. One receptacle was selected as a reference, which is that would be the first one listened to and would be listened to again after each receptacle was reviewed to reset to the baseline.

The checklist:

The listeners:

Myself and 3 members of both the Suncoast and Sarasota Audiophile Society members that have developed sound critical listening skills over the years and are all active in testing various tweaks.

The Music:

Take 5, Dave Brubeck Quartet
Everything Must Change, Randy Crawford
Dirty Work, Jenna Mammina
My Girl, Patricia Barber

If that’s isn’t enough to create a major disagreement, then let’s proceed to the results! There were some very interesting developments..

Furutech:

3 voted best overall
1 voted third best

MIT:
1 voted best overall (he has these in his full system – interesting)
2 voted third best

Hubble:

4 voted second best

The second test was this outlet powering a PS Audio Quintet conditioning powerstrip. With this came a new Audioquest Tornado power cord. The Furutech was a bit better than the Hubble. Both were fine and there was very little if any loss in their characteristics. The MIT was a disaster. It was muddy, slow and dull. It was not at all musical. This was the only conditioning receptacle and with this the circuit was getting double conditioning.

What we took away:

The lowly Hubble is a great value and would most likely be value added to any system. The Furutech is the top dog, but at that price it should be. The MIT is a good unit, but be very careful of double conditioning. You may do more damage than good.

Follow Up Observations:

After the test, I installed the Furutech and the Maestro side by side and ran both. After about 2 weeks I rechecked the two and found to my surprise that the Furutech was now sweeter on top with better high-end extension than the Maestro had been, which was the Maestro’s strong suit to begin with.

01700833df96f87def19a3fdb2c8979d.jpg



Sent from my iPhone using Tapatalk
 
Great thread.

I conducted a blind test with several people about 4 years ago. The process was that each would select one song. They wore sleeping masks which I bought from Walmart. This was an amp shootout. The order of amps was only known to me, but it was written on a piece of paper and taped to the back of the speakers.

It wasn’t meant to be scientific, but more fun. We had a few guys who swore they could tell the solid state amps from the tube amps (they were dead wrong by the way).

The funniest part of the whole test was the guy who brought his amp and ended up picking it dead last. When he had the blindfold on, he called his own amp “like nails on a chalkboard.” It was hard for me not to laugh out loud.


:roflmao:
 
One day they had the AR Ref on Mag 20.7. Pure magic to me. Then they put the Dagostino on the Mag. I walked out of the room. Blind test are not always needed.
Maybe. But wouldn't you be curious to see what happens when your sensory input was reduced to sound>ears?
That seems like a pretty good opportunity there, cover the equipment area, swap the amps as quickly as possible, maybe both connected to same preamp then swap the cables at speaker (I know, might not be possible with some tube amps).
See for yourself how much your total sensory input and stored memory are affecting things beyond sound>ears.
Might be exact same as you described above. Or not.
 
I have 10 Furutech NCF duplex outlets in my system. the outlets are only part of the equation. there is also the matching aluminum bracket, and the carbon fiber/NCF cover...….so the Furutech NCF has a much larger upside (at higher cost) for those wanting that. and agree they need breaking in, i'd say a week to 2 weeks. and then there are the Furutech NCF plugs......which are a similar step up. all my 12 Absolute Fidelity power cords + my 2 Evolution Acoustics power cords have those plugs (at both ends).

using 100% Furutech NCF does require that you like all the truth and nothing but the truth. if you have a need to temper any information, some try to mix in other brands.

I like accidental blind testing, but personally don't care to seek it out. YMMV.
 
I thought it was interesting on how a test can shape perception.
Non tests can shape perception also. That's why tests, specifically blind, were developed, starting over a century ago

Btw, your example of being able to instantly hear blatant differences between AR Ref and Dag at your dealer, seems to belie the position that all valid tests to flesh out differences need be in comforts of ones own home/system.
It appears to support Mikes test above, where if there are audible differences, you will hear them and another system/home/etc doesn't necessarily "mask" them. That's been my experience also. Real audio differences to my ears aren't constrained to my system. YMMV.
 
My point about in home is about the synergy and interactions of my equipment with the test equipment.
Yes, that's how I took it, but your dealer comment implied you easily hearing differences in amps can be done outside your system also.
 
Of course differences can be heard at stores and shows. Especially between AR and D'Agostino. Probably tougher between D'Agostino and PS Audio or some other high power SS amp.

I gather the quality of D'Agostino is good, but not my cup of tea. Others would find my system to full of air and fun. To much background noise. Not analytical enought. No bass thud. I like the natural, open more jazz venue live sound. That's me and I voiced my system that way.
 
Back
Top