Thread: Why you can’t trust measurements

https://youtu.be/Fa1y9JRip68


Sent from my iPad using Tapatalk Pro

I pretty much refuse to watch YouTube videos about anything (with the occasional exception of DIY home maintenance), so I am not sure exactly what is said here. Regardless, measurements that are properly performed can tell you whether a product meets its specifications, and sometimes whether or not it is well engineered. There is no evidence that measurements can tell you how a product sounds, although sometimes a large measured frequency response anomaly is noticeable, but of course that is only one aspect of sound or sound quality.

After more than 50 years of listening to music through home stereos it has never been clear to me why anyone would even think measurements tell you how a piece of equipment sounds. Frequency response, distortion and S/N+D were often helpful in evaluating tape deck performance, especially cassette decks, but even there 2 decks that measured (essentially) the same could sound quite different.

The audio industry has moved on from single sine or swept sine measurements, which are still used by most reviewers that still publish measurements.

In the case of ASR, the use of a single measurement @ 1 kHz is laughable.

ASR has many followers that will argue our ears/brains cannot be trusted and measurements are the only way to shop. I'm struggling to understand why people that don't trust their ears would spend 5 figures on components and I still haven't gotten anyone to suggest a measurement that indicates resolution differences in products. It's fascinating the herd mentality on that site makes discussion seem like interrogation.

Originally Posted by steve59

ASR has many followers that will argue our ears/brains cannot be trusted and measurements are the only way to shop. I'm struggling to understand why people that don't trust their ears would spend 5 figures on components and I still haven't gotten anyone to suggest a measurement that indicates resolution differences in products. It's fascinating the herd mentality on that site makes discussion seem like interrogation.

A widespread comment on many audio boards boils down to: what is the level of enjoyment of those who have systems purchased using measurements as the primary criterion vs. those who have chosen based on actual listening comparisons?

An even bigger question is: which measurements actually matter (e.g., once THD, IMD, S/N+D fall below audible levels) and which if any relate to the component's sound?

Originally Posted by steve59

I'm struggling to understand why people that don't trust their ears would spend 5 figures on components

Because they trust looks, price, weight, country of origin, reviews, awards, etc, etc.
I highly doubt a single person spending 5 figures on components advocates for controlled/blind listening, which by definition, is what "trust ears" is. It's the only option in controlled listening.
Now in completely uncontrolled "listening", you get to trust all ones senses, priori knowledge, biases, preferences, etc, etc. No need whatsoever to "trust ears" or "just listen". Sit back and enjoy it all.
As it should be.

cheers,

AJ

Originally Posted by AJ Soundfield

Because they trust looks, price, weight, country of origin, reviews, awards, etc, etc.
I highly doubt a single person spending 5 figures on components advocates for controlled/blind listening, which by definition, is what "trust ears" is. It's the only option in controlled listening.
Now in completely uncontrolled "listening", you get to trust all ones senses, priori knowledge, biases, preferences, etc, etc. No need whatsoever to "trust ears" or "just listen". Sit back and enjoy it all.
As it should be.

cheers,

AJ

While you selected a sentence of my post I don't think you got the intent of my post. your last line got it, sit back and enjoy.

Originally Posted by steve59

While you selected a sentence of my post I don't think you got the intent of my post. your last line got it, sit back and enjoy.

I 100% got your post as I've seen countless identical ones before and my response is always the same, as above. Yep, agree, sit back and enjoy all your senses, pricetag, etc, as we all do.
Only time I need to "trust my ears" is when doing blind testing during speaker development, as I do vs KEF, Revels et al. Otherwise, I "listen" just like you and everyone else. For enjoyment. No "trust" needed.

cheers,

AJ

I'd be ok with not blind testing knives

Originally Posted by Speedskater

I count 6 to 8 different tests at ASR.

Zero listening. And that SINAD worship...

The way your post is worded makes it sound as if the distortion signature’s sonic characteristics are established fact, but is that really so or are these just empirical observations? Likewise the comment about -105 dB distortion being inaudible (due to masking?) but some think higher levels are still inaudible? I’m not necessarily doubting you, but empirical observations are not “fact” (evidence), although they are often regarded as such in engineering.

Originally Posted by Speedskater

Originally Posted by Speedskater

I count 6 to 8 different tests at ASR.

What kind of a measurement is listening?
Do you mean statistical A/BX tests?

Listening test isn't one of the 6-8.
ABX is one such variety for specific purposes. This is the same Amir who (now) brags about doing such tests while JJ was employed. The revisionist history version. Best be able to show what you're measuring is in any way relevant to audibility, or its just a fancy picture. Ask Toole, JJ and Olive. They post there too.

To add onto Ralph's post, in addition to the audible impact of low-order and high-order harmonics and distortion as a function of frequency, the amount of masking (how much you "need", as a function of frequency, etc.) has also been extensively researched since it plays such a critical part in audio compression algorithm design.

Originally Posted by atmasphere

We've known since at least the 1930s how the ear responds to the various harmonics (see Radiotron Designer's Handbook, 3rd edition). Its also well known that if the circuit has a lot of IMD it will sound fatiguing- IOW harsh and bright.

Its been known for a long time how the 2nd harmonic contributes to 'warmth' (this is the ear converting distortion to a tonality). That should not be controversial- nor should it be that the higher orders contribute to harshness and brightness. Its also easy to demonstrate with very simple test equipment that the ear is using the higher ordered harmonics to sense sound pressure.

The tricky bit is that the lower orders can mask the higher orders, but not surprisingly, this works the same way in electronics as it does with loudspeakers. But this explains how a tube amp can sound smoother than a solid state amp even though its making more higher ordered harmonic distortion. Its masked, so the amp sounds smooth. Tube amps (assuming proper design) without feedback tend sound even smoother as they lack a lot of the higher orders and have a more prodigious 2nd or 3rd harmonic.

People don't trust measurements for simple reasons. First, most of the important measurements are not made or not published. 2nd, even if they are you have to know what they are saying and many people do not.

I am skeptical of "evidence" gathered in the 1920's and 1930's about the audibility of the harmonic spectrum and distortion spectrum of amplifiers and how it might apply to present day audio electronics. Without knowing more I tend to assume (with all that implies) that much of this is empiric data that has not been subjected to appropriate testing to establish it as true evidence. That is not to say that it isn't useful in amplifier design, where the designer will have both measurements and listening tests to attain his design goals, but how it relates to the consumer's (listener's) ability to choose the "best" amplifier for him is unclear.

Rob - there are plenty of modern references on the audibility of linear and non-linear distortion. Just search on the topic, you’ll find a plethora of articles by the likes of Toole, Geddes, Lee, Villchur, and others. They all say pretty much the same thing: normal measurements of IMD, THD, and (gag a maggot) SINAD do not correlate to perceived measures of quality. When you add additional variables like frequency and amplitude and complex stimuli then things get more interesting, as Ralph has noted in his earlier posts.

A lot of work has been done on the mechanics of human hearing perception that explains why these classic measurements fall short. It’s fascinating stuff, psychoacoustics.

Originally Posted by W9TR

It’s fascinating stuff, psychoacoustics.

as are the number of audiophiles that poo-hoo it ........

Originally Posted by W9TR

Rob - there are plenty of modern references on the audibility of linear and non-linear distortion. Just search on the topic, you’ll find a plethora of articles by the likes of Toole, Geddes, Lee, Villchur, and others. They all say pretty much the same thing: normal measurements of IMD, THD, and (gag a maggot) SINAD do not correlate to perceived measures of quality. When you add additional variables like frequency and amplitude and complex stimuli then things get more interesting, as Ralph has noted in his earlier posts.

A lot of work has been done on the mechanics of human hearing perception that explains why these classic measurements fall short. It’s fascinating stuff, psychoacoustics.

I have read a great deal of this research (but not by any means all). As far as I can tell, almost all of these additional measurements overall add little to the traditional measurements you mentioned (except to reflect the designer's priorities and how well those have been achieved); all are audible above a certain threshold and in general designers are wise to keep these measured parameters below those thresholds. However, once that is achieved components still often sound different (even to, or perhaps especially to, the designer), so all these measurements still do not correlate with sound quality, as long as the parameters measure below those thresholds of audibility. And FWIW, some technically minded reviewers, even (yes) Amir, measure many of these extra "distortions", and when they are below the established levels of audibility the product is pronounced "good"; does that tell anyone how it sounds?

Originally Posted by atmasphere

That depends on the component in question!

If an amplifier or preamp, the distortion spectra at 1 Watt is good. Also distortion spectra a 6dB below full power. Also distortion vs frequency at 1 Watt and -6db of full power. If the distortion overall is -105dB or better then the music can mask the distortion and the amp or preamp will be pretty neutral. Many people think that the distortion can be higher and it will still be inaudible. IME, that isn't true.

Here are some things to help out:
1) the ear assigns a tonality to all forms of distortion. As an example the 'sonic signature' of any amplifier is actually its distortion signature.
2) the ear uses higher ordered harmonics to sense sound pressure and so is keenly sensitive to their presence; it has about a 130dB range! The higher ordered harmonics as distortion are sensed as 'bright and harsh'.
3) the lower ordered harmonics (2nd and 3rd) can mask the presence of the higher ordered harmonics if the amplitude of the former is sufficient.

If you know these things then the measurements (if they exist as I described) can tell you a lot!

If you see distortion increasing with frequency (above 2-3KHz) that's a sign that the circuit is losing feedback at those higher frequencies. This can happen if the circuit lacks the Gain Bandwidth Product to support the feedback. The increased distortion will be perceived as brightness and harshness. We've been hearing solid state amps with this problem since the inception of transistors. In addition, if the circuit lacks the feedback needed to really clean things up (+30dB at all frequencies) it will sound harsh and bright since otherwise the feedback will cause distortion of its own thru the process of bifurcation at the feedback node, which is non-linear.

Tubes get around this a bit by having a pronounced 2nd or 3rd harmonic which can mask the presence of the higher orders. In this way they can sound smooth in the mids and highs even though they have more higher ordered harmonics than most solid state amps.

Supporting +30dB of feedback is pretty hard and the semiconductors needed for an amp to do that didn't exist in the 70s or 80s and just barely sometime in the 1990s. Plus the will to deal with this problem for the most part seemed to be absent with many solid state designers. So we have a lot of solid state amps that are harsh and bright as a result.

With class D amps its possible to apply enough feedback so there are class D amps now that lack the harshness of traditional AB amps. Such amps are usually of the self oscillating variety; so much feedback is applied to them that the amp goes into oscillation as soon as its turned on. The oscillation is then used as the switching frequency, killing two birds with one stone. Even then, if the distortion spectra isn't right, the amp can exhibit amusical characteristics. But they can be seen if the amp is tested on the bench as I described.

Many thanks, Ralph.

You are one of few people who both acknowledge and understand the nature of human reaction to distortion. A failure of many folks at ASR is specifically that they don't acknowledge the effect of 2nd & 3rd order distortion.

Forgive me but I don't have your expertise and years of study and design experience, but my observation both of test results in the popular hi-fi press and listening to audiophiles, (most of them subjectivists), convinces me of the following:

• Higher order HD is irritating: harsh, grainy, strident, etc.
• 2nd and maybe less so 3rd order is not only benign but actually pleasant to most listeners. (Hence the preference of many for tube equipment.)
• 2nd/3rd order can MASK higher order and other distortion. (That's why many folks find a tube preamp works well ahead of a S/S power amp.)
• 2nd/3rd order mask distortions in up & downstream components
• 2nd/3rd order also mask brightness and distortions in recordings.

I do think that measurements ought include the harmonic spectra at different base frequencies as will as different powers levels, and probably also different speaker impedances.

Audiophiles, rather than rejecting measurement out of hand, should begin to understand how to interpret measurements and insist on better, more complete measurements.

Originally Posted by rbbert

I have read a great deal of this research (but not by any means all). As far as I can tell, almost all of these additional measurements overall add little to the traditional measurements you mentioned (except to reflect the designer's priorities and how well those have been achieved); all are audible above a certain threshold and in general designers are wise to keep these measured parameters below those thresholds. However, once that is achieved components still often sound different (even to, or perhaps especially to, the designer), so all these measurements still do not correlate with sound quality, as long as the parameters measure below those thresholds of audibility. And FWIW, some technically minded reviewers, even (yes) Amir, measure many of these extra "distortions", and when they are below the established levels of audibility the product is pronounced "good"; does that tell anyone how it sounds?

For my part, I don't entirely trust currently acknowledged "thresholds of audibility". That is, as I recall that some fairly high level of 2nd order distortion, maybe 2%, is what it takes to be heard. But I'm thinking that the level people need to hear as distortion as such is much higher than than what it takes to influence their subjective preferences.

Originally Posted by NekoAudio

To add onto Ralph's post, in addition to the audible impact of low-order and high-order harmonics and distortion as a function of frequency, the amount of masking (how much you "need", as a function of frequency, etc.) has also been extensively researched since it plays such a critical part in audio compression algorithm design.

That's interesting. And I think that lack of understanding of the masking effect 2nd/3rd order HD is a shortcoming of both tech-heads like Amir at ASR and subjectivist audiophile who scoff at measurements.

Measurements serve a useful purpose even when they don’t relate directly to how a piece of equipment “sounds”. I personally would be hesitant to buy a product whose measurements weren’t consistent with good engineering and the designer’s own philosophy. Still, most of the components any of us would consider probably fall into that category; one still has to make a choice for purchase or preference, and the measurements aren’t likely to be of further help (at our current level of understanding). More, I doubt that there is enough interest and money to conduct the sort of listening trials necessary to establish reliable correlations between measurements and subjectively perceived sound quality for music reproduction.

Originally Posted by rbbert

Measurements serve a useful purpose even when they don’t relate directly to how a piece of equipment “sounds”. I personally would be hesitant to buy a product whose measurements weren’t consistent with good engineering and the designer’s own philosophy. Still, most of the components any of us would consider probably fall into that category; one still has to make a choice for purchase or preference, and the measurements aren’t likely to be of further help (at our current level of understanding). More, I doubt that there is enough interest and money to conduct the sort of listening trials necessary to establish reliable correlations between measurements and subjectively perceived sound quality for music reproduction.

Anyway, there will always be personal taste. It's pretty clear that some people prefer a smooth, fuller bodied, and more "holographic" sound, whiles other want maximum detail, transparency, and crisp dynamics.

Originally Posted by Feanor

Anyway, there will always be personal taste. It's pretty clear that some people prefer a smooth, fuller bodied, and more "holographic" sound, whiles other want maximum detail, transparency, and crisp dynamics.

Are there measurements which can indicate which of those sonic characteristics are emphasized (or not?)

Originally Posted by Feanor

For my part, I don't entirely trust currently acknowledged "thresholds of audibility". That is, as I recall that some fairly high level of 2nd order distortion, maybe 2%, is what it takes to be heard.

Again, proving that anything audible can be measured whereas not everything measured can be heard .........

Originally Posted by Mikado463

Again, proving that anything audible can be measured whereas not everything measured can be heard .........

So yes, that is still true. For another thing, I don't trust blind ABX testing as the "be-all and end-all" of prove regarding auditability as all hardcore objectivists seem to.

OTOH, that doesn't mean the measurements are useless in general which is apparently the position of many hardcore subjectivists.

This is a similar conversation that I had with an EE friend of mine and a number of local members on this site.

He builds or rebuilds everything that exists in his audio system.

This week he finished what he referred to as the 48th revision of the crossover for the tweeters in the speakers he designed and the 2 of us built. I said to him. You know what they say the 49th time is a charm. We laughed at the situation and his obsession.

The latest alterations are from changes to the bass driver network that had an effect on mid and high frequencies.

His comment was it sound really good but it measures poorly. I made a comment to the affect that he didn't listen with the meters.
His response was a laugh and saying, I know but, i think it can be a bit better. My guess is it will sound better but as creative people do, you sometimes can't leave things alone because of the "what if factor".

He will have a distraction in a few weeks. He is going to make some upgrades to my amps once the parts and new tubes arrive.

Nicoff,

Other than measurements while replacing parts and measuring the parts themselves before install, probably not. That doesn't mean we won't do a frequency test of the system playing to see if it is different from the one previously taken.

I on the other hand I will be ecstatic if they sound the same though people I have spoken with say it is an improvement.

This is upgrade to keep my ARC 610t amps alive and in good tubes going forward. I hope to never purchase replacements amps.
They currently use 6550s that have gotten harder to get good matched tubes plus the voltage is much higher in the amps.

The change is to use KT120s that was an upgrade option by Audio Research at one time. Lower voltage is easier on all internal parts.

Ordered the KT120s and the person on the phone started laughing at me when I said I needed 38 of them.

I don't want to quote all or parts of your post, it's too long.
I did read the handbook you mentioned, and those references in it that I could find. NONE of those references can pass any kind of methodological scrutiny required to validate their results. That doesn't mean their conclusions aren't valid, only that they aren't proven; as I said, they are essentially observations, not facts. If those conclusions have been proven in more scientifically sound studies since then I would enjoy reading them.

I'm far from anti-measurement; as I posted above in a later post than the one you quoted I would be unlikely to consider a component whose measurements did not indicate both good engineering and meeting its designer's intentions, and it is unfortunate that enough measurements (and the right ones) are often not available. JA's usual measurement suite for amplifiers does include distortion spectrum measurements, and he sometimes comments on how the results might fit with the designer's intention and the resulting sound (as expected, the latter correlation is imperfect at best).

I suspect JA makes more measurements than he publishes in Stereophile; he might well be amenable to a suggestion by you as to which additional measurements would be worthwhile to include in print (IME he has been very open to this type of feedback).

Textbooks are only as good as the evidence they use, and you know as well as I that 90-100 years ago engineers in any field were more likely to accept and use empirical data as fact rather than as observation; even today the distinction between observation (empirical data, no matter how voluminous it might be) and evidence (facts gathered using the scientific method) is often blurred in engineering, occasionally with disastrous results (not so much in electrical engineering, more often in civil/mechanical and chemical engineering).

I play one on TV

Originally Posted by AJ Soundfield

I play one on TV

I thought that was you AJ. Weren't you wearing the blue and white cap in the cab of the locomotive.

Originally Posted by nicoff

What?! Empirical data being used in civil engineering as opposed to evidence gathered using the scientific method? Are you an engineer??

Father, brother, uncles.. they all have/had horror stories about this.

Originally Posted by rbbert

Are there measurements which can indicate which of those sonic [smooth, full-bodied, "holographic", detail, transparent, crisp] characteristics are emphasized (or not?)

It depends on what people really mean when they use those terms, but generally yes there are some measurements that can translate onto those characteristics. An easy one would be transient response, since the faster something can start and stop playing a note, the more crisp or detailed something will likely sound. The more harmonics present, the more full-bodied something will likely sound.

Originally Posted by brad225

I thought that was you AJ. Weren't you wearing the blue and white cap in the cab of the locomotive.

Nah, I was sitting behind a studio recording console, laughing while making an electronic stereo construct based entirely on EE Blumleins patents.
That when transduced by audiophiles, becomes mystically unmeasurable.😉

cheers,

AJ

p.s. Say Hi to Phil for me. He was so smitten by the cardioid bass planars he heard at my old Channeside Loft that he rolled a giant version of his own, after much correspondence. Haven't heard yet but hopefully will soon, I'm sure they will sound great.

p.s. Say Hi to Phil for me. He was so smitten by the cardioid bass planars he heard at my old Channeside Loft that he rolled a giant version of his own, after much correspondence. Haven't heard yet but hopefully will soon, I'm sure they will sound great.[/QUOTE]


That's funny you should mention that. We were talking about my amps upcoming alterations and the subject of engineering. Your name came up and his conversation wafted poetically about listening to your speakers at the loft. Then then went on to say, all of a sudden all he could focus on was the quality of the bass from your cardioid bass creation.

@AJ, sounds like we were neighbors: "at my old Channeside Loft". I lived at Victory Lofts until a few months ago.