Why you can’t trust measurements

I count 6 to 8 different tests at ASR.

What kind of a measurement is listening?
Do you mean statistical A/BX tests?
Listening test isn't one of the 6-8.
ABX is one such variety for specific purposes. This is the same Amir who (now) brags about doing such tests while JJ was employed. The revisionist history version. Best be able to show what you're measuring is in any way relevant to audibility, or its just a fancy picture. Ask Toole, JJ and Olive. They post there too.
 
To add onto Ralph's post, in addition to the audible impact of low-order and high-order harmonics and distortion as a function of frequency, the amount of masking (how much you "need", as a function of frequency, etc.) has also been extensively researched since it plays such a critical part in audio compression algorithm design.
 
We've known since at least the 1930s how the ear responds to the various harmonics (see Radiotron Designer's Handbook, 3rd edition). Its also well known that if the circuit has a lot of IMD it will sound fatiguing- IOW harsh and bright.

Its been known for a long time how the 2nd harmonic contributes to 'warmth' (this is the ear converting distortion to a tonality). That should not be controversial- nor should it be that the higher orders contribute to harshness and brightness. Its also easy to demonstrate with very simple test equipment that the ear is using the higher ordered harmonics to sense sound pressure.

The tricky bit is that the lower orders can mask the higher orders, but not surprisingly, this works the same way in electronics as it does with loudspeakers. But this explains how a tube amp can sound smoother than a solid state amp even though its making more higher ordered harmonic distortion. Its masked, so the amp sounds smooth. Tube amps (assuming proper design) without feedback tend sound even smoother as they lack a lot of the higher orders and have a more prodigious 2nd or 3rd harmonic.

People don't trust measurements for simple reasons. First, most of the important measurements are not made or not published. 2nd, even if they are you have to know what they are saying and many people do not.

I am skeptical of "evidence" gathered in the 1920's and 1930's about the audibility of the harmonic spectrum and distortion spectrum of amplifiers and how it might apply to present day audio electronics. Without knowing more I tend to assume (with all that implies) that much of this is empiric data that has not been subjected to appropriate testing to establish it as true evidence. That is not to say that it isn't useful in amplifier design, where the designer will have both measurements and listening tests to attain his design goals, but how it relates to the consumer's (listener's) ability to choose the "best" amplifier for him is unclear.
 
Rob - there are plenty of modern references on the audibility of linear and non-linear distortion. Just search on the topic, you’ll find a plethora of articles by the likes of Toole, Geddes, Lee, Villchur, and others. They all say pretty much the same thing: normal measurements of IMD, THD, and (gag a maggot) SINAD do not correlate to perceived measures of quality. When you add additional variables like frequency and amplitude and complex stimuli then things get more interesting, as Ralph has noted in his earlier posts.

A lot of work has been done on the mechanics of human hearing perception that explains why these classic measurements fall short. It’s fascinating stuff, psychoacoustics.
 
Rob - there are plenty of modern references on the audibility of linear and non-linear distortion. Just search on the topic, you’ll find a plethora of articles by the likes of Toole, Geddes, Lee, Villchur, and others. They all say pretty much the same thing: normal measurements of IMD, THD, and (gag a maggot) SINAD do not correlate to perceived measures of quality. When you add additional variables like frequency and amplitude and complex stimuli then things get more interesting, as Ralph has noted in his earlier posts.

A lot of work has been done on the mechanics of human hearing perception that explains why these classic measurements fall short. It’s fascinating stuff, psychoacoustics.

I have read a great deal of this research (but not by any means all). As far as I can tell, almost all of these additional measurements overall add little to the traditional measurements you mentioned (except to reflect the designer's priorities and how well those have been achieved); all are audible above a certain threshold and in general designers are wise to keep these measured parameters below those thresholds. However, once that is achieved components still often sound different (even to, or perhaps especially to, the designer), so all these measurements still do not correlate with sound quality, as long as the parameters measure below those thresholds of audibility. And FWIW, some technically minded reviewers, even (yes) Amir, measure many of these extra "distortions", and when they are below the established levels of audibility the product is pronounced "good"; does that tell anyone how it sounds?
 
That depends on the component in question!

If an amplifier or preamp, the distortion spectra at 1 Watt is good. Also distortion spectra a 6dB below full power. Also distortion vs frequency at 1 Watt and -6db of full power. If the distortion overall is -105dB or better then the music can mask the distortion and the amp or preamp will be pretty neutral. Many people think that the distortion can be higher and it will still be inaudible. IME, that isn't true.

Here are some things to help out:
1) the ear assigns a tonality to all forms of distortion. As an example the 'sonic signature' of any amplifier is actually its distortion signature.
2) the ear uses higher ordered harmonics to sense sound pressure and so is keenly sensitive to their presence; it has about a 130dB range! The higher ordered harmonics as distortion are sensed as 'bright and harsh'.
3) the lower ordered harmonics (2nd and 3rd) can mask the presence of the higher ordered harmonics if the amplitude of the former is sufficient.

If you know these things then the measurements (if they exist as I described) can tell you a lot!

If you see distortion increasing with frequency (above 2-3KHz) that's a sign that the circuit is losing feedback at those higher frequencies. This can happen if the circuit lacks the Gain Bandwidth Product to support the feedback. The increased distortion will be perceived as brightness and harshness. We've been hearing solid state amps with this problem since the inception of transistors. In addition, if the circuit lacks the feedback needed to really clean things up (+30dB at all frequencies) it will sound harsh and bright since otherwise the feedback will cause distortion of its own thru the process of bifurcation at the feedback node, which is non-linear.

Tubes get around this a bit by having a pronounced 2nd or 3rd harmonic which can mask the presence of the higher orders. In this way they can sound smooth in the mids and highs even though they have more higher ordered harmonics than most solid state amps.

Supporting +30dB of feedback is pretty hard and the semiconductors needed for an amp to do that didn't exist in the 70s or 80s and just barely sometime in the 1990s. Plus the will to deal with this problem for the most part seemed to be absent with many solid state designers. So we have a lot of solid state amps that are harsh and bright as a result.

With class D amps its possible to apply enough feedback so there are class D amps now that lack the harshness of traditional AB amps. Such amps are usually of the self oscillating variety; so much feedback is applied to them that the amp goes into oscillation as soon as its turned on. The oscillation is then used as the switching frequency, killing two birds with one stone. Even then, if the distortion spectra isn't right, the amp can exhibit amusical characteristics. But they can be seen if the amp is tested on the bench as I described.

Many thanks, Ralph.

You are one of few people who both acknowledge and understand the nature of human reaction to distortion. A failure of many folks at ASR is specifically that they don't acknowledge the effect of 2nd & 3rd order distortion.

Forgive me but I don't have your expertise and years of study and design experience, but my observation both of test results in the popular hi-fi press and listening to audiophiles, (most of them subjectivists), convinces me of the following:
  • Higher order HD is irritating: harsh, grainy, strident, etc.
  • 2nd and maybe less so 3rd order is not only benign but actually pleasant to most listeners. (Hence the preference of many for tube equipment.)
  • 2nd/3rd order can MASK higher order and other distortion. (That's why many folks find a tube preamp works well ahead of a S/S power amp.)
  • 2nd/3rd order mask distortions in up & downstream components
  • 2nd/3rd order also mask brightness and distortions in recordings.
I do think that measurements ought include the harmonic spectra at different base frequencies as will as different powers levels, and probably also different speaker impedances.

Audiophiles, rather than rejecting measurement out of hand, should begin to understand how to interpret measurements and insist on better, more complete measurements.
 
I have read a great deal of this research (but not by any means all). As far as I can tell, almost all of these additional measurements overall add little to the traditional measurements you mentioned (except to reflect the designer's priorities and how well those have been achieved); all are audible above a certain threshold and in general designers are wise to keep these measured parameters below those thresholds. However, once that is achieved components still often sound different (even to, or perhaps especially to, the designer), so all these measurements still do not correlate with sound quality, as long as the parameters measure below those thresholds of audibility. And FWIW, some technically minded reviewers, even (yes) Amir, measure many of these extra "distortions", and when they are below the established levels of audibility the product is pronounced "good"; does that tell anyone how it sounds?

For my part, I don't entirely trust currently acknowledged "thresholds of audibility". That is, as I recall that some fairly high level of 2nd order distortion, maybe 2%, is what it takes to be heard. But I'm thinking that the level people need to hear as distortion as such is much higher than than what it takes to influence their subjective preferences.
 
To add onto Ralph's post, in addition to the audible impact of low-order and high-order harmonics and distortion as a function of frequency, the amount of masking (how much you "need", as a function of frequency, etc.) has also been extensively researched since it plays such a critical part in audio compression algorithm design.

That's interesting. And I think that lack of understanding of the masking effect 2nd/3rd order HD is a shortcoming of both tech-heads like Amir at ASR and subjectivist audiophile who scoff at measurements.
 
Measurements serve a useful purpose even when they don’t relate directly to how a piece of equipment “sounds”. I personally would be hesitant to buy a product whose measurements weren’t consistent with good engineering and the designer’s own philosophy. Still, most of the components any of us would consider probably fall into that category; one still has to make a choice for purchase or preference, and the measurements aren’t likely to be of further help (at our current level of understanding). More, I doubt that there is enough interest and money to conduct the sort of listening trials necessary to establish reliable correlations between measurements and subjectively perceived sound quality for music reproduction.
 
Measurements serve a useful purpose even when they don’t relate directly to how a piece of equipment “sounds”. I personally would be hesitant to buy a product whose measurements weren’t consistent with good engineering and the designer’s own philosophy. Still, most of the components any of us would consider probably fall into that category; one still has to make a choice for purchase or preference, and the measurements aren’t likely to be of further help (at our current level of understanding). More, I doubt that there is enough interest and money to conduct the sort of listening trials necessary to establish reliable correlations between measurements and subjectively perceived sound quality for music reproduction.

Anyway, there will always be personal taste. It's pretty clear that some people prefer a smooth, fuller bodied, and more "holographic" sound, whiles other want maximum detail, transparency, and crisp dynamics.
 
Anyway, there will always be personal taste. It's pretty clear that some people prefer a smooth, fuller bodied, and more "holographic" sound, whiles other want maximum detail, transparency, and crisp dynamics.

Are there measurements which can indicate which of those sonic characteristics are emphasized (or not?)
 
For my part, I don't entirely trust currently acknowledged "thresholds of audibility". That is, as I recall that some fairly high level of 2nd order distortion, maybe 2%, is what it takes to be heard.

Again, proving that anything audible can be measured whereas not everything measured can be heard .........
 
Again, proving that anything audible can be measured whereas not everything measured can be heard .........

So yes, that is still true. For another thing, I don't trust blind ABX testing as the "be-all and end-all" of prove regarding auditability as all hardcore objectivists seem to.

OTOH, that doesn't mean the measurements are useless in general which is apparently the position of many hardcore subjectivists.
 
This is a similar conversation that I had with an EE friend of mine and a number of local members on this site.

He builds or rebuilds everything that exists in his audio system.

This week he finished what he referred to as the 48th revision of the crossover for the tweeters in the speakers he designed and the 2 of us built. I said to him. You know what they say the 49th time is a charm. We laughed at the situation and his obsession.

The latest alterations are from changes to the bass driver network that had an effect on mid and high frequencies.

His comment was it sound really good but it measures poorly. I made a comment to the affect that he didn't listen with the meters.
His response was a laugh and saying, I know but, i think it can be a bit better. My guess is it will sound better but as creative people do, you sometimes can't leave things alone because of the "what if factor".

He will have a distraction in a few weeks. He is going to make some upgrades to my amps once the parts and new tubes arrive.
 
This is a similar conversation that I had with an EE friend of mine and a number of local members on this site.

He builds or rebuilds everything that exists in his audio system.

This week he finished what he referred to as the 48th revision of the crossover for the tweeters in the speakers he designed and the 2 of us built. I said to him. You know what they say the 49th time is a charm. We laughed at the situation and his obsession.

The latest alterations are from changes to the bass driver network that had an effect on mid and high frequencies.

His comment was it sound really good but it measures poorly. I made a comment to the affect that he didn't listen with the meters.
His response was a laugh and saying, I know but, i think it can be a bit better. My guess is it will sound better but as creative people do, you sometimes can't leave things alone because of the "what if factor".

He will have a distraction in a few weeks. He is going to make some upgrades to my amps once the parts and new tubes arrive.

I wonder if he will do 'before' and 'after' measurements on your equipment...
 
Nicoff,

Other than measurements while replacing parts and measuring the parts themselves before install, probably not. That doesn't mean we won't do a frequency test of the system playing to see if it is different from the one previously taken.

I on the other hand I will be ecstatic if they sound the same though people I have spoken with say it is an improvement.

This is upgrade to keep my ARC 610t amps alive and in good tubes going forward. I hope to never purchase replacements amps.
They currently use 6550s that have gotten harder to get good matched tubes plus the voltage is much higher in the amps.

The change is to use KT120s that was an upgrade option by Audio Research at one time. Lower voltage is easier on all internal parts.

Ordered the KT120s and the person on the phone started laughing at me when I said I needed 38 of them.
 
I am skeptical of "evidence" gathered in the 1920's and 1930's about the audibility of the harmonic spectrum and distortion spectrum of amplifiers and how it might apply to present day audio electronics. Without knowing more I tend to assume (with all that implies) that much of this is empiric data that has not been subjected to appropriate testing to establish it as true evidence. That is not to say that it isn't useful in amplifier design, where the designer will have both measurements and listening tests to attain his design goals, but how it relates to the consumer's (listener's) ability to choose the "best" amplifier for him is unclear.

Your skepticism does not serve you well in this case.

If you had looked up the reference, you'd have found a page that referred to the orders of harmonics. This one has tripped me up in the past so I'm going to elaborate on the issue so hopefully you won't be tripped up in the same way.

The 3rd harmonic is 3X the fundamental tone. But musically its a 5th. That is why it can be innocuous if not too loud. But the 7th harmonic, 7X the fundamental, is not a harmonious chord. IOW harmonics, with the exception of the 2nd, are not octaves above the fundamental tone. That is one of the reasons the 7th has to be very low or else it will grate on the ear.
This simple fact has been understood for several centuries.

Now something else is pointed at in that 1930s tome; the simple fact that humans are far more sensitive to higher ordered harmonics than they are the lower orders. This has been repeatedly proven in the succeeding decades, but unfortunately largely ignored by the audio industry for most of the last 70 years. The reason we are so sensitive to these harmonics is the ear uses them to tell how loud sounds are, and since the ear has about a 130dB range, that means it has to really sensitive in this manner!

What I'm talking about might be an amp that is low in distortion, perhaps only 0.01%. Such an amp might measure quite well, but when you listen to it, its harsh and bright. That's because the higher ordered harmonics compose most of the distortion spectra- the lower orders for whatever reason are less prominent or more greatly suppressed. Yet this fact could have been surmised by any designer having read the Radiotron Designer's Handbook from that long ago.

Knowledge is a cumulative thing until something comes along that explains the phenomena better. That's the nature of what we call 'theory'. The simple fact is nothing has supplanted the theory documented in that 1930s text.

Enough about that. The thing now that should be easy to see is that low THD simply isn't enough to tell you anything about how an amp sounds. Further, if you know the distortion spectra then you can take a pretty good guess at what it will sound like. But most audiophiles that promote the idea that we 'can hear things that can't be measured' never got to that bit about distortion spectra. They are still hung up on the THD. So we have this myth out there which was once true; nowadays though we can measure distortion spectra fairly easily and we can measure distortion vs frequency. If we understand what all this can tell us, then the idea that measurements won't tell us how it sounds simply winds up being false.

But like I said, those measurements rarely get published if they are even made. A lot of equipment would be shown up in a big hurry and I have this idea that the industry for the most part really doesn't want that. They'd have to fixs all the bugs in their designs- and that is a lot easier said than done!
 
I don't want to quote all or parts of your post, it's too long.
I did read the handbook you mentioned, and those references in it that I could find. NONE of those references can pass any kind of methodological scrutiny required to validate their results. That doesn't mean their conclusions aren't valid, only that they aren't proven; as I said, they are essentially observations, not facts. If those conclusions have been proven in more scientifically sound studies since then I would enjoy reading them.

I'm far from anti-measurement; as I posted above in a later post than the one you quoted I would be unlikely to consider a component whose measurements did not indicate both good engineering and meeting its designer's intentions, and it is unfortunate that enough measurements (and the right ones) are often not available. JA's usual measurement suite for amplifiers does include distortion spectrum measurements, and he sometimes comments on how the results might fit with the designer's intention and the resulting sound (as expected, the latter correlation is imperfect at best).
 
I don't want to quote all or parts of your post, it's too long.
I did read the handbook you mentioned, and those references in it that I could find. NONE of those references can pass any kind of methodological scrutiny required to validate their results. That doesn't mean their conclusions aren't valid, only that they aren't proven; as I said, they are essentially observations, not facts. If those conclusions have been proven in more scientifically sound studies since then I would enjoy reading them.

I'm far from anti-measurement; as I posted above in a later post than the one you quoted I would be unlikely to consider a component whose measurements did not indicate both good engineering and meeting its designer's intentions, and it is unfortunate that enough measurements (and the right ones) are often not available. JA's usual measurement suite for amplifiers does include distortion spectrum measurements, and he sometimes comments on how the results might fit with the designer's intention and the resulting sound (as expected, the latter correlation is imperfect at best).

The Radiotron Designer's Handbook is meant to be educational; no methodology is required with most textbooks; its assumed that to be a textbook the information is accurate, although I imagine that last bit has been eroded seriously in the past couple of decades... :(

JA does show distortion spectra but he does not show it at a variety of frequencies, you don't always see distortion vs frequency and you certainly don't see distortion spectra at -6dB of full power. So I'm not of the opinion that his measurements are all that useful, being simply incomplete.
 
Back
Top