I agree to some extent - it seems to be recording (and system set-up) related.
I have NEVER had a client who could not hear the difference and express a correct acoustic polarity preference for the majority of the inverted tracks from my RoomPlay playlist. Of course, those cuts may feature transients such as from pianos, drums (most any drum, but especially rimshots, struck wood blocks, etc.), guitar plucks, upright bass (such as Brian Bromberg's Wood II, not only for the bass note attacks, but also for when he uses the fret board as a percussion instrument).
Vocals are harder to diagnose. First, the system set-up needs to be really dialed in (an exceedingly rare occasion, IME). Correct acoustic polarity seems to give some vocalists a bit more presence, as if they are more forward in the sound stage. It's easier with close mic'd recordings, IMO. With some recordings, I simply cannot be sure of which is better.