It's not the size, there's different crossover points for center to compensate for that. It's the timbre, they have to timbre match. You may not understand if you've never heard the difference. If you have panning from side to side, do you want the character of sound to change as it goes across? You should have a solid sound field across the front, a nonmatching center destroys that. The center has to mesh/blend with the mains.
And, the center is not just dialog, it's where most of the main action is on your screen. That's why if serious about your HT set up, not only should the center timbre match but don't get one of those with little 4" drivers. The center is arguably the most important speaker in a surround set up.
And, for multichannel music, it's the same as using mismatch stereo speakers or headphones with different cups on L/R. When was the last time you walked up and really listened to the center when playing? I don't think it was just dialog unless watching the news.
HT doesn't have to be primary but if not going to do it correctly, why bother at all.
NO one is happy about the center being horizontal, it's a compromise manufacturers have to make in order for the speaker to fit under the screen. Ideally, 3 of the same brand, model would be perfect. It's done in better HT systems where the screen is perforated or some other arrangement that allows the center to sit the same way. Most of us with not a dedicated room for just HT have to settle for a horizontal, yet, timbre matched speaker. Most manufacturers who sell a center to match a certain series design them to minimize the negatives of being horizontal.
Just saying