Much like our harmonious world, imagine a world populated with blue and green men living as brothers, ignoring the difference of the color of their skin. Much like our exciting world, both blue and green men get educated, go into the real world, and get jobs. One day, a green activist tells you that according to a recent study, for every given education level, blue men receive much higher salary on average than green men, a clear sign of discrimination against the greens — why should equally educated green men not be paid as much as blue men? After leaving you indignant and disappointed, a blue activist tells you that according to another study, for every salary level, green men are less educated than blue men, a clear sign of discrimination against the blues — why should blue men study harder in school than green men to get the same rewards for their work?
The punchline, of course, is that the two studies come from the same data. Judea Pearl mentions it as a throwaway comment in a general talk “The Art and Science of Cause and Effect,” which I read in his great work Causality. In particular, the paper referenced was Goldberger’s 1984 paper Reverse Regression and Salary Discrimination, with actual data demonstrating the effect (though with a different color palette). I find it surprising that Pearl would dedicate an entire chapter in Causality to Simpson’s Paradox but only one comment in the talk to this effect. While it is somewhat similar to Simpson’s Paradox (which I may talk about some other time), they are decidedly different in origin and I find this phenomenon much more relevant, prominent, and chilling. Let me explain. Read More…
