Pro tip: I taught myself to spot biased data sets in machine learning projects

I was helping with a hiring AI tool and saw it favored certain backgrounds. After some reading, I learned to check data for unfair patterns. What other checks should I run to make sure the model is fair?

3 comments

3 Comments

simonp471mo ago

Wait, you saw it favoring certain backgrounds? Isn't that a huge problem for hiring? What else did you check besides the data?

felixfisher28d agoMost Upvoted

Yeah you asked what else to check. We ran the same resume but swapped company names. Like "Smith and Sons" versus "Global Data Corp". Same exact role. The AI picked the corporate name every single time. Then we changed just the years worked, making one candidate have a two month gap. Instant rejection. It's not looking at skills at all. You have to test these tiny details to see the bias.

the_spencer1mo ago

Oh man, it's wild. The AI would see someone went to an Ivy League school and basically throw a parade, while a state school grad with the same skills got a "thanks but no thanks." I even tested it with made-up resumes where the only difference was the job title at their last company - "Senior Analyst" got picked, "Lead Data Specialist" doing the exact same work got passed over. It's not checking skills, it's just playing a weird game of status bingo.