Caught myself training an AI on biased data without even realizing it

I was feeding my model customer service chats from the last 3 years and didn't notice it was 90% responses from white male agents. My coworker pointed out the gender skew after I ran first tests. Has anyone else accidentally baked in bias from their own training sets?

2 comments

2 Comments

the_paul1mo ago

Oh man, this hit close to home. I had a similar thing happen when I was training a model on support tickets from my company. Turns out our night shift team was mostly women, so the model started associating female names with late night complaints. Completely missed it until a tester noticed the time stamps correlated with gender. It's scary how easy it is to overlook stuff like that when you're just focused on the data.

adam_lewis1mo ago

Got a buddy who works at a healthcare startup and they trained a model to predict patient no-shows. Everything looked great until someone ran a fairness check and found it was flagging people from certain zip codes way more often. Turns out those neighborhoods had less access to public transit and the model was basically learning "poor people miss appointments." They had to retrain the whole thing with commute time data included. It's wild how these hidden variables just sneak in when you're not paying attention.