I tracked my AI hiring tool's rejection rate for 3 weeks and the change scared me
I work at a small insurance office in Boise and we started using an AI resume screener back in January to handle the flood of applications for one entry level job. For the first two weeks it rejected about 60% of resumes, which seemed fine, but then I started noticing something weird. I ran a test where I submitted the same resume twice, once with my name and once with a name I made up that sounded more ethnic, and the second one got auto rejected in under a minute while mine made it to the next round. Over 3 weeks I did this 14 times with fake names and 11 of the ethnic sounding ones got tossed, which is way too many to be a coincidence. The tool was never taught to do that on purpose, it just learned from old hiring data where certain names got picked less. My boss said it saves us time but I can't stop thinking about those 11 people who never had a shot. Has anyone else seen a bias test go this wrong, and did you actually get your company to drop the tool?