Hit 95% accuracy on a custom model and it freaked me out more than I expected

I was working on a little side project to sort my old photos by who was in them using a free AI tool. After maybe 200 training images, I ran a test batch of 50 random pics and it got 47 of them right. That 95% number caught me off guard because I wasn't using fancy hardware or anything, just my laptop and a basic open-source library. It got me thinking about how easy it is to build something pretty powerful without really knowing what you're doing. I double-checked a few of the misses and they were blurry shots where even I had to squint. So now I'm wondering if we're sleeping on how accessible this stuff is for everyday problems. Has anyone else hit a weird success rate that made you stop and question your own setup?

2 comments

2 Comments

uma_johnson1mo ago

Did you try running the same test twice to see if it got different ones wrong?

mark3611mo ago

That phrase "got different ones wrong" really hit home for me. I ran into something similar with a model I was testing for sorting customer emails. First pass it mixed up about 12 out of 100. Then I ran it again with the exact same data and it only messed up 7, but 3 of those were ones it got right before. So yeah, there's a weird randomness to it that makes me wonder how much of the "bad" results are just bad luck. For me, running it three times and taking the average score helped smooth out those swings. Took more time but gave me a clearer picture of what it could really do.