27
Picked a local LLM over a cloud API for a project last month
Ran a Llama 3 8B model locally on my laptop instead of paying for GPT-4 tokens, and after 30 test runs the latency was 2x slower but I saved about $180 in API fees, has anyone else benchmarked local versus cloud for small apps?
3 comments
Log in to join the discussion
Log In3 Comments
johnson.daniel1mo ago
I spent a solid weekend getting a 7B model running on my 2019 laptop and it sounded like a hair dryer taking off every time I hit enter. My roommate thought I was running a crypto miner in my bedroom. The latency was bad enough that I'd make a cup of coffee between queries but hey I saved like 60 bucks and learned that my GPU is basically a toaster with delusions of grandeur.
6
sarah5311mo ago
Wait, I gotta point out that a 2019 laptop probably doesn't have a GPU worth mentioning for this, so you were likely running that 7B model entirely on the CPU which explains both the noise and the coffee breaks. Even a decent integrated GPU from that era would struggle with anything above a 3B or 4B model, so you were basically brute forcing it through raw processor power and laptop fans. Still, it's kind of impressive it ran at all, just don't blame the GPU when it was really your CPU and memory hitting the wall.
5