Just found out GPT-4 training cost way more than I thought

I read a report from SemiAnalysis that said GPT-4 cost around $100 million to train. That's like 10x what I assumed it was. How do smaller companies even compete with that kind of money?

2 comments

2 Comments

miles_sanchez1mo agoMost Upvoted

Man that number still blows my mind every time I see it. The trick is picking a very specific problem people will pay for and building a tiny model that does that one thing better than GPT ever could.

grantschmidt1mo ago

Yeah $100 million is insane but smaller teams can get away with cheaper by not reinventing the wheel just fine tune existing open models and focus on a real niche problem instead of trying to match OpenAI dollar for dollar.