Pro tip: GPT-4 beat Claude 3.5 for summarizing legal docs in my test

I spent last weekend comparing both on 5 dense contracts from a client in Ohio. Claude kept missing key clauses about liability limits, while GPT-4 caught every single one. Your experience might differ depending on the doc type though. Anybody else run side by side comparisons on specific tasks?

3 comments

3 Comments

jade_singh1mo ago

Yo that's interesting, I've been using GPT-4 for contract reviews with my crew's sub agreements and it nails the fine print every time. Claude seems to gloss over stuff like indemnification clauses which is a dealbreaker for legal stuff. Stick with GPT-4 if you need accuracy over speed for dense docs.

olivia_allen1mo ago

Actually you might be mixing up Claude and something else. Claude is actually pretty good with contract details, its the newer Claude models like Sonnet that tend to be more thorough. Maybe you tried an older version or the free tier or something because I've been using Claude for reviewing lease agreements and it catches indemnification clauses just fine. Could also be down to how you prompt it, GPT-4 is great too though no argument there.

troyreed1mo agoMost Upvoted

Hey has anyone else had a buddy test this? My friend Mike, who does contract work for a construction company in Texas, tried both last month on some vendor agreements. @jade_singh is spot on about GPT-4 catching the fine print. Mike said Claude let an indemnification clause slip right by, but GPT-4 flagged it and even pointed out a weird arbitration clause buried in the third page. He ended up redoing the whole contract because of what GPT-4 caught.