I finally spent 8 hours debugging a CSV import that should have taken 20 minutes

I spent 8 hours debugging a CSV import that should have taken 20 minutes. Last Tuesday I was building a data migration tool for a client who needed to move 12,000 customer records from an old system into Salesforce. The CSV file looked clean, headers matched up fine, but the import kept crashing at row 847. I spent like 4 hours checking field types, character encoding, all the usual stuff. Turns out there was a rogue newline character hidden inside a quote in the middle of the 'company name' column on that specific row. Another 2 hours gone figuring out how to strip it without breaking legit line breaks. And then the client emails me saying they had a similar issue six months ago and just manually fixed those rows. Has anyone else wasted a whole day on something that dumb in their data work?

2 comments

2 Comments

jadew6325d ago

Rogue newline char, classic CSV nightmare fuel.

the_evan25d ago

3 separate times last year I had to write custom scripts to clean up CSV exports from a client's old CRM. One had a rogue newline hiding in the middle of a 5000-row address field, broke my whole pipeline before I caught it. Another time a mismatched quote caused about 200 rows to get mangled. That sinking feeling when you realize a file looks clean but has one little invisible character messing everything up, I know it too well. Hope you found it fast and didn't lose any data.