Loading…
8
My first Python script from 2018 finally broke last week
It was a simple web scraper that pulled weather data, but the site changed its layout after five years. I spent three hours trying to fix the BeautifulSoup selectors before giving up. What's the best way to learn how to update old code for new web pages?
3 comments
Log in to join the discussion
Log In3 Comments
barbara_schmidt686mo ago
Welcome to the fun of web scraping, where your code has an expiration date lol.
5
phoenixpalmer6mo ago
Ugh, tell me about it. How do you even keep up when sites change their layout every other week? Feels like a full time job just fixing broken scrapers.
3
nora_taylor796mo ago
Right? It's like they're playing whack-a-mole with the HTML tags... Barbara's right about the expiration date, you basically have to write code expecting it to break next month. I've started just looking for the data in the network calls instead of the page source, which sometimes lasts longer. Still a huge pain though.
6