Mastodon Feed
Boosted by jwz:
jplebreton ("JP") wrote:
Does anyone know the concrete technical reason(s) that LLM website scrapers have been so much nastier to deal with than the ones used by major search engines? Like do these people just not know how to write a scraper that won't DDOS (or equivalent effect) a server? Are they trying to get the data faster or more thoroughly than other scrapers? Do they just not care? Like obviously they don't care but I can't tell if that's the main reason they're so horrible or some more technical point.