Mastodon Feed
Boosted by cstanhope@social.coop ("Your friendly 'net denizen"):
algernon@come-from.mad-scientist.club ("algernon, the ruthless") wrote:
PSA to people who've been using gzip bombs to deter crawlers: I was wrong. It works.
At least to some extent: I started noticing recently that some of the disguising bots set
accept-encoding: identity, likely to avoid those gzip bombs. I can't imagine any other reason to do so than to avoid the bombs, so: CONGRATS, genuinely!You made them play catch up.
The good news is, this header is also a good indicator that they're bots, especially if they pretend to be Chrome (or send no user-agent string). Do with this information as you wish, but first: celebrate!