Mastodon Feed: Post

Mastodon Feed

kornel ("Kornel") wrote:

The https://lib.rs site is getting hammered by residential proxies (probably browser extensions and "monetized" mobile & smart TV apps). They're IPs from all over the world, normal-looking browsers with JS and cookies.

The problem is they're crawling a long tail of junk, causing a cache miss on every request.

I wanted to show users fresh data on every page, but that hits crates-io and github APIs. I ratelimit my requests, so everything piles up and dies timing out.