Mastodon Feed
Boosted by baldur@toot.cafe ("Baldur Bjarnason"):
Codeberg@social.anoxinon.de wrote:
We apologize for the long performance degradation today.
Finally, we identified all of the 'tricks' that AI crawlers found today. They no longer bypass the anubis proof of work challenges.A novelty for us was that AI crawlers seem to not only crawl URLs that are actually presented to them by our frontend, but they converted the URLs into a format that bypassed our filter rules.
By the way, you can track the changes we have been doing via
https://codeberg.org/Codeberg-Infrastructure/scripted-configuration/compare/51618~1..e4aac