Mastodon Feed: Post

Mastodon Feed

fromjason ("fromjason.xyz ❤️ 💻") wrote:

A fairly comprehensive robots.txt disallow list of known AI crawlers that you can use for your website (and hopefully one day for your Bluesky posts). https://github.com/ai-robots-txt/ai.robots.txt/blob/main/robots.txt