Mastodon Feed: Post

Reblogged by cstanhope@social.coop ("Your friendly 'net denizen"):

phooky@octodon.social wrote:

Moving all my web content into "robots.txt" files on the presumption that they'll be ignored by crawlers