• cron@feddit.org
    link
    fedilink
    arrow-up
    4
    ·
    3 months ago

    Does robots.txt really work in the fediverse? At least on lemmy, the content can be retrieved on different hosts, all of which have different robots.txt files. Unless it is somehow “baked” into the protocol.