• ɐɥO@lemmy.ohaa.xyz
    link
    fedilink
    arrow-up
    7
    ·
    4 months ago

    I know that. Thats why I dont ban everyone but only those who dont follow the rules inside my robots.txt. All “sane” search engine crawlers should follow those so its no problem