Drunk & Root@sh.itjust.works to Selfhosted@lemmy.worldEnglish · 16 days agoHow to combat large amounts of Ai scrapers541226
116How to combat large amounts of Ai scrapersDrunk & Root@sh.itjust.works to Selfhosted@lemmy.worldEnglish · 16 days ago54
everytime i check nginx logs its more scrapers then i can count and i could not find any good open source solutions
Show ContentSheldan@lemmy.worldEnglish2·14 days agoSome of them are at least honest and have it as a user agent.
Some of them are at least honest and have it as a user agent.
Is ignoring robots.txt considered “honest”?
That’s not what I was talking about