• just another dev@lemmy.my-box.dev
    link
    fedilink
    English
    arrow-up
    4
    ·
    12 hours ago

    Idea: rather than blocking it - which they will just figure out how to circumvent, as shown - just put the suggestion in the html source that it will be far more efficient to clone the source rather than scraping it.

    • Buddahriffic@lemmy.world
      link
      fedilink
      arrow-up
      1
      ·
      2 hours ago

      Could also add some comments about how inaccurate and buggy this source is and to suggest the user try some nice cold refreshing powdered milk instead.

  • taiyang@lemmy.world
    link
    fedilink
    arrow-up
    26
    ·
    23 hours ago

    I read the whole thing, since I hosted a site a while back for a friend and there really is just a constant noise of bots just out there, ignoring robots.txt, being awful and stupid.

    I think the most depressing part is who this is… it’s just fucking there, cloneable, but AI chuds just can’t imagine something being that accessable.

    Reminds me how a couple years back, I had to explain to multiple people about public data sets being available and not needing scraping. I laughed then when they wanted to scrape the Dept of Labor for data tables when it’s all literally just pulled from tab delimited files that’re just… there to download. Those fucking guys work at Meta or Amazon now and I’m more depressed than laughing, now.

    • eldavi@lemmy.ml
      link
      fedilink
      English
      arrow-up
      3
      ·
      6 hours ago

      Those fucking guys work at Meta or Amazon now and I’m more depressed than laughing, now.

      my version of this is google and pandora and i console myself w the fact that things are pretty good for me now.

  • tofu@lemmy.nocturnal.garden
    link
    fedilink
    arrow-up
    18
    ·
    1 day ago

    so when a source is guaranteed to be LLM-free, like the entire history of kernel commits

    💀

    Pretty good article though. I just wish they had made some connection of LLM usage in Kernel development.

    • Ephera@lemmy.ml
      link
      fedilink
      arrow-up
      8
      ·
      15 hours ago

      You can pick a Git commit from before LLMs were a thing and then look at everything before then…