On Tuesday, September 16th, 2025 at 17:51, Paul Koning via cctalk <cctalk(a)classiccmp.org> wrote: robots.txt doesn't seem to help. If anything it can hurt in this regard. Analyzing my site logs I noticed at least one scraper botnet pulling /robots.txt on its initial connect and then immediately pulling down everything listed in it. Which is also its downfall if you seed the file with traps like a tarpit or a data compression bomb. The Doctor [412/724/301/703/415/510] WWW: https://drwho.virtadpt.net/ Get thee down. Be thou funky.

They do not observe robots .txt Sent from my iPhone

On Wed, Sep 17, 2025 at 09:33:25AM -0400, Paul Koning via cctalk wrote: So, you want to block the whole of the Internet, including every AI company that all ignore robots.txt? All of the AI companies have been already sued by the book publishers for outright pirating all of Z-Lib and Scilib. Several have settled. Meta admited, we only torrented the books, not shared them. (not how that works). Besides Cloudflare (which has a vested interest in this already), the AI constant scraping has prompted solutions such as https://anubis.techaro.lol/ forcing browsers to do proof-of-work to connect to websites to protect their content.

that's an interesting approach for sure. although gopher is probably not a long-term solution, I suppose it does not matter as sooner or later the WWW will be regional webs, protected by corps, governments and netizens themselves. Parts of it will be AI swamps where seldom one ventures and returns unaffected

Yes. paul

Paul In a perfect world yes. Here's a trick you guys can use... Generate a robots.txt and add a few pages not to crawl. Assuming bad bots will ignore, one of the "d onot crawl" pages will have a trigger that blocks the ip address of the session. You would need the ability to communicate the IP address of the offending bot to a process that does the blocking. There are various ways to do that. On Wed, Sep 17, 2025 at 9:46 AM Paul Koning via cctalk < cctalk(a)classiccmp.org&gt; wrote:

On Wednesday, September 17th, 2025 at 09:11, Ethan O'Toole via cctalk &lt;cctalk(a)classiccmp.org&gt; wrote: The same thing happened to me at $dayjob. We wound up putting a bunch of our stuff (wiki, documentation repository, code forge) behind a VPN because the LLM crawler bots were eating up so much bandwidth we couldn't use them for day to day stuff. Which really sucks because they were supposed to be public to begin with. The Doctor [412/724/301/703/415/510] WWW: https://drwho.virtadpt.net/ Get thee down. Be thou funky.

The way I look at it, for my personal site and content, 1. I'm not going to win the arms race. It's not my area of expertise. I can't put hours into devising anti-bot countermeasures. 2. If I do try to implement countermeasures to prevent bots, I will likely also end up impacting/inconveniencing some legitimate users. 3. Ultimately my goal is to help people, so if my content ends up training an AI model,. and that model ends up helping people, then I'm indirectly meeting my goal. 4. Many of the AIs are now citing their sources, and that means I get some level of attribution and recognition. 5. Some archivals, such as wayback machine, I find to be extremely useful to vintage computer research. People die. Providers shut down. A lot of knowledge has been lost. I'll be happy if my content eventually outlives me. I wish there would be more focus on (4). Everyone deserves recognition of their work and their content. I'd support legislation to require that sources are cited/acknowledged when AI results are returned. I think there's some risk of "content laundering", i.e. a bot is trained from your content, someone publishes an AI-generated article, and the next bot is trained from that AI-generated content, losing the original attribution. Without discipline, it can turn into a bunch of slop that nobody knows where it came from, or the accuracy of the information. Scott On Wed, Sep 17, 2025 at 11:31 AM Bill Degnan via cctalk < cctalk(a)classiccmp.org&gt; wrote: