[cctalk] Re: Large language model (LLM) Web Scrapers

Xe has done the Lord's work with Anubis, but I'm not a fan in principle, for two reasons: 1. The bots will get smarter. It's an inevitable arms' race. 2. Specific to our hobby/concern/obsession, it demands more of users' browsers. I have people who prefer Floodgap because I don't enforce HTTPS and the site is still largely browseable by Netscape Navigator 3.0 (and by design, which I selected as an arbitrary cutoff, though older browsers like early Mosaic still somewhat work). You don't even need to provide it a Host: header. It will work with browsers that don't even understand charsets in a MIME type. I can't really ask those users to do a JavaScript proof-of-work. I've started putting high bandwidth items exclusively under Gopher and making the Gopher proxy extremely strict. This has the nice side effect of increasing, slowly, the installed base of Gopher clients. There are a few Gopher bots out there but they're usually single people, easily tracked down, and typically apologetic when they realize it went out of control. The AI bots do none of that, so I have no compunction about blackholing them in huge netblocks. If they were willing to work with sites, they wouldn't constantly toss out new IPs and user agents like germophobes and wet wipes. -- ------------------------------------ personal: http://www.cameronkaiser.com/ -- Cameron Kaiser * Floodgap Systems * www.floodgap.com * ckaiser(a)floodgap.com -- You only live twice. -------------------------------------------------------