i wrote a program that counts requests by IP and frequency. That and some
fancy code to teduce server load and the bots are blocked fairly well. I
capture the attempts and lot them so I can see where and how theyre doing
it. Inhave noticed many bots use the same exact probe progression, there
must be a popular scraper bot you can download
b
On Tue, Jul 7, 2026 at 6:58 PM Dennis Boone via cctalk <
cctalk(a)classiccmp.org> wrote:
I don’t care
why, I’m not changing my browser configuration to
accomodate whatever it is they think they need to make it work, but
it does mean that more and more of the web has become inaccessible
for me.
As a guy who has to keep a variety of web sites running, I can explain
the why:
1. Spiders and robots, many driven by the current AI craze, have
increased in number, mendicacity, lawlessness, and aggressiveness by
orders of magnitude. If you're running a web site that provides content
via some sort of dynamic mechanism (content management system) that runs
code and does database queries to generate each response page, the new
level of offered load squashes your server into a greasy spot.
2. These spiders (a) lie about who they are, pretending to be real
humans; (b) don't obey robots.txt; (c) retrieve the same page multiple
times at the same instant, sometimes using different user agent strings,
to try to be sure they get the page in the face of increasing server
hostility to their bullshit.
3. These bots don't care about request rate; they just expect you to
deal with it. Will the VPS where you host your ccmp Wordpress install
keep up with dozens of page requests per second? Because if the slop
machine gets interested in something, those requests will come, and will
probably flatten you.
4. Search and (especially) AI companies have had to resort to making
end user machines fetch the content they want. Smart TVs, phones,
tablets, and other devices are increasingly being used. These devices
live in IP address space that's assigned to DSL, cable modems, cell
companies, etc.
... and the difficulty of surviving the onslaught:
1. Because these requests come from where real users live, blocking by
IP address blocks the real users.
2. Because the user agent strings are fabricated to be indistinguishable
from real user browsers, blocking by them blocks the real users.
3. Blocking by behavior has some promise, but you really need insight
into behavior across a wide swath of the internet, unless you're that
big yourself. I.e. Cloudflare and Google and Amazon have a chance.
Hobbyists don't.
For a dynamic web site, that basically leaves putting some sort of
challenge in front of your site that humans can solve easily, but the
bots can't (or don't want to bother), coupled with broader intelligence
on trends across the 'net. That looks, unsurprisingly, like Cloudflare
and similar services.
Unfortunately, those kinds of challenges, and the interface between them
and the protected website, is largely dependent on javascript and
cookies. Grumble.
Welcome to late stage capitalism.
The future, for those of us who hate the technologies, budgets, and ick
factor required to survive the post-web sewer, probably lies in static
web sites. Think tools like Hugo, Jekyll, Pelican. Edit the page on
your local machine. Run the generator. Synchronize the updated flat
HTML pages to the server.
Web servers deliver flat files with remarkable efficiency. sendfile(2)
is damn fast.
Additional upside: it's hard to hack a web app that has no code running
on the server.
De