19 Aug
2023
19 Aug
'23
9:21 p.m.
------- Original Message -------
On Friday, August 18th, 2023 at 12:35, Paul Koning via cctalk
<cctalk(a)classiccmp.org> wrote:
Really? It would be interesting to have evidence
supporting that, because if so, they
could be subjected to pain for violating an explicit order not to do so.
There are some of us elsewhere on the Net (in Fedi, if you're around) who, for
various
reasons are pushing back against the big G and Bing due to the generally lousy state of
search these days, and so dropped their crawlers into robots.txt (per those search
engines' documented entries for said file) to tell their crawlers to go away. It was
subsequently discovered that their crawlers (Google's for sure, Bing's less so)
spidered
and indexed new stuff on those sites anyway. Said new stuff still comes up in search
results, just without text summaries. So, we are now looking into iptables and pf rules
for blocking their crawlers.
That's the extent of what I know right now, because my day job hasn't afforded me
as
much continuous time to devote to the discourse.
The Doctor [412/724/301/703/415/510]
WWW: https://drwho.virtadpt.net/
Don't be mean. You don't have to be mean.