18 Aug
2023
18 Aug
'23
9:06 p.m.
Off the original subject but here’s Google’s description of robots.txt and how Google can
show your page in results.
https://developers.google.com/search/docs/crawling-indexing/robots/intro
Sent from my iPhone
On Aug 18, 2023, at 13:39, Sellam Abraham via cctalk <cctalk(a)classiccmp.org>
wrote:
On Fri, Aug 18, 2023 at 12:35 PM Paul Koning via cctalk <
cctalk(a)classiccmp.org> wrote:
On Aug 18, 2023, at 12:48 PM, The Doctor via cctalk <
cctalk(a)classiccmp.org> wrote:
------- Original Message -------
On Thursday, August 17th, 2023 at 13:29, Bill Gunshannon via cctalk <
cctalk(a)classiccmp.org> wrote:
On 8/17/2023 2:14 PM, John Herron via cctalk wrote:
Why would you give all your data to Google?
They pretty much have it all anyway. Even if you use /robots.txt to
block
them, they still spider and index pages,
Really? It would be interesting to have evidence supporting that, because
if so, they could be subjected to pain for violating an explicit order not
to do so.
Not to mention that that could well be a criminal act since it would
constitute unauthorized access of a computer system.
If anyone has actual evidence of this happening now then please show it to
me, and I'll take the necessary legal steps to bring Larry and Sergey to
justice.
Sellam