[cctalk] Re: Large language model (LLM) Web Scrapers

[Paul Koning via cctalk]

> On Sep 17, 2025, at 10:38 AM, Doug McIntyre via cctalk 
> <cctalk@classiccmp.org> wrote:
> 
> On Wed, Sep 17, 2025 at 09:33:25AM -0400, Paul Koning via cctalk wrote:
>> A web crawler that does not obey robots.txt is not a law abiding outfit.  
>> Best would be to block it entirely.  If they are that dismissive of honesty, 
>> they are also unlikely to pay attention to such matters as copyright and 
>> intellectual property ownership.
> 
> So, you want to block the whole of the Internet, including every AI company 
> that all ignore robots.txt?

Yes.

        paul