Package: www.debian.org User: www.debian....@packages.debian.org Usertags: packages Severity: important
Hi, While looking at blocking some web spiders that were causing significant load on picconi (packages.d.o master), I noticed that the robots.txt currently doesn't attempt to block them. Specifically, the current file is: <quote> User-agent: * Disallow: </quote> which is effectively the same as allowing everything. "Disallow: /" might be more logical, unless there is a desire / requirement to allow crawling and indexing of (parts of) the site. Regards, Adam
