Brad and Ozgur,
If your file is in the server's document root, then it is published [1].
For whatever reason, a lot of C-Levels act as if they are unclear on
that. There is also often the false belief among them that security and
usability are mutually exclusive. I don't understand the rules in their
fantasy role playing, but facts and (ALL) opinions seem to have equal
valence there unlike the real world.
Some of the people who have spent time addressing your question are
globally recognized as being top in the area in which you are asking.
That makes their answers on the topic quite relevant. Further, there is
consensus among the answers.
If for some reason, that is not enough, then it should be noted that
what they say is backed up by the specification defining the web traffic
you are asking about:
Hypertext Transfer Protocol -- HTTP/1.1. RFC 2616. W3C. (1999)
http://tools.ietf.org/html/rfc2616
On 2010-3-13 4:18 AM, Brad Tilley wrote:
> I can make the config change myself.
Or you can submit a patch to the vulnerability scanner and get that
fixed rather than trying to break OpenBSD to match the defect in the
broken scanner you found somewhere.
If your system administrator or web administrator has the competence to
configure /robots.txt for your site,
http://www.w3.org/TR/REC-html40/appendix/notes.html#h-B.4.1.1
you can steer the well-behaved spiders away from parts of your
*published* works. Or, failing that, individual authors publishing on
your site can make use of the "robots" meta element:
<meta name="ROBOTS" content="NOINDEX, NOFOLLOW" />
But some spiders will not behave, so you can try to use PF and htaccess
to limit access to what you've published. But if it is on your web
server it is still published, and if you don't what the material
published, don't go to the effort of publishing it.
/Lars
[1] pubB7lish (pubb2lisL8h), transitive verb
"1. to make publicly known; announce, proclaim, divulge,
or promulgate ..."
http://www.yourdictionary.com/publish
[2] "Robots and the META element" HTML 4.01 Specification
W3C Recommendation 24 December 1999. Appendix B: Performance,
Implementation, and Design Notes.
http://www.w3.org/TR/REC-html40/appendix/notes.html#h-B.4.1.2
