On Friday 04 December 2020 12:39:24 Reco wrote: > Hi. > > On Fri, Dec 04, 2020 at 08:39:42AM -0500, Gene Heskett wrote: > > But I asked specifically how to enable it for one bot, and I've > > asked that question several times, getting smoke and mirror answers > > you all assume are helpfull, but which are useless to a new user > > installing the now 7 years old and long out of date package that in > > effect has no "how it works" docs. I asked 3 questions in a previous > > day or so timeline, and no one has actually attempted to actually > > answer even one of them. Here is one line from that log: and that I > > just blocked: > > > > coyote.coyote.den:80 192.99.6.226 - - > > [04/Dec/2020:07:18:20 -0500] "GET > > /gene/toolshed/c3/build/win32/prep/?C=S;O=D HTTP/1.1" 200 673 "-" > > "Mozilla/5.0 (compatible; MJ12bot/v1.4.8; http://mj12bot.com/)" > > Taken directly from the link. > > Bot Type Good crawler (always identifies itself) > IP Range Distributed, Worldwide > Obeys Robots.txt *Yes*
Sorry, they do not, they've read it and ignored it 428 times in the life of that log which I zeroed out around 1 July of this year. They've also used up my upload bandwidth 37760 times in the life of that log. I have a 10 megabit service, means 2.5 going up. That hogging of my upload bandwidth can only be defined as a DDOS attack. Of the 112 DROP rules I currently have, at least 80 were generated by their activities against my site. > Obeys Crawl Delay Yes What the heck is that? > Data served at Majestic.com > > I kindly suggest to all debian-user members to reflect on this, and to > stop this pointless discussion. > > Reco At this point it sounds like you are defending them, but until they read and obey robots.txt, I will continue to block them the instant I can ID that its them. Cheers, Gene Heskett -- "There are four boxes to be used in defense of liberty: soap, ballot, jury, and ammo. Please use in that order." -Ed Howdershelt (Author) If we desire respect for the law, we must first make the law respectable. - Louis D. Brandeis Genes Web page <http://geneslinuxbox.net:6309/gene>
