On Friday 04 December 2020 09:00:05 Nicolas George wrote: > hdv@gmail (12020-12-04): > > Let me offer you an alternative option. (Most) bots work by > > analysing the referrals on each page in your website. Right? So, why > > not add a link to a page that normal users will never visit (e.g. > > because they do not see the link and thus will never click on it), > > but will show up in a bot's analysis? That way you can monitor your > > logs for entries containing that page. Every entity requesting that > > specific URL is blocked. > > This made me think of something. > > A long time ago, a friend of mine implemented, to trap the > badly-behaved robots, something called the Book of Infinity: a set of > deterministic pseud-random pages linking to sub-pages ad infinitum, > with ever growing URLs. > > As it happened, it was not actually a good idea, and released a lot of > CO₂, and the very badly behaved robots had to be blacklisted from > explring it. (At some point, we had the same problem when Googlebot > tried to brute-force our online make-your-own-adventure book, but > Googlebot heeds robots.txt.) > > But it could be coupled with techniques inspired by spam tarpits: have > the server reply at a crawl to force the bots to waste resources, > while keeping the resource consumption on the server strictly bounded. > > Oh, I just noticed I was not the first one to think of it: Wikipedia > tells me it's called a spider trap. > > https://en.wikipedia.org/wiki/Spider_trap > Sounds like a good idea, I'll have to think about it, feed the bots in 256 byte pieces every 5 seconds to keep them from timing out, with 256 bytes from rnd mixed in to make a dos packet? :) Just be sure the crc is good. ;-)
> Regards, Take care now, Nicolas. Cheers, Gene Heskett -- "There are four boxes to be used in defense of liberty: soap, ballot, jury, and ammo. Please use in that order." -Ed Howdershelt (Author) If we desire respect for the law, we must first make the law respectable. - Louis D. Brandeis Genes Web page <http://geneslinuxbox.net:6309/gene>
