Hi,
Hi, I have the following rewrite rule in place on one of our
staging sites to redirect bots and malicious scripts to our
corporate page:
RewriteCond %{HTTP_USER_AGENT} ^$ [OR]
RewriteCond %{HTTP_USER_AGENT}
^.*(<|>|'|%0A|%0D|%27|%3C|%3E|%00).* [NC,OR]
RewriteCond %{HTTP_USER_AGENT}
^.*(HTTrack|clshttp|archiver|loader|email|nikto|miner|python).*
[NC,OR]
RewriteCond %{HTTP_USER_AGENT}
^.*(winhttp|libwww\-perl|curl|wget|harvest|scan|grab|extract).*
[NC,OR]
RewriteCond %{HTTP_USER_AGENT}
^.*(Googlebot|SemrushBot|PetalBot|Bytespider|bingbot).* [NC]
RewriteRule (.*) https://guardiandigital.com$1 [L,R=301]
However, it doesn't appear to always work properly:
66.249.68.6 - - [08/Jul/2024:11:43:41 -0400] "GET /robots.txt
HTTP/1.1" 200 343 r:"-" "Mozilla/5.0 (compatible; Googlebot/2.1;
+http://www.google.com/bot.html)" 0/5493 1145/6615/343 H:HTTP/1.1
U:/robots.txt s:200
Instead of making changes to my rules then having to wait until
the condition is met (Googlebot scans the site again), I'd like to
simulate the above request against my ruleset to see if it
matches. Is this possible?
For the user agent, just install an extension in your browser to
"fake" the value, and make a HTTP request. Alternatively, you can use
curl as well.
I should have mentioned that this was part of a larger effort to
redirect bots while also blocking access to others altogether as well as
allowing authorized users. Here's what I've come up with, which seems to
work quite well. This also all has to appear in .htaccess because it's
processed after the virtualhost config and any requireall/requireany
entries are overridden that already appear there. I also learned that
RequireAny is default deny.
RewriteCond %{HTTP_USER_AGENT} ^$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*(<|>|'|%0A|%0D|%27|%3C|%3E|%00).* [NC,OR]
RewriteCond %{HTTP_USER_AGENT}
^.*(HTTrack|clshttp|archiver|loader|email|nikto|miner|python).* [NC,OR]
RewriteCond %{HTTP_USER_AGENT}
^.*(winhttp|libwww\-perl|curl|wget|harvest|scan|grab|extract).* [NC,OR]
RewriteCond %{HTTP_USER_AGENT}
^.*(Googlebot|SemrushBot|PetalBot|Bytespider|bingbot).* [NC]
RewriteRule (.*) https://guardiandigital.com/$1 [L,R=301]
SetEnvIf user-agent "(?i:GoogleBot)" googlebot=1
SetEnvIf user-agent "(?i:SemrushBot)" googlebot=1
SetEnvIf user-agent "(?i:PetalBot)" googlebot=1
SetEnvIf user-agent "(?i:Bytespider)" googlebot=1
SetEnvIf user-agent "(?i:bingbot)" googlebot=1
<RequireAny>
Require ip 1.2.3.4
Require env googlebot
</RequireAny>
I was also originally trying to associate the rewriterules with the
requireany using <If> but then realized I didn't even have to do that -
it would just automatically get processed independently. It looks so
simple now, but took me a while to make it this simple.
