'if ($args ~ "&$") { return 400; }'
Thanks a lot! Exactly what I needed :)
Posted at Nginx Forum:
https://forum.nginx.org/read.php?2,94128,280124#msg-280124
___
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
That IP resolves to rate-limited-proxy-72-14-199-18.google.com - this is
not the Google search crawler, hence why it ignores your robots.txt. No one
seems to know for sure what the rate-limited-proxy IPs are used for. They
could represent random Chrome users using the Google data saving feature,
he
On Thu, Jun 07, 2018 at 07:57:43PM -0400, shiz wrote:
Hi there,
> Recently, Google has started spidering my website and in addition to normal
> pages, appended "&" to all urls, even the pages excluded by robots.txt
>
> e.g. page.php?page=aaa -> page.php?page=aaa&
>
> Any idea how to redirect/r
'The & to & conversion is another sign of a poor quality crawler.'
I wasn't referring to any of them but to '&'. Important difference.
Also explaining my failure to filter it from parameters since parameters
contains an equal sign. E.g. ...&= something or even &=
& or & would also easy do filt
I see another poster have written this, and deleted it afterwards.
`This is almost certainly not Google as they obey robots.txt. The & to
&
conversion is another sign of a poor quality crawler. Check the RDNS and
you will find it's probably some IP faking Google UA, I suggest blocking at
network l
This is almost certainly not Google as they obey robots.txt. The & to &
conversion is another sign of a poor quality crawler. Check the RDNS and
you will find it's probably some IP faking Google UA, I suggest blocking at
network level.
On Fri, Jun 8, 2018 at 1:57 AM shiz wrote:
> Hi,
>
> Recentl
Hi,
Recently, Google has started spidering my website and in addition to normal
pages, appended "&" to all urls, even the pages excluded by robots.txt
e.g. page.php?page=aaa -> page.php?page=aaa&
Any idea how to redirect/rewrite this?
Posted at Nginx Forum:
https://forum.nginx.org/read.php?2,
rather, $request_uri
Posted at Nginx Forum:
https://forum.nginx.org/read.php?2,276946,276949#msg-276949
___
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
Also - our standardization is not the greatest, so I actually want to
rewrite the entire URI, which is why I have ^/(.*) as the regex. However I
don't think the $1 in the replacement string will still apply to the
original URI. Would this work?
location ~* /v1/device/(.*)/ {
rewrite $uri /api/v1.
I'm trying to setup rewrites so I can automate this more efficiently. Some
of my locations require a rewrite and some do not. I currently have it
hardcoded into the proxy_pass:
location ~* /v1/device/(.*)/ {
proxy_pass http://api.domain.com/api/v1.0/download/$1;
}
Would this accomplish the
10 matches
Mail list logo
