Thanks Zurd for taking the time to answer this. Turns out it was a really simple mistake. After reading this closed bug report <https://github.com/fail2ban/fail2ban/issues/1453> (specifically the comment here <https://github.com/fail2ban/fail2ban/issues/1453#issuecomment-222469723>) I realized that whitespace after the beginning-of-line anchor (^) was actually necessary.
The entry in the wiki <http://www.fail2ban.org/wiki/index.php/MANUAL_0_8#Filters> has this to say, though: > If the failregex is anchored with a leading |^|, then the anchor > refers to the start of the remainder of the line, /after/ the > timestamp and intervening whitespace. So if I'm reading this correctly, this is actually inaccurate, since according to the bug report and that specific comment the whitespace after the timestamp needs to be explicitly included. Unless someone gets back to me telling me why I'm wrong, I'm going to change the entry in the wiki. Thanks for your time everyone. > This was the logical end of a sequence of more general regexes to make > sure I wasn't losing my mind. Here's the one I started out with: > > ^\[error\] \d+#\d+: \*\d+ \S+\(\) \"\S+\" (failed|is not found) \(2\: No > such file or directory\), client\: <HOST>\, server\: \S*\, request\: > \"(GET|POST|HEAD) \/\S+ \S+\"\, .*?$ > > When you expand <HOST> out to |(?:::f{4,6}:)?(?P<host>\S+) |as the wiki > suggests (http://www.fail2ban.org/wiki/index.php/MANUAL_0_8#Filters), > you get > > ^\[error\] \d+#\d+: \*\d+ \S+\(\) \"\S+\" (failed|is not found) \(2\: No > such file or directory\), client\: (?:::f{4,6}:)?(?P<host>\S+)\, > server\: \S*\, request\: \"(GET|POST|HEAD) \/\S+ \S+\"\, .*?$ > > (I tried this one as well), which debuggex.com says should match. > Compiling this last regex in both Python 2 and Python 3, I get a match > when I run it against this line. It's not the over-general regex. The > timestamp, from fail2ban-regex output in my first, should also match. So > what gives? > > > Date: Wed, 6 Jul 2016 21:43:33 -0400 > > From: Zurd <zurd33@...> > > Subject: Re: [Fail2ban-users] fail2ban refuses to match even most > > basic acceptable regex > > To: fail2ban-users@... > > Message-ID: > > <CAFPUJG7_4_LjJCH0HWZJEbKngjaOU_hPizLuuVo_K-LSY2X9wQ@...> > > Content-Type: text/plain; charset="utf-8" > > > > I am by no means expert but I can see that having a filter so simple and so > > small doesn't work. I don't think it's really a bug though as fail2ban > > could be compromised about 1 or 2 years ago and the regex had to be remade. > > I think having as little as possible of .* helps. Maybe an expert can > > comment on this? Isn't there a wiki page on how to create regex on the > > website of fail2ban? I can't seem to find anything. > > > > As for your regex, I can make it work like so below by just adding a few > > characters/words: > > > > $line= > > '2016/07/05 23:10:26 [error] 2359#0: *21 open() > > "/usr/share/nginx/html/wallpaper/technology/Rendered-Blue-Cubes-iPad-Wallpaper/1657" > > failed (2: No such file or directory), client: 198.143.46.17, server: _, > > request: "GET /wallpaper/technology/Rendered-Blue-Cubes-iPad-Wallpaper/1657 > > HTTP/1.1", host: "www.appleipadwallpapers.com"' > > > > $regex= > > '^.*, client: <HOST>, server: _, request:.*$' > > > > Or just copy and paste this: > > fail2ban-regex '2016/07/05 23:10:26 [error] 2359#0: *21 open() > > "/usr/share/nginx/html/wallpaper/technology/Rendered-Blue-Cubes-iPad-Wallpaper/1657" > > failed (2: No such file or directory), client: 198.143.46.17, server: _, > > request: "GET /wallpaper/technology/Rendered-Blue-Cubes-iPad-Wallpaper/1657 > > HTTP/1.1", host: "www.appleipadwallpapers.com"' '^.*, client: <HOST>, > > server: _, request:.*$' > > > > > > > > > > On Wed, Jul 6, 2016 at 4:15 PM, Alan Liddell <alan.c.liddell@...> > > wrote: > > > >> Hi all, > >> > >> I checked the GitHub and asked on IRC (nobody around at the time) and > >> couldn't find anything like this. I'm running fail2ban 0.9.3 on Fedora > >> 24, Python 2.7.11/3.5.1, trying to check Nginx error logs for bots. > >> Here's the line: > >> > >> $ line='2016/07/05 23:10:26 [error] 2359#0: *21 open() > >> > >> "/usr/share/nginx/html/wallpaper/technology/Rendered-Blue-Cubes-iPad-Wallpaper/1657" > >> failed (2: No such file or directory), client: 198.143.46.17, server: _, > >> request: "GET > >> /wallpaper/technology/Rendered-Blue-Cubes-iPad-Wallpaper/1657 HTTP/1.1", > >> host: "www.appleipadwallpapers.com"' > >> > >> Here's the regex: > >> > >> $ regex='^.*<HOST>.*$' > >> > >> This should be the most permissive possible regex on fail2ban, right? > >> But here's the output of fail2ban-regex: > >> > >> $ fail2ban-regex "$line" "$regex" > >> > >> Running tests > >> ============= > >> > >> Use failregex line : ^.*<HOST>.*$ > >> Use single line : 2016/07/05 23:10:26 [error] 2359#0: *21 open() > >> "/u... > >> > >> > >> Results > >> ======= > >> > >> Failregex: 0 total > >> > >> Ignoreregex: 0 total > >> > >> Date template hits: > >> |- [# of hits] date format > >> | [1] Year(?P<_sep>[-/.])Month(?P=_sep)Day > >> 24hour:Minute:Second(?:,Microseconds)? > >> `- > >> > >> Lines: 1 lines, 0 ignored, 0 matched, 1 missed [processed in 0.02 sec] > >> |- Missed line(s): > >> | 2016/07/05 23:10:26 [error] 2359#0: *21 open() > >> > >> "/usr/share/nginx/html/wallpaper/technology/Rendered-Blue-Cubes-iPad-Wallpaper/1657" > >> failed (2: No such file or directory), client: 198.143.46.17, server: _, > >> request: "GET > >> /wallpaper/technology/Rendered-Blue-Cubes-iPad-Wallpaper/1657 HTTP/1.1", > >> host: "www.appleipadwallpapers.com" > >> `- > >> > >> > >> (I'm new to fail2ban and I was worried my timestamp might have been > >> nonstandard, but does the bit under "Date template hits" mean that I'm > >> in the clear there?) By the way, fail2ban-testcases fails a few tests > >> related to this: > >> > >> Regex for filter 'nginx-botsearch' has no samples: 2: '^\\[error\\] > >> \\d+#\\d+: \\*\\d+ \\S+\\(\\) \\"\\S+\\" (failed|is not found) \\(2\\: > >> No such file or directory\\), client\\: > >> (?:::f{4,6}:)?(?P<host>[\\w\\-.^_]*\\w)\\, server\\: \\S*\\, request: > >> \\"(GET|POST|HEAD) \\/\\S+ \\S+\\"\\, .*?$' > >> > >> Regex for filter 'nginx-http-auth' has no samples: 1: '^ \\[error\\] > >> \\d+#\\d+: \\*\\d+ no user/password was provided for basic > >> authentication, client: (?:::f{4,6}:)?(?P<host>[\\w\\-.^_]*\\w), server: > >> \\S+, request: "\\S+ \\S+ HTTP/\\d+\\.\\d+", host: "\\S+"\\s*$' > >> > >> and so forth. Don't know if this specifically is relevant, but thought > >> I'd mention it. Thanks all. > >> > >> > >> > >> > >> > >> ------------------------------------------------------------------------------ > >> Attend Shape: An AT&T Tech Expo July 15-16. Meet us at AT&T Park in San > >> Francisco, CA to explore cutting-edge tech and listen to tech luminaries > >> present their vision of the future. This family event has something for > >> everyone, including kids. Get more information and register today. > >> http://sdm.link/attshape > >> _______________________________________________ > >> Fail2ban-users mailing list > >> Fail2ban-users@... > >> https://lists.sourceforge.net/lists/listinfo/fail2ban-users > >> > >>
signature.asc
Description: OpenPGP digital signature
------------------------------------------------------------------------------ Attend Shape: An AT&T Tech Expo July 15-16. Meet us at AT&T Park in San Francisco, CA to explore cutting-edge tech and listen to tech luminaries present their vision of the future. This family event has something for everyone, including kids. Get more information and register today. http://sdm.link/attshape
_______________________________________________ Fail2ban-users mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/fail2ban-users
