Yes guys, I know that Bugzilla is down. Last night, while I was sleeping, we got a slew of IPs hitting the dependency graph generation. This wouldn't have been a problem normally, but they seemed to hit graphs that took an inordinate amount of memory to generate with GraphViz (collectively 8GiB of RAM and 32GiB of swap). This morning, I got into the box, saw some OOMs of the GraphViz processes, killed off the remaining ones, banned the IPs, and then had to rush off to a work meeting.
I came back this evening, to find the box not responding again, and my last SSH shell was painfully slow then just hung - not died, but hung, the TCP connection is still alive, but the shell isn't responding (shortly after I had seen a loadavg exceeding 1k). Whomever attacked it came back I think. And I can't get in to block them right now. I've contacted the sponsor so that they can hard reboot the box for me, but I don't expect any action from them for the next 5-6 hours at least. Meantime, I'm enacting a plan B, to at least get us some slow Bugzilla functionality, via the second bugzilla box that normally runs the background computations (duplicates etc). I do however fully expect whomever the attacker is to come right back at it - so I'm turning off the dependency graphs. -- Robin Hugh Johnson Gentoo Linux Developer & Infra Guy E-Mail : robb...@gentoo.org GnuPG FP : 11AC BA4F 4778 E3F6 E4ED F38E B27B 944E 3488 4E85
pgp99tTF8eFfT.pgp
Description: PGP signature
