Hi. Please do not top post. This is a mailing list, not a corporate e-mail spamfest.
On Wed, Aug 29, 2018 at 01:18:52PM -0500, Nicholas Geovanis wrote: > Hi, I've never used OVH. How certain are you that the e1000 network driver > is the correct one? > Under VMWare/ESX the network driver choice can be crucial, for example. 1) You cannot run Xen in VSphere/ESXi. 2) No sane public provider will use VSphere/ESXi for hosting, the costs can dim a budget of a small country. 3) e1000e may be bad, but vmxnet3 will make oom-killer an everyday reality. SR-IOV is a way to go. Now, to the issue at hand. > On Wed, Aug 29, 2018 at 6:30 AM Kevin DAGNEAUX <kevin.dagne...@fiitelcom.fr> > wrote: Please note that you seem to have "link down" first: > > Aug 28 15:50:32 ovh-1 kernel: e1000e: enp1s0 NIC Link is Down and 'malfunctioning' netfilter rules next. > > Aug 28 15:50:32 ovh-1 kernel: DROPED packets IN=enp1s0 OUT= > > MAC=ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ SRC=YY.YY.YY.YY > > DST=XX.XX.XX.XX LEN=40 TOS=0x00 PREC=0x00 TTL=55 ID=2229 DF PROTO=TCP > > SPT=9610 DPT=80 WINDOW=0 RES=0x00 RST URGP=0 Observe 'RST' flag at each and every 'DROPPED' message. I find it highly unlikely that whatever PHP load test you did it involved sending mass amounts of TCP Reset packets. It seems that the following scenario has much higher probability: 1) You fired your load generator application. 2) Your hosting provided immediately got a signal of your typical DOS attack (not to be confused with DDOS) coming from that seems to be a typical mobile phone. 3) DDOS protection kicked in: a) Isolating your server from the network to stop DOS. b) Sending forged TCP RST to your server to break existing connections *and* termninate unneeded Apache workers (or whatever you have there). c) Banning the initiator of DOS (i.e. you on mobile network) temporarily. 4) Real network outage of your server was 2 seconds (time between "link down" and "link up"). Reco > > > > Hi, > > > > I've a server in OVH datacenter, on this server i've 7 VMs, on 1 of them > > in run Apache. > > To debug a slow upload (who was ~2Mo/s instead 12Mo/s) i've installed an > > HTML5/PHP speed test application. > > When i use this app, i've no problem in general, but, when a make a speed > > test from a source who have more bandwith than the server (the server is > > limited at 100Mb/s by OVH and i make the test from a 4G+ network where i've > > ~150Mb/s of bandwith), in this case, the DOM0 lost his network connection > > (like the ethernet cable is unplugged) until i reboot the server. > > > > When i check the syslog of DOM0, i see that iptables drop incomming packet > > on port 80 instead of routing them to the VM. > > > > This is my iptables script i use on DOM0 : > > > > #!/bin/bash > > > > IPT="/sbin/iptables" > > > > > > ########################################################################################### > > # Filter > > > > ## Remise par defaut des regles > > $IPT -t filter -P INPUT ACCEPT > > $IPT -t filter -P FORWARD ACCEPT > > $IPT -t filter -P OUTPUT ACCEPT > > > > ## On purge les tables > > $IPT -t filter -F > > > > ## On autorise lo > > $IPT -t filter -A INPUT -i lo -j ACCEPT > > > > ## On ouvre les ports nécéssaires au DOM0 > > $IPT -t filter -A INPUT -m tcp -p tcp --dport 22 -j > > ACCEPT ## SSH > > $IPT -t filter -A INPUT -m udp -p udp --dport 53 -j > > ACCEPT ## DNS > > $IPT -t filter -A INPUT -m icmp -p icmp --icmp-type 8 -j > > ACCEPT ## Ping > > $IPT -t filter -A INPUT -s 10.0.0.0/24 -j ACCEPT > > > > ## On accepte si la connexion est déjà établie > > $IPT -t filter -A INPUT -m conntrack --ctstate ESTABLISHED -j ACCEPT > > > > ## On log ce qui n'a pas été matché par les règles précédente > > $IPT -A INPUT -p tcp -j LOG --log-prefix "DROPED packets " > > > > ## On bloque tout le reste > > $IPT -t filter -P INPUT DROP > > > > > > ############################################################################################ > > # Nat > > > > ## Remise par defaut des regles > > $IPT -t nat -P PREROUTING ACCEPT > > $IPT -t nat -P POSTROUTING ACCEPT > > $IPT -t nat -P INPUT ACCEPT > > $IPT -t nat -P OUTPUT ACCEPT > > > > ## On purge > > $IPT -t nat -F > > > > ### Routage des ports entrants pour la VM "mails" > > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 22030 -j DNAT --to > > 10.0.0.30:22 ## SSH > > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 25 -j DNAT --to > > 10.0.0.30:25 ## SMTP > > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 587 -j DNAT --to > > 10.0.0.30:587 ## SMTP SUBMISSION > > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 465 -j DNAT --to > > 10.0.0.30:465 ## SMTP SSL > > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 143 -j DNAT --to > > 10.0.0.30:143 ## IMAP > > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 993 -j DNAT --to > > 10.0.0.30:993 ## IMAP SSL > > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 9930 -j DNAT --to > > 10.0.0.30:9930 ## IMAP SSL > > > > ### Routage des ports entrants pour la VM "sql" > > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 22020 -j DNAT --to > > 10.0.0.20:22 ## SSH > > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 3306 -j DNAT --to > > 10.0.0.20:3306 ## MariaDB > > > > ### Routage des ports entrants pour la VM "files" > > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 22010 -j DNAT --to > > 10.0.0.10:22 ## SSH > > > > ### Routage des ports entrant pour la VM "web" > > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 22040 -j DNAT --to > > 10.0.0.40:22 ## SSH > > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 80 -j DNAT --to > > 10.0.0.40:80 ## HTTP > > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 443 -j DNAT --to > > 10.0.0.40:443 ## HTTPS > > > > ### Routage des ports entrants pour la VM "monitor" > > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 22050 -j DNAT --to > > 10.0.0.50:22 ## SSH > > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 850 -j DNAT --to > > 10.0.0.50:80 ## HTTP > > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 855 -j DNAT --to > > 10.0.0.50:443 ## HTTPS > > > > ### Routage des ports entrants pour la VM "comm" > > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 22060 -j DNAT --to > > 10.0.0.60:22 ## SSH > > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 5222 -j DNAT --to > > 10.0.0.60:5222 ## Jabber > > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 5269 -j DNAT --to > > 10.0.0.60:5269 ## Jabber > > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 5280 -j DNAT --to > > 10.0.0.60:5280 ## Jabber > > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 5281 -j DNAT --to > > 10.0.0.60:5281 ## Jabber > > > > ### Routage des ports entrants pour la VM "secure" > > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 22070 -j DNAT --to > > 10.0.0.70:22 ## SSH > > > > ### Routage des ports entrants pour la VM "net" > > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 22080 -j DNAT --to > > 10.0.0.80:22 ## SSH > > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 8388 -j DNAT --to > > 10.0.0.80:8388 ## shadowsocks > > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p udp --dport 8388 -j DNAT --to > > 10.0.0.80:8388 ## shadowsocks > > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 11094 -j DNAT --to > > 10.0.0.80:1194 ## OpenVPN > > > > ### Autorise les VMs a accéder a internet > > $IPT -t nat -A POSTROUTING -s 10.0.0.0/24 -j MASQUERADE > > > > And this is what i have in syslogs : > > > > Aug 28 15:50:32 ovh-1 kernel: DROPED packets IN=enp1s0 OUT= > > MAC=ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ SRC=YY.YY.YY.YY > > DST=XX.XX.XX.XX LEN=40 TOS=0x00 PREC=0x00 TTL=55 ID=2226 DF PROTO=TCP > > SPT=9610 DPT=80 WINDOW=0 RES=0x00 RST URGP=0 > > Aug 28 15:50:32 ovh-1 kernel: DROPED packets IN=enp1s0 OUT= > > MAC=ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ SRC=YY.YY.YY.YY > > DST=XX.XX.XX.XX LEN=40 TOS=0x00 PREC=0x00 TTL=55 ID=2227 DF PROTO=TCP > > SPT=9610 DPT=80 WINDOW=0 RES=0x00 RST URGP=0 > > Aug 28 15:50:32 ovh-1 kernel: DROPED packets IN=enp1s0 OUT= > > MAC=ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ SRC=YY.YY.YY.YY > > DST=XX.XX.XX.XX LEN=40 TOS=0x00 PREC=0x00 TTL=55 ID=2228 DF PROTO=TCP > > SPT=9610 DPT=80 WINDOW=0 RES=0x00 RST URGP=0 > > Aug 28 15:50:32 ovh-1 kernel: DROPED packets IN=enp1s0 OUT= > > MAC=ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ SRC=YY.YY.YY.YY > > DST=XX.XX.XX.XX LEN=40 TOS=0x00 PREC=0x00 TTL=55 ID=2229 DF PROTO=TCP > > SPT=9610 DPT=80 WINDOW=0 RES=0x00 RST URGP=0 > > > > *** I've several hundred of similare line where only the ID change, then : > > *** > > > > Aug 28 15:50:32 ovh-1 kernel: e1000e: enp1s0 NIC Link is Down > > Aug 28 15:50:32 ovh-1 systemd-networkd[20998]: enp1s0: Lost carrier > > Aug 28 15:50:34 ovh-1 systemd-networkd[20998]: enp1s0: Gained carrier > > Aug 28 15:50:34 ovh-1 kernel: e1000e: enp1s0 NIC Link is Up 100 Mbps Full > > Duplex, Flow Control: None > > Aug 28 15:50:34 ovh-1 kernel: e1000e 0000:01:00.0 enp1s0: 10/100 speed: > > disabling TSO > > > > The log show that the network is UP but the server is offline. > > In the VM's logs i don't have abnormal things. > > > > OVH have changed the motherboard and the ethernet cable but the problem > > still persist, the syslog show that it's probably a software problem. > > > > Do you have idea of what happen or a way to help me to debug this problem? > > > > Thank you in advance. > > > > Kevin > > -- > > > > DAGNEAUX Kevin > > Service informatique > > 03 29 36 88 85 > > kevin.dagne...@fiitelcom.fr > >
