On Thu, Nov 01, 2018 at 09:46:51AM +0900, Mike Hommey wrote:
> - Looking back at the logs from all the jobs we've had in the past
> failing to reach snapshot.debian.org (or at least, marked as such),
> the IP addresses of the hosts they were running on (as well as the IP
> address of the host I had direct access to and that couldn't connect
> to snapshot.debian.org) were all in the 18.128.0.0/9 block[1].
Traceroute in the other direction (snapshot->18.128/9) looks roughly
like:
lw07% ip -4 ro get 18.213.145.171
18.213.145.171 via 185.17.185.190 dev eth0 src 185.17.185.187
cache
lw07% traceroute !$
traceroute 18.213.145.171
traceroute to 18.213.145.171 (18.213.145.171), 30 hops max, 60 byte packets
1 ge-9-26.ce39.ams-01.nl.leaseweb.net (185.17.185.190) 0.492 ms 0.729 ms
0.782 ms
2 xe-11-3-3.br01.ams-01.nl.leaseweb.net (81.17.33.94) 0.823 ms
xe-2-3-7.br01.ams-01.nl.leaseweb.net (81.17.33.92) 0.290 ms
xe-11-3-3.br01.ams-01.nl.leaseweb.net (81.17.33.94) 0.790 ms
3 ix-xe-5-1-3-0.thar1.hnn-haarlem.as6453.net (195.219.162.73) 0.765 ms
ix-xe-3-3-2-0.thar1.hnn-haarlem.as6453.net (195.219.162.105) 1.380 ms 1.387 ms
4 if-ae-10-2.tcore2.av2-amsterdam.as6453.net (80.231.205.10) 125.026 ms
if-ae-4-2.tcore1.av2-amsterdam.as6453.net (80.231.205.14) 124.579 ms
if-ae-2-2.tcore1.av2-amsterdam.as6453.net (80.231.205.34) 124.774 ms
5 if-ae-2-2.tcore2.av2-amsterdam.as6453.net (195.219.194.6) 132.475 ms
if-ae-14-2.tcore2.l78-london.as6453.net (80.231.131.160) 125.014 ms 124.986 ms
6 if-ae-2-2.tcore1.l78-london.as6453.net (80.231.131.2) 123.790 ms 124.248
ms 123.840 ms
7 if-ae-15-2.tcore3.njy-newark.as6453.net (80.231.130.26) 127.375 ms
127.301 ms 135.540 ms
8 if-ae-1-3.tcore4.njy-newark.as6453.net (216.6.57.6) 127.101 ms
if-ae-15-2.tcore3.njy-newark.as6453.net (80.231.130.26) 127.355 ms 124.983 ms
9 if-ae-12-2.tcore2.aeq-ashburn.as6453.net (216.6.87.222) 124.540 ms
if-ae-12-2.tcore2.aeq-ashburn.as6453.net (216.6.87.200) 126.552 ms
if-ae-1-3.tcore4.njy-newark.as6453.net (216.6.57.6) 124.612 ms
10 if-ae-12-2.tcore2.aeq-ashburn.as6453.net (216.6.87.200) 124.378 ms
if-ae-12-2.tcore2.aeq-ashburn.as6453.net (216.6.87.222) 128.371 ms
if-ae-12-2.tcore2.aeq-ashburn.as6453.net (216.6.87.200) 126.738 ms
11 if-ae-37-3.tcore1.dt8-dallas.as6453.net (66.198.154.69) 125.113 ms
if-ae-2-2.tcore1.aeq-ashburn.as6453.net (216.6.87.2) 126.191 ms 127.846 ms
12 if-ae-37-3.tcore1.dt8-dallas.as6453.net (66.198.154.69) 125.003 ms
216.6.53.53 (216.6.53.53) 156.015 ms if-ae-37-3.tcore1.dt8-dallas.as6453.net
(66.198.154.69) 129.241 ms
13 54.239.105.121 (54.239.105.121) 113.918 ms 54.239.105.115 (54.239.105.115)
115.015 ms 216.6.53.53 (216.6.53.53) 155.924 ms
14 54.239.105.119 (54.239.105.119) 114.677 ms 54.239.105.127 (54.239.105.127)
113.755 ms 54.239.105.125 (54.239.105.125) 117.747 ms
15 176.32.125.157 (176.32.125.157) 122.992 ms * 176.32.125.195
(176.32.125.195) 123.441 ms
16 52.93.129.235 (52.93.129.235) 106.885 ms * 52.93.129.255 (52.93.129.255)
122.297 ms
17 54.239.42.141 (54.239.42.141) 110.397 ms 72.21.222.251 (72.21.222.251)
107.358 ms 178.236.3.31 (178.236.3.31) 109.463 ms
18 * * *
19 * * *
20 54.239.111.156 (54.239.111.156) 119.706 ms 54.239.110.134 (54.239.110.134)
112.157 ms *
21 54.239.110.217 (54.239.110.217) 117.407 ms 54.239.110.149 (54.239.110.149)
125.869 ms 54.239.110.172 (54.239.110.172) 112.527 ms
22 54.239.111.23 (54.239.111.23) 109.229 ms 52.93.25.122 (52.93.25.122)
108.449 ms 54.239.110.131 (54.239.110.131) 114.695 ms
23 52.93.27.215 (52.93.27.215) 154.619 ms 54.239.111.21 (54.239.111.21)
108.260 ms 54.239.108.199 (54.239.108.199) 111.625 ms
24 205.251.244.81 (205.251.244.81) 110.742 ms 72.21.197.19 (72.21.197.19)
108.292 ms 52.93.24.7 (52.93.24.7) 107.111 ms
25 72.21.197.249 (72.21.197.249) 108.529 ms 52.93.24.5 (52.93.24.5) 112.327
ms 72.21.197.241 (72.21.197.241) 107.492 ms
26 * * *
27 * * *
28 * * *
29 * * *
30 * * *
I can confirm that the traceroute traffic does reach the ec2 host:
admin@ip-172-31-16-139:~$ ec2metadata --public-ipv4
18.213.145.171
admin@ip-172-31-16-139:~$ sudo tcpdump -np -i eth0 not tcp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
03:45:49.440533 IP 185.17.185.187.47396 > 172.31.16.139.33522: UDP, length 32
03:45:49.440569 IP 172.31.16.139 > 185.17.185.187: ICMP 172.31.16.139 udp port
33522 unreachable, length 68
03:45:49.445306 IP 185.17.185.187.43076 > 172.31.16.139.33523: UDP, length 32
03:45:49.445321 IP 172.31.16.139 > 185.17.185.187: ICMP 172.31.16.139 udp port
33523 unreachable, length 68
03:45:52.431712 ARP, Request who-has 172.31.16.139 tell 172.31.16.1, length 28
03:45:52.431733 ARP, Reply 172.31.16.139 is-at 0e:d7:93:4b:1c:8c, length 28
>From the original report, we know that the traffic is leaving the Amazon
network. Could we get someone from leaseweb to check for ingres filters
that could be impacting this traffic?
noah
