Home page logo
/

nanog logo nanog mailing list archives

Re: 30% packet loss between cox.net and hetzner.de, possibly at tinet.net
From: "Constantine A. Murenin" <mureninc () gmail com>
Date: Sat, 6 Apr 2013 16:20:42 -0700

On 2013-W14-5 21:27 -0700, Constantine A. Murenin wrote:
On 2013-W14-6 05:04 +0300, Denys Fedoryshchenko wrote:
On 2013-04-06 04:32, Constantine A. Murenin wrote:
Hello,

There has been at least a 25% packet loss between hetzner.de and
cox.net
in the last couple of hours.

Tried contacting hetzner.de, but they said it's not on their network.
This has already happened a couple of days ago, too (strangely, on
April 1),
but then was good for the rest of the week -- no problems whatsoever.

I wouldn't really care about this, if not for ssh:
it just doesn't work on such huge loss.

No other routes or networks seem affected.

Any advice?

Doesnt looks like tinet for me.

Might have been eurorings.net, as your Amazon EC2 to Hetzner 
traceroute seemed to suggest?


This loss was apparent even with their own main websites:


This is now happening again today, in a similar timeframe:


% mtr --report{,-wide,-cycles=60} --order "SRL BGAWV" hetzner.de ; date
HOST: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX    Snt   Rcv Loss%   Best Gmean   Avg  Wrst StDev
  4.|-- COX-68-12-10-121-static.coxinet.net    60    60  0.0%    8.8  12.3  12.9  43.4   5.4
  5.|-- COX-68-12-8-132-static.coxinet.net     60    60  0.0%   11.4  14.5  14.6  27.0   2.5
  6.|-- 68.1.5.161                             60    60  0.0%   56.1  60.4  60.5  74.0   4.0
  7.|-- nyk-s2-rou-1001.US.eurorings.net       60    60  0.0%   55.9  60.5  60.7  93.3   5.4
  8.|-- nntr-s1-rou-1101.FR.eurorings.net      60    60  0.0%  150.1 154.0 154.1 167.0   2.9
  9.|-- kehl-s2-rou-1103.DE.eurorings.net      60    60  0.0%  147.8 152.3 152.5 201.1   7.1
 10.|-- ffm-s1-rou-1102.DE.eurorings.net       60    60  0.0%  143.9 148.0 148.1 174.0   4.6
 11.|-- nbg-s1-rou-1001.DE.eurorings.net       60    60  0.0%  147.4 160.6 163.4 336.0  36.2
 12.|-- kpn-gw.hetzner.de                      60    51 15.0%  167.3 172.5 172.5 184.3   3.5
 13.|-- hos-bb2.juniper3.rz2.hetzner.de        60    60  0.0%  144.4 148.2 148.2 159.0   2.5
 14.|-- hos-tr4.ms-ex3k2.rz1.hetzner.de        60    59  1.7%  145.5 150.1 150.1 157.8   3.0
 15.|-- www.hetzner.de                         60    47 21.7%  167.0 173.2 173.3 203.2   7.1
Sat  6 Apr 2013 15:14:00 PDT

% traceroute -w2 -I hetzner.de; date
traceroute to hetzner.de (213.133.107.227), 64 hops max, 60 byte packets
 4  COX-68-12-10-121-static.coxinet.net (68.12.10.121)  22.012 ms  14.112 ms  11.230 ms
 5  COX-68-12-8-132-static.coxinet.net (68.12.8.132)  23.021 ms  13.239 ms  12.400 ms
 6  68.1.5.161 (68.1.5.161)  60.956 ms  59.738 ms  57.743 ms
 7  nyk-s2-rou-1001.US.eurorings.net (134.222.248.13)  58.906 ms  60.571 ms  70.399 ms
 8  nntr-s1-rou-1101.FR.eurorings.net (134.222.226.162)  152.597 ms  153.552 ms  150.250 ms
 9  kehl-s2-rou-1103.DE.eurorings.net (134.222.227.121)  152.905 ms  149.469 ms  150.648 ms
10  ffm-s1-rou-1102.DE.eurorings.net (134.222.227.177)  148.114 ms  144.579 ms  147.114 ms
11  nbg-s1-rou-1001.DE.eurorings.net (134.222.227.118)  148.728 ms  150.950 ms  148.117 ms
12  kpn-gw.hetzner.de (134.222.107.21)  170.385 ms *  167.488 ms
13  hos-bb2.juniper3.rz2.hetzner.de (213.239.240.129)  147.476 ms  149.074 ms  144.824 ms
14  hos-tr4.ms-ex3k2.rz1.hetzner.de (213.239.193.243)  161.621 ms  148.504 ms  147.931 ms
15  * www.hetzner.de (213.133.107.227)  169.137 ms  176.884 ms
Sat  6 Apr 2013 15:28:32 PDT


Cns# mtr --report{,-wide,-cycles=60} --order "SRL BGAWV" cox.net ; date
HOST: XXXXXXXXXX                                  Snt   Rcv Loss%   Best Gmean   Avg  Wrst StDev
  1.|-- static.33.203.4.46.clients.your-server.de    60    60  0.0%    0.6   1.2   1.5   3.8   1.1
  2.|-- hos-tr1.juniper1.rz13.hetzner.de             60    60  0.0%    0.2   0.3   1.8  34.6   5.8
  3.|-- hos-bb2.juniper4.rz2.hetzner.de              60    60  0.0%    2.7   3.2   5.2  77.2  10.9
  4.|-- r1nue1.core.init7.net                        60    60  0.0%    2.8   4.6   5.7  13.3   3.9
  5.|-- r1nue2.core.init7.net                        60    60  0.0%    2.9   4.1   4.7  13.9   2.9
  6.|-- r1fra2.core.init7.net                        60    60  0.0%    5.7   7.9   8.6  17.8   3.8
  7.|-- r1fra1.core.init7.net                        60    60  0.0%    5.9   8.9   9.8  17.1   4.3
  8.|-- xe-4-2-2.fra23.ip4.tinet.net                 60    60  0.0%    5.9   6.2   6.9  59.5   7.0
  9.|-- xe-10-2-2.was14.ip4.tinet.net                60    52 13.3%  115.3 119.8 120.1 163.4   9.1
 10.|-- cox-communications-gw.ip4.tinet.net          60    51 15.0%  119.5 123.7 124.0 175.8  10.3
 11.|-- dukedsrj02-ge210.0.rd.at.cox.net             60    48 20.0%  132.4 135.6 135.7 165.3   5.8
 12.|-- 68.1.15.238                                  60    47 21.7%  132.9 135.0 135.1 146.5   2.0
 13.|-- 68.99.123.4                                  60    33 45.0%  135.1 136.7 136.7 139.1   1.1
 14.|-- ww2.cox.com                                  60    50 16.7%  135.1 136.8 136.8 139.2   1.1
Sat Apr  6 15:13:49 PDT 2013

Cns# traceroute -w2 -I cox.net; date
traceroute to cox.net (68.99.123.161), 64 hops max, 60 byte packets
 1  static.33.203.4.46.clients.your-server.de (46.4.203.33)  0.790 ms  3.658 ms  0.486 ms
 2  hos-tr1.juniper1.rz13.hetzner.de (213.239.224.1)  0.244 ms  0.241 ms  0.251 ms
 3  hos-bb2.juniper4.rz2.hetzner.de (213.239.240.138)  2.735 ms  2.772 ms  2.752 ms
 4  r1nue1.core.init7.net (77.109.135.101)  2.811 ms  2.833 ms  2.769 ms
 5  r1nue2.core.init7.net (77.109.140.154)  2.955 ms  2.928 ms  2.896 ms
 6  r1fra2.core.init7.net (77.109.140.49)  14.891 ms  5.720 ms  13.291 ms
 7  r1fra1.core.init7.net (77.109.128.137)  5.935 ms  7.804 ms  5.905 ms
 8  xe-4-2-2.fra23.ip4.tinet.net (77.67.76.237)  5.885 ms  5.920 ms  5.912 ms
 9  xe-10-2-2.was14.ip4.tinet.net (141.136.110.18)  115.127 ms  121.543 ms *
10  cox-communications-gw.ip4.tinet.net (77.67.79.234)  119.574 ms  119.245 ms  119.170 ms
11  dukedsrj02-ge210.0.rd.at.cox.net (68.1.1.123)  132.619 ms  132.431 ms  132.363 ms
12  68.1.15.238 (68.1.15.238)  132.718 ms  132.689 ms  132.599 ms
13  68.99.123.4 (68.99.123.4)  134.947 ms *  134.671 ms
14  * ww2.cox.com (68.99.123.161)  135.139 ms  135 ms
Sat Apr  6 15:44:12 PDT 2013



It seems like hetzner.de <-> Amazon EC2 connectivity is indeed 
also affected, in addition to hetzner.de <-> cox.net:


Cns# mtr --report{,-wide,-cycles=60} --order "SRL BGAWV" www.tarsnap.com ; date
  2.|-- hos-tr3.juniper2.rz13.hetzner.de             60    60  0.0%    0.2   0.3   2.8  50.1   9.6
  3.|-- hos-bb2.juniper4.rz2.hetzner.de              60    60  0.0%    2.7   3.5   5.9  51.2  10.4
  4.|-- r1nue1.core.init7.net                        60    60  0.0%    2.8   4.7   5.6  14.2   3.5
  5.|-- r1nue2.core.init7.net                        60    60  0.0%    2.9   4.3   5.2  13.2   3.8
  6.|-- r1fra2.core.init7.net                        60    60  0.0%    5.7   8.8   9.7  18.4   4.6
  7.|-- r1fra1.core.init7.net                        60    60  0.0%    5.9   8.2   9.0  24.7   4.3
  8.|-- xe-4-2-2.fra23.ip4.tinet.net                 60    60  0.0%    5.9   6.4   7.0  43.7   5.4
  9.|-- xe-4-1-1.was14.ip4.tinet.net                 60    45 25.0%  115.1 117.3 117.6 173.2   9.6
 10.|-- vadata-gw.ip4.tinet.net                      60    50 16.7%  117.0 117.6 117.7 130.0   2.1
 11.|-- 72.21.220.31                                 60    50 16.7%  118.1 120.2 120.2 140.4   4.6
 12.|-- 205.251.245.55                               60    49 18.3%  120.2 121.1 121.1 132.0   2.1
 13.|-- ???                                          60     0 100.0    0.0   0.0   0.0   0.0   0.0
 14.|-- ???                                          60     0 100.0    0.0   0.0   0.0   0.0   0.0
 15.|-- ???                                          60     0 100.0    0.0   0.0   0.0   0.0   0.0
 16.|-- 216.182.224.129                              60    54 10.0%  118.2 118.9 118.9 130.3   1.7
 17.|-- ???                                          60     0 100.0    0.0   0.0   0.0   0.0   0.0
 18.|-- ???                                          60     0 100.0    0.0   0.0   0.0   0.0   0.0
 19.|-- ???                                          60     0 100.0    0.0   0.0   0.0   0.0   0.0
 20.|-- ec2-23-21-149-109.compute-1.amazonaws.com    59    53 10.2%  118.6 119.4 119.4 139.8   2.9
Sat Apr  6 15:21:06 PDT 2013



On the other hand, comcast.net is not affected, att.net not affected, 
he.net not affected, lots of other routes not affected.


Cns# mtr --report{,-wide,-cycles=60} --order "SRL BGAWV" comcast.net ; date
  2.|-- hos-tr3.juniper2.rz13.hetzner.de                      60    60  0.0%    0.2   0.3   3.9  56.6  11.6
  3.|-- hos-bb2.juniper8.rz1.hetzner.de                       60    60  0.0%    2.8   2.9   3.1  24.9   2.9
  4.|-- nbg-s1-rou-1001.DE.eurorings.net                      60    60  0.0%    3.0   3.6   7.3 177.0  24.2
  5.|-- mchn-s1-rou-1021.DE.eurorings.net                     60    60  0.0%    6.9   7.6   8.0  24.1   3.8
  6.|-- mchn-s1-rou-1103.DE.eurorings.net                     60    60  0.0%   91.8  93.7  93.9 133.0   6.7
  7.|-- kehl-s2-rou-1103.DE.eurorings.net                     60    60  0.0%   93.9  95.8  96.1 134.2   7.3
  8.|-- nntr-s1-rou-1101.FR.eurorings.net                     60    60  0.0%   91.8  94.2  94.3 127.6   5.3
  9.|-- nyk-s2-rou-1021.US.eurorings.net                      60    60  0.0%   92.1  95.7  96.0 128.3   8.2
 10.|-- nyk-s2-rou-1001.US.eurorings.net                      60    60  0.0%   91.7  93.2  93.7 188.9  12.6
 11.|-- te-1-10-0-4-pe01.111eighthave.ny.ibone.comcast.net    60    60  0.0%   91.7  92.3  92.3  93.7   0.3
 12.|-- pos-1-6-0-0-cr01.newyork.ny.ibone.comcast.net         60    60  0.0%   92.2  94.0  94.0  95.8   1.1
 13.|-- he-0-0-0-0-cr01.350ecermak.il.ibone.comcast.net       60    60  0.0%  111.5 117.1 117.2 123.3   3.6
 14.|-- pos-0-3-0-0-ar02.northlake.il.ndcchgo.comcast.net     60    60  0.0%  112.8 113.0 113.0 113.5   0.1
 15.|-- te-0-3-0-0-ur04.northlake.il.ndcchgo.comcast.net      60    60  0.0%  112.8 113.0 113.0 113.3   0.1
 16.|-- ge-0-1-0-0-ur03.northlake.il.ndcchgo.comcast.net      60    60  0.0%  112.9 113.0 113.0 113.3   0.1
 17.|-- ???                                                   60     0 100.0    0.0   0.0   0.0   0.0   0.0
Sat Apr  6 15:47:41 PDT 2013


Cns# mtr --report{,-wide,-cycles=60} --order "SRL BGAWV" att.net ; date
  2.|-- hos-tr3.juniper2.rz13.hetzner.de             60    60  0.0%    0.2   0.2   1.0  45.2   5.8
  3.|-- hos-bb2.juniper4.rz2.hetzner.de              60    60  0.0%    2.7   3.7   8.0  86.2  17.5
  4.|-- ae55.edge7.Frankfurt1.Level3.net             60    60  0.0%    7.2   7.9   8.9  57.8   7.9
  5.|-- ???                                          60     0 100.0    0.0   0.0   0.0   0.0   0.0
  6.|-- ae-62-62.ebr2.Frankfurt1.Level3.net          60    60  0.0%   85.3  85.3  85.3  87.2   0.3
  7.|-- ae-23-23.ebr2.London1.Level3.net             60    60  0.0%   85.3  85.3  85.3  85.4   0.0
  8.|-- ae-43-43.ebr1.NewYork1.Level3.net            60    60  0.0%   85.4  85.5  85.5  88.6   0.4
  9.|-- ae-71-71.csw2.NewYork1.Level3.net            60    60  0.0%   85.4  87.6  87.7  97.7   3.9
 10.|-- ae-2-70.edge3.NewYork1.Level3.net            60    60  0.0%   85.5  87.5  88.0 154.7  10.9
 11.|-- att-level3.newyork1.level3.net               60    60  0.0%   86.5  88.3  88.3  91.8   1.2
 12.|-- cr2.n54ny.ip.att.net                         60    60  0.0%  111.5 116.7 116.8 124.9   4.9
 13.|-- cr2.wswdc.ip.att.net                         60    60  0.0%  110.1 115.4 115.5 123.9   5.1
 14.|-- cr1.attga.ip.att.net                         60    60  0.0%  111.3 116.7 116.8 124.9   4.8
...
Sat Apr  6 15:36:40 PDT 2013


Cns# mtr --report{,-wide,-cycles=60} --order "SRL BGAWV" he.net ; date
  2.|-- hos-tr3.juniper2.rz13.hetzner.de             60    60  0.0%    0.2   0.3   4.2  71.3  13.9
  3.|-- hos-bb2.juniper4.ffm.hetzner.de              60    60  0.0%    5.8   5.9   6.0   9.9   0.7
  4.|-- 30gigabitethernet4-3.core1.fra1.he.net       60    60  0.0%    5.7   8.3   8.8  17.6   3.3
  5.|-- 10gigabitethernet1-4.core1.par2.he.net       60    60  0.0%   15.1  18.1  18.5  26.0   3.7
  6.|-- 10gigabitethernet7-1.core1.ash1.he.net       60    60  0.0%   93.1  95.9  95.9 105.7   3.5
  7.|-- 10gigabitethernet11-1.core1.pao1.he.net      60    60  0.0%  163.4 167.5 167.6 177.0   4.1
  8.|-- 10gigabitethernet1-2.core1.fmt1.he.net       60    60  0.0%  164.5 170.2 170.5 248.5  12.6
  9.|-- he.net                                       60    59  1.7%  164.0 164.6 164.6 165.2   0.2
Sat Apr  6 15:52:46 PDT 2013



Although hetzner.de claims that this whole loss is outside of their own 
network, I'm inclined to deduce that the loss might actually be 
concentrated on their own KPN / eurorings.net router -- 
kpn-gw.hetzner.de (134.222.107.21), and perhaps occurs only 
in one direction.

Although there is no traffic loss from he.net if you try to traceroute 
the router itself (I'm not sure what that means, though, other than a 
potential attack vector from exposing a router globally like that):

# mtr --report{,-wide,-cycles=60} --order "SRL BGAWV" -4 134.222.107.21 ; date
HOST: XXXXXXXXX                                Snt   Rcv Loss%   Best Gmean   Avg  Wrst StDev
  1. router1-fmt.linode.com                    60    60  0.0%    0.5   1.5   4.8  70.2  10.6
  2. 10gigabitethernet2-3.core1.fmt1.he.net    60    60  0.0%    0.6   1.0   3.5  72.1  11.9
  3. 10gigabitethernet1-1.core1.pao1.he.net    60    60  0.0%    1.0   1.2   1.4  12.4   1.5
  4. eqix-sv8.kpn.com                          60    60  0.0%    1.8   2.5   5.9  77.9  15.2
  5. chg-s1-rou-1041.US.eurorings.net          60    60  0.0%  161.5 167.1 167.3 197.9   7.5
  6. ahbn-s1-rou-1041.US.eurorings.net         60    60  0.0%   71.5  74.0  74.3 108.9   7.5
  7. nyk-s2-rou-1021.US.eurorings.net          60    60  0.0%   71.6  75.7  76.2 131.7  10.3
  8. nntr-s1-rou-1101.FR.eurorings.net         60    60  0.0%  161.6 164.5 164.6 188.8   6.8
  9. kehl-s2-rou-1103.DE.eurorings.net         60    60  0.0%  161.5 162.9 163.0 206.3   6.0
 10. ffm-s1-rou-1102.DE.eurorings.net          60    60  0.0%  157.9 166.9 167.1 193.8   9.4
 11. nbg-s1-rou-1001.DE.eurorings.net          60    60  0.0%  161.6 163.6 163.9 235.2  10.3
 12. kpn-gw.hetzner.de                         60    59  1.7%  161.3 161.9 161.9 167.5   1.0
Sat Apr  6 15:58:04 PDT 2013

li163-159:~# mtr --report{,-wide,-cycles=60} --order "SRL BGAWV" -4 hetzner.de ; date
HOST: xxxxxxxxx                                Snt   Rcv Loss%   Best Gmean   Avg  Wrst StDev
  1. router1-fmt.linode.com                    60    60  0.0%    0.5   8.6  34.8 396.6  66.0
  2. 10gigabitethernet2-3.core1.fmt1.he.net    60    60  0.0%    0.6   2.7   6.7  75.1  12.2
  3. 10gigabitethernet1-1.core1.pao1.he.net    60    60  0.0%    0.9   2.2   3.8  22.1   4.3
  4. 10gigabitethernet9-3.core1.ash1.he.net    60    60  0.0%   71.2  74.9  75.1  88.7   4.9
  5. 10gigabitethernet1-2.core1.par2.he.net    60    60  0.0%  149.0 151.9 152.0 159.8   3.5
  6. 10gigabitethernet4-4.core1.fra1.he.net    60    60  0.0%  158.2 161.3 161.4 169.1   3.8
  7. decix-gw.hetzner.de                       60    60  0.0%  158.6 161.2 161.4 209.3   8.9
  8. hos-bb1.juniper3.rz2.hetzner.de           60    60  0.0%  161.8 173.2 174.0 220.2  18.4
  9. hos-tr3.ms-ex3k2.rz1.hetzner.de           60    60  0.0%  162.6 163.7 163.7 166.0   1.1
 10. www.hetzner.de                            60    59  1.7%  161.8 162.4 162.4 163.1   0.3
Sat Apr  6 16:14:48 PDT 2013



I've been a fan of hetzner.de, but I think it's staggering that 
they won't do anything about this huge and persistent packet loss.


Best regards,
Constantine.


  By Date           By Thread  

Current thread:
[ Nmap | Sec Tools | Mailing Lists | Site News | About/Contact | Advertising | Privacy ]
AlienVault