nanog mailing list archives

Re: [CloudOps] [outages] Major loss/latency on Twelve99 around NY/SJO

From: Phil Lavin via NANOG <nanog () nanog org>
Date: Thu, 15 Aug 2024 11:09:29 +0100

Fwiw, still ongoing

I raised an AWS ticket and they’re looking into it. If there’s any friendly AWS Net Engineers on list, your help 
removing Twelve99 from route would be appreciated. Cross-posting to Nanog for wider reach.

On 15 Aug 2024, at 09:44, Matt Almgren <malmgren () backblaze com> wrote:

There's a planned SJC maintenance happening now. Expected end time is 2am PST. 

On Thu, Aug 15, 2024, 1:38 AM Phil Lavin via Outages <outages () outages org <mailto:outages () outages org>> wrote:

It’s a good week for the Internet. Third outages list post this week :(

Seeing major loss on Twelve99 around SJO:

UK to AWS USW2:

                                My traceroute  [v0.95]
VONMVPCVXYVFD (172.20.0.14) -> 216.147.3.209 (216.147.3.209) 2024-08-15T09:29:13+0100
Keys:  Help   Display mode   Restart statistics   Order of fields   quit
                                             Packets               Pings
 Host                                      Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. 172.20.0.1                              0.0%     9    1.1   1.2   1.0   1.7   0.2
 2. lo0-0.bng2.ixn-lon.zen.net.uk <http://lo0-0.bng2.ixn-lon.zen.net.uk/>           0.0%     9    5.5   6.8   5.4  
11.2   1.9
 3. lag-5.p2.ixn-lon.zen.net.uk <http://lag-5.p2.ixn-lon.zen.net.uk/>             0.0%     8    5.5   6.0   5.2   
6.5   0.5
 4. ldn-b13-link.ip.twelve99.net <http://ldn-b13-link.ip.twelve99.net/>           25.0%     8    5.8   5.5   5.0   
6.0   0.4
 5. ldn-bb2-link.ip.twelve99.net <http://ldn-bb2-link.ip.twelve99.net/>            0.0%     8    6.6   6.8   5.9   
7.5   0.7
 6. nyk-bb1-link.ip.twelve99.net <http://nyk-bb1-link.ip.twelve99.net/>            0.0%     8   75.2  75.2  74.5  
75.9   0.5
 7. sjo-b23-link.ip.twelve99.net <http://sjo-b23-link.ip.twelve99.net/>           57.1%     8  296.2 312.5 296.2 
343.7  27.0
 8. (waiting for reply)
 9. 15.230.28.67                           62.5%     8  294.4 295.7 294.4 296.8   1.2
10. 54.240.242.119                         25.0%     8  337.1 305.7 294.6 337.1  16.2
11. (waiting for reply)
12. 52.93.15.36                            57.1%     8  398.2 400.8 395.4 408.8   7.1
13. 52.93.15.31                            42.9%     8  388.7 395.5 388.7 403.3   6.2
14. 52.93.14.34                            28.6%     8  431.8 432.2 393.2 488.2  35.8
15. 52.93.246.177                          42.9%     8  397.7 399.3 397.7 401.1   1.4
16. 52.93.246.81                           25.0%     8  393.4 396.9 393.2 400.6   3.0
17. (waiting for reply)
18. (waiting for reply)19. (waiting for re 28.6%     8  394.3 397.5 394.3 400.2   3.0

AWS USW2 to UK:

 Host                                                       Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. ip-10-96-50-153.us-west-2.compute.internal               0.0%    87    0.2   0.1   0.1   0.2   0.0
 2. ec2-44-233-118-55.us-west-2.compute.amazonaws.com <http://ec2-44-233-118-55.us-west-2.compute.amazonaws.com/>    
    0.0%    87    7.3  11.9   1.0  66.9  16.4
    ec2-34-221-151-47.us-west-2.compute.amazonaws.com <http://ec2-34-221-151-47.us-west-2.compute.amazonaws.com/>
    ec2-44-233-117-187.us-west-2.compute.amazonaws.com <http://ec2-44-233-117-187.us-west-2.compute.amazonaws.com/>
 3. 240.0.212.4                                              0.0%    87    0.5   0.4   0.3   1.7   0.2
    240.0.72.7
 4. 242.2.110.95                                             0.0%    87    0.8   1.8   0.4  20.6   3.4
 5. 240.2.144.13                                             0.0%    87    5.8   5.8   5.7   5.9   0.1
 6. amazon-ic-375418.ip.twelve99-cust.net <http://amazon-ic-375418.ip.twelve99-cust.net/>                    0.0%    
87    5.7   5.6   5.6   6.1   0.1
 7. port-b4-link.ip.twelve99.net <http://port-b4-link.ip.twelve99.net/>                             0.0%    87    
6.2   6.1   5.9   6.5   0.1
 8. port-b3-link.ip.twelve99.net <http://port-b3-link.ip.twelve99.net/>                             0.0%    86    
5.6   6.3   5.6  18.5   1.8
 9. palo-b24-link.ip.twelve99.net <http://palo-b24-link.ip.twelve99.net/>                            0.0%    86   
21.4  21.3  21.1  21.6   0.1
10. nyk-bb2-link.ip.twelve99.net <http://nyk-bb2-link.ip.twelve99.net/>                            19.8%    86  
165.3 172.2 165.3 206.1   4.7
11. ldn-bb1-link.ip.twelve99.net <http://ldn-bb1-link.ip.twelve99.net/>                            68.2%    86  
391.5 401.2 387.3 411.8   6.3
12. ldn-b3-link.ip.twelve99.net <http://ldn-b3-link.ip.twelve99.net/>                             57.0%    86  397.1 
401.4 395.5 409.0   2.7
13. zeninternet-ic-363994.ip.twelve99-cust.net <http://zeninternet-ic-363994.ip.twelve99-cust.net/>              
40.7%    86  397.0 412.3 397.0 473.3  18.0
14. lag-1.bng2.ixn-lon.zen.net.uk <http://lag-1.bng2.ixn-lon.zen.net.uk/>                           63.5%    86  
387.2 402.5 382.5 422.8   9.0
15. 62-3-100-30.dsl.in-addr.zen.co.uk <http://62-3-100-30.dsl.in-addr.zen.co.uk/>                       55.8%    86  
391.6 404.5 391.6 412.5   6.1
16. (waiting for reply)
_______________________________________________
Outages mailing list
Outages () outages org <mailto:Outages () outages org>
https://puck.nether.net/mailman/listinfo/outages


This email, including its contents and any attachment(s), may contain confidential and/or proprietary information and 
is solely for the review and use of the intended recipient(s). If you have received this email in error, please 
notify the sender and permanently delete this email, its content, and any attachment(s).  Any disclosure, copying, or 
taking of any action in reliance on an email received in error is strictly prohibited.

Current thread:

Re: [CloudOps] [outages] Major loss/latency on Twelve99 around NY/SJO Phil Lavin via NANOG (Aug 15)
- Re: [CloudOps] [outages] Major loss/latency on Twelve99 around NY/SJO Phil Lavin via NANOG (Aug 15)