AS2828 XO / Verizon Route Flapping?

[Stephen Frost via NANOG <nanog () nanog org>]

Greetings,

We're seeing a pretty consistent drop of traffic every 60s for about 5s
inside of AS2828 (formerly xo.net, now Verizon).

Started seeing alerts due to this at Sun, 06 Sep 2020 06:04:04 +0000.

During the 5s we're getting 'No route to host' bounces, eg:

connect to address 2a02:c0:301:0:ffff::31 and port 80: No route to host

MTR:

Keys:  Help   Display mode   Restart statistics   Order of fields   quit

                             Last 181 pings
 1. 2604:a580:2::2           
.....................................................................................................................................................................................
 2. 2604:a580::1:0:0:12      
.....................................................................................................................................................................................
 3. 2604:a580::1:0:0:d       
.....................................................................................................................................................................................
 4. 2610:18:18c:4000::5      
.....................................................................................................................................................................................
 5. 2610:18::2046            
.......................................?????..........................>......................?????......................................................................?????........
 6. 2001:7f8:4:1::847:1      
.......................................?????.................................................?????......................................................................?????.......?
 7. 2001:4600:0:100::29      
.......................>...............?????.................................................?????......................................................................?????.......
 8. 2001:4600:0:100::3       
.......................................?????.................................................?????......................................................................?????.......
 9. 2001:4600:0:100::2a      
.........>.....>.....>...............>?????.........................>.......................?????......................................>.............................>.?????........
10. 2001:4600:0:200::62      
...>..................................?????........>..........>..................>>.........?????.....>.......>............................>>...>......................?????........
11. 2001:4600:9:100::2e      
..............>.......................?????............................................>....?????.............................>...................>....................?????........
12. 2a02:c0:1:1::6           
......................................?????.................................................?????......................................................................?????........
13. 2a02:c0:1::4:1           
......................................?????.....?????????????????????????????????????????????????.....??????????????????????????????????????????????????????????????????????.....???
14. 2a02:c0:1:401::25        
......................................?????.................................................?????......................................................................?????........
15. 2a02:c0:301:0:ffff::34   
......................................?????.................................................?????......................................................................?????........

Scale:  .:3 ms  1:8 ms  2:14 ms  3:26 ms  a:56 ms  b:85 ms  c:137 ms

Traceroute:

traceroute to wrigleys (2a02:c0:301:0:ffff::34), 30 hops max, 80 byte packets
 1  2604:a580:2::2 (2604:a580:2::2)  0.357 ms  0.320 ms  0.243 ms
 2  2604:a580::1:0:0:12 (2604:a580::1:0:0:12)  0.641 ms  0.302 ms  0.271 ms
 3  2604:a580::1:0:0:d (2604:a580::1:0:0:d)  0.472 ms  0.427 ms  0.313 ms
 4  2610:18:18c:4000::5 (2610:18:18c:4000::5)  0.873 ms  0.906 ms  0.855 ms
 5  cir1.london2-eng.us.xo.net (2610:18::2046)  91.539 ms  91.471 ms  91.413 ms
 6  2001:7f8:4:1::847:1 (2001:7f8:4:1::847:1)  92.361 ms  92.570 ms  92.513 ms
 7  ti3004c400-lo0-0.ti.telenor.net (2001:4600:0:100::5)  123.541 ms  123.460 ms  123.388 ms
 8  ti0001c360-lo0-0.ti.telenor.net (2001:4600:0:100::23)  123.516 ms  123.474 ms  123.419 ms
 9  ti0275c360-lo0-0.ti.telenor.net (2001:4600:0:100::2)  123.305 ms  123.332 ms  123.264 ms
10  ti0169a400-lo0-0.ti.telenor.net (2001:4600:0:200::62)  129.801 ms  129.767 ms  129.717 ms
11  2001:4600:9:100::2e (2001:4600:9:100::2e)  128.751 ms  128.703 ms cir1.london2-eng.us.xo.net (2610:18::2046)  
91.658 ms !H

MTR going the other way:

regula (2a02:c0:301:0:ffff::38)                                                                                         
                                                                  2020-09-07T13:24:34+0000
Keys:  Help   Display mode   Restart statistics   Order of fields   quit

                             Last 181 pings
 1. 2a02:c0:301:0:ffff::1    
.....................................................................................................................................................................................
 2. 2a02:c0:1:401::22        
.....................................................................................................................................................................................
 3. 2a02:c0:1::1             
.....................................................................................................................................................................................
 4. 2a02:c0:1:1::7           
.....................................................................................................................................................................................
 5. 2001:4600:9:100::2d      
.....................................................................................................................................................................................
 6. 2001:4600:0:100::2       
.....................................................................................................................................................................................
 7. 2001:4600:0:100::23      
.....................................................................................................................................................................................
 8. 2001:4600:0:100::5       
.....................................................................................................................................................................................
 9. 2001:4600::4             
.....................................................................................................................................................................................
10. 2001:7f8:4:1::b0c:1      
...............................?????................................................?????......................................................................?????.................
11. 2610:18::30c4            
...............................?????................>...............................?????......................................................................?????.................
12. 2610:18:18c:4000::6      
...............................?????................................................?????......................................................................?????.................
13. 2604:a580::1:0:0:e       
..........>....................?????.>.........................>....................?????........................>............................>................?????...........>>..??
14. 2604:a580:2:0:5254:ff:fe 
...............................?????................................................?????......................................................................?????................

Anyone from Verizon watching who could look into this?  Anyone else
seeing similar issues?

Feel free to contact me off-list if that'd help.  We're also trying to
reach out to our provider to see if they can help.

Thanks!

Stephen

Attachment: signature.asc
Description: