Re: Sites unreachable while traversing Dallas IXP

[Pedro Prado via NANOG <nanog () lists nanog org>]

IMHO, the key info here is that a known set of subnets was affected. This rules out some stuff:

- LACP manages link bundling, as in “can this interface be added to the bundle?”. The effect of bundling should be to 
have multiple links to choose from when egressing a packet. RFC7130 is a nice addition to bundles as it uses BFD to 
manage each link - meaning a bad member is removed quickly (LACP timers are not that fast and LACP itself is not 
designed to react fast).

- Hashing (which is used for load balancing traffic in hardware switches) is not managed by LACP - it’s always local to 
each device and as said before, each side usually has a different view of the ideal hashing. A classical example is 
when there are many IPs behind one firewall doing NAT - you can’t rely on diversity of IPs and MACs to select an egress 
link, so you usually change the hashing to be per port.

- Link errors would affect random traffic to any destination / from any source

So, none of the above technologies would affect traffic connectivity _selectively_. Perhaps a malformed bundle could 
blackhole traffic, but that wouldn’t be specific to certain subnets unless someone is *extremely* unlucky and _only_ 
his subnets hashed to the “bad bundle member” :)


It simply looks like a routing issue through this path. 
Perhaps the flapping of the BGP session re-advertised this path to some place that previously wasn’t using it, and 
apparently can't? 



Pedro Martins Prado
pedro.prado () gmail com / +353 83 036 1875

> On 28 Sep 2025, at 06:24, William Herrin via NANOG <nanog () lists nanog org> wrote:
>
> On Sat, Sep 27, 2025 at 7:31 PM Bruce Wainer via NANOG
> <nanog () lists nanog org> wrote:
> > Excuse my ignorance about this IXP and your equipment, but is
> > Micro-BFD (RFC 7130) supported? And if so, is it enabled or can you
> > enable it? While configuration wise it will use the single IP
> > addresses of the aggregate, separate BFD instances are set up for each
> > underlying link and will confirm whether Layer 3 is working on that
> > point-to-point connection.
>
> Hi Bruce,
>
> I'm also not familiar with this particular IXP but generally with IXPs
> we're not talking about point to point connections. The multiple
> participants' routers are part of a shared layer-2 fabric (a switch or
> switches) over which they trade layer-3 packets directly with each
> other. The route advertisements may transit the route servers but the
> routed packets do not.
>
> You can get into some really finicky errors where both participants
> successfully talk to the route server and thereby exchange routes, but
> for one reason or another can't get packets back and forth to each
> other. Bonded circuits (LAGs) add complexity which makes
> troubleshooting that much harder.
>
> If it were me, I would have considered building this connection
> differently. For speed, I'd have chosen a 100G link instead of two 10G
> links. Had my objective been reliability, I'd have built that at layer
> 3 instead of layer 2 -- two routers each with its own 10G link, and
> then done some balancing of the advertised routes. But in all fairness
> to Andy, I don't have anywhere near complete information here and the
> details matter a lot.
>
> Regards,
> Bill Herrin
>
> -- 
> William Herrin
> bill () herrin us
> https://bill.herrin.us/
> _______________________________________________
> NANOG mailing list 
> https://lists.nanog.org/archives/list/nanog () lists nanog org/message/RXKZDTZBXR4VTGE7R6E55FHAC7QEVDHY/

_______________________________________________
NANOG mailing list 
https://lists.nanog.org/archives/list/nanog () lists nanog org/message/PNQLZYKGZRP5HTTDWOYSRB4372XHXUWO/