Re: Distributed Router Fabrics

[Phil Bedard <bedard.phil () gmail com>]

All major vendors espouse both chassis and fabrics depending on what you are doing.  I’m typically more of a fan of 
fabric based models but as others mentioned depends what you are doing.   When I say fabric I mean something using 
Ethernet and a standard control plane, not proprietary interconnects and fabric encapsulation which is what you see in 
some BRCM based solutions.  Those are basically virtual/multi-chassis systems, which also have their own pros and cons. 
  What vendor/box you choose is mostly dependent on the feature set you need.

There is a giant list of pros and cons between traditional chassis and distributed “fabrics” but here are a few.

Management and control plane scale can be an issue until that gets figured out.  Doing a 1:1 replacement of traditional 
large chassis with fabrics can add a lot of routers to a network.

Power depends on the chassis and power distribution design.  However, you can build a fabric as you need, it doesn’t 
require the power day one like a traditional chassis.

Upgrading chassis switch fabrics and moving/mixing generations of line cards is almost always a painful experience.

Phil

From: NANOG <nanog-bounces+bedard.phil=gmail.com () nanog org> on behalf of David Sinn <dsinn () uw edu>
Date: Tuesday, December 24, 2024 at 14:12
To: Mike Hammett <nanog () ics-il net>
Cc: NANOG <nanog () nanog org>
Subject: Re: Distributed Router Fabrics
From experience I can tell you that once you fully operationalize the pizza box model you will never go back to the 
chassis model. Why would you trade, open, standards based model for interconnect (OSPF and BGP work great at scale) for 
proprietary black boxes that do stupid router tricks to make a bunch of discrete components pretend to be one along 
with giving you the benefit of a huge blast-radius when the software inevitably breaks? Distributed ARP/ND, solved. 
Actually distributed BFD (not "it's all running on one line card because customers like LACP bundles spread between 
line cards and that's really hard to distribute reliably), solved. Pizza box models means the boxes are fungible. So 
you can competitively bid between multiple suppliers and pick and choose who you want to buy from depending on what is 
the most important thing at the time (delivery dates? price? which of them is annoying you the least at that moment in 
time?). They are also infinitely more scaleable(*) than any big chassis model. State of the art 5 years ago had 
Internet edge systems deploying with 8k of 400G ports and datacenter deployments with 65k 400G ports using the same 
fundamental design.

The real downside: vendors don't like the flexibility that it affords the customer and the meaninglessness of 
differentiation between vendors that it drives the operators to avoid.

David

(*) - Among some of the critical things to get right from the outset is what peak scale you want to have for the fabric 
because recabling is not something to be taken lightly...



On Dec 23, 2024, at 7:15 AM, Mike Hammett <nanog () ics-il net> wrote:

"Obviously, at some point, buying a big chassis..."

Actually, as I read more about it and watch more videos about it, it seems like that isn't necessarily true. The claims 
they have at the top end surpass what any chassis platform I've seen is capable of, though I don't know that they 
actually have pushed the upper bounds of what's possible in the real world.


-----
Mike Hammett
Intelligent Computing 
Solutions<https://urldefense.com/v3/__http:/www.ics-il.com/__;!!K-Hz7m0Vt54!iatSqn4yHM3x91laULnIOllnvAEqZBClRhLXcW0CJw8ZrVGDtYCggOnkHCfTy97OA0E8DBeZtQ$>
[Image removed by 
sender.]<https://urldefense.com/v3/__https:/www.facebook.com/ICSIL__;!!K-Hz7m0Vt54!iatSqn4yHM3x91laULnIOllnvAEqZBClRhLXcW0CJw8ZrVGDtYCggOnkHCfTy97OA0HlU5g15g$>[Image
 removed by 
sender.]<https://urldefense.com/v3/__https:/plus.google.com/*IntelligentComputingSolutionsDeKalb__;Kw!!K-Hz7m0Vt54!iatSqn4yHM3x91laULnIOllnvAEqZBClRhLXcW0CJw8ZrVGDtYCggOnkHCfTy97OA0Fd4VS5Aw$>[Image
 removed by 
sender.]<https://urldefense.com/v3/__https:/www.linkedin.com/company/intelligent-computing-solutions__;!!K-Hz7m0Vt54!iatSqn4yHM3x91laULnIOllnvAEqZBClRhLXcW0CJw8ZrVGDtYCggOnkHCfTy97OA0HbHzd3kA$>[Image
 removed by 
sender.]<https://urldefense.com/v3/__https:/twitter.com/ICSIL__;!!K-Hz7m0Vt54!iatSqn4yHM3x91laULnIOllnvAEqZBClRhLXcW0CJw8ZrVGDtYCggOnkHCfTy97OA0Fj7RvhHQ$>
Midwest Internet 
Exchange<https://urldefense.com/v3/__http:/www.midwest-ix.com/__;!!K-Hz7m0Vt54!iatSqn4yHM3x91laULnIOllnvAEqZBClRhLXcW0CJw8ZrVGDtYCggOnkHCfTy97OA0Gzrk6nZQ$>
[Image removed by 
sender.]<https://urldefense.com/v3/__https:/www.facebook.com/mdwestix__;!!K-Hz7m0Vt54!iatSqn4yHM3x91laULnIOllnvAEqZBClRhLXcW0CJw8ZrVGDtYCggOnkHCfTy97OA0F4YUnLMg$>[Image
 removed by 
sender.]<https://urldefense.com/v3/__https:/www.linkedin.com/company/midwest-internet-exchange__;!!K-Hz7m0Vt54!iatSqn4yHM3x91laULnIOllnvAEqZBClRhLXcW0CJw8ZrVGDtYCggOnkHCfTy97OA0HgozL-IA$>[Image
 removed by 
sender.]<https://urldefense.com/v3/__https:/twitter.com/mdwestix__;!!K-Hz7m0Vt54!iatSqn4yHM3x91laULnIOllnvAEqZBClRhLXcW0CJw8ZrVGDtYCggOnkHCfTy97OA0GiPThd0g$>
The Brothers 
WISP<https://urldefense.com/v3/__http:/www.thebrotherswisp.com/__;!!K-Hz7m0Vt54!iatSqn4yHM3x91laULnIOllnvAEqZBClRhLXcW0CJw8ZrVGDtYCggOnkHCfTy97OA0GRvdikNA$>
[Image removed by 
sender.]<https://urldefense.com/v3/__https:/www.facebook.com/thebrotherswisp__;!!K-Hz7m0Vt54!iatSqn4yHM3x91laULnIOllnvAEqZBClRhLXcW0CJw8ZrVGDtYCggOnkHCfTy97OA0FzIY3xBA$>[Image
 removed by 
sender.]<https://urldefense.com/v3/__https:/www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg__;!!K-Hz7m0Vt54!iatSqn4yHM3x91laULnIOllnvAEqZBClRhLXcW0CJw8ZrVGDtYCggOnkHCfTy97OA0GJBCPlCg$>
________________________________
From: "Mike Hammett" <nanog () ics-il net<mailto:nanog () ics-il net>>
To: "NANOG" <nanog () nanog org<mailto:nanog () nanog org>>
Sent: Friday, December 20, 2024 10:06:36 AM
Subject: Distributed Router Fabrics

I've noticed that the whitebox hardware vendors are pushing distributed router fabrics, where you can keep buying pizza 
boxes and hooking them into a larger and larger fabric. Obviously, at some point, buying a big chassis makes more 
sense. Does it make sense building up to that point? What are your thoughts on that direction?



-----
Mike Hammett
[ 
http://www.ics-il.com/<https://urldefense.com/v3/__http:/www.ics-il.com/__;!!K-Hz7m0Vt54!iatSqn4yHM3x91laULnIOllnvAEqZBClRhLXcW0CJw8ZrVGDtYCggOnkHCfTy97OA0E8DBeZtQ$>
 | Intelligent Computing Solutions ]
[ 
https://www.facebook.com/ICSIL<https://urldefense.com/v3/__https:/www.facebook.com/ICSIL__;!!K-Hz7m0Vt54!iatSqn4yHM3x91laULnIOllnvAEqZBClRhLXcW0CJw8ZrVGDtYCggOnkHCfTy97OA0HlU5g15g$>
 ] [ 
https://plus.google.com/+IntelligentComputingSolutionsDeKalb<https://urldefense.com/v3/__https:/plus.google.com/*IntelligentComputingSolutionsDeKalb__;Kw!!K-Hz7m0Vt54!iatSqn4yHM3x91laULnIOllnvAEqZBClRhLXcW0CJw8ZrVGDtYCggOnkHCfTy97OA0Fd4VS5Aw$>
 ] [ 
https://www.linkedin.com/company/intelligent-computing-solutions<https://urldefense.com/v3/__https:/www.linkedin.com/company/intelligent-computing-solutions__;!!K-Hz7m0Vt54!iatSqn4yHM3x91laULnIOllnvAEqZBClRhLXcW0CJw8ZrVGDtYCggOnkHCfTy97OA0HbHzd3kA$>
 ] [ 
https://twitter.com/ICSIL<https://urldefense.com/v3/__https:/twitter.com/ICSIL__;!!K-Hz7m0Vt54!iatSqn4yHM3x91laULnIOllnvAEqZBClRhLXcW0CJw8ZrVGDtYCggOnkHCfTy97OA0Fj7RvhHQ$>
 ]
[ 
http://www.midwest-ix.com/<https://urldefense.com/v3/__http:/www.midwest-ix.com/__;!!K-Hz7m0Vt54!iatSqn4yHM3x91laULnIOllnvAEqZBClRhLXcW0CJw8ZrVGDtYCggOnkHCfTy97OA0Gzrk6nZQ$>
 | Midwest Internet Exchange ]
[ 
https://www.facebook.com/mdwestix<https://urldefense.com/v3/__https:/www.facebook.com/mdwestix__;!!K-Hz7m0Vt54!iatSqn4yHM3x91laULnIOllnvAEqZBClRhLXcW0CJw8ZrVGDtYCggOnkHCfTy97OA0F4YUnLMg$>
 ] [ 
https://www.linkedin.com/company/midwest-internet-exchange<https://urldefense.com/v3/__https:/www.linkedin.com/company/midwest-internet-exchange__;!!K-Hz7m0Vt54!iatSqn4yHM3x91laULnIOllnvAEqZBClRhLXcW0CJw8ZrVGDtYCggOnkHCfTy97OA0HgozL-IA$>
 ] [ 
https://twitter.com/mdwestix<https://urldefense.com/v3/__https:/twitter.com/mdwestix__;!!K-Hz7m0Vt54!iatSqn4yHM3x91laULnIOllnvAEqZBClRhLXcW0CJw8ZrVGDtYCggOnkHCfTy97OA0GiPThd0g$>
 ]
[ 
http://www.thebrotherswisp.com/<https://urldefense.com/v3/__http:/www.thebrotherswisp.com/__;!!K-Hz7m0Vt54!iatSqn4yHM3x91laULnIOllnvAEqZBClRhLXcW0CJw8ZrVGDtYCggOnkHCfTy97OA0GRvdikNA$>
 | The Brothers WISP ]
[ 
https://www.facebook.com/thebrotherswisp<https://urldefense.com/v3/__https:/www.facebook.com/thebrotherswisp__;!!K-Hz7m0Vt54!iatSqn4yHM3x91laULnIOllnvAEqZBClRhLXcW0CJw8ZrVGDtYCggOnkHCfTy97OA0FzIY3xBA$>
 ] [ 
https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg<https://urldefense.com/v3/__https:/www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg__;!!K-Hz7m0Vt54!iatSqn4yHM3x91laULnIOllnvAEqZBClRhLXcW0CJw8ZrVGDtYCggOnkHCfTy97OA0GJBCPlCg$>
 ]