Re: Distributed Router Fabrics

[Saku Ytti <saku () ytti fi>]

On Wed, 25 Dec 2024 at 05:34, Matthew Petach <mpetach () netflight com> wrote:

> Power is a *huge* part of the equation that I think many people overlook.
> When you look at what a really big chassis takes in terms of power feeds,
> it's not uncommon to need relatively specialized 3-phase 240V power feeds
> for the very-high-end chassis box that give you the same type of high speed
> port densities that a pizza-box fabric folded Clos model can yield.
> (not to pick on any vendor, but here's an example of the types of power feeds
> a large chassis can require:)

This and most of the differences are implementation details, not fundamental.

If we assume some fantastical world, where you can make anything
appear out of thin air with no cost, no one will use the same
interfaces to connect customers and to build interconnect between
chips. Because the serdes fabric interfaces are lower power, pin,
thermal, cost than the real customer facing port and you can have
higher density of them.

On very high and impractical level, that's the difference, how do you
interconnect chips, do you use some specialised solution that
understands that both ends are going to be the chips in the same rack
or do you use generic solution that makes no assumption on what is
going to be connected on the other end.

In practice people do these stack of switches, because they want
rightsized platforms and there isn't quite the right size available
from anyone that they care to deploy. So the rightsized stack ends up
being commercially more viable and more energy efficient, because the
right box for the application has no commercial availability.


Luckily for most of us, these problems do not matter, as chip
densities are front-running even most hyperscalers, when Amazon
presented their stack-of-switches solutions couple years ago, giving
the densities and how many front-facing ports are 'wasted' on internal
interconnect, vendors were already shipping single chip solutions
matching the stack-of-switches non-interconnect port densities, i.e.
already this hyperscaler solution could have been single chip device,
not stack-of-switches, not fabric box.
And this is true for almost every buyer in the market, you need just a
single chip box today, densities are absurd for almost everyone.


-- 
  ++ytti