Why is IPv6 broken?

[Bob Network <networkjoe () hotmail com>]

Why is IPv6 broken?

It's broken, first and foremost, because not all network providers who claim to be tier 1 are tier 1.

Even worse, some of these providers run 6to4 relays or providers to home users.  A user has no choice which provider is 
running their 6to4 relay...so, they might end up using a relay that is run by a provider who doesn't peer with their 
intended destination.  I don't think the IETF saw that one coming.  But the result is to make 6to4 even more broken.  
Now, I know some people want 6to4 to die, but while it still exists in some form, user experience is worse than it 
could be.  The temporary fix is for any provider to run their own 6to4 relay for their own customers (assuming that 
they themselves have full connectivity).

Right now, unless you buy transit from multiple tier 1s, and do so with carefully chosen tier ones, you have only part 
of the IPv6 internet.  Many tier 1s are unsuitable even as backup connections, since you still want your backup 
connection to have access to the whole internet!  Good tier 2 providers might be an excellent choice, sine good 
providers have already done this leg work and can monitor their providers for compliance.

A few myths...

Routing table size has nothing to do with completeness of routes.  Google may be one route, through aggregation.  And 
SmallCo may advertise a large route through one provider, and, due to traffic engineering, a smaller route through a 
second one - in many cases, anyone that had the large route would be able to contact SmallCo, even without the smaller 
route being present.  So routing table size doesn't work.  In addition, some providers aggregate their routing tables 
to reduce routing load and such.  Others intentionally don't or deaggregate it intentionally so that they can brag 
about having bigger routing tables.  What you need to ask is: "How many /64s can you get to from your network, and how 
many of these /64s are reachable from at least one other major provider (you don't care about internal-only networks, 
after all)?"  They can give you that information, but many won't want to.

It's also not about technical people not getting along.  It's about business players trying to make money, but not just 
that either.  It's also about ensuring that providers don't end up assuming more than their share of costs for a link.  
Just because you have a common peering point doesn't mean that turning peering on would reduce your costs.  In some 
cases it may increase costs tremendously, particularly on your long haul backbone links, because the other party would 
like to take advantage of an attitude of trust on the internet.  That's why we end up with peering policies and 
contracts.

What is the issue?

Let's take Hurricane.  This is no different than other providers...basically, they want to say, "We shouldn't need to 
pay for IPv6 transit from anyone."  This is what Cogent said on IPv4 a few years ago.  Google used to say this too for 
IPv6, not sure if they are still saying it.  Basically, "We know we're big enough that you won't want to screw your 
users by not peering with us."

A small network couldn't do this tactic - a 100 node network who said to the IPv4 tier 1s: "Hey, I'm in the Podunk 
Internet Exchange, so are you, so I'm going to peer from you so I don't have to buy any bandwidth for my web server 
(placed in the Podunk exchange).  Sure, they would like to - it would save a ton of money if their site got lots of 
hits.  I mean, who wouldn't want free connectivity?

In IPv6, we're going through what we settled years ago in IPv4 - who has to pay who to connect.  After all, even free 
peering connections have a cost in manpower, debugging, traffic engineering, documentation, etc.

Some players who aren't getting free interconnection to tier 1s in IPv4 want to get it in IPv6.  So they've worked to 
attract lots of users, and done so under the guise of "We like IPv6 and want to promote it."  Others have not bothered 
with trying to attract the users, but have said, "We're too big for you to not want to give us connectivity for free, 
since it would piss off your users if you don't" (Google did this at one point in the past, may still be doing it).  
The Google example is basically trying to use a monopoly position to force business decisions.

Now, HE, Google, and others would want you to think, "Hey, IPv6 is all new, and these $#@! other providers just want to 
make a buck on something they have no right to."  Well, perhaps.  But what they aren't saying is, "We can turn on BGP 
for IPv6 on our existing connections to other providers, with no cost to us, and actually have full connectivity."  The 
issue isn't about cost today - nobody is charging extra for IPv6 in addition to IPv4 on a pipe where you already buy 
IPv4 bandwidth.  And Google and HE already buy IPv4 bandwidth.  What they are thinking of is the future, 15 years from 
now, when there is no IPv4 - in that future, IPv6 isn't insignificant bandwidth, it's everything.  Wouldn't it be nice 
to be a tier 1 and not pay for that?  Of course!  And certainly one can argue for or against the current tier 1 club's 
exclusivity.  But it's the way the internet works right now, for better or worse.  In the meantime, in pursuit of this 
future, today's customers are screwed by these providers trying to position themselves to make more profit margin down 
the road.

Which is better for the customer?  A system where they are screwed today so that their provider can have a better 
negotiating position in business discussions OR a position where they do whatever they have to take to provide the 
customer with full connectivity?  (To HE's credit, they are giving away transit today on IPv6, so it's not like you are 
losing anything of value by not having the full internet routing tables, but it's a huge reason to not pay HE anything 
in other services, such as data center colocation - go with a provider that you pay and which gives you what you pay 
for - full transit).

A bit about peering...

Lots of people who aren't running big networks don't understand peering.  They think, "Doesn't this benefit everyone if 
everyone exchanges traffic?"  Maybe, on a pure level, but the business doesn't work that way.

I'll give you an example.  Let's say you are a little ISP, and located in Virginia, near a major peering point.  You 
say, "All the tier 1s are there, I can pull fiber to that peering point, which is only a block away, and have free 
internet, other than the cost of the line."  So, let's say you run the line, and, let's say that all the tier 1s agree 
to let you peer for free, since they want your traffic too.  Now, let's say your user downloads 1,000 TB from a server 
in California, on Qwest's network.

You paid, let's say, $15,000 for your piece of fiber going a block.  You needed to hire contractors and buy permits and 
such, after all.  So you shared in the costs of letting the user get to the server.  What did Qwest pay?  Well, they 
dug trenches, pulled fiber, negotiated with cities, counties, and states, paid taxes on their work, lit this fiber, 
etc.  It cost a lot because they went a lot further than your one block.  And a lot more than $15,000.

You say, "So what! Their customer benefits too!"  That's true, but let's go a bit further.  Let's say you have a 
network that extends to California - you by DS3s from Sprint to do it.  There's some cost in that, but your user in 
Virginia would need more bandwidth than your DS3s.  So you decide NOT to peer in California, just in Virginia.  That 
way you don't have to upgrade your lines for your Virginia user.  Maybe you even legally break your company into two 
entities, so that you can peer in California and Virginia both, but you can say with a straight face, "We only have 
Virginia offices for this user - the other company is a separate entity, and not the entity that owns either the server 
or the end user."

In other words, you found a way to shift most of the traffic burden and infrastructure costs to Qwest, away from your 
user.

This is why Qwest has some sort of peering policy.  Among other things, it will require multiple exchange points, and 
Qwest will probably say they will send traffic to the closest peering point, to minimize their costs.  You get to do 
the same (more on that later).

Let's say that you currently buy bandwidth from NTT - you're not big enough to get free peering from everyone, but 
Qwest agrees to peer with you.  Of course Qwest and NTT also have a business relationship, to give each other free 
peering.  If Qwest gives you and many other customers free peering, however, you'll send less traffic across NTT's 
network.  That might be good from a technical standpoint, but NTT now is selling you a smaller pipe - and making less 
money.  In effect, Qwest undercut NTT's business and lowered NTT's profits on the connection.  How will NTT respond to 
that, when they were also giving free peering to and from Qwest?  Well, they might decide that Qwest isn't a very nice 
partner and tell Qwest, "Pay us for transit or get lost."  That could be ugly - both NTT and Qwest could lose, but 
Qwest, if they actually care about stable service, won't want to risk it.  So generally you don't give peering to 
anyone who is a customer of one of your free peers.  You don't hurt their business.  In fact, it's often a requirement 
in the peering connection, legally.  (that said, you could argue whether or not there is an abuse of monopoly 
here...that's a different issue)

Going one further, let's say you have the server, and Qwest has the end-user.  That doesn't change anything - the 
economics are still such that Qwest has the cost, you don't.  That said, it's convention that the person receiving the 
traffic pays for most of the backhaul.

Asymmetry in the Internet:

What's the path between your host and a remote server?  How do you find it?  If you said "traceroute", you might be 
right, but are probably wrong.  You need to trace route both sides.

Every provider on the internet is trying to minimize costs.  This means that you want traffic to leave your network and 
go to the destination network with as little distance traveled as possible, because costs go up with distance.  It's 
cheaper to increase the size of pipes within a city to get to a peering point than to increase your backbone pipe size. 
 So, peering contracts typically specify that you dump traffic to the peer as soon as possible.  That means the person 
receiving the traffic generally pays more.  It also means that any traffic that crosses an AS boundary almost certainly 
travels a completely different path each way.  In many cases, one third party provider may be used in one direction, 
another in the other direction.  So seeing packet loss on your traceroute at some random tinet router doesn't mean that 
this router is the cause of any problem, since the return path for that packet from that provider's router might 
actually cross yet another network that is never transited in either direction for your network connection.  (I'm 
ignoring that most large providers also don't always send ICMP reliably BECAUSE they limit this intentionally to spare 
the router CPU from overload - it takes router CPU to generate an ICMP TTL exceeded, but it doesn't take router CPU to 
forward a packet - so traceroute or ping indicating loss at a router doesn't mean anything in itself - the path itself 
likely has zero percent loss).

So, here's the scenerio.

Let's say a user and a server are on two seperate networks, U (user) and S (server).

Let's say they both utilize transit provider T.  So the path could be:  U -- T -- S.  S buys an OC12 from T, while U 
buys a T1.

But let's say that the user has a second transit provider, BIG, who is a free peer of T.  He bought an OC3 from BIG.  
So there's another path between U and S: U -- BIG -- T -- S.  Likely this path is much faster than U -- T -- S.

So, the path for the traffic to S goes U -- T -- S.

Now, what path does the traffic from T's router go, when T's router generates an ICMP TTL exceeded in response from a 
traceroute from a user?  Does it go straight over the T1 line, or does it go over the peering connection to BIG and 
then to the customer?  The answer, it turns out, depends on network configuration and policy.  Let's say it goes out 
over the T1, but the T1 is congested.  It will look like the congestion is at the connection between BIG and T, because 
this is the first hop that will show packet loss.  BUT...the congestion is actually at the U's connection to T, which 
is irrelevant to the actual traffic path between U and S.  So the user, at this point, calls up BIG and T and bitches 
about "Your peering congestion is congested" when the real problem is that traffic completely unrelated to the user's 
problem is going via a congested path that is never used for connectivity between U and S.

If you add several providers into this loop, you can end up with a situation where traffic uses Sprint in one 
direction, but never hits a Sprint router in the other.  This is actually very common.  A user with slow downloads 
might be experiencing packet loss on the path from server to user, but not the other way around.  In other words, the 
problem is a provider that never shows up on the user's traceroute!

Remember that the providers hand off the traffic as soon as possible to 
their peer.  So, whoever receives the larger amount of traffic needs the
 bigger cross-country (or trans-oceanic) links.  If one side transmits a
 T1's worth of data, the other side transmits an OC48's worth of data, 
only one needs the OC48s across the country - the one receiving the 
traffic.  That's why you hear about "traffic ratios".  If the traffic is even both ways, both sides have to pay for the 
same amount of cross-country infrastructure to carry that traffic.  So most providers won't peer with someone for free 
that sends, say, 10 times the amount of traffic that they will receive.  It would end up costing a lot of money

Back to IPv6...that's interesting, but what does it have to do with IPv6?

Some providers want to do away with traffic ratio policies, mutliple location peering, not providing free services to 
the other's customers, etc.

THAT is why you can't ping some sites from your HE tunnel.  It's not just that providers won't peer.  It's also that 
providers have rules to keep themselves from getting screwed.

Certainly, there's ways around some of this (for example, traffic ratios - if I make sure my network is used for the 
cross-country traffic I send, not yours, then I've addressed that concern at a bit of increased expense for myself).  
But it's generally not worth doing until the size of the providers is sufficiently large.  Other things don't have a 
good technical fix, like not peering with your peer's customer - that's a business rule.