mailing list archives
RE: Verizon FiOS - is BGP an option?
From: "Ralph E. Whitmore, III" <ralphw () interworld net>
Date: Mon, 6 Aug 2012 04:53:38 +0000
What I think most people are objecting to is that most of the issue with maintain service is not related to technical
capabilities, but related to the cost of providing these support services impacting the profit margins of the large
monopolistic carriers. Verizon with their FIOS offerings, at least in my area is CO based and all optical, so all I
have to provide is power to my own FIOS terminal which is easy to do, floated on batteries (not the POS that VZ
provided but a real battery bank) and I stay online. We have been through 10 outages at least 10 hours in duration and
one 3 day outage all courtesy of SCE who can't figure out how to replace the 57 year old wires that keep breaking from
corrosion here in So. Calif. As a subscriber, I paid for the copper Wire that VZ installed, I paid for the Copper that
Edison installed, I pay to maintain both of these services on a 7x24x365 basis. And each and every month, SCE and VZ
take a mandatory deduction out of my bill specifically to replace the copper every 50 years (the design life of the
infrastructure). Edison squanders that money and just patches the infrastructure with no regard for the customers, and
VZ replaces the copper with FIOS so they don't have to allow any competition from anyone else and demos the copper when
they are done so no one else can use it. All of these companies fail to understand that they were granted a license
and handed the keys to provide a public service and we expect them to perform that job rain or shine 7x24x365.
Hurricanes (while not a problem where I am) are a known problem in many parts of the company and it is their job to
maintain service despite the hurricanes. These fiber huts/RSU's were installed to minimize VZ's (insert your favorite
carrier here) cost of maintenance for their network . This way, they can increase their profits by laying off more
workers and hiring more subcontractors. So be it, that is their business model. What people/PUC/ Regulatory bodies
fail to follow up on is that just because they are allowed to install Fiber Huts/RSU's the customers should expect the
same level of service and redundancy that is provided by a brick and mortar CO built to the ATT/Bellcore standards for
stability and reliability. I am all for the carriers pushing the edge closer to the customers, but it should not be
allowed to occur at a substandard level. They certainly aren't offering a discount for substandard service received by
some. My customers get 99.999% reliability from my infrastructure, I expect the carriers to do at least as well,
obviously that doesn't happen.
All my Roadside cabinets have a DC plant that is engineered to hold the facility for at least 18 hours based on the
equipment in the box. All my facilities have an external Transfer switch and a generator plug. A single 5kw generator,
with one cable and padlock, with one tank of fuel (generally propane or diesel)(about 5 gallons) will run the hut for
8-10 hours, fully recharge the DC plant inside buying you another 18 hours on battery (therefore 28 additional hours
before fuel is needed) and to boot have electronic monitoring to tell me if there is another AC failure of the
generator during that 8-10 hours. One contractor, god forbid we wouldn't do this with actual employees of the
company, with a pickup truck and a small trailer should be able to support between 12-20 Fiber Huts /RSU's in a single
shift. If your site is bigger than a 5KW site then you should have generator backup onsite in the first place. If I
can do this Verizon/ATT/Centurylink should have no trouble supporting their facilities in a similar fashion. I am a
little dinky company with 12 employees, but Verizon( your carrier name here) If you want to use Wisper Watt 25KW
generators, more power to you, they will just pass the costs on to us anyways in the next rate hike. The argument
about lack of fuel availability is total bunk as well as I can't believe that any carrier doesn't already have 1000's
of gallons onsite already to support their CO's, trucks, ect. if you need more you can order a 5000gallon tank like
everyone else in the real world does, you can probably have it delivered tomorrow afternoon if you place the order
tomorrow morning before noon. Lack of power at an Fiber hut/RSU is negligence at best on a carriers part, not an
unsolvable technical problem
My uncle who retired from Bell Atlantic/VZ after 30 years was in a rural area of Virginia and now works as a contractor
for emergency restoration after storms. He was called in to work on Storm restoration he along with 20 other crews,
were told report to the staging yard and wait for an assignment. It took three days to get to the staging area, They
waited 4 days in the yard before being assigned work and was then told that they couldn't work more than 12 hours in a
day, because VZ would only pay for a max of 4 hours OT for any personnel in a 24 hour period. They worked for 4 days
and were then sent home as they were no longer needed. To the best of my recollection people were without service for
weeks if not months. The safety wackos in the world will say well we don't want to tax our employees and more than 12
hours isn't safe. Ok, I will not argue the validity of their argument, personally I disagree with it, but this is an
emergency response not a shareholders meeting, but sending the crews home weeks before all service is restored doesn't
resonate well. IMHO Restoration was not about serving the people who were out of service, but rather spending at
little as they could get away with without regard for the people who pay for the infrastructure and pay their bills.
While we are at it, my uncle installed the RSU he is serviced from at his house before he retired from Bell
Atlantic/VZ, they engineered only 4 hours of battery backup for their site near his house, the fiber hut/RSU is approx.
8'x8'x5' and it has only 4 -105AH batteries installed in it, there are three vacant shelves for more batteries and one
totally vacant compartment left in the cabinet for battery expansion. Given the rural nature of his service RSU, the
nearest stocking yard for VZ is better than an hour's drive from his RSU. Given the Union contract gives the workers 2
hours to show up at the yard from the time they are contacted ( which could take hours to do in the first place if his
RSU is down) there is almost no way that you can callout a tech and make it to the site before the battery back is
exhausted and Dominion Power is highly un-likely to arrive onsite in less than 12 hours from the time of call. So this
site is doomed to go down from the start.
A single house, my house, your house, despite what we think are not critical infrastructure to the carriers, but the
fiber huts that service them certainly is and needs to be supported better than they are now.
Back to the original topic, BGP works great over FIOS if you have a business account with static IP's and a Cooperative
Colo. I have IPv4 and IPv6 running at my house and office (we microwave the Home's business FIOS to the office)
through a GRE tunnel. It's the cheapest 35x35Mb connection around for about $200.00 with 32 static IP's around TWC
wants about $2k for the same service. It took a little bit of fiddling to get the MTU right across the tunnel (ip tcp
adjust-mss is your friend) I could upgrade the service for another $100.00 to 150Mb x 65Mb, but the wireless link
won't support that without a large expenditure and for 12 people it's not needed.
Boy that was a lot, glad I had time on a Sunday night to write this after fueling my generator.
From: Frank Bulk [mailto:frnkblk () iname com]
Sent: Sunday, August 05, 2012 7:41 PM
To: 'William Herrin'; Andy Koch
Cc: nanog () nanog org
Subject: RE: Verizon FiOS - is BGP an option?
I think the term "critical" is being used in different senses in this discussion. Are people's lives critical? Yes,
but the regulations for wired and wireless infrastructure don't require service providers to expend any and all costs
to maintain connectivity. And we don't have ambulances and fire trucks at every corner and hospitals in every
subdivision. Despite the almost incalculable value of life, there are still limitations on all the services provided
to residences. Would I like to have the same uptime at my home as we have in the CO? or data center? Sure, but
collectively we aren't willing, nay, able, to pay that price.
From: William Herrin [mailto:bill () herrin us]
Sent: Saturday, August 04, 2012 11:15 PM
To: Andy Koch
Cc: nanog () nanog org
Subject: Re: Verizon FiOS - is BGP an option?
On Sat, Aug 4, 2012 at 10:26 PM, Nathan Eisenberg <nathan () atlasnetworks us> wrote:
Residences aren't critical infrastructure, no matter how angry the
911 access isn't a critical service? Fire and security panels aren't
If basic life safety and property protection aren't critical services,
not sure what is.
Whether each individual's residence contains critical infrastructure is a decision best left up to that individual. By
necessity that makes the upstream aggregation components critical infrastructure. No different than it was for POTS 20
The Internet isn't just a toy any more. It's the primary communications channel in to many folks homes and well on its
way to becoming the primary channel period.
William D. Herrin ................ herrin () dirtside com bill () herrin us
3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004
RE: Verizon FiOS - is BGP an option? Peter Kristolaitis (Aug 05)
Re: Verizon FiOS - is BGP an option? Peter Kristolaitis (Aug 05)