Home page logo
/

nanog logo nanog mailing list archives

Re: Verizon damage and service disruptions
From: Frank Coluccio <fcoluccio () dticonsulting com>
Date: Thu, 13 Sep 2001 06:37:23 GMT


re: the NYSE not having tridundancy, and the Broad Street CO Power uncertainties:

Several things. 

The Pearl Street Central Office [which sits adjacent to the Brooklyn Bridge on 
the East Side of City Hall] is capable and does provide back up to both the Broad 
Street and West Street Central Offices on a special assembly basis. This, in 
addition to supporting its own subscribers, many of whom are residential and 
small business with routes running into the Chinatown area. 

Some firms near the NY Stock Exchange [I know of two on Broad Street] had elected 
long ago to have tri-dundancy built into their local access networks, by adding 
Pearl Street to the mix. So, the NYSE has to tread lightly when they look to why 
they don't have the same. All they had to do is place an order. This does raise 
an interesting question, though, concerning why their other local providers (of 
which, I believe, there are at least four) can't pick up some of the slack. 

As for the Broad Street Central Office having some power problems, I would remind 
all here that the Seaport Substation [which is just south of the Pearl Street CO] 
is also having its difficulties at this time. I believe that this substation 
failed yesterday under the load when 7WTC collapsed. You might recall that the 
Seaport substation was the cause of another of the Wall Street's major disasters 
back in the early 1990s. 

A substantial part of the financial district down Water Street and Broad Street 
are fed directly from this substation. Major re-grooming of feeders in this area 
was done back then when the failure occurred, in order to shore up switching to 
alternate substations in the event that it failed again. But this situation is 
different, because now it is the Seaport station itself that the West Side grid 
would depend on, and neither is working at this time. Most of what is working at 
this time in the area is actually on generator.

It's going to take a lot of doing to support trading when the Stock Exchanges are 
re-opened on Friday, or Monday at the latest. Another thing that I remain wary of 
is the integrity of many of the underground duct bank systems supporting newly-
run fiber routes in the area. Some of these sit atop older ducts and are only a 
foot or two beneath street asphalt in many places, despite standards to the 
contrary, and they run up and down Church Street, which becomes Tinity Place.

These will not be tested fully until the downtown trading floors and data centers 
are reopened in a couple of days. Some rings in the area are now showing failed 
links on remote surveillance systems in offsite data centers, but it is not known 
whether this is due to carriers taking their nodes down temporarily, or due to 
underground problems, or due to anomalies at the customers own premises that are 
now almost entirely unpersonneled below 14th Street and working on generator.

-FAC


First, 1 Verizon employee and 3 Genuity employees are still missing, my
prayers, and I'm sure everyone else's prayers go out to their families.
No one wants to distract or impair the continuing rescue efforts in the area.


As Verizon continues its assessment of the damage, it is starting to
look worse than original reports.  Verizon hasn't filed an outage
report with the FCC, or the FCC has delayed its release.  So this is
based on Verizon's news conference and news reports.  The reports
aren't very clear in all the details.

The walls of the 140 West ST CO adjacent to the WTO Building 7 were breached
when Building 7 collapsed, and the building has flooded and the equipment
is covered with debris.  140 West St has no power.  Verizon needs to pump
out the building because it is 5 stories below ground.

Also mentioned is the Broad Street CO, which is experiencing intermittent
power problems.  The Broad Street CO supplies 80% of the New York Stock
Exchange lines, and the West Steet CO serves as the alternate CO supplying
20% of the NYSE lines.

The CO's supply 200,000 voice lines and 3 million separate private lines.
Although the NYSE had backup facilities, it looks like the disaster in
NYC is disrupting service from both CO's.  The telecommunication directors
of the exchanges have been meeting during the day.

According to news reports, Verizon left it unclear why a backup facility
somewhere else wouldn't be an adequate substitute for Wall Street's
financial community and stock trading.  According to one news report,
NYSE Chairman Grasso acknowledged that work is needed to come up with
a better telecommunications plan in the future.







  By Date           By Thread  

Current thread:
[ Nmap | Sec Tools | Mailing Lists | Site News | About/Contact | Advertising | Privacy ]