Home page logo

nanog logo nanog mailing list archives

Re: power failure causes and effects
From: "Marshall Eubanks" <tme () multicasttech com>
Date: Fri, 15 Aug 2003 00:51:48 -0400

On Thu, 14 Aug 2003 19:12:04 -0700
 Fred Heutte <aoxomoxoa () sunlightdata com> wrote:

It looks like DC may have had a close call - the University of Maryland was
dark for about 1 hour, and I have heard several news reports here stating
that the local (DC regional) power grid just managed to 
decouple from wider East Coast grid in time to avoid a collapse here.

Marshall Eubanks

Most of the early rumors about causes of the power failure
have proven incorrect -- fire at a New York City power plant,

Most likely is a congestion failure in the Niagara-Mohawk
grid, which covers a large part of New York state and has
feeders across into Canada.

Congestion failures happen when power flows through a
particular switching station are high, and a component
fails either directly or because of a power surge caused by
a failure elsewhere.  

The imbalance caused by an open or short circuit will then
immediately spread through the rest of the grid unless action 
is taken to disconnect, or point failures occur (resulting in fires 
and explosions from degradation of power transmission 
equipment, transformers, etc., not a pretty outcome at all) -- 
or both.  In general, transmission grids are designed from a
failsafe perspective, meaning that it is much safer to cause
rolling brownouts or blackouts than to let key components
such as transmission substations or power plants have a
catastrophic failure.  

Since the entire grid has to be in sync and supply and demand 
must be in relative parity at all times, the usual strategy in 
these cases is to isolate the affected area, "island it" by
shutting down power ingress and egress (tripping safety
breakers at major crossing points), and shutting down
power plants in the vicinity that will have stress failures
if they don't have sufficient load to balance their output.

The problem is that power travels faster than even the
highest-speed switching equipment can operate, so the
surges causing a cascading failure like this afternoon's can 
spread very quickly, like ripples in a pond.  

The weather was hot and humid but completely within
range for the time of year, so this has to be counted as a
"normal accident."  It's likely that whatever component
initially failed and triggered the shutdown was within 
its usual tolerances and simply had an ordinary breakdown.
Of course, to spread, a massive outage like this also exposes
other weaknesses and hidden dependencies in the system,
which might be other physical components, software,
operator error, etc.

The system is remarkably resilient in most circumstances,
which is what five-nines or more is all about.  But rust never
sleeps, and underinvestment in key transmission corridors in
New England, New York north of Manhattan and in
parts of the Midwest is no doubt an underlying cause.

As to the root cause of that engineering problem -- the
answer is politics, some of it congressional, and I will say no 
more in this forum.

Fred Heutte

Portland, Oregon
energy policy analyst and net geek

  By Date           By Thread  

Current thread:
[ Nmap | Sec Tools | Mailing Lists | Site News | About/Contact | Advertising | Privacy ]