Re: Never push the Big Red Button (New York City subway failure)

[Daniel Seagraves <dseagrav () humancapitaldev com>]

> On Sep 15, 2021, at 2:20 PM, Fred Baker <fredbaker.ietf () gmail com> wrote:
>
> One of the many stories that came out of 9/11 was a switching center in NY City that had a diesel generator as a 
> power backup - which of course acted as primary when the city power is off. After a few days of operation, it needed 
> to be refueled, so a truck was sent in carrying gasoline. The generator was refueled and restarted, and - oops - 
> diesel != gasoline. So then they needed to bring in a new generator.
>
> Yup, it happens, and it happened.

I distinctly remember something like this - Someone built a datacenter with large fuel storage tanks in the basement 
and the actual generators up on the roof, or some higher floor. It was tested several times, everything seemed to be 
working as expected, and life went on. Then one day the power went out, the generators came on, but after about 10 
minutes the generators started to crap out. It was then discovered that they had forgotten to include the transfer 
pumps for getting the fuel up from the basement to the generators in the list of things powered by said generators…