Re: Famous operational issues

[Dovid Bender <dovid () telecurve com>]

On Mon, Feb 22, 2021 at 2:05 PM Warren Kumari <warren () kumari net> wrote:

> On Mon, Feb 22, 2021 at 12:50 PM Regis M. Donovan <regis-nanog () offhand org>
> wrote:
>
> > On Thu, Feb 18, 2021 at 07:34:39PM -0500, Patrick W. Gilmore wrote:
> > > And to put it on topic, cover your EPOs
> >
> > I worked somewhere with an uncovered EPO, which was okay until we had a
> > telco tech in who was used to a different data center where a similar
> > looking button controlled the door access, so he reflexively hit it
> > on his way out to unlock the door.  Oops.
> >
> > Also, consider what's on generator and what's not.  I worked in a
> > corporate
> > data center where we lost power.  The backup system kept all the machines
> > running, but the ventilation system was still down, so it was very warm
> > very
> > fast as everyone went around trying to shut servers down gracefully while
> > other folks propped the doors open to get some cooler air in.
>
> That reminds me of another one...
>
> In parts of NYC, there are noise abatement requirements, and so many
> places have their generators mounted on the roof -- it's cheap real-estate,
> the exhaust is easier, the noise issues are less, etc.
>
> The generators usually have a smallish diesel tank, and then a much larger
> one in the basement (diesel is heavy)...
>
> So, one of the buildings that I was in was really good about testing thier
> gensets - they'd do weekly tests (usually at night), and the generators
> always worked perfectly -- right up until the time that it was actually
> needed.
> The generator fired up, the lights kept blinking, the disks kept spinning
> - but the transfer pump that pumped diesel from the basement to the roof
> was one of the few things that was not on the generator....
When we were looking at one of the big carrier hotels in NYC  they said
that they had the same issue (could be it was the same one). The elevators
were also out as well. They resorted to having techs climb up an down 9
flights of stairs all day long with 5 gallon buckets of diesel and throwing
it into the generator.

>