Datacenter on Emergency Status.

UH-Matt · 3rd June 2003, 04:48 PM

Quote from EV1:

Quote:

At just after 7 PM CST, a major transformer in the EV1 data center parking lot exploded. At this time, it is still on fire.

We have implemented our emergency response plans.

Our backup generators are working as designed and a fuel truck has been dispached.

We are also bringing in 2 truck mounted generators as additional backup due to teh length of time we anticipate running on generator.

LIMITED staff is working in the building at this time as a precaution.

HOWEVER, our full management and administrative staff has been dispached to the facility.

Please bear with us if you have reboot or restore tickets pending, we will get to them as fast as possible given the limited in building staff.

Just to put the magnitude of this in perspective, we currently have 8 fire trucks, 6 vpolice cars, and two ambulances on our property and in the roadway ... this in addition to several news helicopters.

Thanks for your patience.

At this time, our network is functioning normally.

UH-Matt · 3rd June 2003, 05:04 PM

pictures direct from the scene...

We will keep you updated.

UH-Simon · 3rd June 2003, 05:30 PM

Quote from EV1:

Quote:

All personnel are safe and accounted for including the security guard on duty in the parking lot.

During the switch to generator power, we lost several a/c units due to fuses blowing. Fuses are on hand and being replaced now. All A/c units should be functional again by 9 PM CDT. Backup building units are functioning normally and keeping the temperature within normal paramaters.

The fire is officially out buy the fire department is still cooling it. Throughout the emergency, the fire department was spraying water on our generator housings to ensure that they remained safe and cool.

Special thanks to the emergency crews that responded so quickly and thanks to teh Rackshack staff for working calmly throughout the emergency.

Additional backup generators are enroute as backup to the units that are running as part of our emergency response plan. These units should be onsite by 2 AM and wired into the bypass ports shortly thereafter.

A fuel truck has been scheduled for 7 AM and every 12 hours thereafter. Fuel tanks were full when the explosion hit and we anticipate that no more than 40% of our fuel will have been used by the time we refuel.

Staffing should be back to normal and any backlog on tickets will be cleared shortly.

More updates as they develop.

UH-Matt · 3rd June 2003, 07:00 PM

This is what blew:

Backups are in place:

Running on petrol.

corenominal · 3rd June 2003, 08:49 PM

Wow! thats totally cool in a "I LUV FIRE" kind of way. Excellent pics too!

UH-Matt · 3rd June 2003, 09:00 PM

Its pretty amazing that servers didnt experience any downtime. I know the Datacenter has excellent facilities and full backup plans, but when things like this happen for real its great to see the plans go so smoothly.

L Plates · 4th June 2003, 12:03 AM

My first thought was 'I wish my lawn was that green' and then I noticed the fire!

spells · 4th June 2003, 03:51 AM

Been in this situation myself. Thanks for keeping us totally up to date with everything Matt & Simon. Pictures very much appreciated.

Reptilian Feline · 4th June 2003, 08:06 AM

Oh my!! That was incredible! Those backups are impressive. I'm glad I'm not running a DC. Those fuel trucks must cost a fortune.

UH-Matt · 4th June 2003, 09:12 AM

$4000 a day in petrol to keep the place ticking over. New transforming is going in as we speak.

UH-Matt · 4th June 2003, 04:38 PM

Update from EV1

Quote:

Backup generators continue to function and power is being delivered to the building normally. We are being refueled about every 12 hours just to keep the tanks as full as possible.

We currently have a heavy rain and lightning so outside work on the electrical components is being hampered.

It is still believed that utility power will return on Thursday.

Thanks for yoru continued patience during this crisis.

DrPoppers · 6th June 2003, 07:10 AM

Looked like it might need a new 13amp fuse, I reckon. Good to see nothing went down.

PotentialCustomer · 9th June 2003, 07:46 PM

[quote author=DrPoppers link=board=11;threadid=1073;start=0#msg6756 date=1054879818]
Looked like it might need a new 13amp fuse, I reckon. Good to see nothing went down.
[/quote]

are they 13 amp in the US ?

djnoz · 11th June 2003, 02:07 AM

Hmm, I wonder how many websites would disappear if they lost all power to that building? quite a lot i'd imagine. :-

UH-Matt · 11th June 2003, 09:04 AM

Just to let you all know that domestic power was returned a few days ago so Its all straight again.