Ongoing power incident at our UK facility

UH-Simon · 25th June 2008, 08:29 PM

Please note that the intention of this post is to speedily alert customers to a potential issue. The issue may well be resolved without any customer affecting issues, however in these circumstances we issue a notification that redundancy is impacted as soon as possible so customers are kept up to date.

Earlier this evening (around 7PM) there was an issue with the utility high voltage supply to the LHC datacenter. The power infrastructure at the datacenter detected that the supply was outside of the acceptable tolerances, so automatically started up the diesel backup generators. The UPS systems on each floor of the LHC briefly went to battery power to assume the load whilst the backup generators started. Once the generators were started and synchronised, the UPS systems received power from the generators and went off battery power.

Soon after this occurred, the utility mains power was restored, and the power infrastructure automatically changed over to utility power.

This passed without incident throughout the datacenter, apart from the 2nd floor, which houses our UK based server racks. The redundant UPS equipment that supports this floor detected an internal failure, and as such did not allow incoming power to be transferred from the generators to the mains power, to protect against any potential issues with the battery and UPS management systems from inadvertently stopping supply to the IT equipment hosted on that floor.

At the moment, the UPS systems on the 2nd floor are in "bypass" mode, with power being supplied by the standby diesel generators. It is not possible in this mode to switch back to utility power from the mains without a break in the power supply to the IT equipment, so generators will remain online whilst the fault with the UPS equipment is diagnosed.

We will keep you updated with further developments as the issue progresses.

UH-Matt · 25th June 2008, 09:58 PM

We are back on mains high voltage utility power, the UPS systems are still in bypass mode.

Engineers from the UPS manufacturer are analysing the data from the management controllers on the 2nd floor UPS equipment following the switch back to utility power, and are deciding on a course of action. The alarms previously active on the UPS equipment have been successfully cleared, and the engineers hope to restore battery backup protection shortly.

No power was lost at any point and resuming UPS protection should be completed shortly without inturruption. It was re-assuring to witness the diesel generators kick in and do their job without any inturruption of service to our servers.

Many thanks,