UnitedForums.co.uk - View Single Post - Ongoing power incident at our UK facility

UH-Simon · 25th June 2008, 09:29 PM

Please note that the intention of this post is to speedily alert customers to a potential issue. The issue may well be resolved without any customer affecting issues, however in these circumstances we issue a notification that redundancy is impacted as soon as possible so customers are kept up to date.

Earlier this evening (around 7PM) there was an issue with the utility high voltage supply to the LHC datacenter. The power infrastructure at the datacenter detected that the supply was outside of the acceptable tolerances, so automatically started up the diesel backup generators. The UPS systems on each floor of the LHC briefly went to battery power to assume the load whilst the backup generators started. Once the generators were started and synchronised, the UPS systems received power from the generators and went off battery power.

Soon after this occurred, the utility mains power was restored, and the power infrastructure automatically changed over to utility power.

This passed without incident throughout the datacenter, apart from the 2nd floor, which houses our UK based server racks. The redundant UPS equipment that supports this floor detected an internal failure, and as such did not allow incoming power to be transferred from the generators to the mains power, to protect against any potential issues with the battery and UPS management systems from inadvertently stopping supply to the IT equipment hosted on that floor.

At the moment, the UPS systems on the 2nd floor are in "bypass" mode, with power being supplied by the standby diesel generators. It is not possible in this mode to switch back to utility power from the mains without a break in the power supply to the IT equipment, so generators will remain online whilst the fault with the UPS equipment is diagnosed.

We will keep you updated with further developments as the issue progresses.

25th June 2008, 09:29 PM	#1 (permalink)
UH-Simon Administrator Join Date: Oct 2002 Location: Hertfordshire, UK Posts: 4,601	Ongoing power incident at our UK facility Please note that the intention of this post is to speedily alert customers to a potential issue. The issue may well be resolved without any customer affecting issues, however in these circumstances we issue a notification that redundancy is impacted as soon as possible so customers are kept up to date. Earlier this evening (around 7PM) there was an issue with the utility high voltage supply to the LHC datacenter. The power infrastructure at the datacenter detected that the supply was outside of the acceptable tolerances, so automatically started up the diesel backup generators. The UPS systems on each floor of the LHC briefly went to battery power to assume the load whilst the backup generators started. Once the generators were started and synchronised, the UPS systems received power from the generators and went off battery power. Soon after this occurred, the utility mains power was restored, and the power infrastructure automatically changed over to utility power. This passed without incident throughout the datacenter, apart from the 2nd floor, which houses our UK based server racks. The redundant UPS equipment that supports this floor detected an internal failure, and as such did not allow incoming power to be transferred from the generators to the mains power, to protect against any potential issues with the battery and UPS management systems from inadvertently stopping supply to the IT equipment hosted on that floor. At the moment, the UPS systems on the 2nd floor are in "bypass" mode, with power being supplied by the standby diesel generators. It is not possible in this mode to switch back to utility power from the mains without a break in the power supply to the IT equipment, so generators will remain online whilst the fault with the UPS equipment is diagnosed. We will keep you updated with further developments as the issue progresses. __________________ . Simon UnitedHosting Staff For official support please use our helpdesk at UnitedSupport.co.uk UnitedHosting proudly hosting more than 20,000 sites since 1998.