Blackbird server - outtage

UH-Matt · 4th October 2007, 03:32 PM

Blackbird server has rebooted twice this afternoon and we are working to rectify the problem and cause of this, several minutes of downtime would have been noticed for customers on this server and we do apologise for the inconvenience.

UH-Simon · 4th October 2007, 04:21 PM

The Blackbird server issue is ongoing. We have our techs at the DC working on the server now and will keep you updated via. this thread.

Thank you for your patience.

UH-Matt · 4th October 2007, 04:49 PM

We are just finishing checking over the hardware on blackbird due to the intermittant reboots (we suspect a hardware issue is causing) and the server will hopefully be back online again shortly.

We apologise for this unavoidable unscheduled outtage.

UH-Matt · 4th October 2007, 04:58 PM

The server is back online now after our hardware inspection did not find any faults. This does mean there could be further intermittant problems this afternoon as we continue to find the cause of todays outtages, but we will do our best to minimise or avoid any further disruption.

Thank you for your continued patience as we fully resolve this matter.

UH-Simon · 4th October 2007, 09:54 PM

Unfortunately Blackbird is once again playing up. We have a couple of diagnostics to run to try and identify/eliminate the fault. We will keep you all updated via. this thread.

Thank you for your understanding.

UH-Simon · 4th October 2007, 10:38 PM

Update. We suspect one of the memory modules has gone back. We're planning to swap the memory out and try bringing the box backup. We will update you again shortly.

UH-Simon · 4th October 2007, 11:06 PM

Update. The server is now back online but is only using 1 dual core opteron CPU, the 2nd CPU appears to be faulty. If the server remains stable over the next few hours then it will confirm our diagnosis and we will schedule a maintenance window to fit a new 2nd CPU.

Thank you for your patience during this outage.

UH-Simon · 9th October 2007, 11:26 PM

We will be shutting down Blackbird at 23:40 this evening to put the new 2nd CPU in. The outage should be no more than 10 minutes.

Thank you for your understanding.

UH-Simon · 9th October 2007, 11:59 PM

2nd CPU has been installed and the server is back online.

UH-Simon · 25th October 2007, 11:41 AM

We have noted that Blackbird has been playing up again today. We're checking the logs and will update you all asap.

UH-Simon · 25th October 2007, 12:03 PM

We suspect either the CPU issue has returned or we have a bad stick of RAM. We are initially replacing the RAM (immediately) and if this fails to resolve the issue we will remove the 2nd CPU for testing.

Apologies for any inconvenience this unexpected issue may be causing.

UH-Simon · 25th October 2007, 10:20 PM

Apologies for the lack of updates. We have been monitoring various things on the server throughout the day. We are trying a different kernel to try and rule out software and will then need to re-look at hardware.

UH-Matt · 25th October 2007, 11:48 PM

The server is now on a 3rd set of RAM and a roll back of its kernal by 1 version.

We are now monitoring to see if things are stabalised.