Datacenter / Connecitivity Issues 15:15 Today

UH-Matt · 26th November 2007, 03:45 PM

Hello

Around 30 minutes ago we lost all connectivity from our UK facility. THis is now fully restored but I am opening this thread to keep you informed on the outcomes of how/why/when this all occured.

So far we are led to believe there has been some sort of flood at the exchange which handles a lot of connectivity to our facility. I hope to give more details as they become available, but everything should now be online and re-routed.

Thanks for your understanding.

UH-Matt · 26th November 2007, 04:51 PM

Further details:

Quote:

There was a massive traffic flood on the LINX peering exchange earlier
today, which triggered an issue in our (and several other ISPs using Juniper routers on LINX) routers. Layer 2 connectivity was fine, causing our routing protocols to assume the router was up, however no layer 3 traffic was passed, causing packet loss to some destinations on the Internet.

We have resolved this for the moment by removing this router from service entirely and will be troubleshooting this with JTAC (the Juniper Technical Assistance Center).

Please note that we did not experience a full outage - to most destinations and monitoring systems our network was online, however people connecting via LINX (e.g. most UK based DSL/cable users) would've experienced significant packet loss for a short while.