Yesterday’s Ushahidi service outage, and and all associated sub-domains and services were offline for approximately 7 hours and 30 minutes yesterday (August 10, 2011). Our hosting provider’s upstream data center suffered a major data center outage.

You count on us to provide reliable open source solutions. We apologize for this inconvenience to you and your deployment teams. Our hosting provider is closely following up with us and their data center provider to conduct an incident review and action plan. We, too, are discussing how we can improve.

What Happened:
An ATS (automatic transfer switch) completely failed at our hosting provider’s data center. The ATS is responsible for flipping load between the active and backup power (which is why backup power failed to start as the brain that is responsible for it is fried). A temporary ATS was installed, wired and tested to allow power to be restored. The Colo4 status page outlines their full technical response. Cartika Hosting, our hosting company, provided ongoing updates throughout the incident. We appreciated their communications, which we shared with you via Twitter, our developer mailing lists and our Ushahidi Dev Skype Chat.

Our next steps:
We are thankful for our community members offering to assist us throughout the day. We are determined to use experience to improve and ensure we can provide you with quality, reliable services. To that end, we will be holding discussions and share our next steps.

Thank you again for your remarkable patience during this difficult day.

One Response to “Yesterday’s Ushahidi service outage”