Crowdmap Maintenance

Crowdmap is currently in maintenance. We apologize for the inconvenience caused to you and your users. We introduced a technical change which required a roll-back. The change issued was to tag and organize orphan reports. Unfortunately, this resulted in some public reports becoming private reports. After troubleshooting, we determined the best course of action was to rollback to a database snapshot from Monday, March 5, 02:30 UTC. Crowdmap.com began necessary maintenance at 15:00 UTC.  All Crowdmap deployments will be restored to active data as of Monday, March 5 02:30 UTC. This ensures that no active data will be corrupted. There may be some data lost in this restoration. Once we complete the restoration, we will do a database comparison. The expected maintenance time for the database restorations and associated testing will be up to 15 hours. (Approximately Tuesday, March 6, 2012 at 10:00:00 UTC). (Your local time.) We will post updates as we receive them. We will provide an incident summary and next steps once we complete our review. In the past month, we were already been working on a longer term architecture changes for more a more robust system with redundancy.

Update: Tuesday, March 6, 2012: 11:30 UTC (Your local time)

The Crowdmap Database restoration progress has been slower than expected. Approximately an hour ago, a decision was made to attempt a different course of action with the database restore. The original pathway was taking an extended time. Our hosting provider advised that it may be an additional 2 - 3 more hours to resolve this issue. We have been closely working with them to troubleshoot. Our hosting provider expects that this changed course may result in the database and Crowdmap being restored by 14:00 UTC (your local time).

Update #2: Tuesday, March 6, 2012: 16:00 UTC (Your local time)

Our hosting provider has tried two different methods to complete the database restoration faster. (As mentioned, we started the second method this morning).  Our provider has been in contact with their backup software provider to investigate why this issue is taking so long. Unfortunately, the original estimated completion time was wrong. The rebuild is at 25% complete. At this rate, we are looking at approximately another 12 hours of a service outage. We are working very closely with our partner to try to complete the restoration in a timely manner. The technical team is doing everything we can to resolve this as soon and as effectively as possible. Each one of us is mindful that your Crowdmap services and your users are important. As such, we are focused on fixing this issue, then improving the services.

Update #3: March 6, 2012: 20:30UTC (your local time)

The Crowdmap Database restoration continues. The rebuild is 32% complete. We are hoping to test and restore services by tomorrow morning. The rebuild is very slow. The most recent estimate has services online by tomorrow morning. Not ideal, but servers don't seem to like *virtual coffee*.

Update #4: March 7, 2012: 04:00 UTC (your local time)

Crowdmap services remain unavailable . Our technical team is closely monitoring the rebuild, which is around 44% complete. Once this step is done, we will test services and determine the next steps. Our humblest apologies for this service outage. We would prefer to be posting about your great deployments. We are also updating our @crowdmap twitter account with regular updates. Again, we apologize for these service delays. If you have any questions about your deployment, please contact us.