Google App Engine Stalled Out For About 6 Hours Today

A little over two hours ago, a Google employee posted a note in this Google Groups thread indicating that Google App Engine was “seeing elevated Datastore latency and error-rates, as well as elevated serving error-rates.” He noted that the problem began around 6:30 AM Pacific time and that the team was looking into it. A few minutes later he updated that Google App Engine was going into “unplanned maintenance mode” — over 4 hours later, it’s still not back up.

That’s a long time for any service to be broken, but especially one that is the backbone for many startups’ web apps. What’s worse is that while Google is updating the Google Groups thread, the actual App Engine Status page has been down the entire time as a result of the problems, so people are going there for updates and seeing nothing.

The last update from Google came about an hour ago:

Read-only mode continues. Elevated latency and error-rates persist for Datastore reads. Memcache writes have been reenabled to better soak read-only load. Our engineering teams are looking into the root cause of the problem. Will post more information as soon as it’s available.

Obviously, the natives are getting restless on Twitter. This outage follows the popular hosting service Rackspace experiencing some rare downtime earlier this week.

Update: And it looks like this downtime is even hampering the development of Chrome for the Mac. Here’s was lead developer Mike Pinkerton just tweeted out: “Popup blocking UI is done, but appspot apps are all horked so I can’t get it reviewed. #chrome”

Update 2: 6 hours later, it looks like things are finally back in working order. The latest update from Google in the Google Group thread:

Datastore writes are reenabled and functioning normally! Overall App Engine health is back to normal! We will update this thread if anything else develops, but at this time we anticipate no additional problems. Thank you for your patience.

[thanks Adam and everyone else who sent this in]