Do Staging Procedures Need a Rethink?

“Has everyone begun owning conversations with their CIO/CEO about going back again to an in-residence mail server? I advocate for it”

Specified the scale of its person base and with a deal value up to $10 billion in the bag to run the back again-close of a superpower’s army, Microsoft may well want to start off considering about how it can establish a staging course of action for its Azure cloud that will allow it to deploy changes and reliably roll back again all those changes when things break.

(We know, it is straightforward to say so from a safe and sound distance…)

Redmond was at it all over again late Monday, knocking an (seemingly sizeable) “subset of consumers in the Azure Public and Azure Govt clouds” offline for three hours with swathes of customers globally encountering faults doing authentication operations a number of companies ended up affected, like Microsoft 365.

The enterprise blamed the challenge on a “recent configuration transform [that] impacted a backend storage layer, which prompted latency to authentication requests.” (Study, customers couldn’t login to Teams, Azure and additional for hours due to the fact of the snafu).

A whole root lead to evaluation is pending. (We will update this piece when we see it). The blockage was felt for customers from 22:25 BST on Sep 28 2020 to 01:23 BST.

The challenge arrives a fortnight just after a protracted outage in Microsoft’s United kingdom South region brought on by a cooling program failure in a facts centre. With temperatures climbing, automatic methods shut down all community, compute, and storage assets “to secure facts durability” as engineers rushed to choose handbook regulate.

Earlier this thirty day period in the meantime Gartner reported it “continues to have considerations relevant to the total architecture and implementation of Azure, inspite of resilience-targeted engineering initiatives and improved services availability metrics throughout the earlier year”.

Microsoft Azure CTO Mark Russinovich in July 2019 reported that Azure experienced formed a new Good quality Engineering group inside of his CTO office environment, working along with Microsoft’s Web site Reliability Engineering (SRE) group to “pioneer new approaches to produce an even additional reliable platform” adhering to shopper concern at a string of outages.

He wrote at the time: “Outages and other services incidents are a obstacle for all community cloud companies, and we proceed to increase our comprehension of the sophisticated ways in which aspects these kinds of as operational procedures, architectural layouts, hardware problems, application flaws, and human aspects can align to lead to services incidents.

“Has everyone begun owning conversations with their CIO/CEO about going back again to an in-residence mail server? I advocate for it” 1 frustrated person famous on a world Outages mailing listing meanwhile… If cloud is your compressed audio stream that you’re not guaranteed you own, it might not be extensive in advance of in-residence mail servers become the vintage high quality vinyl of the IT globe outdated, but really a lot back again in desire.

Stranger things have happened.