Starting around 9 a.m. Pacific, several Microsoft services simultaneously went down, including the cloud platform Azure, Microsoft 365, Xbox, and more.
Over on the outage tracking site Down Detector, we've seen spikes of close to 10,000 reports on multiple Microsoft platforms. Those spikes have started to dip for Azure, Teams, and 365.
For reasons we haven't been able to determine, the gaming services like Xbox Live and Minecraft stayed affected longer than the more business-oriented Azure services. As of this writing, the Xbox support website is still down.
Initially according to the Microsoft Azure status page, it appears the outage was caused by DNS issues, though they don't specify if it was an attack or something internal. The team there has said that it's attempting to roll back to an older version.
Meanwhile, a number of services like Starbucks.com, Chris Hemsworth's fitness app Centr, and others are among the affected services.
Starting around 1:30 pm Pacific, a deployment of the 'last known good' configuration of Azure was deployed, with the company expecting recovery by 3:20 pm Pacific.
The outage lasted about five hours. Toward the end of the outage, the Azure status page claimed the issues were caused by an "inadvertent configuration change" that triggered the problems.
Unfortunately, at the same time, the Amazon Web Services platform also seemed to be having issues, meaning that a huge chunk of the internet was potentially affected by both of these platforms getting hit.
We tracked both. Keep it here for live updates.
Microsoft outage — live updates
Outage began around 9 am Pacific
We first started seeing reports on Down Detector start spiking around 9 a.m. Pacific.
Affected Microsoft services
Here's what Microsoft-owned services we've seen so far as having issues.
We're certain there are more plus other companies that use Azure like Starbucks but this is what we've specifically seen spike at the moment.
- Microsoft 365
- Microsoft Azure
- Micrsoft Copilot
- Microsoft Entra
- Microsoft Store
- Microsoft Teams
- Minecraft
- Xbox
AWS is also down...again
Simultaneously, Amazon Web Services has gone down again. Specifically the US-Region-1 site.
We're already seeing that impact on multiple services.
If the internet isn't working for you today, these simultaneous outages may be why.
Azure status notice
Microsoft has acknowledged the outage on the Azure status page. Here's what it currently says:
Starting at approximately 16:00 UTC, we began experiencing DNS issues resulting in availability degradation of some services. Customers may experience issues accessing the Azure Portal. We have taken action that is expected to address the portal access issues here shortly. We are actively investigating the underlying issue and additional mitigation actions. More information will be provided within 60 minutes or sooner.
It seems to be a similar problem that affected and is affecting AWS. Unlike that outage, Microsoft's appears more global.
Some services coming back online?
I am seeing a drop in reports on Down Detector.
Specifically for Azure, the Microsoft Store and 365.
However, services like Minecraft and Xbox Live are going up. It's not clear at this moment if those are separate issues.
Have you been affected?
Azure, unlike AWS, has more government contracts and business ones. Though it does power some services like Walmart.
Have you noticed a degradation of service because of the Microsoft problems today?
Let us know by emailing me at scott.younker@futurenet.com
Azure update
The Microsoft Azure status page has another update and it looks like there may be some solutions.
Customers can attempt to use programmatic methods (PowerShell, CLI, etc.) to access/utilize resources if they are unable to access the portal directly. We have failed the portal away from Azure Front Door (AFD) to attempt to mitigate the portal access issues and are continuing to assess the situation.
We are actively assessing failover options of internal services from our AFD infrastructure. Our investigation into the contributing factors and additional recovery workstreams continues. More information will be provided within 60 minutes or sooner.
That's specifically for Azure customers but it may help bring things back online.
Steep drop in reports
Since the solutions message appeared on Azure reports on Down Detector have taken a steep drop.
Here's the thing, though, other servics from Microsoft like Minecraft and Xbox are going up.
Additionally, places that use Azure like Starbucks are still totally down. Starbucks.com is inaccessible as of this writing.
Another Microsoft update
Microsoft Azure's status page has a new update showing that the team there is trying to get back to a version that wasn't facing issues. However, it appears to be partially working and the rollback may not be ready yet:
We suspect that an inadvertent configuration change as the trigger event for this issue. We are taking two concurrent actions where we are blocking all changes to the AFD services and at the same time rolling back to our last known good state.
We have failed the portal away from AFD to mitigate the portal access issues. Customers should be able to access the Azure management portal directly.
We do not have an ETA for when the rollback will be completed.
What readers are telling us
Tom's Guide readers have reached out to confirm that some services they use aren't working properly.
Some of these include the Chris Hemsworth fitness app Centr, Workfront via the Adobe Experience Platform, the Pearson Vue testing system for schools, and at least one university where a student said they couldn't upload documents.
Every region affected
You'll notice that the Azure status page has changed to show issues across every region of that they service. This includes AFD a portion of the cloud distribution service being down.
This will likely have widespread issues as problems spike up and down for services that use Azure.
Microsoft Status page down
We have been using the Microsoft Azure status site to see what the Azure team is saying about updates and how it appears to be going.
However, even that page is down now!
As you can see in the screenshot above the only thing I can see is a "We'll be back soon!" message. It's not clear how long that will last.
A statement from Microsoft
I reached out to Microsoft toward the beginning of the outage, and we just received a response from the company.
"We are working to address an issue affecting Azure Front Door that is impacting the availability of some services. Customers should continue to check their Service Health Alerts and the latest update on this issue can be found on the Azure status page.”
It's not an explanation and directs us to the status page we've already been reviewing. But if you haven't looked there, please do.
I will note that the "failover strategies" page it currently links to is broken.
Mitigation deployed
According to the Azure status page, the Microsoft team has deployed its fix, which should be ready in about 30 minutes.
We have initiated the deployment of our 'last known good' configuration. This is expected to be fully deployed in about 30 minutes from which point customers will start to see initial signs of recovery. Once this is completed, the next stage is to start to recover nodes while we route traffic through these healthy nodes.
Customer configuration changes will remain blocked during this time as we work towards mitigation. We will communicate to customers when this block is reverted.
Reports down
Over on Down Detector the main Microsoft services from Azure to 365 and Teams have steadily declined. It's not quite over but Azure's reports are under 2,500.
However, the gaming services Minecraft and Xbox Live remain seeing high reports.
We are currently waiting for the Azure mitigation to go through. That should happen in about 10 or 15 minutes.
Xbox struggling
With Xbox Live and Minecraft still showing outage issues, I tried checking the Xbox status page and can't access it at all.
All I can see is a tiny Xbox logo with the spinning loading circle.
I have reached out to Xbox about the outage, but have yet to receive a response.
Azure update - Recovery
According to the Azure status page, the team there has rolled back to the "last known good" configuration. This means that recovery signs should start appearing. However, it may not be a full one yet.
We have pushed our ‘last known good’ configuration, and customers may begin to see initial signs of recovery. We are currently recovering nodes and routing traffic through healthy nodes, and as we make progress in this workstream, customers will continue to see improvement.
Customer configuration changes will remain temporarily blocked while we continue mitigation efforts. We will notify customers once this block has been lifted.
More reader reports
From the outside, it looks like Azure is finally coming back online after over three hours of issues.
That said, we have received more emails from Tom's Guide readers. Here's what they're saying.
More people at schools reached out saying that their 365 programs aren't working properly and their student systems down.
One reader from Spain noted that Microsoft's Parental Control service relies on Family Center, run on Azure. With that down, kids might not be able to access their computers for homework or fun.
In other cases, virtual desktop environments have been borked and specific apps for health services offices can't run. One business owner said they weren't able to take credit card payments.
Still can't access Xbox
The Xbox status page is still giving me an unending loading circle.
Over on Down Detector reports are currently sitting at just over 3,000. One user reported that an Xbox customer service tech told them things would be back to normal at 6 pm Mountain Standard Time still several hours from now.
What about AWS?
We've been tracking a separate outage over at Amazon Web Services.
AWS has officially denied that an outage occurred with its service but did acknowledge that "an operational issue at another infrastructure provider" could be impacting connections. They don't name Azure but with everything happening today that is the implication.
Minecraft down still?
Despite Azure seemingly back on track, Xbox Live and Minecraft are still going through it.
The Minecraft tracking site PebbleHost is showing that server sessions may be down.
I am seeing comments asking why some people are able to play online and others aren't. My suspicion is that the servers players are joining are cracked, meaning they don't use the official Mojang authentication servers and don't require a subscription account to access.
Official Mojang servers still appear to be down.
Microsoft Azure says its deployment has been completed
According to the Microsoft Azure status page they have "successfully completed" the deployment of the 'last known good' configuration.
"We initiated the deployment of our ‘last known good’ configuration, which has now successfully completed. We are currently recovering nodes and re-routing traffic through healthy nodes," the site reads.
It goes on to say that they expect recovery to fully hit by 3:20 pm Pacific.
You should be seeing signs of recovery. Configuration changes by customers are still blocked.
The outage appears to be over
After about 5 hours of issues, it appears that the Microsoft outage is finally petered out.
The remaining Microsoft properties in Minecraft and Xbox Live are finally seeing drops in reports on Down Detector. The Mojang server tracker shows things back to normal.
I'm still not able to access the Xbox status web page but I'm seeing comments from multiple users on both DD and Twitter saying they're able to get back in.
