Many of the world’s biggest apps and websites have been broken for hours in a worldwide outage.
Snapchat, Roblox, Fortnite, Duolingo and Canva were all suffering problems, according to tracking website Down Detector.
The issues began around 8am in the UK, or midnight pacific time, and have been continuing through the day.
The problems appear to be related to an issue at Amazon Web Services (AWS), which offers infrastructure that underpins much of the modern internet.
Amazon Web Services provides a host of internet infrastructure services that allow companies to hire computers and servers to run their apps and websites. As such, any problems at AWS can quickly affect much of the rest of the internet, bringing down websites that might have no apparent connection to Amazon itself.
It is now the most popular provider of such cloud services in the world. It made $108 billion last year, and it now accounts for the majority of Amazon’s profits.
AWS internet outage: Key points
- Snapchat, Wordle, HMRC: Which sites are down?
- Amazon says it is experiencing problems
- What is AWS - Amazon Web Services?
- Downdetector suddenly turns red
Why is the internet broken? How one small problem can take down the whole web
15:23 , Andrew GriffinThe scale of today’s outage has prompted a somewhat familiar question: do we rely on a few small companies too much? (But it’s not so familiar, which might also be an indication of why we do rely on them.)
-at-its-exhibitor-stall-at-the-Indi.jpeg?trim=0,140,0,139&width=1200&height=800&crop=1200:800){kind=logo}
Outage might not be over
15:19 , Andrew GriffinAmazon’s service status page has just been updated again. The problem might not be entirely fixed.
“We can confirm significant API errors and connectivity issues across multiple services in the US-EAST-1 Region,” it reads. “We are investigating and will provide further update in 30 minutes or soon if we have additional information.”
In simpler terms: things seem to be breaking again.
Is the internet too centralised?
14:39 , Andrew GriffinThat’s the question asked in a new piece from our colleagues on the Voices team. The internet was originally intended as a self-sustaining network, kept alive by all of its users, notes Chris Stokel-Walker. But these days it relies on a very very small number of companies to keep much of the web running.
Read the full piece here.
EA App stops working, users complain
14:27 , Andrew GriffinSome players are saying that their EA app is offline, and they are unable to play games (such as has the newly released Battlefield 6) as a result.
“The app couldn’t connect to EA’s servers,” an error message reads. “Check your internet connection, and try again.”
EA’s status website doesn’t yet show any problem with any of its platforms.
It’s not clear whether it’s related to the AWS outage. (If it is, then it would be strange if it began now, given that everything else is coming back to normal.) But either way it’s been a big day of things not working.
Problems could continue for the 'rest of the day'
14:14 , Andrew GriffinThe underlying issue might be fixed. But it could take some time before everything is working as expected, says one expert.
“This morning an issue affecting Amazon’s US East caused significant disruption, with the underlying cause attributed to a problem with DNS - the ‘phone book’ of the internet that helps computers find each other,” Graeme Bragg, a computer networking expert at the University of Southampton.
“This has affected lots of services hosted in different regions because Amazon host a lot of their management infrastructure that other services depend on in that region. This is a significant outage because of how many companies rely on Amazon and the global scale of the impact.
“The underlying issue has now been resolved but it will take some time to propagate across the internet. Amazon’s servers also have a backlog of requests to work through, so it’s likely that we will see some disruption for the rest of the day.”
Reddit says it has fixed its issues
12:48 , Andrew GriffinA post on Reddit’s status page says that it is aware of an earlier issue but that it has since been fixed. It doesn’t say whether it was actually related to the broader AWS outage. (It certainly seemed to begin later than that.)
Outage might be making it harder to respond to the outage
12:41 , Andrew GriffinThe nature of the outage might actually make it difficult to respond to the outage, points out Jon Crowcroft FRS FREng, Marconi Professor of Communications Systems at the University of Cambridge.
“One interesting challenge is that the back channels a lot of tech people use to communicate information/tech details about ongoing outages are also taken down by this outage - hence our usual ways of learning (e.g. via signal or slack) are both currently stymied by the AWS outage,” he said.
Complaints seem to be falling – but problems persist
12:40 , Andrew GriffinDown Detector, the tracking website for outages, has been an alarming red all day. And it’s still that colour, but it does seem to be calming down a little.
Amazon has said that it is still working towards full recovery, so it makes sense that some people will continue to run into problems with some apps.
Amazon says problem has been solved – but some issues could remain
12:04 , Andrew GriffinThe latest update from Amazon Web Services says that it believes the problem has been fixed. But it notes that might be some issues – slow responses, and so on – while everything gets back online.
The full thing is a little technical but can be seen below:
“The underlying DNS issue has been fully mitigated, and most AWS Service operations are succeeding normally now,” it reads. “Some requests may be throttled while we work toward full resolution. Additionally, some services are continuing to work through a backlog of events such as Cloudtrail and Lambda.
“While most operations are recovered, requests to launch new EC2 instances (or services that launch EC2 instances such as ECS) in the US-EAST-1 Region are still experiencing increased error rates. We continue to work toward full resolution.
“If you are still experiencing an issue resolving the DynamoDB service endpoints in US-EAST-1, we recommend flushing your DNS caches. We will provide an update by 4:15 AM, or sooner if we have additional information to share.”
Outage shows a 'stark reality', another expert says
11:41 , Andrew GriffinRimesh Patel, a cyber security specialist, says that the outage is a reminder of the way that small problems can become big very quickly, when everyone depends on the same things.
“This major online outage underscores a stark reality: Business operations associated with one critical vendor in a region can cascade into global instability. What began as a service interruption has rippled outward, potentially compromising key systems at the very start of the business week – an illustration of how supply chain and infrastructure resiliency must be front of mind for every organisation,”
“Amazon has reportedly committed full resources to restoring affected services, but in the interim the burden falls to other organisations to mobilise rapid responses, isolate impacts, and limit service degradation wherever possible.”
Problem might be something 'relatively simple', expert says
11:23 , Andrew GriffinAlan Woodward, visiting professor of computing at the University of Surrey, has said that while we still don’t know the cause of the problem it is likely to be something straightforward. (See previous post about the possibility of a hack.)
“Although we don’t yet know the exact cause of the outage at AWS history suggests it will be something relatively simple like a misconfiguration in DNS or BGP,” he said. “Once these errors propagate across the Internet it takes a while for the update to reach the far corners of the Internet so the outage can appear longer than you might expect for such minor errors.
“What this episode has highlighted is just how interdependent our infrastructure is. So many online services rely upon third parties for their physical infrastructure, and this shows that problems can occur in even the largest of those third-party providers. Small errors, often human made, can have widespread and significant impact.”
... but problems still seem to remain
11:15 , Andrew GriffinDespite Amazon’s reports that the problems are fixed, many services are still continuing to struggle.
From personal experience, that includes apps and websites that had previously been working. Reddit and Slack, for instance, had mostly been fairly stable but they are both failing to load properly over the last few minutes.
Amazon says things are continuing to recover
11:12 , Andrew GriffinThe latest update on Amazon Web Service’s status page suggests that it believes the problem is fixed, and that everything should be recovering.
“We continue to observe recovery across most of the affected AWS Services,” it reads. “We can confirm global services and features that rely on US-EAST-1 have also recovered. We continue to work towards full resolution and will provide updates as we have more information to share.”
Could HMRC and DWP payments be affected by the outage?
11:11 , Albert TothThe giant reach of Amazon Web Services (AWS) has become clear this morning as many services and applications that people use everyday suddenly stopped working.
The company provides internet infrastructure services to hundreds of companies which use its servers to run their apps and websites.
Not confined to the private sector, several UK government departments are included in AWS's vast clientele. Both HMRC and Department for Work and Pensions (DWP) have contracts worth multi-millions with the company.
Many users have reported having issues with HMRC this morning, as outage-monitoring website Down Detector shows nearly 800 people reporting issues. Being unable to access the tax authority's online portal will mean difficulties for those looking to pay certain taxes, especially business owners.
The DWP has not reported any issues so far, but AWS's hybrid cloud services are responsible for all the department's hosting requirements. Anyone who is expecting a benefit payment soon should keep an eye out for the latest communication from the department and, if it does not arrive in due course, contact its helpline.
Could the internet have been hacked?
10:16 , Andrew GriffinWhen there’s a big outage of this kind, it’s always tempting to wonder whether it’s the result of a cyber attack. Security experts have long been warning that the infrastructure underpinning the internet – and as a result underpinning much of the world – could be more fragile than we realise, and that it might one day be broken down by hackers.
There is nothing to indicate that this is a cyber attack. Amazon Web Services doesn’t seem to know exactly what the problem is, and there has been nothing explicit to rule it out, but there is no reason to believe yet that it was an intentional hack.
Such hacks are of course on the rise. In recent months we have seen companies such as Jaguar Land Rover brought to a complete standstill by cyber attacks.
But, more often than not, outages of the kind being seen at Amazon Web Services are the result of an error rather than intentional sabotage. They can stem from a misapplied update, a physical problem at a data centre or just a wrong change to an important setting.
Amazon says it has found a 'potential root cause'
10:12 , Andrew GriffinIt’s now been two hours since the problems began. Amazon says that it has found a “potential root cause” of the problem – and so is presumably at least some of the way to fixing it.
In slightly more technical terms, this is what it has to say: “We have identified a potential root cause for error rates for the DynamoDB APIs in the US-EAST-1 Region. Based on our investigation, the issue appears to be related to DNS resolution of the DynamoDB API endpoint in US-EAST-1.”
(That doesn’t really tell us much either about the precise nature of the problem or when or how it might be fixed.)
UK banks Lloyds and Halifax stop working
10:01 , Andrew GriffinLloyds, Halifax and Bank of Scotland – which are part of the same parent company – are all seeing problems too, according to tracking website Down Detector and user reports. Affected customers say they are unable to get onto their internet banking.
Amazon Web Services says it is still looking for the cause of the problems
09:44 , Andrew GriffinThe outage has now been going on for more than 90 minutes. Amazon has posted a range of updates to its status page, but they make clear that it’s still not entirely clear what the problem is.
“Engineers were immediately engaged and are actively working on both mitigating the issue, and fully understanding the root cause,” the latest update reads.
What is Amazon Web Services?
09:42 , Andrew GriffinAmazon Web Services might be described as the biggest thing on the internet that you’ve never heard of. (But that might not be true, because actually it’s been growing in prominence, and Amazon itself is spending money on advertising it, among other things. Still, it’s a nice line that’s not entirely untrue either.)
AWS powers the internet, offering cloud services and other web infrastructure to companies. It is not alone in doing that – it competes with Microsoft Azure and Google Cloud, most notably – but it is the biggest in the industry.
It means in short that a company can rent computers, servers and other technologies from Amazon. For the companies, that means access to the infrastructure required to show apps to people without having to build their own data centres; for Amazon, it benefits from economies of scale that allow it to build massive data centres and share out those computing resources as and when they are needed.
AWS officially launched in 2006, though its roots go further back than that. In the 20 years since it has grown rapidly, and now powers much of the internet and is responsible for most of Amazon’s profits.
Problems present everywhere
09:37 , Andrew GriffinThe issue at Amazon Web Services appears to be related to a specific facility in northern Virginia. But the problems are affecting users around the world – tracking website Down Detector shows reports from just about everywhere, across the various affected apps.
Here’s a big list of everything that is showing as having problems on Down Detector. It includes everything from online games (Clash Royale and Roblox) to the UK’s tax authority:
Snapchat, Ring, Roblox, Clash Royale, Life360, My Fitness Pal, Xero, Canva, Amazon, Amazon Web Services, Amazon Music, Prime Video, Clash of Clans, Fortnite, Wordle, Duolingo, Coinbase, HMRC, Vodafone, Playstation, Pokémon Go.
Amazon says it is experiencing problems
08:41 , Andrew GriffinAn update has been posted on Amazon’s service status website, which makes clear that it is experiencing problems at its facilities in north Virginia. The issues are affecting Amazon DynamoDB and Amazon Elastic Computer Cloud, both of which allow companies to rent storage and computers to run their services.
Downdetector suddenly turns red
08:38 , Andrew GriffinDowndetector, a website that tracks complaints about websites and web services not working, shows the sudden and widespread nature of the outage.
Hello and welcome...
08:34 , Andrew Griffin... to The Independent’s live coverage of an ongoing internet outage that has taken down much of the internet.
