Summary
We’re going to close our coverage of the stock market outage for the day with a quick summary of what happened and what we do and don’t know.
- The New York Stock Exchange abruptly shut down at 11.32am local time and did not resume trading until about 3.10pm. NYSE president Thomas Farley said that the outage was caused by technical “configuration problem” that was “fixed”, but he did not describe what exactly went wrong and forced the midday shutdown. He said an investigation would begin on Wednesday night.
- Homeland Security secretary Jeh Johnson said that there was no indication the outage was linked to “nefarious” actors or a cyberattack, but law enforcement, SEC and Treasury officials said that although were in close communication with Wall Street nonetheless.
- Stocks closed down on the day by about 1.5%, but without any apparent panic over the technical failures of the US’s premier stock exchange. Chinese markets continued a free fall on the other hand, and with debt fears in Europe rattled the market.
- The outage revives questions about the automation of Wall Street, with some traders raising concerns about the dependency on machines, unintended consequences and having recovery systems in the case of a true disaster.
- Officials said there was no reason to believe the technical problem was linked to coincidental problems in United Airlines computer systems or at the Wall Street Journal.
The outage was caused by a “configuration problem in our system that needed fixing,” NYSE president Tom Farley has told CNBC, but adds: “whether that came up as a result of an update is premature” to say.
“We found what was wrong and we fixed what was wrong and we have no evidence whatsoever to suspect that it was external.
“Tonight and overnight starts the investigation of what exactly we need to change. Do we need to change those protocols? Absolutely. Exactly what those changes are I’m’ not prepared to say.”
Farley argues that suspending trading on the floor was actually the lesser of two evils compared to shifting trading to the emergency recovery center in Chicago. “It was never really an option for us to go to the disaster recovery,” he says.
The NYSE would have shifted to Chicago only in the event of a “catastrophe, a hurricane” or something similar, he adds.
“We have a playbook that we follow, and that involves making a decision like I did today,” he says. “One of the important concepts or precepts of our plan is to choose the least disruptive option to customers wherever possible.”
“Ultimately while we think of the cloud and the ether and the internet, ultimately you’re connecting to a place … so in order to go to a disaster recovery system you need to point to that place.”
Finally he adds that the NYSE was in “near constant communication, not just today” with government and security agencies.
No matter how bad a day today was for the tech developers at the NYSE, traders in China had a worse day as their stock market continued its three-week nosedive. They’ve also had a bit of fun at New York’s expense.
Buzzfeed has collected and translated some of the better quips:
“In order to set an example of trading halt for China’s stock markets, NYSE did its best1”
“Is it because NYSE traders bought Chinese stocks?”
“Maybe it’s done by our men.”
You can read the set here.
Market closes
The bells sound and the traders clap: it’s closing time on Wall Street after a three and a half hour outage blamed on technical problems.
Although the NYSE was closed for half the day, trading continued through other venues, and the day of decline continued as concerns about the Chinese market and Greek debt influenced the market.
All in all, the Dow was down about 260pts (1.47%), the S&P down 35pts (1.67%) and the Nasdaq down 87pts (1.75%) – not a great day for Wall Street, though nothing like the free fall and panic from the day in China.
Need more dystopia in your day of Wall Street semi-shutdowns? Molly Crabapple has your apocalyptic vision over at Comment is Free.
Outside, I take in the scene: street preachers denouncing Gnosticism, a lone banker trying to garrote himself with ticket tape, and the Bull – that gold, beautiful bull – running through the streets like Zeus. I chase after it for a quote, but, like the dubious financial transactions powered by super-compressors, it is too quick.
Smoke. Weeping. Screams.
I hire now-former JP Morgan CEO Jamie Dimon as my local fixer. “Tell me the ways of your people, caught as they are between the present and the ancient past,” I demand, offering him half a hotdog as payment. Instead, he weeps. He tried to seek shelter at the dungeon of his favorite pro domme, he tells me, but when his black card bounced she slammed the door in his face.
In Zuccotti Park, once the home of Occupy, the ex-Goldman Sachs boys have built a squatters city out of Hermès gift boxes that were meant for their mistresses. They communicate only by wiggling their fingers. No cops try to roust them. Behind us, Tiffany’s burns.
Then we hear the chanting. Dear god. The chanting. Dimon and I run towards it.
You can read the full piece – eldritch offerings, taurine testicles and all – here.
Four hours later and less than half an hour before closing time, the New York Stock Exchange announces that it’s back to business as usual.
BREAKING: NYSE says "all systems are functioning normally" after trading resumed after 3+ hour trading halt » http://t.co/jvkuQmTVYV
— CNBC Now (@CNBCnow) July 8, 2015
Market outages are neither uncommon nor much of a problem for investors, according to the Wall Street Journal, which has published a story about closures not long after the newspaper’s own site went down on Wednesday.
No exchange has exclusive rights to trade any stock or exchange-traded fund, except during the opening and closing of trading. Apart from those brief periods, trading takes place at any of a dozen exchanges and more than 40 private trading venues, known as dark pools.
Outages could hurt traders and investors if they have orders sitting on the order book of an exchange that has a problem. So if an institution has an offer at an exchange to buy 1,000 shares of Apple Inc at a certain price, and the market stops trading, it can cause uncertainty.
The institution might not be able to find out whether its order traded, and it could have trouble canceling the order. The NYSE said in a note to traders that all open orders had been canceled except the long-term requests to buy or sell put in by institutions that are known as “good-till-cancel” orders.
“Outages have hit the markets every few months in recent years,” the Journal continues, adding that the real problem would have been the prevention of closing auctions when most trading takes place.
Those final auctions also “set the closing price for NYSE-listed stocks” and set the markets in place for businesses and investors to take stock (sorry) of the day.
Many firms use those final prices to mark the value of their portfolios and set risk parameters. Without those closing prices, they might be forced to use less-accurate prices, and that could affect other trading decisions.
There is some precedent for resolving problems with the closing auctions. In November 2012, technical issues forced the NYSE to find an alternative when it couldn’t hold closing auctions for 216 stocks, according to a note put out by Credit Suisse trading researchers Wednesday afternoon. In that case, the NYSE determined a closing price by monitoring transactions at other venues.
As the machines once again match bids and quotes, the stock exchange tweets its official blessing upon the markets.
There’s no fanfare on the floor – just numbers gradually starting to change again on monitors and a little more activity among the traders.
Trading on NYSE and NYSE MKT has resumed
— NYSE (@NYSE) July 8, 2015
Stock exchange resumes trading
The New York Stock Exchange has returned to business after more than three and a half hours of an unexpected closure, which officials blamed on technical problems.
The exchange closed at 11.32am ET and re-opened just under an hour before the close of markets at 4pm ET.
The secretary of the Department of Homeland Security said that there’s no indication the abrupt stoppage was linked to “nefarious” actors. NYSE president Thomas Farley said that the technical problem was “identified” and “addressed” but did not describe what happened.
Officials said there was no reason believe that the NYSE glitch is at all linked to coincidental technical problems at United Airlines and the Wall Street Journal.
Although trading continued apparently on other open exchanges apparently unperturbed by the outage, the Dow is currently more than 200 points down.
Updated
“It was a mess from the open,” a broker has told Reuters, which runs down what happened this morning and the importance of closing time on Wall Street.
Some of the internet ports to the NYSE would not connect, or would unexpectedly disconnect, and this was happening from before market opened until the halt, the broker said.
The exchange does a lot of its business at the close of trading, and if it’s still down then, the problems could increase.
“That could be a major problem,” said Peter Costa, president of boutique trading firm Empire Executions Inc. “The bigger issue is that there are unfilled orders that would likely not get filled at the closing price.”
“They don’t need human beings,” an options trader opines on the tech woes of Wall Street, in video shot by my colleague Jana Kasperkevic.
Officials at the NYSE have just announced that they have a plan, CNBC reports.
Officials have said they’re going to try to open the American Stock Exchange at 3.05pm ET and the New York Stock Exchange at 3.10pm ET, leaving just under an hour of trading if the gates re-open without a hitch.
Does the automation of the stock market mean many more days of glitches, breaks and malfunctions interrupting the rivers of cash moving around the world?
At least one trader foresees problems in the rise of the machines – but says that doesn’t mean the markets need people per se. My colleague Jana Kasperkevic (@kasperka) reports from outside the NYSE.
Hours into the unexpected break in trading, an options trader who gave his name as Chris stepped outside to puff away on his e-cigarette, and eventually relented to questions about the hours-long break in trading.
“We need better technology. I mean, this hasn’t happened ever on the New York Stock exchange, for what they say,” he said. “I’m in options. I am not on the equity floor. I have never seen this happen on the New York Stock exchange ever, where all symbols halt at the same time.
“It makes me worried that they are trying to make this a fully automated exchange, and what they are doing, they are slowing things down on purpose, saying ‘we don’t need people.’
“If we had different technology and we didn’t have people running certain things, we wouldn’t have this problem. They are trying to make everything electronic. That’s what happened.”
But does the exchange need people? “They don’t need human beings.”
Fortune Magazine has also postulated that a problem in a software update is what brought the NYSE to its knees – or at least froze it in place, since stocks are trading in other venues. (via @nicflatow).
Eric Scott Hunsader, an expert in Wall Street trading systems who heads market data firm Nanex, said that it appears that a faulty system upgrade brought trading on the exchange to a halt. The NYSE has reportedly also told floor traders the exchange had to suspend trading due to an error with a systems upgrade that was rolled out before the market opened on Wednesday.
The Guardian’s Sam Thielman has a couple follow-up questions.
If Fortune is right that a "systems upgrade" broke the NYSE, A) who makes that software B) is that company publicly traded C) can I short it
— Sam Thielman (@samthielman) July 8, 2015
Farley projected the NYSE to resume trading within the hour, which would give the exchange roughly an hour of running normally before the daily “close” of 4pm ET.
Meanwhile the New York Times’ Nathaniel Popper reports that the NYSE had to manually cancel some 700,000 existing orders so that it could restart the exchange, citing a trader on the floor.
Floor trader says the NYSE is rebooting its systems, which can take 45 minutes, and hopes everything is working when it's up again
— Nathaniel Popper (@nathanielpopper) July 8, 2015
NYSE president: exchange will re-open today
The NYSE will be up and running again before close today, president Thomas Farley has just told CNBC.
“We identified the issue and we have that addressed,” he says, estimating between a return to business around 2.45-3pm.
He adds that it’s “premature” to say what the actual issue is. “Our system is more than just software, the network connectivity, the various external internal sources, and the actual hardware.
“It’s not a good day and I don’t feel good for our customers who continue to deal with the fallout.”
Farley adds: “I wanted my team to take their time and get it right, be open for the close.”
He says that he does not believe there’s any connection between the glitches that occurred today at different organizations, and that if there had been a “catastrophic issue” then there is a disaster recovery plan in place. “You need to have absolute resiliency.”
Updated
Guardian editors weigh in. Tech support from national security editor Spencer Ackerman.
Have they tried turning the stock exchange off & turning it back on?
— Spencer Ackerman (@attackerman) July 8, 2015
The usual suspects, courtesy business editor Dominic Rushe.
#NYSE shutdown was it a) Time Warner Cable went down or b) Y2K?
— dominic rushe (@dominicru) July 8, 2015
And a fear of time itself from live editor Paul Owen.
Was it the leap second that caused all this chaos? http://t.co/Lm5zinMB1e
— Paul Owen (@PaulTOwen) July 8, 2015
“At this point it’s unclear what kind of impact this particular glitch will have,” adds Earnest, to whom Guardian DC bureau chief Dan Roberts was also listening.
Earnest says it’s “too early to offer an assessment [on any economic impact]” but said the Treasury department and Securities and Exchange Commission were closely monitoring the situation.
The NYSE has not given any indication as to its timeline for resolving the problem, and trading continues on other exchanges. On Wall Street life goes on.
President Barack Obama was briefed this morning on the suspension on Wall Street, White House press secretary Josh Earnest has just said during a press briefing.
Earnest’s message (paraphrased): don’t panic.
“The officials at the stock exchange are working feverishly, as you would expect.”
Earnest said that “there is no indication that malicious actors are involved in these technology issues” and that NYSE officials “had been in close touch with the Department of Homeland Security, with the [SEC] as well as the Treasury Department.”
Obama briefed on NYSE outage by homeland security adviser Lisa Monaco who says 'no indication of malicious actors involved' says @PressSec
— Dan Roberts (@RobertsDan) July 8, 2015
Earnest goes on to say that more generally, the government is “keenly aware of the risk in cyberspace that exists right now” and wants to “improve communication between the private sector and the federal government.”
China and Russia have been recently blamed for massive data breaches and cyber attacks against the US government, including on data of people with security clearances and on the White House email system.
“Our security posture is continually evolving,” Earnest says.
Updated
Two hours into the stoppage, the apparent electronic glitch is not inspiring Wall Street about the quality of the exchange, Guardian US business editor Dominic Rushe (@dominicru) reports:
Jack Ablin, chief investment officer of BMO Private Bank, said he initially worried that China’s woes had spread to the US when he heard the news and was relieved that it appeared to be a technical glitch. “That said it doesn’t add a lot of confidence in the markets when investors don’t have a lot to begin with,” he said.
China’s stock markets have fallen more than 30% in the last three weeks after massive gains in the last year.
At CNBC, journalists and investors are avidly discussing the foibles and flaws of one of the most important stock exchanges in the world.
“At the NYSE this is a manual process,” said Sal Arnuk, a principal at Themis Trading. “Is the NYSE technologically the most (robust) exchange in the world? No. The fact of the matter is the different exchange operators have diverse standards, different architecture. Some of them are more legacy than others. This is to be expected from time to time.”
Meanwhile a trader has told the New York Times on condition of anonymity that after trading stopped, they were told “that the problem was related to updated software that was rolled out before markets opened”.
According to the trader, the exchange said that the new software caused problems soon after trading began on Wednesday and the exchange decided to shut down trading all together to fix the problem.
The exchange has been closed for more than an hour and a half now, since 11.32am. The longer it goes the more traders emerge to smoke, notes my colleague Jana Kasperkevic – but the traders are pretty insouciant about the break, at least so far.
At about 1 pm – an hour and a half after the trading halted at 11.32am, one of the traders walked over to the barricades. He had been standing a few feet away in the comfortable shade of the building he works at. After a reporter beckoned him, he strolled over and other reporters flocked to him.
“What? Do you want my autograph now?” he asked. “I can’t comment,” he said.
When asked if the trading being halted for more than hour and a half, he smiled: “What’s scary is you asking me all these questions.”
Updated
Homeland Security secretary Jeh Johnson is delivering a speech on cybersecurity in Washington, and opened by saying the stoppage is not due to “nefarious actions”. The Guardian’s Ben Jacobs is listening in.
Johnson: It appears that today we had a system malfunctions at United, the New York Stock Exchange and the Wall Street Journal
— Ben Jacobs (@Bencjacobs) July 8, 2015
Johnson says it appears that the malfunctions at United and NYSE was not the result of nefarious actions. We know less about the WSJ
— Ben Jacobs (@Bencjacobs) July 8, 2015
My colleague Jana Kasperkevic is on Wall Street, where the exchange looms up over the sidewalk. She reports:
The New York State Exchange stand regal in the downtown Manhattan, one side of it draped in huge American flag. Even as tourists snap selfies and family portraits nearby, the building remains inaccessible as the traders who work on its floors are separated from the public by metal barricades.
The exchange has claimed at least a ten-feet wide sidewalk, where traders can smoke in peace. And smoke they did on Wednesday.
About an hour after the trading halted, one walked out. “What happened?” two photographers standing nearby asked. He shrugged, “the system crashed.”
Bunch of tourists snapping selfies outside #nyse. Maybe it's the flag? pic.twitter.com/Na2BRR67Et
— Jana Kasperkevic (@kasperka) July 8, 2015
A US official has told the Guardian that there is no indication anything that happened today was the result of any malicious actor, cyber attack, etc.
The day is apparently full of coincidences of a technically glitchy nature, however. United Airlines grounded all US flights over computer issues earlier on Wednesday, telling my colleague Amanda Holpuch (@holpuch) the major delays were due to a “network connectivity issue”. The Wall Street Journal’s website also briefly shut down around midday.
Meanwhile in Washington FBI director James Comey warned senators about the dangers of commercial encryption and urged them to allow “backdoor” access into encryption software. Homeland Security secretary Jeh Johnson is scheduled to give a speech on cybersecurity later today.
For the more conspiracy-minded readers out there, hacker collective Anonymous has the tweet for you, sent late last night.
Wonder if tomorrow is going to be bad for Wall Street.... we can only hope.
— Anonymous (@YourAnonNews) July 8, 2015
The NYSE has elaborated a bit on the closure, saying it was caused by “an internal technical issue and is not the result of a cyber breach”.
A spokesperson for the NYSE released this statement: ““We’re currently experiencing a technical issue that we’re working to resolve as quickly as possible. We will be providing further updates as soon as we can, and are doing our utmost to produce a swift resolution, communicate thoroughly and transparently, and ensure a timely and orderly market re-open.”
Even should the NYSE resolve its technical issue promptly, the Intercontinental Exchange that owns it will inevitably face questions as to why it apparently gave no forewarning that it would shut down trading in the middle of an otherwise normal workday.
(2 of 3) We chose to suspend trading on NYSE to avoid problems arising from our technical issue.
— NYSE (@NYSE) July 8, 2015
(3 of 3) NYSE-listed securities continue to trade unaffected on other market centers.
— NYSE (@NYSE) July 8, 2015
The New York Stock Exchange unexpectedly stopped all trading at about 11.30am ET on Wednesday after a “major technical issue”.
The NYSE’s website only reads: “Additional information will follow as soon as possible.”
Other exchanges such as the Nasdaq remain open and functioning, and although law enforcement agencies have said they are investigating, the Department of Homeland Security said there was no sign of suspicious activity.
The largest stock exchange in the US has suffered from technical problems before, though never of such duration or scale.
The apparent technical issue coincides with sharp falls in the Chinese stock market and fears in Europe over the Greek debt crisis.
We’ll include updates on the NYSE, markets and other developments here, assisted by my colleague Jana Kasperkevic (@kasperka) on Wall Street, and Dan Roberts (@robertsdan) and Ben Jacobs (@bencjacobs) will report on any developments on Washington about the stoppage.
Updated
