The CrowdStrike fail and next global IT meltdown already within the making

When computer screens went blue worldwide on Friday, flights were grounded, hotel check-ins became inconceivable, and freight deliveries were dropped at a stand-still. Businesses resorted to paper and pen. And initial suspicions landed on some kind of cyberterrorist attack. The truth, nevertheless, was rather more mundane: a botched software update from the cybersecurity company CrowdStrike.

“On this case, it was a content update,” said Nick Hyatt, director of threat intelligence at security firm Blackpoint Cyber.

And since CrowdStrike has such a broad base of consumers, it was the content update felt all over the world.

“One mistake has had catastrophic results. That is an excellent example of how closely tied to IT our modern society is — from coffee shops to hospitals to airports, a mistake like this has massive ramifications,”  Hyatt said.

On this case, the content update was tied to the CrowdStrike Falcon monitoring software. Falcon, Hyatt says, has deep connections to watch for malware and other malicious behavior on endpoints, on this case, laptops, desktops, and servers. Falcon updates itself routinely to account for brand spanking new threats.

“Buggy code was rolled out via the auto-update feature, and, well, here we’re,”  Hyatt said. Auto-update capability is standard in lots of software applications, and is not unique to CrowdStrike. “It’s just that as a consequence of what CrowdStrike does, the fallout here is catastrophic,” Hyatt added.

Though CrowdStrike quickly identified the issue, and lots of systems were back up and running inside hours, the worldwide cascade of injury is not easily reversed for organizations with complex systems.

“We predict three to 5 days before things are resolved,” said Eric O’Neill, a former FBI counterterrorism and counterintelligence operative and cybersecurity expert. “It is a bunch of downtime for organizations.”

It didn’t help, O’Neill said, that the outage happened on a summer Friday with many offices empty, and IT to assist to resolve the problem in brief supply. 

Software updates needs to be rolled out incrementally

One lesson from the worldwide IT outage, O’Neill said, is that CrowdStrike’s update must have been rolled out incrementally.

“What Crowdstrike was doing was rolling out its updates to everyone without delay. That isn’t the very best idea.  Send it to at least one group and test it. There are levels of quality control it should undergo,” O’Neill said.

“It must have been tested in sandboxes, in lots of environments before it went out,” said Peter Avery, vp of security and compliance at Visual Edge IT.

He expects more safeguards are needed to stop future incidents that repeat this sort of failure.

“You wish the fitting checks and balances in corporations. It might have been a single individual that decided to push this update, or any individual picked the fallacious file to execute on,” Avery said.

The IT industry calls this a single-point failure — an error in a single a part of a system that creates a technical disaster across industries, functions, and interconnected communications networks; a large domino effect. 

Call to construct redundancy into IT systems

Friday’s event could cause corporations and individuals to heighten their level of cyber preparedness.

“The larger picture is how fragile the world is; it is not only a cyber or technical issue. There are a ton of various phenomena that could cause an outage, like solar flares that may take out our communications and electronics,” Avery said.

Ultimately, Friday’s meltdown wasn’t an indictment of Crowdstrike or Microsoft, but of how businesses view cybersecurity, said Javad Abed is an assistant professor of data systems at Johns Hopkins Carey Business School. “Business owners have to stop viewing cybersecurity services as merely a price and as a substitute as a vital investment of their company’s future,” Abed said.

Businesses needs to be doing this by constructing redundancy into their systems.

“A single point of failure shouldn’t have the ability to stop a business, and that’s what happened,” Abed said. “You possibly can’t depend on just one cybersecurity tool, cybersecurity 101,” Abed said.

While constructing redundancy into enterprise systems is dear, what happened Friday is dearer.

“I hope it is a wake-up call, and I hope it causes some changes within the mindsets of the business owners and organizations to revise their cybersecurity strategies,” Abed said.

What to do about ‘kernel-level’ code

On a macro level, it’s fair to assign some systemic blame inside a world of enterprise IT that always views cybersecurity, data security, and the tech supply chain as “nice-to-have things” as a substitute of essentials, and a general lack of cybersecurity leadership inside organizations, said Nicholas Reese, former Department of Homeland Security official and instructor at Latest York University’s SPS Center for Global Affairs.

On a micro level, Reese said the code that caused this disruption was kernel-level code, impacting every computer hardware and software communication aspect. “Kernel-level code should get the best level of scrutiny,” Reese said, with approval and implementation needing to be entirely separate processes with accountability.

That is an issue that can proceed for the complete ecosystem, awash in third-party vendor products, all with vulnerabilities.

“How will we look across the ecosystem of third-party vendors and see where the following vulnerability will probably be? It is nearly inconceivable, but now we have to try,” Reese said. “It isn’t a possibly, but a certainty until we grapple with the variety of potential vulnerabilities. We want to concentrate on backup and redundancy and put money into it, but businesses say they cannot afford to pay for things which may never occur. It’s a tough case to make,” he said.