Earlier this year, a downright chilling cyberattack against our nation’s critical infrastructure was exposed and reported in Oldsmar, Florida, a town of fewer than 14,000 people just outside of Tampa. The attack was targeted against a local water treatment facility and – if successful – could have managed to poison the area’s water supply.
While this sounds like a plot straight out of the Adam West Batman television series, it was very real, and it was luckily thwarted by individuals exposing and correcting the malicious activity before anything harmful could happen. But it still creates a lot of questions about how this could have happened, and what could have been the result should the attack been better executed, or had employees been less vigilant.
Unfortunately, this attack in Florida is not an isolated incident or outlier. It’s yet another attack against our nation’s critical infrastructure, which is seemingly increasing as the systems and applications that run these facilities and manage these necessary services become network-enabled and connected.
To learn more about why our nation’s critical infrastructure is vulnerable to cyberattack, what kind of malicious actor would want to hack our critical infrastructure, and what these overworked, under-funded organizations can do to protect themselves, we sat down with Elia Zaitsev, the Chief Technology Officer (CTO) at cybersecurity juggernaut, CrowdStrike.
During our discussion, we asked Elia hard questions about why these attacks are so common, why the systems that manage our critical infrastructure are even connected to networks and the Internet, and what the government can and should be doing better to protect citizens. Here is what he told us:
GovCyberHub (GCH): Why are we suddenly starting to hear about critical infrastructure falling victim to cyberattacks? Is this a result of changes in the equipment and systems employed within these organizations, advancements in the tools and strategies used by malicious actors, or some other factor?
Elia Zaitsev: I would argue that we aren’t suddenly starting to hear about these types of attacks. The Center for Strategic and International Studies (CSIS), a Washington-based nonprofit think tank, has been tracking significant cyber incidents since 2006. As far back as 2008, they cite CIA officials with knowledge of several overseas incidents where power supplies were disrupted to foreign cities. NSA Director, General Alexander, described a seventeen-fold increase in cyber incidents against American infrastructure companies from 2009-2011.
Targeting of critical infrastructure significantly picked up in 2012, when DHS issued warnings of campaigns targeting US gas pipelines; a SCADA security company was attacked by adversaries linked to China’s People’s Liberation Army; and the “Mahdi” malware family was involved in reconnaissance against hundreds of critical infrastructure engineering firms, hackers stole SCADA system data from a Canadian industrial automation company, and two US power plants were infected through rogue USB drivers.
The earliest example I’ve come across took place in the year 2000 – and, no, it was not a Y2k incident – at the Maroochy Water Services sewage facility in Australia, when a disgruntled insider accessed control systems and caused 800,000 liters of raw sewage to spill into parks, rivers, and the grounds of a local hotel.
“…let’s not fall for the illusion that air gaps are a panacea, particularly against determined and well-resourced nation-state adversaries. I previously mentioned power plants in the US being infected with malware from a malicious USB drive. That’s not an isolated incident.” – Elia Zaitsev
I could go on, but the list only grows as we approach the present. The examples I’ve selected only include targets that would traditionally be categorized as “critical infrastructure”. One can easily make the case that there are many more digital services we rely on heavily today that could be considered just as critical in areas such as telecommunications, banking, cloud services, etc. We are only just beginning to pay attention for the first time and take notice as they have started to hit closer to home and begun to impact our everyday lives in the real physical world.
The majority of these and other examples did not actually rely on any new or sophisticated tools and tactics not used previously. Rather they typically leveraged tried and true techniques like spear phishing, malicious removable media, unpatched vulnerabilities, and weak or default passwords.
GCH: Why are the organizations that run and manage our critical infrastructure implementing equipment and solutions that are network-enabled and connected? What benefits are they gaining from network-enabling these systems? Wouldn’t it be better if the systems were completely air-gapped from the outside world?
Elia Zaitsev: There are a variety of reasons and justifications for having network-enabled equipment in these environments. Some obvious ones include remote monitoring and maintenance of the connected equipment, as well as software and firmware updates–including those that patch vulnerabilities.
If you think about the sheer physical size and number of locations involved in things like our energy and utility grids, it becomes clear why some type of remote management is a necessity. Without significant increases in funding and staffing for these organizations, it’s just not feasible to have human operators present at all these sites.
A great example is the recent attack against a Florida water treatment facility. There are an estimated 54,000 similar water systems in the US, the majority serve areas with under 50,000 residents, and many in rural areas serve just a few hundred people. Increasing funding is not a trivial matter as most of these entities are heavily regulated and are funded by usage fees, meaning politicians face a delicate balance between badly needed upgrades to decades-old systems and anger from consumers over rising bills.
Even with limitless resources, many elements of our critical infrastructure – things like offshore oil rigs, wind farms, or nuclear reactors – reside in hard-to-reach, distant, and isolated locations, or involve dangerous working conditions.
Lastly, let’s not fall for the illusion that air gaps are a panacea, particularly against determined and well-resourced nation-state adversaries. I previously mentioned power plants in the US being infected with malware from a malicious USB drive. That’s not an isolated incident. Similar tactics were used in the widely-reported Stuxnet attack against the Iranian nuclear program.
“We did get lucky that the crude nature of the attack quickly set off alarms…but it’s not unreasonable to believe that – given enough time and understanding of the systems involved – a more sophisticated and determined adversary could have succeeded.” – Elia Zaitsev
Humans in the form of system operators and contractors will always remain a potential threat vector as well, either by social engineering or even direct recruitment by foreign intelligence services.
Air-gapping these types of sites would come with other safety trade-offs including delays in the ability to respond to problems, and an inability to monitor for possible intrusions and attacks of the kind I just outlined that does not require an active network connection.
GCH: What kinds of malicious actors and hackers would want to target these systems? What motivation is behind an attack like the attack in Florida which – if successful – could have made people really sick?
Elia Zaitsev: As is the case with the majority of cyberattacks, adversaries typically fall into one of three categories that vary in the danger they present and level of sophistication.
First, there are the hacktivists. This is generally the least sophisticated and presents the least danger. It’s a broad category that includes both “lone wolves” and groups of people with motivations that can vary from curiosity to fame and attention-seeking, to attempts to advance a political cause or agenda. They are usually – but not always – the least sophisticated and organized actors of the three categories.
Next, there are the e-criminals. This category is primarily motivated by financial gain, and the ransomware attack is the most common and widely known tool that they employ. There are a number of examples of e-crime attacks in recent years, with varied targets ranging from private companies, hospitals, schools, municipal governments, and even critical infrastructure. The skills and tradecraft exhibited by some of these groups have been quite sophisticated in recent years, borrowing from and – in some cases – blurring the lines between them our next category, the nation-state actors.
The nation-state actors are the most sophisticated and well-resourced category, these groups are typically directly controlled and funded by military and intelligence organizations. Their objectives range from espionage and reconnaissance to full-on cyber warfare preparation. Such adversaries have demonstrated an interest in U.S. critical infrastructure.
Unfortunately, due to our size and heavy reliance on technology, America is uniquely vulnerable to a hostile cyberattack. The recent power outages in Texas that were a result of cold weather serve as a stark reminder about the economic and human toll that can accompany infrastructure disruptions
Turning back to the recent attack in Florida we don’t know for sure yet who was behind the attack or their true motivations. The likely attack vector seems to have been a remote access tool called Teamviewer, and legitimate credentials that were somehow learned or guessed. The attackers did little to cover their tracks, were quickly spotted, and appeared – or wanted to appear – as if they did not have a good understanding of the systems and settings involved.
GCH: It sounds like the attack in Florida was thwarted at almost the last minute by someone noticing the changes being implemented or requested and then reversing them. Did it have to get that far? What steps could they have taken to mitigate this problem earlier?
Elia Zaitsev: We did get lucky that the crude nature of the attack quickly set off alarms and controls before any actual changes to the water supply happened. Specifically, a setting controlling the amount of sodium hydroxide (lye) was set to 100x its normal value, which quickly triggered various safety measures and physical limitations of the system. It’s unclear if this tactic could have been executed properly and injured people, but it’s not unreasonable to believe that – given enough time and understanding of the systems involved – a more sophisticated and determined adversary could have succeeded.
For any breach, the object is for defenders to identify and remediate the issue as soon as possible, and before the adversary can achieve their objectives There are several steps that might have prevented this specific attack vector – the abuse of a remote access tool with legitimate credentials – and other common threats faced by critical infrastructure organizations.
First, these organizations should be implementing password hygiene policies. It’s unclear how the adversary was able to obtain the credentials involved, but in the absence of other more effective controls – at a minimum – passwords should meet minimum complexity and rotation requirements.
“This specific attack was thwarted because a human operator noticed unusual activity in the industrial control systems (ICS) involved and quickly took action. Beyond monitoring the ICS control interfaces, the underlying systems need to be instrumented and monitored continuously and constantly.” – Elia Zaitsev
Next, these organizations should embrace identity and access management, including multi-factor authentication – No critical system should be accessible with just a password. Additional steps can and should be taken to ensure that only trusted individuals can access these devices, including the use of encryption certificates – like SSH keys – and multi-factor authentication (MFA). Zero-trust philosophies and approaches are also very relevant here.
Critical infrastructure organizations should also explore implementing network access controls, which – depending on the nature of the environment – could enable some additional security. For example, if remote access software is only intended to be used from authorized facilities and terminals, technologies such as firewalls and VPNs could be configured to only allow connections from those locations.
It’s also essential to get back to basics and work to install proper IT hygiene across the organization. While unpatched vulnerabilities do not seem to be a factor in this specific incident, they are in countless others where they were. Robust IT hygiene programs that include patch, asset, and identity management are a must for every organization.
Finally, these organizations should implement robust visibility and threat hunting tools and services. Ultimately, we must assume that no matter how many controls and safeguards are in place, a determined adversary or malicious insider will always find a way. If we keep this assumption in mind, then the need for visibility and proactive threat hunting becomes obvious.
This specific attack was thwarted because a human operator noticed unusual activity in the industrial control systems (ICS) involved and quickly took action. Beyond monitoring the ICS control interfaces, the underlying systems need to be instrumented and monitored continuously and constantly. Endpoint Detection and Response software (EDR) is a must and should be used whenever possible to watch and record all activity throughout the environment.
Many organizations will try and shortcut this with simpler network-level visibility, but these are poor substitutes for true endpoint visibility due to the pervasive use of encryption and legitimate credentials. Software alone is not the only answer; trained and experienced human operators must monitor and proactively hunt for threats around the clock. The skill-set and human resources required to do this will not be available to every critical infrastructure provider. Partnerships with government organizations and private sector providers can help close the gap.
GCH: With water treatment facilities and other critical infrastructure organizations, the main mission isn’t IT or cybersecurity-related and they’re often struggling to accomplish their primary mission with the tools, resources, and staff they have. How can organizations like this be better protected? What can we do at the organizational level? Local government level? Federal level?
Elia Zaitsev: While there will be unique methods and challenges in different areas, there are common standards and principles that should be adopted by everyone – whether they are related to water treatment, power, and energy, telecommunications, or other critical areas. Mandatory minimum controls, reporting standards, and disclosure requirements form a baseline.
Today we have multiple industry groups that attempt to facilitate the sharing of information and best practices, such as the Water ISAC, and we think that model is promising. These are voluntary groups that specifically promote collaboration and pooling of resources, without using heavy-handed approaches. For example, the Multi-State ISAC offers a variety of shared cybersecurity services to its members including access to cybersecurity software, incident response services, and a 24/7 security operations center that go well above and beyond simple information sharing.
These were critical in securing the 2020 US elections despite the complexity of working with 50 autonomous states and thousands of independent municipalities. Similar groups can learn from this approach.
