Blog / Water treatment security – It’s not just up to the contractor
I was reading an article the other day about how a water treatment plant in Florida was recently hacked. Someone connected to the remote software (Teamviewer) used to connect to a machine that handled mixing the water, then cranked up the levels of sodium hydroxide (lye) to dangerous levels (more then 100 times normal). Thankfully the change was noticed and reversed immediately so no harm was done.
This got me thinking about water treatment plants and how contracting companies often handle remote access. I’ve looked at the networks of several different water treatment plants across northern Alberta over the years. I’ve seen how they are managed and it seems to depend very much on the contracting company. It’s all over the place and there’s no standard.
This subsequently got me thinking about the relationship that municipalities have with contractors. It seems like most of the time a machine is set up to run software that controls the equipment and local IT is not allowed any sort of administrative access. This is likely because the software being used is “finicky” and could react badly to software updates. (This is typically the result of hardware manufacturers and not contractors, but that’s another issue entirely)
I looked into the Alberta municipal guidelines for waterworks and section 2.7.4 ‘SCADA security’ deals with cyber security. The section is split into six different subsections totaling about one full page. There are pages of detailed information regarding physical security (locks, alarms, fences, etc.), but when it comes to electronic security the advice is non-specific, vague and very, very brief. The last time Alberta updated it’s municipal guidelines for water treatment was almost a decade ago (April 2012). Case in point: the guidelines still say the installation of antivirus software “should be considered” on SCADA controllers, but is not required. Worse, nothing in that section is actually required, and everything is stuff that “should” be done.
The thing is, water treatment is important. If those mixtures get messed up it could cause damage to equipment, people could get sick, or worse, even die. It’s important to make sure that any sort of access to the control equipment is handled properly. That’s why for this newsletter I figured I would outline how to make sure your water treatment setup is reasonably secure. While you can’t dictate exactly which tools contractors use, you can put some rules in with regards to how they are used.
i: All outside access must be through a VPN with multifactor authentication
Regardless of the exact tool the contractor wants to use for remote access (Teamviewer, RDP, Etc.) there must be absolutely no direct access to any of the equipment from outside your network. In order to connect remotely the contractor should be required to login to a VPN that has multifactor authentication setup. Ideally this should only be allowed from their offices as well. Each individual authorized to work on the equipment should have their own VPN login (no generic logins, no sharing). This setup helps ensure that only authorized individuals can connect to the equipment and won’t not cause any software compatibility issues, though it may slow down their connection a bit.
ii: There should be a long term log of VPN access
By default most log settings are configured so that logs don’t stick around for a long time (sometimes minutes, sometimes seconds). Storing logs for any length of time can require a fair bit of space. In order to have a record of who logged in and when, you need to make sure that your VPN logs are stored somewhere on a long term basis. This will give you an audit trail of which user connected and where they connected from. If you can also get access logs from the computer and control software that’s even better, but it may not be possible.
iii: Your contractors needs to agree to your password policy
Since they are a 3rd party organization the rules they have for employee conduct may be different from yours, and because they will be connecting remotely to your network, you can require that they agree to your password policy (and impose penalties if they violate it). This helps maintain your security and ensures that contractors use appropriately secure passwords for everything they are responsible for. And note that by “agree” I’m talking about officially including these terms and conditions in contracts. Contractors should be made legally subject to many of the same rules that regular employees need to follow.
iv: The controlling computer and hardware should use a zero access approach
A lot of times devices placed into a network are treated as though they can go anywhere in that network, and then you start reducing accessibility. However, when it come to high security equipment, you need to use the opposite approach. Start by giving new devices zero access to your network and other devices, then begin providing access as needed. If a connection to the internet is required that’s fine, but ask if that connection can it be restricted to specific ports and locations? The same goes for access from the outside; you don’t simply open the device up to everyone. What about access to or from different devices within your own network? If there’s no compelling reason for that access, then it should be prevented by the design of your network.
If a bad actor gains access to your water treatment controls they can hurt or even kill a lot of people. Unfortunately, as the attack in Florida shows, these kinds of attacks happen in real life. When you setup sensitive or important equipment, you need to make sure it’s properly secured in addition to testing that it works correctly.
At the end of the day, the official recommended guidelines for waterworks aren’t bad when it comes to the electronic security of the equipment. Most of the items they suggest should be required rather then recommended, and they need to go into more detail. However, at the end of the day the recommendations are still good. You can find the guidelines on the Open Government Alberta website in their publications section. You can also find it directly with some fairly simply Google searching. I recommend that any municipality that operates a water treatment facility take a look at the guidelines, particularly the SCADA security section on page 2-50, and maybe consider being a little more pushy when it comes to some of your contractors security practices.
To take a line from Henry IV, “Tis dangerous to take a cold, to sleep, to drink; but I tell you, my lord fool, out of this nettle, danger, we pluck this flower, safety”
If you have any questions about monitoring your computer equipment, please reach out to your TRINUS Account Manager for some stress-free IT.
By Kind Courtesy of Your Friendly Neighbourhood Cyber-Man.
