What's in your server?

ControlGlobal.com
Security threats to control system networks are a fact of life. Senior Technical Editor Rich
Merritt shares tips and techniques he culled from end users and vendors on how you can
keep the Barbarians at bay.
By Rich Merritt, Senior Technical Editor

E verybody is trying to scare the beejeezus out of you these days with talk of nasty

olhackers and crackers, how assorted bad guys are trying to break into your process
control system, and all the risks you take by using Web, Ethernet, wireless, and Microsoft
technology in your control system. While we dont exactly subscribe to the theory that
terrorists are plotting to destroy your recipe for making chocolate, we realize that threats
do exist from hackers, viruses, and competitors. Maybe even terrorists, too.
To help you bolster your defenses, we've assembled a timely list of tips and techniques
you can use to build a fortress to secure your data and protect your control system from
intruders. (See Top ten defenses sidebar below.)

Some of these tips and techniques dont cost a thing. Some are just common sense, some
require a change in the way you do things, and some require the purchase of a little
hardware.
Justin Lowe, a security analyst at PA Consulting, explains it best: There is no silver
bullet,he says. A suite of security measures are required but
only around 30% of the solution is technical. The remainder is
procedure, process and management.

The Nature of the Threats

Weve heard the classic stories about security problems ad nauseum: the wastewater
plant in Australia, the nuclear plant in Ohio, and the SCADA system at an electric power
plant in California (which turned out to be a hoax). Ernie Rakaczky, Director of Process
Control Network Security at Invensys Process Systems, says theyve seen a few in
process control, too:
Internet worms such as CodeRed Nimda and SQL Slammer have attacked web
servers
Outsiders have tapped into wireless communications paths
An intruder connected via a modem
A maintenance worker accidentally inserted a virus via an infected floppy or CD
Unauthorized personnel gained access to an unprotected PC in an unlocked lab
A remote user inadvertently introduced a virus into the network
An intruder entered through a Remote Access Services (RAS) link

Except for a few incidents like these, the process control industry has remained relatively
immune from the huge number of problems that plague commercial web sites, banks,
and government institutions. Maybe the bad guys havent discovered us yet, or maybe we
dont have anything they want. Or maybe companies in our industry just dont talk about
it when they take a hit.
Bad guys are definitely out there. One of our contributors, a control engineer at a large
Midwest refinery, is worried. We have been written by name on terrorist lists, so our
physical security is very tight,he says. He asked to remain anonymous as did other
contributors. Nevertheless, there does not appear to be a major, organized attack on
process control systems yet.
It certainly appears that the two biggest problems are (1) external random attacks by
worms, viruses and similar software that roam cyberspace looking for vulnerabilities, and
(2) internal problems caused by disgruntled employees, careless operators, and bad
procedures.
In the first case, nobody outside is trying to destroy the chocolate recipe they dont
even know you make chocolate. If they get you, you are probably just the victim of a
random Internet crime. In the second case, you do it to yourself because of poor security
or poor training. Both situations are preventable.
The tips and techniques that follow will help you create a fortress and tighten up security,
but nothing will stop someone who is determined to take your plant down. No firewall is
safe from a talented hacker, no anti-virus software gets them all, and dealing with
disgruntled employees and actual terrorists is beyond the scope of this article. We can,
however, help you make it more difficult for them. So lets build a fortress.
Get Off the Networks!
End users and vendors alike universally advise disconnecting your process control system
from the Internet, corporate networks, business LANs, or any network not needed for
actual control. One engineer at a chemical plant said it bluntly, We do not allow any
outside connections into our control system. There are no modems and certainly no
Ethernet connections to the Web or business system.Its the fortress mentality, but it
works.
Carl King, senior engineer at Cinergy Services in Owensville, Ind, agrees. If you do not
have a strategic reason to connect your control system to your corporate network, don't
connect it,says King. That provides the best security for attacks from external
sources. If possible, provide a separate data collection system for information that needs
to be available to the corporate network from the control system. No data traffic should
be allowed to directly access the control network from the corporate network or the
internet.
Some people call the separate data collection system a replicated systemor a shadow
server.Essentially, data that is needed by external systems such as maintenance, ERP
software or corporate IT is sent to a computer outside the control area, where a
duplicate image of the real-time data base or process historian is maintained. The
external systems can take whatever they want from the shadow serverwithout affecting
the process control system. If the shadow server is attacked by a virus or worm, this
does not affect the control system.
Communications between the control system and the shadow server go through a firewall

and protective software, which we will cover below.

The Shadow Server allows process data to come from the control LAN PI Server and be
moved to the business LAN side, with replication,says our anonymous refinery engineer.
"All the functionality of collecting process data, creating spreadsheets and other uses we
make of the data are available to people on the business LAN without having any risk of
them being on the control LAN pulling data. Our control system is physically isolated from
our business network and the outside world.
Weve banned e-mail and web browsing from the control system,says an anonymous
Digital Security Advisor at a major oil company. If a plant operator needs to access these
facilities then we provide them with a corporate desktop alongside the control system.
Vendors agree. It is my recommendation that the local control networks should be 100%
separated from the Internet,says Scott Saunders, Director, Strategic Marketing at Moore
Industries. This will stop hackers from outside coming in.
Metso agrees, too. A DCS should not be connected directly or indirectly to the
Internet,says Roger Leimbach, marketing manager at Metso Automation.
Siemens agrees with the shadow server concept. Siemens recommends that customers
keep their automation networks isolated from their corporate IT networks if at all
possible,says Todd Stauffer, Manager Product Marketing, Siemens Energy & Automation.
It is also recommended that a dedicated computer be selected within the system to
perform all software imports, as well as introducing externally-engineered data.

Working Inside a Fortress

Although complete isolation sets up a fortress, the logic is sound. Most of the threats
arrive electronically, via networks, wireless or the Internet. Eric Byres, a member of the
research faculty for Critical Infrastructure Security at the British Columbia Institute of
Technology, says 70% of the security threats to process control and SCADA systems are
external. Deloitte & Touche says that 90% of security breaches in financial companies are
external. Therefore, our users and vendors say to disconnect yourself from all networks,
and eliminate the major source of problems. Besides, why would you need access to the
Web to control a process?
The most secure systems we have are those that are not networked to anything
else,says Mike Larocca, control engineer at Solutia in St. Louis, Mo. These tend to be
older systems.Older legacy control systems are fairly immune
to external attack, mainly because their networks are
proprietary.
Older systems usually were not connected to Internet protocol networks,says King.
They were usually proprietary in their communications protocols and as such were
difficult to connect to other systems. This, however, did not make them more secure in
nature. There were just fewer opportunities for systems to be attacked.In other words,
you really have to know what you are doing to get into a legacy system from the outside.
You might want to reconsider replacing that old legacy system, especially if it is still
working well. Unfortunately, hardware support issues usually dictate replacement sooner
rather than later,notes King.

Disconnecting yourself from networks does pose one problem: What about all the modem
connections on your equipment that allow vendors to perform remote maintenance,
software upgrades and diagnostics? Protection can only be enforced if the end user
institutes a Do not connectpolicy and periodically verifies that rogue modems or high
speed internet and non-DCS LAN connections do not exist,says Leimbach.
Just about every piece of equipment in your plant purchased in the past few years has a
modem connection. All of these seemingly innocent roguephone connections are, in
fact, a back doorinto your system. Byres reports that the Slammer worm infiltrated at
least four different control systems last year, and one of them got into a paper machines
HMI via a dial-up modem.
Remember, virus writers are not targeting you specifically. They are just looking for any
vulnerable port, and they scour cyberspace relentlessly and automatically. Someone like
me, who sits on a cable modem eight hours a day, gets hundreds of port probes a day.
Any system connected to any network will experience the same. You must devise a
system that denies access to your system via back-door modems, and permits outside
vendors to call up their equipment only under carefully controlled, supervised conditions.
This isnt going to be easy, warns Joe Weiss, a security consultant at KEMA. In his
testimony before Congress, Joe Weiss said you may not know about all the phone lines in
your plant. He says an audit at an electrical utility turned up 100-200 phone lines in
power plants and substations that were not owned by the utility. These phone lines were
owned, installed and paid for by control and diagnostic system vendors,he explained.
Since the phone lines belonged to the vendors, (they) were not identified. This is a
common occurrence on many control system implementations.
Then there are all the new wireless systems with handheld PDAs that let control
engineers and techs wander around the plant. What to do with wireless? Hook it into the
shadow server, of course, where a security breach cant hurt anything.
Finally, there are all the wonders of the Web, such as remote tuning and loop analysis
software, maintenance management packages, batch management software,
manufacturing execution systems, and so on, all of which need access to real time
information. What to do with them? Hook them into the shadow server, too.
Lock Up the Hardware
A few years ago, an engineer explained to me how control systems in some Third World
countries are installed. All the hardware is installed in 19-in. racks behind padlocked bars,
he said. The racks are in locked cabinets, and the cabinets are in a high-security room
with a locked steel door inside a secure building. The only access operators have to the
system is via HMI terminals (not PCs) in the control room, which is located elsewhere in
the building.
The purpose is to keep unauthorized personnel from tinkering with control settings, but
the technique improves security, too. Thats because even a system that is disconnected
from networks is still vulnerable to software brought in by operators and technicians.
The concern I am wrestling with is operators bringing in homemade CDs to listen to
music on control system PCs,says one control engineer. What else are they bringing
with them? I have found card games and the like on some computers after a slow
weekend.

Weiss agrees. At least one facility with no external connections suffered a forced outage
when a controls technician brought in an infected disk with games that shut down the
plant,he adds.
Larocca says they limit access to prevent such incidents. We keep computers and
controllers in locked or otherwise protected rooms where access to CD and DVD drives,
floppy drives, USB ports, and so on is limited,he explains. We also keep network hubs,
switches, routers, etc. in limited access areas.
Siemens' Stauffer said that "automation system owners should also implement a
standard operating procedure for ensuring that only authorized individuals have access to
the automation system data. This policy should include user administration procedures
which are based on Windows security (such as password expiration and lockout after
number of retries) and controlling access to project data stored on the hard drive. To
further prevent unauthorized access to the automation system, the key assets such as
controllers, PCs, servers, and engineering workstations should be physically isolated and
protected in a locked room. Additionally each controller has a physical switch can be
enabled to prevent downloading of unwanted configuration changes."

Dont Upgrade or Install Patches Automatically

Microsoft says it is vital to install all the security patches they develop to keep ahead of
the bad guys. If you are running Microsoft's most recent operating system, Windows XP
SP2 or Windows Server 2003, which contain significant security enhancements, you've
already taken a huge step toward reducing your security vulnerability,says Ron
Sielinski, Senior Industry Technical Strategist, Manufacturing Industry Unit, Microsoft.
"Keeping your operating systems patched via Windows Update is equally important.
Stay current,says Mike Pursiful, security analyst at Cryptek. By far, the greatest
number of non-trivial intrusions, interruptions and systems disasters happen in
environments where components are forgotten, out-of-date, and unpatched.
Microsoft told us the latest versions of Windows are designed for better security. Bill
Gates himself made security a number one priority. That may certainly be true. However,
some engineers advise against making patches. Weiss says that traditional methods
employed by IT departments, such as installing patches, dont
always work in a process control system.
IT security policies and technologies used to secure traditional IT systems can potentially
impact control systems if applied without understanding or inappropriately applying them
to the control system environment,says Weiss. Automatically implementing security
patches on control system workstations can, and have, shut down control systems.
If you have isolated your control system into a fortress by disconnecting all the networks,
you may not need to install patches on the control system PCs, but you probably should
put them on the Shadow Server. A key benefit of isolation with Microsoft based systems
is that you then don't have to install every little security patch that Microsoft
releases,notes Larocca.
You can be a little judicious about what patches you apply. Microsoft's security patches
can be a real headache, particularly if they have not been tested with your control
software.

This is true of any operating system used in a control system,notes King. However,
ignoring security patches may be at your systems peril.
The older the operating system, the less you may have to worry about patches. Users
have talked to us about the operating system paradox that showed itself when the Sasser
Virus was launched in 2004,says Todd Stauffer, Manager Product Marketing, Siemens
Energy & Automation. This virus attacked only the newer Microsoft operating systems,
such as Windows 2000 and Windows XP, but left Windows NT alone. This means that
users of older, seemingly less secure, operating systems were actually less vulnerable to
threats since hackers do not typically target older operating systems.
Have a well-defined policy for immediate testing of new Microsoft Security patches and
Virus scanner profiles and for notification of testing results,advises Stauffer. Have a
well-defined policy regarding whether new Microsoft Security patches can be installed as
soon as they are available, or whether users must wait for compatibility test results by
the host vendor.
Install Firewalls Everywhere
Entire technical articles have been written about firewalls, and many process control
vendors seem to base most of their security advice on firewalls. A hardware firewall sits
between two networked devices such as between the shadow server and the control
system, or between the IT network and the control network and monitors network
traffic.
If you have a strategic need to connect networks, make sure the corporate network is
protected by a firewall between it and the internet,advises Cinergys King. The control
system network should then be protected by a firewall between it and the corporate
network.
Esssentially, a firewall examines the data and decides if it meets your criteria. Eric Byres
defines three kinds of firewalls:
Packet filtering: Compares header information, including IP addresses and TCP
port numbers, in each packet against a set of criteria before forwarding the
packet.
Stateful inspection: Filters packets at the network layer, determine if session
packets are legitimate, and evaluate contents at the application layer. Also called
Dynamic Packet Filtering.
Application proxy: Examines packets at the application layer and filters traffic
based on specific application rules. (For a more complete description, see Eric
Byres White Paper.)
In a nutshell, you can configure a packet to pass only specific kinds of information from
specific sources. For the shadow server, for example, you probably want to establish
criteria that carefully defines the kind of information that the shadow server can send to
the process control system, limiting it to process control-related data in a narrowly
defined secret format. This may require software on both sides to format the data
properly.
It is highly unlikely that a hacker, even if he penetrates the shadow servers security,
would know the secret format for talking to the control system. If he does know the
format, then you are dealing with a truly determined, knowledgeable opponent, and that

is beyond the scope of this article.

A firewall will stop most inappropriate traffic, if you set it up properly. But thats the key:
you must set it up. Just putting a firewall between the process control network and the
rest of the network, without configuring it to know what data is essential and what is not,
could waste time and money, without adding protection,warns InvensysRakaczky. Even
decisions of where to implement firewalls must be policy-driven. Our cybersecurity
consultants, for example, typically break out the following security zones: the public
Internet, the data center, the plant network, the control network and the field I/O zone,
and then deploy different brand firewalls between each. In addition to working out these
configurations, some installations also require protection for sub zones.(See the
Invensys White Paper for examples of sub zones).
We segregated the control system from the corporate network by installation of a fully
managed and monitored firewall,says our anonymous digital security supervisor. Fully
managed and monitored means that a security company examines the log files from the
firewall every five minutes continuously and alerts us to any suspicious activity.
Be careful, though. Many DCS vendors base much of their security on firewalls. Or, as one
control engineer put it, Firewalls are the best you can hope for from most vendors.
As Rakaczky points out, you may want to put different firewalls in different locations in
the system. For example, you could install shadow servers in various locations, including
remote sites and individual process units, that would allow for remote multiplexing,
access by vendors to equipment, and wireless access.
Keep in mind that every access port that is allowed through the firewall is another
potential security hazard, just waiting to be exploited by someone,cautions King.

Outside the Fortress

If you subscribe to the Fortress Mentality recommended here, and plan to set up
a shadow server, then all the conventional protective devices firewalls, antivirus software, passwords, and so on should be applied with vigor to the
server. The operating system should be one of the vendors latest, and it should
be kept updated with all necessary fixes, patches and service packs.
Listed below are White Papers at ControlGlobal.com that go into great detail on
anti-virus software, password protection, operational procedures, and other IT
techniques needed to protect computers that are connected to external
networks. Some of the White Papers address specific control systems so, if you
have one of those, you may want to read what your vendor has to say very
carefully.
You may also want to ask your IT department for help
in protecting the server. But be careful. As Joe Weiss warns, IT departments
working without Control System Staff can do more harm than good: Performing
system-wide diagnostics, maintenance, and/or scans can and have shutdown
control systems,he says. Performing penetration testing of control systems
can, and has, shut down control systems. Whats worse, Weiss says, even
after control system people complain about the tests, IT wants to continue
performing them. IT can help if you work with them on process control
systems.

William Collins, control engineer at Constellation, says that IT can help. Our IT
security ran a routine vulnerability scan and found a hole in our Unix servers on
the control system and knocked them down, says Collins. He says they told IT
to stay out of their control systems, but he still goes to them for help. Together,
they developed a private process network with limited access to the corporate
intranet. Even so, Collins offers this advice: If at all possible, stay disconnected
from your corporate LAN.
Collins does not feel particularly secure. On a scale of 1-10, Id say I am a
five, he laments. I am running Unix, which seems to be less of a target. So I
feel protected from the outside, but am open for inside attacks.
We have experienced very few security problems, and we have no evidence of
specific directed attacks,The security advisor reports, The activity we observe
most is from non-directed Internet worms.
Appoint a Security Czar
Except for setting up a fortress that limits access, there is no hardware or software that
prevents attacks by a disgruntled employee, a bad guy that got inside your company, or
even a well-meaning operator or technician that makes a mistake in configuring a
controller, maintaining a device, or loading up tainted software.
Such problems are dealt with by setting up operational procedures, passwords, levels of
access, approval processes, and security rules and regulations.
This requires a corporate commitment. Make the integrity of your systems a business
responsibility and a priority, says Pursifull. Unless someone is explicitly responsible for
this -- and empowered to act or establish procedure -- it will not get done, except
perhaps sporadically.
In other words, appoint a security czar and give that person responsibility for control
system security, enough authority to command respect, and a budget to carry out the
task.
Much is happening on the security front these days, and it takes a full time person to
track down all the standards, recommended procedures, and approved hardware to keep
your system safe. A few of the buzzwords, organizations, standards and coalitions flying
around these days include Common Criteria EAL 4, FIPS 140-2, ISA 99, CIDX
Cybersecurity Initiative, MS-MUG, TPEP B2, DTTS CAP, and more.
Maintaining security on a process control system is a full time job, requires constant
vigilance and training, and deserves a corporate commitment. The tips and techniques
presented here are just the tip of the iceberg.
=
= closed
=