Designing a Robust Defense for Operational Technology Using Azure Defender for IoT

Many IT executives are concerned about the security of Operational Technology (OT).  This concern is valid based on my experience, but sometimes the approaches to alleviating this anxiety creates a divide between the IT and OT sides of the ‘house’.  This blog will attempt to address this divide with practical suggestions about how to get the best results from a thoughtful approach.  It will also address methods to accomplish useful but non-intrusive monitoring in the OT environment.  It will provide specific technical examples to guide you.  If this tickles your interest, read further.

Passive network monitoring is one of the most effective and least intrusive tools to gain visibility into OT networks. Installed properly it provides information on inventory, network topology, protocols in use, endpoint types, switches, and routers, etc.  Much of this information is not generally well documented and is only vaguely known by enterprise security teams. It lives below OT edge firewalls and is carefully guarded by the engineers who are responsible to make sure their factories continue to operate reliably.  As most security experts know, it is impossible to protect equipment you don’t know you have.

There is a natural push-pull between enterprise security teams who are tasked with overall business protection and operational engineers who are more focused on production.  It is common for operational engineers to express concern that a network monitoring tool will affect the reliability of the OT equipment.  While they may be honestly concerned about cyber security, they fear repercussions if reliability is affected.  If corporate policy mandates monitoring, the security team is usually instructed to install their monitoring equipment as far away from the production equipment as possible.  This usually results in an installation at or near the enterprise edge firewall.  The most common argument is that anything bad will come from the internet which is on the other side of that firewall.  This is usually NOT the best location for OT network monitoring and the assumption relating to the source of threats is not accurate either. However, based on the urgency of schedules, this location is often accepted as better than nothing.  It is important to understand that the AD4IoT sensor is completely passive.  It only listens to copies of network traffic and as such is not a threat to operational technology.

I would like to suggest a more reasoned approach, which admittedly takes more time and possibly resources, but results in a win-win for both groups if implemented well.  OT networks are often complicated by a variety of interconnected systems as shown in the next diagram.  The red sections of this diagram show the ideal locations for connections to the AD4IoT.  It is important to start implementation with a diagram of the OT system.  Diagrams of this sort are often provided as proposal documentation when Industrial Control Systems are purchased.  They may often be found on control house walls, or in the OT engineer’s office. Because these systems continue to evolve and are often upgraded in piecemeal fashion, these drawings are seldom up to date.  However, they still provide a reasonable starting point for understanding the best placement for sensors.  The point is to accurately and completely document the inventory of control equipment and the network architecture of the system.  If a sensor is only installed in the outgoing DMZ, much of this inventory information will not be available.  Information identifying the types and versions of Purdue level 0 to 2  devices will not be available.  To determine this information, the actual downloads to these devices must be seen by the AD4IoT sensor.

This is an example of an ICS diagram with the recommended locations of sensors (in red).

How can we identify if we are located at the best location?

A sample traffic taken too high in the network is analyzed below using the sensor.

In these screenshots, the sensor is too high in the network, too close to the enterprise firewall.  Note that no devices in the Process Control Level 0/1 are shown.  The monitoring in this network shows the workstations and their interactions with the database server, engineering, HMIs, and AD, some exiting traffic, but no PLC control traffic.

In the inventory, no firmware or model information is identified because the traffic to the PLCs is not being seen at this location in the network.

Another similar instance is where the majority of the traffic is broadcast or multicast.  While some industrial control systems use this method for information transfer, the indications here are that the sensor is not seeing much of the control traffic.  Only one PLC is shown in the Process Control area and most devices are sending multicast traffic.  The switches are seen, the HMIs and database servers but not much control traffic as shown in the inventory view below.

A properly configured system will look like this.  Notice the OT Protocols; Profinet DCP, Profinet Real-Time, Siemens S7 and S7 Plus.  Notice the balance between Supervisory and Process Control.  The sensor is seeing the traffic between the engineering workstation and the PLCs when they are downloaded as evidenced by the presence of firmware versions and PLC model numbers in the inventory. 

Why not just monitor the enterprise edge?

I would like to address the reason for monitoring networks in the ICS in addition to monitoring at the enterprise edge.  Many people assume that this is adequate since they see this as the source of all threats.  I will use the sample ICS network shown above to discuss some potential access points for malware or data compromise, see below.

With your security ‘blue team’ hat on, think about these scenarios identified by numbers in blue ovals on the diagram:

1. The ICS (or DCS) may be maintained by an external contractor, possibly the system integrator in the case of PLC systems, or generally the Original Equipment Maker (OEM), for DCS equipment. Sometimes these people are authorized to utilize laptops with specialized OEM software to perform upgrades, troubleshoot problems, install new hardware or do routine system maintenance.  Even if they are not permitted to utilize their laptops, they may install software, OS and firmware upgrades, and other activities utilizing programs they bring in on USB or other devices. 
2. Many large organizations have network engineers who manage all or most network devices, including but not necessarily limited to switches, routers, firewalls, and the like.  Smaller organizations may contract networking engineers.  This being a rather specialized function, these folks usually operate somewhat independently of the normal operations personnel.  Exceptions would be when the ICS or DCS supplier either utilizes unmanaged devices or provides the management function as a part of their service.  Switch management and required firmware upgrades in addition to reasonable hardening is not normally on the ‘radar screen’ for many system upgrades.  The adage, ‘if it ain’t broke, don’t fix it’ is commonly the norm.
3. Variable Frequency  Drives (VFDs) are generally maintained by the supplier.  Problem-solving, firmware upgrades, and system modifications are accomplished through contracts or purchase orders with the equipment provider.  These changes once again introduce uncontrolled laptops into the OT environment where these devices may be networked to the ICS.
4. Very expensive process analyzers and industrial robots may be leased from the manufacturers.  This equipment often comes with a required data connection to the manufacturer for usage monitoring and troubleshooting purposes.  These connections should be and often are firewalled but may allow incoming traffic for firmware updates and other related activities.
5. Most large organizations have physical security operations handled by separate internal organizations or through an externally contracted firm.  It is common to see security cameras that are used for both ICS and security functions.  Sometimes, the operator can even view the perimeter cameras or other cameras on his/her operator screens.
6. It is also common to see voice communication equipment sharing switches or infrastructure devices with OT networks.  While these are generally on different VLANs, errors can connect these devices with OT equipment. 
7. Additionally, there may be data links to Uninterruptible Power Systems (UPSs), again usually maintained by the OEM.
8. Plant historian packages often have links to share plant data, inventory, and other information with the enterprise.
9. Sometimes contracts are established for the maintenance of corporate printers. Since most of these devices have unpatched apache web servers, maintenance could introduce issues carried over from enterprise equipment.
10. Operators have even been known to utilize USB ports on HMI devices to charge their phones thereby unknowingly placing the HMIs on a cellular network.
11. Cleaning contracts, maintenance of support systems such as HVAC and fire protection generally allow access to controlled areas where physical access to ICS equipment could be leveraged by unscrupulous parties.

And the list can go on… with every industrial facility having different variations on this theme.  As any security-minded individual can readily see, the opportunities for compromise, malware infection, and data exfiltration in any large industrial campus are numerous. 

Conclusions

Coordination with operational engineers is the starting point to a win-win engagement.  The benefits are apparent to both enterprise and operations personnel.  With correct sensor placement, a complete inventory with full device information, firmware versions and model numbers can be derived. This is a benefit to both parties.  Additionally, the actual network flows can be confirmed, unexpected paths can be identified and potential vulnerabilities can be found and corrected.   

Monitoring Industrial Control Systems at the enterprise edge, while important, is by no means adequate.  Malware introduced, even if prevented from beaconing home by enterprise edge firewall rules, can still damage operational equipment and affect production or operational safety.  Data can be modified, control system programs could be changed to perform dangerous actions, company secrets could be stolen, and system backups corrupted.