Network Security within a Converged Plantwide Ethernet Architecture
The prevailing trend in Industrial Automation and Control System (IACS) networking is the convergence of technology, specifically IACS operational technology (OT) with information technology (IT). Converged Plantwide Ethernet (CPwE) helps to enable IACS network and security technology and OT-IT persona convergence through the use of standard Ethernet, Internet Protocol (IP), network services, security services, and EtherNet/IP. A reliable and secure converged plant-wide IACS architecture helps to enable the Industrial Internet of Things (IIoT).
As access methods to plant-wide IACS networks expand, the complexity of managing network access security and controlling unknown risks continues to increase. With a growing demand for in-plant access by trusted industry partners (for example, system integrator, OEM, or IACS vendor), IACS applications within the CPwE architecture (Figure 1) face continuous threats such as malware propagation, data exfiltration, network scanning, and so on. Furthermore, industrial operations face additional challenges such as legacy systems, lack of visibility on what type of IACS assets and devices are on the IACS network, and lack of security skills for the OT team.
No single product, technology, or methodology can fully secure plant-wide architectures. Protecting IACS assets requires a holistic defense-in-depth security approach that addresses internal and external security threats. This approach uses multiple layers of defense (administrative, technical, and physical) utilizing diverse technologies for threat detection and prevention, implemented by different personas, and applied at separate levels of the IACS architecture.
Defense-in-depth applies policies and procedures that address many different types of threats. The CPwE Industrial Security Framework (Figure 2), using a defense-in-depth approach, is aligned to industrial security standards such as IEC-62443 (formerly ISA99), Industrial Automation and Control Systems (IACS) Security, and NIST 800-82 Industrial Control System (ICS) Security.
With all the opportunities and challenges faced by industrial operations, there is a strong need in manufacturing and heavy industry markets for the following requirements:
- Visibility —Visibility of the current network devices and IACS assets present in the IACS network is very critical for the OT-IT security team to design and deploy a comprehensive industrial security access policy. Existing IT network monitoring tools are unable to gain full visibility of IACS network devices and IACS assets in a plant-wide network because the IACS assets communicate with IACS protocols. There is a need for a network monitoring tool (NMT) that can gain full visibility of IACS assets present in a plant-wide IACS network and pass this information to a security access policy design and implementation solution.
Note Cisco and Rockwell Automation recommend that the OT-IT security team be composed of a multi-discipline team of operations, engineering, safety, maintenance, and IT representatives to develop an industrial security access policy based on your risk tolerance and risk management.
- Segmentation —Segmentation (zoning) is an important piece of network architecture required by the OT-IT network design team for improving security and performance by grouping and separating network assets. Cyber criminals study ways to infiltrate the IACS network by looking at the most vulnerable point. Segmentation helps to prevent the spread of the infection and limits it only to those endpoints that an infected host can reach. A common segmentation method adopted by industrial operations is to segment the IACS network Industrial Zone (Figure 1) from the Enterprise Zone via an industrial DMZ (IDMZ), then use logical segmentation within that zone (following the IEC 62443-3-2 Zones and Conduits model). OT-IT then collaborates to design the access policy in the Industrial Zone by using access control lists (ACLs). However, the management of ACLs can be tedious and their larger size can affect the performance of network devices. Industrial operations are looking for a better solution to segment access control policies for the IACS network Industrial Zone that is easier to deploy and manage.
- Anomaly detection and Mitigation —When little to no access control methods to a plant-wide architecture are enabled, the possibility of IACS assets getting infected increases. When such an event happens, the OT-IT security teams need to identify the infected device, then based on the OT-IT industrial security access policy, decide how to address the threat based on the level of risk. Industrial operations need a method to detect anomalies, have the option to block threats, and identify compromised IACS assets. This detection and remediation method deployed in the plant-wide IACS network by the OT-IT team must be scalable and also should not change the currently deployed architecture.
- Intent-based security for OT —In many industrial operations, IT helps to defines industrial security policies, architecture, and design. OT depends on IT to enable and manage those policies. However, given that OT requirements are often fluid, the OT-IT security team needs a process that allows OT to express operational intent that results in dynamic industrial security access policy changes without having to depend on IT. For example, consider the network security use case associated with remote access. The IT team can create the general centralized access policy for remote access that has rules to allow a remote trusted industry partner expert to connect to an IACS asset. When the remote access is no longer needed, the OT team informs IT to revoke the access for the remote expert. Since this process is manual, in some cases there might be delays in providing or revoking the remote access. To overcome these challenges, an automated self-service process is needed where an OT engineer can request the remote access without IT intervention.
CPwE is the underlying architecture that provides standard network and security services for control and information disciplines, devices, and equipment found in modern IACS applications. The CPwE architectures (Figure 1) provide design and implementation guidance, test results, and documented configuration settings that can help to achieve the real-time communication, reliability, scalability, security, and resiliency requirements of modern IACS applications.
CPwE Network Security describes several network security use cases that are solved using diverse security solutions and technologies. CPwE Network Security is brought to market through a strategic alliance between Cisco and Rockwell Automation.
Figure 1 CPwE Architecture
There are many personae managing the plant-wide security architecture, with diverse technologies, as shown in Figure 2.
- Control System Engineers (highlighted in tan)—IACS asset hardening (for example, physical and electronic), infrastructure device hardening (for example, port security), network monitoring and change management, network segmentation (trust zoning), industrial firewalls (with inspection) at the IACS application edge, and IACS application authentication, authorization, and accounting (AAA).
- Control System Engineers in collaboration with IT Network (highlighted in blue)—Computer hardening (OS patching, application white listing), network device hardening (for example, access control, resiliency), network monitoring and inspection, and wired and wireless LAN access policies.
- IT Security Architects in collaboration with Control Systems Engineers (highlighted in purple)—Identity and Mobility Services (wired and wireless), network monitoring with anomaly detection, Active Directory (AD), Remote Access Servers, plant firewalls, and Industrial Demilitarized Zone (IDMZ) design best practices.
Figure 2 CPwE Industrial Security Framework
Note This release of the CPwE architecture focuses on EtherNet/IP which uses the ODVA, Inc. Common Industrial Protocol (CIP™), and is ready for the Industrial Internet of Things (IIoT). For more information on EtherNet/IP, and CIP Security™, see odva.org at the following URL:
CPwE Network Security Solution Use Cases
There are four network security solution use cases that are addressed by CPwE Network Security:
- Visibility and Identification of network devices and IACS assets in Cell/Area Zone(s).
- Security Group Policy segmentation of IACS assets in Industrial Zone (Level 3 Site Operations and Cell/Area Zone(s)).
- Network flow and threat (e.g., malware) detection of network devices and IACS assets in the Industrial Zone.
- OT managed remote user (employee, partner) access (enterprise, internet) for network devices and IACS assets in the Industrial Zone.
These network security solution use cases apply to both brown field (legacy) and green field (new) deployments and follow the best practice framework of CPwE.
IACS asset and network device visibility is a continuous process of discovering and identifying all the different IACS assets in the plant-wide network. From the industrial security perspective, it is imperative to have visibility of the IACS assets and network devices due to the following reasons:
- Gaining the visibility of all the IACS assets would allow an OT-IT security administrative team to logically group these IACS assets based on the function of the asset. Once all the assets are grouped into different sets, it is easier to create a security group access level policy, which is more efficient than an individual policy.
- Helps to detect malicious activity. Knowing the infected device type helps identify if there is a known vulnerability to remediate similar endpoints in the network.
To gain visibility of assets in the enterprise networks, IT has used Cisco Identity Service Engine (ISE) with Cisco ISE Profiling Services (explained below). Cisco ISE is a security administration product that enables an OT-IT security administrative team to create and enforce access level security policies. One of the salient features of Cisco ISE provides profiling services, detecting and classifying endpoints connected to the network. Using MAC addresses as the unique identifier, ISE collects various attributes for each network endpoint to build an internal endpoint database. The classification process matches the collected attributes to pre-built or user-defined conditions, which are then correlated to an extensive library of profiles. These profiles include a wide range of device types, including mobile clients (iPads, Android tablets, Blackberry phones, and so on), desktop operating systems (for example, Windows 7, Mac OS X, Linux, and others), and numerous non-user systems such as printers, phones, cameras, and game consoles.
However, for IACS assets, the ISE built-in probes will not be able to get all the information from the IACS asset to create a granular profiling policy. This is due to the fact that the IACS assets may not support some traditional IT protocols that ISE relies on to profile the device. To gain visibility of IACS assets CPwE Network Security uses the Industrial Network Director from Cisco and the FactoryTalk Network Manager from Rockwell Automation, network monitoring tool (NMT). The NMT product was built to help the OT team gain full visibility of IACS network devices and IACS assets in the context of industrial operations and provides improved system availability and performance, leading to increased overall effectiveness. NMT uses industrial protocols such as the ODVA, Inc. Common Industrial Protocol (CIP) and PROFINET to enable a dynamic, integrated view of the connected IACS assets and network infrastructure. NMT is a lightweight and highly scalable network monitoring tool, which was built mainly for OT industrial operations.
NMT interfaces with Cisco ISE using Cisco pxGrid, which is an open, scalable, and IETF standards-driven data sharing and threat control platform to communicate device information through attributes to ISE. This integration allows exporting of the endpoints discovered by NMT to ISE. NMT also exports several attributes to ISE that would be used to create profiling policies for IACS assets, which is shown in Figure 3.
Figure 3 NMT Exporting Attributes to ISE
The integration between NMT and ISE provides the following benefits:
- Automatically enrolls IACS assets into the ISE endpoint database.
- Enables an OT-IT security administrative team to create granular profiling policies based on the attributes received from NMT.
- Allows the OT engineers to leverage the integration between NMT and ISE to automatically deploy new security policies in the network.
Segmentation is a practice of zoning the IACS network to create smaller domains of trust to help protect the IACS network from the known and unknown risks in the network. As shown in Figure 1, CPwE segments the IACS plant-wide architecture into different zones: Cell/Area Zone, Industrial Zone, IDMZ, and Enterprise Zone. OT-IT teams control the communication between the Enterprise and Industrial Zones through the IDMZ. This zoning creates strong boundaries and helps to reduce the risk of unauthorized communications.
The segmentation between Cell/Area Zones was typically done using VLANs with ACLs at the Layer 3 distribution switch. A group of IACS assets that are part of the same functional area (zone) and need to communicate with each other were put in the same VLAN. When IACS assets need to communicate with IACS assets located in a different functional zone, communication occurs via the distribution switch which uses ACLs to either permit or deny traffic. There are many benefits associated with segmentation, such as creating functional areas (building block approach for scalability), creating smaller connected LANs for smaller broadcast/fault domains and smaller domains of trust (security groups), and helping to contain any security incidents. For example, if there is a security group access policy to restrict the communication between the VLANs (zones), traffic from an infected host is contained within the VLAN. However, as the size of the ACL increases, the complexity of managing the ACL also increases.
To provide more flexibility and simplicity to network segmentation, CPwE Network Security uses Cisco TrustSec technology to define access policies using security groups. This allows the segmentation of IACS assets using Security Group Tags (SGT) which group the assets regardless of their location in the plant-wide network. This technology is available on the Allen-Bradley® Stratix 5400/5410 and the Cisco IE 4000/5000 industrial Ethernet switch (IES). As shown in Figure 4, the IACS assets in Cell/Area Zone 10 are given an SGT of 10, the IACS assets in Cell/Area Zone 20 are given a tag of 20, and the FactoryTalk application(s) located within Level 3 Site Operations is given an SGT of 100.
Figure 4 Secure Group Assignment
Once the IACS assets are put in logical groups by the OT-IT security administrative team, the next step is to enforce the Secure Group Access Control List (SGACL) on the distribution switch. Enforcement of security access policy is achieved by defining a policy matrix in ISE; an example of such a policy is shown in Figure 5.
Figure 5 An Example of Secure Group Access Control List
As shown in Figure 5, all IACS assets in Cell/Area Zone 10 (SGT 10) are allowed to talk to each other, and all IACS assets in Cell/Area Zone 20 (SGT 20) are allowed to talk to each other. However, IACS assets in Cell/Area Zone 10 are not allowed to talk to IACS assets in Cell/Area Zone 20. The key point to observe is that FactoryTalk application(s) (SGT 100) is allowed to talk to all IACS assets in Cell/Area Zone 10 and Cell/Area Zone 20. This is required because the FactoryTalk application(s) may need to have access to all the IACS assets for managing industrial operations.
After the IACS assets are tagged, and the security access policy matrix is defined in ISE, the last step is to enforce the access policy in the Cell/Area Zone. As IACS assets attach to the network, they are authenticated to ISE using MAC Authentication Bypass (MAB), which is a port-based access control method using the MAC address of the IACS asset. An SGT assignment is also done. For example, as shown in Figure 4, when PAC_10 attaches to the IES in the Cell/Area Zone 10, it is assigned an SGT of 10. The distribution switch connecting the Cell/Area Zones needs to download the SGACL that is shown in Figure 5. Figure 6 shows the ordered sequence:
1. All of the IES are configured with MAB Open Access.
2. The OT user discovers IACS assets with NMT and tags them with custom attributes.
3. NMT sends the asset details to ISE via pxGrid.
4. The IT user pre-defines profiling rules in ISE to match custom attributes and assigns the SGT in Authorization policies. All the IACS assets attached to Cell/Area Zone 10 are assigned a SGT of 10, all the IACS assets attached to Cell/Area Zone 20 are assigned a SGT of 20, and the FactoryTalk application(s) is assigned a SGT of 100.
5. ISE distributes the TrustSec policy to the distribution switch to enforce Zone segmentation.
Figure 6 Policy Enforcement of All the Traffic Going East-West and North-South between the Zonesr
Flow-based Anomaly Detection Using Stealthwatch Technology
Network flows are the communications between network devices. Having visibility to those devices allows the OT-IT security administrative team to have a baseline idea of typical traffic patterns within the plant-wide architecture. Complete visibility information has the following benefits:
- Is my security access policy working correctly?
- Are there any unauthorized network connections occurring in the network?
- Are there any abnormal connections established to the outside world?
- Is there any active malware spreading in the network?
- Is this occurring for the first time or it has been occurring for a while?
Cisco Stealthwatch helps industrial operations to address all the questions that are important for doing any incident or regular operation analysis. CPwE Network Security integrates Stealthwatch technology and enables the OT-IT security administrative team to monitor real-time traffic and also detect if there is any network anomaly or if malware is propagating in the network. Cisco Stealthwatch collects the data on the switches themselves using NetFlow technology, which is more scalable than the traditional SPAN (switched port analyzer) method.
The SPAN method involves dedicating a source port for collecting the traffic and a destination port for analyzing the traffic. If the traffic analyzer is not directly attached to the source IES, then there are two alternatives:
- Add a cable directly from the source IES to the destination switch.
- Configure remote SPAN (RSPAN) on the source IES, implement a dedicated RSPAN VLAN, then configure RSPAN on the destination switch.
Configuring remote SPAN allows the source traffic to be carried across multiple switches, but it increases the complexity of deployment. Second, if the captured traffic exceeds the interface bandwidth, then the traffic may be dropped. Third, if RSPAN is enabled on multiple IES, then the captured traffic coming from all the IES may impact the performance of the distribution/aggregation switch. Fourth, the traffic analyzer needs to be managed to see if it can handle the load coming from all the IES.
Furthermore, with the NMT and Stealthwatch integration (Figure 7), the OT-IT security administrative team may get contextual flow information. For example, if a PAC were communicating with a PAC, then Cisco Stealthwatch will provide visibility of the flow as well as IACS asset information.
Figure 7 Detecting Network Anomalies Using Cisco Stealthwatch
OT Influenced Remote Access—For Example Downtime
Securely Traversing IACS Data Across the IDMZ Design and Implementation Guide (CPwE IDMZ DIG) outlines the current best practices for deploying remote access in an IACS network environment. As described in the CPwE IDMZ DIG, the remote access user must be able to access the remote desktop server in the IDMZ zone and then use the remote desktop server to access the IACS assets in the Industrial Zone. The CPwE Network Security solution enhances this process by enabling OT staff to express intent using NMT and ISE (Figure 8), thereby automating the process of granting remote access as well as removing it.
CPwE Network Security design uses NMT, ISE, and TrustSec technology to meet the remote access requirement. The OT team can create groups in NMT for remote access. When remote access is required, the IACS assets are moved into those security groups and access is granted. When remote access is no longer required, the IACS assets are moved back to their normal security groups. NMT communicates these changes to ISE automatically, which configures network devices like the ASA firewall within the IDMZ.
Figure 8 OT Influenced Remote Accessing NMT Solution
CPwE is a collection of tested and validated architectures that are developed by subject matter authorities at Cisco and Rockwell Automation. The testing and validation follow the Cisco Validated Design (CVD) and Cisco Reference Design (CRD) methodologies.
The content of CPwE, which is relevant to both OT and IT disciplines, consists of documented architectures, best practices, guidance and configuration settings to help industrial operations and OEMs with design and deployment of a scalable, reliable, secure and future-ready plant-wide industrial network infrastructure. CPwE also helps industrial operations and OEMs achieve cost reduction benefits using proven designs that can facilitate quicker deployment while helping to minimize risk in deploying new technology.
Industrial IoT (IIoT) offers the promise of business benefits using innovative technologies such as mobility, collaboration, analytics, and cloud-based services. The challenge for industrial operations and OEMs is to develop a balanced security stance to take advantage of IIoT innovation while maintaining the integrity of industrial security best practices. Deploying Network Security within a Converged Plantwide Ethernet Architecture Design and Implementation Guide (DIG) outlines several network security use cases for plant-wide Industrial Automation and Control System (IACS) network infrastructure to help enable IIoT innovation within the CPwE framework. CPwE Network Security was architected, tested and validated by Cisco Systems and Rockwell Automation.
More information on CPwE Design and Implementation Guides can be found at the following URLs:
- Rockwell Automation site:
- Cisco is the worldwide leader in networking that transforms how people connect, communicate and collaborate. Information about Cisco can be found at www.cisco.com. For ongoing news, please go to http://newsroom.cisco.com. Cisco equipment in Europe is supplied by Cisco Systems International BV, a wholly owned subsidiary of Cisco Systems, Inc.
- Americas Headquarters
- Cisco Systems, Inc.
- San Jose, CA
- Asia Pacific Headquarters
- Cisco Systems (USA) Pte. Ltd.
- Europe Headquarters
- Cisco Systems International BV
- Amsterdam, The Netherlands
- Cisco has more than 200 offices worldwide. Addresses, phone numbers, and fax numbers are listed on the Cisco Website at www.cisco.com/go/offices.
Cisco and the Cisco logo are trademarks or registered trademarks of Cisco and/or its affiliates in the U.S. and other countries. To view a list of Cisco trademarks, go to this URL: www.cisco.com/go/trademarks. Third-party trademarks mentioned are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (1721R)
- Rockwell Automation is a leading provider of power, control and information solutions that enable customers to be more productive and the world more sustainable. In support of smart manufacturing concepts, Rockwell Automation helps customers maximize value and prepare for their future by building a Connected Enterprise.
- Rockwell Automation
- 1201 South Second Street
- Milwaukee, WI 53204-2496 USA
- Tel: (1) 414.382.2000
- Fax: (1) 414.382.4444
- Asia Pacific:
- Rockwell Automation
- Level 14, Core F, Cyberport 3
- 100 Cyberport Road, Hong Kong
- Tel: (852) 2887 4788
- Fax: (852) 2508 1846
- Europe/Middle East/Africa:
- Rockwell Automation
- NV, Pegasus Park, De Kleetlaan 12a
- 1831 Diegem, Belgium
- Tel: (32) 2 663 0600
Fax: (32) 2 663 0640
- Allen-Bradley, FactoryTalk, FactoryTalk Network Manager, Rockwell Automation, and Stratix are trademarks of Rockwell Automation, Inc.
- Trademarks not belonging to Rockwell Automation are property of their respective companies.
- CIP, CIP Security and EtherNet/IP are trademarks of the ODVA, Inc.
- © 2019 Cisco Systems, Inc. and Rockwell Automation, Inc. and all rights reserved.
Publication ENET-WP023B-EN-P April 2019