This guide provides guidance on a typical operational technology (OT) network deployment profile that uses Cisco DNA Center and Cisco SD-Access. You can use this guide as a validation reference.

OT networks have followed the Purdue model (which was released in 1990) for the last couple of decades. Since its release, significant changes have occurred in the networking industry and these innovations are leveraged in the OT industry today.

The following sections describe the key considerations for a large evolving Manufacturing IT/OT network that needs to meet today's automation requirements.

Resiliency, Redundancy, and High Availability

As OT networks have become critical to an organization's functionality (as in many cases, an OT network is the core of a company’s standing), it's critical to provide strict network- and service-level resiliency. Network-level resiliency is achieved with a robust fabric network design that includes dual-fabric borders nodes, dual-fabric control plane nodes, dual-anchor borders and control plane nodes, dual wireless controllers, fabric switches with either hardware stacking or StackWise Virtual, a REP ring of IE extended nodes, and dual-fabric transit control plane nodes (where applicable). Service-level resiliency is achieved by deploying the following:

• A Cisco DNA Center three-node cluster.

• A distributed Cisco Identity Services Engine (ISE) cluster with multiple Policy Administration Nodes (PAN), Monitoring Nodes (MNT), as well as active and standby Platform Exchange Grid (pxGrid) and local Policy Service Nodes (PSN).

Security and Network Segmentation

Today's OT network sometimes provides limited segmentation capabilities. Mostly these networks are VLAN based. When deeper segmentation is needed, many organizations create physically separate OT networks. To solve this issue, network administrators use logical segmentation, which has the strength of physical separation and the simplicity of a single network. This segmentation can also help address cybersecurity issues. If segmentation is instantiated by IP Access Control Lists (ACLs), it can become difficult to scale, troubleshoot, and maintain over time in most simple deployments.

Within the Cisco SD-Access architecture, Cisco DNA Center and Cisco ISE work together to provide automation for planning, configuration, segmentation, identity, and policy services. Note that Cisco ISE is responsible for device profiling, identity services, policy services, and dynamically exchanging information with Cisco DNA Center.

The Cisco SD-Access solution addresses the need for complete data and control plane isolation between IT devices and manufacturing floor devices by using macrosegmentation. By creating and putting different devices into different overlay virtual networks (VNs), the manufacturing OT network achieves complete data isolation and provides security among different IT departments.

Cisco SD-Access also addresses the need for granular data plane isolation between endpoints within the same VN by using microsegmentation with scalable group tags (SGT) for Group-Based Policy (GBP). Cisco DNA Center IT administrators create groups, place employees in those groups by their roles, and define the policies that control how these groups can interact with each other.

Silent Hosts Handling

In the OT network, one of the biggest issues is the silent hosts on the manufacturing floor. Some endpoints on this floor are silent hosts. These silent endpoints do not broadcast their presence in the network. Cisco DNA Center offers a solution to silent hosts through the Layer 2 flooding capability. Layer 2 flooding enables the flooding of broadcast, link-local multicast, and ARP traffic for a given overlay subnet. This capability maps the overlay subnet to a dedicated multicast group in the underlay, encapsulates the targeted traffic in a fabric Virtual Extensible LAN (VXLAN), and then sends the data to the destination underlay multicast group. This solution makes use of PIM Any-Source Multicast (PIM-ASM) in the underlay, which can be configured automatically (using the Cisco DNA Center LAN Automation workflow) or manually (in the later phase of deployment). This solution offers flexibility to OT networks by accommodating various OT devices with different embedded capabilities.

Layer 2-Only VN (Gateway Outside the Fabric)

OT networks have naturally a high number of Layer 2 networks in their factory floors for various use cases and business purposes. To gain a high level of security, you need to have the Layer 2 traffic inspected by a firewall outside of the fabric network. This requirement means that the first hop for all the traffic needs to be outside the fabric. This implementation uses a combination of Layer 2-only VN and underlay multicast features.

Network Assurance and Analytics

Network administrators can efficiently manage and monitor their networks to quickly respond to the OT system's dynamic needs. The deployment proactively predicts network-related and security-related risks by using telemetry to improve the performance of the network, devices, and applications. Cisco DNA Assurance, with the use of Cisco AI Network Analytics, collects telemetry data, monitors the performance of network devices, flags any issues that it detects, and offers remediation steps.

With Cisco DNA Assurance and Cisco AI Network Analytics, network administrators can monitor not only the overall health of the network devices and connected endpoints (both wired and wireless) but can determine the individual health of the devices, endpoints, and their applications. With this 360-degree analytics, administrators can identify the individual issues that network elements are facing, such as wireless laptop connectivity or an OT device's connection to a wireless SSID.

Cisco AI Endpoint Analytics

Modern security threats seek to exploit a single vulnerable point of entry in an Enterprise network for Enterprise information. When an entry point is breached, a lateral spread of threats across devices can spread in mere seconds. Granular network segmentation, like the Cisco SD-Access solution, is the preferred method to prevent this kind of lateral spread of threats. The OT network traditionally contains thousands of similar devices, such as automated guided vehicles (AGVs), programmable logic controllers (PLCs), and various manufacturing equipment and monitoring devices. Finding and identifying all the devices on a network is time-consuming and tedious. Cisco AI Endpoint Analytics addresses this issue by identifying the devices by type, manufacturer, communication protocols, and ports through passive network telemetry monitoring and deep packet inspection of the network.

Cisco DNA Center also shares the endpoint classification attributes with Cisco ISE. When new devices onboard through identity-based authentication, they can be automatically identified by the manufacturer and type and then are added to the appropriate group. It's easier to define and enforce security policies when these policies are applied to groups rather than individual endpoints. Group-based policies can be easily edited to adapt to new circumstances, such as security breakage by endpoints, and applied globally to an entire network.

The solution is validated with the hardware and software listed in the following table. For the complete list of hardware supported, see the Cisco Software-Defined Access Compatibility Matrix.

The following use cases were validated on the Manufacturing Vertical profile using the topology shown in Figure 1.

• Service and network resiliency

  • High Availability can be achieved throughout the network with dual Cisco SD-Access borders and dual control plane nodes, a border StackWise Virtual link and border/edge stack, and dual-transit control planes in the transit network. If failover and recovery of the network failure occurs, there should be no or minimal interruption to the traffic flow.

  • Administrators can configure Cisco DNA Center in the three-node High Availability mode. If services or node failure occur in the Cisco DNA Center cluster, the system should recover without user intervention.

  • The Cisco ISE distributed deployment model should be recovered with the PAN, PSN, and pxGrid service failover.

  • Administrators can implement a critical VLAN for fabric edges if Cisco ISE is unreachable.

  • Cisco DNA Center can back up the configuration and data on demand or by schedule. The backup file can be restored onto Cisco DNA Center to roll back to previous configurations.

• Implement multitiered security to protect sensitive OT network data

  • Administrators can segment users, guests, and Internet of Things (IoT) and OT devices into their logical network to limit the movement of network threats.

  • Administrators can enable Closed Authentication Onboarding (the dot1x authentication) or MAC Authentication Bypass (MAB) for wired and wireless endpoints to prevent unauthorized access.

  • Administrators can create groups, place users and endpoints into those groups by their identities, and define group-based policies that control the traffic between groups.

  • Administrators can implement a high scale of access control policies for factory floors and Security Group ACLs (SGACL) that are properly installed on edge devices when clients are onboarding.

  • Administrators can monitor Cisco DNA Center activities with the audit logs that record system events (which includes what occurred, when and where it occurred, and which users initiated the event).

  • Administrators can create granular role-based users with different privileges to access Cisco DNA Center.

• Simplified management

  • Cisco DNA Center provides the central management of device inventory and allows user to view device information, including the IP address, provision status, software releases, and inventory insights.

  • In Cisco DNA Center, administrators can use the Software Image Management (SWIM) function to upgrade the Golden image on switches, routers, extended nodes, and wireless controllers.

  • In Cisco DNA Center, the Fabric Border and Control Plane RMA workflow makes device replacement seamless.

  • The Site Border Layer 3 Handoff VLAN consumption optimization provides administrators VLAN assignment flexiblity in a scaled multisite environment.

  • LAN Automation with the overlapping pool option provides significant IP address optimization by allowing the underlay network to reuse the same address across different fabric sites.

• Network services

  • Administrators can implement the Layer 2-Only VN feature, which is intended to support Cisco SD-Access fabric endpoints. For security reasons, this has a strict entry point to the network (for example, through a firewall), which resides outside the fabric.

  • Administrator can enable redundancy with a recovery time of less than 50 ms for network failures with extended nodes and can configure a Resilient Ethernet Protocol (REP) ring for the fabric site.

  • Administrators can enable Layer 2 flooding to handle silent OT devices as well as broadcast, unknown unicast, and multicast (BUM) traffic within fabric sites.

• Monitor a network and its clients using Cisco DNA Assurance and Cisco AI Endpoint Analytics

  • Administrators can use Cisco DNA Assurance to monitor network health and identify network issues. Cisco DNA Assurance can report issues triggered by various network failures, including link down, AP down, and switch stack member down.

  • Administrators can use Cisco DNA Assurance to monitor wired and wireless client health and identify client onboarding issues.

  • Administrators can enable Telemetry Data Logger (TDL)-based assurance for better scale and performance in reporting client health.

  • Administrators can monitor a large number of concurrent endpoints, with Assurance charts showing information for 100,000 concurrent endpoints and 250,000 transient endpoints.

  • Administrators can use Cisco AI Endpoint Analytics to identify and profile endpoints and IoT devices.

  • Administrators can monitor wireless network performance with wireless sensors.

  • Administrators can enable Application telemetry and use Cisco DNA Assurance to monitor application health for latency, jitter, and packets drops.

  • Administrators can visualize communications between existing endpoints to assess the need and impact of introducing new access controls.

  • Administrator can use group-based policy analytics capabilities to create and implement microsegmentation policies.

Solution test verified the scale numbers listed in the following table. For the hardware capacity, see the Cisco DNA Center Data Sheet.

• Cisco SD-Access Solution Design Guide (Cisco Validated Design)

• Cisco Software-Defined Access for Distributed Campus Prescriptive Deployment Guide

• Cisco Software-Defined Access Compatibility Matrix