Disclaimers

■The content of this CVD applies mainly to utilities who have adopted Ethernet-connected intelligent end devices (IEDs).

■Although substation zones are mentioned, this release of the SA LAN and Security CVD version 2.3.2 focuses mainly on enhancements to the ESP zone design.

■Refer to older releases of the solution document for designs relevant to endpoints communicating using serial-based protocols such as Modbus or DNP3.

■If you do not have access to any of the Cisco SalesConnect links in Related Documentation, ask your Cisco account team to help provide you with the documentation. However, some of the documents require a non-disclosure agreement (NDA) with Cisco.

Prerequisite Knowledge

To fully comprehend the information in this document, you should:

■Have a strong foundation in how the utility operational technology (OT) world functions

■Be familiar with relevant utility industry standards and mandates, such as IEC 61850 and NERC CIP

What is Substation Automation?

Substation Automation is an intelligent electrical delivery system integrated with communications and information technology to enhance grid operations, improve customer service, lower costs, and help enable new environmental benefits. The Cisco advanced substation automation solution describes how to use the utility network to monitor and manage electrical systems from power generation, through transmission and distribution, to end users in smart buildings, smart homes, and other sites connected to the electrical utility network.

Cisco Substation Automation Solution release 1.5 addressed the following use case scenarios:

■Substation automation with and without IEC 61850 GOOSE messaging

■Substation automation, including phase measurement units (PMUs)

■Physical security (video surveillance and access)

■Remote workforce management (wired only)

■Precise timing distribution

■Remote engineering access to substation devices

■Network management

Cisco Substation Automation Solution release 2.2.1 covered the following security topics:

■Restricting access

■Protecting data

■Logging events and changes

■Monitoring activity in the substation

Cisco Substation Automation Solution release 2.3.1 focused on:

■High Availability (HA) in the ESP zone topology with PRP and REP

■GOOSE validation

■Dying Gasp

■Cisco CGR2010 4G/LTE offload

■CIP zone badge reader

■PTP in LAN

■Firewall redundancy

Ethernet-Based Substation Automation Implementation Benefits

Substation automation goes beyond traditional supervisory control and data acquisition (SCADA) to help provide added capabilities and information that can further improve operations and maintenance, increase system and staff efficiencies, and leverage and defer major capital investments.

An intelligent communications network is a foundation to building a smart grid. Utilities are investing in communications networks to improve their situational awareness of grid assets in order to control, automate, and integrate systems. Value is created when utilities “smooth out” peak load demand, forgo the use of costly spinning reserves, and alleviate the need for long-term capital investments in new generation plants and other capital investments such as reconducting for capacity improvement.

Cisco believes that substation automation is a key first step to achieving a smarter grid. The grid must be observable and measurable before it can be controlled and automated. Substation automation helps utilities add sophisticated protection and control functions while also providing greater visibility into the performance and health of grid infrastructure.

Ethernet Evolution and Standardization for Utilities

In order to integrate substation protection, control, measurement, and monitoring applications, new communication protocols have been developed and standardized under the umbrella of International Electrotechnical Commission (IEC) 61850, Communication Networks and Systems in Substations. These protocols leverage and build upon already existing Ethernet standards.

Legacy serially-connected devices now have modern Intelligent electronic device (IED) counterparts available with Ethernet ports that implement these new protocols. IEDs typically contain multiple protection, control, monitoring, and communication functions.

One specific IED that warrants special consideration because of its unique latency requirements is the phasor measurement unit (PMU). PMUs are devices capable of measuring voltages and reporting data. PMUs are used to help synchronize grid devices to ensure phase imbalance does not occur across segments of the power grid.

IEC 61850 Goals

IEC 61850, a set or over 25 standards and technical reports (TRs), has been adopted by many electrical utilities around the world. The Utility Communications Architecture International Users Group (UCA IUG) that developed the standards did so with a desire for interoperability (between devices and systems), ease of configuration (allocation of functions to devices), long term stability (layered, object-model based design), and extensibility.

IEC 61850 Protocol Stack

The IEC 61850 protocol stack defines a network architecture for the substation LAN. Figure 2 shows the basic IEC 61850 protocol stack as derived from IEC-61850-90-4 Ed1.

Figure 2 IEC 61850 Protocol Stack

The IEC 61850 protocol stack is divided into a hard-real time stack supporting the services of Sampled Values (SV), Generic Object-Oriented Substation Events (GOOSE), and the Precision Time Protocol (PTP). The IEC 61850 protocol stack also offers a soft real-time stack supporting the network time synchronization SNTP, the manufacturing message specification (MMS) communication and ancillary services mentioned in IEC 61850-8-1. These protocols rely on the services of the Media Access Control (MAC) layer, which may support 802.1Q VLANs and priorities and redundancy.

Cisco Alignment to IEC 61850 Standards

Considering the importance of IEC 61850 and the alignment of many utilities around the globe to the standards’ recommendations, Cisco decided it was important to be actively involved in some of the standard body’s technical working groups. Cisco also participates in conformance and interoperability testing conducted by the UCA IUG membership (composed of end users and vendors). Cisco substation automation architectures align with the IEC 61850 modern substation standards recommendations.

New Developments for Substation Automation

The capabilities offered by the Cisco SA LAN and Security solution have evolved since the previous validation effort. This version of the SA LAN and Security CVD version 2.3.2 emphasizes some of the more significant developments since the validation cycle of late 2016:

■An evolution in network resiliency protocols with the availability of:

–High-Availability Seamless Redundancy (HSR) singly-attached node (SAN)

–Parallel Redundancy Protocol (PRP)-HSR dual RedBox

■An evolution of network-based timing with the introduction of:

–Global Navigation Satellite System (GNSS) and Global Positioning System (GPS) support

–Precision Time Protocol (PTP) 1588 v2 timing protocol over both PRP LANs (A and B)

■Security advancements with Cisco NetFlow and Stealthwatch for traffic flow anomaly monitoring

■QoS to predictably service a variety of network applications and traffic types

■Validate a recently introduced Industrial Ethernet switch, Cisco IE 4010, for use in a substation LAN

This document starts with an overview of Ethernet in the electronic security perimeter (ESP) zone, explores some challenges associated with Ethernet-based substation automation implementations, and then expands on how Cisco technological advancements can help overcome these challenges.

A new industrial switch has since been built by Cisco, namely the Cisco Industrial Ethernet (IE) 4010. Similar to the Cisco IE 4000, the Cisco IE 4010 boasts a robust feature set applicable to many industries. It differs from the Cisco IE 4000 in form factor. There are differences in resilient protocol support, the details of which will be covered later in this document. The Cisco IE 4010 is at a better price point for those customers who do not need all of the high-bandwidth and timing capabilities of the Cisco IE 5000.

This document then explores the addition of High-Availability Seamless Redundancy (HSR) single attached node (SAN) protocol. If a customer decided to jointly implement HSR and Parallel Redundancy Protocol (PRP) lossless protocols, Cisco now offers the capability for some IE switches to serve as dual RedBoxes (that can bridge HSR and PRP networks). This section also examines the pros and cons for PRP verses HSR, since resiliency protocols are not created equally. Each has advantages and disadvantages.

An evolution of network timing functionality on Cisco IE switches has also occurred since the validation from late 2016. Cisco IE 5000 switches now support GNSS and GPS via the onboard receiver. Native GNSS and GPS capabilities on a Cisco IE 5000 switch alleviates the need for procuring and maintaining external clock sources in the substation automation environment, a capability that can save money particularly in greenfield deployment scenarios. The GNSS and GPS receiver allows the Cisco IE 5000 to directly act as a Precision Time Protocol (PTP) 1588 v2 grandmaster (GM) for the substation LAN and the Cisco IE 5000 can directly provide PTP 1588 v2 power profile to the intelligent end devices (IEDs) that require a high-precision, network-based timing input.

The Cisco solution now supports and recommends the deployment of PTP 1588 v2 over both PRP LANs and PTP frames will be protected like other protocol traffic.

This document then explores security advancements with the evolution of Cisco NetFlow support (beyond just NetFlow-lite). Cisco Stealthwatch can now leverage Cisco NetFlow for Layer 3 and higher traffic visibility. Stealthwatch provides anomaly detection of Layer 3+ traffic flows on Cisco IE switches.

The document ends with design considerations for the Cisco substation automation solutions by reviewing quality of service (QoS). QoS is what facilitates the network to predictably transport a variety of network applications and traffic types with service differentiation.

New Software Feature Support Matrix

The SA LAN and Security CVD version 2.3.2 of the solution has the features and recommended hardware platform combinations shown in Table 1.

Summary

The SA LAN and Security 2.3.2 CVD builds on previous solution releases through new technology validations in Cisco solution labs in the areas of the Cisco IE 4010 switch, new IOS software release 15.2.6E2a for Cisco IE switches, and new software features related to resiliency, timing, and security. This CVD helps to provide guidance on where the new technology fits in the Cisco overall substation automation solution. This new technology can enhance the design and deployment of a substation automation network by Cisco.

Multiservice Zone

The Multiservice Critical Infrastructure Protection (CIP) zone contains physical security components like Ethernet-connected badge readers, video surveillance cameras, local authentication, authorization, and accounting (AAA), and logging applications. If remote access from a control center into the substation ESP zone is required, Cisco recommends that a jump server, or a computer used to manage devices in a separate security zone, be installed in the multiservice zone. The multiservice zone is a likely location for security applications such as Cisco Identity Services Engine (ISE), Splunk, and downstream utility applications requiring services such as an application gateway or broker functions. Segmentation of these applications and services is highly recommended even within this zone and it can be achieved with virtual LAN (VLAN).

Corporate Substation Zone

The Corporate Substation (CorpSS) zone is an extension of the corporate network in the substation. It is where wireless Ethernet connectivity (Wi-Fi), voice services, and general Ethernet connectivity for employees to access email, web, or the Internet (via the central site) are provided. This is an extended enterprise located remotely within the substation. This zone is the least secure zone and includes devices and services such as IP phones, video end points, and Wi-Fi connected or hardwired PCs for corporate applications.

Electronic Security Perimeter Zone

The Electronic Security Perimeter (ESP) zone includes all grid operations infrastructure and is the highest security zone. It is highly recommended that this be further segmented by application such as SCADA, protection services, transformer ops, and so on. The ESP is the most critical zone in the substation and requires the highest level of security and availability. One method of achieving Ethernet network segmentation is with VLANs terminating at the substation edge firewall.

Deployment models are typically based on the size of the substation’s ESP zone. Substation IEDs can connect to Cisco IE switches built in one of a variety of topological options, namely hub and spoke, ring, or tree. Cisco offers high-availability redundancy mechanisms such as Resilient Ethernet Protocol (REP), Parallel Redundancy Protocol (PRP), and Highly-Available Seamless Ring (HSR). Choice of the topology style and redundancy protocol will depend on application requirements. Redundancy and resiliency are described in more detail later in this CVD.

Station Bus

The station bus connects the entire substation and helps provide connectivity between central management and individual bays. The station bus connects IEDs within a bay, distributed controllers, and human machine interfaces (HMIs). It connects bays to each other and connects bays with the gateway/gateway router. It may connect up to hundreds of IEDs, often segmented physically or logically, based on communication parameters or application/purpose.

Process Bus

The process bus connects primary measurement and control equipment to the IEDs. The process bus conveys unprocessed power system information (voltage and current samples and apparatus status) from the switch-yard source devices—such as current transformers (CTs), potential transformers (PTs), data acquisition units (DAUs), and merging units (MUs)—to the IEDs and relays that process data into measurements and control and protection decisions.

Typically, the process bus is limited to a bay, however busbar protection and differential protection traffic might span multiple bays.

ESP Traffic Classes Per IEC 61850

The following are the traffic class definitions as taken from IEC-61850-107D C160 9073 EB38 A493 0266 7153 D440 18AD 53F4 Ed1:

[Manufacturing Message Specification] MMS traffic defined in IEC 61850-8-1, which allows an MMS client such as the SCADA, an OPC server or a gateway to access 'vertically' all IED objects. This traffic flows both on the station bus and on the process bus, although some process bus IEDs do not support MMS. The MMS protocol is a client-server (unicast) protocol operating at the network layer (Layer 3). Therefore, it operates with IP addresses and can cross routers. In one operating mode, the MMS client (generally the SCADA or the gateway) sends a request for a specific data item to the MMS server of an IED, identified by its IP address. The server returns the requested data in a response message to the IP address of the client. In another mode, the client can instruct the server to send a notification spontaneously upon occurrence of an event.

[Generic Object Oriented Substation Events] GOOSE allows IEDs to exchange data “horizontally” in a bay or between bays. It is used for tasks such as interlocking, measurements, and tripping of circuit breakers. Based on Layer 2 Multicast traffic, GOOSE usually flows over the station bus but can extend to the process bus and even the WAN. GOOSE uses short informational messages and GOOSE requirements specify a low probability of loss and a budget delay of only a few milliseconds.

The Sampled Values protocol (SV; specified in IEC 61850-9-2) is mainly used to transmit analogue values (current and voltage) from the sensors to the IEDs. This traffic flows normally on the process bus but can also flow over the station bus, for instance, for busbar protection and phasor measurement.

ESP Traffic Requirements

As per the IEC 61850-8-1 standards, GOOSE uses publisher/subscriber communications for time sensitive and critical communications. GOOSE is a control model in which any format of data (status, value) is grouped into a data set and transmitted. GOOSE data is directly embedded into Ethernet data frames and includes mechanisms to help ensure transmission speed and reliability.

GOOSE allows IEDs to exchange data “horizontally” in a bay or between bays. It is used for tasks such as interlocking, measurements, and tripping of circuit breakers. Based on Layer 2 Multicast traffic, GOOSE usually flows over the station bus but can extend to the process bus and even the WAN. GOOSE uses short informational messages and GOOSE requirements specify a low probability of loss and a budget delay of only a few milliseconds.

GOOSE is one of the IEC 61850 traffic types within the substation that is time sensitive in nature and requires low-latency forwarding. It uses well known EtherType of 0x88b8 for easy identification and classification within the Layer 2 domain. SV packets, on the other hand, use a well-known EtherType of 0x88bA.

GOOSE traffic can deal with some jitter or some delay in interarrival time. GOOSE can have a slightly lower priority treatment when compared to SV traffic (also Layer 2 multicast).

IEC 61850 prescribes that GOOSE and Sampled Values (SV) frames are priority-tagged using a VLAN ID of 0, marked by IEDs, in order for the network to use PCP for classification and help provide preferential treatment. IEEE C37.238-2011 mandates the use of VLAN tags. Future revisions may make VLANs optional. Defaults for GOOSE, SV, and C37.238-2011 are priority-tagging with priority code point (PCP) value of 4.

IED QoS priority markings are assigned at the power systems engineering stage and recorded in the substation configuration description (SCD) file. Consider the impact to engineering design if the network decides to remark QoS values.

There are multiple types and classes of GOOSE traffic that have latency requirements ranging from 3ms to 100ms. IEC 61850-90-4 QoS classification states that GOOSE frames for tripping and inter-tripping should have high priority. GOOSE frames for interlocking should have medium priority. Finally, other GOOSE frames like heartbeats and analog values should be assigned medium priority.

Table 2 highlights the different GOOSE, SV, MMS, and time synchronization messages along with details that can help distinguish their application and communication requirements.

Protocol Locations in Station Bus and Process Bus

The protocols typically found on a station bus include GOOSE (Layer 2 multicast), MMS, SNTP, SNMP, FTP, and others (Transmission Control Protocol—TCP/IP or User Datagram Protocol—UDP/IP Layer 3 unicast).

The protocols found on a process bus are SV (Layer 2 multicast), sometimes GOOSE (Layer 2 multicast), and often MMS (Layer 3 unicast) traffic. The infrastructure connecting process bus devices is expected to provide real-time quality of service to critical traffic.

There is no hard requirement forcing SV traffic out of the station bus; in fact bus-bar protection might dictate the need for SV traffic in the station bus. If this is the case, QoS would need to be in place to preserve the lower jitter and latency tolerance of such SV traffic in the station bus.

Figure 5 is derived directly from IEC 61850 standards and illustrates where in the station and process buses you would typically find MMS, GOOSE, and SV traffic.

Figure 5 Where to Find MMS, GOOSE, and SV in Station and Process Bus

Combining the Station Bus and Process Bus

While it is possible to fit station bus and process bus into one network structure from a networking perspective (if sufficient bandwidth is available, such as 1 Gbit/s or higher), it is prudent to separate them for various reasons. For instance, consider separating buses to reduce station bus load due to chatty application traffic on the process bus. If you must combine them, think about avoiding single points of failure when coupling process and station buses.

Refer to IEC 61850-90-4 for additional details, including many possible topology design options.

Summary

The traffic flows in the substation are modeled on the 61850 standards. GOOSE and SV traffic demand high priority treatment and low latency requirements. GOOSE data has latency requirements in the 3-100 msec range, while Sample Values are in the 3-10 msec range. Timing is also extremely important when transported within and between substations. Network-based timing can be carried across the station over Ethernet using the IEEE-based Precision Time Protocol 1588 v2 (PTP v2).

Given this summary of the basics about utility substation automation Ethernet-connected applications and advantages in an Ethernet-based substation design, what are some of the main challenges that need to be overcome in order to design an efficient Ethernet-based substation ESP network?

Challenge 1—Latency and Jitter

The first challenge is to design for low latency. Latency is directly impacted by buffering in the network. Buffering happens when links are oversubscribed and congested. Latency therefore correlates to bandwidth availability and quality of service.

Cisco offers different Industrial Ethernet switches with varying interface bandwidth capacities. Do your applications require 10 Mb Ethernet connections? What about 100 Mb? Have you analyzed the bandwidth consumption once application traffic is enabled?

At Layer 2, latency is kept to a minimum because switches are designed to forward frames using hardware lookups. Layer 3 boundaries (where routing comes into the picture) add software lookups and buffering, hence traffic crossing a Layer 3 boundary will incur a higher latency. Layer 2 traffic that is restricted within a local substation zone will experience much lower latency than traffic that needs to cross a Layer 3 boundary (through the DMZ).

Jitter is variation in the latency or inter-arrival time of frames/packets. Assuming you complete your capacity planning activity, what happens if and when link oversubscription occurs? Not all traffic is sensitive to latency and jitter, meaning not all traffic classes require the lowest possible latency. Are the important traffic classes being offered preferential treatment via QoS classification, marking, and prioritization?

In summary, to deal with low latency, choose network switches that offer a high enough bandwidth capacity for your applications and design a proper QoS model that offers differentiated services in order for higher priority traffic to be passed through your network before medium and lower priority traffic. With the help of a well-planned QoS design for packets granted the same treatment, latency should not vary much between one frame/packet and the next. Hence, jitter would be kept to a minimum.

Challenge 2—Fast Convergence

The second challenge is to design for fast convergence. Convergence is all about recovery time; how long does it take for your traffic flows to recover in the event of a link or node failure.

As discussed, certain applications and protocols have differing latency requirements. Note what makes ESP traffic latency requirements more difficult to achieve is that if there ever was a network outage due to a link or node failure, the ESP traffic is still expected to abide by the protocol requirements. Convergence needs to be as fast as possible because convergence events will add latency when a failure occurs.

Protocol selection should be based on the most demanding application’s requirements in the substation’s ESP zone. A summary of resilient protocol convergence performance times is provided in Table 3.

Challenge 3—Scale

The third challenge is to design for scale. From a pre-emptive planning perspective, a network can be designed for scale and protected from oversubscription with a proper traffic segmentation model. Knowing that substation traffic like GOOSE and SV are Layer 2 multicast traffic types, which effectively behave like a Layer 2 broadcast, it is important to consider which GOOSE/SV endpoints need to subscribe to one another and hence need to “listen” to each other’s traffic. Scale can be planned using:

■VLANs—Assign VLANs to limit broadcast storms. Assign switch access ports to a VLAN and only devices on the same VLAN would ever receive the traffic from devices on the same VLAN. Note that the intelligent end devices themselves may have the ability to natively define the VLAN so that traffic egressing is already marked. Ports on the switches need to understand and look for the VLAN headers by configuring them as dot1q trunk ports.

■Multicast filtering—Layer 2 access lists can be implemented that filter and only allow traffic with specific Layer 2 multicast addresses. Insight into what multicast addresses are in use and for which devices across the network is critical to create meaningful access lists.

Cisco utility substation automation designs recommend the network administrator consider both methods of planning for scale.

One might think scale can be resolved by adding more equipment into the infrastructure. Adding more equipment with more ports is fine for scaling available access ports for device connectivity, but do not lose sight of what happens to all of the access traffic as it traverses the zone between switches. The appropriate amount of bandwidth should be available at aggregation points to allow the traffic through.

Some protocols have a limit in terms of recommended number of nodes participating. Ring size will ultimately be impacted by the recommendations for each protocol. Consider the protocols in use, the amount of traffic for each application flow, and factor this information into your ring size planning activities.

Challenge 4—Security

The forth challenge is to design for security. Security can be achieved using traffic segmentation.

Figure 6 Cisco Substation Automation Reference Architecture

Referring to the reference architecture in Figure 6, Cisco recommends that the substation network be segmented in the following ways:

■Zones—At this point, the CVD already highlighted the need for separation of functional zones (multiservice, CorpSS, and ESP zones). Extended enterprise and other traffic should never mix with the critical (ESP) grid monitoring and controlling application traffic.

■Inside the ESP zone, Layer 2 traffic can and should be separated by VLAN.

■Beyond the ESP zone Layer 2 boundary, a router or firewall can encrypt outgoing traffic and map logically-segmented traffic zone traffic destined to the control center.

Storm control, a Layer 2 network security protocol, prevents traffic on a LAN from being disrupted by a broadcast, multicast, or unicast storm on one of the physical interfaces. A LAN storm occurs when frames flood the LAN, creating excessive traffic and degrading network performance. Errors in the protocol-stack implementation, mistakes in network configurations, or users issuing a denial-of-service attack can cause a storm.

Be mindful when implementing security protocols that the effectiveness quickly diminishes should any devices or network switches lack support for the security protocols. Utilities with mixed vendor deployments cannot fully implement a single vendor’s proprietary technology. Hence, mixed vendor environments are limited to implementing industry standard protocols on all devices.

For Layer 3 traffic in the ESP zone (for example, MMS, Modbus TCP, and DNP3 IP), consider enabling features like Cisco NetFlow and leveraging security applications like Stealthwatch. CIsco NetFlow aides Stealthwatch to provide anomaly detection within the ESP zone because Cisco NetFlow helps provide visibility into the traffic traversing the zone’s infrastructure. Stealthwatch can aid with troubleshooting and highlighting abnormal behaviors due to malware infections. With the Cisco IE 4000, Cisco IE 4010, and Cisco IE 5000, Cisco NetFlow can be enabled to help provide data flow metrics to Stealthwatch. Stealthwatch takes the flow data from the network then helps offer real-time situational awareness of all users, devices, and traffic. Stealthwatch has many in-built machine learning algorithms that can assist in analyzing all of this data and detecting anomalies in the network.

Challenge 5—Field Serviceability

Devices in the ESP zone are designed to be easily replaceable should the need arise for field maintenance. How do Cisco Industrial Ethernet switches and routers accomplish this?

First, the devices can be configured to send what is called a “dying gasp” to alert the operations team that a critical failure has occurred. A dying gasp is a notification message which signifies a critical event, such as loss of power or a link going down. Dying Gasp notifications are available through Ethernet Operations and Maintenance (OAM) frames, SYSLOG messages, and SNMP traps.

Next, Cisco hardened switches and routers leverage replaceable flash disks containing the software and configuration files necessary for the devices to function. Simply swapping out the flash disks from the old to the new devices helps ensure that the new devices will boot and function exactly like the devices they are replacing.

Challenge 6—Network Management

Utility substation deployments can mimic in size an enterprise branch office deployment, which in total can contain hundreds or even thousands of switches, routers, and security appliances. Network management of large installations can be challenging. To meet these challenges, Cisco provides a robust set of mechanisms referred to as the network management system (NMS) through which the operator configures and monitors the performance of the substation LAN deployment. Several management platform options are available, including Cisco Industrial Network Director (IND), Cisco Prime Infrastructure (PI), and each of the Cisco IE switches that are configurable using an embedded device manager.

During the validation phase of this CVD, the embedded device manager, which is a web-based graphical user interface, was tested as an alternative to the functionality provided by the command line interface (CLI). We recommend using the embedded device manager as an alternative to the CLI in some cases, however not all CLI-available functionality is present in the device manager at this time. See the implementation sections for more details.

Note that Cisco IND and Cisco PI were not validated in this release. However, they are both regularly used in Cisco IoT solution validation.

Network management traffic would span between control center and substations, traversing the WAN. A dedicated management L3VPN is recommended to segregate management traffic from other traffic types that travel across the utility’s distributed architecture. Once inside the substation, network management traffic should map to a consistent VLAN across the ESP zone. For all network management traffic on IE switches, Cisco recommends that the administrator designate a management port and an IP address for device communication.

This document now examines in more detail how Cisco hardware and software products can help build a better substation automation network by overcoming the ESP zone design challenges.

New Hardware Availability—Cisco IE 4010

This section describes the newest Cisco IE switch, the Cisco IE 4010. The IE designation means this switch is meant to be deployed in non-carpeted, rugged locations that experience extreme high or low temperatures and that require passive cooling capabilities and no internal moving parts.

■The Cisco IE 4010 switch boasts a robust feature set applicable to many industries.

■The Cisco IE 4010 form factor fits into a 19” rack mount (rather than being DIN-rail mounted like the Cisco IE 4000).

■The Cisco IE 4010 is at a better price point for those customers who do not need the higher 10 Gbps uplink bandwidth or built-in timing capabilities (GNSS, GPS, and IRIG-B) of the Cisco IE 5000.

■The Cisco IE 4010 comes in a either a 12- or a 24-port configuration with 200 watts of power over Ethernet (PoE) budget usable for IP cameras and access points (APs).

■The Cisco IE 4010 offers 1 Gigabit up/downlinks.

■The Layer 2 and Layer 3 feature sets on the Cisco IE 4010 are similar to the Cisco IE 4000 and Cisco IE 5000 offerings, however there are some differences:

–Cisco IE 5000 supports PTP GM

–Cisco IE 4000 supports HSR-PRP RedBox

Deployment Locations for the Cisco IE 4010

Consider deploying the Cisco IE 4010 switch in the following places in the substation automation network:

■Electronic Security Perimeter (ESP) zone

–Station bus

–Process bus

■Multiservice zone

–Physical security and badge-reader connectivity

–Wireless access point connectivity

■Corporate zone—Providing connectivity to corporate users

ESP Design Challenges the Cisco IE 4010 Addresses

The Cisco IE 4010 was purpose-built with industrial environment demands in mind. It forwards Layer 2 frames using hardware lookups, so Ethernet frames are forwarded with ultra-low latency. With a proper QoS configuration to protect real-time, latency sensitive traffic, latency and jitter should not be a problem as long as traffic stays inside the Layer 2 ESP zone.

The Cisco IE 4010 supports all the variants of fast convergence protocols like RSTP, REP, PRP, and HSR. It supports segmentation using VLANs and multicast filtering. It is one of the platforms that supports Cisco NetFlow collectors and works with Stealthwatch. It supports field serviceability and network management.

Legacy Resiliency Protocols

Depending on whether or not a utility decides to fully embrace IEC 61850 design recommendations, several resiliency protocols are available from which to choose. For utilities that need more time before they can migrate the substation implementation to Cisco and IEC 61850-based standards, a few legacy resiliency protocols are available to choose from within ring topology deployments:

■Rapid Spanning Tree (RSTP)—A variant of spanning tree protocol (STP) that is known, used, and trusted by IT professionals who have previously used Cisco switches.

■Resilient Ethernet Protocol (REP)—A Cisco proprietary protocol described in Resilient Ethernet Protocol.

Rapid Spanning Tree Protocol

Rapid Spanning Tree Protocol (RSTP; IEEE 802.1w) is an evolution of the STP IEEE 802.1D standard. Like its predecessor, RSTP creates spanning-tree instances. RSTP addresses many convergence issues that existed with STP. The CPU and memory requirements are higher than what is needed for 802.1D. RSTP speeds the recalculation of the spanning tree when the Layer 2 network topology changes. It does so by immediately changing a blocked port to a forwarding state without waiting for the network to converge. RSTP can achieve much faster convergence in a properly configured network. RSTP is compatible with STP networks.

Cisco IE switches support RSTP. RSTP, however, takes on the order of milliseconds (at best) to seconds (at worst) to converge in the event of a link or a node failure in the ring topology.

Resilient Ethernet Protocol

Resilient Ethernet Protocol (REP) is a Cisco-proprietary protocol designed to meet fast convergence requirements in a large scale, Layer 2 network, particularly for ring topologies. REP avoids the need for Spanning-tree in simple ring-based topologies and is designed to operate with standard Ethernet hardware. REP is implemented on Cisco Connected Grid (CG), Industrial Ethernet (IE), and Carrier Ethernet (CE) platforms. It delivers fast and predictable convergence in a ring topology with convergence typically in the 50 - 250 msec range in most cases. It is deterministic, scalable, and it coexists with spanning tree. An industry standard protocol, G.8032, was later derived from REP.

Other Resiliency Protocols

When following IEC 61850 implementation standards in the station bus and process bus, high performance applications in the utility substation mandate a number of key requirements that should be addressed. The substation architecture must meet design requirements for GOOSE and Sample Values, both of which are multicast traffic types. This includes high availability (HA) and topology choices to meet scale, segmentation, and communications requirements. IEC 61850-5 provides guidance for HA and communication requirements. There are two choices:

■Parallel Redundancy Protocol (PRP) supports either tree or ring topologies with no limits on node counts and it can deliver a zero msec failover/recovery requirement. However, PRP has one drawback. PRP requires duplicate LANs (named LAN-A and LAN-B) and double the networking equipment hardware.

■Highly Available Seamless Ring (HSR) also delivers a zero msec fail-over/recovery requirement but is only available in a ring topology. HSR scales to a limited number of devices based on application bandwidth requirements. HSR does not require duplicate LANs (double the switching infrastructure) in the ESP.

Protecting the Resilient Network With Storm Control

With HSR and PRP resiliency protocols, Cisco recommends protecting the network from storms as they can very quickly multiply even faster considering the packet replication that occurs by applying a redundancy feature. Storm control prevents traffic on a LAN from being disrupted by a broadcast, multicast, or unicast storm on one of the physical interfaces. A LAN storm occurs when packets flood the LAN, creating excessive traffic and degrading network performance.

How Lossless Resiliency Protocols Solve ESP Design Challenges

Lossless resiliency protocols like PRP and HSR ultimately help ensure that critical, real-time traffic in the substation ESP zone gets delivered in time, even in the event of a network failure. An Ethernet link or an entire switch can suffer downtime without leading to any overall loss of critical application traffic. Hence, latency requirements are maintained.

PRP Support—New Features on Cisco IE Switches

At the time the previous Cisco SA LAN and Security CVD version 2.3.1 solution was validated, PRP interoperated with PTP 1588 v2 on Cisco IE switches. However, support then was limited to carrying PTP frames over a single LAN (LAN-A). It was Cisco’s recommendation to block PTP frames from traversing LAN-B.

With the current SA LAN and Security CVD version 2.3.2, PTP protocol support has been extended to operate over both PRP LANs. Once the network administrator enables PTP frames to be carried over both LANs, PTP frames become protected just like other Layer 2 protocol frames and the loss of a single node or a link in either of the LANs will result in a loss-less convergence of the PTP 1588 v2 protocol. This means that IEDs will continue to stay in synchronization with a precise time source should one of the paths experience a failure.

PRP resiliency support, including PTP over PRP, is available on 8/16 port Cisco IE 2000U, Cisco IE 4000, Cisco IE 4010, and Cisco IE 5000 series switches. Operations staff can rest assured that Cisco’s current PRP implementation offers as a resiliency protocol that meets the need for applications requiring precise timing via PTP 1588 v2 protocol.

HSR Loop Avoidance

To avoid loops and use network bandwidth effectively, an HSR RedBox does not transmit frames that have already been transmitted in the same direction. See Figure 9 for an illustration of this behavior.

To avoid loops, when a an HSR RedBox injects a frame into the ring, the frame is handled as follows:

■Unicast frame with destination inside the HSR ring—When the unicast frame reaches the destination node, the frame is consumed by the respective node and is no longer forwarded.

■Unicast frame with destination not inside the ring:

–Since this frame does not have a destination node in the ring, it is forwarded by every node in the ring until it reaches the originating node.

–Every node has a record of the frames it had previously sent, along with the direction in which it was sent.

–Every node will check whether it has already forwarded the received packet through its outgoing interface.

–Once the originating node detects that frame has completed the loop, it drops the frame.

■Multicast frame:

–A multicast frame is forwarded by each HSR node because there can be more than one consumer (intended receiver) of this frame.

–For this reason, a multicast frame always reaches the originating node.

–Every node will check whether it has already forwarded the received packet through its outgoing interface.

–Once the originating node detects that frame has completed the loop, it drops the frame.

HSR RedBox Modes of Operation

An HSR RedBox can operate in one of the following modes:

■HSR-SAN—This is the most basic mode. In this mode, the RedBox connects SAN devices to an HSR Ring. No other PRP or HSR network is involved in this configuration. In this mode, the traffic on the upstream switch port does not have HSR/PRP tags and the RedBox represents the SAN device as a VDAN in the ring.

■HSR-PRP—This configuration is used to bridge HSR and PRP networks. The RedBox extracts data from the PRP frame and generates the HSR frame using this data. It performs the reverse operation for packets in the opposite direction. Figure 10 depicts HSR SAN and HSR-PRP operating modes and how they work.

Figure 10 HSR SAN and HSR-PRP Operating Modes

GNSS and GPS on Cisco IE 5000

Cisco Industrial Ethernet switches are capable of accurate time distribution using PTP, but the switches previously relied on an external source to help provide accurate time.

A significant enhancement is the new, built-in Global Navigation Satellite System (GNSS) receiver on the Cisco IE 5000 switch. The GNSS and GPS receiver enables the switch to determine its location and get an accurate time reference from a satellite constellation. The Cisco IE 5000 switch can now serve as the PTP grandmaster clock and offer precise time in the substation LAN.

Note: Select Cisco IE 5000 switches with SKUs that have Version ID (VID) v05 or higher and GNSS firmware version 1.04 or higher.

PTP Over PRP

Precision Time Protocol (PTP) is defined in IEEE 1588 as Precision Clock Synchronization for Networked Measurements and Control Systems and was developed to synchronize the clocks in packet-based networks that include distributed device clocks of varying precision and stability. PTP is designed specifically for industrial, networked measurement and control systems and is optimal for use in distributed systems because it requires minimal bandwidth and little processing overhead.

Previously, PTP traffic was allowed only on LAN-A of PRP. However, if LAN-A went down, PTP synchronization was lost. PTP can now leverage the redundancy benefits offered by PRP infrastructure, however PTP packets are handled differently than other types of traffic over PRP networks. The current Cisco implementation of PTP over PRP does not append an RCT to PTP packets and bypasses the PRP duplicate/discard logic for PTP packets.

PTP Grandmaster Redundancy

The following are possible ways that the PTP GM can be positioned in a PRP topology:

■A single PTP GM can be a Redbox that connects to both PRP LANs (LAN-A and LAN-B).

■A single PTP GM can be a VDAN that connects to a PRP RedBox.

■Dual Star Topology—Two PTP GMs can be Redboxes and each PTP GM connects to both PRP LANs (LAN-A and LAN-B). This is the Cisco recommended approach.

As with any critical protocol in the ESP zone, timing redundancy should be considered when designing the ESP network. As such, Cisco recommends deploying redundant PTP GMs in the ESP zone in a dual-star topology.

Security Principles

The following fundamental principles should be adopted by the utility network operator to ensure secure systems:

■Visibility of all devices in the substation LAN network—Traditionally, enterprise devices such as laptops, mobile phones, printers, and scanners were identified by the enterprise management systems when these devices accessed the network. This visibility must be extended to all devices in the substation.

■Segmentation and zoning of the network—Segmentation is a process of bounding the reachability of a device and zoning is defining a layer where all the members in that zone will have identical security functions. Segmenting a network using zones helps provide an organized way of managing access within and across zones. Segmenting the network-connected devices further reduces the risk of an infection spread if or when a device is subjected to malware.

■Identification and restricted data flow— All the devices in the substation must be identified, authenticated, and authorized. The network must enforce policies when users and utility assets attach to the network.

■Network anomalies—Any unusual behavior in network activity must be detected and examined to determine if the new behavior is intended or is due to a malfunction of the device. Detecting network anomalies as soon as possible gives operations teams the means to remediate network abnormality sooner, thereby reducing possible downtimes.

■Malware detection and mitigation—Unusual behavior displayed by an infected device must be detected immediately and security tools should allow remediation actions to the infected device.

■Deep packet inspection—Traditional firewalls are not built for industrial environments. There is a need for industrial firewalls which can perform deep packet inspection on industrial protocols to identify anomalies in traffic flow.

■Environmental—Secure infrastructure hardening of the networking assets in the substation.

■Security standards and assessments—Abiding by North American Electric Reliability Corporation Critical Infrastructure Protection (NERC CIP) mandates for critical substations in North America and following guidelines from NERC CIP and NIST elsewhere. Security risk assessments identify control systems and their level of criticality.

Security Using Cisco NetFlow and Stealthwatch for Anomaly Detection

This version of SA LAN and Security CVD heavily leverages the validation outcomes available in Networking and Security in Industrial Automation Environments Design and Implementation Guide (IA CVD):
https://www.cisco.com/c/en/us/td/docs/solutions/Verticals/Industrial_Automation/IA_Horizontal/DG/Industrial-AutomationDG.html

The IA CVD offers design guidance for enabling Cisco NetFlow and leveraging Stealthwatch to help provide anomaly detection. Visibility into the traffic traversing the utility network can aid with troubleshooting and highlight abnormal behaviors. With the Cisco IE 4000, Cisco IE 4010, and Cisco IE 5000, Cisco NetFlow can be enabled on these devices that can help provide data flow metrics to Stealthwatch. Stealthwatch takes the flow data from the network and has many inbuilt machine learning algorithms that can assist an IT security professional in detecting possible malware propagation in the network.

Note that within substation ESP zones, Cisco NetFlow and Stealthwatch can be used to detect anomalies with Layer 3 traffic flows (for example, MMS, MODBUS TCP, and DNP3 IP). However, since GOOSE and SV protocols are Layer 2 only, Cisco NetFlow collections are not applicable and Stealthwatch dashboards would not show GOOSE and SV Layer 2 frame flows.

A more detailed CVD solution release is being planned that will be devoted to the topic of network security for utilities. For now, refer to the Networking and Security in Industrial Automation Environments Design and Implementation Guide for ideas on designing and implementing Cisco NetFlow and Stealthwatch for a utility substation ESP zone.

HSR SAN

A maximum of two ports are configurable for HSR SAN on Cisco IE switches. Specific interfaces are reserved for use of the HSR SAN feature. HSR is configurable using CLI and the web-based, embedded device manager tool available for Cisco IE switches.

When configuring a Cisco IE switch in SAN mode, a non-HSR aware IED can be connected as a Virtual Doubly Attached Node (VDAN). The Cisco IE switch in SAN mode becomes a proxy for the VDAN and allows the VDAN traffic to be protected.

Commands are available for troubleshooting the HSR SAN feature, including the ability to view:

■Configuration details

■HSR ring state

■Statistics for HSR components

■All MAC addresses accessible to the SAN switch using the HSR interface (other HSR nodes in the ring and VDANs)

■The HSR VDAN (proxy node) table

■Neighboring switch information

■HSR alarms

The HSR SAN implementation on Cisco IE switches supports VLAN-tagging of the HSR supervision frames.

HSR SAN is interoperable with many protocols and features on Cisco IE switches, including but not limited to the following:

■HSRP

■LLDP

■CDP

■SNMP

■QoS (Layer 2 CoS)

■VLAN untagged and tagged traffic

Note that HSR SAN is not interoperable with the following protocols on Cisco IE switches:

■Flex Link

■Ether Channel

■REP

■PTP

Single HSR Ring Process Bus

High-availability Seamless Redundancy (HSR) is defined in International Standard IEC 62439-3-2016 clause 5. HSR is designed to work in a ring topology. Instead of two parallel independent networks of any PRP topology (LAN-A and LAN-B), HSR defines a ring with traffic in opposite directions. Port-A sends traffic counter clockwise in the ring and Port-B sends traffic clockwise in the ring.

Figure 11 shows an example of an HSR Ring for Substation Automation Process Bus using Cisco IE 4000 variant switches. Traffic in the network consisted of Sample Values (tagged—VLAN 0 and VLAN 110), GOOSE (VLAN 120), and untagged IP packets.

Figure 11 Process Bus as HSR Ring

To validate the resiliency and the corresponding latency requirements of the network, failures like switch failure and link flap were introduced at different points as highlighted and numbered in Figure 11:

■(1)—A redundant network switch

■(2)—A redundant network link

Refer to Table 8 for device role information for HSR SAN.

Configuring HSR SAN Using CLI

Follow these steps to configure HSR-PRP mode on the switch. Enabling HSR-PRP mode creates an HSR ring and a PRP channel.

Before you begin, note:

■HSR ring ports can only be configured in Layer 2 mode.

■It is recommended to configure settings like media-type, speed, and duplex settings of the port before configuring ring membership.

■Once a port is part of HSR ring, the port cannot be shut down. Instead the HSR Ring interface can be shut if required. However, this operation would shut down both of the member ports.

■HSR VLAN(s) should be enabled on the switch using the following commands:

■When an IED sends VLAN 0 tagged packets, it is recommended to configure the IED-facing interface and the uplink interfaces as trunk port allowing VLAN 1 along with other required VLANs.

Summary of Steps

1. Activate HSR feature mode:

Note: Reload the switch for the change to take effect. Confirm the reload when prompted and wait for the switch to reload and boot.

Refer to the following configuration guide for more details on licensing:
https://www.cisco.com/c/en/us/td/docs/switches/lan/cisco_ie4010/software/release/15-2_4_EC/configuration/guide/scg-ie4010_5000.pdf

2. Verify that the HSR feature is activated:

3. Enter global configuration mode:

4. Enter interface configuration mode and disable PTP on the ports to be assigned to the HSR ring:

Note: PTP feature over HSR ring is currently not supported on Cisco IE switches.

5. Shut down the ports before configuring the HSR ring:

6. Create the HSR ring interface:

7. Assign HSR ring to the physical interfaces:

8. Repeat the steps on remaining switches part of the HSR ring.

Verifying HSR SAN Using CLI

Verify the functionality of HSR SAN using the following commands:

Recommended Practices for HSR-SAN Configuration

■Disable PTP on interfaces where PTP is not necessary.

■Enable storm control for broadcast and multicast traffic on access facing interfaces:

■Configure different VLANs for different IEDs so as to avoid flooding of multicast and broadcast messages to other devices.

Note: PTP over HSR ring is not currently supported on Cisco IE switches.

Configuring HSR-SAN Using Device Manager

This section assumes that the Cisco IE switch has been installed and pre-configured with an IP address for remote access. For more details on setting up a Cisco IE switch, refer to the Cisco IE switch installation guides.

1. Log in to the switch using Device Manager credentials, as shown in Figure 12.

Figure 12 Device Manager Login Screen

2. After a successful login, the Dashboard for the switch loads, as shown in Figure 13.

Figure 13 Cisco IE 4000 Device Manager Dashboard

3. Enable the HSR feature on the Cisco IE switch using the options highlighted in Figure 14.

Figure 14 Enable HSR Feature

4. Configure HSR Ring and its related parameters on the Cisco IE switch using the options highlighted in Figure 15.

Figure 15 HSR Ring Parameters

Configuring HSR-PRP Dual RedBox Using CLI

Follow these steps to configure HSR-PRP mode on the switch. Enabling HSR-PRP mode creates an HSR ring and a PRP channel. Before you begin, note:

■Enabling HSR-PRP mode will disable all ports other than two HSR ports and one PRP port and all port settings for these disabled ports will return to default values. A warning message is displayed to notify you that interface configurations will be removed. Before enabling or disabling HSR-PRP mode, check for cables connected to the switch and verify the ports’ status.

■HSR-PRP RedBox mode uses ports Gi1/3 and Gi1/4 as HSR ring 2 interfaces and Gi1/1 (for RedBox A) or Gi1/2 (for RedBox B) as PRP channel 1 interfaces. These port assignments are fixed and cannot be changed. Therefore, HSR-PRP Dual RedBox mode is supported only on HSR ring 2.

■PRP uplink interfaces can be configured as access, trunk, or routed interfaces.

■PRP Dual Attached Nodes and RedBoxes add a 6-byte PRP trailer to the frame. To help ensure that all packets can flow through the PRP network, increase the MTU size for switches within the PRP LAN-A and LAN-B network to 1506 as follows:

■PRP VLANs can be enabled on the switch using the following commands:

■When an IED sends VLAN 0-tagged packets, it is recommended to configure the IED-facing and uplink interfaces as trunk ports allowing VLAN 1 along with other required VLANs:

Summary of Steps

1. Activate HSR feature mode:

For more details refer to the following configuration guide:
https://www.cisco.com/c/en/us/td/docs/switches/lan/cisco_ie4010/software/release/15-2_4_EC/configuration/guide/scg-ie4010_5000.pdf

Note: Reload the switch for the license change to take effect by confirming the reload prompt and waiting for the switch to boot.

2. Verify that the HSR feature is activated:

3. Enter global configuration mode:

4. Enable HSR-PRP mode and select LAN-A or LAN-B and the PRP Net ID:

Note: PRP LAN: prp-lan-a-RedBox Interlink is connected to lan-A. prp-lan-b-RedBox Interlink is connected to lan-B.

5. Enter yes to confirm enabling HSR-PRP mode. To disable HSR-PRP RedBox mode, use the command:

6. Enter interface configuration mode and disable PTP on the ports to be assigned to the HSR ring:

Note: PTP feature over HSR ring is currently not supported.

7. Shut down the ports before configuring the HSR ring:

8. Create the HSR ring interface:

9. Assign HSR ring to the physical interfaces:

10. Create PRP LAN interface. Repeat the step on the second HSR-PRP RedBox:

11. Assign PRP channel to the physical interface. Follow the guidelines for identifying the switch role and the corresponding interface:

■Refer to Implementing HSR SAN to configure HSR on Cisco IE switches that participate in the HSR ring.

■Refer to Implementing PRP RedBox to configure PRP on Cisco IE switches that participate in the PRP network.

■Per VLAN Spanning Tree protocol is enabled by default on Cisco Industrial Ethernet switches. Refer to the following link to configure other modes of Spanning Tree Protocol on Cisco Industrial Ethernet Switches:
https://www.cisco.com/c/en/us/td/docs/switches/lan/cisco_ie4010/software/release/15-2_4_EC/configuration/guide/scg-ie4010_5000/swmstp.html

Verifying HSR-PRP Dual RedBox Using CLI

Verify HSR-PRP functionality using the following commands:

Recommended Practices for HSR-PRP Dual RedBox

■Disable PTP on interfaces where PTP is not necessary.

■Enable storm control for broadcast and multicast traffic on access facing interfaces:

■Configure separate VLANs for different IEDs to avoid flooding of unnecessary multicast and broadcast messages to non-participating devices.

Configuring HSR-PRP Dual RedBox Using Device Manager

This section assumes that the Cisco IE switch has been installed and pre-configured with an IP address for remote access. For more details on setting up a Cisco IE switch, refer to the Cisco IE switch installation guides.

1. Log in to the switch using Device Manager credentials, as shown in Figure 18.

Figure 18 Device Manager Login Screen

2. After a successful login, the Dashboard for the switch loads, as shown in Figure 19.

Figure 19 Cisco IE 4000 Device Manager Dashboard

3. Enable the HSR feature on the Cisco IE switch using the options highlighted in Figure 20. Select the Admin tab, select the Feature Mode option, and then select HSR as the required feature mode.

Figure 20 Enable HSR Feature

4. Configure HSR-PRP Redbox by selecting the highlighted steps in Figure 21.

Figure 21 Configure HSR Feature

5. Configure HSR-PRP Rebox parameters on the Cisco IE switch using the options highlighted in Figure 22.

Figure 22 Configure HSR-PRP Parameters

Configuring PRP RedBox Using CLI

Summary of Steps

To create a PRP channel and PRP group on the switch, follow these steps:

1. Enter global configuration mode:

2. Create PRP LAN interface:

3. Attach PRP channel to the physical interface. Follow the guidelines for identifying the switch role and the corresponding interface:

Refer to the following configuration guide for licensing if required:
https://www.cisco.com/c/en/us/td/docs/switches/lan/cisco_ie4010/software/release/15-2_4_EC/configuration/guide/scg-ie4010_5000.pdf

Verifying PRP RedBox Using CLI

Verify HSR-PRP functionality using the following commands:

Configuring PRP RedBox Using Device Manager

This section assumes that the Cisco IE switch has been installed and pre-configured with an IP address for remote access. For more details on setting up a Cisco IE switch, refer to the Cisco IE switch installation guides.

1. Log in to the switch using Device Manager credentials, as shown in Figure 23.

Figure 23 Device Manager Login Screen

2. After a successful login, the Dashboard for the switch loads, as shown in Figure 24.

Figure 24 Device Manager Dashboard

3. Configure PRP by selecting the PRP option under the Configure tab as highlighted in Figure 25.

Figure 25 Configure PRP Using Device Manager

4. Configure PRP Channel properties like channel number, switchport mode, allowed VLANs, native VLANs, and so on as highlighted in Figure 26.

Figure 26 Configure PRP Channel Properties

5. The status of the PRP Channel would be reflected as soon as the configuration of the PRP channel is completed as highlighted in Figure 27.

Figure 27 PRP Channel Status

6. Select the PRP option listed under the Monitor tab to check details of VDAN and Node table details, as shown in Figure 28.

Figure 28 Monitor PRP

Configuring PTP over PRP Using CLI

Follow these steps to configure PTP over PRP network. When the grandmaster clock requires tagged packets, make one of the following configuration changes:

■Force the switch to send tagged frames by entering the global vlan dot1q tag native command:

■Configure the grandmaster clock to send and receive untagged packets. If you make this configuration change on the grandmaster clock, you can configure the switch port as an access port.

■Force the switch to tag PTP packets by entering interface level command ptp vlan < vlanID >. With this configuration change, the switch would tag all ptp packets traversing through the interface with the corresponding VLAN.

Note: PTP VLAN configuration is supported only in PTP boundary clock mode while using Cisco IOS Version 15.2.6E2a. PTP VLAN configuration for PTP Transparent clock mode will be supported in future releases.

Note: When PRP LANs use any of the spanning tree protocols like RSTP for loop avoidance, some type of PTP packets are looped. This issue is observed while using Cisco IOS Version 15.2.6E2a. The corresponding defect is CSCvm17877. In such scenarios, it is recommended to use a Cisco IOS version with a fix for defect CSCvm17877.

When the network requires CoS values for PTP packets to be set, make one of the following configuration changes:

■The switch by default sets the CoS value to 4 to all tagged PTP packets as per IEEE C37.238 standard in PTP Power profile mode.

■Force the switch to set CoS value for PTP packets by entering global ptp packet < cos > command:

Summary of Steps

1. Enter global configuration mode:

2. Set the Power Profile:

3. Specify the synchronization clock mode:

–mode boundary pdelay-req—Configures the switch for boundary clock mode using the delay-request mechanism. In this mode, the switch participates in the selection of the most accurate master clock. Use this mode when overload or heavy load conditions produce significant delay or jitter.

–mode p2ptransparent—Configures the switch for peer-to-peer transparent clock mode and synchronizes all switch ports with the master clock. The link delay time between the participating PTP ports and the message transit time is added to the resident time. Use this mode to reduce jitter and error accumulation. This is the default in Power Profile mode.

–mode forward—Configures the switch to pass incoming PTP packets as normal multicast traffic.

4. Specify TLV settings:

5. Specify synchronization algorithm if the switch is configured as PTP Boundary Clock:

–feedforward—Very fast and accurate. No PDV filtering.

–filter linear—Provides a simple linear filter (default).

Verifying PTP Over PRP Using CLI

Use the following commands to check PTP clock type, grandmaster properties, and clock source:

On a PRP Redbox and PTP GM Cisco IE 5000 switch, the PTP state on PRP member ports can be checked as follows. The PTP port state of both PRP member ports should be MASTER:

On a PRP Redbox and PTP boundary clock Cisco IE 5000 switch, the PTP state on PRP member ports can be checked as follows. The active port state should be SLAVE and the standby port state should be PASSIVE_SLAVE. If the active port fails, the standby port changes to SLAVE state:

Recommended Practices

■Disable PTP on interfaces where PTP is not necessary.

■Configure peer-to-peer transparent mode for PTP transparent clocks to reduce jitter and delay accumulation of PTP packets.

■Configure the switch to process a non-compliant PTP grandmaster’s announce messages without Organization_extension and Alternate_timescale TLVs using the following command:

■In interoperability scenarios, it is best to use default PTP domain value, which as per C37.238:2011 standard is 0 (zero). The default PTP domain value on IE switches is set to 0 (zero). It can also be configured using the following command:

Configuring PTP Over PRP Using the Device Manager

The task of configuring PTP over PRP using the embedded device manager is no different than configuring each of the features independently. Refer to:

■Configuring GNSS and GPS on Cisco IE 5000 Using Device Manager

■Configuring PRP RedBox Using Device Manager

Cisco IE 5000 GNSS and GPS

Cisco Industrial Ethernet switches are capable of accurate time distribution using PTP or IRIG-B, but previously relied on an external source to provide accurate time. The Cisco IE 5000 switch has a built-in GNSS receiver that helps enable the switch to determine its own location and get accurate time from a satellite constellation. The switch can become the PTP grandmaster clock for time distribution in the network.

Configuring GNSS and GPS on Cisco IE 5000 Using CLI

Before you begin, note:

■GNSS is supported only on Cisco IE 5000 switches with SKUs that have Version ID (VID) v05 or higher and GNSS firmware version 1.04 or higher. Verify using show version output:

■GNSS is available for all Cisco IOS software feature sets for Cisco IE switches (lanbase, ipservices) and does not require a separate license.

■GNSS can be used as time source for PTP default and power profiles only.

■GNSS can be used as time source for PTP in GMC-BC mode only.

Summary of Steps

1. Enter global configuration mode:

2. Enable GNSS:

3. Configure the switch for grandmaster-boundary clock mode:

Verifying GNSS and GPS on Cisco IE 5000

Use the following commands to verify GNSS operation on Cisco IE 5000:

Configuring GNSS and GPS on Cisco IE 5000 Using Device Manager

This section assumes that the Cisco IE switch has been installed and pre-configured with an IP address for remote access. For more details on setting up a Cisco IE switch, refer to the Cisco IE switch installation guides.

1. Log in to the switch using Device Manager credentials, as shown in Figure 30.

Figure 30 Device Manager Login Screen

2. After a successful login, the Dashboard for the switch loads, as shown in Figure 31.

Figure 31 Cisco IE Switch Dashboard in Device Manager

3. Configure PTP by selecting the PTP option under the Configure tab as highlighted in Figure 32.

Figure 32 Navigating PTP Settings in Device Manager

4. Select the PTP Power profile as highlighted in Figure 33.

Figure 33 Selecting PTP Power Profile in Device Manager

5. Select NTP-PTP mode if the Cisco IE 5000 needs to be configured as a PTP grandmaster. The options can be selected as highlighted in Figure 34.

Figure 34 Selecting PTP Mode in Device Manager

Note: In case of other Cisco IE switches like the Cisco IE 4000 and Cisco IE 4010, the PTP grandmaster function is not supported. Choose an appropriate operating mode according to your requirements.

Verifying QoS in the Substation LAN

Use the following commands to verify QoS operations:

For additional details on QoS deployment methodology and specific examples, see Enterprise QoS Solution Reference Network Design Guide :
http://www.cisco.com/en/US/docs/solutions/Enterprise/WAN_and_MAN/QoS_SRND/QoSPref.html