Deploying Parallel Redundancy Protocol within a Converged Plantwide Ethernet Architecture CVD

Initial Configuration

Before configuring IES features for the PRP network, switches should be configured according to general recommendations and best practices for IACS networks:

• Apply initial configuration using web-based Express Setup procedure (recommended), command line Interface (CLI) using serial console connection, or by transferring the configuration file to the Secure Digital (SD) card.
• Make sure that all IES in the PRP network are assigned unique management IP addresses.
• Create necessary VLANs, including the Native VLAN, per network segmentation requirements
• Configure switch ports according to their function using Smartport roles. Smartports optimize switch port configuration according to the type of device connected to the port.
• Configure NTP servers on each IES to make sure that network events are logged with accurate timestamps.
• Configure network protocols, security and management settings on the switch as appropriate per your company’s policy and standards.
• It is recommended to apply the latest version of IES firmware. Configuration steps in this section assume the IOS version 15.2(8)E or later, and IOS XE version 17.07.01 or later.

Infrastructure IES Configuration

Configuration of infrastructure switches in LAN A and LAN B depends on the chosen topology and resiliency protocol in the LAN (if applicable). For example, infrastructure switches may need to be configured first for DLR, Spanning Tree, REP, EtherChannel and so on. Refer to the Stratix switch user manuals, corresponding application guides and CPwE design guides for more information (see Appendix A).

Next steps describe required or recommended settings for infrastructure IES that are specific to the PRP operation.

Step 1 Configure System Maximum Transmission Unit (MTU) size to 1506 bytes or greater.

Note After submitting the System MTU change, the Allen-Bradley 5700, 5400 and 5410 switch will restart to apply the change. The Stratix 5800 switches do not require restart.

Step 2 If the infrastructure switch connects to a RedBox IES using VLAN trunking (Smartport Switch for Automation), configure PortFast Trunk mode on the ports connected to the RedBox.

PTP (CIP Sync) Configuration

If time synchronization with PTP (CIP Sync) is required for multiple VLANs in the network, it is recommended to use infrastructure IES that supports PTP in the boundary clock (BC) mode. Configure PTP settings as follows:

Step 3 Configure PTP Boundary Clock mode.

Step 4 Set PTP Priority1 as 255 (lowest priority).

Step 5 Configure on all PTP-enabled ports:

a. Announce Interval as 0 (equals 1 second in base 2 logarithmic scale)

b. Sync Fault Limit as 10,000 (nanoseconds)

Step 6 Configure PTP properties using CLI:

Note If IES in end-to-end transparent clock or forward mode are used in the topology, PTP can only be implemented in a single VLAN. In this case, the PTP-enabled VLAN must be specified on the upstream ports of the closest BC switch.

IGMP Snooping Configuration

Multicast management with IGMP Snooping is recommended if multicast IACS data is present, for example when ControlLogix Redundancy or CIP Sync is used.

Step 7 Disable IGMP Querier on the infrastructure IES. Leave IGMP Snooping enabled for all VLANs.

Step 8 Enable Extended Flood option with the default value of 10 seconds (not applicable to Stratix 5800).

Step 9 Configure static mrouter on all ports in the possible path to the Layer 3 RedBox IES with HSRP (IGMP queriers) for every VLAN that has multicast traffic. This step is necessary to help prevent multicast loss in case of the querier change and recovery, e.g., after the HSRP failover. This is CLI only configuration.

Note In a star or linear topology, configure uplink ports to the aggregation or Layer 3 RedBox IES as static mrouter ports. In a ring topology (REP or DLR), configure both ports in the ring as static mrouter ports.

The above recommendations assume that Layer 3 RedBox IES with HSRP are configured with lowest physical IP addresses in each VLAN and take the querier role in the election process.

RedBox IES Configuration—Layer 2 (Access)

The next steps describe required or recommended settings for RedBox IES that are specific to the PRP operation. These steps apply to the access layer RedBox IES (Layer 2 switches) with a single PRP channel.

PRP Channel Configuration

Step 1 Configure ports that will be in the PRP channel group 1 for VLAN trunking using the Switch for Automation Smartport template. The applicable ports are shown in Table 3-1 .

Step 2 Configure PortFast Trunk mode for ports that will be in the PRP channel.

Step 3 Add the PRP Channel Group 1 with these settings:

a. Administrative mode: Trunk

b. STP PortFast Edge: Enabled (CLI only on Stratix 5800)

c. Native VLAN and Allowed VLAN list should match exactly the settings on the individual ports

The CLI command to enable PortFast Trunk mode on the PRP channel group is:

Step 4 Configure PRP Supervision Frame Option as VLAN Tagged and select VLAN ID that is used for VDAN devices.

Note A RedBox IEs can also be connected to the infrastructure with PRP ports and the PRP channel in the access mode (single VLAN, Smartport Multiport Automation Device). In this case, the management interface of the RedBox and all VDANs are assigned to the same VLAN and IP subnet. The access mode for RedBox IES is out of scope for CPwE PRP.

Step 5 If PRP-enabled ports are using fiber media, Unidirectional Link Detection (UDLD) on the ports must be disabled. UDLD is not supported with PRP and will cause fiber ports to go to error-disable mode.

a. UDLD is automatically disabled on PRP ports starting with IOS 15.2(8)E and IOS XE 17.x

b. For earlier versions, use CLI configuration:

Note After configuring a PRP channel group, do not change settings for individual ports in the PRP channel, such as switchport mode (access or trunk) or VLAN ID. Doing so may cause the port to be suspended.

Note If a PRP group channel is deleted, physical ports in the group will be administratively shutdown to help prevent unintentional loops in the network. After adding the PRP channel back, the ports will be enabled again.

Note Adding a second PRP channel group to a Layer 2 RedBox IES (Stratix 5410 or Stratix 5800) is possible but out of scope for CPwE PRP.

PTP (CIP Sync) Configuration

If time synchronization with PTP (CIP Sync) is required, configure PTP settings as follows:

Step 6 Configure PTP Boundary mode.

Step 7 Configure Priority1 value as 10 (or any value higher than the GM priority but lower than default 128). Configure Priority2 value as 1

Step 8 Configure on all PTP-enabled ports:

a. Announce Interval as 0

b. Sync Fault Limit as 10,000 (nanoseconds)

Step 9 Configure PTP properties using CLI:

RedBox IES Configuration—Layer 3 (HSRP)

The next steps describe required or recommended settings for the distribution layer RedBox IES (Layer 3 switches with HSRP) in the CPwE PRP architecture.

PRP Channel Configuration

The PRP channel configuration is almost identical to the Layer 2 RedBox steps. The only difference is that a second PRP channel can potentially be used in a large architecture with Stratix 5410 or Stratix 5800 RedBox IES (one pair of HSRP switches connected to two separate PRP Cell/Area Zones).

Step 1 Configure ports that will be in the PRP channel for VLAN trunking using the Switch for Automation Smartport template. The allowed ports are shown in Table 3-2.

Table 3-2 PRP Channel Ports

Make sure that ports have exactly the same settings such as port speed, trunk mode, native VLAN, list of allowed VLANs and so on.

Step 2 Configure PortFast Trunk mode for ports that will be in the PRP channel.

Step 3 Configure PRP Channel Group 1 or 2 with these settings:

a. Administrative mode: Trunk

b. STP PortFast Edge: Enabled (CLI only on Stratix 5800)

c. IGMP General Query: Enabled

d. Native VLAN and Allowed VLAN list should be the same as settings on the individual ports

The CLI command to enable Portfast Trunk mode on the PRP channel group is:

Step 4 Configure PRP Supervision Frame Option as VLAN Tagged and select one of the VLAN ID configured for IACS.

Step 5 If PRP-enabled ports are using fiber media, disable Unidirectional Link Detection (UDLD) on the ports. UDLD is not supported with PRP and will cause fiber ports to go to err-disable mode.

a. UDLD is automatically disabled on PRP ports starting with IOS 15.2(8)E and IOS XE 17.x

b. For earlier versions, use CLI configuration:

HSRP Configuration

Hot Standby Routing Protocol (HSRP) is enabled and configured on Layer 3 RedBox IES for each VLAN in the PRP-enabled Cell/Area Zone. This section describes how to configure HSRP features to achieve optimum performance and fast convergence for routed traffic.

Note HSRP feature is only available in the Layer 3 firmware type on Stratix 5400 switches (catalog numbers ending with -R) and Stratix 5410 switches (catalog numbers ending with -R, -RDC, and -RAC).

Note Stratix Device Manager configuration for HSRP is available starting with IOS 15.2(8)E2 on Stratix 5410 and Stratix 5400 IES

• HSRP commands are applied to the Switch Virtual Interface (SVI) of the VLANs.
• HSRP is enabled by configuring an instance, specified by an ID value, and the virtual IP that will be shared between the HSRP peers. The virtual IP will be used as the default gateway address for hosts in the PRP VLAN.
• The primary HSRP peer should be configured with the lower physical IP address so that it will win elections for protocols that do not rely on the virtual IP, such as IGMP. The secondary HSRP peer is typically assigned the next IP address in the subnet.
• The desired active peer should be configured with a higher HSRP priority so that it consistently wins the election.
• HSRP timers (hello and hold timers) should be decreased from default values to provide sub-second protocol convergence.
• HSRP preemption should be disabled. As a result, when the active HSRP RedBox IES reboots, it assumes the standby HSRP role, which minimizes routing convergence.
• The HSRP process should be delayed on startup to help prevent a new HSRP peer from assuming too quickly that it is the only peer in the network and taking on the active role.

Step 6 Configure each SVI on the primary Layer 3 RedBox IES for HSRP as follows:

a. Standby (virtual IP address): the lowest address in the subnet (typically x.x.x.1)

b. Physical IP address: second lowest in the subnet (typically x.x.x.2)

c. HSRP version 2

d. HSRP hello timer: 200 milliseconds

e. HSRP hold time: 750 milliseconds

f. Delay timers: Minimum 30 seconds, reload 60 seconds

g. Priority: 150 (or any value higher than default 100)

h. Preempt: Disabled

This is a typical CLI configuration on the primary HSRP switch for an SVI (VLAN ID and IP addresses are examples only):

Step 7 Configure each SVI on the secondary Layer 3 RedBox IES for HSRP as follows:

a. Standby (virtual IP address): the lowest address in the subnet (typically x.x.x.1)

b. Physical IP address: third lowest in the subnet (typically x.x.x.3)

c. HSRP version 2

d. HSRP hello timer: 200 milliseconds

e. HSRP hold time: 750 milliseconds

f. Delay timers: Minimum 30 seconds, reload 60 seconds

g. Priority: default 100 (should be lower than the primary HSRP switch)

h. Preempt: Disabled

This is a typical CLI configuration on the secondary HSRP switch for an SVI (VLAN ID and IP addresses are examples only):

Layer 3 EtherChannel Configuration

For additional resiliency, Layer 3 RedBox IES should be connected to the distribution switch layer and to each other with Layer 3 (routed) EtherChannel links. Note that Layer 2 connections are not allowed between the RedBoxes except for the PRP channel ports.

Each Layer 3 RedBox IES is configured with two Layer 3 EtherChannels: one for the uplink connection to the distribution switch, and another for a peer connection to the other Layer 3 RedBox IES.

Step 8 Configure ports that will be part of the Layer 3 EtherChannel groups as routed ports (No IP Address) in the port settings.

Step 9 Configure two EtherChannel groups using previously configured routed ports. LACP Active mode is recommended. The channel mode should be compatible with the mode on the connected switch.

Step 10 Configure IP address for each routed EtherChannel port according to the IP scheme in the routed network.

Step 11 Verify that EtherChannel status is up on both ends of the channel and ports are not suspended after connecting ports.

EIGRP Configuration

The following steps are provided only as an example of the EIGRP configuration that was used for the CPwE PRP testing. Note that routing protocol configuration can be very specific to the network environment and EIGRP parameters in your environment may be different.

Other routing protocols such as OSPF can be implemented but are out of scope for CPwE PRP.

Note Dynamic routing protocols like EIGRP or OSPF are only available in the Layer 3 firmware type on Stratix 5400, Stratix 5800 and Stratix 5410 IES (catalog numbers ending with -R, -RDC, or -RAC).

The following steps apply to both Layer 3 RedBox IES:

Step 12 Enable routing on the switch.

Step 13 Configure the EIGRP instance on the switch. In most cases, default settings are sufficient.

Step 14 Add network addresses and wildcard masks for IP subnets that should be routed by EIGRP. The network range should include all IP subnets associated with the PRP VLANs and IP sub-nets configured for all routed ports.

Step 15 As best practice, suppress routing updates (EIGRP Passive mode) on all ports not participating in EIGRP. For example, passive mode should be enabled on the PRP channel ports.

Step 16 It is recommended to redistribute the default route information using EIGRP from the core/distribution layer. Alternatively, a static default route to the distribution switch can be configured.

IGMP Snooping Configuration

The following configuration steps are recommended for the distribution RedBox IES with the IGMP snooping querier role. In the CPwE PRP architecture, distribution IES (active and standby HSRP gateway) should be assigned the lowest IP addresses in each PRP VLAN to win the querier election.

Step 17 Enable IGMP Snooping for PRP VLANs where multicast traffic management is necessary. Enable IGMP Querier.

Step 18 Enable Extended Flood option with the default value of 10 seconds (not applicable to Stratix 5800).

PTP (CIP Sync) Configuration - Boundary Clock

This section describes steps for configuring Layer 3 RedBox IES in the recommended architecture for plant-wide or site-wide time synchronization with PTP (see Figure 2-27). In this architecture, the Grandmaster clock is connected in the distribution layer and switches in the PRP Cell/Area Zone are configured as boundary clocks.

Step 19 Configure PTP Boundary mode.

Step 20 Configure Priority1 value as 10 (or any value higher than the GM priority but lower than default 128). Configure Priority2 value as 1.

Step 21 Configure on all PTP-enabled ports:

a. Announce Interval as 0

b. Sync Fault Limit as 10,000 (nanoseconds)

Step 22 Configure PTP properties using CLI:

PTP (CIP Sync) Configuration - NTP/PTP mode

The following steps are required only if the Layer 3 RedBox IES are primary and backup Grandmaster clocks (NTP/PTP mode) for the PTP-enabled VLANs in the network. In this case, switches use NTP time source in the plant-wide or site-wide network to distribute time in the PTP-enabled VLANs.

For recommendations on deploying and selecting NTP servers for CPwE, refer to:

• Deploying Scalable Time Distribution within a Converged Plantwide Ethernet Architecture https://literature.rockwellautomation.com/idc/groups/literature/documents/td/enet-td016_-en-p.pdf

Steps 23-29 below replace steps 19-22 in the previous section if NTP/PTP mode is used for Layer 3 RedBox IES.

Step 23 Configure at least 2 NTP servers on each IES. Three or more NTP servers are recommended to be able to detect and reject bad clock sources. Verify that the switches are successfully synchronized to the NTP source.

Step 24 Configure both Layer 3 RedBox IES (the active and standby HSRP gateway roles) in the NTP-PTP Clock mode.

Step 25 Configure PTP priorities:

a. Configure the first switch with Priority1 value 1 and Priority2 value 1 (primary Grandmaster)

b. Configure the second switch with Priority1 value 1 and Priority2 value 2 (secondary Grandmaster).

Step 26 Verify that PTP is enabled on the PRP channel ports and on the peer link EtherChannel.

Step 27 Disable PTP on the uplinks to the distribution switch (Layer 3 EtherChannel ports).

Step 28 Configure on all PTP-enabled ports:

a) Announce Interval as 0

b) Sync Fault Limit as 10,000 (nanoseconds)

Step 29 Configure PTP properties using CLI:

Distribution Switch Configuration

This section describes settings for the Cisco Catalyst distribution stack that are relevant to the CPwE PRP architecture. Common configuration steps such as basic setup, configuring routed interfaces, routing protocols and EtherChannels on a Catalyst 9000 series switch, are not covered in this guide.

• For more information on configuring Cisco Catalyst 9300 platform, refer to the latest guide: https://www.cisco.com/c/en/us/support/switches/catalyst-9300-series-switches/products-installation-and-configuration-guides-list.html

PTP Configuration

This configuration assumes that PTP GMs (primary and secondary) are connected to the distribution stack.

Upgrade the switch stack to IOS XE 17.07.01 or later. Network Advantage license type is needed for PTP support.

Step 1 Configure PTP in the boundary mode using CLI:

Step 2 Disable PTP on ports where PTP time distribution is not needed, for example on the uplinks to the core layer.

Step 3 Configure on all PTP-enabled ports:

Step 4 Connect primary and secondary GM devices to different switches in the stack. Verify PTP status on the distribution stack and in the rest of the network.

PTP Grandmaster Configuration

The recommended CPwE PRP architecture with PTP (CIP Sync) uses redundant time modules in the distribution layer as primary/secondary GM and primary/secondary NTP servers, with GPS as the reference clock.

The following steps apply to Aparian A-TSM modules that have been used in the CPwE PRP testing.

Step 1 Select Time Source as GPS / PTP. Enable PTP and NTP time services.

Step 2 Set CIP Sync Priority1 on both time modules to 1.

Step 3 Set CIP Sync Priority2 to 1 on the primary module, and 2 on the secondary module.

Step 4 Verify GPS status on the time module and that it displays the correct UTC time.

If GPS signal is not available at the location, select NTP/PTP as the time source, and configure two NTP server IP addresses for redundancy.