Topology Diagram

This section describes the high-level solution validation topology that has been used in this Feeder Automation Implementation Guide. Figure 3 depicts the high-level solution validation topology.

Figure 3 Cisco DA Feeder Automation Solution Validation Topology

The multiple layers of topology include:

■The Headend or Control Center Block, which hosts the DSO Control Center, includes:

–DA application servers (for example, SCADA application server):

• They could also host other application servers.

–Network Operations Center (NOC), which hosts the following headend components:

• Certificate Authority (RSA encryption), Dynamic Host Configuration Protocol (DHCP), Field Network Director (FND), FND Database, Authentication Authorization and Accounting (AAA), Active Directory (AD), Certificate Authority (ECC encryption), Fog Director (FD), Registration Authority (RA), Tunnel Provisioning Server (TPS), and Cluster of Headend Routers.
• These components are essential for the ZTD of the Cisco IOS Routers, which could be DA Gateways (IR1101, IR807, IR800) that are positioned along the Distribution Feeder or CGR1000 series of routers positioned as FARs.

–Headend block, which includes:

• Private network, where the protected part of the headend is located, along with SCADA and other application servers.
• DMZ network, where the exposed part of the headend is located; it includes TPS, RA, and HER Cluster.

■The WAN Block commonly refers to the public Internet over Ethernet/cellular backhaul. It could also be a private IP network.

■The Distribution Block, which comprises the following three major sub-blocks:

–Cisco Cellular DA Gateways, which refer to Cisco IOS Routers like IR1100, IR807, and IR809.

–Cisco Field Area Routers, which refer to Cisco IOS Routers like CGR1240 and CGR1120. These routers are used for aggregating the Cisco Resilient Mesh Endpoints (also referred as CR Mesh DA Gateways). The NAN Block is a subset of the Distribution Block, comprising CR Mesh devices, including Cisco FAR and CR Mesh endpoints.

–Cisco Resilient (CR) Mesh DA Gateways with Edge Compute, which refer to the Cisco IR510 WPAN Industrial Router.

■The Utility Controller Devices Block, in which the Utility controller devices (real/simulated) are connected to the Cisco DA Gateways (Cellular DA Gateway or Mesh DA Gateway) over an Ethernet/Serial interface. The following components are simulated using the Triangle Micro Works (Distributed Test Manager or DTM) tool:

–SCADA Master located in DSO Control Center

–IEDs located in the Utility Controller Devices Block layer

■The NAN Block, which is comprised of three Personal Area Networks (PANs):

–CR Mesh—PAN1

–CR Mesh—PAN2

–CR Mesh—PAN3

PAN3 has been validated over LTE backhaul. PAN1 and PAN2 have been validated over Ethernet backhaul. Cisco IOx Edge Compute functionality has been validated over PAN2. Fog Director (FD) located in the DSO control center has been used for the lifecycle management of Edge compute applications on the IOx platform of CR Mesh DA Gateway.

For implementation involving dual control scenarios, please refer to the Distribution Automation - Feeder Automation Implementation Guide.

IPv4 and IPv6 Addressing

This section, which provides detail about the addressing used at every layer of the Figure 1 Cisco DA Feeder Automation solution validation topology, includes the following sections:

■Addressing in the DSO Control Center Block

■Addressing in the WAN Block

■Addressing in the Distribution Block

■Addressing in the Utility Controller Devices Block

Addressing in the DSO Control Center Block

Figure 4 captures the granular details of the DSO Control Center.

Figure 4 DSO Control Center Block—Zoom In

The DSO Control Center is comprised of two types of network: the Private Network and the DMZ Network

■The Private Network hosts an UCS server (with all the required head end components like FND, Certificate Authority, DHCP server, and so on), SCADA Master as well as Fog Director. Private Network leverages the Cisco NTP for time synchronization, as well as Cisco DNS servers for name resolution.

■The DMZ Network hosts a cluster of Headend Routers (ASR 1000), TPS, and Registration Authority. These components connect to the DMZ Network on one side and the Private Network on the other side.

For more details about implementing the headend in the DSO Control Center, please refer to the Cisco FAN-Headend Deep Dive Implementation and FAN Use Cases Guide.

Addressing in the Private Network

Table 1 captures the addressing details of the components located in the private network of DSO Control Center.

Addressing in the DMZ Network

The previous topology in Figure 4 shows that components that are located in the DMZ Network (reachable over WAN) include the following:

■Registration Authority (RA)

■Tunnel Provisioning Server (TPS)

■HER Cluster of ASR 1000 series of routers

Table 2 captures the addressing details of the components located in the DMZ network of DSO Control Center.

Note: The Virtual IP for FAN-PHE-HER1, FAN-PHE-HER2, and FAN-PHE-HER3 is 10.10.100.100.

Addressing in the WAN Block

The Public IP WAN has been validated in this implementation guide. Addressing in the WAN block is typically service provider managed. As long as the Cisco FARs or Cisco Cellular IoT Gateways in the Distribution Block receive a dynamically-assigned IP address from the service provider and are able to reach the components in the DMZ network, the requirement would be met.

Addressing in the Distribution Block

Addressing in the Distribution blocks is discussed granularly in the following sections:

■Addressing used in Cisco Cellular DA Gateways

■Addressing used in Cisco Field Area Routers

■Addressing used in Cisco Resilient Mesh DA Gateways

Addressing used in Cisco Cellular DA Gateways

Figure 5 captures the various interfaces on the Cisco Cellular DA Gateways that are involved in the solution.

Figure 5 Addressing used in Cisco Cellular DA Gateways 256396

Table 3 captures the addressing used in Cisco Cellular DA Gateways.

Note: Some Cisco FAR devices available are CGR1120, CGR1240, IR1101 and IR807.

Addressing used in Cisco Field Area Routers

Figure 6 captures the various interfaces on the Cisco FARs that are involved in the solution.

Figure 6 Addressing used in Cisco Field Area Routers

Table 4 captures the various interfaces used in the Cisco FAR and its associated addressing.

Addressing used in Cisco Resilient Mesh DA Gateways

Figure 7 captures the various interfaces on the Cisco Resilient Mesh DA Gateways that are used in this solution.

Figure 7 Addressing used in Cisco Resilient Mesh DA Gateways

IR510 receives the IPv6 address for the LoWPAN interface from CGR. The IPv6 address of IR510 LoWPAN interface and the CGR WPAN interface are on the same IPv6 subnet. The CGR would serve as the default gateway for IR510.

Table 5 captures the various interfaces used in the CR Mesh DA Gateway and its associated addressing.

Addressing in the Utility Controller Devices Block

The Ethernet-based Utility Controller devices is to be configured with 192.168.0.3. It can be connected to the Ethernet ports of the Cisco Cellular DA Gateway or the CR Mesh DA Gateway. In this implementation, controller devices were simulated using Triangle Micro Works (Distributed Test Manager) tool. This simulated controller device is configured with 192.168.0.3 during this validation.

Topology Diagram for FLISR

This Linear and Aggregated Mesh topology constructed using RF coax cables, power splitters and attenuators, enabling signal variations to construct a 10-hop linear and 23 nodes aggregated mesh network. In mesh network nodes that can hear each other, in that the RSSI (Reverse Signal Strength Indication) is within the acceptable range for a specific modulation (OFDM) fixed modulation and data rate established between parent, child, and neighbor nodes.

The RF connectivity between the DA gateways designed for IEEE 802.15.4 Option 2 (OFDM fixed modulation PHY mode149 on Cisco Resilient Mesh) which corresponds to a physical layer data rate of 800kbps. The OFDM 800kbps maximum Receive Signal Strength Indicator (RSSI) is -101db. To avoid node flapping and instability in the network a new node joining the mesh network for the first time must have minimum RSSI of -91db with respect to its neighbor. So, for a best practice design rule that the link between DA devices is designed the average link RSSI range between -70db to -90db.

The mesh radio parameter configured using IEEE 802.15.4g and Routing Protocol for Low Power and Lossy Networks (RPL) timers. Mesh is also configured to operate in Storing Mode to support peer to peer communication.

This section describes the solution validation topology that has been used in this DA 2.0 FLISR Implementation Guide.

Linear Mesh lab topology for FLISR

In linear topology each node has two neighbors, one parent from upper rank close to CGR and one child from lower rank. The RSSI also designed for same RSSI range as showing in the topology. On lower ranks, as the hop counts increase, the latency values also increase due to each node adds its own processing delays. So the end to end, i.e. each hop to control center path delay will be longer.

Figure 8 depicts the DA 2.0 Linear Mesh Lab Topology.

Figure 8 Linear Mesh lab topology diagram

In the linear topology, fixed and variable attenuators are added to achieve an RSSI range of -70 to -90dB. RF Splitters are added at appropriate RF links, as shown in above lab topology figure, for creating a linear CR mesh.

Each SEL-3505 RTAC is connected to each IR510 device via Ethernet connection. SEL-3530 RTAC, which act as a SCADA Master and DAC Controller is located in Control Center.

Refer to Addressing in the DSO Control Center Block section for the Control Center details.

Aggregated Mesh lab topology for FLISR

In aggregate topology the distance between DA Grid device is shorter and nodes can aggregate traffic from multiple children. The ratio of child to parent is higher and the parent available bandwidth is shared among the children. To simulate this network the 2nd, 3rd, and 4th rank nodes were designed to establish physical layer 1 connection with first node of parent rank. The aggregation topology can be designed in multiple way to select their parent, limitations are applied due to lab environment and worst conditions. Refer to the topology for this implementation.

Note: This implementation is purely based on the topology provided in this section.

Figure 9 depicts the DA 2.0 Aggregated Mesh Lab Topology.

Figure 9 Aggregate Mesh lab topology diagram

In the aggregate topology, fixed and variable attenuators are added to achieve an RSSI range of -70 to -90dB. RF Splitters are added at appropriate RF links, as shown in above lab topology figure, for creating a linear CR mesh.

Each SEL-3505 RTAC is connected to each IR510 device via ethernet connection. SEL-3530 RTAC, which act as a SCADA Master and DAC Controller is located in Control Center.

Refer to DSO Control Center Block section for the Control Center details.

IPv4 and IPv6 Addressing

For general and complete IPv4 and IPv6 addressing please refer to the “Solution Network Topology and Addressing” section in this document. The specific FLISR configurations are shown below.

Table 6 Additional components for Field Block for FLISR

CGR 1240 Configuration

Please refer to Zero Touch Enrollment of Cisco Resilient Mesh Endpoints for IR510 device.

Tunnel Provisioning Server/Field Network Director Categories

Bootstrapping TPS/FND

The TPS/FND located in the staging/bootstrapping environment that helps with PnP bootstrapping of the IoT Gateways are referred to as the bootstrapping TPS and bootstrapping FND.

Network Operating Center

The TPS/FND located in the NOC/Control Center environment that helps with ZTD of IoT Gateways is referred to as the NOC or Control Center TPS/FND. This TPS/FND located in the DSO Control Center helps with management of the IoT Gateways.

Note: The bootstrapping TPS/FND could be the same as or different from the NOC TPS/FND depending on the chosen approach.

Since Approach 1 is chosen for implementation in this guide, two different pairs of TPS/FND have been implemented:

■Bootstrapping TPS/FND

■NOC TPS/FND

For general implementation of TPS/FND, please refer to the detailed steps covered in the following sections of the Cisco FAN-Headend Deep Dive Implementation and FAN Use Cases Guide:

■Implementing Tunnel Provisioning Server

■Implementing Field Network Director

The Cisco IoT Field Network Director Installation Guide could also be referred to for implementation of TPS/FND.

Note: This guide focuses on the implementation details for enhancing the TPS/FND servers to also serve the functionality of Bootstrapping TPS and Bootstrapping FND.

Certificate Considerations for PnP and ZTD

Common Name and Subject Alternate Name requirements must be considered while creating certificates for the Bootstrapping TPS/FND and NOC TPS/FND. Table 7 captures the sample certificate parameter requirements of the certificate that are to be installed on the TPS/FND server.

PnP TPS and FND need to have their subject alternative name (and optionally their corresponding IP addresses) set to FQDN. Also, the Common Name must match the hostname FQDN used in the URL during a https communication from the IoT Gateways. ZTD, TPS, and FND must have Common Name entries match the hostname FQDN used in the URL during https communication from the IoT Gateways.

Note: If https communication is attempted on https://tps-san.ipg.cisco.com:9120, then the Common Name of the certificate installed on the target server must match the FQDN ( tps-san.ipg.cisco.com) accessed in the URL.

Note: If https communication is attempted on https://10.10.242.242:9120, and if the Common Name of the certificate installed on the target server only has FQDN (and not IP), the SSL connection may not establish.

Bootstrapping the IoT Gateway

Bootstrapping can also be referred to with the following terminology:

■Day 0 provisioning

■ZTD staging

■PnP staging

■Application of manufacturing configuration onto IoT Gateway

■Generation of Express Configuration

On the bootstrapping FND, import the bootstrapping csv file and then assign the IoT Gateways to the correct bootstrapping group. Bootstrapping will occur automatically when the IoT gateway is powered on.

Note: To bootstrap the IoT Gateway, in the case of Approach 1, just connect the IoT Gateway to the Ethernet PnP Staging switch, and then power it on. In the case of Approach 2, just insert the LTE SIM cards (or connect the Ethernet link) with internet access on the IoT Gateway and power it on.

Bootstrapping is achieved with the help of the Cisco Network PnP solution. This section focuses on building the infrastructure required for bootstrapping to happen. The “Cisco Network PnP - Available Methods" section of the Design Guide discusses multiple methods for PnP server discovery. Three PnP server discovery methods, which have been implemented as part of this guide, are:

■PnP server discovery through Cisco PnP Connect—validated with Approach 2

■PnP server discovery through DHCP server—validated with Approach 1

■PnP server discovery through manual PnP profile—validated with Approach 1

Preparing the Bootstrapping Infrastructure

The bootstrapping infrastructure, which involves multiple actors, is captured in Table 8.

This section is discussed in the following phases:

■Prerequisites

■Certificate Creation and Installation

■Installation of Bootstrapping TPS

■Installation of Bootstrapping FND

■Configuration of Bootstrapping TPS

■Configuration of Bootstrapping FND

Prerequisites

■The TPS and FND server must be up and running.

■This section focuses only on the incremental portions to make the regular TPS/FND a bootstrapping TPS/FND.

■Routing reachability over IPv4 and/or IPv6 networks from IoT Gateways to TPS.

■Routing reachability between TPS and FND.

Certificate Creation and Installation

This section captures the parameters that need to be considered while creating the certificate for the TPS (PnP Proxy) and FND (PnP server).

Note: For detailed instructions about certificate creation, please refer to the section “Creation of Certificate Templates and Certificates” of the Cisco FAN-Headend Deep Dive Guide.

Certificate Creation for Bootstrapping TPS

The certificate for the TPS must be created with both the Subject Name and the Subject Alternative Name fields populated.

Figure 10 TPS Certificate Parameters for PnP Bootstrapping

The Subject Name is the Common Name that must be set to the FQDN of the PnP Proxy. The Subject Alternative Name must be set to the FQDN of the PnP Proxy, along with the optional IP address. The Subject Alternative Name is required for PnP to work. The enrolled certificate is exported as PnP-TPS.pfx and is protected with a password.

Certificate Creation for Bootstrapping FND

The FND certificate must be created with both the Subject Name and Subject Alternative Name fields populated.

Figure 11 FND Certificate Parameters for PnP Bootstrapping

The Subject Name is the Common Name that must be set to the FQDN of the PnP Server. The Subject Alternative Name must be set to the FQDN of the PnP Server, along with the optional IP address. The Subject Alternative Name is required for PnP to work. The enrolled certificate is exported as PnP-FND.pfx and is protected with a password.

Installation of Bootstrapping TPS

The bootstrapping procedure in this implementation considers the use of TPS as PnP Proxy.

Note: As TPS is used in this implementation, TPS would represent itself as the PnP server for the IoT Gateways. Therefore, TPS is referred to as the PnP Proxy. For installation of TPS, please refer to the detailed steps covered under the section “Implementing Tunnel Provisioning Server” of the Cisco FAN-Headend Deep Dive Implementation and FAN Use Cases Guide.

TPS Certificate Installation on the Bootstrapping TPS

For installation of the certificate on the Bootstrapping TPS, please refer to the detailed steps covered under the section “Certificate Enrollment Phase for TPS Proxy Server” of the Cisco FAN - Headend Deep Dive Implementation and FAN Use Cases Guide.

Note: Please use PnP-TPS.pfx while enrolling the certificate on the TPS.

The following are the brief steps:

# To view the content of the "Pnp-TPS.pfx" certificate:

# To import the certificate:

Cisco SUDI Certificate Installation on the Bootstrapping TPS

Cisco SUDI CA can be installed into the cgms_keystore of TPS using the following command:

The Cisco SUDI CA file "cisco-sudi-ca.pem" can be fetched from the FND, from the following location “/opt/cgms/server/cgms/conf/ciscosudi/cisco-sudi-ca.pem”

Installation of Bootstrapping FND

For installation of FND, please refer to the detailed steps covered under the section “Implementing Field Network Director” of the Cisco FAN-Headend Deep Dive Implementation and FAN Use Cases Guide.

FND Certificate Installation on the Bootstrapping FND

For installation of the certificate on the Bootstrapping FND, please refer to the detailed steps covered under the section “Certificate Enrollment onto FND's Keystore” of the Cisco FAN Headend Deep Dive Implementation and FAN Use Cases Guide.

Note: Please use PnP-FND.pfx while enrolling the certificate on the FND.

Cisco SUDI Certificate Installation on the Bootstrapping FND

Cisco SUDI CA can be installed into the cgms_keystore of FND using the following command:

Configuration of Bootstrapping TPS

This section covers the configuration steps and the final verification steps on the TPS.

TPS Proxy Properties Configuration TPS

Proxy Properties file needs to be configured with the following details:

■inbound-bsproxy-destination: Address to which the bootstrapping requests be forwarded.

■enable-bootstrap-service: Is bootstrapping service enabled/disabled?

■bootstrap-proxy-listen-port: Port on which the PnP Proxy must be listening for processing bootstrapping requests (default port is 9125).

Name resolution entries have to be present for FND FQDN in the /etc/hosts file.

Mandatory Verification Checks on TPS Proxy

The verification checks include the following:

■FND FQDN entry in /etc/hosts.

■TPS must have three certificates installed into the cgms_keystore:

–Certificate signed by Utility PKI for TPS (with private key)

–Public Certificate of the Utility PKI CA server

–Public Certificate of the Cisco SUDI CA

■Hostname consistency with the certificate.

■There shouldn't be any unreachable name servers in /etc/resolv.conf.

■NTP daemon should be running. Time should be synchronized.

■Necessary firewall ports must have been opened up, if the firewall/iptables/ip6tables are enabled:

–TCP Port 9125 to process http communication

–TCP port 9120 to process https communication FND FQDN entry in /etc/hosts:

TPS must have three certificates installed into the cgms_keystore:

■The certificate entry 'root' represents the Utility PKI CA certificate.

■The certificate entry 'sudi' represents the Cisco SUDI CA certificate.

■The certificate entry 'cgms' represents the private certificate of the TPS server signed by the (custom) Utility PKI CA server.

Hostname should match certificate Common Name/SAN:

Note: No unreachable name servers should exist. Either the name servers should be present and reachable or they should be empty. Any unreachable name server address entry must be taken care or removed under the network interface configuration.

NTP daemon should be running. Time should be synchronized:

Note: The TPS server should be time synchronized. Otherwise, the https communication from the IoT Gateway might not reach the TPS Proxy Application.

Configuration of Bootstrapping FND

This section covers the configuration steps and the final verification steps on the FND.

CGMS Properties Configuration

The CGMS Properties file needs to be configured with the following details:

■proxy-bootstrap-ip—Address of the PnP Proxy from which the bootstrapping requests are processed

■enable-bootstrap-service—Enable/Disable the bootstrapping service

■bootstrap-fnd-alias—The trust point alias to be used during bootstrapping of the IoT Gateway

■ca-fingerprint—fingerprint of the 'root' trustpoint

Name resolution entries have to be present for TPS FQDN in the /etc/hosts file.

Mandatory Verification Checks on FND

Verification checks include the following:

■TPS FQDN entry in the /etc/hosts file.

■FND must have three certificates installed into the cgms_keystore:

–Certificate signed by Utility PKI for FND (with private key)

–Public Certificate of the Utility PKI CA server

–Public Certificate of the Cisco SUDI CA

■Hostname must be consistent with the certificate.

■No unreachable name servers in /etc/resolv.conf should exist.

■NTP daemon should be running. Time should be synchronized.

■Necessary firewall ports must have been opened up if the firewall/iptables/ip6tables are enabled:

–TCP Port 9125 to process http communication

–TCP port 9120 to process https communication

TPS/FND FQDN entry in the /etc/hosts file:

FND must have three certificates installed into the cgms_keystore:

■The certificate entry 'root' represents the Utility PKI CA certificate.

■The certificate entry 'sudi' represents the Cisco SUDI CA certificate.

■The certificate entry 'cgms' represents the private certificate of the FND server signed by the (custom) Utility PKI CA server.

Hostname should match the certificate Common Name/SAN:

Note: No unreachable name servers should exist. Either the name servers should be present and reachable or they should be empty. Any unreachable name server address entry must be taken care or removed under the network interface configuration:

NTP daemon should be running. Time should be synchronized:

Note: The FND server should be time synchronized. Otherwise, the https communication from the IoT Gateway might not reach the FND (cgms) application.

Csv File Import on FND GUI

A sample csv file that can be imported into FND for bootstrapping of IoT Gateway is shown below:

Note: Ensure that there aren’t any blank spaces while using this csv file.

Figure 12 Bootstrapping CSV Import at Bootstrapping FND

In bootstrapping FND:

1. From Devices > Field Devices, click Router in the left pane.

2. Click the Inventory tab on the middle pane.

3. Click Add Devices.

4. Browse the csv file created in the previous step.

5. Then click Add to import the IoT Gateway CSV list into the bootstrapping FND.

DHCP Server-Assisted PnP Provisioning

This section is discussed in the following phases:

■Prerequisites

■Bootstrapping in the IPv4 Network

■Bootstrapping in the IPv6 Network

■Logical Call Flow

Prerequisites

PnP Proxy must be reachable either over the LAN or over the WAN/Internet. As TPS is used in this implementation, TPS acts as the PnP server for the IoT Gateways. The DHCP server advertises TPS details in place of the PnP server details.

Bootstrapping in the IPv4 Network

This section discusses the DHCP server-assisted bootstrapping of the IoT Gateways over the IPv4 network. In Figure 13, IoT Gateways obtain the IP address dynamically from the DHCP server along with details of the PnP server (which, in this case, is actually that of PnP Proxy, as TPS is deployed).

■The PnP server details are received using DHCP option 43.

■The PnP agent (residing on the IoT Gateway) then reaches out to PnP Proxy over IPv4 LAN/WAN network over http on port 9125 and then over https on port 9120.

Figure 13 DHCP Server-Assisted Bootstrapping of IoT Gateways over IPv4 Network

Bootstrapping in the IPv6 Network

This section discusses the DHCP server-assisted bootstrapping of the IoT Gateways over the IPv6 network.

■IoT Gateways obtains the IP address dynamically from the DHCP server along with details of the PnP server (which, in this case, is actually that of PnP Proxy, as TPS is deployed).

■The PnP server details are received using DHCP option 9.

■The PnP agent (residing on the IoT Gateway) then reaches out to PnP Proxy over IPv6 LAN/WAN network over http on port 9125 and then over https on port 9120.

Logical Call Flow

This section discusses the logical call flow sequence with the DHCP server-assisted bootstrapping of the IoT Gateways over the IPv4/IPv6 network. Figure 14 shows the following actors:

■PnP Agent (IoT Gateway)

■DHCP Server

■DNS Server

■PnP Proxy (TPS)

■PnP Server (FND)

Figure 14 DHCP Server-Assisted Bootstrapping of IoT Gateways—Logical Call Flow

1. When the IoT Gateway is powered on, the PnP Agent on the IoT Gateway checks for the presence of the startup configuration. If the startup configuration is not found, then the PnP agent performs “no shut" and enables DHCP on all the interfaces.

2. The IOS on the IoT Gateway sends out a DHCP request, which reaches the DHCP server (either directly or with the help of DHCP relay agent).

3. The DHCP server responds back with the IPv4 address along with option 43, or the IPv6 address along with option 9. The option contains the FQDN of the PnP server to talk to (for example, tps-san.ipg.cisco.com) and the port number (for example, 9125) on which the PnP Proxy/Server is expected to be listening. The PnP server detail advertised as part of the DHCP option is the IP address of the PnP Proxy instead of the actual PnP server (with TPS deployed as part of the solution).

4. The IoT Gateway then sends out a name resolution request to DNS server to resolve the FQDN to its corresponding IPv4/IPv6 address.

5. The PnP Agent attempts its communication with the PnP Proxy over port 9125 (over http). PnP Proxy, in turn, communicates with the FND on port 9125. Bootstrapping begins at the FND from this point. The prerequisite to processing this bootstrapping request from the IoT Gateway is the addition of IoT Gateway details into the FND with the loading of the csv file.

6. The FND installs the trust point on the IoT Gateway.

7. The IoT Gateway sends out a Get CA Certificate request to PnP Proxy, which, in turn, proxies the communication to the FND. The FND would respond back with the CA certificate of the FND's trust point, which would then be installed on the IoT Gateway.

The following PnP States would have transitioned at the FND:

–CONFIGURING_HTTP_FOR_SUDI

–CONFIGURED_HTTP_FOR_SUDI

–CREATING_FND_TRUSTPOINT

–AUTHENTICATING_WITH_CA

–AUTHENTICATED_WITH_CA

8. From this point onwards, the further communication switches over to https on port 9120. The IoT Gateway would communicate with the TPS IP on port 9120, which, in turn, is sent to the FND IP on port 9120. The rest of the IoT Gateway bootstrapping happens over this secure https communication established on port 9120.

Note: Since the communication is over https, time synchronization and certificate parameters matching must be taken care of:

–For example, if https://<TPS_FQDN>:9120 is attempted, then the certificate installed on the TPS must have CN/SAN configured with <TPS_FQDN>.

–Similarly, if the https://<TPS_IP>:9120 is attempted, then the certificate installed on the TPS must also have CN/SAN configured with <TPS_IP>. Otherwise, SSL failure might occur and the https message from IoT Gateway might not reach the TPS Proxy Application on port 9120.

FND would transition through the following PnP states while the bootstrapping progresses:

–UPDATING_ODM

–UPDATING_ODM_VERIFY_HASH

–UPDATED_ODM

–COLLECTING_INVENTORY

–COLLECTED_INVENTORY

–VALIDATING_CONFIGURATION

–VALIDATED_CONFIGURATION

–PUSHING_BOOTSTRAP_CONFIG_FILE

–PUSHING_BOOTSTRAP_CONFIG_VERIFY_HASH

–PUSHED_BOOTSTRAP_CONFIG_FILE

–CONFIGURING_STARTUP_CONFIG

–CONFIGURED_STARTUP_CONFIG

–APPLYING_CONFIG

–APPLIED_CONFIG

–TERMINATING_BS_PROFILE

–BOOTSTRAP_DONE

9. Bootstrapping would be complete with the "BOOTSTRAP_DONE" PnP State.

Custom PnP Profile for PnP Server

This section is discussed in the following phases:

■Prerequisites

■Bootstrapping over IPv4 Network

■Bootstrapping over IPv6 Network

■Logical Call Flow

As a gateway of last resort, if dynamic ways of learning the PnP Server are not an option, an option does exist to enable learning about the PnP server with minimal manual configuration.

Manual PnP profile configuration with PnP server details:

Note: Only the PnP Server detail is manually configured. Bootstrapping and Deployment (the rest of ZTD) still happens dynamically.

Prerequisites

■The PnP server must be reachable either over the LAN or over the WAN/Internet.

■As TPS is used in this implementation, TPS acts as a PnP server for the IoT Gateways.

Bootstrapping over IPv4 Network

This section focuses on the bootstrapping of the IoT Gateways over the IPv4 network in the absence of the DHCP server, DNS server, and Cisco Cloud redirector server to provide the PnP server details. IoT Gateways are informed about the PnP server detail directly through the Cisco IOS configuration commands.

In Figure 15, the manual PnP profile configuration on the IoT Gateways lets the IoT Gateways learn about the PnP server that should be reached out to and the desired PnP port number. For example, the custom PnP profile is configured to reach out to the PnP server ( tps-san.ipg.cisco.com) over the http on port 9125.

Figure 15 Custom PnP Profile-Assisted Bootstrapping of IoT Gateways over IPv4 Network

Based on the manual PnP profile configuration on the IoT Gateways, communication is initially established with PnP Proxy on http://tps-san.ipg.cisco.com:9125. Later, the communication is established with the PnP Proxy on https://tps-san.ipg.cisco.com:9120.

Note: Only the PnP server discovery is made manual. The rest of the bootstrapping procedure is the same as the DHCP server-assisted PnP provisioning discussed above.

Bootstrapping over IPv6 Network

This section focuses on the bootstrapping of the IoT Gateways over the IPv6 network in the absence of the DHCP server, DNS server, and Cisco Cloud Redirector Server to provide the PnP server details. IoT Gateways are informed about the PnP server detail directly through the Cisco IOS configuration commands in order to enable bootstrapping of the IoT Gateways over the IPv6 network.

In Figure 16, based on the manual PNP profile configuration on the IoT Gateways, initially communication is established with the PnP Proxy on http://tps-san.ipg.cisco.com:9125. Later, the communication is established with PnP Proxy on https://tps-san.ipg.cisco.com:9120.

Name resolution happens to an IPv6 address, and the bootstrapping happens over an IPv6 network.

Note: Only the PnP server discovery is made manual. The rest of the bootstrapping procedure (PnP communication on port 9120 and 9125) is still dynamic.

Figure 16 Custom PnP Profile-Assisted Bootstrapping of IoT Gateways over IPv6 Network

Logical Call Flow

This section discusses the logical call flow sequence with the Custom PnP profile-assisted bootstrapping of the IoT Gateways over the IPv4/IPv6 network.

Figure 17 Custom PnP Profile-Assisted Bootstrapping of IoT Gateways—Logical Call Flow

In Figure 17:

■PnP server detail is learned out of the custom PnP profile, configured manually.

■The IoT Gateway reaches out to the PnP server in the configuration, which is http://tps- san.ipg.cisco.com:9125.

■The communication reaches TPS, and is then sent to FND. Bootstrapping of the IoT Gateway begins at the FND.

■The rest of the procedure is exactly the same as the bootstrapping steps discussed as part of DHCP server-assisted PnP Provisioning.

–Initial communication happens on http://tps-san.ipg.cisco.com:9125

–Later communication happens on https://tps-san.ipg.cisco.com:9120

PnP Server Discovery through Cisco PnP Connect and Bootstrapping

■Prerequisites

■Bootstrapping

■Logical Call Flow

Prerequisites

PnP Proxy must be reachable either over the WAN/Internet. As TPS is used in this implementation, TPS acts as the PnP server for the IoT Gateways. The controller profile on "software.cisco.com" should be configured with the correct TPS address. The controller profile advertises TPS details in place of the PnP server details.

To create the controller profile, login to software.cisco.com. Go to Network Plug and Play > Select controller profile from the toolbar and add the details.

Figure 18 shows the controller profile added on software.cisco.com.

Figure 18 Controller Profile

When a device is ordered through CCW, the device must be attached with the Smart account. For the PnP discovery to be successful using PnP Connect, a device must be added on the software.cisco.com portal. The device can be added either manually or by uploading a csv file. You can refer to "PnP Server Discovery Through Cisco PnP Connect" in the Cisco Distribution Automation Feeder Automation Design Guide. Figure 19 shows adding a device manually.

Figure 19 Manual Addition of Device

After manually adding the device in the PnP Connect portal, the request is yet to received from the device and the status for PnP redirection will be pending. This is shown in Figure 20.

Figure 20 PnP Redirect Pending after Manual Device Addition

Finally, when the device is added successfully, it should be populated in the devices list as shown in Figure 20, which lists the devices for when the Redirect was successful.

Bootstrapping

This section discusses the PnP Connect-Assisted bootstrapping of the IoT Gateways over the IPv4 network.

In Figure 21, IoT Gateways obtain the IP address dynamically from the service provider.

■The PnP agent (residing on the IoT Gateway) then reaches out to PnP Proxy over IPv4 LAN/WAN network over http on port 9125 and then over https on port 9120.

Figure 21 PnP Connect—Assisted Bootstrapping of IoT Gateways

Logical Call Flow

This section discusses the logical call flow sequence with the DHCP server-assisted bootstrapping of the IoT Gateways over the IPv4/IPv6 network.

The actors shown in Figure 22 are the following:

■PnP Agent (IoT Gateway)

■Service Provider

■PnP Cloud Re-direction Service PnP Connect Portal

■PnP Proxy (TPS)

■PnP Server (FND)

Figure 22 PnP Connect-Assisted Bootstrapping of IoT Gateways -Logical Call Flow

1. When the IoT Gateway is powered on, the PnP Agent on the IoT Gateway checks for the presence of the startup configuration. If the startup configuration is not found, then the PnP agent performs "no shut" on all the cellular interfaces.

2. The IOS on the IoT Gateway sends out a request to the service provider.

3. The service provider responds back with the IPv4 address.

4. The IOT gateway proceeds for PnP server discovery and connects to the PnP cloud re-direction service connect portal. After successfully connecting the server devicehelper.cisco.com, the server PnP Connect portal sends the publicly reachable TPS DMZ IP(A.B.C.D) PnP proxy IP and the port number (9125) on which the proxy server is listening. The serial number of the gateway should be added to the Cisco Cloud PnP Connect portal for the re-direction service to be successful.

5. Once the PnP discovery is successful, the PnP profile is configured on the device with the publicly reachable TPS DMZ IP. Once the profile is configured, the bootstrapping begins.

6. The rest of the procedure is exactly the same as the bootstrapping steps discussed as part of PnP server discovery through DHCP server.

Bootstrapping Configuration Template on Bootstrapping FND

The bootstrapping template is a configuration template residing on the bootstrapping FND. As part of the bootstrapping procedure, when the bootstrapping request is received from the IoT Gateway, this bootstrap configuration template is used to derive the Cisco IOS configuration, which is then pushed onto the IoT Gateway.

Once this Cisco IOS configuration is pushed onto the IoT Gateway and copied onto a running configuration successfully, the bootstrapping is said to be SUCCESSFUL.

This bootstrapping of Cisco IoT Gateways from Cisco IoT FND (PnP Server) is entirely Zero Touch. This implementation section includes the following sections:

■Creation of Bootstrap Configuration Template Group

■Router Bootstrap Configuration Groups—Populating Templates

Creation of Bootstrap Configuration Template Group

This section covers the steps required for configuring the bootstrapping group.

Figure 23 CREATE Bootstrap—CONFIG—Tunnel Provisioning

1. From the CONFIG Menu, select the Tunnel Provisioning option.

Figure 24 CREATE Bootstrap—Add Group

2. With the Router Group selected in the left pane, click the "+" sign (Add Group icon) located on the top right of the left pane.

Figure 25 CREATE Bootstrap—Add IPv4 Group

3. Configure the group name IPv4-BOOTSTRAP, and click Add.

Figure 26 CREATE Bootstrap—Add IPv6 Group

4. Similarly, configure another group name IPv6-BOOTSTRAP for bootstrapping over the IPv6 network. Click Add.

Figure 27 CREATE Bootstrap—List of Bootstrap Groups

The two newly created bootstrapping groups are displayed in the left pane:

■IPv4-BOOTSTRAP (Created to handle bootstrapping over the IPv4 network)

■IPv6-BOOTSTRAP (Created to handle bootstrapping over the IPv6 network)

Moving Devices under the Bootstrapping Group

Multiple bootstrapping groups could be configured on the bootstrapping FND. IoT Gateways have to be moved under the correct group in order to have it bootstrapped with the appropriate configuration.

Complete the following steps to move IoT Gateways under the correct bootstrapping group.

Figure 28 CHANGE Tunnel Group—Device Under Default Group

1. In Figure 28, two IoT Gateways are under the default group. The devices need to be moved to the newly created IPv4-BOOTSTRAP group. In the middle pane, select the Router in the pull-down menu, select the IoT Gateways to be moved under the new bootstrapping group, and then click Change Tunnel Group.

Figure 29 CHANGE Tunnel Group—Pull-Down Menu

2. Choose the correct bootstrap group IPv4-BOOTSTRAP. To perform bootstrapping over the IPv6 network, choose the IPv6-BOOTSTRAP tunnel group.

Figure 30 CHANGE Tunnel Group—Select IPv4 Group

3. With the appropriate bootstrap group chosen, click Change Tunnel Group to move the IoT Gateway from the default group to the desired group.

Figure 31 CHANGE Tunnel Group—Updated IPv4 Group

Device migration to the desired group was successful.

Figure 32 CHANGE Tunnel Group—Devices Moved under IPv4 Group

In Figure 30, it can be seen that IoT Gateways were moved under the correct bootstrapping group.

Router Bootstrap Configuration Groups—Populating Templates

This section shows where to populate the bootstrapping template in FND, and the template that needs to be chosen for bootstrapping of the IoT Gateways according to the network in which the IoT Gateway would be deployed (for example, IPv4/IPv6 network, located/not located behind NAT, etc).

Note: Working versions of bootstrapping templates can be found in Appendix A: PnP Profiles.

Figure 33 captures the Router Bootstrap Configuration section that needs to be populated for the purpose of bootstrapping.

Figure 33 Router Bootstrap Configuration

Every bootstrap group (referred as Tunnel Group in the left pane) can be populated with a unique Router bootstrap configuration.

With reference to Table 10, for bootstrapping the IoT Gateways for deployment over the IPv4 network:

■If IoT Gateways are located behind NAT, then the bootstrapping template IPv4- BOOTSTRAP-NAT could be used.

■If IoT Gateways are not located behind NAT, then the bootstrapping template IPv4- BOOTSTRAP could be used.

Similarly, for bootstrapping the IoT Gateways for deployment over IPv6 network:

■If IoT Gateways are located behind NAT, then the bootstrapping template IPv6- BOOTSTRAP-NAT could be used.

■If IoT Gateways are not located behind NAT, then the bootstrapping template IPv6- BOOTSTRAP could be used.

Deployment of the Cisco IoT Gateway

This section includes the following topics:

■Prerequisites for Deployment

■Deployment over IPv4 Cellular Network with NAT

■Deployment over IPv4 Network without NAT

■Deployment over Native IPv6 Ethernet Network

Prerequisites for Deployment

■Cisco IoT Gateway should have gone through the bootstrapping procedure mentioned in Bootstrapping the IoT Gateway, with the device being part of the appropriate bootstrapping group.

■Bootstrapping is said to be complete, when the Cisco IOS Routers received the bootstrapping configuration from the Bootstrapping FND.

■The bootstrapping status for the router on the Bootstrapping FND must be in ‘Bootstrapped’ state.

Deployment Infrastructure Readiness

■Cisco IoT Gateway should be assigned an IPv4/IPv6 address dynamically over Ethernet/Cellular. If a static address needs to be used on the Cisco IoT Gateway, then assignment of address to the Cisco IoT Gateway's interface needs to be taken care as part of Bootstrapping.

Tip: If any extra configuration is required to receive IP address dynamically, the delta configuration should be fed back into the bootstrapping profile, that was used to bootstrap the IoT Gateway.

■Cisco Field Area Network—Headend (DSO Control Center1) should be UP and running.

–If it needs to be set up, the Cisco FAN-Headend Deep Dive Implementation and FAN Use Cases' guide could be referenced to set up the headend in the DSO Control Center or NOC.

■All the required headend components like the CA server (RSA), AAA, AD, Registration Authority, NOC TPS/FND, DHCP server, and HERs are expected to be up and running in the DSO Control Center.

■NOC TPS, RA, and HERs must have static IP addresses configured and should be reachable from the Cisco IoT Gateways that are located along the Distribution network.

Note: If the prerequisites for deployment are addressed, ZTD of the IoT Gateways should happen successfully after the gateway is deployed at the desired location and powered on, with the Ethernet cable connected or the LTE SIM card inserted.

Deployment over IPv4 Cellular Network with NAT

Note: This section has no implementation steps. As the term "ZTD" states, it's a zero touch deployment. As long as bootstrapping happened successfully by having the IoT Gateway part of the correct bootstrapping group, this deployment should happen successfully with no manual steps.

Figure 34 captures the deployment steps for IoT Gateway over LTE Cellular.

Figure 34 Deployment over IPv4 Cellular Network

Note: This scenario has been validated with the headend located in the Cisco DMZ.

The following is the summary sequence of steps that occurs during the deployment:

1. The IoT Gateway is powered on. When up, it obtains the IP address over LTE Cellular interface.

2. The EEM Script for ZTD kicks in and waits for the time to be synchronized. Then, SCEP enrollment happens over port 80 with RA-DMZ-IP.

3. Once the certificate is received for the IoT Gateway (from the RA/CA), the ZTD script disables itself and activates the CGNA profile for tunnel provisioning (cgna initiator-profile cg-nms-tunnel).

Note: "cgna initiator-profile cg-nms-tunnel" must be used when the IoT Gateway is behind NAT, whereas "cgna profile cg-nms-tunnel" must be used when no NAT exists between IoT Gateway and TPS. This CGNA profile is configured as part of bootstrapping.

4. TPS/FND provisions the secure FlexVPN tunnel with the HER Cluster located in the DSO Control Center1.

5. As an overlay routing, FND and SCADA routes are advertised (by the HER) to the IoT Gateway through the secure FlexVPN tunnel.

6. The IoT Gateway sends out a registration request to FND on port 9121. Once registered successfully, the IOT Gateway is remotely manageable from the FND.

7. As part of the device registration with the FND, FND also pushes ICT enablement configurations to the IoT Gateway, which enables the communication between the SCADA Master in the Control Center and the SCADA Outstation located in the Feeder Automation/Distribution Network.

8. ZTD of the IoT Gateway is successful.

9. Utility Application Traffic - READY TO GO.

Deployment over IPv4 Network without NAT

Note: This section has no implementation steps. As the term "ZTD" states, it's a zero touch deployment. As long as bootstrapping happened successfully by having the IoT Gateway part of the right bootstrapping group, this deployment should happen successfully with no manual steps.

Figure 33 captures the deployment steps for IoT Gateway without NAT over the IPv4 network.

Figure 35 Deployment over IPv4 Ethernet Network

Note: This scenario has been validated with the headend located in the Engineering Lab.

The following is the summary sequence of steps that happens during the deployment:

1. The IoT Gateway is powered on. When up, it obtains the IP address over the Ethernet interface.

2. The EEM Script for ZTD kicks in and waits for the time to be synchronized. Then, SCEP enrollment happens with RA IP (172.16.241.2) on port 80.

3. Once the certificate is received for the IoT Gateway (from the RA/CA), the ZTD script disables itself, and activates the CGNA profile for tunnel provisioning (cgna profile cg-nms- tunnel).

Note: "cgna profile cg-nms-tunnel" must be used when there is no NAT between IoT Gateway and TPS. This CGNA profile has already been configured as part of IoT Gateway bootstrapping. TPS/FND provisions secure FlexVPN tunnel with the HER Cluster located in the DSO Control Center1.

4. As an overlay routing, FND (172.16.103.100 and 2001:db8:16:103::100) and SCADA (172.16.107.11 and 2001:db8:16:107::11) routes are advertised (by HER) to the IoT Gateway through the secure FlexVPN tunnel.

5. IoT Gateway sends out a registration request to FND IPv4 address 172.16.103.100 (or) IPv6 address 2001:db8:16:103::100 on port 9121. Once registered successfully, the IOT Gateway is remotely manageable from the FND.

6. As part of the device registration with the FND, FND also pushes ICT enablement configurations to the IoT Gateway, which enables the communication between SCADA Master in the Control Center and the SCADA Outstation located in the Feeder Automation/Distribution Network.

7. ZTD of the IoT Gateway is successful.

8. Utility Application Traffic - READY TO GO.

Deployment over Native IPv6 Ethernet Network

Note: This section has no implementation steps. As the term "ZTD" states, it's a zero touch deployment. As long as bootstrapping happened successfully by having the IoT Gateway part of the right bootstrapping group, this deployment should happen successfully with no manual steps.

Figure 36 captures the deployment steps for the IoT Gateway over the Native IPv6 network.

Figure 36 Deployment over Native IPv6 Ethernet Network

Note: This scenario has been validated with the headend located in the Engineering Lab over a native IPv6 network. It could be dual stack as well.

The following is the summary sequence of steps that happens during the deployment:

1. The IoT Gateway is powered on. When up, it obtains the IPv6 address over the Ethernet interface.

2. The EEM script for ZTD kicks in and waits for the time to be synchronized. Then, SCEP enrollment happens with RA IPv6 address (2001:db8:10:241::5921) on port 80.

3. IPv4 communication could be retained between RA and CA in the Control Center private network.

4. Once the certificate is received for the IoT Gateway (from the RA/CA), the ZTD script disables itself, and activates the CGNA profile for tunnel provisioning.

Note: "cgna initiator-profile cg-nms-tunnel" must be used when the IoT Gateway is behind NAT, whereas "cgna profile cg-nms-tunnel" must be used when there is no NAT between IoT Gateway and TPS. This CGNA profile has already been configured as part of the IoT Gateway bootstrapping.

5. TPS/FND provisions secure the FlexVPN tunnel with the HER Cluster located in the DSO Control Center, over the Native IPv6 network.

6. As an overlay routing, FND (172.16.103.100 and 2001:db8:16:103::100) and SCADA (172.16.107.11 and 2001:db8:16:107::11) routes are advertised (by HER) to the IoT Gateway through the secure FlexVPN tunnel.

7. IoT Gateway sends out a registration request to FND IPv4 address 172.16.103.100 (or) IPv6 address 2001:db8:16:103::100 on port 9121. Once registered successfully, IOT Gateway is remotely manageable from the FND.

8. As part of the device registration with the FND, FND also pushes ICT enablement configurations to the IoT Gateway, which enables the communication between SCADA Master in the Control Center and the SCADA Outstation located in the Feeder Automation/Distribution Network.

9. ZTD of the IoT Gateway is successful.

10. Utility Application Traffic - READY TO GO.

Tunnel Provisioning Template Profiles

Tunnel Provisioning Template profiles, which are needed for Tunnel establishment, are captured in Appendix B: FND Zero Touch Deployment Profiles.

Device Configuration Template Profiles

Device Configuration Template profiles, which are needed for ICT SCADA Traffic enablement, are captured in Appendix C: Device Configuration Profiles.

Bootstrapping and ZTD of the Cisco IoT Gateway at the Deployment Location

This section describes the bootstrapping and Deployment of the Cisco IoT gateway at the deployed location. Unlike the previous section, one TPS and FND is sufficient to complete both bootstrapping and ZTD. Although the previous two sections and this section overlap, minor changes in the implementation of TPS and FND need to be done in order for the deployment to be successful.

This section, which covers the minor changes that have to be implemented in the headend setup, describes these phases:

■Prerequisites

■Certificate Creation and Installation

■Installation of TPS

■Installation of FND

■Configuration of TPS

■Configuration of FND

■Device Bootstrapping

■Device Deployment

Prerequisites

Prerequisites include the following:

■TPS and FND server must be up and running.

■This section focuses on portions required for TPS and FND to carry out both bootstrapping and ZTD.

■Routing reachability over IPv4 and/or IPv6 networks from IoT Gateways to TPS.

■Routing reachability between TPS and FND.

Certificate Creation and Installation

This section captures the parameters that need to be considered while creating the certificate for the TPS and FND.

Note: For detailed instructions about certificate creation, please refer to the section "Creation of Certificate Templates and Certificates" of the Cisco FAN-Headend Deep Dive Implementation and FAN Use Cases Guide at the following URL:

■ https://docs.cisco.com/share/proxy/alfresco/url?docnum=EDCS-15726915

Certificate Creation for TPS

The certificate for the TPS must be created with both the Subject Name and the Subject Alternative Name fields populated.

Figure 37 TPS Certificate Parameters

The Subject Name is the Common Name that must be set to the FQDN of the TPS.

The Subject Alternative Name must be set to the FQDN - tps.ipg.cisco.com of the TPS, along with the IP address (A.B.C.D - Public reachable DMZ IP). The Subject Alternative Name is required for PnP to work. The IP address must be reachable from the IoT Gateway. TPS is located in DMZ. The IP address is not optional in this implementation. FQDN is optional, but the IP address is not.

The enrolled certificate is exported as PnP-ZTD-TPS.pfx and is protected with a password.

Certificate Creation for FND

The FND certificate must be created with both the Subject Name and Subject Alternative Name fields populated.

Figure 38 FND Certificate Parameters

The Subject Name is the Common Name that must be set to the FQDN of the PnP Server.

The Subject Alternative Name must be set to the FQDN of the FND, along with the optional IP address. The Subject Alternative Name is required for PnP to work. The IP address in Figure 38 will be reachable after tunnel is established between IoT gateway and the headend.

The enrolled certificate is exported as PnP-ZTD-FND.pfx and is protected with a password.

Installation of TPS

The bootstrapping procedure in this implementation guide considers the use of TPS as PnP Proxy. For installation of TPS, please refer to Installation of TPS.

Installation of FND

For installation of FND, please refer to the detailed steps covered under the section "Implementing Field Network Director" of the Cisco FAN-Headend Deep Dive Implementation and FAN Use Cases Guide.

Configuration of TPS

This section covers the configuration steps and the final verification steps on the TPS.

TPS Proxy Properties Configuration

TPS Proxy Properties file needs to be configured with the following details:

■inbound-bsproxy-destination: Address to which the bootstrapping requests be forwarded.

■enable-bootstrap-service: Is bootstrapping service enabled/disabled?

■bootstrap-proxy-listen-port: Port on which the PnP Proxy must be listening for processing bootstrapping requests (default port is 9125).

Name resolution entries have to be present for FND FQDN in the /etc/hosts file.

Mandatory Verification Checks on TPS Proxy

The verification checks include the following:

■FND FQDN entry in /etc/hosts.

■TPS must have three certificates installed into the cgms_keystore:

–Certificate signed by Utility PKI for TPS (with private key)

–Public Certificate of the Utility PKI CA server

–Public Certificate of the Cisco SUDI CA

■Hostname consistency with the certificate.

■There shouldn't be any unreachable name servers in /etc/resolv.conf.

■NTP daemon should be running. Time should be synchronized.

■Necessary firewall ports must have been opened up, if the firewall/iptables/ip6tables are enabled:

–TCP Port 9125 to process http communication

–TCP port 9120 to process https communication

■FND FQDN entry in /etc/hosts:

TPS must have three certificates installed into the cgms_keystore:

■The certificate entry 'root' represents the Utility PKI CA certificate.

■The certificate entry 'sudi' represents the Cisco SUDI CA certificate.

■The certificate entry 'cgms' represents the private certificate of the TPS server signed by the (custom) Utility PKI CA server.

Keytool -list -keystore /opt/cgms-tpsproxy/conf/cgms_keystore:

Hostname should match the certificate Common Name/SAN:

No unreachable name servers should exist. Either the name servers should be present and reachable or they should be empty. Any unreachable name server address entry must be taken care or removed under the network interface configuration.

NTP daemon should be running. Time should be synchronized:

Note: The TPS server should be time synchronized. Otherwise, the https communication from the IoT Gateway might not reach the TPS Proxy Application.

Configuration of FND

This section covers the configuration steps and the final verification steps on the FND.

CGMS Properties Configuration

The CGMS Properties file needs to be configured with the following details:

■proxy-bootstrap-ip: Address of the PnP Proxy from which the bootstrapping requests are processed

■enable-bootstrap-service: Enable/Disable the bootstrapping service

■bootstrap-fnd-alias: The trust point alias to be used during bootstrapping of the IoT Gateway

■ca-fingerprint: fingerprint of the 'root' trustpoint

Name resolution entries have to be present for TPS FQDN in the /etc/hosts file.

In our lab setup, the proxy-bootstrap-ip is a DMZ IP. In cases where FQDN is globally resolvable, then FQDN can be used instead of IP.

Mandatory Verification Checks on FND

Verification checks include the following:

■TPS FQDN entry in the /etc/hosts file.

■FND must have three certificates installed into the cgms_keystore:

–Certificate signed by Utility PKI for FND (with private key)

–Public Certificate of the Utility PKI CA server

–Public Certificate of the Cisco SUDI CA

■Hostname must be consistent with the certificate.

■No unreachable name servers in /etc/resolv.conf should exist.

■NTP daemon should be running. Time should be synchronized.

■Necessary firewall ports must have been opened up if the firewall/iptables/ip6tables are enabled:

–TCP Port 9125 to process http communication

–TCP port 9120 to process https communication

TPS/FND FQDN entry in the /etc/hosts file:

FND must have three certificates installed into the cgms_keystore:

■The certificate entry 'root' represents the Utility PKI CA certificate.

■The certificate entry 'sudi' represents the Cisco SUDI CA certificate.

■The certificate entry 'cgms' represents the private certificate of the FND server signed by the (custom) Utility PKI CA server.

Hostname should match the certificate Common Name/SAN:

No unreachable name servers should exist. Either the name servers should be present and reachable or they should be empty. Any unreachable name server address entry must be taken care or removed under the network interface configuration:

NTP daemon should be running. Time should be synchronized:

Note: The FND server should be time synchronized. Otherwise, the https communication from the IoT Gateway might not reach the FND (cgms) application.

Csv File Import on FND GUI

A sample csv file that can be imported into FND for bootstrapping of IoT Gateway is shown below:

Device Bootstrapping

After the above sections have been implemented, the headend is now ready for both provisioning and deployment.

The device bootstrapping is an important process as it eliminates the manual intervention to create and copy the express config to the device.

Device bootstrapping using Cisco PnP Connect has been clearly elucidated in PnP Server Discovery through Cisco PnP Connect and Bootstrapping.

Device Deployment

After the device has been successfully bootstrapped using Cisco PnP Connect, the device is now ready to undergo ZTD. No manual interface is required for the ZTD to begin.

Deployment over IPv4 Cellular Network with NAT, elucidates the ZTD process that would begin as soon as bootstrapping using Cisco PnP Connect is complete.

IoT Gateway Validation Matrix

Table 12 captures the Bootstrapping and ZTD validation matrix across the various platform types, supported as IoT Gateways.

From Table 12, Platform IR1101 has been validated for:

■Bootstrapping over IPv6 Ethernet

■ZTD over IPv6 Ethernet

Similarly, Platform IR1101 has been validated for:

■Bootstrapping over IPv4 Ethernet

■ZTD over IPv4 Ethernet/Cellular

Similarly, Platform IR807 has been validated for:

■Bootstrapping over IPv4 Ethernet

■ZTD over IPv4 Ethernet/Cellular

Similarly, platforms CGR1120 and CGR1240 have been validated for:

■Bootstrapping over IPv4 Ethernet

■ZTD over IPv4 Ethernet/Cellular

All other platform types have been validated for:

■Bootstrapping over IPv4 Ethernet

■ZTD over IPv4 Ethernet network

With this, the Cellular DA Gateways or Cisco Field Area Routers could be on boarded and registered with FND, enabling further remote management and monitoring from FND.

The next section discusses in detail the implementation steps required to onboard the Cisco Resilient Mesh Endpoints like the Cisco IR510 WPAN Industrial Router, to serve the functionality of the DA Gateway.

Staging

This section describes the implementation steps needed to bring up the CR Mesh using IR510 DA Gateways (also referred to as FDs). The IR510 connects to the CGR (also referred to as the FAR) via the Connected Grid Module (CGM) WPAN-OFDM-FCC module that needs to be installed within the FAR.

Note: For information on setting up the WPAN module, please refer to the Connected Grid Module (CGM) WPAN-OFDM-FCC Module - Cisco IOS at following URL:

■ https://www.cisco.com/c/en/us/td/docs/routers/connectedgrid/modules/cgm_wpan_ofdm/cgm_wpan_ofdm.html#pgfId-157681

Table 13 lists the basic components along with their software versions needed to bring up the CR Mesh topology depicted in Figure 1.

Certificate Creation

The prerequisites for deploying a CR Mesh include obtaining all the necessary ECC certificates from the CA server and configuring the AAA RADIUS server to authenticate the IR510 using a certificate-based authentication method. The FAR facilitates dot1x authentication between the IR510 and AAA server, thereby acting as the dot1x authenticator. The ECC certificate mentioned earlier is part of the configuration binary file (.bin) used to program the IR510 node. The ECC certificates and procedures for generating the config file for IR510 are described in further sections.

Note: While the FD need ECC CA certificates for zero touch enrollment, FAR use RSA type certificate for ZDT.

The following certificates need to be obtained from the ECC CA to program an IR510:

■The X.509 certificate of the IR510 node in PKCS#12 format (.pfx) contains its private key and is used to program the node.

■The DER-encoded X.509 certificate (.cer) of the IR510 node without the private key is used to enroll the node with the Active Directory.

■The DER-encoded X.509 certificate (.cer) of the ECC CA server is also used for programming the IR510 node.

■The CSMP certificate downloaded from the IoT FND in binary format (.cer) to validate node CSMP registration with IoT FND.

For details on setting up and configuring the ECC CA and AAA server and on obtaining all of the above certificates, please refer to Secure Onboarding of Mesh Nodes into CR Mesh.

The following section describes the process for generating a configuration binary file (.bin) used to program the IR510 node.

Bin File Creation

The configuration file for the IR510 nodes is prepared in binary format using the Configuration Writer utility (cfgwriter).

Note: To obtain the cfgwriter utility discussed below, please check with your Account team or Sales representative.

cfgwriter is a java-based utility that takes as input an XML file with the node configuration information and produces a binary (.bin) memory file. This utility may be executed on any host platform with Java Run Time Environment installed. In this deployment, a Windows 10 machine with Java pre-installed was used to host the cfgwriter utility. The node configuration information, among other items, includes the SSID of the WPAN it must join and the security certificates. The schema of the XML configuration file and the corresponding documentation are packaged with the cfgwriter utility as a ZIP file.

Figure 39 cfgwriter Utility

The following XML file is used in this deployment to program the IR510 node:

Note: In the above schema, phy mode 149 refers to OFDM modulation with a data rate of 800kb/s.

The cfgwriter utility converts the input XML file into a binary format (.bin) output. Successful execution of the cfgwriter utility with the XML file and necessary certificates as input will return a '0' numeric code to Standard Output (stdout).

From the command prompt on a Windows PC, navigate to the folder where the cfgwriter utility and all the necessary certificates described in Table 14 are placed.

The following is the command syntax used to generate the config (.bin) file needed to program the IR510 node:

java -jar cfgwriter-6.0.20.jar -x <IR510.pfx> -p <password> -ca <CAcert.cer> -w <config.xml> --nmscert <csmpcert.cer> <outputfile.bin>

The command line parameters used in the above command are explained in Table 14:

Figure 40 shows a sample command issued to generate the.bin file needed for IR510 programming.

Figure 40 Bin File Generation

Bin File Programming

The binary configuration file (.bin) prepared in the previous step, along with the correct firmware, is programmed into the IR510 node using another utility known as HostOne tool (fwubl). This tool is also placed on the same Windows machine where the cfgwriter utility was placed.

Note: To obtain the HostOne (fwubl) tool discussed below, please check with your Account team or Sales representative.

From the same Windows machine, connect to the IR510 console port using an USB to serial converter connected through a Cisco RJ45 to DB9 (female) blue serial console cable. From the command prompt on Windows PC, navigate to the folder where the fwubl tool is placed along with the firmware image and config bin files of the IR510.

Note: Do not power on the IR510 unit without any attenuators, antenna, or RF cabling in place. It is highly recommended to keep the RF port on the node always connected; don't leave it to transmit in free air since without the right connector/RF cables, the radio has a high likelihood of becoming damaged.

Once the node is powered on, issue the following command to verify that the node is in bootloader mode first. If it isn't, power cycle the node and check again as it would re-enter into the bootloader mode.

fwubl_win732bit_1.0.5.exe com<port>

The above command output would show the current bootloader version on the node besides few other parameters. Figure 41 shows the sample output of an IR510 unit initially in bootloader mode.

Figure 41 IR510 in Bootloader State

The next step is to program the firmware version on the IR510 into the memory location specified in the following command:

fwubl_win732bit_1.0.5.exe -w <IR510 firmware.bin> -a 0x8020000 com<port>

Figure 42 shows the sample output of firmware push issued to an IR510 unit.

Figure 42 Firmware Push on IR510

The next step is to program the config.bin file generated for the IR510 into the memory location specified in the following command:

fwubl_win732bit_1.0.5.exe -w <IR510 config.bin> -a 0x80E0000 com<port>

Figure 43 shows the sample output of config bin push issued to an IR510 unit:

Figure 43 Config Bin Push on IR510

The final step is to enable CR Mesh on IR510 by bringing it out of bootloader mode by issuing the following command:

fwubl_win732bit_1.0.5.exe -g 0x8020000 com<port>

Figure 44 shows the sample output to run CG-mesh software on the IR510 unit.

Figure 44 CR Mesh enabled on IR510

Secure Onboarding of Mesh Nodes into CR Mesh

Staging provided details on how to set up an IR510 node to securely join the mesh network. This section discusses the components needed to enable secure onboarding of IR510 nodes into the mesh network.

CR Mesh Endpoint - Authentication Call Flow

The FAR router provides security services such as 802.1x port-based authentication, encryption, and routing to provide a secure connection for the mesh endpoint all the way to the control center. IEEE 802.1x using X.509 certificates is the process used to securely authenticate a mesh node before allowing it to join the PAN or to even send packets into the network.

For details regarding authentication call flow using dot1x, please refer to figure "IEEE 802.1x Device Authentication" under the section "Network Security" in the Design Guide.

CR Mesh Endpoint Onboarding - Associated Touchpoints in the Headend

Table 15 lists the associated touchpoints that should be set up and configured as a prerequisite step before enabling secure onboarding process of mesh nodes.

Associated CGR Configurations for Onboarding of the Cisco WPAN Industrial Router (IR510)

Note: The following configurations are for reference purposes only. They would be dynamically provisioned by the FND as part of Zero Touch Deployment (ZTD) of CGR.

WPAN Configuration on CGR to Enable Secure Mesh

The following is the sample configuration of a CGR1240 for the WPAN interface. Please note that the SSID configured on the WPAN interface below matches what was configured in the IR510 XML schema shown in an earlier section.

AAA RADIUS Client Configuration on CGR

The following is the RADIUS client configuration needed on CGR1240 for enabling dot1x authentication of the mesh endpoint with the AAA server:

Note: The secret key above configured on the CGR must match the secret key configured on NPS when adding CGR as a radius client.

Mesh Key Configuration on CGR

As part of ZTD, the FAR is provisioned with a mesh key pushed from FND that is used to provide link layer encryption for the communication between the IR510 and the FAR.

The following command is used to verify if the key is indeed present on the CGR:

DHCPv6 Server Configuration on CGR for Address Allocation

The CR Mesh nodes need to be assigned an IPv6 address for reachability from the CGR as well as from the control center. For this purpose, a local IPv6 DHCP pool is configured on the CGR as shown below. However, a central DHCP server option, if available is recommended.

From the above mesh prefix, the first address 2001:DB8:ABCD:1::1/64 is assigned to the CGR WPAN interface while the mesh nodes are allocated an IPv6 address from the remaining pool. The sub-option 1 address specifies the IPv6 address of the IoT FND to the mesh nodes.

Note: Please refer to Appendix E: HER and CGR Configurations for the complete configuration of CGR tested to bring up the CR Mesh.

MAP-T Infrastructure in DA Feeder Automation

Basic Overview of MAP-T

MAP-T refers to address and port mapping using a translation mechanism and is used to provide connectivity to IPv4 hosts over IPv6 domains by performing double translation (IPv4 to IPv6 and vice versa) on customer edge (CE) devices and border routers.

A MAP-T domain is comprised of one or more MAP CE devices (IR510) and a border relay router (HER), all of which are connected to the same IPv6 network.

For a MAP-T domain to be operational, mapping rules known as basic mapping rules (BMR) and a default mapping rule (DMR) must be configured. While BMR is configured for the MAP IPv6 source address prefix, DMR is used to map IPv4 information to IPv6 addresses for destinations outside a MAP-T domain. Some port parameters like share-ratio and start-port are also configured for the MAP-T BMR whereas EA bits refer to the IPv4 embedded address bits within the MAP-T IPv6 address identifier of the MAP-T CPE.

For more details on MAP-T, please refer to “Mapping of Address and Port Using Translation” at the following URL:

■ https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/ipaddr_nat/configuration/15-mt/nat-15-mt-book/iadnat-mapt.pdf

Packet Flow in MAP-T network:

The following is the logical packet flow between a SCADA client and the SCADA Master:

SCADA Client --> IPv4 --> IR510 --> IPv6 --> CGR --> IPv6 --> HER --> IPv4 --> SCADA Master

An actual sample packet flow, including MAP-T parameters like BMR and DMR used in this implementation, is illustrated in Figure 45.

Figure 45 MAP-T Packet Flow

While configuring MAP-T, the DMR prefix, the IPv6 user prefix, and the IPv6 prefix plus the embedded address (EA) bits must be less than or equal to 64 bits.

Note: MAP-T parameters like the BMR IPv6 prefix and associated prefix length unique to each node are configured as part of the.csv file uploaded to IoT FND whereas the DMR IPv6 and the BMR IPv4 prefixes and their associated lengths along with EA bit length are configured via the configuration template in IoT FND which is later applied to the nodes, as shown later in Configuration Options from FND.

MAP-T Points in the Network

IR510 - MAP-T CE

A MAP-T CE device connects a user's private IPv4 address and the native IPv6 network to the IPv6-only MAP-T domain by first doing a NAT44 translation from the private to public (inside to outside) address within the v4 domain and then subsequently doing a v4 to v6 translation.

MAP-T BMR Prefix Selection for IR510.csv

The BMR prefix is used by the MAP-T CE to configure itself with an IPv4 address, an IPv4 prefix from an IPv6 prefix. As shown in Figure 45, the Rule IPv6 prefix represents the BMR IPv6 prefix used in the MAP-T network. As such, the BMR IPv6 prefix of 2001:DB8:267:1515::/56 corresponds to the MAP-T IPv4 address of 10.153.10.21 of an IR510 node.

HER - MAP-T Border Relay Router

The following configuration is needed on the HER to enable MAP-T border relay functionality:

Additionally, the CLI command nat64 enable needs to be enabled as shown below on the HER interfaces participating in the MAP-T translations (such as the interface where the SCADA Master connects and the tunnel interface towards CGR).

The HER interface connecting to the control center side where SCADA Master resides is IPv4 based whereas the virtual-template interface of the HER connecting to the CGR on the WAN side is IPv6 based, as shown logically below:

CGR --> IPv6 --> (VTI) HER (Gig port) --> IPv4 --> SCADA Master

Enabling nat64 on the SCADA Master-facing interface of the HER below:

Enabling nat64 on the FAR-facing Virtual-Template interface of HER below:

Note: For the complete running configuration of the HER, please refer to Appendix E: HER and CGR Configurations.

Configuration Options from FND

Csv File Import at FND

The following template can be used to add mesh endpoints to the FND database.

eid,deviceType,function,enduseripv6prefix,bmripv6prefixlen

The above fields are explained in Table 16:

The following are the contents of a sample csv file used in this implementation:

1. To upload the CSV file into IoT FND, navigate to the GUI.

2. From Inventory tab > Devices > Field Devices > Add Devices, click Browse to upload the file as shown in Figure 46

3. Click Add.

Figure 46 CSV File Upload to IoT FND

Once added, the devices will initially be in Unheard state. Once mesh nodes start registering with the FND, their device status turns green as shown in Figure 47.

Figure 47 Mesh Endpoint Status in FND

The nodes must register successfully with IoT FND before other settings like MAP-T, NAT44, and other serial configuration profiles be properly pushed/applied to the nodes. However, if those settings are pre-linked via the default profiles, the configuration would be automatically pushed to the nodes upon device registration.

Creation of MAP-T Group

1. To configure the MAP-T settings in FND, navigate to Config > Device Configuration.

2. Under Config Profiles and click the Add Profile icon (+).

3. Create a new MAP-T profile with the correct settings for BMR and DMR rules, as shown in Figure 48.

Figure 48 Creating a MAP-T Profile

Creation of NAT44 Group on FND

1. To configure the NAT44 settings for mesh endpoints in FND, navigate to Config Profiles > Config > Device Configuration.

2. Click the Add Profile icon (+).

3. Create a new NAT44 profile with the correct Internal IPv4 address, internal, and external ports, as shown in Figure 49.

Figure 49 Creating a NAT44 Profile

In Figure 49, the IPv4 address and prefix length of the IR510 are specified under Ethernet Settings.

The Internal IPv4 address refers to the internal address of the NAT44-configured device like the SCADA client, which is connected behind IR510. The internal port refers to the internal port number on which the SCADA client would be listening. The external port refers to the external port number of the SCADA client accessed by devices from outside MAP-T domain.

Note: Since 192.168.0.2 is reserved for the Guest OS inside the IOX portion of the IR510 unit, it is recommended to use a different address such as 192.168.0.3 for the SCADA client and, accordingly, multiple NAT44 mappings like the one shown above could be created for different ports.

Creation of Configuration Group on FND

Initially all the IR510s added to the FND are placed in the Default-IR500 group. Depending on the deployment, some of them can be moved to a newly created configuration group in which the corresponding MAP-T, NAT44 profiles can be selectively applied and a config pushed to these nodes.

1. To create a configuration group, navigate to the Groups tab > Config > Device Configuration.

2. Click the Add Group icon (+).

3. Then create a new group of type Endpoint as shown in Figure 50.

Figure 50 Creating an Endpoint Configuration Group

4. Move some of the mesh nodes from the default endpoint group to the newly created group based on the deployment.

5. Navigate to the default endpoint group, select the nodes of interest and click Change Configuration Group.

6. Then select the newly created config group in the drop-down menu as shown in Figure 51.

Figure 51 Moving IR510 to the New Configuration Group

7. Once devices are moved to the newly created configuration group, from the Edit configuration template, select the MAP-T and NAT44 profiles created earlier.

8. Click Save Changes for these settings to be applied to the devices part of this group, as shown in Figure 52.

Figure 52 Editing the Configuration Template

9. Finally, push the configuration to the devices in this group by navigating to the Push Configuration tab, selecting Push Endpoint Configuration.

10. Click Start as shown in Figure 53.

Figure 53 Push Configuration Operation

This completes the configuration settings from FND to the mesh node that are needed to operate as a DA gateway.

11. The final step is to verify that all the configuration settings are properly applied to the IR510. Click on the node inside the configuration group and navigate to the Device Info tab, as shown in Figure 54.

Figure 54 Verify Configuration Settings on IR510 (1)

12. On scrolling further down, the MAP-T settings applied to the device can be verified, as shown in Figure 55.

Figure 55 Verify Configuration Settings on IR510 (2)

Routing Advertisements from FAR to HER

Note: HER advertises a default route to all the FARs in order to provide connectivity to control center components.

Advertising Summary Route of LoWPAN Prefix

Once the CR Mesh has been formed, the IR510 nodes have reachability only to the FAR. The mesh nodes need a way to communicate all the way to control center components like IoT FND for management purposes. To achieve this, the IPv6 LoWPAN address subnet assigned to the mesh endpoints is advertised to the HER (which has reachability to the control center components) using the IKEv2 prefix injection over the FlexVPN tunnel. Specifically, the mesh prefix is advertised as part of the IPv6 ACL, which is part of the FlexVPN authorization policy as shown below.

Note: The config shown below is for reference purposes only since ZTD takes addresses it.

Advertising MAP-T BMR IPv6 Prefix using Snapshot Routing

As discussed above, besides advertising the Mesh LoWPAN prefix of the IR510 nodes to the HER, even the MAP-T BMR IPv6 prefix of the nodes needs to be reachable from the control center to communicate with the SCADA clients connected to the IR510. To achieve this, the IKEv2 snapshot routing feature is implemented wherein the BMR IPv6 prefix assigned to the mesh endpoints is included in the route map redistributed inside the FlexVPN authorization policy, as shown below.

Note: The config shown below is for reference purposes only since ZTD takes addresses it. Basically, the BMR IPv6 /128 address of the nodes that appear/disappear from the HER routing table are the ones that match the route-map snapshot shown below.

SCADA Control Center Point-to-Point Implementation Scenarios Over Cellular Gateways

In this scenario, the DSO will be hosting SCADA applications (Master) in a Control Center. The SCADA Slave is connected to the DA Gateway via the serial or Ethernet interface. The SCADA Master residing in the DSO Control Center can communicate with the Slave using the DNP3 or DNP3 IP protocol.

Operations that can be executed when the communication protocol is DNP3, DNP3 IP. or DNP3-DNP3 IP translation are as follows:

■Poll (Master > Slave)

■Control (Master > Slave)

■Unsolicited Reporting (Slave > Master) - Notification

The operations have been executed using a SCADA simulator known as the Distributed Test Manager (DTM), which has the capability of simulating both the Master and the Slave devices.

■If the endpoint is connected to the DA Gateway via the Ethernet port, then it is pure IP traffic. The IP address of the endpoint (i.e., IED) can be NAT'd so that the same subnet between the IED and the Ethernet interface of the DA Gateway can be re-used. This approach will ease the deployment.

■If the endpoint is connected using asynchronous serial (RS-232 or RS-485), then the DNP3 could be tunneled to the control center using Raw Socket, and the SCADA Master would consume as DNP3 or DNP3 to be converted to DNP3 IP at the gateway and the SCADA Master would consume as DNP3/IP.

This document focuses on SCADA protocols such as the DNP3, DNP3 IP, and DNP3-DNP3 IP translation protocols widely used in the U.S. Region with a Control Center.

Figure 56 Feeder Automation Lab Topology

IR1101 and IR807 are implemented as Cellular DA Gateways. ASR 1000s implemented in clustering mode act as a HER, which terminates FlexVPN tunnels from DA Gateways.

The following sections focus upon:

1. SCADA Communication with IP intelligent devices

2. SCADA Communication with Legacy devices

a. Raw Socket TCP

b. Protocol Translation

SCADA Communication with IP Intelligent Devices

Protocols Validated

The protocol we have validated for this release is DNP3 IP.

Flow Diagram

Figure 57 DNP3 IP Control Flow

As shown in Figure 57, the SCADA Master DTM can perform a read and write operation to a remote Slave via the DA Gateway. The Slave can send the Unsolicited Reporting to the SCADA Master via the DA Gateway over the IP network.

As per the topology, the interface connected to SCADA Slave has the following configuration. This configuration is only for reference purpose only since ZTD of Cellular gateways will address it. Please refer to Appendix D: SCADA ICT Enablement Profiles.

IR807 DA Gateway Configuration

IR1101 DA Gateway Configuration

SCADA Master Configuration

As per the topology, the SCADA Master is residing in the Control Center. The following configuration must be required for the SCADA Master to communicate with SCADA Slave.

1. Open the SCADA Master Application and add a new DNP3 Master.

2. From the Channel tab, configure the SCADA Master, as per Figure 58.

3. SCADA Master, in this case, is configured as a TCP Client interacting with the SCADA Slave, which is configured to act as TCP Server.

4. Populate the remote address field with the Loopback IP of the Cellular gateway.

5. Populate the port with 20000, which is the port used in the Cisco IOS configuration.

Figure 58 SCADA Master Configuration

SCADA Slave Configuration

As per the topology, the SCADA Slave resides in the field area. The following configuration must be required for the SCADA Slave to communicate with the SCADA Master.

1. Open the SCADA Slave Application and add a new DNP3 Slave.

2. From the Channel tab, configure the SCADA Master, as per Figure 59.

3. Populate the remote address field with SCADA Master IP.

4. Populate the port with 20000, which is the port used in SCADA Master.

Figure 59 SCADA Slave Configuration

SCADA Operations

The Master and the Slave can communicate via Poll, Control, and Unsolicited Reporting. Poll and Control operations are initiated from the Master. Unsolicited Reporting is sent to the Master from the Slave. Figure 60 and Figure 61 show the Poll operation from the SCADA Master. Similarly, Control and Unsolicited Reporting can be seen on the Master Analyzer logs.

Poll

The Poll operation is performed by the Master. The Master can execute a general Poll in which all the register values are read and sent to the Master. In Figure 60 and Figure 61, we see a general Poll executed on the Master side. As Figure 60 shows, the Master Analyzer is initially empty.

Figure 60 Master Analyzer Logs before Poll Operation

However, when the General Interrogation command is executed, the values of all the registers are displayed on the Master Analyzer, as shown in Figure 61.

Figure 61 Master Analyzer Logs after Poll Operation

Control

The Control operation basically sends the control command from the SCADA Master to the SCADA Slave in order to control the operation of end devices. The control command can be executed and the results can be seen on the analyzer. The value of Control Relay Output is changed and is notified to the Master. Figure 62 shows control relay output status before sending the control command to the Slave.

Figure 62 Slave Register before Control Operation

Figure 63 shows how SCADA Master sends the control command.

Figure 63 Master Control Operation

Figure 64 show the Control Command and Control Relay Output status changed on the SCADA Master.

Figure 64 Slave Register after Control Operation

Unsolicited Reporting

Unsolicited Reporting is initiated by the Slave, which is connected to the DA Gateway. Changes to the value of the Slave register are notified to the SCADA Master. This notification can be seen on the Master Analyzer. Figure 65 shows the SCADA Master Analyzer before any unsolicited reporting.

Figure 65 Master Analyzer

Figure 66 shows that the binary input of the Slave is going to change. Initially the value of binary input is OFF.

Figure 66 Slave Registers

Figure 67 shows that the binary input of the Slave is changed from OFF to ON.

Figure 67 Change in Slave Register Value

Figure 68 show the Unsolicited Reporting on the analyzer. The value of Binary Inputs is changed and the same is notified to the Master.

Figure 68 Master Analyzer after Change in Register Value

Legacy SCADA (Raw Socket TCP)

Protocols Validated

The protocol we have validated for this release is DNP3.

Flow Diagram

Figure 69 DNP3 Control Flow

As shown in Figure 69, the DTM Master can read and write the Slave via the DA Gateway using TCP Raw Socket. In addition, the Slave can send the Unsolicited Reporting to the Master via the DA Gateway using TCP Raw Socket. For more details about Raw Socket, refer to the Distribution Automation - Feeder Automation Design Guide.

IR807 DA Gateway Raw Socket Configuration

As per the topology, the interface connected to SCADA Slave has the following configuration:

IR1101 DA Gateway Raw Socket Configuration

As per the topology, the interface connected to SCADA Slave has the following configuration:

SCADA Master Configuration

As per the topology, the SCADA Master is residing in the Control Center. The following configuration is required for the SCADA Master to communicate with SCADA Slave. In this implementation, we used the SCADA DTMW simulator instead of a real SCADA device.

1. Open the SCADA Master Application and click Add a new DNP3 Master.

2. From the Channel tab, configure the SCADA Master as per Figure 70.

3. On the SCADA Master, select the appropriate serial port, baud rate, data bits, stop bits, and parity matching for your device configuration.

Figure 70 Master Configuration

SCADA Slave Configuration

As per the topology, the SCADA Slave is residing in the field area. The following configuration must be required for the SCADA Slave to communicate with the SCADA Master. In this implementation, we used the SCADA DTMW simulator instead of a real SCADA device.

1. Open the SCADA Slave Application and click Add a new DNP3 Slave.

2. From the Channel tab, configure the SCADA Master as per Figure 71.

3. On the SCADA Slave, select the appropriate serial port, baud rate, data bits, stop bits and parity matching for your device configuration.

Figure 71 Slave Configuration

SCADA Operations

The Master and the Slave can communicate via the network. Poll and Control operations are initiated from the Master. Unsolicited Reporting is sent to the Master from the Slave. Figure 72 and Figure 73 show the Poll operation from the SCADA Master. Similarly, Control and Unsolicited Reporting can also be seen on the Master Analyzer logs.

Poll

The Poll operation is performed by the Master, which can execute a general Poll in which all the register values are read and sent to the Master. In Figure 72 and Figure 73, we see a general Poll executed on the Master side. As Figure 72 shows, the Master Analyzer is initially empty.

Figure 72 Master Analyzer Logs before Poll Operation

However, when the General Interrogation command is executed, the values of all the registers are displayed on the Master Analyzer shown in Figure 73.

Figure 73 Master Analyzer Logs after Poll Operation

Control

The Control operation basically sends the control command from the SCADA Master to SCADA Slave for the purpose of controlling the operation of end devices. The control command can be executed and the results can be seen on the analyzer. The value of Control Relay Output is changed, which is notified to the Master. Figure 74 shows control relay output status before sending the control command to the Slave.

Figure 74 Slave Register before Control Operation

Figure 75 shows how SCADA Master sends the control command.

Figure 75 Master Control Operation

Figure 76 shows the Control Relay Output status changed on SCADA Master.

Figure 76 Slave Register after Control Operation

Unsolicited Reporting

Unsolicited Reporting is initiated by the Slave, which is connected to the DA Gateway. Changes to the value of the Slave register are reported to the SCADA Master. This notification can be seen on the Master Analyzer. Figure 77 shows an empty screen of the SCADA Master Analyzer before any unsolicited reporting.

Figure 77 Master Analyzer

Figure 78 shows that the binary input of the Slave is going to change. Initially the value of binary input is OFF.

Figure 78 Slave Registers

Figure 79 shows the binary input of the Slave is changed from OFF to ON.

Figure 79 Change in Slave Register Value

Figure 80 show the Unsolicited Reporting on the analyzer. The value of Binary Inputs is changed and the same is notified to the Master.

Figure 80 Master Analyzer after Change in Register Value

SCADA Gateway

Protocols Validated

The protocols we have validated for this release are DNP3 and DNP3 IP.

Flow Diagram

Figure 81 DNP3-to-DNP3 IP Protocol Translation Control Flow

As shown in Figure 81, the DTM Master can read and write the Slave via the DA Gateway using protocol translation. The Slave can send the Unsolicited Reporting to the Master via the DA Gateway using protocol translation.

IR807 DA SCADA Gateway Configuration

As per the topology, the interface connected to SCADA Slave has the following configuration:

IR1101 DA SCADA Gateway Configuration

As per the topology, the interface connected to SCADA Slave has the following configuration:

SCADA Master Configuration

As per the topology, the SCADA Master is residing in the Control Center. The following configuration is required in order for the SCADA Master to communicate with SCADA Slave:

1. Open the SCADA Master Application and click Add a new DNP3 Master.

2. From the Channel tab, configure the SCADA Master as per Figure 82.

3. SCADA Master (in this case configured as TCP Client), interacts with the SCADA Slave, which is configured to act as a TCP Server.

4. Populate the remote address field with the Loopback IP of Cellular Gateway.

5. Populate the port with 21000, which is the port used in Cisco IOS Configuration.

Figure 82 Master Configuration

SCADA Slave Configuration

As per the topology, the SCADA Slave is residing in the field area. The following configuration must be required for the SCADA Slave to communicate with SCADA Master. In this implementation, we used SCADA DTMW simulator instead of a real SCADA device.

1. Open the SCADA Slave Application and click Add a new DNP3 Slave.

2. From the Channel tab, configure the SCADA Master, as per Figure 83.

3. On the SCADA Slave, select the appropriate serial port, baud rate, data bits, stop bits, and parity matching your device configuration.

Figure 83 Slave Configuration

SCADA Operations

The Master and the Slave can communicate via the network. Poll and Control operations are initiated from the Master. Unsolicited Reporting is sent to the Master from the Slave. Figure 84 and Figure 85 show the Poll operation from the SCADA Master. Control and Unsolicited Reporting can also be seen on the Master Analyzer logs.

Poll

The Poll operation is performed by the Master, which can execute a general Poll in which all the register values are read and sent to the Master. In Figure 84 and Figure 85, we see a general Poll executed on the Master side.

As Figure 84 shows, the Master Analyzer is initially empty.

Figure 84 Master Analyzer Logs before Poll Operation

However, when the General Interrogation command is executed, the values of all the registers are displayed on the Master Analyzer, as shown in Figure 85.

Figure 85 Master Analyzer Logs after Poll Operation

Control

The Control operation basically sends the control command from the SCADA Master to SCADA Slave for the purpose of controlling the operation of end devices. The control command can be executed, and the results can be seen on the analyzer. The value of Control Relay Output is changed and the same is notified to the Master. Figure 86 shows the control relay output status before sending the control command to the Slave.

Figure 86 Slave Register before Control Operation

Figure 87 shows how the SCADA Master sends the control command.

Figure 87 Master Control Operation

Figure 88 shows the Control Relay Output status changed on the SCADA Master.

Figure 88 Slave Register after Control Operation

Unsolicited Reporting

Unsolicited Reporting is initiated by the Slave, which is connected to the DA Gateway. Changes to the value of the Slave register changes are notified to the SCADA Master. This notification can be seen on the Master Analyzer. Figure 89 shows an empty screen of the SCADA Master Analyzer before any unsolicited reporting.

Figure 89 Master Analyzer

Figure 90 shows the binary input of the Slave that is going to change. Initially the value of binary input is OFF.

Figure 90 Slave Registers

Figure 91 shows the binary input of the Slave is changed from OFF to ON.

Figure 91 Change in Slave Register Value

Figure 92 show the Unsolicited Reporting on the analyzer. The value of Binary Inputs is changed and the same is notified to the Master.

Figure 92 Master Analyzer after Change in Register Value

SCADA Communication Scenarios over CR Mesh Network (IEEE 802.15.4)

In this scenario, the DSO will be hosting SCADA applications (Master) in a Control Center. The SCADA Slave is connected to the mesh node via the serial or Ethernet interface. The SCADA Master residing in the DSO Control Center can communicate with the Slave using the DNP3 or DNP3 IP protocol.

Operations that can be executed when the communication protocol is DNP3 or DNP3 IP are as follows:

■Poll (Master > Slave)

■Control (Master > Slave)

■Unsolicited Reporting (Slave > Master) - Notification

The operations have been executed using a SCADA simulator known as the DTM and Test Harness tool, which has the capability of simulating both the Master and the Slave devices.

■If the endpoint is connected to the mesh node via the Ethernet port, then it is pure IP traffic. The IP address of the endpoint (i.e., IED) can be NAT'd so that the same subnet between the IED and the Ethernet interface of the DA Gateway can be re-used. This approach will ease the deployment.

■If the endpoint is connected using asynchronous serial (RS-232 or RS-485), then tunneling of serial traffic using Raw Sockets (DNP3) must happen at the mesh node only.

This document focuses on SCADA protocols such as DNP3 and DNP3 IP protocols widely used in the Americas Region with a Control Center.

Figure 93 Feeder Automation CR Mesh Lab Topology

The IR510 is implemented as a mesh node, The CGR1240 is implemented as a FAR, and the ASR 1000s implemented in clustering mode act as a HER, which terminates FlexVPN tunnels from the FAR and the HER.

IP-Enabled SCADA

Protocols Validated

The protocol we have validated for this release is DNP3 IP.

Flow Diagram

Figure 94 DNP3 IP Control Flow

As shown in Figure 94, the SCADA Master can perform a read and write operation to a remote Slave via the DA Gateway. The Slave can send the Unsolicited Reporting to the SCADA Master via the DA Gateway over the IP network.

IR510 Mesh Node Configuration

This section describes the NAT44 configuration of the IR510 device. Basically IPv4 address assignment of the SCADA Slave and the gateway IPv4 address and the port SCADA Slave listens.

Figure 95 IR510 Mesh Node Configuration

Note: Enable the front panel Ethernet Port on the Configuration template.

For information on NMS management and MAP-T, please refer to Zero Touch Enrollment of Cisco Resilient Mesh Endpoints.

SCADA Master Configuration

As per the topology, the SCADA Master is residing in the Control Center. The following configuration must be required for the SCADA Master to communicate with the SCADA Slave.

1. Open the SCADA Master Application and click Add a new DNP3 Master.

2. From the Channel tab, configure the SCADA Master as per Figure 96.

The SCADA Master, in this case, is configured as TCP Client, interacting with SCADA Slave, which is configured to act as the TCP Server.

3. Populate the Remote Address field with the Loopback IP of the Cellular Gateway.

4. Populate the port with 20000, which is the port used in Cisco IOS Configuration.

For information on MAP-T, please refer to Zero Touch Enrollment of Cisco Resilient Mesh Endpoints.

Figure 96 SCADA Master Configuration

SCADA Slave Configuration

As per the topology, the SCADA Slave is residing in the field area. The following configuration is required for the SCADA Slave to communicate with SCADA Master.

1. Open the SCADA Slave Application and click Add a new DNP3 Slave.

2. From the Channel tab, configure the SCADA Master as per Figure 97.

3. Populate the Remote Address field with the SCADA Master IP.

4. Populate the port with 20000, which is the port used in the SCADA Master.

Figure 97 SCADA Slave Configuration

SCADA Operations

The Master and the Slave can communicate via the network. Poll and Control operations are initiated from the Master. Unsolicited Reporting is sent to the Master from the Slave. Figure 98 and Figure 99 show the Poll operation from the SCADA Master. Control, and Unsolicited Reporting can also be seen on the Master Analyzer logs.

Poll

The Poll operation is performed by the Master. The Master can execute a general Poll in which all the register values are read and sent to the Master. In Figure 98 and Figure 99, we see a general Poll executed on the Master side.

As per Figure 98 shows, the Master Analyzer is initially empty.

Figure 98 Master Analyzer Logs before Poll Operation

However, when the General Interrogation command is executed, the values of all the registers are displayed on the Master Analyzer, as shown in Figure 99.

Figure 99 Master Analyzer Logs after Poll Operation

Control

The Control operation basically sends the control command from the SCADA Master to SCADA Slave for the purpose of controlling the operation of end devices. The control command can be executed, and the results can be seen on the analyzer. The value of Control Relay Output is changed and the same is notified to the Master. The SCADA Control operation has been validated in the following sequence of steps.

The Initial Control Relay Output status would be noted on the SCADA Slave.

Figure 100 shows the control relay output status before sending the control command to the Slave.

Figure 100 Slave Register before Control Operation

As Figure 101 shows, a control operation is then performed to modify the value of the control relay output register on the SCADA Slave. This operation is performed from the SCADA Master on the SCADA Slave.

Figure 101 Master Control Operation

After the control operation is issued from the SCADA Master, the control relay output register of the SCADA Slave is noted. As Figure 102 shows, successful modification of the register value on the SCADA Slave signifies the successful Control operation.

Figure 102 Slave Register after Control Operation

Unsolicited Reporting

Unsolicited Reporting is initiated by the Slave, which is connected to the DA Gateway. Changes to the value of the Slave register are reported to the SCADA Master. This notification can be seen on the Master Analyzer. Figure 103 shows an empty screen of the SCADA Master Analyzer before any unsolicited reporting.

Figure 103 Master Analyzer

Figure 104 shows binary input of the Slave that is going to change. Initially the value of binary input is OFF.

Figure 104 Slave Registers

Figure 105 shows the binary input of the Slave is changed from OFF to ON.

Figure 105 Change in Slave Register Value