Replacement of Nexus 9236C Spine Switch - CPAR

Introduction

This document describes the steps required to replace a faulty Spine switch (Nexus 9236C) in an Ultra-M setup.

This procedure applies for an Openstack environment through NEWTON version where ESC does not manage Cisco Prime Access Registrar (CPAR) and CPAR is installed directly on the VM deployed on Openstack.

Background Information

Ultra-M is a pre-packaged and validated virtualized mobile packet core solution designed to simplify the deployment of VNFs. The servers that are part of the Ultra-M setup are connected to three different types of switches :

• Catalyst Switch
• Leaf Switch
• Spine Switch

This image shows the network topology of a Ultra-M setup:

Note: The Network topology is only a representation, the connections between the switches might slightly vary, it depends upon the solution deployed.

This document is intended for the Cisco personnel who are familiar with Cisco Ultra-M setup and Catalyst Switch operations.

Abbreviations

Workflow of the MoP

This image shows the high level workflow of Replacement Procedure.

Spine Switch in UltraM Setup

Prerequisite

Take a backup of the configuration file from the Spine switch through FTP/TFTP before it proceedes with the switch replacement. 

 
AL03-N9236c-POD2-spine1# copy running-config sftp: 
 Enter destination filename: [AL03-N9236c-POD2-spine1-running-config] backup-spine-cfg-2 
 Enter vrf (If no input, current vrf 'default' is considered): management 
 Enter hostname for the sftp server: 10.1.1.1
 Enter username: root 

The authenticity of host '10.1.1.1 (10.1.1.1)' can't be established.
ECDSA key fingerprint is SHA256:dnHSXQ6fyYX5v8DU87oqVkCoA8CSPW7CpT+o2qilNI4.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.1.1.1' (ECDSA) to the list of known hosts. 
root@10.1.1.1's password: 
Connected to 10.1.1.1.
sftp> put /var/tmp/vsh/backup-spine-cfg-2 backup-spine-cfg-2 
Uploading /var/tmp/vsh/backup-spine-cfg-2 to /root/backup-spine-cfg-2
 /var/tmp/vsh/backup-spine-cfg-2 0% 0 0.0KB/s --:-- ETA /var/tmp/vsh/backup-spine-cfg-2 100% 11KB 11.2KB/s 00:00 
sftp> exit 
Copy complete, now saving to disk (please wait)...
Copy complete.

Health Checks

Step 1. Check the current software version that runs in the switch and make a note of it.

AL03-N9236c-POD2-spine1# show ver 
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (C) 2002-2018, Cisco and/or its affiliates.
All rights reserved.
The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under their own
licenses, such as open source. This software is provided "as is," and unless
otherwise stated, there is no warranty, express or implied, including but not
limited to warranties of merchantability and fitness for a particular purpose.
Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or 
GNU General Public License (GPL) version 3.0 or the GNU
Lesser General Public License (LGPL) Version 2.1 or 
Lesser General Public License (LGPL) Version 2.0. 
A copy of each such license is available at
http://www.opensource.org/licenses/gpl-2.0.php and
http://opensource.org/licenses/gpl-3.0.html and
http://www.opensource.org/licenses/lgpl-2.1.php and
http://www.gnu.org/licenses/old-licenses/library.txt.

Software
 BIOS: version 07.59
 NXOS: version 7.0(3)I7(3)
 BIOS compile time: 08/26/2016
 NXOS image file is: bootflash:///nxos.7.0.3.I7.3.bin
 NXOS compile time: 2/12/2018 13:00:00 [02/12/2018 19:13:48]


Hardware
 cisco Nexus9000 C9236C chassis 
 Intel(R) Xeon(R) CPU @ 1.80GHz with 16400560 kB of memory.
 Processor Board ID FDO211613CA

 Device name: AL03-N9236c-POD2-spine1
 bootflash: 53298520 kB
Kernel uptime is 134 day(s), 14 hour(s), 48 minute(s), 25 second(s)

Last reset at 705173 usecs after Fri Mar 2 17:56:31 2018
 Reason: Reset due to upgrade
 System version: 7.0(3)I5(2)
 Service: 

plugin
--More-- Core Plugin, Ethernet Plugin

Active Package(s):

Step 2. Check the current license usage.

AL03-N9236c-POD2-spine1# show license uasgesage 
Feature Ins Lic Status Expiry Date Comments
 Count
--------------------------------------------------------------------------------
N9K_LIC_1G No - Unused -
VPN_FABRIC No - Unused -
FCOE_NPV_PKG No - Unused -
SECURITY_PKG No 0 Unused -
N9K_UPG_EX_10G No - Unused -
TP_SERVICES_PKG No - Unused -
NXOS_ADVANTAGE_GF No - Unused -
NXOS_ADVANTAGE_M4 No - Unused -
NXOS_ADVANTAGE_XF No - Unused -
NXOS_ESSENTIALS_GF No - Unused -
NXOS_ESSENTIALS_M4 No - Unused -
NXOS_ESSENTIALS_XF No - Unused -
SAN_ENTERPRISE_PKG No - Unused -
PORT_ACTIVATION_PKG No 0 Unused -
NETWORK_SERVICES_PKG No - Unused -
NXOS_ADVANTAGE_M8-16 No - Unused -
NXOS_ESSENTIALS_M8-16 No - Unused -
FC_PORT_ACTIVATION_PKG No 0 Unused -
LAN_ENTERPRISE_SERVICES_PKG Yes - In use Never -
--------------------------------------------------------------------------------

Step 3. Ensure that the physical cables connected to the switch port-channel, Vlan and also the port status are good as expected.

AL03-N9236c-POD2-spine1# show interface status | grep connected 
mgmt0 -- connected routed full 100 -- 
Eth1/1 Connected to leaf1 connected trunk full 100G QSFP-100G-AOC5M
Eth1/2 Connected to leaf1 connected trunk full 100G QSFP-100G-AOC5M
Eth1/4 Connected to leaf2 connected trunk full 100G QSFP-100G-AOC5M
Eth1/5 Connected to leaf2 connected trunk full 100G QSFP-100G-AOC5M
Eth1/7 Connected to Leaf3 connected trunk full 100G QSFP-100G-AOC5M
Eth1/8 Connected to Leaf3 connected trunk full 100G QSFP-100G-AOC5M
Eth1/10 connect leaf4 connected trunk full 100G QSFP-100G-AOC5M
Eth1/11 connect leaf4 connected trunk full 100G QSFP-100G-AOC5M
Eth1/21/1 Connected to NMNET connected 101 full 10G QSFP-40G-SR4
Eth1/29 Connected to AL04- connected trunk full 100G QSFP-100G-AOC5M
Eth1/30 Connected to AL04- connected trunk full 100G QSFP-100G-AOC5M
Eth1/36/1 connected to ARG | connected routed full 10G QSFP-40G-SR4
Eth1/36/2 connected to ARG | disabled routed auto auto QSFP-40G-SR4
Eth1/36/3 connected to ARG | disabled routed auto auto QSFP-40G-SR4
Eth1/36/4 connected to ARG | disabled routed auto auto QSFP-40G-SR4
Po10 -- connected routed full 10G -- 
Po10.3101 -- connected routed full 10G -- 
Po10.3109 -- connected routed full 10G -- 
Po18 -- connected trunk full 100G -- 
Po21 -- connected trunk full 100G -- 
Po22 -- connected trunk full 100G -- 
Po25 -- connected trunk full 100G -- 
Po26 -- connected trunk full 100G -- 
Lo1 -- connected routed auto auto -- 
Vlan101 -- connected routed auto auto --
Vlan1025 vPCRF 1 and vPCRF connected routed auto auto --
Vlan1026 Replication for vP connected routed auto auto --
Vlan1031 External Traffic v connected routed auto auto --
Vlan1032 External External connected routed auto auto --
Vlan1033 External Traffic connected routed auto auto --
Vlan1034 External Traffic v connected routed auto auto --
Vlan1035 External Traffic v connected routed auto auto --
Vlan1036 External Traffic v connected routed auto auto --
Vlan1041 External Traffic A connected routed auto auto --
Vlan1042 External Traffic A connected routed auto auto --
Vlan1043 External Traffic A connected routed auto auto --
Vlan2601 Global Interchassi connected routed auto auto --
Vlan2602 NMNET Interchassis connected routed auto auto --

Step 4. Ensure that BGP neighbors are up.

AL03-N9236c-POD2-spine1# show ip bgp summary vrf all 
BGP summary information for VRF NMNET, address family IPv4 Unicast
BGP router identifier 10.225.247.201, local AS number 65361
BGP table version is 2, IPv4 Unicast config peers 2, capable peers 1
0 network entries and 0 paths using 0 bytes of memory
BGP attribute entries [0/0], BGP AS path entries [0/0]
BGP community entries [0/0], BGP clusterlist entries [0/0]

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.0.100.6 4 65361 193834 193833 2 0 0 19w1d 0 
10.250.180.185 4 65137 0 0 0 0 0 19w1d Idle 

BGP summary information for VRF default, address family IPv4 Unicast
BGP router identifier 10.0.200.1, local AS number 65361
BGP table version is 161, IPv4 Unicast config peers 2, capable peers 2
18 network entries and 29 paths using 5700 bytes of memory
BGP attribute entries [3/480], BGP AS path entries [1/6]
BGP community entries [0/0], BGP clusterlist entries [0/0]

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.0.100.2 4 65361 3868481 3868419 161 0 0 19w1d 12 
10.163.140.225 4 65137 3868521 3868365 161 0 0 19w1d 5   

 AL03-N9236c-POD2-spine1# show ipv6 bgp summary vrf all

Step 5. Ensure the BFD neighbors state is up.

AL03-N9236c-POD2-spine1# show bfd neighbors vrf all 

OurAddr NeighAddr LD/RD RH/RS Holdown(mult) State Int Vrf 
10.163.140.226 10.163.140.225 1090519042/1090519042 Up 107(3) Up Po10.3101 default 

AL03-N9236c-POD2-spine1# show bfd ipv6 neighbor vrf all
                           

Switch Replacement procedure

Step 1. Install the new switch in the rack and connect the cables to the switch as noted. The steps for switch installation can be found in this link:  Spine Nexus 236C NX-OS mode Switch Hardware Installation Guide

Step 2. Check the software version of the Nexus Spine switch and upgrade/downgrade to the software version as per the given link:  Cisco Nexus 9000 Series NX-OS Software Upgrade and Downgrade Guide, Release 7.x

Step 3. Transfer the configuration backup to the new switch and copy that configuration to the startup-config.

server-backup$ sftp admin@10.1.1.1
The authenticity of host '10.1.1.1 (10.1.1.1)' can't be established.
RSA key fingerprint is SHA256:fnbUmd2mL5yE94zxrRoKAlvYfQbheXJfQox7m3XfpIU.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.1.1.1' (RSA) to the list of known hosts.
User Access Verification
Password: 
Password: 
Connected to 10.1.1.1.
sftp> ls          
20170607_193430_poap_15489_1.log     20170607_193430_poap_15489_2.log     20170607_193430_poap_15489_init.log  backup-spine-cfg-2                   backup-spine.cfg                     
backup_run_02152018                  bios_daemon.dbg                      cfg-july25th                         cfg-july25th-spine1                  flash:                               
flash:cfg-aug8th-ybattina-afterNSO   flash:cfg-jul28th                    home                                 l3-cfg-aug10th                       license_FDO211406K1_16.lic           
lost+found                           nxos.7.0.3.I4.4.bin                  nxos.7.0.3.I5.2.bin                  nxos.7.0.3.I7.3.bin                  platform-sdk.cmd                     
scripts                              spine-config-bkp                     starat                               virtual-instance                     vlan.dat                                                                                                                                                                  100%   33KB  21.7KB/s   00:01    
sftp> put backup-spine-cfg-2
Uploading backup-spine-cfg-2 to /backup-spine-cfg-2
backup-spine-cfg-2                                                                                                                                      100%   33KB  23.5KB/s   00:01    
sftp> bye

AL03-N9236c-POD2-spine1# copy bootflash:///backup-spine-cfg-2 startup-config
Copy progress 100% 33KB
Copy complete, now saving to disk (please wait)...
Copy complete.

Step 4. Reload the switch after loading the backup configuration.

AL03-N9236c-POD2-spine1#reload

Verification of Replaced Spine Switch

After the switch is available for access, perform the health checks through these commands on the spine switch to verify that replaced switch is available in expected state.

AL03-N9236c-POD2-spine1# show port-channel summary

AL03-N9236c-POD2-spine1# show ip bgp summary vrf all

AL03-N9236c-POD2-spine1# show ipv6 bgp summary vrf all

AL03-N9236c-POD2-spine1# show bfd neighbor vrf all

AL03-N9236c-POD2-spine1# show bfd ipv6 neighbor vrf all