Contents

Introduction

Background Information

Abbreviations

Workflow of the MoP

Spine Switch in an Ultra-M Setup

Prerequisite

Health Checks

Switch Replacement Procedure

Verify the Replaced Spine Switch

Introduction

This document describes the steps required in order to replace a faulty Spine Switch (Nexus 9236C) in an Ultra-M setup that hosts StarOS Virtual Network Functions (VNFs).

Background Information

Ultra-M is a pre-packaged and validated virtualized mobile packet core solution designed to simplify the deployment of VNFs. The servers that are part of the Ultra-M setup are connected to three different types of switches :

- Catalyst Switch

- Leaf Switch

- Spine Switch

The network topology of an Ultra-M setup:

UltraM Network Topology

Note: The Network topology is only a representation, the connections between the switches might slightly vary depending on the solution deployed.

This document is intended for the Cisco personnel who are familiar with Cisco Ultra-M setup and Catalyst Switch operations.

Abbreviations

VNF

Virtual Network Function

SPINE

Nexus 9236C Switch as Spine

MOP

Method of Procedure

LAN

Local Area Network

FTP

File Transfer Protocol

TFTP

Trivial File Transfer Protocol

CIMC

Cisco Integrated Management Controller

Workflow of the MoP

Highlevel Workflow of the Replacement Procedure

Spine Switch in an Ultra-M Setup

Prerequisite

Take a backup of the configuration file from the Spine switch with the use of ftp/tftp before you proceed with the switch replacement.

Nexus-POD1-spine2# copy running-config sftp:

Enter destination filename: [Nexus-POD1-spine2-running-config] backup-spine-cfg-2

Enter vrf (If no input, current vrf 'default' is considered): management

Enter hostname for the sftp server: 10.1.1.1

Enter username: admin

The authenticity of host '10.1.1.1 (10.1.1.1)' can't be established.

RSA key fingerprint is SHA256:fnbUmd2mL5yE94zxrRoKAlvYfQbheXJfQox7m3XfpIU.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added '10.1.1.1' (RSA) to the list of known hosts.

User Access Verification

Password:

Connected to 10.1.1.1.

sftp> put  /var/tmp/vsh/backup-spine-cfg-2  backup-spine-cfg-2

Uploading /var/tmp/vsh/backup-spine-cfg-2 to /backup-spine-cfg-2

/var/tmp/vsh/backup-spine-cfg-2                                                                                                                                                                                          100%   33KB  33.2KB/s   00:00

sftp> exit

Copy complete, now saving to disk (please wait)...

Copy complete.

Nexus-POD1-spine2#

Health Checks

1. Check the current software version running in the switch and make a note of it.

Nexus-POD1-spine2# show ver

Cisco Nexus Operating System (NX-OS) Software

TAC support: http://www.cisco.com/tac

Copyright (C) 2002-2018, Cisco and/or its affiliates.

All rights reserved.

The copyrights to certain works contained in this software are

owned by other third parties and used and distributed under their own

licenses, such as open source.  This software is provided "as is," and unless

otherwise stated, there is no warranty, express or implied, including but not

limited to warranties of merchantability and fitness for a particular purpose.

Certain components of this software are licensed under

the GNU General Public License (GPL) version 2.0 or

GNU General Public License (GPL) version 3.0  or the GNU

Lesser General Public License (LGPL) Version 2.1 or

Lesser General Public License (LGPL) Version 2.0.

A copy of each such license is available at

http://www.opensource.org/licenses/gpl-2.0.php and

http://opensource.org/licenses/gpl-3.0.html and

http://www.opensource.org/licenses/lgpl-2.1.php and

http://www.gnu.org/licenses/old-licenses/library.txt.

Software

BIOS: version 07.59

NXOS: version 7.0(3)I7(3)

BIOS compile time:  08/26/2016

NXOS image file is: bootflash:///nxos.7.0.3.I7.3.bin

NXOS compile time:  2/12/2018 13:00:00 [02/12/2018 19:13:48]

Hardware

cisco Nexus9000 C9236C chassis

Intel(R) Xeon(R) CPU  @ 1.80GHz with 16400560 kB of memory.

Processor Board ID FDO21120SSN

Device name: Nexus-POD1-spine2

bootflash:   53298520 kB

Kernel uptime is 108 day(s), 13 hour(s), 15 minute(s), 12 second(s)

Last reset

Reason: Unknown

System version: 7.0(3)I7(3)

Service:

plugin

Core Plugin, Ethernet Plugin

Active Package(s):

2. Check the current license usage.

Nexus-POD1-spine2# show license usage

Feature                      Ins  Lic   Status Expiry Date Comments

Count

--------------------------------------------------------------------------------

N9K_LIC_1G                    No    -   Unused             -

VPN_FABRIC                    No    -   Unused             -

FCOE_NPV_PKG                  No    -   Unused             -

SECURITY_PKG                  No    0   Unused             -

N9K_UPG_EX_10G                No    -   Unused             -

TP_SERVICES_PKG               No    -   Unused             -

NXOS_ADVANTAGE_GF             No    -   Unused             -

NXOS_ADVANTAGE_M4             No    -   Unused             -

NXOS_ADVANTAGE_XF             No    -   Unused             -

NXOS_ESSENTIALS_GF            No    -   Unused             -

NXOS_ESSENTIALS_M4            No    -   Unused             -

NXOS_ESSENTIALS_XF            No    -   Unused             -

SAN_ENTERPRISE_PKG            No    -   Unused             -

PORT_ACTIVATION_PKG           No    0   Unused             -

NETWORK_SERVICES_PKG          No    -   Unused             -

NXOS_ADVANTAGE_M8-16          No    -   Unused             -

NXOS_ESSENTIALS_M8-16         No    -   Unused             -

FC_PORT_ACTIVATION_PKG        No    0   Unused             -

LAN_ENTERPRISE_SERVICES_PKG   Yes   -   In use Never       -

--------------------------------------------------------------------------------

3. Ensure  the physical cables connected to the switch port-channel , Vlan and also the port status are good as expected

Nexus-POD1-spine2# show interface status | grep connected

mgmt0         --                 connected routed    full    100     --

Eth1/1        --                 connected trunk     full    100G    QSFP-100G-AOC5M

Eth1/2        --                 connected trunk     full    100G    QSFP-100G-AOC5M

Eth1/5        --                 connected trunk     full    100G    QSFP-100G-AOC5M

Eth1/6        --                 connected trunk     full    100G    QSFP-100G-AOC5M

Eth1/11/1     Connected to NMNET connected 101       full    10G     QSFP-40G-SR4

Eth1/21/1     --                 connected routed    full    10G     QSFP-40G-SR4

Eth1/21/2     --                 connected routed    full    10G     QSFP-40G-SR4

Eth1/21/3     --                 connected routed    full    10G     QSFP-40G-SR4

Eth1/21/4     --                 connected routed    full    10G     QSFP-40G-SR4

Eth1/22/1     --                 connected routed    full    10G     QSFP-40G-SR4

Eth1/22/2     --                 connected routed    full    10G     QSFP-40G-SR4

Eth1/22/3     --                 connected routed    full    10G     QSFP-40G-SR4

Eth1/22/4     --                 connected routed    full    10G     QSFP-40G-SR4

Eth1/23/1     --                 connected routed    full    10G     QSFP-40G-SR4

Eth1/23/2     --                 connected routed    full    10G     QSFP-40G-SR4

Eth1/23/3     --                 connected routed    full    10G     QSFP-40G-SR4

Eth1/23/4     --                 connected routed    full    10G     QSFP-40G-SR4

Eth1/29       "spine-spine link" connected trunk     full    100G    QSFP-100G-AOC5M

Eth1/30       "spine-spine link" connected trunk     full    100G    QSFP-100G-AOC5M

Eth1/31       "spine-spine link" connected trunk     full    100G    QSFP-100G-AOC5M

Po22          portchannel to Lea connected trunk     full    100G    --

Po24          portchannel to Lea connected trunk     full    100G    --

Po30          --                 connected routed    full    10G     --

Po30.3201     --                 connected routed    full    10G     --

Po30.3202     --                 connected routed    full    10G     --

Po30.3203     --                 connected routed    full    10G     --

Po30.3204     --                 connected routed    full    10G     --

Po30.3205     --                 connected routed    full    10G     --

Po30.3206     --                 connected routed    full    10G     --

Po99          spine1-spine2      connected trunk     full    100G    --

Lo1           --                 connected routed    auto    auto    --

Lo2           --                 connected routed    auto    auto    --

Lo3           --                 connected routed    auto    auto    --

Lo4           --                 connected routed    auto    auto    --

Lo5           --                 connected routed    auto    auto    --

Vlan2601      Global VRF spine-2 connected routed    auto    auto    --

Vlan2602      GI VRF spine-2 <-> connected routed    auto    auto    --

Vlan2603      IPV6SUB VRF spine- connected routed    auto    auto    --

Vlan2604      METROE-E VRF spine connected routed    auto    auto    --

Vlan2605      NMNET VRF spine-2  connected routed    auto    auto    --

Vlan2721      Global VRF spine-2 connected routed    auto    auto    --

Vlan2722      GI VRF spine-2 <-> connected routed    auto    auto    --

Vlan2723      IPV6SUB VRF spine- connected routed    auto    auto    --

Vlan2724      METROE-E VRF spine connected routed    auto    auto    --

Vlan2741      Global VRF spine-2 connected routed    auto    auto    --

Vlan2742      GI VRF spine-2 <-> connected routed    auto    auto    --

Vlan2743      IPV6SUB VRF spine- connected routed    auto    auto    --

Vlan2744      METROE-E VRF spine connected routed    auto    auto    --

4. Ensure bgp neighbors are up.

Nexus-POD1-spine2# show ip bgp summary vrf all

BGP summary information for VRF GI, address family IPv4 Unicast

BGP router identifier 10.0.202.2, local AS number 65145

BGP table version is 28, IPv4 Unicast config peers 4, capable peers 4

1 network entries and 1 paths using 236 bytes of memory

BGP attribute entries [1/160], BGP AS path entries [1/6]

BGP community entries [0/0], BGP clusterlist entries [0/0]

Neighbor        V    AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd

10.0.92.1       4 65145  312621  312617       28    0    0    15w3d 1

10.0.122.2      4 65251  312624  312616       28    0    0    15w3d 0

10.0.142.2      4 65252  312623  312617       28    0    0    15w3d 0

10.178.240.185  4 65137  312618  312616       28    0    0    12w0d 0

BGP summary information for VRF IPV6SUB, address family IPv4 Unicast

BGP summary information for VRF METROE-E, address family IPv4 Unicast

BGP router identifier 10.0.202.4, local AS number 65145

BGP table version is 10, IPv4 Unicast config peers 4, capable peers 4

1 network entries and 1 paths using 236 bytes of memory

BGP attribute entries [1/160], BGP AS path entries [1/6]

BGP community entries [0/0], BGP clusterlist entries [0/0]

Neighbor        V    AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd

10.0.94.1       4 65145  312619  312617       10    0    0    15w3d 1

10.0.124.2      4 65251  312623  312616       10    0    0    15w3d 0

10.0.144.2      4 65252  312618  312617       10    0    0    15w3d 0

10.178.240.181  4 65137  312618  312616       10    0    0    12w0d 0

BGP summary information for VRF NMNET, address family IPv4 Unicast

BGP router identifier 10.0.202.5, local AS number 65145

BGP table version is 3, IPv4 Unicast config peers 1, capable peers 1

1 network entries and 1 paths using 236 bytes of memory

BGP attribute entries [1/160], BGP AS path entries [0/0]

BGP community entries [0/0], BGP clusterlist entries [0/0]

Neighbor        V    AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd

10.0.95.1       4 65145  312616  312616        3    0    0    15w3d 1

BGP summary information for VRF default, address family IPv4 Unicast

BGP router identifier 10.0.202.1, local AS number 65145

BGP table version is 261, IPv4 Unicast config peers 4, capable peers 3

35 network entries and 49 paths using 10108 bytes of memory

BGP attribute entries [5/800], BGP AS path entries [3/22]

BGP community entries [0/0], BGP clusterlist entries [0/0]

Neighbor        V    AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd

10.0.91.1       4 65145  312681  312635      261    0    0    15w3d 35

10.0.121.2      4 65251  312629  312630      261    0    0    15w3d 0

10.0.141.2      4 65252  312683  312620      261    0    0    15w3d 14

10.178.240.177  4 65137       0  124577        0    0    0    15w3d 12

Nexus-POD1-spine2# show ipv6 bgp summary vrf all

BGP summary information for VRF GI, address family IPv6 Unicast

BGP router identifier 10.0.202.2, local AS number 65145

BGP table version is 64, IPv6 Unicast config peers 4, capable peers 4

1 network entries and 1 paths using 248 bytes of memory

BGP attribute entries [1/160], BGP AS path entries [1/6]

BGP community entries [0/0], BGP clusterlist entries [0/0]

Neighbor        V    AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd

fc00:420:81:122::2

4 65251  312637  312629       64    0    0    15w3d 0

fc00:420:81:142::2

4 65252  312637  312630       64    0    0    15w3d 0

fc00:420:81:992::1

4 65145  312635  312630       64    0    0    15w3d 1

fd01:976a:c002:1:4d13:36f3:0:22

4 65137  312631  312627       64    0    0    12w0d 0

BGP summary information for VRF IPV6SUB, address family IPv6 Unicast

BGP router identifier 10.0.202.3, local AS number 65145

BGP table version is 44, IPv6 Unicast config peers 4, capable peers 4

0 network entries and 0 paths using 0 bytes of memory

BGP attribute entries [0/0], BGP AS path entries [0/0]

BGP community entries [0/0], BGP clusterlist entries [0/0]

Neighbor        V    AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd

fc00:420:81:123::2

4 65251  312632  312629       44    0    0    15w3d 0

fc00:420:81:143::2

4 65252  312636  312629       44    0    0    15w3d 0

fc00:420:81:993::1

4 65145  312631  312629       44    0    0    15w3d 0

fd01:976a:c002:1:4d13:36f3:0:24

4 65137  312629  312628       44    0    0    12w0d 0

BGP summary information for VRF METROE-E, address family IPv6 Unicast

BGP summary information for VRF NMNET, address family IPv6 Unicast

BGP summary information for VRF default, address family IPv6 Unicast

5. Ensure the BFD neighbors state is UP.

Nexus-POD1-spine2# show bfd neighbors vrf al

OurAddr         NeighAddr       LD/RD                 RH/RS           Holdown(mult)     State       Int                   Vrf

10.0.124.1      10.0.124.2      1090519045/1090519205 Up              672(3)            Up          Vlan2724              METROE-E

10.0.122.1      10.0.122.2      1090519046/1090519206 Up              639(3)            Up          Vlan2722              GI

10.0.92.2       10.0.92.1       1090519050/1090519094 Up              585(3)            Up          Vlan2602              GI

10.0.95.2       10.0.95.1       1090519051/1090519095 Up              585(3)            Up          Vlan2605              NMNET

10.0.121.1      10.0.121.2      1090519052/1090519208 Up              639(3)            Up          Vlan2721              default

10.0.94.2       10.0.94.1       1090519054/1090519097 Up              585(3)            Up          Vlan2604              METROE-E

10.0.91.2       10.0.91.1       1090519055/0          Up              N/A(3)            Up          Vlan2601              default

10.0.141.1      10.0.141.2      1090519056/1090519212 Up              610(3)            Up          Vlan2741              default

10.0.144.1      10.0.144.2      1090519057/1090519213 Up              610(3)            Up          Vlan2744              METROE-E

10.0.142.1      10.0.142.2      1090519059/1090519214 Up              610(3)            Up          Vlan2742              GI

10.178.240.182  10.178.240.181  1090519061/0          UP              N/A(3)            Up          Po30.3202             METROE-E

10.178.240.186  10.178.240.185  1090519062/0          Up              N/A(3)            Up          Po30.3203             GI

Nexus-POD1-spine2#

Nexus-POD1-spine2# show bfd ipv6 neighbor vrf all

OurAddr                          NeighAddr

LD/RD                 RH/RS           Holdown(mult)     State       Int                   Vrf

fc00:420:81:123::1               fc00:420:81:123::2

1090519044/1090519204 Up              615(3)            Up          Vlan2723              IPV6SUB

fc00:420:81:122::1               fc00:420:81:122::2

1090519047/1090519207 Up              731(3)            Up          Vlan2722              GI

fc00:420:81:142::1               fc00:420:81:142::2

1090519048/1090519210 Up              696(3)            Up          Vlan2742              GI

fc00:420:81:143::1               fc00:420:81:143::2

1090519049/1090519211 Up              696(3)            Up          Vlan2743              IPV6SUB

fc00:420:81:993::2               fc00:420:81:993::1

1090519053/1090519096 Up              544(3)            Up          Vlan2603              IPV6SUB

fc00:420:81:992::2               fc00:420:81:992::1

1090519058/1090519099 Up              544(3)            Up          Vlan2602              GI

Switch Replacement Procedure

1. Install the new switch in the rack and connect the cables to the switch as noted. The steps for switch installation can be found in this link: Spine Nexus 236C NX-OS mode Switch Hardware Installation Guide

2. Check the software version of the Nexus Spine switch and upgrade/downgrade to the software version as per the link given here: Cisco Nexus 9000 Series NX-OS Software Upgrade and Downgrade Guide, Release 7.x

3. Transfer the configuration backup to the new switch and copy that configuration to the startup-config

server-backup$ sftp admin@10.1.1.1

The authenticity of host '10.1.1.1 (10.1.1.1)' can't be established.

RSA key fingerprint is SHA256:fnbUmd2mL5yE94zxrRoKAlvYfQbheXJfQox7m3XfpIU.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added '10.1.1.1' (RSA) to the list of known hosts.

User Access Verification

Password:

Password:

Connected to 10.1.1.1.

sftp> ls

20170607_193430_poap_15489_1.log     20170607_193430_poap_15489_2.log     20170607_193430_poap_15489_init.log  backup-spine-cfg-2                   backup-spine.cfg

backup_run_02152018                  bios_daemon.dbg                      cfg-july25th                         cfg-july25th-spine1                  flash:

flash:cfg-aug8th-ybattina-afterNSO   flash:cfg-jul28th                    home                                 l3-cfg-aug10th                       license_FDO211406K1_16.lic

lost+found                           nxos.7.0.3.I4.4.bin                  nxos.7.0.3.I5.2.bin                  nxos.7.0.3.I7.3.bin                  platform-sdk.cmd

scripts                              spine-config-bkp                     starat                               virtual-instance                     vlan.dat                                                                                                                                                                  100%   33KB  21.7KB/s   00:01

sftp> put backup-spine-cfg-2

Uploading backup-spine-cfg-2 to /backup-spine-cfg-2

backup-spine-cfg-2                                                                                                                                      100%   33KB  23.5KB/s   00:01

sftp> bye

Nexus-spine1# copy bootflash:///backup-spine-cfg-2 startup-config

Copy progress 100% 33KB

Copy complete, now saving to disk (please wait)...

Copy complete.

4. Reload the switch after loading the backup configuration.

Nexus-POD1-spine2#reload

Verify the Replaced Spine Switch

After the switch is available for access perform the health checks using the below commands on the spine switch to verify that replaced switch is available in the expected state.

Nexus-POD1-spine2# show port-channel summary

Nexus-POD1-spine2# show ip bgp summary vrf all

Nexus-POD1-spine2# show ipv6 bgp summary vrf all

Nexus-POD1-spine2# show bfd neighbor vrf all

Nexus-POD1-spine2# show bfd ipv6 neighbor vrf all