Verify Network Convergence System (NCS) 2000 Database Integrity

Introduction

This document describes the best practices regarding procedures involving reset, reseat, and replacement of controller cards in the NCS 2000.

Prerequisites

Requirements

There are no specific requirements for this document.

Components Used

The information in this document is based on these software and hardware versions:

• NCS2006-SA
• NCS2K-TNCS-O-K9
• NCS2006-DC20
• M6-FTA2
• NCS2006-LCD
• M6-ECU2
• Cisco Transport Controller (CTC) release 11.123-022-D2911-S-SPA

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Background Information

The NCS 2000 stores all system provisioning on the shelf 1 controller cards in a provisioning database, with a backup database on the LCD module. All other shelves within a multi-shelf node do not store a copy of the database. These practices help to mitigate risk during operations requiring changes to the node controller (shelf 1). Without taking these steps, specific conditions present on the system or controller cards in rare circumstances lead to a loss of the provisioning database, resulting in the loss of all services on the node.

Supported NCS 2000 controller cards include:

• TNC-E
• TSC-E
• TNCS
• TNCS-O
• TNCS2
• TNCS2-O

Procedure

Before taking any action requiring a change to the controller card state, perform these pre-checks. Changes to the controller card can include:

• Soft reset via Cisco Transport Controller (CTC) or the command line
• Cold restarted via the command line
• Removal and re-insertion
• Replacement
• Software upgrade

Step 1. Check for Standing Alarms and Conditions

In CTC check the Alarms and Conditions tabs in order to verify none of these conditions exist on the system, controller cards, or LCD module. Ensure the filter in the bottom right corner does not prevent the display of any entries. This does not represent an exhaustive list. For additional information on these and other alarms, refer to the Cisco NCS 2000 Series Troubleshooting Guide.

• Active Volume Software Signature Verification Failed
• Protect Volume Software Signature Verification Failed
• Software Download Failed
• Software Mismatch
• Bad Database Detected
• Standby Database Out of Synchronization
• Improper Removal
• Protection Unit Not Available
• No Valid USB DB
• USB Mount Fail
• USB Synchronization

If possible, clear these alarms before proceeding with any changes to the active controller card.

Step 2. Check Database Integrity

In order to verify database integrity on the node, connect to the VxWorks command line with telnet or SSH. In order to determine the available access type, check Provisioning > Security > Access > Shell Access.

NCS 2000 Shell Access Configuration

Caution: Cisco does not recommend the use of the VxWorks command line for troubleshooting or diagnostic purposes. Improper use of VxWorks commands can result in unexpected system behavior, up to and including loss of system provisioning.

• Log in with Superuser credentials
• Enter enablec in order to allow the break sequence, using Ctrl+C.

-> enablec
value = 3 = 0x3

Verify Node Information

• Verify the node name, IP address, and software version with flmStat.
  
  

-> flmStat

 System Database: /myst/system.db
        NodeName: 154-SMR20
    Node IP addr: 10.1.1.154     maskLen: 24
  Secure IP addr: 10.1.1.154     maskLen: 24
        Rtr addr: 10.1.1.1
         version: 9
       IIOP port: 57790
     SSLIOP port: 57791
        shelf ID: 1
multishelf enbld: Y
  multishelf LAN: 1
SwDwld fail flag: 0

Current running software is:
Software version: 11.123-022-D2911-S-SPA
Package: NCS2K_F_SONET

This is TNC-1, redState: FULL ACT

      Other TCC is: Present and Alive

Active Code Volume: 0,  avail: 1
    Flash0 release: 11.123-022-D2911-S-SPA  qualified: YES  sel: YES  pkg: NCS2K_F_SONET
    Flash1 release: 11.13-023E-16.17-S-SPA  qualified: YES  sel:  NO  pkg: NCS2K_F_SONET
      No Init Sync: NO

Active Database is: J,  avail: A
     active seqnum: 34126
      avail seqnum: 34117

value = 0 = 0x0

Verify LCD Status

• Ensure the LCD has the status FULL ACTIVE in the output of flmStat2.

-> flmStat2

This is TNC-1, type: 7, redState: FULL ACT

    Running Software version: 11.123-022-D2911-S-SPA
    Package: NCS2K_F_SONET

    Flash0 release: 11.123-022-D2911-S-SPA K  qualified: YES  sel: YES  pkg: NCS2K_F_SONET
    Flash1 release: 11.13-023E-16.17-S-SPA K  qualified: YES  sel:  NO  pkg: NCS2K_F_SONET
      No Init Sync: NO

         Databases: DB  Seq#  Release
            Active:  H 34244   11.123-022-D2911-S-SPA
             Avail:  I 34235   11.123-022-D2911-S-SPA
            Revert:  R 30593   11.123-022-D2911-F-SPA

        NodeName: 154-SMR20
    Node IP addr: 10.1.1.154     maskLen: 24
  Secure IP addr: 10.1.1.154     maskLen: 24
        Rtr addr: 10.1.1.1

Other is TNC-2, type 5, Present and Alive

 USB Flash is FULL ACTIVE

    USB Flash0 release: 11.123-022-D2911-S-SPA K  qualified: YES  sel: YES
    USB Flash1 release: 11.13-023E-16.17-S-SPA K  qualified: YES  sel: NO
     USB Databases: DB  Seq#  Release
            Active:  A 34244   11.123-022-D2911-S-SPA
             Avail:  B 34235   11.123-022-D2911-S-SPA
            Revert:  R 30593   11.123-022-D2911-F-SPA

value = 0 = 0x0

Identify Active Database Information

• Enter flmDbAll in order to view the current provisioning and system databases.

-> flmDbAll

Database ==> /DBJ/prov.db       status: ACTIVE
  Node Name: 154-SMR20   IP Address: 10.1.1.154
 SizetDbHdr: 188                SizetDb: 1153209
     Seqnum: 34126
     CRCHdr: 3d31a07d           CRCDb: 2a4cac02
    Node Id: 1b35d760           Ne Type: 12
    MacAddr: f4:f:1b:35:d7:60
  Sys Swrel: 11.123-022-D2911-S-SPA
 LastChange: 9-26-2023 12:12:2

Database ==> /DBA/prov.db       status: AVAIL
  Node Name: 154-SMR20   IP Address: 10.1.1.154
 SizetDbHdr: 188                SizetDb: 1153221
     Seqnum: 34117
     CRCHdr: 4fc81200           CRCDb: ad3b314a
    Node Id: 1b35d760           Ne Type: 12
    MacAddr: f4:f:1b:35:d7:60
  Sys Swrel: 11.123-022-D2911-S-SPA
 LastChange: 9-26-2023 10:49:5

Database ==> /DBB/prov.db       status: AVAIL
  Node Name: 154-SMR20   IP Address: 10.1.1.154
 SizetDbHdr: 188                SizetDb: 1153221
     Seqnum: 34118
     CRCHdr: 92cc5796           CRCDb: 818fa953
    Node Id: 1b35d760           Ne Type: 12
    MacAddr: f4:f:1b:35:d7:60
  Sys Swrel: 11.123-022-D2911-S-SPA
 LastChange: 9-26-2023 10:57:53

Database ==> /DBC/prov.db       status: AVAIL
  Node Name: 154-SMR20   IP Address: 10.1.1.154
 SizetDbHdr: 188                SizetDb: 1153221
     Seqnum: 34119
     CRCHdr: 22869e45           CRCDb: da4dba9a
    Node Id: 1b35d760           Ne Type: 12
    MacAddr: f4:f:1b:35:d7:60
  Sys Swrel: 11.123-022-D2911-S-SPA
 LastChange: 9-26-2023 11:4:4

Database ==> /DBD/prov.db       status: AVAIL
  Node Name: 154-SMR20   IP Address: 10.1.1.154
 SizetDbHdr: 188                SizetDb: 1153209
     Seqnum: 34120
     CRCHdr: fbb74309           CRCDb: e2aee96d
    Node Id: 1b35d760           Ne Type: 12
    MacAddr: f4:f:1b:35:d7:60
  Sys Swrel: 11.123-022-D2911-S-SPA
 LastChange: 9-26-2023 11:12:2

Database ==> /DBE/prov.db       status: AVAIL
  Node Name: 154-SMR20   IP Address: 10.1.1.154
 SizetDbHdr: 188                SizetDb: 1153209
     Seqnum: 34121
     CRCHdr: c5e4bdbb           CRCDb: 79957918
    Node Id: 1b35d760           Ne Type: 12
    MacAddr: f4:f:1b:35:d7:60
  Sys Swrel: 11.123-022-D2911-S-SPA
 LastChange: 9-26-2023 11:19:5

Database ==> /DBF/prov.db       status: AVAIL
  Node Name: 154-SMR20   IP Address: 10.1.1.154
 SizetDbHdr: 188                SizetDb: 1153209
     Seqnum: 34122
     CRCHdr: a259ae4a           CRCDb: a042ba4a
    Node Id: 1b35d760           Ne Type: 12
    MacAddr: f4:f:1b:35:d7:60
  Sys Swrel: 11.123-022-D2911-S-SPA
 LastChange: 9-26-2023 11:34:4

Database ==> /DBG/prov.db       status: AVAIL
  Node Name: 154-SMR20   IP Address: 10.1.1.154
 SizetDbHdr: 188                SizetDb: 1153209
     Seqnum: 34123
     CRCHdr: f418d972           CRCDb: 8eb25209
    Node Id: 1b35d760           Ne Type: 12
    MacAddr: f4:f:1b:35:d7:60
  Sys Swrel: 11.123-022-D2911-S-SPA
 LastChange: 9-26-2023 11:36:29

Database ==> /DBH/prov.db       status: AVAIL
  Node Name: 154-SMR20   IP Address: 10.1.1.154
 SizetDbHdr: 188                SizetDb: 1153209
     Seqnum: 34124
     CRCHdr: e68d1dd1           CRCDb: 799bc08d
    Node Id: 1b35d760           Ne Type: 12
    MacAddr: f4:f:1b:35:d7:60
  Sys Swrel: 11.123-022-D2911-S-SPA
 LastChange: 9-26-2023 11:49:4

Database ==> /DBI/prov.db       status: AVAIL
  Node Name: 154-SMR20   IP Address: 10.1.1.154
 SizetDbHdr: 188                SizetDb: 1153209
     Seqnum: 34125
     CRCHdr: 42d39a79           CRCDb: 643378ab
    Node Id: 1b35d760           Ne Type: 12
    MacAddr: f4:f:1b:35:d7:60
  Sys Swrel: 11.123-022-D2911-S-SPA
 LastChange: 9-26-2023 12:4:4

Database ==> /DBR/prov.db       status: REVERT
  Node Name: 154-SMR20   IP Address: 10.1.1.154
 SizetDbHdr: 188                SizetDb: 1091869
     Seqnum: 30593
     CRCHdr: e368fd7b           CRCDb: a8bf1b35
    Node Id: 1b35d760           Ne Type: 12
    MacAddr: f4:f:1b:35:d7:60
  Sys Swrel: 11.123-022-D2911-F-SPA
 LastChange: 8-15-2023 15:29:35

 System Database: /myst/system.db
        NodeName: 154-SMR20
    Node IP addr: 10.1.1.154     maskLen: 24
  Secure IP addr: 10.1.1.154     maskLen: 24
        Rtr addr: 10.1.1.1
         version: 9
       IIOP port: 57790
     SSLIOP port: 57791
        shelf ID: 1
multishelf enbld: Y
  multishelf LAN: 1
SwDwld fail flag: 0
value = 0 = 0x0

• Note the letter designation (DBJ in this instance) Seqnum, SizetDb, and CRCDb values from the database with ACTIVE status.
• For the REVERT database (DBR) note the software version, Seqnum, SizetDb, and CRCDb.

Log in to Standby Controller

• Log in to the standby controller with telnet "otherTcc".
  

-> telnet "otherTcc"

In order to discontinue logging in, type control-d or wait 60 seconds.
After logging in, type 'enablec' to enable abort key (control-c).

-> enablec
value = 3 = 0x3

• Verify you have entered the standby controller with flmStat.

-> flmStat

 System Database: /myst/system.db
        NodeName: 154-SMR20
    Node IP addr: 10.1.1.154     maskLen: 24
  Secure IP addr: 10.1.1.154     maskLen: 24
        Rtr addr: 10.1.1.1
         version: 9
       IIOP port: 57790
     SSLIOP port: 57791
        shelf ID: 1
multishelf enbld: Y
  multishelf LAN: 1
SwDwld fail flag: 0

Current running software is:
Software version: 11.123-022-D2911-S-SPA
Package: NCS2K_F_SONET

This is TNC-2(1), redState: FULL SBY

      Other TCC is: Present and Alive

Active Code Volume: 0,  avail: 1
    Flash0 release: 11.123-022-D2911-S-SPA  qualified: YES  sel: YES  pkg: NCS2K_F_SONET
    Flash1 release: 11.13-023E-16.17-S-SPA  qualified: YES  sel:  NO  pkg: NCS2K_F_SONET
      No Init Sync: NO

Active Database is: H,  avail: I
     active seqnum: 34126
      avail seqnum: 34117

value = 0 = 0x0

Verify Standby Controller Database Information

• Again enter flmDbAll in order to confirm the active database information matches between the active and standby controllers. The letter designation does not need to match, but the Seqnum, SizetDb, and CRCDb values must match exactly.

-> flmDbAll

Database ==> /DBH/prov.db       status: ACTIVE
  Node Name: 154-SMR20   IP Address: 10.1.1.154
 SizetDbHdr: 188                SizetDb: 1153209
     Seqnum: 34126
     CRCHdr: 3d31a07d           CRCDb: 2a4cac02
    Node Id: 1b35d760           Ne Type: 12
    MacAddr: f4:f:1b:35:d7:60
  Sys Swrel: 11.123-022-D2911-S-SPA
 LastChange: 9-26-2023 12:12:2

Database ==> /DBI/prov.db       status: AVAIL
  Node Name: 154-SMR20   IP Address: 10.1.1.154
 SizetDbHdr: 188                SizetDb: 1153221
     Seqnum: 34117
     CRCHdr: 4fc81200           CRCDb: ad3b314a
    Node Id: 1b35d760           Ne Type: 12
    MacAddr: f4:f:1b:35:d7:60
  Sys Swrel: 11.123-022-D2911-S-SPA
 LastChange: 9-26-2023 10:49:5

///

Database ==> /DBR/prov.db       status: REVERT
  Node Name: 154-SMR20   IP Address: 10.1.1.154
 SizetDbHdr: 188                SizetDb: 1091869
     Seqnum: 30593
     CRCHdr: e368fd7b           CRCDb: a8bf1b35
    Node Id: 1b35d760           Ne Type: 12
    MacAddr: f4:f:1b:35:d7:60
  Sys Swrel: 11.123-022-D2911-F-SPA
 LastChange: 8-15-2023 15:29:35

 System Database: /myst/system.db
        NodeName: 154-SMR20
    Node IP addr: 10.1.1.154     maskLen: 24
  Secure IP addr: 10.1.1.154     maskLen: 24
        Rtr addr: 10.88.130.1
         version: 9
       IIOP port: 57790
     SSLIOP port: 57791
        shelf ID: 1
multishelf enbld: Y
  multishelf LAN: 1
SwDwld fail flag: 0
value = 0 = 0x0
->

• If the database information does not match, do not proceed with any changes to the active controller card.
• Enter logout in order to return to the active controller card.
• As the database can update at any time, confirm the sequence number has not been updated between issuing the first and second flmDbAll commands. This can lead to an apparent mismatch of database information.

Verify LCD Database Information

• Verify the backup database on the LCD module likewise matches with the active controller using flmUSBDbAll.

-> flmUSBDbAll

Database ==> /uDBH/prov.db       status: ACTIVE
  Node Name: 154-SMR20   IP Address: 10.1.1.154
 SizetDbHdr: 188                SizetDb: 1153209
     Seqnum: 34126
     CRCHdr: 3d31a07d           CRCDb: 2a4cac02
    Node Id: 1b35d760           Ne Type: 12
    MacAddr: f4:f:1b:35:d7:60
  Sys Swrel: 11.123-022-D2911-S-SPA
 LastChange: 9-26-2023 12:12:2

Database ==> /uDBI/prov.db       status: AVAIL
  Node Name: 154-SMR20   IP Address: 10.1.1.154
 SizetDbHdr: 188                SizetDb: 1153221
     Seqnum: 34117
     CRCHdr: 4fc81200           CRCDb: ad3b314a
    Node Id: 1b35d760           Ne Type: 12
    MacAddr: f4:f:1b:35:d7:60
  Sys Swrel: 11.123-022-D2911-S-SPA
 LastChange: 9-26-2023 10:49:5

///

Database ==> /uDBR/prov.db       status: REVERT
  Node Name: 154-SMR20   IP Address: 10.1.1.154
 SizetDbHdr: 188                SizetDb: 1091869
     Seqnum: 30593
     CRCHdr: e368fd7b           CRCDb: a8bf1b35
    Node Id: 1b35d760           Ne Type: 12
    MacAddr: f4:f:1b:35:d7:60
  Sys Swrel: 11.123-022-D2911-F-SPA
 LastChange: 8-15-2023 15:29:35

 System Database: /umyst/system.db
        NodeName: 154-SMR20
    Node IP addr: 10.1.1.154     maskLen: 24
  Secure IP addr: 10.1.1.154     maskLen: 24
        Rtr addr: 10.1.1.1
         version: 9
       IIOP port: 57790
     SSLIOP port: 57791
        shelf ID: 1
multishelf enbld: Y
  multishelf LAN: 1
SwDwld fail flag: 0
value = 0 = 0x0
->

Step 3. Backup the Database

Before making a change to the active controller, backup the database before proceeding. In CTC navigate to Maintenance > Database > Backup. Save the file to an external location as a txt or bin file.

NCS 2000 Database Backup

If the database backup fails from CTC, contact Cisco TAC for alternative methods to backup the database.

Step 4. Verify Database Backup

After retrieving the database from the node, open the file in order to verify that the header matches the expected format. This output provides an example of a functional database header.

OTBU01000271
Info.FileType=Cisco ONS NE Database File
Info.Version=1.0
Info.NeDesc=Cisco NCS2006 FLEX
Info.NeType=NCS2006 FLEX
Ne.Software-Version=11.123-022-D2911-S-SPA
Ne.Node-Name=154-SMR20
Ne.Node-Id=0x1b35d760
Ne.Date=09/26/2023
Ne.Time=13:28:12 CST
Ne.Secure-Mode=N

Additional Best Practices

In situations with a risk of database loss, such as the presence of a system alarm noted earlier, exercise caution before taking any action on the active controller. Take these steps after taking a database backup if possible.

Remove Standby Controller

In order to prevent unwanted database loss or reversion after a reset or restart of the active controller, remove the standby controller from the node and leave it unseated.

Remove LCD Module

In specific situations, the loss of the active database can lead to reversion to a previous database stored on the LCD module. Remove the LCD module and leave it unseated.

Remove MultiShelf Management (MSM) Cables

In the event of complete database loss, all subtended shelves return to the default configuration, resulting in the loss of all services on the node. Remove the MSM cables in order to mitigate this risk. The subtended shelves continue to operate in headless mode for the duration.

Match Active and Protect Software Versions

The REVERT database uses the software listed in Protect Version in CTC under Maintenance > Software. Database reversion or software downgrade restores the provisioning database associated with the protect software. Ensure the software versions are compatible. For example, the SSON (-S-SPA) software cannot interoperate with the Dense Wavelength Division Multiplexing (DWDM) software so do not have these versions present at the same time. Download the same SSON software for the Protect Version in order to prevent the loss of the database on a reversion.

Complete the reset or restart of the active controller after completing these best practices, then restore the node to normal operation:

• Insert the standby controller after confirming the successful boot of the active controller card.
• After the standby controller boots, verify the databases match between active and standby controllers.
• Insert the LCD module and wait for the USBSYNC condition to clear.
• Verify the LCD database synchronizes with the active controller.
• Plug in the MSM cables.