Guest

Cisco MGX 8200 Series Edge Concentrators

MGX 8250 and MGX 8850 (PXM1) - Boot Code and Firmware Graceful Upgrade Script

Cisco - MGX 8250 and MGX 8850 (PXM1) - Boot Code and Firmware Graceful Upgrade Script

Document ID: 6934

Updated: Apr 17, 2009

   Print

Introduction

This document describes the Cisco recommended 28-step process for an MGX 8850 Edge Switch graceful upgrade.

Before You Begin

Conventions

For more information on document conventions, see the Cisco Technical Tips Conventions.

Prerequisites

Graceful upgrades cause little or no service disruption and are recommended when upgrading:

  • To a compatible firmware version.

  • To a compatible database / Management Information Base (MIB) structure.

  • To a redundant MGX 8850 with two Processor Switch Modules (PXMs).

The MGX 8850 graceful upgrade uses the following commands. All commands are case sensitive.

Command Switch Software Upgrade Equivalent Function
install first loadrev to new version Loads the new firmware version.
newrev runrev to new version Executes the new firmware version. Results in a switchcc from the Active PXM / Primary Service Module to the Standby PXM / Secondary Service Module.
commit second loadrev to new version Completes the upgrade to the new firmware version. Graceful downgrade to original firmware version is lost.
abort loadrev to old version Restores the PXM to the original firmware version. Must be issued prior to the commit command. Not supported for Service Module firmware.

The MGX 8850 firmware provides redundancy by providing support for hot insertion and removal of the PXM module, as well as 1:1 hot standby redundancy for high availability of the MGX 8850. The active and standby PXM have exactly the same database in the local memory at any given time. The active PXM is responsible for updating the standby PXM whenever the database is changed. When the active PXM fails, the standby PXM will switchover in 100 milliseconds (msec). The switchover is transparent to the RPM and service modules.

In some cases, older firmware versions are incompatible with newer versions due to incompatible database structures or incompatible MIB structures, and the MGX 8850 Boot Code and Firmware Upgrade Script for Non-Redundant Switches script should be used. To determine compatibility, please refer to the Release Notes for the desired firmware.

The tasks listed in this document are recommended for redundant MGX 8850 firmware upgrades using two PXMs. The tasks were verified in the order shown in a laboratory test of a redundant MGX 8850 upgrade from release 1.1.21 to release 1.1.24. To maintain database integrity an interim PXM runtime firmware upgrade to release 1.1.23 was required. The path of the graceful upgrade was:

  • 1.1.21 -> 1.1.23 -> 1.1.24.

This document lists the minimum required steps, and then addresses each step in some detail. The MGX 8850 is based on the same platform as the MGX 8220, and it is recommended that the MGX 8220 Upgrade and Downgrade Matrices, Concepts and Definitions be reviewed to familiarize the reader with general upgrade concepts. The screen displays used to illustrate the tasks were taken from laboratory equipment and are in no way intended to specify Internet Protocol (IP) addressing or naming schemes.

caution Caution: 

  • Only one image must be loaded onto the PXM per Trivial File Transfer Protocol (TFTP) session.

  • Multiple TFTP sessions are required to load boot code and firmware images onto a PXM.

  • If multiple firmware images are loaded in one TFTP session, all files copied after the initial image will be corrupted.

  • This document is intended to be used as an aid for conducting successful firmware upgrades, but is not a substitute for proper planning with your Cisco Sales Engineer, Systems Engineer, or Account Manager.

Components Used

The information in this document is based on the software and hardware versions below.

  • Graceful PXM runtime firmware upgrades are not supported from release 1.1.21 to release 1.1.24. This document includes an interim PXM runtime firmware upgrade to 1.1.23, which ensures database integrity and user traffic continuity.

  • No graceful downgrade is supported from release 1.1.24 or later to version 1.1.21 or below due to MIB changes.

Background

This section explains IP addressing on the MGX 8850 shelf in general. There are three separate IP addresses for on MGX 8850 shelf with two PXMs.

  • One cnfifip IP address, also known as the shelf IP address

  • Two bootChange IP addresses, also known as the PXM IP address

The cnfifip IP address or the shelf IP address is the live IP Address of the Active PXM Ethernet port on the MGX 8850. It is the IP address used to manage the MGX 8850 shelf. If a switchcc occurs, the new MAC address of the Standby PXM card is automatically broadcast out and takes over the cnfifip IP address.

To verify the existing IP address, issue the dspifip command. The dspifip output also displays the ATM and SLIP addresses assigned to the MGX 8850 shelf.

  • The ATM address is used for inband IP routing (NWIP) management of the MGX 8850 shelf.

  • The SLIP address is a legacy assigned to the MGX 8850.

The SLIP interface does not support statistics collection. The cnfifip and bootChange IP addresses are retained after the clrallcnf command is issued.

bootChange is a Service-level command that is used as necessary for MGX 8850 bring up when the PXMs have no run time firmware. The bootChange IP address or the PXM IP address should be different than the cnfifip IP address.

The bootChange IP address of the Active PXM should also be different than the bootChange IP address of the Standby PXM. The bootChange IP address is active only when the PXM is in the boot mode or when the PXM is in the Standby mode and is used to load firmware and boot code directly into the PXM. Refer to Bringing up the PXM with no runtime firmware for more information. Once the PXM is booted up, the cnfifip IP address is active. The bootChange gateway address specifies the next hop that allows the shelf to communicate with a laptop (PC) or Cisco WAN Manager (CWM) station on a different LAN segment while the MGX 8850 is in the boot mode. To view the bootChange IP address of the PXM when the MGX 8850 shelf is using runtime firmware, issue the version command.

sj_core.1.7.PXM.a > bootChange 
'.' = clear field;  '-' = go to previous field;  ^D = quit 

boot device          : lnPci 
processor number     : 0 
host name            : solwandbg1 
file name            : 
inet on ethernet (e) : 10.1.2.15:ffffff00 
inet on backplane (b): 
host inet (h)        : 
gateway inet (g)     : 10.1.1.1 
user (u)             : autoprog 
ftp password (pw) (blank = use rsh): 
flags (f)            : 0x0 
target name (tn)     : pxm-7 
startup script (s)   : 
other (o)            : 
  

sj_core.1.7.PXM.a > dspifip 

    Interface        Flag  IP Address       Subnetmask       Broadcast Addr 
    ---------------  ----  ---------------  ---------------  --------------- 
    Ethernet/lnPci0  UP    10.1.2.44        255.255.255.0    10.1.1.1 
    SLIP/sl0         DOWN  0.0.0.0          255.255.255.0    (N/A) 
    ATM/atm0         DOWN  0.0.0.0          255.255.255.0    0.0.0.0 

sj_core.1.7.PXM.a >

To assign a bootChange IP address to the Standby PXM, issue the Service level shellCon command and the bootChange command. The Ethernet port of the Standby PXM must be cabled to a hub or similar network device to load files using the bootChange IP address. Cisco recommends using two LAN connections when loading the ComMat.dat file onto the Active and Standby PXMs. If you only use one LAN connection, move the cable from the Active PXM to the Standby PXM to download the ComMat.dat file.

sj_core.1.7.PXM.a >cc 8 
(session redirected) 

sj_core.1.8.PXM.s >shellCon 

-> bootChange

To abort the command use Ctrl-C. To exit from the shellCon mode issue quit.

Task Detail

Stage 1: Planning

The following summarizes the planning steps that are necessary for a successful upgrade. All steps should be completed irrespective of network size.

  1. Evaluate known anomalies in the selected release.

    Some anomalies may require additional preparation in order to ensure a smooth upgrade. This may mean:

    • Additional upgrade steps

    • Parameter changes

    • Workarounds

  2. Review release notes for upgrade steps specific to this release.

    As in Task 1, this task may result in:

    • Additional upgrade steps

    • Parameter changes

    • Workarounds

  3. Write scripts, which is an optional task to aid the parameter changes required in certain sections of Stage 3.

    Writing and testing scripts will:

    • Make the parameter change process easier to execute

    • Highlight any commands that have changed in the new firmware release.

    There are various products that can be used to aid in setting parameters in preparation for a network upgrade.

Stage 2: Network Preparation

The following summarizes the network preparation steps that are necessary for a successful upgrade. All steps should be completed irrespective of network size.

Note: This stage needs to be completed one week before firmware upgrade.

  1. Network health check.

    See Appendix A.

  2. Monitor network closely until time of upgrade.

    Step 1 should highlight any existing network issues, but it is prudent to monitor the network for new firmware errors and card errors right up to the time of the upgrade. Report recurring errors to Cisco TAC.

    See Appendix A for details on checking for firmware errors and card errors.

  3. Verify network management connectivity to network nodes.

    Ensure that every network MGX 8850 shelf can be connected to using Out of Band access. Using TELNET, connect to each MGX 8850 in the network.

  4. Verify the CardState of both PXMs.

    Verify that one PXM is Active and the other Standby. Issue the dspcds command to verify the state of both PXMs. If the PXM states are not Active and Standby, do not proceed with the upgrade.

    A sample dspcds output that displays the correct state of both PXMs is provided below. Note that for this document, only the first page of dspcds output is provided.

    jet.1.7.PXM.a > dspcds 
    
        Slot  CardState    CardType     CardAlarm  Redundancy 
        ----  -----------  --------     ---------  ----------- 
        1.1   Active       FRSM-2E3     Clear 
        1.2   Active       FRSM-2CT3    Clear 
        1.3   Active       FRSM-2E3     Clear 
        1.4   Active       VISM-8T1     Clear 
        1.5   Empty                     Clear 
        1.6   Empty                     Clear 
        1.7   Active       PXM1-OC3     Clear 
        1.8   Standby      PXM1-OC3     Clear 
        1.9   Empty                     Clear 
        1.10  Active       RPM          Clear 
        1.11  Active       VISM-8E1     Clear 
        1.12  Empty                     Clear 
        1.13  Empty                     Clear 
        1.14  Empty                     Clear 
        1.15  Empty                     Clear 
        1.16  Empty                     Clear 
        1.17  Empty                     Clear 
        1.18  Empty                     Clear 
        1.19  Empty                     Clear 
    
    Type <CR> to continue, Q<CR> to stop: 
  5. Verify bootChange address configuration on each of the PXMs.

    Use the Service level bootChange command to assign a unique IP address to each PXM in the MGX 8850 shelf. The bootChange IP address is used to load runtime firmware onto a PXM. The bootChange IP address must also be different from the IP address assigned to the MGX 8850 shelf using the cnfifip command.

    jet.1.7.PXM.a > bootChange 
    '.' = clear field;  '-' = go to previous field;  ^D = quit 
    
    boot device          : lnPci 
    processor number     : 0 
    host name            : solwandbg1 
    file name            : 
    inet on ethernet (e) : 192.168.1.65:ffffff00 
    inet on backplane (b): 
    host inet (h)        : 
    gateway inet (g)     : 192.168.1.1 
    user (u)             : autoprog 
    ftp password (pw) (blank = use rsh): 
    flags (f)            : 0x0 
    target name (tn)     : pxm-7 
    startup script (s)   : 
    other (o)            : 
    

    To verify the bootChange IP address of the Active PXM issue the version command.

    jet.1.7.PXM.a > version 
    VxWorks (for POPEYE) version 5.3.1. 
    Kernel: WIND version 2.5 
    Made on Mar 30 1999, 12:20:01. 
    Boot line: 
    lnPci(0,0)solwandbg1: e=192.168.1.65 g=192.168.1.1 u=autoprog tn=pxm-7
    PXM firmware version : 1.0.00
    Boot Image version   : 1.0.00Dc1
    

    To assign bootChange IP address to the Standby PXM, issue the Service level shellCon command and then use the bootChange command.

    jet.1.7.PXM.a >cc 8 
    (session redirected) 
    
    jet.1.7.PXM.s >shellCon 
    
    -> 
    -> bootChange 
    bootChange 
    
    '.' = clear field;  '-' = go to previous field;  ^D = quit 
    
    boot device          : lnPci 
    processor number     : 0 
    host name            : solwandbg1 
    file name            : 
    inet on ethernet (e) : 192.168.1.30:ffffff00 
    inet on backplane (b): 
    host inet (h)        : 
    gateway inet (g)     : 192.168.1.1 
    user (u)             : autoprog 
    ftp password (pw) (blank = use rsh): 
    flags (f)            : 0x0 
    target name (TN)     : pxm-7 
    startup script (s)   : 
    other (o)            : 
    
    value = 0 = 0x0 
    -> quit 
    quit 
    
    (session resumed) 
    
    jet.1.8.PXM.s >  version 
    VxWorks (for POPEYE) version 5.3.1. 
    Kernel: WIND version 2.5. 
    Made on Jun  6 2000, 23:05:55. 
    Boot line: 
    lnPci(0,0)solwandbg1: e=192.168.1.30:ffffff00 g=192.168.1.1 u=autoprog TN=pxm7 
    PXM firmware version : 1.1.21 
    Boot Image Version   : 1.1.21
    

    Issue the cnfifip command to assign the IP address used to connect to the MGX 8850 shelf. The IP address assigned by the cnfifip command is the IP address used to connect to the MGX 8850 when the shelf is in a normal operating state.

    jet.1.7.PXM.a > cnfifip 26 192.168.1.23 255.255.255.0 192.168.1.255
    

    To verify the shelf IP address issue the dspifip command.

    jet.1.7.PXM.a > dspifip 
        Interface        Flag  IP Address       Subnetmask       Broadcast Addr 
        ---------------  ----  ---------------  ---------------  --------------- 
        Ethernet/lnPci0  UP    192.168.1.23     255.255.255.0    192.168.1.255 
        SLIP/sl0         DOWN  0.0.0.0          255.255.255.0    (N/A) 
        ATM/atm0         DOWN  0.0.0.0          255.255.255.0    0.0.0.0
    

    The ATM address is used for in-band management of the MGX 8850 shelf over the feeder trunk to the Cisco BPX 8600 Series Switch.

Stage 3: The Upgrade

The following summarizes the steps that are necessary for a successful upgrade. All steps should be completed irrespective of network size.

  1. Provisioning freeze starts.

    Halt provisioning of new services until completion of upgrade.

  2. As a precautionary step, save MGX 8850 PXM and Service Module (SM) configuration.

    Save a snapshot of the MGX 8850 configuration on a CWM (SV+) workstation. If the MGX 8850 configuration is not saved, the entire configuration must be manually re-entered.

    jet.1.7.PXM.a > saveallcnf 
    jet.1.7.PXM.a > ll C:/CNF 
      size          date       time       name 
    --------       ------     ------    -------- 
         512    MAY-21-1999  17:46:12   .                 <
    DIR> 
         512    MAY-21-1999  17:46:12   ..                <DIR> 
      182762    JUL-06-2000  15:33:45   jet_1533000602.zip 
      182762    JUL-06-2000  15:33:48   jet.zip 
    
    In the file system : 
        total space :  819200 K bytes 
        free  space :  712933 K bytes 
    

    From the TFTP server issue the following commands to save the configuration file to the server. The TFTP server can be a Unix workstation or CWM workstation.

    unix-prompt>tftp 192.168.1.23 
    tftp>bin 
    tftp>get CNF/jet_1533000602.zip 
    Received 182762 bytes in 2.4 seconds 
    tftp>quit 
    
    
  3. View and record card errors and clear all error log files.

    On all nodes to be upgraded record the card errors and clear the card errors using the following commands on the respective cards:

         dspcderrs on the PXM, FRSM, AUSM, VISM, CESM. 
         clrcderrs on the FRSM, AUSM. 
         clrerr on the PXM. 
         clrlog on the PXM.  
    
  4. Load new revision into CWM (SV+) stations.

    Load new firmware version into CWM (SV+) stations. Verify that the images have loaded successfully by comparing file sizes to those listed in the Firmware Release Notes.

  5. Remove the cause of all MAJOR alarms and if possible all MINOR alarms.

    Ideally, the network should be alarm free at the time of the firmware upgrade. If this is not possible, at least the reason for all major alarms should be identified and noted, and then suitable reconfiguration should be made in order to remove the alarm. Verify connection totals by issuing the dsptotals command as described in Appendix A.

    Any minor alarms should be noted so that, after the upgrade, a comparison can be made.

  6. Load target boot code revision into PXM.

    Upload the new PXM boot code to the MGX 8850 using the TFTP process and verify the checksum. The byte count and checksum below is just an example. It will be different for different images. For this test, the intermediate PXM boot code version of 1.1.23 is not required.

    unix-prompt>tftp 192.168.1.23 
    tftp>bin 
    tftp>put pxm_bkup_1.1.24.fw POPEYE@PXM.BT 
    Sent 1274256 bytes in 7.2 seconds 
    tftp>quit
     
    jet.1.7.PXM.a > 
    Program length = 1274256 
    Calculated checksum = 0xb5fb283e stored checksum = 0xb5fb283e 
    Fw checksum passed

    The PXM executes boot code sequentially, so if there is an older image loaded, the PXM will execute the oldest image. To avoid this problem, either delete the existing boot code image or rename the filename with a .old extension. If the existing boot code image is renamed, the FW directory contents will have two boot code files, one with a .old extension. A sample FW directory is provided below. To view the contents of the FW directory; from the C: drive issue the cd FW command and then the ll command. The current boot code file and two old boot code files are highlighted.

    jet.1.7.PXM.a > ll
    size          date       time       name 
    --------       ------     ------    -------- 
         512    JUL-21-2000  17:13:30   .                 <DIR> 
         512    JUL-21-2000  17:13:30   ..                <DIR> 
     2105328    JUL-20-2000  14:30:12   pxm_1.1.11_fw.old
    620368    JUL-20-2000  16:49:48   sm90.fw 
    799440    MAY-11-2000  18:53:24   sm35.fw 
    1178168    MAY-11-2000  18:54:40   sm50.fw 
    934356    JUL-21-2000  11:47:08   sm130.fw 
    1246872    JUL-20-2000  15:54:40   pxm_bkup_1.1.12.old 
    21    JUL-24-2000  15:58:44   ComMat.dat
    1265620    JUL-24-2000  10:36:14   pxm_bkup_1.1.21.old
    1253388      NOV-16-1999  06:42:38 pxm_bkup_1.1.13.fw 
    1246872      OCT-20-1999  11:07:28 pxm_bkup_1.1.12.old 
    2105328      OCT-20-1999  11:58:34 pxm_1.1.11.fw 
    644624       OCT-20-1999  12:07:38 pxm_bkup_1.1.01.old 
    2006664      OCT-20-1999  12:02:16 pxm_1.1.01.fw 
    2117676      NOV-16-1999  06:45:22 pxm_1.1.12.fw 
    1274256    JUL-24-2000  13:42:42   pxm_bkup_1.1.24.fw 
    2183088    JUL-24-2000  13:47:42   pxm_1.1.24.fw 
    2182548    JUL-24-2000  14:45:18   pxm_1.1.23.fw 
    
    In the file system : 
    total space :  819200 K bytes 
    free  space :  727272 K bytes 

    Note: The firmware files displayed using the ll command are a superset of the firmware files displayed by the dspfwrev command.

    jet.1.7.PXM.a > dspfwrevs 
    Card Type   Date       Time     Size     Version             File Name 
    ----------- ------------------- -------- ------------------- ------------------ 
    CESM-8T1E1  07/20/2000 16:49:48 620368   10.0.04             sm90.fw 
    FRSM-8T1E1  05/11/2000 18:53:24 799440   10.0.11             sm35.fw 
    AUSM-8T1E1  05/11/2000 18:54:40 1178168  10.0.11             sm50.fw 
    FRSM-VHS    07/21/2000 11:47:08 934356   10.0.11             sm130.fw 
    PXM1        07/24/2000 11:21:48 2147060  1.1.21              pxm_1.1.21.fw 
    VISM-8T1E1  07/24/2000 12:04:34 1315400  1.0.02              sm150.fw 
    PXM1        07/24/2000 13:42:42 1274256  1.1.24              pxm_bkup_1.1.24.fw 
    PXM1        07/24/2000 13:47:42 2183088  1.1.24              pxm_1.1.24.fw 
    PXM1        07/24/2000 14:45:18 2182548  1.1.23              pxm_1.1.23.fw

    The newly uploaded firmware files will be automatically replicated on to the Standby PXM in few seconds. To verify the files on the Standby PXM, issue the following commands:

    1. cc <card_number>

    2. CD FW

    3. ll

    The listing of firmware images residing on the Standby PXM in slot 8 is provided below.

    jet.1.8.PXM.s > ll 
      size          date       time       name 
    --------       ------     ------    -------- 
         512    MAY-12-2000  00:03:16   .                 <DIR> 
         512    MAY-12-2000  00:03:16   ..                <DIR> 
     2105328    JUL-20-2000  14:30:12   pxm_1.1.11_fw.old 
      620368    JUL-20-2000  16:49:48   sm90.fw 
      799440    MAY-11-2000  18:53:24   sm35.fw 
     1178168    MAY-11-2000  18:54:40   sm50.fw 
      934356    JUL-21-2000  11:47:08   sm130.fw 
     1265620    JUL-24-2000  10:36:14   pxm_bkup_1.1.21.old 
     2147060    JUL-24-2000  11:21:48   pxm_1.1.21.fw 
          21    JUL-24-2000  15:58:44   ComMat.dat 
     1246872    JUL-20-2000  15:54:40   pxm_bkup_1.1.12.old 
     1315400    JUL-24-2000  12:04:34   sm150.fw 
     1274256    JUL-24-2000  13:42:42   pxm_bkup_1.1.24.fw 
     2183088    JUL-24-2000  13:47:42   pxm_1.1.24.fw 
     2182548    JUL-24-2000  14:45:18   pxm_1.1.23.fw 
    
    In the file system : 
        total space :  819200 K bytes 
        free  space :  682019 K bytes 
    
    jet.1.8.PXM.s >
    
  7. Load intermediate and target runtime firmware versions into PXMs.

    Upload the intermediate and target runtime firmware versions to the MGX 8850 using the TFTP process and verify the checksum. The byte count and checksum below is shown for illustration and the values will be different for other images. Note that for this test, both 1.1.23 and 1.1.24 versions of runtime firmware are loaded. Storing multiple versions of runtime firmware can be accomplished as long as the order of firmware upgrade steps is followed.

    unix-promt>tftp 192.168.1.23 
    tftp>bin 
    tftp>put pxm_1.1.23.fw POPEYE@PXM.FW 
    Sent 2182548 bytes in 10.4 seconds 
    tftp>quit 
    jet.1.7.PXM.a > 
    Program length = 2182548 
    Calculated checksum = 0xa65cb14f stored checksum = 0xa65cb14f 
    Fw checksum passed 
    
    unix-promt>tftp 192.168.1.23 
    tftp>bin 
    tftp>put pxm_1.1.24.fw POPEYE@PXM.FW 
    Sent 2182548 bytes in 10.4 seconds 
    tftp>quit 
    
    jet.1.7.PXM.a > 
    Program length = 2182548 
    Calculated checksum = 0xcb8h24ac stored checksum = 0xcb8h24ac 
    Fw checksum passed

    To verify the uploaded versions on each of the PXMs issue the dspfw command.

    jet.1.7.PXM.a > dspfw 
    PXM FW versions: 
    "1.1.21" in pxm_1.1.21.fw 
    "1.1.24" in pxm_1.1.24.fw 
    "1.1.23" in pxm_1.1.23.fw 
    
    jet.1.7.PXM.a > cc 8 
    
    (session redirected) 
    
    jet.1.8.PXM.s > dspfw 
    PXM FW versions: 
    
    "1.1.21" in pxm_1.1.21.fw 
    "1.1.24" in pxm_1.1.24.fw 
    "1.1.23" in pxm_1.1.23.fw
    
  8. Install the intermediate version ComMat.dat file on both the PXMs

    The ComMat.dat file contains compatibility matrix data that specifies the firmware version ranges that support graceful upgrades. Different versions of the ComMat.dat file can not be stored on the PXM. Each version of the ComMat.dat file will need to uploaded prior to each installation of runtime firmware. Upload the 1.1.23 ComMat.dat file and then copy to the C:/FW directory of the Active PXM.

    UNIX-prompt>tftp 192.168.1.23 
    tftp>bin 
    tftp>put ComMat.dat 
    Sent 21 bytes in 0.3 seconds 
    tftp>quit 
    jet.1.7.PXM.a > pwd 
    C: 
    
    jet.1.7.PXM.a >mv ComMat.dat C:/FW/ComMat.dat
    

    To upload the ComMat.dat file to the Standby PXM, use the bootChange IP address for the TFTP. The bootChange IP address is functional when the PXM is in the Standby state. Copy the ComMat.dat file to the C:/FW directory of the Standby PXM.

    UNIX-prompt>tftp 192.168.1.30 
    tftp>bin 
    tftp>put ComMat.dat 
    Sent 21 bytes in 0.3 seconds 
    tftp>quit 
    jet.1.8.PXM.s > pwd 
    C: 
    
    jet.1.8.PXM.s > MV ComMat.dat C:/FW/ComMat.dat
    
  9. If network has been stable for 30 minutes after the successful firmware downloads, install the boot code into the PXM flash.

    Issue the install bt command to upload the boot code file into the PXM flash memory. This command will download the boot code to both the Active and Standby PXM.

    jet.1.7.PXM.a > install bt "1.1.24" 
    writing pxm_bkup_1.1.24.fw to flash... 
    Board recognised as a PXM1B board ... 
    Checksum size is 1274256 ... 
    Erasing the flash .... 
    FLASH erase complete 
    Downloading C:/FW/pxm_bkup_1.1.24.fw into the flash ... 
    verifying flash contents .... 
    Flash ok .... 
    Flash download completed ... 
    copying pxm_bt_1.1.24.fw to standby... 
    writing flash on other card... 
    command completed OK on both pxms. 
    The new boot code will be used after the next reset
  10. Upgrade to the intermediate PXM runtime firmware version using the install, newrev, and commit commands.

    Issue the install 1.1.23 command to install the intermediate PXM runtime firmware. The Standby PXM will reset and go into the Hold state. This will take a few seconds.

    jet.1.7.PXM.a > install 1.1.23 
    this may take a while ... 
    install command completed OK 
    please wait for the other card to enter the hold state. 
    
    jet.1.7.PXM.a > dspcds 
    
      Slot  CardState    CardType     CardAlarm  Redundancy 
        ----  -----------  --------     ---------  ----------- 
        1.1   Active       FRSM-2E3     Clear 
        1.2   Active       FRSM-2CT3    Clear 
        1.3   Active       FRSM-2E3     Clear 
        1.4   Empty                     Clear 
        1.5   Empty                     Clear 
        1.6   Empty                     Clear 
        1.7   Active       PXM1-OC3     Clear 
        1.8   Hold         PXM1-OC3     Clear 
        1.9   Empty                     Clear 
        1.10  Active       RPM          Clear 
        1.11  Active       VISM-8E1     Clear 
        1.12  Empty                     Clear 
        1.13  Empty                     Clear 
        1.14  Empty                     Clear 
        1.15  Empty                     Clear 
        1.16  Empty                     Clear 
        1.17  Empty                     Clear 
        1.18  Empty                     Clear 
        1.19  Empty                     Clear 
    
    Type <CR> to continue, Q<CR> to stop:
    

    Issue the newrev 1.1.23 command after the Standby PXM is in the Hold state. After the newrev 1.1.23 command is issued, the Active PXM will reset and go to Hold state and the Standby PXM will be Active.

    jet.1.7.PXM.a > newrev 1.1.23
    reset type: 0x00000002 
    pio input: 0xf00f5771 
    Error EPC: 0x800c6e70 
    Status Reg: 0x3040ff05 
    Cause Reg: 0x00000000 
    CacheErr Reg: 0xb0000000 
    
    Reset L2 cache... 
    DRAM size: 0x08000000 
    Reset L1 cache... 
    
    Backup Boot Version: 1.1.24 
    Verify Checksum... Valid 
    jumping to romStart 
    ............................................. 
    .........................................
    

    To verify the PXM state, login to the console port of the PXM in slot 8.

    Login: 
     card going active.. 
    SM Feature Bit Map is = 0 
    SM Feature Bit Map is = 0

    After the newrev command is issued, the output of the dspcd command on the PXM in slot 8 will show the interim firmware version. The MGX 8850 is now running the interim firmware and health and status as wells as user traffic should be verified.

    jet.1.8.PXM.a > dspcd 
        ModuleSlotNumber:          8 
        FunctionModuleState:       Active 
        FunctionModuleType:        PXM1-OC3 
        FunctionModuleSerialNum:   SCK03160179 
        FunctionModuleHWRev:       A0 
        FunctionModuleFWRev:       1.1.23 
        FunctionModuleResetReason: Upgrade Reset 
        LineModuleType:            PXM-UI 
        LineModuleState:           Present 
        SecondaryLineModuleType:   MMF-4-155 
        SecondaryLineModuleState:  Present 
        mibVersionNumber:          0.0.00 
        configChangeTypeBitMap:    No changes 
        cardIntegratedAlarm:       Clear 
        cardMajorAlarmBitMap:      Line Alarm 
        cardMinorAlarmBitMap:      Line Statistical Alarm 
        BkCardSerialNum:           SBK02420284 
        TrunkBkCardSerialNum:      SAK0320005M 
        FrontCardFabNumber:        800-05086-03 
    

    After the PXM in slot 7 is reset and successfully enters the Hold state, issue the commit 1.1.23 command. The commit 1.1.23 command completes the runtime firmware upgrade on both PXMs and the PXM in slot 7 will now enter the Standby state

    mgx1.1.8.PXM.a > commit 1.1.23 
    this may take a while ... 
    commit command completed OK
    
  11. Verify the intermediate version and the CardState of each MGX 8850 PXM.

    To verify the CardState of the PXMs issue the dspcds command. Note that the PXM which was previously in the Standby state is now Active. Issue the version command to verify the firmware version on each of the PXMs.

    jet.1.8.PXM.a > dspcds 
    
        Slot  CardState    CardType     CardAlarm  Redundancy 
        ----  -----------  --------     ---------  ----------- 
        1.1   Active       FRSM-2E3     Clear 
        1.2   Active       FRSM-2CT3    Clear 
        1.3   Active       FRSM-2E3     Clear 
        1.4   Active       VISM-8T1     Clear 
        1.5   Empty                     Clear 
        1.6   Empty                     Clear 
        1.7   Standby      PXM1-OC3     Clear 
        1.8   Active       PXM1-OC3     Clear 
        1.9   Empty                     Clear 
        1.10  Active       RPM          Clear 
        1.11  Active       VISM-8E1     Clear 
        1.12  Empty                     Clear 
        1.13  Empty                     Clear 
        1.14  Empty                     Clear 
        1.15  Empty                     Clear 
        1.16  Empty                     Clear 
        1.17  Empty                     Clear 
        1.18  Empty                     Clear 
        1.19  Empty                     Clear 
    
    Type <CR> to continue, Q<CR> to stop:
  12. Verify PXM functionality.

    To verify PXM functionality, issue the switchcc command. After the command is executed, the Active PXM will be in slot 7 and the Standby PXM will be in slot 8. Report any alarms incurred during the switchcc command to the Cisco TAC.

  13. Install the target version ComMat.dat file into PXMs.

    The ComMat.dat file contains compatibility matrix data that specifies the firmware version ranges that support graceful upgrades. Different versions of the ComMat.dat file can not be stored on the PXM. Each version of the ComMat.dat file will need to uploaded prior to each installation of runtime firmware. Upload the 1.1.24 ComMat.dat file and then copy to the C:/FW directory of the Active PXM.

    unix-prompt>tftp 192.168.1.65  
    tftp>bin 
    tftp>put ComMat.dat 
    Sent 21 bytes in 0.3 seconds 
    tftp>quit 
    
    jet.1.7.PXM.a > pwd 
    C: 
    
    jet.1.7.PXM.a >mv ComMat.dat C:/FW/ComMat.dat
    

    To upload the ComMat.dat file to the Standby PXM, use the bootChange IP address for the TFTP. The bootChange IP address is functional when the PXM is in the Standby state. Copy the ComMat.dat file to the C:/FW directory of the Standby PXM.

    UNIX-prompt>tftp 192.168.1.30 
    tftp>bin 
    tftp>put ComMat.dat 
    Sent 21 bytes in 0.3 seconds 
    tftp>quit 
    jet.1.8.PXM.s > pwd 
    C: 
    
    jet.1.8.PXM.s > MV ComMat.dat C:/FW/ComMat.dat
    
  14. If network has been stable for 30 minutes after the successful upgrade to the intermediate firmware version, upgrade to the target PXM runtime firmware version using the install, newrev, and commit commands.

    Repeat steps 9 and 10 in Stage 3 in order to upgrade the PXM runtime firmware from 1.1.23 to 1.1.24. Replace occurrences of 1.1.23 with 1.1.24 in each command.

  15. Load target Service Module boot code and firmware versions into the PXM.

    The PXM evaluates all firmware on the MGX 8850 Service Modules. If the PXM detects any incompatibilities between the PXM and Service Module runtime firmware versions an error or Mismatch condition will result.

    If the new firmware version does not require a Service Module boot code upgrade, omit the boot code step. Upload the target firmware and boot code for each Service Module to the shelf. Note that the checksum results are only displayed for firmware uploads.

    Service Module boot code must be loaded per slot. Service Module firmware is copied onto the MGX 8850 PXM hard drive into the /FW directory. If no slot is specified when loading Service Module firmware by using the 0, any Service Module can be inserted into a valid slot and retrieve necessary firmware from the PXM. Loading Service Module firmware without specifying a slot will overwrite the old version of firmware if it exists on the hard drive.

    Boot code and firmware files will be automatically replicated to the Standby PXM a few seconds after they are loaded onto the Active PXM.

    To upload the new Service Module boot code :

    unix-prompt>tftp 192.168.1.23 
    tftp> bin 
    tftp>put frsm_vhs_VHS_BT_1.0.02.fw POPEYE@SM_1_1.BOOT
    Sent 457988 bytes in 14.2 seconds 
    tftp>quit
    

    Syntax of the put command is

    put <backup boot> popeye@SM_1_<slot#>.BOOT

    To upload the new firmware so that it applies to all Service Modules of the same model:

    unix-prompt>tftp 192.168.1.23 
    tftp> bin 
    tftp>put frsm_vhs_10.0.12.fw POPEYE@SM_1_0.FW
    Sent 913360 bytes in 18.3 seconds 
    tftp>quit 
    
    jet.1.7.PXM.a > 
    Program length = 913360 
    Calculated checksum = 0xe2f5ca1b stored checksum = 0xe2f5ca1b 
    Fw checksum passed 
    

    The syntax of the put command to apply firmware to all Service Modules of the same model is:

    put <firmware_filename> POPEYE@SM_1_0.FW

  16. Upgrade Service Module boot code and firmware version.

    Install the uploaded Service Module firmware for each service module. For ungraceful upgrades associated with non-redundant Service Modules, issue the resetcd <card_number> command from the Active PXM. The resetcd <card_number> command forces the Service Module to execute the new boot code and firmware. The resetcd <card_number> command will cause service interruption to the connections for approximately five minutes as there is no redundant Service Module.

    For graceful Service Module upgrades, redundancy must be configured and used. The redundant Service Module firmware upgrade uses the same steps as the redundant PXM firmware upgrade, except the abort command is not supported.

    The MGX 8850 offers 1:1 and 1:N redundancy depending on the Service Module. For this document, 1:1 redundancy is addressed. To configure 1:1 redundancy a secondary Service Module must be available to back-up the primary Service Module. The primary and secondary Service Modules must be the same model, type, and use the same Line Module or back card. To activate 1:1 redundancy between the Service Modules in 2 slots, issue the addred command from the Active PXM. The redundant slots do not need to be contiguous, but a disperse configuration makes cable management and troubleshooting difficult. To identify redundancy on an MGX 8850, issue the dspred command from the Active PXM. Once a Service Module is configured as secondary in a 1:1 redundant scenario, the state changes from Active to Standby. The state change indicates that many commands will not work when issued directly on a Service Module in the Standby state. Commands that do not work on a Service Module in the Standby state include install, newrev, and commit.

    mgx1.1.8.PXM.a > dspred 
      Primary  Primary  Primary Secondary Secondary  Secondary  Red.  Red.Slot 
      SlotNum   Type    State   SlotNum   Type       State      Type  Cover 
      ------- -------   ------- --------- ---------  ---------  ----  -------- 
         1    FRSM-2E3  Active     3      FRSM-2E3   Standby    1:1    0 
    
    

    Issue the install bt sm <slot_number> <boot_code_version> to execute the target version of boot code.

    Issue the following commands to execute the target version of Service Module firmware:

    Command Syntax Function
    install install smslot_number> <firmware_version> to new Loads the new firmware version.
    newrev newrev smslot_number> <firmware_version> Executes the new firmware version. Results in a switchcc from the Primary Service Module to the Secondary Service Module.
    commit commit smslot_number> <firmware_version> Completes the upgrade to the new firmware version. Graceful downgrade to original firmware version is lost.
    abort The abort command is not supported for graceful Service Module firmware upgrades.

    jet.1.7.PXM.a > install sm 1 10.0.12 
    Do you want to proceed (Yes/No)? yes 
    
    jet.1.7.PXM.a > newrev sm 1 10.0.12 
    Do you want to proceed (Yes/No)? yes 
    
    jet.1.7.PXM.a > dspcds 
    
     Slot  CardState    CardType     CardAlarm  Redundancy 
        ----  -----------  --------     ---------  ----------- 
        1.1   Boot         FRSM-2E3     Clear     Covered by slot 3  
        1.2   Active       FRSM-2CT3    Clear 
        1.3   Active       FRSM-2E3     Clear     Covering slot 1  
        1.4   Active       VISM-8T1     Clear 
        1.5   Active       VISM-8T1     Clear 
        1.6   Empty                     Clear 
        1.7   Active       PXM1-OC3     Clear 
        1.8   Standby      PXM1-OC3     Clear 
        1.9   Empty                     Clear 
        1.10  Active       RPM          Clear 
        1.11  Active       VISM-8E1     Clear 
        1.12  Empty                     Clear 
        1.13  Empty                     Clear 
        1.14  Empty                     Clear 
        1.15  Empty                     Clear 
        1.16  Empty                     Clear 
        1.17  Empty                     Clear 
        1.18  Empty                     Clear 
        1.19  Empty                     Clear 
    
    Type <CR> to continue, Q<CR> to stop:
    
    jet.1.7.PXM.a > commit sm 1 10.0.12 
    Do you want to proceed (Yes/No)? yes 
  17. Let network settle and run customer specific validation tests.

    After 10 minutes, login to the target node and verify health using the following commands:

    • dsplog

    • dsperr -en

    • dsptotals

    This period provides an ideal time to run tests to check that the new firmware is functioning correctly.

    Interrogate all external management systems that are used to manage any routers that are connected to the MGX 8850 network. This interrogation is done to ensure that all devices are reachable.

    If possible, end users should be contacted and asked to check that all network connections are in proper working order.

    Note: In the unlikely event that a decision is taken to revert back to the previous firmware revision, Cisco TAC should be contacted prior to switching to the old revision. Important information as to why the new firmware is not functioning correctly will be lost after switching back to the old revision.

  18. Network health check.

    See Appendix A

  19. Save MGX 8850 PXM and Service Module (SM) configuration.

    See step 2 of Stage 3.

  20. Provisioning freeze ends.

Appendix A - Network Health Check

Follow these steps to check the network health:

  1. Audit the parameters within the following commands.

    Settings should be consistent across all nodes of the same type within the network. Document differences and any variations from the default values.

    dsptotals 
          dsplog 
          dspalms 
          dspshelfalm
    
  2. Audit network for recent errors (active and standby controller cards), card errors, load model inconsistencies and alarms.

    Use the following commands to accomplish these tasks:

    dsperr -en 
         dsplog s 
         dsplog 
         printlog 
         dspcderrs or the dspcderrs <slot #> 
         dspalms 
    
    
  3. Investigate the following:

    • Recent firmware errors: Any nodes that continually log errors or have logged recent errors should be reported to Cisco TAC.

    • Card errors: Cards that are logging failures or have a history of hardware errors should be investigated by Cisco TAC.

    • Any trunks that are logging errors: Should be fixed for the duration of the upgrade.

    • All alarms should be accounted for. The real purpose of this check is to make sure that there are no alarms that will require special intervention before the upgrade.

  4. Ensure that any necessary corrections are made before the start of the upgrade.

Related Information

Updated: Apr 17, 2009
Document ID: 6934