Actualización de software en funcionamiento (ISSU) en switches Catalyst serie 3850 y 9000

Introducción

El objetivo de este documento es ayudar a comprender los pasos que implica la realización de ISSU en los switches Catalyst serie 9000 y 3850.

¿Qué es ISSU?

In-Service Software Upgrade (ISSU) es un proceso que actualiza una imagen a otra imagen en un dispositivo mientras la red continúa reenviando paquetes. ISSU ayuda a los administradores de red a evitar interrupciones en la red al realizar una actualización de software. Las imágenes se actualizan en modo de instalación, en el que cada paquete se actualiza individualmente.

ISSU es compatible con todas las series Catalyst 3850 y Catalyst 9000 compatibles con Stackwise-Virtual y también con los chasis independientes Catalyst 9400/9600 con supervisores duales.

StackWise Virtual (SVL) consta de dos switches que se conectan para formar un switch virtual. SVL admite actualizaciones de software en servicio.

ISSU admite actualizaciones, reversiones y reversiones.

Plataformas admitidas y matriz de soporte de versiones

Antes de continuar con ISSU, verifique si la plataforma admite realmente ISSU. Verifique también si ISSU es compatible entre el código actual y el código de destino. Los detalles del modo en las plataformas soportadas y la matriz de compatibilidad de ISSU se pueden encontrar aquí - Soporte de ISSU entre Versiones

Requisitos previos antes de ISSU

Nota: El ejemplo de este documento se basa en los switches Cisco Catalyst 9500 configurados como Stackwise-Virtual. Los pasos son aplicables a los chasis independientes Cat9400 / Cat9600 con motores supervisores duales y también a los dispositivos Catalyst 3850 / Catalyst 9000 configurados como Stackwise-Virtual.

1. Comprobar la versión de código actual

C9500#show version | in IOS XE 
Cisco IOS XE Software, Version 16.09.02

2. Verifique el modo de arranque

ISSU se soporta solamente si ambos switches en Stackwise Virtual se inician en el modo INSTALL.

C9500#show ver | in INSTALL
*    1 50    C9500-40X          16.9.2            CAT9K_IOSXE           INSTALL
     2 50    C9500-40X          16.9.2            CAT9K_IOSXE           INSTALL

On Catalyst 9400, the above output is not available. Check if the switch booted from packages.conf file 
C9400#show version | in System image
System image file is "flash:packages.conf"
 

Si el chasis se inicia en un modo BUNDLE, ISSU no se soporta. Verá errores como los siguientes si intenta ejecutar ISSU cuando los switches se ejecutan en el modo BUNDLE.

*Nov 13 14:55:57.338: %INSTALL-5-INSTALL_START_INFO: Chassis 1 R1/0: install_engine: Started install one-shot ISSU flash:cat9k_iosxe.16.09.02.SPA.bininstall_add_activate_commit: Adding ISSU
ERROR: install_add_activate_commit: One-Shot ISSU operation is not supported in bundle boot mode
FAILED: install_add_activate_commit  exit(1) Tue Nov 13 14:56:03 UTC 2018

3. Compruebe si hay suficiente memoria disponible en la memoria flash

C9500#dir flash: | in free
11353194496 bytes total (8565174272 bytes free)

C9500#dir stby-flash: | in free
11353980928 bytes total (8566865920 bytes free)

Asegúrese de tener al menos 1 GB de espacio en la memoria flash para expandir una nueva imagen. Limpie los archivos de instalación antiguos en caso de espacio insuficiente mediante el comando "install remove idle".

4. Compruebe si los switches están en modo SSO

C9500#show redundancy 
Redundant System Information :
------------------------------
       Available system uptime = 4 minutes
Switchovers system experienced = 0
              Standby failures = 0
        Last switchover reason = none

                 Hardware Mode = Duplex
    Configured Redundancy Mode = sso
     Operating Redundancy Mode = sso
              Maintenance Mode = Disabled
                Communications = Up

Current Processor Information :
-------------------------------
               Active Location = slot 1
        Current Software state = ACTIVE
       Uptime in current state = 30 minutes
                 Image Version = Cisco IOS Software [Fuji], Catalyst L3 Switch Software (CAT9K_IOSXE), Version 16.9.2, RELEASE SOFTWARE (fc4)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2018 by Cisco Systems, Inc.
Compiled Mon 05-Nov-18 19:32 by mcpre
                          BOOT = flash:packages.conf;
                   CONFIG_FILE = 
        Configuration register = 0x102

Peer Processor Information :
----------------------------
              Standby Location = slot 2
        Current Software state = STANDBY HOT 
       Uptime in current state = 26 minutes
                 Image Version = Cisco IOS Software [Fuji], Catalyst L3 Switch Software (CAT9K_IOSXE), Version 16.9.2, RELEASE SOFTWARE (fc4)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2018 by Cisco Systems, Inc.
Compiled Mon 05-Nov-18 19:32 by mcpre
                          BOOT = flash:packages.conf;
                   CONFIG_FILE = 
        Configuration register = 0x102
        

5. Compruebe si Auto-Boot está activado

C9500#show boot system 
---------------------------
Switch 1
---------------------------
Current Boot Variables:
BOOT variable = flash:packages.conf;

Boot Variables on next reload:
BOOT variable = flash:packages.conf;
Manual Boot = no <<<<< Manual Boot should be set to "no"   
Enable Break = no
Boot Mode = DEVICE
iPXE Timeout = 0
---------------------------
Switch 2
---------------------------
Current Boot Variables:
BOOT variable = flash:packages.conf;

Boot Variables on next reload:
BOOT variable = flash:packages.conf;
Manual Boot = no
Enable Break = no
Boot Mode = DEVICE
iPXE Timeout = 0 

Si Auto-Boot no está habilitado, esto puede cambiarse como se muestra a continuación:

C9500(config)#no boot manual

6. Verifique los estados ISSU e Install actuales.

     NOTE: ESTE PASO ES MUY IMPORTANTE

C9500#show issu state detail
--- Starting local lock acquisition on switch 1 ---
Finished local lock acquisition on switch 1

No ISSU operation is in progress <<<<<<<<< If see anything else, abort ISSU before proceeding. Check below on how to manually abort ISSU.



 C9500#show install summary
[ Switch 1 2 ] Installed Package(s) Information:
State (St): I - Inactive, U - Activated & Uncommitted,
            C - Activated & Committed, D - Deactivated & Uncommitted
--------------------------------------------------------------------------------
Type  St   Filename/Version    
--------------------------------------------------------------------------------
IMG   C    16.9.2.0.2433  <<<<<<<< State should be Activated & Committed for current version alone. If not clear install state before proceeding. Check below on how to clear install state. 
--------------------------------------------------------------------------------
Auto abort timer: inactive
--------------------------------------------------------------------------------

7. Copie la nueva imagen para actualizar o degradar.

Es suficiente copiar la imagen sólo en el chasis Activo (en caso de Stackwise) o en el supervisor Activo (en caso de que el Cat9400 doble SUP).

C9500#copy tftp: bootflash:
Address or name of remote host []? X.X.X.X
Source filename []? cat9k_iosxe.16.09.02.SPA.bin
Destination filename [cat9k_iosxe.16.09.02.SPA.bin]? 
 

Flujos de trabajo de ISSU - Actualización real

Una vez verificados todos los requisitos previos, puede continuar con la actualización real usando uno de los métodos siguientes:

•            Flujo de trabajo de 1 paso (solo un paso y no admite la reversión)
•            Flujo de trabajo de 3 pasos (implica 3 pasos y admite la retroceso en caso de fallo)

Flujo de trabajo de 1 paso

•    Este flujo de trabajo implica sólo un paso y ayuda en la optimización.

NOTE: No se puede deshacer porque la actualización se ha confirmado automáticamente. Vaya a Flujo de trabajo de 3 pasos si desea retroceder.

// Below example has SW-2 as Active and Sw-1 as Standby before starting ISSU


C9500#install add file flash:cat9k_iosxe.16.09.02.SPA.bin activate issu commit
install_add_activate_commit: START Fri Feb  8 10:07:51 jst 2019

*Feb  8 10:07:52.456 jst: %INSTALL-5-INSTALL_START_INFO: Switch 2 R0/0: install_engine: Started install one-shot ISSU flash:cat9k_iosxe.16.09.02.SPA.bininstall_add_activate_commit: Adding ISSU

--- Starting initial file syncing ---
[2]: Copying flash:cat9k_iosxe.16.09.02.SPA.bin from switch 2 to switch 1
[1]: Finished copying to switch 1  
Info: Finished copying flash:cat9k_iosxe.16.09.02.SPA.bin to the selected switch(es)
Finished initial file syncing

--- Starting Add ---
Performing Add on all members
  [1] Add package(s) on switch 1
  [1] Finished Add on switch 1
  [2] Add package(s) on switch 2
  [2] Finished Add on switch 2
Checking status of Add on [1 2]
Add: Passed on [1 2]
Finished Add

install_add_activate_commit: Activating ISSU

NOTE: Going to start Oneshot ISSU install process
 STAGE 0: Initial System Level Sanity Check before starting ISSU
===================================================
--- Verifying install_issu supported ---
--- Verifying standby is in Standby Hot state ---
--- Verifying booted from the valid media ---
--- Verifying AutoBoot mode is enabled ---
Finished Initial System Level Sanity Check


STAGE 1: Installing software on Standby
===================================================
--- Starting install_remote ---
Performing install_remote on Chassis remote
[1] install_remote package(s) on switch 1
[1] Finished install_remote on switch 1
install_remote: Passed on [1]
Finished install_remote

 STAGE 2: Restarting Standby
===================================================
--- Starting standby reload ---
Finished standby reload

--- Starting wait for Standby to reach terminal redundancy state ---     <<<<< Standby (Sw-1) reloads here!!!


<<<<<<<< After Standby (Sw-1) comes up >>>>>>>>>>>

*Feb  8 10:19:10.223 jst: %REDUNDANCY-3-IPC: IOS versions do not match.

*Feb  8 10:19:48.421 jst: %HA_CONFIG_SYNC-6-BULK_CFGSYNC_SUCCEED: Bulk Sync succeeded

*Feb  8 10:19:49.422 jst: %RF-5-RF_TERMINAL_STATE: Terminal state reached for (SSO) <<<<< Standby (Sw-1) comes up on new code and joins as Hot Standby..


*Feb  8 10:21:02.975 jst: %PLATFORM-6-HASTATUS_DETAIL: RP switchover, received chassis event became active. Switch to primary (count 1)  <<<<<< Switchover happens after this log and the previous active (Sw-2) reloads.

<<<<<<<< After new Standby (Sw-2) comes up >>>>>>>>

*Feb  8 10:27:09.715 jst: %HA_CONFIG_SYNC-6-BULK_CFGSYNC_SUCCEED: Bulk Sync succeeded

*Feb  8 10:27:10.717 jst: %RF-5-RF_TERMINAL_STATE: Terminal state reached for (SSO). <<<< ISSU commit starts after this automatically..


*Feb  8 10:28:27.302 jst: %INSTALL-5-INSTALL_START_INFO: Switch 2 R0/0: install_engine: Started install commit
%IOSXEBOOT-4-ISSU_ONE_SHOT: (rp/0): ISSU finished successfully

*Feb  8 10:29:32.127 jst: %INSTALL-5-INSTALL_COMPLETED_INFO: Switch 2 R0/0: install_engine: Completed install commit ISSU   

 Continúe con después de que ISSU se complete.

Flujo de trabajo de 3 pasos

• Este flujo de trabajo implica tres pasos: agregar, activar y confirmar. Después de la activación, todos los switches se actualizan a la nueva versión de software excepto que el software no se compromete automáticamente, sino que se debe realizar manualmente a través del comando install commit.
• La ventaja de este enfoque es que el sistema se puede deshacer a una versión de software anterior.
• El sistema se reinicia automáticamente si el temporizador de reversión no se detiene usando el comando install auto-abort-timer stop o el comando install commit. Si se detiene el temporizador de reversión, la nueva versión de software se podría ejecutar en el dispositivo durante cualquier tiempo y, a continuación, volver a la versión anterior.

PASO 1: Install_add

 Este comando descarga la imagen en la memoria flash de inicialización y la expande en ambos switches.

// Below example has SW-1 as Active and Sw-2 as Standby before starting ISSU

C9500#install add file flash:cat9k-universalk9.SPA.16.09.03.BETA.E1.SSA.bin.bin    
install_add: START Fri Feb  8 09:22:00 jst 2019

*Feb  8 09:22:02.055 jst: %INSTALL-5-INSTALL_START_INFO: Switch 1 R0/0: install_engine: Started install add flash:cat9k-universalk9.SPA.16.09.03.BETA.E1.SSA.bin.bininstall_add: Adding PACKAGE

--- Starting initial file syncing ---
[1]: Copying flash:cat9k-universalk9.SPA.16.09.03.BETA.E1.SSA.bin.bin from switch 1 to switch 2
[2]: Finished copying to switch 2  
Info: Finished copying flash:cat9k-universalk9.SPA.16.09.03.BETA.E1.SSA.bin.bin to the selected switch(es)
Finished initial file syncing

--- Starting Add ---
Performing Add on all members
  [1] Add package(s) on switch 1
  [1] Finished Add on switch 1
  [2] Add package(s) on switch 2
  [2] Finished Add on switch 2
Checking status of Add on [1 2]
Add: Passed on [1 2]
Finished Add

SUCCESS: install_add  Fri Feb  8 09:26:26 jst 2019 <<<< Wait until install_add says SUCCESS. If fails, manually abort ISSU.

PASO 2: Install_activar

Al ejecutar este comando, se produce la siguiente secuencia de eventos:

    i) Se inicia un temporizador de reversión. Si caduca el temporizador de reversión, el sistema vuelve al mismo estado antes del inicio del ISSU. El temporizador de reversión se puede detener usando el comando install auto-abort-timer stop. ISSU se puede deshacer usando el comando install abort issu.

    ii) El switch en espera se aprovisiona con el nuevo software y se recarga con la nueva versión de software. A continuación, el switch activo se aprovisiona con el nuevo software y se recarga. El switch en espera con la nueva imagen ahora se convierte en el switch activo y el switch activo antiguo se convierte en el standby.

    Al final de este procedimiento, ambos switches se ejecutan con la nueva imagen de software.

C9500#install activate issu
install_activate: START Fri Feb  8 09:28:27 jst 2019
install_activate: Activating ISSU

*Feb  8 09:28:28.905 jst: %INSTALL-5-INSTALL_START_INFO: Switch 1 R0/0: install_engine: Started install activate ISSU
NOTE: Going to start Activate ISSU install process

STAGE 0: Initial System Level Sanity Check before starting ISSU
===================================================
--- Verifying install_issu supported ---
--- Verifying standby is in Standby Hot state ---
--- Verifying booted from the valid media ---
--- Verifying AutoBoot mode is enabled ---
Finished Initial System Level Sanity Check

 STAGE 1: Installing software on Standby
===================================================
--- Starting install_remote ---
Performing install_remote on Chassis remote

*Feb  8 09:28:31.880 jst: %INSTALL-5-INSTALL_AUTO_ABORT_TIMER_PROGRESS: Switch 1 R0/0: rollback_timer: Install auto abort timer will expire in 7200 seconds


[2] install_remote package(s) on switch 2
[2] Finished install_remote on switch 2
install_remote: Passed on [2]
Finished install_remote


STAGE 2: Restarting Standby
===================================================
--- Starting standby reload ---
Finished standby reload

--- Starting wait for Standby to reach terminal redundancy state ---      <<<<<<<< Standby (Sw-2) reloads here!!!


*Feb 8 09:35:16.489 jst: %REDUNDANCY-3-IPC: IOS versions do not match.
 
*Feb  8 09:36:00.238 jst: %HA_CONFIG_SYNC-6-BULK_CFGSYNC_SUCCEED: Bulk Sync succeeded
*Feb  8 09:36:01.240 jst: %RF-5-RF_TERMINAL_STATE: Terminal state reached for (SSO)     <<<< At this point, Standby (Sw-2) comes up with new code and joins as Hot Standby  Finished wait for Standby to reach terminal redundancy state 
 STAGE 3: Installing software on Active
===================================================
--- Starting install_active ---
Performing install_active on Chassis 1


1] install_active package(s) on switch 1
[1] Finished install_active on switch 1
install_active: Passed on [1]
Finished install_active



Chassis 1 reloading, reason - Non participant detected
STAGE 4: Restarting Active (switchover to standby)         <<<<<<<< At this point, there is a switchover and then the previous Active (Sw-1) reloads. ===================================================
--- Starting active reload ---
 New software will load after reboot process is completed
SUCCESS: install_activate  Fri Feb  8 09:37:14 jst 2019

Al final del estado Activate, verifique los estados ISSU.

C9500#show issu state detail
--- Starting local lock acquisition on switch 2 ---
Finished local lock acquisition on switch 2

Operation type: Step-by-step ISSU
Install type  : Image installation using ISSU
Current state : Activated state
Last operation: Switchover
 Completed operations: Operation Start time ------------------------------------------------------- Activate location standby Chassis 2 2019-02-08:09:28:32 Activate location active Chassis 1 2019-02-08:09:36:03 Switchover 2019-02-08:09:37:16

State transition: Added -> Standby activated -> Active switched-over 

Auto abort timer: automatic, remaining time before rollback: 01:43:55
Running image: flash:packages.conf
Operating mode: sso, terminal state reached                 <<<<< Wait until SSO terminal state before proceeding to commit.

PASO 3: Install_commit

El comando commit realiza la limpieza necesaria, habilita el nuevo software como permanente (eliminando la versión anterior del software) y detiene el temporizador de reversión. Cualquier reinicio después de la confirmación se iniciará con el nuevo software.

C9500#install commit 
install_commit: START Fri Feb  8 09:45:22 jst 2019
install_commit: Committing ISSU

*Feb  8 09:45:23.533 jst: %INSTALL-5-INSTALL_START_INFO: Switch 2 R0/0: install_engine: Started install commit

NOTE: Going to start Commit ISSU install process

STAGE 0: Initial System Level Sanity Check before starting ISSU
===================================================
--- Verifying install_issu supported ---
--- Verifying standby is in Standby Hot state ---
--- Verifying booted from the valid media ---
--- Verifying AutoBoot mode is enabled ---
Finished Initial System Level Sanity Check

--- Starting install_commit_2 ---
Performing install_commit_2 on Chassis 2
[2] install_commit_2 package(s) on switch 2
[2] Finished install_commit_2 on switch 2
install_commit_2: Passed on [2]
Finished install_commit_2


STAGE 1: Dispatching the commit command to remote
===================================================
--- Starting install_commit_remote ---
Performing install_commit_remote on Chassis 1
Feb  8 09:48:33.364: %INSTALL-5-INSTALL_START_INFO: R0/0: install_engine: Started install commit

*Feb  8 09:48:33.352 jst: %INSTALL-5-INSTALL_START_INFO: Switch 1 R0/0: install_engine: Started install commit


Feb  8 09:51:27.505: %INSTALL-5-INSTALL_COMPLETED_INFO: R0/0: install_engine: Completed install commit ISSU
[1] install_commit_remote package(s) on switch 1
[1] Finished install_commit_remote on switch 1
install_commit_remote: Passed on [1]
Finished install_commit_remote

SUCCESS: install_commit  Fri Feb  8 09:51:27 jst 2019     <<<<< ISSU is completed here!!!!! 

Publicar lista de comprobación ISSU

Una vez que ISSU se haya completado con éxito,

• Verifique si ambos switches se están ejecutando en el nuevo software.
• Verifique que la salida "show issu state detail" esté limpia y no muestre ningún ISSU en curso.
• Verifique la salida "show install issu history" para asegurar el funcionamiento correcto de ISSU (NOTA: Comando disponible sólo a partir de la versión 16.10.1)
• Se recomienda dar suficiente tiempo de inactividad en el nuevo software antes de activar cualquier nueva función.

Acción sobre la Falla de ISSU

• Si falla ISSU, se espera que auto-abort recupere el sistema de nuevo al estado inicial (imagen anterior). Sin embargo, si esto también falla, se espera recuperación manual del chasis.
• Durante la recuperación manual, verifique si tanto activo como en espera están ejecutando la imagen anterior (si no es así, recupere el chasis individual)
• Después de asegurarse de que ambos chasis estén ejecutando la imagen antigua, ejecute "install remove idle" para eliminar cualquier paquete de imagen sin utilizar.
• Una vez que ambos chasis ejecutan el software antiguo, limpie manualmente todos los estados internos del funcionamiento de ISSU. (Consulte a continuación sobre cómo limpiar los estados ISSU internos)

1. Abortar ISSU

• En el flujo de trabajo de 3 pasos, durante el proceso de activación de ISSU, si caduca el temporizador de anulación, el sistema anulará automáticamente la imagen anterior.

Nota: Si el modo en espera no llega al SSO durante el aborto, se requiere el aborto manual. Además, si por alguna razón desea anular el ISSU en el intervalo, se requiere aborto manual.

EXAMPLE : During install add, we notice the below errors:

C9400#install add file flash:cat9k_iosxe.16.09.02.SPA.bin
install_add: START Tue Nov 13 20:47:53 UTC 2018

*Nov 13 20:47:54.787: %INSTALL-5-INSTALL_START_INFO: Chassis 1 R1/0: install_engine: Started install add flash:cat9k_iosxe.16.09.02.SPA.bininstall_add: Adding PACKAGE

--- Starting initial file syncing ---
[1]: Copying flash:cat9k_iosxe.16.09.02.SPA.bin from chassis 1 to chassis 2
[2]: Finished copying to chassis 2  
Info: Finished copying flash:cat9k_iosxe.16.09.02.SPA.bin to the selected chassis
Finished initial file syncing

--- Starting Add ---
Performing Add on all members
  [1] Add package(s) on chassis 1
  [1] Finished Add on chassis 1
  [2] Add package(s) on chassis 2
    cp: cannot stat '/tmp/packages.conf': No such file or directory
  [2] Finished Add on chassis 2
Checking status of Add on [1 2]
Add: Passed on [1]. Failed on [2]
Finished Add

FAILED: install_add exit(1) Tue Nov 13 20:51:58 UTC 2018 <<<<<< install_add failed. If see any such error, first do a manual abort before proceeding further. 

C9400#install abort issu 
install_abort: START Tue Nov 13 20:57:40 UTC 2018
install_abort: Abort type ISSU subtype NONE smutype NONE

*Nov 13 20:57:41.759: %INSTALL-5-INSTALL_START_INFO: Chassis 1 R1/0: install_engine: Started install abort ISSU
NOTE: Going to start Abort ISSU install process

STAGE 0: Initial System Level Sanity Check before starting ISSU
===================================================
--- Verifying install_issu supported ---
--- Verifying booted from the valid media ---
--- Verifying AutoBoot mode is enabled ---
Finished Initial System Level Sanity Check

FAILED: ABORT operation is not allowed in ADDED state
ERROR: install_abort  exit(2 ) Tue Nov 13 20:57:49 UTC 2018

*Nov 13 20:57:49.756: %INSTALL-5-INSTALL_COMPLETED_INFO: Chassis 1 R1/0: install_engine: Completed install abort ISSU 

2.Limpieza del estado ISSU

Si el upgrade/downgrade/abort/auto-abort de ISSU no se realiza correctamente, se requiere la limpieza manual de los estados internos de ISSU.

C9400#sh issu state detail
--- Starting local lock acquisition on chassis 1 ---
Finished local lock acquisition on chassis 1

Operation type: One-shot ISSU
Install type  : Image installation using ISSU
Current state : Added state Last operation: Activate location standby Chassis 2   <<<< Previous Add is still pending. This needs to be cleared before starting ISSU 
Completed operations:

Operation                           Start time
-------------------------------------------------------
Activate location standby Chassis 2 2018-11-13:16:26:34   

State transition: Added 

Auto abort timer: inactive
Running image: flash:packages.conf
Operating mode: sso, terminal state not reached

NOTE: Enable Service Internal before run the below command

C9400#clear install state 
clear_install_state: START Tue Nov 13 17:05:47 UTC 2018
--- Starting clear_install_state ---
Performing clear_install_state on all members
  [1] clear_install_state package(s) on chassis 1
  [1] Finished clear_install_state on chassis 1
Checking status of clear_install_state on [1]
clear_install_state: Passed on [1]
Finished clear_install_state 
 C9400#sh issu state detail 
--- Starting local lock acquisition on chassis 1 ---
Finished local lock acquisition on chassis 1

No ISSU operation is in progress