Online Insertion and Removal (OIR) Support in Routers

Introduction

This document provides some generic information about Online Insertion and Removal (OIR), and provides a list of platforms and modules supporting this operation.

Prerequisites

Requirements

Readers of this document should have a basic knowledge of router modules hardware installation.

Components Used

The information in this document is based on these software and hardware versions:

• All Cisco IOS® software versions

• Cisco 3600 Series Routers

• Cisco AS5800 Universal Access Server

• Cisco 7200 Series Routers

• Cisco 7500 Series Routers

• Cisco 12000 Series Internet Routers

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.

Conventions

For more information on document conventions, refer to the Cisco Technical Tips Conventions.

About OIR

OIR was developed to enable you to replace faulty parts without affecting system operation. When a card is inserted, power is available on the card, and it initializes itself to start working.

Hot swap functionality allows the system to determine when a change occurs in the unit's physical configuration, and reallocate the unit's resources to allow all interfaces to function adequately. This feature allows interfaces on the card to be reconfigured while other interfaces on the router remain unchanged. The interrupt routine must ensure that the interrupt line has reached a stable state.

The software performs the necessary tasks involved in handling the removal and insertion of the card. A hardware interrupt is sent to the software subsystem when a hardware change is detected, and the software reconfigures the system as such:

• When a card is inserted, it is analyzed and initialized in such a way that the end user can configure it properly. The initialization routines used during OIR are the same as those called when the router is powered on. System resources, also handled by software, are allocated to the new interface.

• When a card is removed, the resources associated with the empty slot must either be freed or altered to indicate the change in its status.

What Does an OIR Do?

When an OIR is performed, the router:

1. Rapidly scans the backplane for configuration changes.

2. Initializes all newly inserted interfaces and places them in the administratively shut down state.

3. Brings all previously configured interfaces on the card back to the state they were in when they were removed. Any newly inserted interfaces are put in the administratively shut down state.

The only effect on the routing tables is that routes through a removed interface are deleted, as are routes learned through that interface. The Address Resolution Protocol (ARP) cache is selectively flushed, and routing caches are completely flushed (this may also occur during normal operation of the router, without OIR, and this is nothing to worry about).

If a card is reinserted into the same slot from which it was removed, or if an identical card is inserted in its place, many of the control blocks from the previous installation are reused. This is necessary due to the implementation by Cisco IOS software of certain control blocks, and has the benefit of saving the configuration from the previously installed card.

How Do I Perform an OIR?

It is always safer to power down the router when you perform any hardware changes, but here are some recommendations if you need to perform an OIR. The system may indicate a hardware failure if you do not follow proper procedures.

• Insert only one card at a time; you must allow the system time to complete the preceding tasks before you remove or insert another interface processor. If you disrupt the sequence before the system completes its verification, it can cause the system to detect spurious hardware failures.

• Insert the cards swiftly and firmly, but do not shove them in.

• If present, be sure to use the little plastic levers on the side of the card to lock the card in.

• If the OIR is successful, there is absolutely no need to schedule a reload of the router.

If you get a LONGSTALL message after an OIR, or a CPUHOG during the OIR process, but encounter no other problems, you may safely ignore those messages. Detailed information about CPUHOG messages and their implication in OIR events can be found in the What Causes %SYS-3-CPUHOG Messages? document.

Cisco Routers Which Support OIR

Cisco 3600 Series Routers

Platform	OIR Supported?
3620	No
3640	No
3660	OIR supported for network modules (NMs) OIR not supported for WAN interface cards (WICs) OIR not supported for power supplies

Restrictions

• A network module can only be replaced by a similar one (if OIR is used). For example, an NM-12DM can be replaced only by another NM-12DM, and not by an NM-6DM.

• If a module has a T1/E1 interface, the T1/E1 controller should first be disabled before the network module is swapped.

• The new network module is functional only if the router has adequate input and output (I/O) memory to support its operation (use the 2600/3600/3700 Memory Calculator (registered customers only) to find out the memory requirements for your configuration).

You may experience an OIR-related error message on your Cisco 3600 Series Router, similar to this one:

%OIRINT: OIR Event has occurred oir_ctrl 5000 oir_stat F02

For help troubleshooting any OIR-related error messages, refer to Troubleshooting OIR Events on 3600 Series Routers.

Cisco AS5800 Universal Access Server

Follow this procedure to perform an OIR on the AS5800 platform:

1. Cleanly bring down the card prior to removing it from the chassis. For information on the various terms associated with the AS5800, refer to Hardware Troubleshooting for the AS5800.

2. Issue the show dial-shelf slot slot_number command from the enable prompt on the router shelf and note the numeric value associated with the type of feature board (in slot y) you want to reset. In this sample output, the board type for slot 3 is 259:

   as5800RS-VXR#show dial-shelf slot 3 
   Slot: 3, Type: Channelised T3 (259)
   

3. Open a virtual connection to the Dial Shelf Controller (DSC) with the dsip console slave dsc_slot_number command. The value for dsc_slot_number should be either 12 or 13. The value depends on the slot number of the DSC that the affected feature board belongs to. For example:

   routershelf#dsip console slave 12
   Trying Dial shelf slot 12 ...
   Entering CONSOLE for slot 12

4. Issue the oir testport enable command to enable OIR simulation.

5. Issue the oir slot slot_number remove command to remove the card using the software. The value for slot_number should be the slot number of the feature board on which you want to enable OIR (slot y in the error message above).

6. Remove the card and insert the replacement card in any slot.

7. To re-insert the card, issue the oir slot slot_number insert command (use the board type from step 1).

8. Issue the oir log command to check the OIR events.

   For more information, refer to AS5800: Replacing or Installing Dial Shelf Cards.

Cisco 7200 Series Routers

Cisco 7200 Series Router	OIR Supported?
NPE/NSE	No
I/O Controller	No
Port Adapters (PAs)	Yes
Power Supply	Yes

More details about the different port adapters can be found on the Port Adapters Documentation pages.

Cisco 7500 Series Routers

Cisco 7500 Series Routers	OIR Supported?
Master Route-Switch Processor (RSP)	No (unless Standby present)
Standby RSP	Yes
Channel Interface Processor (CIP)	OIR supported from 11.1(5)
Interface Processors (xIP)	OIR supported from 11.1(6)
Port Adapters (PAs)	No
Power Supply	Yes

Restrictions

• The master RSP should normally not be removed while the system is operating. However, if a Standby RSP is present, it takes over as per the configured HA redundancy mode (for more information on this, refer to Route Processor Redundancy and Fast Software Upgrade on Cisco 7500 Series Routers), but the traffic is interrupted. For Cisco 7500 Series routers, the redundancy mode configured has no influence in case of an Online Removal of the active RSP. Online removal of the active RSP causes all line cards to reset and reload, which is equivalent to an RPR switchover, and results in a longer switchover time. When it is necessary to remove the active RSP from the system, first issue the switchover command to switch from the active RSP to the standby RSP. When a switchover is forced to the standby RSP before the previously active RSP is removed, the network operation benefits from the continuous forwarding capability of Stateful Switchover (SSO), which is supported in Cisco IOS software release 12.0(22)S and above.

  For more information, refer to Stateful Switchover.

• Never insert a Versatile Interface Processor (VIP) without port adapters; this configuration is not supported. Each unused interface processor slot must contain an interface processor filler (which is an interface processor carrier without a printed circuit board), to keep dust out of the router and to maintain proper airflow through the interface processor compartment.

• OIR of port adapters is not supported, but you can remove the VIP card completely, then add, remove, or replace the port adapter(s), and plug the VIP card back in.

Note: In Cisco 7507/7507-MX or Cisco 7513/7513-MX routers with the high system availability (HSA) feature active, online insertion and removal of any interface processor in either CyBus might cause the slave RSP2 to reboot with a bus error or a processor memory parity error. The master RSP recovers from this event and issues a "cBus Complex Restart" message. Systems that are configured with an RSP4 or an RSP8 as the system slave are not affected and do not experience this problem. This issue is described in detail at Field Notice: Cisco 7507 and Cisco 7513: RSP2 HSA OIR.

Cisco 7600 Series Routers

Cisco 7600 Series Router	OIR Supported?
FlexWAN and Enhanced FlexWAN Module	Yes
Supervisor Engine 720-3BXL	Yes
2-Port ATM Optical Services Module	Yes
2-Port OC-48c OC-48c/1-port OC-48c DPT OSM	Yes
4- and 8-Port OC-3c/STM-1 POS Enhanced OSM	Yes
Enhanced Gigabit Ethernet Optical Services Module	Yes
2- and 4-Port OC-12c/STM-4 POS Enhanced OSM	Yes
1-Port OC-48c/STM-16 POS Enhanced Optical Services Module	Yes

Although the FlexWAN and Enhanced FlexWAN modules support hot swapping, individual port adapters do not. To replace port adapters, you must first remove the FlexWAN module from the chassis and then replace port adapters as required.

Cisco ESR 10000 Series Routers

Cisco 10000 Series Router	OIR Supported?
Performance Routing Engine (PRE-1)	Yes
Performance Routing Engine (PRE-2)	Yes
Power Supply	Yes
Line Card	Yes

Cisco 12000 Series Internet Routers

Cisco 12000 Series Routers	OIR Supported?
Primary Gigabit Route Processor (GRP)	No (unless Secondary present)
Secondary Gigabit Route Processor (GRP)	Yes
Clock Scheduler Card (CSC)	No (unless redundant)
Switch Fabric Card (SFC)	Yes
Line Card (LC)	Yes
Blower Module	Yes
Power Supply	Yes
Alarm Card	Yes

Restrictions

• If you replace an Engine type A line card with an Engine type B line card, the line card configuration is not maintained across the swap, even if the line cards are of the same media type. For example, if you replace a 4xOC12 POS Engine-2 line card with a 4xOC12 POS Engine-3 line card, all configurations for the Engine-2 line card are lost and are not applied to the Engine-3 line card.

• The primary GRP should normally not be removed while the system is operating. However, if a secondary GRP is present, it takes over. Make sure to run a Cisco IOS software release that supports Route Processor Redundancy Plus (RPR+). With RPR+, the secondary GRP is fully initialized and configured. This feature dramatically shortens the switchover time if the primary GRP fails, or if the primary GRP is removed from the system. More information on RPR+ is available at Cisco 12000 Series Internet Router Architecture: Route Processor.

  For Cisco 10000 and 12000 series Internet routers that are configured to use Stateful Switchover (SSO), online removal of the active GRP automatically forces a stateful switchover to the standby GRP. SSO is supported since Cisco IOS software release 12.0(22)S. For more information, refer to Stateful Switchover.

• The CSC can be removed and replaced, only if a second (redundant) CSC is installed in the system. One CSC must be present and operational at all times to maintain normal system operations. The switch to the redundant CSC occurs in the order of seconds, during which time there can be loss of data on some/all LCs.

  On the 12406, 12416 and 12816, redundant CSCs should be removed only after shutting them down first with the hw-module slot command, with the shutdown keyword.

  On the 12406, 12416, and 12816, secondary CSC insertion causes traffic interruption and spurious CRC error reports. Post 32S3 traffic loss will not be seen

• The SFC can be removed and replaced, without disrupting normal system operations, only if a second (redundant) CSC is installed in the system (the redundant CSC can function as either the CSC or the SFC).

  On the 12406, 12410, 12416, 12810 and 12816, SFCs should be physically removed only after they are first shutdown with the hw-module slot command, with the shutdown keyword. Failure to do this could result in a linecard crash.

  On the 12406, 12410, and 12416, SFC insertion causes both traffic loss and transient errors. Post 32S3 traffic loss will not be seen

Note: The Cisco 12008, 12012, and 12016 can run with only one CSC and no SFC if it uses only Engine 0 line cards. Other line cards are automatically shut down.

Note: The 12404 has one board that contains all the CSC/SFC functionality. For the 12404, there is no redundancy. The consolidated switch fabric cannot be OIRed while the router is in function.

Details about Cisco 12000 maintenance can be found at Cisco 12000 Series Internet Routers Documentation.

Related Information

• What Causes %SYS-3-CPUHOG Messages?
• What Causes a "%RSP-3-RESTART: cbus complex"?
• Field Notice: Cisco 7507 and Cisco 7513: RSP2 HSA OIR
• Cisco 12000 Series Internet Routers Documentation
• Cisco 7500 Series Installation and Configuration Guide
• Cisco 7200 Series Hardware Installation and Maintenance
• Hardware Installation Documents for Cisco 3600 Series
• Stateful Switchover
• Routers Product Support Page
• Technical Support - Cisco Systems