> Redundancy and Availability Features
> Every minute of downtime and every dropped session represents lost revenue to the wireless operator resulting in potential customer loss and reduced profitability. With this understanding, we have developed a system that exceeds the availability features found in the majority of today's wireless and wireline access devices.
> Service Availability Features
> In its recommended redundant configuration, the system provides the highest level of service assurance. Following is detailed information describing the service availability features found in the system.
> Hardware Redundancy Features
> In addition to providing the highest transaction rates and session capacity, the system is designed to provide robust hardware reliability and service assurance features.
> Features of the hardware design include:
> ASR 5000

> • | System Management Card (SMC) redundancy

> • | 1:n Packet Services Cards (PSC/PSC2) redundancy, allowing redundancy of multiple active to multiple redundant for up to 14 total PSCs or PSC2s

> Important: 1:1 redundancy is supported for these cards however some subscriber sessions and accounting information may be lost in the event of a hardware or software failure even though the system remains operational.

> • | 1:1 Optical (ATM) line card (LC) redundancy (OLC and OLC2)

> • | 1:1 Channelized (STM-1/OC-3) line card (LC) redundancy (CLC and CLC2)

> • | 1:1 Quad Gigabit Ethernet Line Card (QGLC)

> • | 1:1 10 Gigabit Ethernet Line Card (XGLC)

> • | 1:1 Switch Processor I/O (SPIO) card redundancy

> • | 1:1 10/100 Ethernet Line Card (FELC)

> • | 1:1 1000 Gigabit Ethernet Line Cards (GELC)

> • | Configurable line card port redundancy (Ethernet, ATM, and SPIO line cards)

> • | Redundancy Crossbar Card (RCC) for processor-card-to-line card failover using the 280 Gbps Redundancy Bus

> • | Self-healing redundant 320 Gbps switching fabric

> • | Redundant 32 Gbps Control Bus

> • | Redundant Power Filter Units (PFUs)

> • | Hot-swappable cards, allowing dynamic replacement while the system is operational

> Hardware Redundancy Configuration
> The maximum redundant configuration for a fully loaded system supporting data services consists of the following:

> • | 2 SMCs: 1 active and 1 standby (redundant)

> • | 14 processing cards: 13 active and 1 standby

> • | 2 SPIOs: 1 active and 1 standby

> • | 26 Ethernet/Gigabit Ethernet line cards: 13 active and 13 standby (10/100 Ethernet Line Card (FELC), 1000 Gigabit Line Card (GELC), and Quad Gigabit Ethernet Line Card (QGLC))

> • | 2 1000 Gigabit Ethernet Line Cards (XGLC): 1 active, 1 standby. Note that the XGLC, which is a full-height line card that populates both the upper and lower slots of the chassis, uses a side-by-side redundancy scheme. Refer to the Hardware Installation and Administration Guide for more information.

> • | 26 Optical (ATM) line cards: 13 active and 13 standby (OLC and OLC2)

> • | 26 Channelized line cards: 13 active and 13 standby (CLC and CLC2)

> • | 2 RCCs: 2 standby

> This configuration allows for the highest session capacity while still providing redundancy. The following figures depict this recommended maximum redundant configuration.
> Recommended Redundant Configuration for Data Services - Front View
> Recommended Redundant Configuration for Data Services - Rear View
> Maintenance and Failure Scenarios
> The following table shows various maintenance and failure scenarios involving the SMC and SPIO cards; and explains how each situation is resolved.
> Service Assurance Features for the SMC and SPIO

> Hardware Failure Scenario | Action Taken | Effect on Accounting Data | Effect on User Sessions | Effect on the Flow of User Data Packets | Effect on User Control Transactions | Effect on Management Traffic
> SMC - Planned maintenance | Tasks are switched over to standby SMC. SPIO remains active. | No impact | No impact | No impact | No impact | < 1 sec. Interrupt
> Unplanned SMC failure | Standby SMC takes control of all system & management processes as SPIO remains active. | No impact | No impact | < 2 sec. interrupt | < 1 min. interrupt | < 1 min. interrupt
> SPIO failure | Standby SPIO takes over, using active SMC. | No impact | No impact | No impact | No impact | < 1 sec. interrupt
> Software upgrade | After applying a soft busy-out to the system, performs a soft boot after the last session disconnects. | Service interrupt for the duration of system boot (~4 min) | Service interrupt for the duration of system boot (~4 min) | Service interrupt for the duration of system boot (~4 min) | Service interrupt for the duration of system boot (~4 min) | Service interrupt for the duration of system boot (~4 min)

> ‌
> Important: When an SMC or SPIO failover occurs, the standby SMC or SPIO automatically becomes active. However, should the failed card's error condition be corrected (by replacement or configuration change), the state of the repaired SMC or SPIO does not automatically return to the active state. This migration must occur through manual intervention by a system administrative user.
> With the ability of performing on-line process migration, supporting 1:1 SMC and SPIO redundancy, and utilizing the fully redundant switching fabric and control bus, single points of failure are eliminated from the switch fabric and system management capabilities.
> The following table shows various maintenance and failure situations involving the processing cards (PSC, PSC2, PPC), Line Cards (LCs), and RCC cards; and explains how each situation is resolved. Note that LCs are not needed behind the standby processing cards that provide redundancy.
> Service Assurance Features for Processing and Line Cards

> Hardware Failure Scenario | Action Taken | Effect on Accounting Data | Effect on User Sessions | Effect on the flow of Data Packets | Effect on Control Transactions
> Processing Card Planned maintenance | Session managers are migrated to standby processing card. Other tasks are restarted on standby card. Network connection is maintained on existing LC via RCC. | No impact | No impact | < 2 sec. interrupt to user traffic on affected processing card (user application will retransmit data) | < 2 sec. interrupt to new call setups (PCF/SGSN and mobile nodes will retransmit requests)
> Unplanned processing card failure, no Session Recovery | Tasks are restarted on standby processing card. Network connection is maintained on existing LC via RCC. | AAA Acct_Stop record is generated for all sessions in the affected subgroup | Sessions lost on affected processing card only | Lost only for the effected sessions | < 5 sec. interrupt until new A11/GTP-C manager is available (new sessions only)NOTE: Applies only when A11/GTP-C manager is on failed card
> Unplanned processing card failure, with Session Recovery | Sessions are recovered on the standby processing card. Network connection is maintained on existing LC via RCC | No impact (less interim update interval) | No impact | < 5 sec. interrupt | < 5 sec. interrupt (new sessions only)
> Unplanned LC failure | Standby LC becomes active if installed in 1:1 redundant configuration. | No impact (less update interval) | No impact | < 1 sec. interrupt | < 1 sec. interrupt
> Unplanned LC port failure | With LC port redundancy enabled, standby port is enabled. | No impact | No impact | < 1 sec. interrupt | < 1 sec. interrupt
> 1. This does not apply to for deployments containing only 1 active processing card.

> ‌
> Important: If the session recovery feature is enabled, then a processing card hardware failure will not cause any loss of fully established HA subscriber sessions. This feature does, however, require a minimum processing card configuration per chassis of three active cards and two standby to prevent all data loss and session recovery.
> Important: When a processing or line card failover occurs, the redundant component (when installed) automatically begins providing service. However, once the failed card's error condition is corrected (by replacement or configuration change), there is no automatic return of control to the repaired processing or line card. This migration must occur through manual intervention by a system administrative user.
> Software Assurance Features
> Numerous features are built into the system software to ensure the continuation of service in the case of software process failures. SMC software controls the management contexts and overall system control, while processing card software controls the PPP sessions, AAA, and VPN processes.
> The following table shows various software process failure situations involving the SMC and SPIO cards, provides impact analysis (if any), and explains how each situation is resolved using rapid failure detection techniques found in the system.
> Service Assurance Features for the SPC/SMC Software

> Software Process Failure Scenario | Action Taken | Effect on Accounting Data | Effect on User Sessions | Effect on the Flow of User Data Packets | Effect on User Control Transactions | Effect on Management Traffic
> SMC - Management task failure | Cleanup process performs automatically, and process is restarted | No impact | No impact | No impact | No impact | < 1 sec. interrupt
> SMC - System control task failure | The same process for unplanned hardware failure (table above) is applied | No impact | No impact | < 2 sec. interrupt | No impact | < 1 min. interrupt

> ‌
> The following table shows various software process failure situations involving the processing cards, provides impact analysis (if any), and explains how each situation is resolved using rapid failure detection techniques found in the system.
> Service Assurance Features for the Processing Cards Software

> Software Process Failure Scenario | Action Taken | Effect on Accounting Data | Effect on User Sessions | Effect on the flow of Data Packets | Effect on Control Transactions
> Processing Cards - Session Manager Task failure | Cleanup process performs automatically, and process is restarted | AAA Acct._Stop record is generated for all sessions in the affected subgroup | Affected subgroup sessions are lost For PSC/PSC2: up to 13200 for PDSN 13200 for PDIF 13200 for ASN GW, 26400 for HA, and 26400 GGSN) | Lost only for the affected subgroup | Lost only for the affected subgroup
> Processing Cards - AAA failure | Cleanup process performs automatically, and process is restarted | No impact | No impact | No impact | No impact
> Processing Cards - VPN context failure | Cleanup process performs automatically, and process is restarted | No impact | No impact | < 1 sec. interrupt for VPN context | < 1 sec. interrupt for VPN context
> 1. This Assumes that there is more than 1 active processing card. 2. The information in this row applies to systems on which the Session Recovery feature is not implemented. With the Session Recovery Feature enabled, no sessions are lost.

> ‌
> Session Recovery Feature
> This licensed software feature performs an automatic recovery of all fully established subscriber sessions should a session manager task failure occur. This functionality is available for the following call types:

> • | PDSN PDIF services supporting simple IP, Mobile IP, and Proxy Mobile IP

> • | HA services supporting Mobile IP and/or Proxy Mobile IP session types with or without per-user Layer 3 tunnels

> • | GGSN services for IPv4 and PPP PDP contexts

> • | LNS session types

> With this feature enabled, there is no loss of session information as described in table above. Session recovery consists of the migration and recreation of control and data packet state information, subscriber session statistics, or session time parameters such as idle timer and others.
> Typical recovery time for a single session manager failure is not expected to exceed 10 seconds. Should a processing card hardware failure occur during a migration, then the time to recover all tasks and subscriber sessions should not exceed 60 seconds.
> This feature is enabled/disabled on a chassis-wide basis and requires additional processing card hardware to ensure that enough reserve resources (memory, processing, etc.) are available to fully recover session in the event of a software or hardware failure.
> Interchassis Session Recovery
> The Interchassis Session Recovery feature provides the highest possible availability for continuous call processing without interrupting subscriber services. This is accomplished through the use of redundant chassis. The chassis are configured as primary and backup with one being active and one inactive. Both chassis are connected to the AAA server. When calls pass the checkpoint duration timer, checkpoint data is sent from the active chassis to the inactive chassis. If the active chassis handling the call traffic goes out of service, the inactive chassis transitions to the active state and continues processing the call traffic without interrupting the subscriber session.
> The chassis determine which is active through a propriety TCP-based connection called a redundancy link. This link is used to exchange status messages between the primary and backup chassis and must be maintained for proper system operation. In the event the redundancy link goes out of service, interchassis session recovery is maintained through the use of authentication probes and BGP peer monitoring. BGP routing must be enabled.
> Interchassis Session Redundancy is currently supported on chassis configured for GGSN service or HA services in support of Mobile IP and Proxy Mobile IP session types.
> Mean Time Between Failure and System Availability
> Mean Time Between Failure (MTBF) data is used to provide statistical information as to the length of time that should expire before a particular card or system fails. This information is calculated using the following method:
> Calculated MTBF - Expected elapsed time before failure occurs using the method defined in Telcordia TR-NWT-000332-CORE. This is based on reliability of components and design factors.
> Failure per million hours (Fpmh) identifies the predicted failure rate per one million hours (for every 1,000,000 hours of operation, “FITS number” of failures would be expected to occur) for a component of the system.
> MTBF Table
> The following table shows the MTBF characteristics of each major component of the system.
> Mean Time Between Failure Statistics

> Part Number | Description | MTBF (Hours) | MTBF (Years) | Fpmh (Failure per million hours)
> 600-00-1111 | Chassis with Midplane | 16,386,995 | 1869.38 | 0.061
> 600-00-3026 | System Management Card | 104,372 | 11.91 | 9.58
> 600-00-3025 | Packet Services Card (PSC or PSC2) | 102,294 | 11.68 | 9.78
> 600-00-5052 | 10 Gigabit Ethernet Line Card (XGLC) | 247,720 | 28.28 | 4.04
> 600-00-5038 Multi-Mode 600-00-5051 Single Mode 600-00-5039 Copper | Quad Gig-E Card (QGLC) | 258,606 | 29.52 | 3.867
> 600-00-5016 | ATM/POS OC-3 SM IR-1 Card optical daughter card | 214,492 1,419,581 | 48.6 73.4 | 4.66 0.70
> 600-00-5001 | Switch Processor I/O Card | 333,999 | 38.13 | 2.99
> 600-00-5002 | Redundancy Crossbar Card | 555,862 | 63.46 | 1.79
> 600-00-5003 | Ethernet 10/100 Card (FELC) | 495,886 | 56.61 | 2.01
> 600-00-5101 | Ethernet 1000 Card (GELC) | 396,715 | 45.29 | 2.52
> 600-00-1112 | Power Filter Unit (165A) | 967,118 | 110.40 | 1.03
> 600-00-1104 | Fan Tray Unit - Lower | 70,517 | 8.05 | 19.51
> 600-00-1103 | Fan Blower Unit - Upper | 120,178 | 13.72 | 18.72

> ‌
> System Availability
> System-level Mean Time To Failure (MTTF), is the average interval of time that a component will operate before failing. Reliability information is based on the number of overall anticipated failures of the individual components, in conjunction with any redundancy schemes employed to minimize the impact of such failures.
> The following table provides service availability calculations (based on reliability modeling) for the ASR 5000 platform.
> Platform Service Availability Calculations

> Platform | Operational Uptime | Yearly Downtime | MTTF
> (%) | (minutes) | Hours | Years
> ASR 5000 | 99.999978 | 0.12 | 14,077,473 | 1605.91

> ‌
> One suggestion to help improve overall system availability is to institute an on-site spares program, wherein key components are housed locally with the deployed equipment. The following section defines a recommended spares program and quantities for the system.
> Mean Time To Repair (MTTR) is the amount of time needed to repair a component, recover the system, or otherwise restore service after a failure. System availability calculations are based on the industry standard of four hours.
> Spare Component Recommendations
> This section provides a recommended quantity of spare parts to be used as part of a spare components program for the system. The information contained is for informational purposes only, and should only be used as a guideline for designing a spares program that meets your company's design, deployment, and availability goals.
> It is recommended that your company either has fully-trained personnel available to effect the exchange of Field Replaceable Units (FRUs) within your network, or requests on-site or field engineering resources to perform such duties.
> Based on industry-leading redundancy and failover features found in the system, the following minimum spare parts levels for any planned deployment are recommended.
> Recommended FRU Parts Sparing Quantities

> Component Name | Minimum number of spares | For every “n” number of deployed components
> ASR 5000 Chassis with Midplane | 1 | 20
> System Management Card (SMC) | 1 | 10
> Packet Services Card (PSC or PSC2) | 1 | 12
> Ethernet 1000/Quad Gig-E (QGLC) Card | 1 | 20
> 10 Gigabit Ethernet Line Card (XGLC) | 1 | 20
> Optical Line Card (OLC or OLC2) | 1 | 20
> Channelized Line Card (CLC or CLC2) | 1 | 20
> Switch Processor I/O Card (SPIO) | 1 | 18
> Redundancy Crossbar Card (RCC) | 1 | 30
> Ethernet 10/100 Line Card (FELC) | 1 | 25
> Gigabit Ethernet Line Card (GELC) | 1 | 25
> Power Filter Unit (165A) | 1 | 30
> Upper Fan Tray Unit | 1 | 8
> Lower Fan Tray Unit | 1 | 5
> Particulate Air Filter | 1 | 1

> ‌

Cisco Systems Inc.
Tel: 408-526-4000
Fax: 408-527-0883