Introduction
This document describes troubleshooting the FanFailed SNMP alert, caused by the rear fan tray despite appearing to originate from the front tray..
Problem
SNMP traps indicated a 'Multiple Fan Failure' on the front fan tray. Despite reseating the front tray, the issue persisted. When the tray was removed, the fans were not spinning, confirming they had no power or control. This suggested the fault was not with the front tray itself but with the rear fan tray responsible for powering and controlling it.
Wed Dec 31 10:09:15 2024 Internal trap notification 1509 (CiscoFruFanStatusChanged) FRU entity Fan tray fan status changed to Multiple Fan Failure at Upper Front
Wed Dec 31 10:09:15 2024 Internal trap notification 22 (FanFailed) fan number 1 fan status Multiple Fan Failure
Wed Dec 31 10:09:15 2024 Internal trap notification 1503 (EntStateOperDisabled) Fan Tray(Upper Front) Admin state:"Unknown", Alarm severity:"Major"
The output of show fans verbose showed that while the Upper Front tray showed all fans failed with 0 RPM and 0 mA, the Rear trays were functioning normally.
******** show fans verbose *******
Location State Speed Temp Version WD Tach Current
----------- ----------------------- ----- ---- ------- -- ---- -------
Lower Rear Normal 60% 28C 0.4.1 E -- --
...
Upper Rear Normal 65% 47C 0.4.1 E -- --
Fan 00 Normal -- -- -- -- 4280 740mA
Fan 01 Normal -- -- -- -- 4280 696mA
Fan 02 Normal -- -- -- -- 4280 720mA
Fan 03 Normal -- -- -- -- 4280 706mA
Fan 04 Normal -- -- -- -- 4280 686mA
Fan 05 Normal -- -- -- -- 4280 706mA
Fan 06 Normal -- -- -- -- 4280 706mA
Fan 07 Normal -- -- -- -- 4320 686mA
Fan 08 Normal -- -- -- -- 4280 706mA
Lower Front Normal 55% 29C 0.4.1 E -- --
...
Upper Front Multiple Fan Failure 55% 46C 0.4.1 E -- --
Fan 09 Single Fan Failure -- -- -- -- 0 0mA
Fan 10 Single Fan Failure -- -- -- -- 0 0mA
Fan 11 Single Fan Failure -- -- -- -- 0 0mA
Solution
The front fan trays receive power and control from the rear fan trays. If the front fans are not spinning—as shown by show fans verbose with 0 RPM and 0 mA readings—you must also visually confirm that the fans are not rotating when the tray is removed. If confirmed, the root cause is likely a failure in both rear fan trays or the midplane connections.
To troubleshoot and resolve:
- Review logs and
show fans verbose output to identify discrepancies and verify failure symptoms.
- Re-seat trays one at a time:
- Start with the front fan tray. After reseating, run
show fans verbose to check if the fans resume operation.
- If the issue persists, re-seat the corresponding rear fan tray. Again, use
show fans verbose to confirm whether the fans regain proper speed and current.
- Visually confirm fan movement during tray removal. If fans are not spinning, this validates the telemetry data.
- Inspect air filters while trays are removed. Clean or replace if obstructed. Blocked airflow can trigger thermal issues.
- Refer to the Cisco ASR 5500 Installation Guide for detailed procedures on removing and reinstalling fan trays and filters.
- Replace the rear fan tray if reseating and filter inspection do not resolve the issue.
- Verify normal operation via
show fans verbose. All fans must report proper RPM and current, and no new SNMP traps must appear.
- If the issue persists, collect
show support details and open a TAC case with Cisco for further analysis and assistance.
Feedback