Introduction
This document describes Cisco bug ID CSCwa79915 conditions, symptoms, trigger, and mitigation options to recover the appliance.
Prerequisites
Requirements
Cisco recommends that you have knowledge of these topics:
- Firepower eXtensible Operating System (FXOS)
- Adaptive Security Appliance (ASA)
- LInux NAtively (LINA)
- Firepower Threat Defense (FTD)
Components Used
The information in this document is based on this hardware model and software version:
- Firepower 2110
- FTD 6.6.5 (bundled with FXOS version 2.8.1.165)
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.
Known Conditions for Susceptibility
Current known conditions related to Cisco bug ID CSCwa79915 include:
1. Firepower 2100 series appliance.
2. One or more external directed chassis ports that run in Half-duplex mode (whether intentional, or as a duplex mismatch result).
3. Configured with any affected release of Adaptive Security Appliance (ASA) or Firepower Threat Defense (FTD) software.
Bug Symptoms
1. Packets Sourced by ASA/LINA (FTD) Never Leave the Appliance.
The most classic/common observation of this state is that all data interfaces show very little traffic from its interfaces.
When a capture is placed in this condition, it reveals that Address Resolution Protocol (ARP) Requests are sent by other hosts on the same subnet and query for the Layer 2 address of the LInux NAtively (LINA) IP address is received and the LINA capture does show a reply. However, those ARP replies are not seen to leave the chassis, as revealed when a Switched Port Analyzer (SPAN) is performed on the external switch in which the respective chassis interfaces assigned to the LINA are connected.
For instance:
firepower# show capture arp
4 packets captured
1: 14:43:44.185872 arp who-has 10.255.255.1 tell 10.255.255.2
2: 14:43:44.186132 arp reply 10.255.255.1 is-at b0:8b:cf:8c:61:4f
3: 14:43:45.205906 arp who-has 10.255.255.1 tell 10.255.255.2
4: 14:43:45.206166 arp reply 10.255.255.1 is-at b0:8b:cf:8c:61:4f
Where 10.255.255.2 is the IP address of an external host that sends ARP queries to 10.255.255.1, which belongs to one of the LINA data interfaces.
ARP replies that are seen as transmitted by the LINA on its capture are never actually seen to leave the respective physical chassis port.
2. FXOS Interface Counters for TX Packets do not Increment.
Similar to the symptom where external hosts never receive any packets from the affected appliance, as evidenced by all packets sent by the LINA do not leave the chassis, we have the symptom where the external port counters for transmitted (TX) packets do not increment.
In this example, the affected interface is Ethernet1/12. A check of the Firepower eXtensible Operating System (FXOS) interface counters for TX packets showed the counters never incremented, despite the indication from the LINA that those packets were transmitted to the internal chassis switch.
firepower# scope eth-uplink
firepower/eth-uplink # scope fabric
firepower/eth-uplink/fabric # scope interface 1 12 <<< interface Eth1/12
firepower/eth-uplink/fabric/interface # show stats ether-tx-stats
Ether Tx Stats:
Time Collected: 2021-12-09T17:29:45.621 <<< first execution of the command
Monitored Object: sys/switch-A/slot-1/switch-ether/port-8
Suspect: No
Total Packets (packets): 4823522 <<< Counter of packets transmitted
Unicast Packets (packets): 4823515
Multicast Packets (packets): 0
Broadcast Packets (packets): 7
Total Bytes (bytes): 606771974
Jumbo Packets (packets): 0
Thresholded: 0
firepower/eth-uplink/fabric/interface # show stat ether-tx-stats
Ether Tx Stats:
Time Collected: 2021-12-09T17:30:15.726 <<< second execution of the command
Monitored Object: sys/switch-A/slot-1/switch-ether/port-8
Suspect: No
Total Packets (packets): 4823522 <<< Counter of packets transmitted (No delta seen)
Unicast Packets (packets): 4823515
Multicast Packets (packets): 0
Broadcast Packets (packets): 7
Total Bytes (bytes): 606771974
Jumbo Packets (packets): 0
Thresholded: 0
3. Drops on Internal Data/Backplane Interface Between the Internal Chassis Switch and ASA/LINA.
Interface Internal1/3 is used as a backplane/uplink interface between the switch on the logical device that runs on the chassis.
firepower#
firepower# connect local-mgmt
firepower(local-mgmt)# show portmanager counters internal 1 3 <<< first execution of the command
Good Octets Received : 23510696205
Bad Octets Received : 0
MAC Transmit Error : 0
Good Packets Received : 49729185
Bad Packets Received : 0
BRDC Packets Received : 1704250
MC Packets Received : 320755
Size 64 : 21746457
Size 65 to 127 : 112073389
Size 128 to 255 : 7536865
Size 256 to 511 : 3053841
Size 512 to 1023 : 2490597
Size 1024 to Max : 0
Good Octets Sent : 27203100553
Good Packets Sent : 122656923
Excessive Collision : 0
MC Packets Sent : 1095115
BRDC Packets Sent : 90585686
Unrecognized MAC Received : 0
FC Sent : 0
Good FC Received : 0
Drop Events : 16837069
Undersize Packets : 0
Fragments Packets : 0
Oversize Packets : 0
Jabber Packets : 0
MAC RX Error Packets Received : 0
Bad CRC : 0
Collisions : 0
Late Collision : 0
bad FC Received : 0
Good UC Packets Received : 47704180
Good UC Packets Sent : 30976122
Multiple Packets Sent : 0
Deferred Packets Sent : 0
Size 1024 to 15180 : 0
Size 1519 to Max : 0
txqFilterDisc : 0
linkChange : 1
firepower(local-mgmt)# show portmanager counters internal 1 3 <<< second execution of the command
Good Octets Received : 23510700469
Bad Octets Received : 0
MAC Transmit Error : 0
Good Packets Received : 49729250 >>>> 49729250 – 49729185 = 65 packets received from FTD
Bad Packets Received : 0
BRDC Packets Received : 1704261
MC Packets Received : 320759
Size 64 : 21746518
Size 65 to 127 : 112074355
Size 128 to 255 : 7536866
Size 256 to 511 : 3053847
Size 512 to 1023 : 2490606
Size 1024 to Max : 0
Good Octets Sent : 27203179868
Good Packets Sent : 122657901
Excessive Collision : 0
MC Packets Sent : 1095130
BRDC Packets Sent : 90586649
Unrecognized MAC Received : 0
FC Sent : 0
Good FC Received : 0
Drop Events : 16837134 >>>>> 16837134 – 16837069 = 65 packets dropped (matching above counter)
Undersize Packets : 0
Fragments Packets : 0
Oversize Packets : 0
Jabber Packets : 0
MAC RX Error Packets Received : 0
Bad CRC : 0
Collisions : 0
Late Collision : 0
bad FC Received : 0
Good UC Packets Received : 47704230
Good UC Packets Sent : 30976122
Multiple Packets Sent : 0
Deferred Packets Sent : 0
Size 1024 to 15180 : 0
Size 1519 to Max : 0
txqFilterDisc : 0
linkChange : 1
firepower(local-mgmt)#
Note: For a live traffic environment, interface counter verification can be difficult due to noise, so first verify and correct the half duplex mode.
Symptom Trigger
A check of the active interfaces status shows one of the active/UP data interfaces is in in Half-duplex mode, which is unusual to be seen in general.
firepower#
firepower# connect local-mgmt
firepower(local-mgmt)# show portmanager switch status
Dev/Port Mode Link Speed Duplex Loopback Mode
--------- ---------------- ----- ----- ------ -------------
0/0 QSGMII Down 1G Half None
0/1 QSGMII Up 1G Full None
0/2 QSGMII Down 1G Half None
0/3 QSGMII Down 1G Half None
0/4 QSGMII Down 1G Half None
0/5 QSGMII Down 1G Half None
0/6 QSGMII Up 100 Half None <<<<< Up and Half-duplex
0/7 QSGMII Down 1G Half None
0/8 QSGMII Down 1G Half None
0/9 QSGMII Up 1G Full None
0/10 QSGMII Up 1G Full None
0/11 QSGMII Up 1G Full None
0/12 QSGMII Up 1G Full None
0/13 QSGMII Down 10 Half None
0/14 QSGMII Down 10 Half None
0/15 QSGMII Down 10 Half None
0/16 n/a Down n/a Full N/A
0/17 n/a Down n/a Full N/A
0/18 n/a Down n/a Full N/A
0/19 n/a Down n/a Full N/A
0/20 n/a Down n/a Full N/A
0/21 n/a Down n/a Full N/A
0/22 n/a Down n/a Full N/A
0/23 n/a Down n/a Full N/A
0/24 KR Up 10G Full None
0/25 KR Up 10G Full None
0/26 KR Down 10G Full None
0/27 KR Up 10G Full None
This table provides the mapping of the physical chassis interface to the internal switch port number. This mapping is required to understand the output from show portmanager switch status. Based on the table, we can see that for the internal switch port ID 0/6 (seen on the previous output of show portmanager switch status), the associated physical chassis port is Ethernet1/8.
Interface Name Internal Switch Port (2110/2120) Internal Switch Port (2130/2140)
Ethernet 1/1 1 1
Ethernet 1/2 0 0
Ethernet 1/3 3 3
Ethernet 1/4 2 2
Ethernet 1/5 5 5
Ethernet 1/6 4 4
Ethernet 1/7 7 7
Ethernet 1/8 6 6
Ethernet 1/9 9 49
Ethernet 1/10 8 48
Ethernet 1/11 11 51
Ethernet 1/12 10 50
Ethernet 1/13 12 59
Ethernet 1/14 13 58
Ethernet 1/15 14 57
Ethernet 1/16 15 56
Ethernet 2/1 N/A 70
Ethernet 2/2 N/A 71
Ethernet 2/3 N/A 69
Ethernet 2/4 N/A 68
Ethernet 2/5 N/A 66
Ethernet 2/6 N/A 67
Ethernet 2/7 N/A 65
Ethernet 2/8 N/A 64
Internal 1/1 26 81 (Eventing Port - NOT visible at Service Manager)
Internal 1/2 27 80 (Unused - NOT visible at Service Manager)
Internal 1/3 24 52 (Internal backplane uplink to logical device, whether ASA or FTD)
Options to Mitigate the Trigger and Recover the Appliance
Correction of any duplex mismatch is the only way to prevent the side effect seen on the backplane interface, and this can be done by one of these methods and a reload of the appliance.
1. If peer device is not configured with Duplex Auto, change it to Auto (preferred method).
2. If there is no management access to the peer device:
2.1. For FTD managed by Firepower Management Center (FMC), disable option Auto-Negotiation under Edit Physical Interface on FMC.
2.2. For FTD managed by Firepower Device Manager (FDM), change Duplex option from Auto to Full under Interface Advanced Options.
2.3. For ASA, disable Duplex Auto negotiation from chassis level as follows:
firepower /eth-uplink # scope
firepower /eth-uplink # scope fabric a
firepower /eth-uplink/fabric # scope interface 1 1
firepower /eth-uplink/fabric/interface # set auto-negotiation
no No
yes Yes
firepower /eth-uplink/fabric/interface # set auto-negotiation no
firepower /eth-uplink/fabric/interface* # commit-buffer
firepower /eth-uplink/fabric/interface #
Note: Methods listed in step 2 are theoretical options that can work under normal conditions.
3. Connect the Firepower interface in half-duplex mode to a different switch that supports auto-negotiation of duplex settings or configure the switch port to enable auto-negotiation of duplex settings.
Note: A reload of the whole appliance is still necessary after the execution of any of the steps, in order to recover the backplane interface from its failed state.
Cisco bug ID Information
This bug was raised to track a software resolution of the symptom in which the backplane Internal1/3 interface is unable to process any traffic received from the LINA after some time.
Cisco bug ID CSCwa79915 Physical port in Half Duplex causes all packets from LINA to be dropped by the chassis.
Feedback