Contents

Introduction

Prerequisites

Requirements

Components Used

Known Conditions for Susceptibility

Bug Symptoms

1. Packets Sourced by ASA/LINA (FTD) Never Leave the Appliance.

2. FXOS Interface Counters for TX Packets do not Increment.

3. Drops on Internal Data/Backplane Interface Between the Internal Chassis Switch and ASA/LINA.

Symptom Trigger

Options to Mitigate the Trigger and Recover the Appliance

Cisco bug ID Information

Introduction

This document describes Cisco bug ID CSCwa79915 conditions, symptoms, trigger, and mitigation options to recover the appliance.

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics:

- Firepower eXtensible Operating System (FXOS)

- Adaptive Security Appliance (ASA)

- LInux NAtively (LINA)

- Firepower Threat Defense (FTD)

Components Used

The information in this document is based on this hardware model and software version:

- Firepower 2110

- FTD 6.6.5 (bundled with FXOS version 2.8.1.165)

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Known Conditions for Susceptibility

Current known conditions related to Cisco bug ID CSCwa79915 include:

1. Firepower 2100 series appliance.

2. One or more external directed chassis ports that run in Half-duplex mode (whether intentional, or as a duplex mismatch result).

3. Configured with any affected release of Adaptive Security Appliance (ASA) or Firepower Threat Defense (FTD) software.

Bug Symptoms

1. Packets Sourced by ASA/LINA (FTD) Never Leave the Appliance.

The most classic/common observation of this state is that all data interfaces show very little traffic from its interfaces.

When a capture is placed in this condition, it reveals that Address Resolution Protocol (ARP) Requests are sent by other hosts on the same subnet and query for the Layer 2 address of the LInux NAtively (LINA) IP address is received and the LINA capture does show a reply. However, those ARP replies are not seen to leave the chassis, as revealed when a Switched Port Analyzer (SPAN) is performed on the external switch in which the respective chassis interfaces assigned to the LINA are connected.

For instance:

firepower# show capture arp

4 packets captured

1: 14:43:44.185872       arp who-has 10.255.255.1 tell 10.255.255.2

2: 14:43:44.186132       arp reply 10.255.255.1 is-at b0:8b:cf:8c:61:4f

3: 14:43:45.205906       arp who-has 10.255.255.1 tell 10.255.255.2

4: 14:43:45.206166       arp reply 10.255.255.1 is-at b0:8b:cf:8c:61:4f

Where 10.255.255.2 is the IP address of an external host that sends ARP queries to 10.255.255.1, which belongs to one of the LINA data interfaces.

ARP replies that are seen as transmitted by the LINA on its capture are never actually seen to leave the respective physical chassis port.

2. FXOS Interface Counters for TX Packets do not Increment.

Similar to the symptom where external hosts never receive any packets from the affected appliance, as evidenced by all packets sent by the LINA do not leave the chassis, we have the symptom where the external port counters for transmitted (TX) packets do not increment.

In this example, the affected interface is Ethernet1/12. A check of the Firepower eXtensible Operating System (FXOS) interface counters for TX packets showed the counters never incremented, despite the indication from the LINA that those packets were transmitted to the internal chassis switch.

firepower# scope eth-uplink

firepower/eth-uplink # scope fabric

firepower/eth-uplink/fabric # scope interface 1 12   <<< interface Eth1/12

firepower/eth-uplink/fabric/interface # show stats ether-tx-stats

Ether Tx Stats:

Time Collected: 2021-12-09T17:29:45.621   <<< first execution of the command

Monitored Object: sys/switch-A/slot-1/switch-ether/port-8

Suspect: No

Total Packets (packets): 4823522   <<< Counter of packets transmitted

Unicast Packets (packets): 4823515

Multicast Packets (packets): 0

Broadcast Packets (packets): 7

Total Bytes (bytes): 606771974

Jumbo Packets (packets): 0

Thresholded: 0

firepower/eth-uplink/fabric/interface # show stat ether-tx-stats

Ether Tx Stats:

Time Collected: 2021-12-09T17:30:15.726   <<< second execution of the command

Monitored Object: sys/switch-A/slot-1/switch-ether/port-8

Suspect: No

Total Packets (packets): 4823522   <<< Counter of packets transmitted (No delta seen)

Unicast Packets (packets): 4823515

Multicast Packets (packets): 0

Broadcast Packets (packets): 7

Total Bytes (bytes): 606771974

Jumbo Packets (packets): 0

Thresholded: 0

3. Drops on Internal Data/Backplane Interface Between the Internal Chassis Switch and ASA/LINA.

Interface Internal1/3 is used as a backplane/uplink interface between the switch on the logical device that runs on the chassis.

firepower#

firepower# connect local-mgmt

firepower(local-mgmt)# show portmanager counters internal 1 3   <<< first execution of the command

Good Octets Received : 23510696205

Bad Octets Received : 0

MAC Transmit Error : 0

Good Packets Received : 49729185

Bad Packets Received : 0

BRDC Packets Received : 1704250

MC Packets Received : 320755

Size 64 : 21746457

Size 65 to 127 : 112073389

Size 128 to 255 : 7536865

Size 256 to 511 : 3053841

Size 512 to 1023 : 2490597

Size 1024 to Max : 0

Good Octets Sent : 27203100553

Good Packets Sent : 122656923

Excessive Collision : 0

MC Packets Sent : 1095115

BRDC Packets Sent : 90585686

Unrecognized MAC Received : 0

FC Sent : 0

Good FC Received : 0

Drop Events : 16837069

Undersize Packets : 0

Fragments Packets : 0

Oversize Packets : 0

Jabber Packets : 0

MAC RX Error Packets Received : 0

Bad CRC : 0

Collisions : 0

Late Collision : 0

bad FC Received : 0

Good UC Packets Received : 47704180

Good UC Packets Sent : 30976122

Multiple Packets Sent : 0

Deferred Packets Sent : 0

Size 1024 to 15180 : 0

Size 1519 to Max : 0

txqFilterDisc : 0

linkChange : 1

firepower(local-mgmt)# show portmanager counters internal 1 3   <<< second execution of the command

Good Octets Received : 23510700469

Bad Octets Received : 0

MAC Transmit Error : 0

Good Packets Received : 49729250     >>>> 49729250 – 49729185 = 65 packets received from FTD

Bad Packets Received : 0

BRDC Packets Received : 1704261

MC Packets Received : 320759

Size 64 : 21746518

Size 65 to 127 : 112074355

Size 128 to 255 : 7536866

Size 256 to 511 : 3053847

Size 512 to 1023 : 2490606

Size 1024 to Max : 0

Good Octets Sent : 27203179868

Good Packets Sent : 122657901

Excessive Collision : 0

MC Packets Sent : 1095130

BRDC Packets Sent : 90586649

Unrecognized MAC Received : 0

FC Sent : 0

Good FC Received : 0

Drop Events : 16837134   >>>>> 16837134 – 16837069 = 65 packets dropped (matching above counter)

Undersize Packets : 0

Fragments Packets : 0

Oversize Packets : 0

Jabber Packets : 0

MAC RX Error Packets Received : 0

Bad CRC : 0

Collisions : 0

Late Collision : 0

bad FC Received : 0

Good UC Packets Received : 47704230

Good UC Packets Sent : 30976122

Multiple Packets Sent : 0

Deferred Packets Sent : 0

Size 1024 to 15180 : 0

Size 1519 to Max : 0

txqFilterDisc : 0

linkChange : 1

firepower(local-mgmt)#

Note: For a live traffic environment, interface counter verification can be difficult due to noise, so first verify and correct the half duplex mode.

Symptom Trigger

A check of the active interfaces status shows one of the active/UP data interfaces is in in Half-duplex mode, which is unusual to be seen in general.

firepower#

firepower# connect local-mgmt

firepower(local-mgmt)# show portmanager switch status

Dev/Port      Mode         Link  Speed Duplex Loopback Mode

--------- ---------------- ----- ----- ------ -------------

0/0          QSGMII        Down   1G    Half   None

0/1          QSGMII         Up    1G    Full   None

0/2          QSGMII        Down   1G    Half   None

0/3          QSGMII        Down   1G    Half   None

0/4          QSGMII        Down   1G    Half   None

0/5          QSGMII        Down   1G    Half   None

0/6          QSGMII         Up    100   Half   None    <<<<< Up and Half-duplex

0/7          QSGMII        Down   1G    Half   None

0/8          QSGMII        Down   1G    Half   None

0/9          QSGMII         Up    1G    Full   None

0/10         QSGMII         Up    1G    Full   None

0/11         QSGMII         Up    1G    Full   None

0/12         QSGMII         Up    1G    Full   None

0/13         QSGMII        Down   10    Half   None

0/14         QSGMII        Down   10    Half   None

0/15         QSGMII        Down   10    Half   None

0/16          n/a          Down   n/a   Full   N/A

0/17          n/a          Down   n/a   Full   N/A

0/18          n/a          Down   n/a   Full   N/A

0/19          n/a          Down   n/a   Full   N/A

0/20          n/a          Down   n/a   Full   N/A

0/21          n/a          Down   n/a   Full   N/A

0/22          n/a          Down   n/a   Full   N/A

0/23          n/a          Down   n/a   Full   N/A

0/24          KR            Up    10G   Full   None

0/25          KR            Up    10G   Full   None

0/26          KR           Down   10G   Full   None

0/27          KR            Up    10G   Full   None

This table provides the mapping of the physical chassis interface to the internal switch port number. This mapping is required to understand the output from show portmanager switch status. Based on the table, we can see that for the internal switch port ID 0/6 (seen on the previous output of show portmanager switch status), the associated physical chassis port is Ethernet1/8.

Interface Name Internal Switch Port (2110/2120) Internal Switch Port (2130/2140)

Ethernet 1/1                1                  1

Ethernet 1/2                0                  0

Ethernet 1/3                3                  3

Ethernet 1/4                2                  2

Ethernet 1/5                5                  5

Ethernet 1/6                4                  4

Ethernet 1/7                7                  7

Ethernet 1/8                6                  6

Ethernet 1/9                9                  49

Ethernet 1/10                8                  48

Ethernet 1/11                11                  51

Ethernet 1/12                10                  50

Ethernet 1/13                12                  59

Ethernet 1/14                13                  58

Ethernet 1/15                14                  57

Ethernet 1/16                15                  56

Ethernet 2/1                N/A                  70

Ethernet 2/2                N/A                  71

Ethernet 2/3                N/A                  69

Ethernet 2/4                N/A                  68

Ethernet 2/5                N/A                  66

Ethernet 2/6                N/A                  67

Ethernet 2/7                N/A                  65

Ethernet 2/8                N/A                  64

Internal 1/1                 26                  81 (Eventing Port - NOT visible at Service Manager)

Internal 1/2                 27                  80 (Unused - NOT visible at Service Manager)

Internal 1/3           24                      52 (Internal backplane uplink to logical device, whether ASA or FTD)

Options to Mitigate the Trigger and Recover the Appliance

Correction of any duplex mismatch is the only way to prevent the side effect seen on the backplane interface, and this can be done by one of these methods and a reload of the appliance.

1. If peer device is not configured with Duplex Auto, change it to Auto (preferred method).

2. If there is no management access to the peer device:

2.1. For FTD managed by Firepower Management Center (FMC), disable option Auto-Negotiation under Edit Physical Interface on FMC.

2.2. For FTD managed by Firepower Device Manager (FDM), change Duplex option from Auto to Full under Interface Advanced Options.

2.3. For ASA, disable Duplex Auto negotiation from chassis level as follows:

firepower /eth-uplink # scope

firepower /eth-uplink # scope fabric a

firepower /eth-uplink/fabric # scope interface 1 1

firepower /eth-uplink/fabric/interface # set auto-negotiation

no No

yes Yes

firepower /eth-uplink/fabric/interface # set auto-negotiation no

firepower /eth-uplink/fabric/interface* # commit-buffer

firepower /eth-uplink/fabric/interface #

Note: Methods listed in step 2 are theoretical options that can work under normal conditions.

3. Connect the Firepower interface in half-duplex mode to a different switch that supports auto-negotiation of duplex settings or configure the switch port to enable auto-negotiation of duplex settings.

Note: A reload of the whole appliance is still necessary after the execution of any of the steps, in order to recover the backplane interface from its failed state.

Cisco bug ID Information

This bug was raised to track a software resolution of the symptom in which the backplane Internal1/3 interface is unable to process any traffic received from the LINA after some time.

Cisco bug ID CSCwa79915 Physical port in Half Duplex causes all packets from LINA to be dropped by the chassis.