This document provides information on how to troubleshoot Address
Resolution Protocol (ARP) or Content Addressable Memory (CAM) table-related
issues on Catalyst 6500/6000 Switches.
There are no specific requirements for this document.
This document is not restricted to specific software and hardware
Technical Tips Conventions for more information on document
Catalyst switches maintain several types of tables that are tailored
for Layer 2 switching or multilayer switching (MLS), and are kept in very fast
memory so that many fields within a frame or packet can be compared in
ARP—Maps an IP address to a MAC address in order to
provide IP communication within a Layer 2 broadcast domain. For example, Host B
wants to send information to Host A but does not have the MAC address of Host A
in its ARP cache. Host B generates a broadcast message for all hosts within the
broadcast domain to obtain the MAC address associated with the IP address of
Host A. All hosts within the broadcast domain receive the ARP request, and only
Host A responds with its MAC address.
CAM—All Catalyst switch models use a CAM table for
Layer 2 switching. As frames arrive on switch ports, the source MAC addresses
are learned and recorded in the CAM table. The port of arrival and the VLAN are
both recorded in the table, along with a timestamp. If a MAC address learned on
one switch port has moved to a different port, the MAC address and timestamp
are recorded for the most recent arrival port. Then, the previous entry is
deleted. If a MAC address is found already present in the table for the correct
arrival port, only its timestamp is updated.
Ternary Content Addressable Memory (TCAM)—In
multilayer switches, all the processes that access control lists (ACLs) provide
in traditional routing, such as matching, filtering, or control specific
traffic, are implemented in hardware. TCAM allows a packet to be evaluated
against an entire access list in a single table lookup. Most switches have
multiple TCAMs so that both inbound and outbound security, as well as QoS ACLs,
can be evaluated simultaneously, or entirely in parallel with a Layer 2 or
Layer 3 forwarding decision.
In distributed switching, each Distributed Feature Card (DFC) is
responsible for maintaining each own CAM table. This means that each DFC learns
the MAC address and ages them, which depends on the CAM aging and traffic
matching that particular entry. With distributed switching, it is normal that
the supervisor engine does not see any traffic for a particular MAC address for
a while, so the entry might expire. There are currently two mechanisms
available to keep the CAM tables consistent between the different engines, such
as DFC (present in line modules) and Policy Feature Card (PFC) (present in
Flood to Fabric (FF)
MAC Notification (MN)
When a MAC address entry is aged out on the PFC, the show
mac-address address <MAC_Address>
all command displays the DFC or PFC that holds this MAC
In order to prevent the age out of an entry on a DFC or PFC, even if
there is not traffic for that MAC address, enable the MAC address
synchronization. Issue these commands in order to enable the
!--- This is a global configuration command and is used to enable the synchronization.
!--- This is a privileged EXEC command and is used to clear dynamic MAC addresses.
Cat6K-IOS#clear mac-address-table dynamic
The mac-address-table synchronize command
is available from Cisco IOS® Software Releases
12.2(18)SXE4 and later. After you enable it, it is possible to still see
entries that are not present in PFC or DFC. However, the module has a way to
learn it from others that use Ethernet Out of Band Channel (EOBC).
Caution: The mac-address-table synchronize command
purges the routed MAC entires. In order to avoid this, disable the routed MAC
purging with the mac-address-table aging-time
0 routed-mac global configuration
Cisco Express Forwarding (CEF) is a Layer 3 IP switching technology
that provides superior performance compared to other switching technologies,
especially in networks with dynamic traffic patterns. CEF maintains data
structures called Forwarding Information Base (FIB) and adjacency tables. The
FIB table mirrors the information in the routing table and is used to make
forwarding decisions. The adjacency table contains the precomputed link-layer
header for next hop devices. Based on the next hop interface, entries in the
FIB table are mapped to entries in the adjacency table. A device is not able to
perform CEF switch packets if the adjacency table is not populated with the
If CEF drops packets at regular intervals, interspaced by periods of
normal operation, it is probably due to the adjacency table being cleared
periodically. This is caused by the aging of the ARP entry. Packets are not CEF
switched for the duration in which the adjacency table is repopulated with the
required next hop information. While ARP entries are refreshed by default every
four hours, configuring a very small value of ARP timeout is disruptive to CEF
command in the interface configuration mode in
order to change the time in which an entry remains in the ARP cache.
Refer to Cisco bug ID
(registered customers only)
for more information on this
vulnerability. Refer to
Incomplete Adjacencies with CEF for more information on CEF
The switch filter frames with a source MAC address of
00-00-00-00-00-00, which is an invalid source MAC, from the CAM table. This is
an example of the syslog error output when this occurs:
%SYS-4-P2_WARN: 1/Filtering MAC address 00-00-00-00-00-00 on port 2/48 from host table
These messages are informational and tell you that a frame that has a
source MAC address of 00-00-00-00-00-00 is found, and the switch will never add
that to the CAM table. However, the switch will forward traffic sourced from an
all-zero MAC address.
The workaround is to identify the end station that generates frames
with an all-zero source MAC address. Typically, one of these devices transmits
A traffic generator, such as Spirent SmartBits
Certain types of servers, such as load-balancing IBM WebSphere
A misconfigured router or end station, such as a device that
transmits all-zeros broadcasts
A faulty NIC
LAN switches use forwarding tables, such as Layer 2 and CAM tables, to
direct traffic to specific ports based on the VLAN number and the destination
MAC address of the frame. When there is no entry that corresponds to the
destination MAC address of the frame in the incoming VLAN, the (unicast) frame
is sent to all forwarding ports within the respective VLAN. This causes
flooding. The very cause of flooding is that the destination MAC address of the
packet is not in the Layer 2 forwarding table of the switch. In this case, the
packet is flooded out of all forwarding ports in its VLAN, except the port it
is received on.
The default ARP table aging time is 4 hours while the CAM holds the
entries for only 5 minutes. The switch sends out a frame to all forwarding
ports within the respective VLAN when the destination MAC address is aged out
from the CAM table. You need a CAM aging timer greater or equal to the ARP
timeout in order to prevent unicast flooding. As a workaround, you can issue
one of these commands in order to increase the CAM aging timer for the VLAN you
are having trouble with to match the ARP aging time:
Note: In any Catalyst environment that runs a Hot Standby Router Protocol
(HSRP), it is recommended that you ensure the CAM and ARP timers are
Flooding in Switched Campus Networks for information on possible causes
and implications of unicast packet flooding in switched networks.
In Hybrid mode, the supervisor engine runs CatOS and the Multilayer
Switch Feature Card (MSFC) runs Cisco IOS. CatOS operates at Layer 2 and builds
the CAM address table to hold the VLAN, MAC address and port number
information. Cisco IOS in MSFC operates in Layer 3 and builds the ARP table to
hold the IP address to MAC address resolution.When you change the IP address of
any device, such as a printer or a server, you might not be able to ping that
new IP address. However, you are able to ping the new IP address from the same
VLAN. This can be an ARP issue on the MSFC.
This workaround can help to isolate and resolve the issue:
Clear the ARP table on the MSFC.
MSFC2#clear arp int vlan 40
Verify the ARP timeout value. The default value is 4 hours. If the
ARP timeout in the VLAN is high, you can set the timeout value back to the
default or optimal value.
MSFC2#show int vlan 40
Vlan40 is up, line protocol is up
Hardware is Cat6k RP Virtual Ethernet, address is 00d0.0050.33fc (bia 00d0.005
Internet address is 18.104.22.168/24
MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive not supported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:01:44, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Enter configuration commands, one per line. End with CNTL/Z.
MSFC2(config)#int vlan 40
MSFC2(config-if)#arp timeout ?
MSFC2(config-if)#arp timeout 240
Reload the MSFC.
Proceed with reload? [confirm]
This is an example of the syslog error output when you have this
%EARL-2-EARL4LOOKUPRAMERROR:Address eac6, data 0-0-8000-0, count 8
This appears when you perform a CAM table lookup. This occurs due to a
parity error when you access memory. This error is usually generated when you
command in order to access the CAM table. In some
cases, the switch also resets when the show cam
command is issued.
%EARL-2-EARLLOOKUPRAMERROR: Address [hex], data [hex]-[hex]-[hex]-[hex], count [dec]
This error message indicates that a lookup RAM parity error has been
detected. The address [hex] field is the address in the forwarding table where
the error was detected. The data [hex]-[hex]-[hex]-[hex] field is the word0,
word1, word2, and word3 of RAM data that generated the parity error. The count
[dec] field is the total number of parity errors.
This message is not catastrophic and might not result in outage
situations if you only have isolated occurrences of it. If you receive this
message continuously, it indicates that the switch is trying to write to a bad
DRAM sector when it adds a new entry to the CAM table. Then, you need to
replace the DRAM or the supervisor itself.
Static CAM entries that are configured on the active supervisor engine
are lost after fast switchover. As a workaround to this issue, you must
reconfigure CAM entries after fast switchover.
Refer to Cisco bug IDs
(registered customers only)
(registered customers only)
for more information on this
If the TCAM is full and you attempt to add new ACLs, or access control
entries (ACEs) to ACLs that exist, the commit or map process fails. Any prior
configuration remains in effect. In the case of Router Access Control Lists
(RACLs), the ACL is enforced in software on the Multilayer Switch Feature Card
(MSFC) with the corresponding performance penalty.
On a switch that runs hybrid software, if you configure a Virtual Local
Area Network Access Control List (VACL) or QoS ACL ACEs that exceed the pattern
or mask capacity of the TCAM, a syslog message similar to this prints to the
%ACL-5-TCAMFULL: acl engine TCAM table is full
On Supervisor IOS systems, or on the MSFC in a hybrid system, if you
configure RACL ACEs that exceed the capacity of the TCAM, a syslog message
similar to this prints to the console:
%FM-4-TCAM_ENTRY: Hardware TCAM entry capacity exceeded
On Supervisor IOS systems, or on the MSFC in a hybrid system, issue the
show fm summary command in order to see which
interfaces enforce ACLs in hardware (ACTIVE) and which interfaces enforce ACLs
in software (INACTIVE).
The workaround for this issue is to remove the unused ACL or QoS from
the swich configuration. Refer to
ACL on Catalyst 6500 Series Switches for more information.
When you ping a VLAN interface, an ARP request with a source IP of that
VLAN is sent to the Default Router (MSFC), but the router does not respond to
the ARP request and the debug ARP shows this error message:
IP ARP req filtered src [ip-address] [mac-address] dst [ip-address]
[mac-address] wrong cable, interface-id
For each ARP datagram, an ARP reply is discarded if the destination IP
address does not match the local host address. An ARP request is discarded if
the source IP address is not in the same subnet. It is desirable that this test
be overridden by a configuration parameter in order to support the infrequent
cases where more than one subnet can co-exist on the same cable.
An ARP reply is generated only if the destination protocol IP address
is reachable from the local host, as determined by the routing algorithm, and
the next hop is not through the same interface. If the local host functions as
a gateway, this can result in ARP replies for destinations not in the same
subnet. This shows that to drop the ARP request is justifiable.
This can be resolved by making the Catalyst 6500 not respond to all ARP
requests because the source IP address in the ARP request is on a different
subnet than the target IP address in the ARP. Therefore, the MSFC/Router
concludes that the ARP did not remain in the same Layer 2 domain and shows the
wrong cable type. In other words, the wrong cable debug message is generated
when the ARP source and destination do not belong to the same Layer 2 domain.
In order to make ARP work in this scenario, the destination protocol IP must be
reachable with the use of the static route as a workaround.
Two entries show for MAC address in the MAC address table.
Cat6K#show mac-address-table int gi 6/11
Displaying entries from Line card 6:
Legend: * - primary entry
age - seconds since last seen
n/a - not available
vlan mac address type learn age ports
* 100 0011.857c.4d10 dynamic Yes 0 Gi6/11
* 100 0011.857c.4d10 dynamic Yes 95 Gi6/11
Cat6K#show module 6
Mod Ports Card Type Model Serial No.
--- ----- -------------------------------------- ------------------ -----------
6 48 CEF720 48 port 10/100/1000mb Ethernet WS-X6748-GE-TX SADxxxxxxxx
Mod MAC addresses Hw Fw Sw Status
--- ---------------------------------- ------ ------------ ------------ -------
6 001d.45fd.xx4a to 001d.45fd.xx79 2.6 12.2(14r)S5 12.2(18)SXF8 Ok
Mod Sub-Module Model Serial Hw Status
---- --------------------------- ------------------ ----------- ------- -------
6 Distributed Forwarding Card WS-F6700-DFC3B SALxxxxxxxx 4.6 Ok
Mod Online Diag Status
Two Layer 2 forwarding lookup engines exist in the DFC environment. It
is common in the dCEF environment that the FE1 and FE2 learn the same MAC
address on the same port on a CEF720/dCEF720 architecture line cards.
Cisco routers require an ARP (Address Resolution Protocol) entry for
every virtual IP address. While network load balancing uses Level 2 multicast
for the delivery of packets. In Cisco's implementation of the RFC, multicast is
used only for IP multicast. Therefore, when the router does not see a multicast
IP address, it does not automatically create an ARP entry, and you must
manually add it to the router.
Normally, Cisco devices do not put a multicast MAC address (clusters
virtual MAC address) in the ARP table if it was resolved through a unicast IP
address (cluster's virtual address). In order to resolve this issue, you need a
static mapping of the unicast virtual IP address to the multicast MAC
For more information, refer to the
Mode section of the Catalyst Switches for Microsoft Network
Load Balancing Configuration Example.