What do you do when a Nexus 5000 switch displays the "FWM-2-STM_LOOP_DETECT" message in the log?
%FWM-2-STM_LOOP_DETECT: Loops detected in the network among ports
Eth x/y and Eth x/y vlan xx -
Disabling dynamic learn notifications for 180 seconds
%FWM-2-STM_LEARNING_RE_ENABLE: Re enabling dynamic learning on all interfaces
This message indicates that the switch receives frames with the same source MAC address on these two interfaces and that the swtich learns the same MAC address on these interfaces at a very high speed. The switch detects this as a loop. The switch disables MAC address learning in order to protect its control plane. This is implemented on all VLANs even if the loop occurred on only one VLAN.
Possible Causes
- MAC addresses move because of incorrect Spanning Tree Protocol (STP)-port state convergence.
- MAC addresses move because the source of the data is physically moved across all switches while STP states are converged and in correct states.
- MAC addresses can move between interfaces if the server Network Interface Cards (NICs) are configured for teaming/bonding, but the connected switch interfaces are not. This can be avoided if you use Link Aggregation Control Protocol (LACP) in order to bond the interfaces on both ends, or configure the server interfaces to use the NICs in an "active/standby" mode.
How is the loop actually detected?
Forwarding Manager (FWM) has a mechanism to count the number of MAC-move-backs and weigh them based on the number of times the MAC address moves. It determines the total MAC-move-backs count (switch-wide across all VLANs, MACs, and interfaces), declares the %FWM-2-STM_LOOP_DETECT, and disables learning to protect FWM in loopy conditions.
Threshold Math: 28,000 MAC move-backs count in a given aging scan period of 10 seconds switch-wide. It is declared as %FWM-2-STM_LOOP_DETECT and learning is disabled.
Example Messages
2011 Jan 30 16:14:23 Nexus-5000 %FWM-2-STM_LOOP_DETECT:
Loops detected in the network among ports Eth119/1/13 and Po90 vlan 218 -
Disabling dynamic learn notifications for 180 seconds
2011 Jan 30 16:17:23 Nexus-5000 %FWM-2-STM_LEARNING_RE_ENABLE:
Re enabling dynamic learning on all interfaces
The logic for MAC-move notifications should be noted. It is possible to notify MAC-moves when the MAC-address-table notification for MAC-moves is enabled. This adds notification logs on the console but no action is taken. A move is declared when a given MAC address has moved three times back and forth across a given pair of ports on a VLAN within an aging scan period of 10 seconds.
Troubleshoot
You can enable MAC-move notification on the switch to find out which MAC addresses move.
Nexus-5000# conf t
Nexus-5000(config)# mac address-table notification mac-move
With Nexus 5000 switches, it is not always sufficient to enable the MAC-move notification in order to generate a syslog message about MAC-move notification.
In order to ensure syslog message generation, enter these commands in conjunction with the previous command.
Nexus-5000# conf t
Nexus-5000(config)# Logging level spanning-tree 6
Nexus-5000(config)# Logging level fwm 6
Nexus-5000(config)# Logging monitor 6
The addition of these commands ensures that the syslog for FWM detect displays when there is a MAC address move.
In order to verify the STP port state across VLANs on the switches, enter these commands.
Nexus-5000# show spanning-tree
Nexus-5000# show spanning-tree vlan <id>
Nexus-5000# show spanning-tree internal interaction
Example
In order to check if MAC addresses move, enter this command:
Nexus-5000# show mac address-table notification mac-move
MAC Move Notify Triggers: 1206
Number of MAC Addresses added: 944088
Number of MAC Addresses moved: 265
Number of MAC Addresses removed: 943920
MAC address moves are also logged with a minimum logging level of six required to display which MAC addresses move.
2012 Jun 12 16:05:31.564 Nexus-5000 %FWM-6-MAC_MOVE_NOTIFICATION:
Host 0000.0000.fe00 in vlan 85 is flapping between
port Eth104/1/8 and port Eth104/1/9
Solution
- Check for a correct STP convergence and for STP port-states across all switches in the topography. Also confirm that there are no disputes or incorrect port states.
- If the source of the data frames that are physically moving is identified, control the source in order to halt rapid and continuous moves.
- By default, dynamic learning is reenabled after 180 seconds. At that point, any STP disputes or inconsistencies should be resolved. If not, the dynamic learning is disabled again.
Related Enhancement on the Nexus 5000 Switch
Cisco bug ID CSCug28099 - Enh: Knob to Disbable ports after loop is detected on the Nexus 5000.
The current behavior on earlier code (pre - 6.0(2)N2(1)) is described here.
When loop messages (FWM-2-STM_LOOP_DETECT: Loops detected in the network among ports <port_id> and <po_id> vlan >vlan_id> - Disabling dynamic learn notifications for 180 seconds) are detected, after 120 seconds of loop detection you should rapid age out all the MAC addresses and then relearn them rather than aging the whole MAC address table. Due to this behavior you will not learn the new MAC addresses for 120 seconds, but if the loop is consistently present it can cause significant impact to the network as you would rapid age the MAC addresses from all VLANs.
This enhancement is filed in order to have a CLI knob where after a loop is detected, the switch shut downs the port in question (the port where the loop is detected) in order to avoid complete outage.
Here are the commands that are implemented in the code in Versions 6.0(2)N2(1) and later:
swo2-371(config)# mac address-table loop-detect ?
port-down Take port-down action for mac loop detection
swo2-371(config)# mac address-table loop-detect port-down
swo2-371(config)# no mac address-table loop-detect port-down
Feedback