The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.
This document describes why a Fibre Channel (FC) link between a Cisco Multilayer Director Switch (MDS) and another Cisco switch doesn't come Up when trunking or in a port-channel, however, the same link might come Up when it is not in a port-channel and trunking is disabled.
This includes FC links between MDS switches and other MDS switches, Nexus switches, and Unified Computing System (UCS) Fabric Interconnects (FIs).
Tip: For more detailed information about port-channels and trunking, refer to the appropriate port-channel and trunking configuration guides.
Port-channel Guide: https://www.cisco.com/c/en/us/td/docs/switches/datacenter/mds9000/sw/8_x/config/interfaces/cisco_mds9000_interfaces_config_guide_8x/configuring_portchannels.html
Trunking Guide: https://www.cisco.com/c/en/us/td/docs/switches/datacenter/mds9000/sw/8_x/config/interfaces/cisco_mds9000_interfaces_config_guide_8x/configuring_trunking.html
All MDS switches can aggregate multiple physical links into a single virtual link via port-channels, as well as transport multiple Virtual Storage Area Networks (VSANs) over a link with the trunking feature when connected with another Cisco device.
In order to negotiate port-channels and trunking links, MDS switches use Exchange Peer Parameters (EPP) services to communicate across peer ports in an Inter-Switch Link (ISL). Part of the ISL negotiation process verifies that the peer is a Cisco device with the peer's Organizational Unique Identifier (OUI).
If a Cisco switch's OUI is not listed in the peer's OUI database as a Cisco OUI, the link fails to come up when added to a port-channel or if trunking is enabled on the interface.
If the link is in a port-channel, check the port-channel interface haschannel mode active configured.
MDSswitch# show run interface port-channel 1
...
interface port-channel1
channel mode active <<<<
If the link is connected to a N_Port Virtualization (NPV) switch, like a FI, check the NPIV and fport-channel-trunk features are enabled.
MDSswitch# show feature | include npiv
npiv 1 enabled
MDSswitch# show feature | include fport
fport-channel-trunk 1 enabled
If the link is trunking, check the trunking protocol is enabled globally.
MDSswitch# show trunk protocol
Trunk Protocol is enabled
Check trunking is enabled on the interface.
MDSswitch# show interface port-channel 1
port-channel1 is down (No operational members)
Hardware is Fibre Channel
Port WWN is xx:xx:00:2a:6a:xx:xx:xx
Admin port mode is auto, trunk mode is on <<<<
If the link is connected at 8Gbps, ensure the fill pattern, or fill word, must be configured the same for both sides of the link. By default, the MDS, N5K/N6K, and UCS FI 6300 series devices use ARBFF. N9K and UCS FI 6400 series only support IDLE as the 8G fill pattern.
Use this command to confirm the current fill pattern on MDS/N5K/N6K interfaces fc x/y:
slot x show hardware internal fcmac port y port-info | i FILL
MDSswitch# slot 1 show hardware internal fcmac port 2 port-info | i FILL
FILL_WORD(raw) : ARBFF (0xbc94ffff)
Note: The output on Nexus switches can be 0x0 (interface not up at 8Gbps), 0xBC94FFFF (ARBFF), or 0xBC95B5B5 (IDLE).
When an N9K or UCS FI 6400 is connected to a device that uses ARBFF by default, you must configure the interface on the connected device to use IDLE.
MDSswitch# conf t
Enter configuration commands, one per line. End with CNTL/Z.
MDSswitch(config)# interface fc1/2MDSswitch(config-if)# switchport fill-pattern IDLE speed 8000
On UCS FI 6300 series, you see FC uplinks' fill pattern and configured in the UCS Manager (UCSM) at SAN > SAN Cloud > Fabric > Uplink FC Interfaces > FC Interface x/yin the General tab.
Note: It is possible for the switch OUI to be absent from the peer's OUI database on one or both sides of a link. Both sides must be investigated.
For MDS and Nexus switches:
The switch without the peer's OUI in its OUI database records "physical flogi rejected, waiting for the port mode" and "EPP_SYNC" errors.
MDSswitch# show flogi internal event-history errors | include reject previous 1 next 1
...
10) Event:E_DEBUG, length:124, at 727676 usecs after Tue May 14 17:44:47 2013
[102] fs_fc2_msg_flogi: ifindex[0x12ae000] pwwn[xx:xx:00:2a:6a:xx:xx:xx] physical flogi rejected, waiting for the port mode
11) Event:E_DEBUG, length:124, at 661821 usecs after Tue May 14 17:44:39 2013
[102] fs_fc2_msg_flogi: ifindex[0x12ae000] pwwn[xx:xx:00:2a:6a:xx:xx:xx] physical flogi rejected, waiting for the port mode
12) Event:E_DEBUG, length:73, at 772303 usecs after Tue May 14 17:43:11 2013
[102] fs_flogi_send_flogi_reject: mts_q == 0, ifindex 0x12ae000, port 0x0
MDSswitch# show port internal event-history errors | include EPP previous 1 next 1
...
19) Event:E_DEBUG, length:48, at 465145 usecs after Tue May 14 17:44:49 2013
[102] epp_ac_accept_sync: Error sending EPP_SYNC
...
27) Event:E_DEBUG, length:48, at 280537 usecs after Tue May 14 17:44:43 2013
[102] epp_ac_accept_sync: Error sending EPP_SYNC
To confirm the interface of the errors, translate the index withshow interface snmp-ifindexand filter by the ifindex in the error message. The example uses the ifindex 12ae000 found in theshow flogi internal event-history errors.
MDSswitch# show interface snmp-ifindex | include 12ae000
fc6/47 0019587072 (00012ae000)
For UCS FIs: The FI without the peer's OUI in its OUI database records "PI_FSM_EV_PORT_CONFIG_FAILURE" and "port fcx/y attempting vf tagging on non-cisco switch" errors.
UCS(nxos)# show port internal event-history errors
1) Event:E_DEBUG, length:117, at 225850 usecs after Wed Aug 1 10:18:26 2018
[102] pi_fsm_port_attr_change_init: Ifindex (fc1/47)0x102e000, Err disabled event (PI_FSM_EV_PORT_CONFIG_FAILURE)0xd7
2) Event:E_DEBUG, length:100, at 222587 usecs after Wed Aug 1 10:18:26 2018
[102] pm_process_fport_vf_tagging_capability: port fc1/47 attempting vf tagging on non-cisco switch
Extract the OUI from each switch. The OUI is the third, fourth, and fifth octet of the World Wide Name (WWN).
peer-switch# show wwn switch
Switch WWN is xx:xx:00:2a:6a:xx:xx:xx
In this example, the switch OUI is0x002A6A.
Check the switch's OUI database for the peer switch's OUI. If the command has no output, the OUI is absent from the database.
MDSswitch# show wwn oui | include 0x002a6a
MDSswitch#
Note: UCS FIs don't support theshow wwn ouicommand. Refer to Table 1. and Table 2. in order to see when OUIs were added to the FI OUI database.
Switch OUI databases are updated through switch software upgrades. Until a switch can be upgraded to a software version with an OUI database that includes the peer switch's OUI, use single links configured outside of port-channels with trunking disabled.
Thewwn oui <hex oui> command on MDS and Nexus switches can manually add Cisco OUIs to the switch's OUI database.
On MDS switches, this feature is available in NX-OS Version 7.3(0)D1(1)and later.
On Nexus 5K/6K switches, this feature was added in NX-OS Version 7.3(3)N1(1)and later.
On Nexus 9000 switches on NX-OS 7.x, this feature is available in NX-IS Version 7.3(1)N1(1) and later.
On Nexus 9000 switches on other versions of NX-OS, this feature is available in NX-OS Version9.3(3)and later.
Example: MDSswitch# configure terminal
MDSswitch(config)# wwn oui 0x0000fc
MDSswitch# show wwn oui
OUI Vendor Default/Static
----------------------------------------------------
0x0000fc Cisco Static
Nexus 9000 switches on ACI images can add an OUI to the library in run time from the APIC in15.0(1k) and later.
OUI |
Software that includes OUI |
0x547FEE |
All versions of NX-OS 5.0(1) and later. |
0x00351A 0x003A7D |
NX-OS 5.x at 5.2(8h) and later. NX-OS 6.x at 6.2(17) and later. NX-OS 7.x at 7.3(1)D1(1) and later. All versions of NX-OS 8.3(1) and later. |
0x0018BA 0x001B54 |
All versions of NX-OS 5.2(6) and later. |
0x508789 0xFC5B39 |
All versions of NX-OS 6.2(11) and later. |
0x000831 0x74A02F |
NX-OS 5.x at 5.2(8g) and later. NX-OS 6.x at 6.2(11c) and later. All versions of NX-OS 7.3(1)D1(1) and later. |
0x70EA1A 0xC4F7D5 |
All versions of NX-OS 8.4(2) and later. |
0x00FD22 0x10B3D5 |
NX-OS 6.x at 6.2(33) and later. All versions of NX-OS 8.4(2) and later. |
0xDC774C |
NX-OS 8.4(x) at 8.4(2c) and later. NX-OS 9.x at 9.2(1) and later. |
0x3C13CC 0x4CE176 |
To be determined |
OUI |
Software that includes OUI |
0x002A6A 0x00DEFB 0x8C604F |
NX-OS 5.1 at 5.1(3)N2(1) and later. NX-OS 5.2 at 5.2(1)N1(4) and later. All versions of NX-OS 6.0(2)N1(2) and later. |
0x50EB1A 0x9371D5 |
All versions of NX-OS 7.3(7)N1(1) and later. |
0x3C13CC 0x4CE176 0xDC774C |
All versions of NX-OS 7.3(10)N1(1) and later. |
OUI |
Software that includes OUI |
||||
0x000831 0x001086 |
0x18E728 0x1C6A7A |
0x70DB98 0x70DF2F |
0xBADBAD 0xBC26C7 |
All versions of NX-OS9.2(4) and later. |
|
0x3C13CC 0x4CE176 0xDC774C |
All versions of NX-OS10.1(2) and later. |
OUI |
Software that includes OUI |
0x0018BA 0x001B54 |
Bundle software 2.x at 2.2(8j) and later. Bundle software 3.x at 3.1(3b) and 3.1(3h) and later. Bundle software 3.2(3d) and later. |
0x002651 |
Bundle software 2.x at 2.2(8j) and later. Bundle software 3.x at 3.1(3h) and later. Bundle software 3.2(3d) and later. |
0x0025B5 0xA44C11 |
Bundle software 3.x at 3.1(3h) and later. Bundle software 3.2(3d) and later. |
0x000831 0x003A9C 0xD0A5A6 |
Bundle software 3.x at 3.2.(3h) and later. All versions of bundle software at 4.0(1x). Bundle software at 4.0(2b) and later. |
0x6400F1 |
Bundle software 3.3(5b) and later. |
0x00EEAB 0x0476B0 |
Bundle software 4.1(3a) and later. |
0xAC7A56 |
Bundle software 4.1(2a) and later. |
0x3C13CC 0x4CE176 0xDC774C |
Bundle software 4.2(1d) and later. |
Cisco bug ID CSCut41864 - Port process crashes when interface > 64 used for NPV-NPIV connections
Cisco bug ID CSCuz52695 - UCS FI FC Uplinks Error Disable Immediately Connected NPV to MDS9396S
Cisco bug ID CSCvp95480 - 6454 FI - Restrict allowed WWxN addresses during UCSM Configuration (avoid multicast address)
Multiple bugs have been filed to document OUIs that were absent from switches' OUI libraries. They include:
MDS/UCS Bugs
Cisco bug ID CSCtr01652- Add N7K OUIs to liboui's list of recognized Cisco OUIs. MDS.
MDS/N5K/N6K Bugs
Cisco bug ID CSCty04686- Add MAC OUI "002a6a", "8c604f", "00defb" for 5k/UCS-FI
MDS Bugs
Cisco bug ID CSCta97375 - GLDN:SABRE: Sabre oui is not recognized as cisco oui 547FEE.
Cisco bug ID CSCur44218 - Add MiniUCS FI OUIs to liboui's list of recognized Cisco OUIs
Cisco bug ID CSCuz11494 - FC port channel does not come up with peer UCS OUI 00:c8:8b
Cisco bug ID CSCuu03717 - Add OUIs 00:3a:9c, 00:08:31, d0:a5:a6 to default OUI list
Cisco bug ID CSCuu26045 - Add MiniUCS FI OUI 0x74a02f to MDS list of recognized Cisco OUIs
Cisco bug ID CSCvx67383 - MDS Port-Channel towards ACI LEAF N9K with switch OUI 0x3C13CC (3C:13:CC) does not come up or trunk
Cisco bug ID CSCvx49403 - Port-Channel with Cisco switch OUI 0xDC774C does not come up or trunk.
Cisco bug ID CSCvr60193 - Add OUI C4:F7:D5 to the default OUI list
Cisco bug ID CSCvp19788- MDS 9000 Add OUI 70:EA:1A to OUI library
Cisco bug ID CSCvt28351 - Add six OUI's to the default OUI list
Cisco bug ID CSCvw14504 - Add OUI 0x4CE176 to OUI Database - Port-channels and trunking not working with OUI 0x4CE176
Cisco bug ID CSCvx67383 - MDS Port-Channel towards ACI LEAF N9K with switch OUI 0x3C13CC (3C:13:CC) does not come up or trunk
N5K/N6K Bugs
Cisco bug ID CSCvx72821 - Port-Channel with Cisco switch OUI 0xDC774C or 0x4CE176 or 0x3C13CC does not come up or trunk.
Cisco bug ID CSCvp54881 - Unable to create San Port Channel between UCS & Nexus 56128p
N9K Bugs
Cisco bug ID CSCvp21786 - Nexus 9000:::Unable to bring up F mode san-port-channel or trunking between MDS and N9K
Cisco bug ID CSCvx68507 - Port-Channel with Cisco switch OUI 0xDC774C or 0x4CE176 or 0x3C13CC does not come up or trunk.
UCS Bugs
Cisco bug ID CSCvk63036 - UCS Missing OUI 003a9c, 000831, d0a5a6 -- Prevents FC/FCoE Port-Channel From Forming
Cisco bug ID CSCvv70634 - UCS FI needs to add additional N9K OUI's
Cisco bug ID CSCvx68517 - Port-Channel with Cisco switch OUI 0xDC774C or 0x4CE176 or 0x3C13CC does not come up or trunk. Cisco bug ID
Revision | Publish Date | Comments |
---|---|---|
1.0 |
23-Feb-2022 |
Initial Release |