Contents

Introduction

Background Information

Troubleshoot MDS Trunking/Port-Channel Links that Fail to Come Up

Verify

Symptoms

Log Messages

OUI Database

Work Around

OUI Database Additions

Table 1.1. MDS OUI Database Additions

Table 1.2. Nexus 5000/5500/5600/6000 OUI Database Additions

Table 1.3. Nexus 9000 OUI Database Additions

Table 1.4. UCS FI OUI Database Additions

Caveats

Introduction

This document describes why a Fibre Channel (FC) link between a Cisco Multilayer Director Switch (MDS) and another Cisco switch doesn't come Up when trunking or in a port-channel, however, the same link might come Up when it is not in a port-channel and trunking is disabled.

Background Information

This includes FC links between MDS switches and other MDS switches, Nexus switches, and Unified Computing System (UCS) Fabric Interconnects (FIs).

Tip: For more detailed information about port-channels and trunking, refer to the appropriate port-channel and trunking configuration guides.

Port-channel Guide: https://www.cisco.com/c/en/us/td/docs/switches/datacenter/mds9000/sw/8_x/config/interfaces/cisco_mds9000_interfaces_config_guide_8x/configuring_portchannels.html

Trunking Guide: https://www.cisco.com/c/en/us/td/docs/switches/datacenter/mds9000/sw/8_x/config/interfaces/cisco_mds9000_interfaces_config_guide_8x/configuring_trunking.html

All MDS switches can aggregate multiple physical links into a single virtual link via port-channels, as well as transport multiple Virtual Storage Area Networks (VSANs) over a link with the trunking feature when connected with another Cisco device.

In order to negotiate port-channels and trunking links, MDS switches use Exchange Peer Parameters (EPP) services to communicate across peer ports in an Inter-Switch Link (ISL). Part of the ISL negotiation process verifies that the peer is a Cisco device with the peer's Organizational Unique Identifier (OUI).

If a Cisco switch's OUI is not listed in the peer's OUI database as a Cisco OUI, the link fails to come up when added to a port-channel or if trunking is enabled on the interface.

Troubleshoot MDS Trunking/Port-Channel Links that Fail to Come Up

If the link is in a port-channel, check the port-channel interface haschannel mode active configured.

MDSswitch# show run interface port-channel 1

...

interface port-channel1

channel mode active      <<<<

If the link is connected to a N_Port Virtualization (NPV) switch, like a FI, check the NPIV and fport-channel-trunk features are enabled.

MDSswitch# show feature | include npiv

npiv 1 enabled

MDSswitch# show feature | include fport

fport-channel-trunk 1 enabled

If the link is trunking, check the trunking protocol is enabled globally.

MDSswitch# show trunk protocol

Trunk Protocol is enabled

Check trunking is enabled on the interface.

MDSswitch# show interface port-channel 1

port-channel1 is down (No operational members)

Hardware is Fibre Channel

Port WWN is xx:xx:00:2a:6a:xx:xx:xx

Admin port mode is auto, trunk mode is on <<<<

If the link is connected at 8Gbps, ensure the fill pattern, or fill word, must be configured the same for both sides of the link. By default, the MDS, N5K/N6K, and UCS FI 6300 series devices use ARBFF. N9K and UCS FI 6400 series only support IDLE as the 8G fill pattern.

Use this command to confirm the current fill pattern on MDS/N5K/N6K interfaces fc x/y:

slot x show hardware internal fcmac port y port-info | i FILL

MDSswitch# slot 1 show hardware internal fcmac port 2 port-info | i FILL

FILL_WORD(raw) : ARBFF (0xbc94ffff)

Note: The output on Nexus switches can be 0x0 (interface not up at 8Gbps), 0xBC94FFFF (ARBFF), or 0xBC95B5B5 (IDLE).

When an N9K or UCS FI 6400 is connected to a device that uses ARBFF by default, you must configure the interface on the connected device to use IDLE.

MDSswitch# conf t

Enter configuration commands, one per line. End with CNTL/Z.

MDSswitch(config)# interface fc1/2MDSswitch(config-if)# switchport fill-pattern IDLE speed 8000

On UCS FI 6300 series, you see FC uplinks' fill pattern and configured in the UCS Manager (UCSM) at SAN > SAN Cloud > Fabric > Uplink FC Interfaces > FC Interface x/yin the General tab.

Verify

Symptoms

- Interfaces come up as single ports with trunking disabled, but not in a port-channel.

- A Single interface fails to come up with trunking enabled.

- When in a port-channel or added to a port-channel, interfaces become error-disabled immediately.

Log Messages

Note: It is possible for the switch OUI to be absent from the peer's OUI database on one or both sides of a link. Both sides must be investigated.

For MDS and Nexus switches:

The switch without the peer's OUI in its OUI database records "physical flogi rejected, waiting for the port mode" and "EPP_SYNC" errors.

MDSswitch# show flogi internal event-history errors | include reject previous 1 next 1

...

10) Event:E_DEBUG, length:124, at 727676 usecs after Tue May 14 17:44:47 2013

[102] fs_fc2_msg_flogi: ifindex[0x12ae000] pwwn[xx:xx:00:2a:6a:xx:xx:xx] physical flogi rejected, waiting for the port mode

11) Event:E_DEBUG, length:124, at 661821 usecs after Tue May 14 17:44:39 2013

[102] fs_fc2_msg_flogi: ifindex[0x12ae000] pwwn[xx:xx:00:2a:6a:xx:xx:xx] physical flogi rejected, waiting for the port mode

12) Event:E_DEBUG, length:73, at 772303 usecs after Tue May 14 17:43:11 2013

[102] fs_flogi_send_flogi_reject: mts_q == 0, ifindex 0x12ae000, port 0x0

MDSswitch# show port internal event-history errors | include EPP previous 1 next 1

...

19) Event:E_DEBUG, length:48, at 465145 usecs after Tue May 14 17:44:49 2013

[102] epp_ac_accept_sync: Error sending EPP_SYNC

...

27) Event:E_DEBUG, length:48, at 280537 usecs after Tue May 14 17:44:43 2013

[102] epp_ac_accept_sync: Error sending EPP_SYNC

To confirm the interface of the errors, translate the index withshow interface snmp-ifindexand filter by the ifindex in the error message. The example uses the ifindex 12ae000 found in theshow flogi internal event-history errors.

MDSswitch# show interface snmp-ifindex | include 12ae000

fc6/47 0019587072 (00012ae000)

For UCS FIs: The FI without the peer's OUI in its OUI database records "PI_FSM_EV_PORT_CONFIG_FAILURE" and "port fcx/y attempting vf tagging on non-cisco switch" errors.

UCS(nxos)# show port internal event-history errors

1) Event:E_DEBUG, length:117, at 225850 usecs after Wed Aug 1 10:18:26 2018

[102] pi_fsm_port_attr_change_init: Ifindex (fc1/47)0x102e000, Err disabled event (PI_FSM_EV_PORT_CONFIG_FAILURE)0xd7

2) Event:E_DEBUG, length:100, at 222587 usecs after Wed Aug 1 10:18:26 2018

[102] pm_process_fport_vf_tagging_capability: port fc1/47 attempting vf tagging on non-cisco switch

OUI Database

Extract the OUI from each switch. The OUI is the third, fourth, and fifth octet of the World Wide Name (WWN).

peer-switch# show wwn switch

Switch WWN is xx:xx:00:2a:6a:xx:xx:xx

In this example, the switch OUI is0x002A6A.

Check the switch's OUI database for the peer switch's OUI. If the command has no output, the OUI is absent from the database.

MDSswitch# show wwn oui | include 0x002a6a

MDSswitch#

Note: UCS FIs don't support theshow wwn ouicommand. Refer to Table 1. and Table 2. in order to see when OUIs were added to the FI OUI database.

Work Around

Switch OUI databases are updated through switch software upgrades. Until a switch can be upgraded to a software version with an OUI database that includes the peer switch's OUI, use single links configured outside of port-channels with trunking disabled.

Thewwn oui <hex oui> command on MDS and Nexus switches can manually add Cisco OUIs to the switch's OUI database.

On MDS switches, this feature is available in NX-OS Version 7.3(0)D1(1)and later.

On Nexus 5K/6K switches, this feature was added in NX-OS Version 7.3(3)N1(1)and later.

On Nexus 9000 switches on NX-OS 7.x, this feature is available in NX-IS Version 7.3(1)N1(1) and later.

On Nexus 9000 switches on other versions of NX-OS, this feature is available in NX-OS Version9.3(3)and later.

Example: MDSswitch# configure terminal

MDSswitch(config)# wwn oui 0x0000fc

MDSswitch# show wwn oui

OUI Vendor Default/Static

----------------------------------------------------

0x0000fc Cisco Static

Nexus 9000 switches on ACI images can add an OUI to the library in run time from the APIC in15.0(1k) and later.

OUI Database Additions

Table 1.1. MDS OUI Database Additions

OUI

Software that includes OUI

0x547FEE

All versions of NX-OS 5.0(1) and later.

0x00351A

0x003A7D

0x004268

0x0062EC

0x007888

0x00C164

0x00C88B

0x00F28B

0x00FEC8

0x046273

0x188B9D

0x5897BD

0x58AC78

0x5C838F

0x64F69D

0x70E422

0x80E01D

0x84B261

0x84B802

0xA46C2A

0xCC167E

0xCC46D6

0xD8B190

0xE00EDA

0xE4AA5D

NX-OS 5.x at 5.2(8h) and later.

NX-OS 6.x at 6.2(17) and later.

NX-OS 7.x at 7.3(1)D1(1) and later.

All versions of NX-OS 8.3(1) and later.

0x0018BA

0x001B54

0x002255

0x0023AC

0x002498

0x0024F7

0x002651

0x002698

0x002A6A

0x00DEFB

0x04C5A4

0x108CCF

0x18EF63

0x1CDF0F

0x405539

0x68BDAB

0x8C604F

0x8CB64F

0xA8B1D4

0xB41489

0xC0626B

0xF02572

0xF866F2

All versions of NX-OS 5.2(6) and later.

0x508789

0x58F39C

0x7426AC

0x7C0ECE

0x881DFC

0x88F031

0x8C604F

0xA0ECF9

0xF07F06

0xF40F1B

0xF44E05

0xF8C288

0xFC5B39

All versions of NX-OS 6.2(11) and later.

0x000831

0x003A9C

0x74A02F

0xD0A5A6

NX-OS 5.x at 5.2(8g) and later.

NX-OS 6.x at 6.2(11c) and later.

All versions of NX-OS 7.3(1)D1(1) and later.

0x70EA1A

0xC4F7D5

All versions of NX-OS 8.4(2) and later.

0x00FD22

0x10B3D5

0x10B3D6

0x4C710D

0xC4B239

0xD4E880

NX-OS 6.x at 6.2(33) and later.

All versions of NX-OS 8.4(2) and later.

0xDC774C

NX-OS 8.4(x) at 8.4(2c) and later.

NX-OS 9.x at 9.2(1) and later.

0x3C13CC

0x4CE176

To be determined

Table 1.2. Nexus 5000/5500/5600/6000 OUI Database Additions

OUI

Software that includes OUI

0x002A6A

0x00DEFB

0x8C604F

NX-OS 5.1 at 5.1(3)N2(1) and later.

NX-OS 5.2 at 5.2(1)N1(4) and later.

All versions of NX-OS 6.0(2)N1(2) and later.

0x50EB1A

0x9371D5

All versions of NX-OS 7.3(7)N1(1) and later.

0x3C13CC

0x4CE176

0xDC774C

All versions of NX-OS 7.3(10)N1(1) and later.

Table 1.3. Nexus 9000 OUI Database Additions

OUI

Software that includes OUI

0x000831

0x001086

0x0024FF

0x0027E3

0x002A10

0x002CC8

0x00351A

0x003A7D

0x003A99

0x003A9C

0x004268

0x005D73

0x0062EC

0x006BF1

0x007888

0x0081C4

0x0090FA

0x00A2EE

0x00A38E

0x00A6CA

0x00A742

0x00B771

0x00BE75

0x00C164

0x00C88B

0x00D78F

0x00EABD

0x00F28B

0x00F663

0x00FCBA

0x00FEC8

0x043110

0x046273

0x0896AD

0x1005CA

0x14F0C5

0x188090

0x188B9D

0x18E728

0x1C6A7A

0x286F7F

0x28AC9E

0x2C27D7

0x2C3311

0x2C44FD

0x2C5A0F

0x2CD02D

0x380E4D

0x3890A5

0x3C08F6

0x3C4A92

0x40017A

0x40CE24

0x4403A7

0x4C776D

0x500F80

0x502FA8

0x5061BF

0x508789

0x54A274

0x58971E

0x5897BD

0x58AC78

0x58F39C

0x5C838F

0x641225

0x64A0E7

0x64F69D

0x68BDAB

0x6C9CED

0x6CB2AE

0x700F6A

0x70695A

0x70708B

0x7079B3

0x707DB9

0x70DB98

0x70DF2F

0x70E422

0x7426AC

0x74A02F

0x74A2E6

0x780CF0

0x78725D

0x78BAF9

0x78DA6E

0x78E3B5

0x7C0ECE

0x7C69F6

0x80E01D

0x843DC6

0x8478AC

0x84B261

0x84B802

0x881DFC

0x885A92

0x88F031

0x8CB64F

0xA0239F

0xA03D6F

0xA09351

0xA0E0AF

0xA0ECF9

0xA44C11

0xA46C2A

0xA80C0D

0xA89D21

0xA8B1D4

0xB02680

0xB08BCF

0xB0AA77

0xB41489

0xB4DE31

0xB83861

0xBADBAD

0xBC26C7

0xC0626B

0xC067AF

0xC08C60

0xCC167E

0xCC46D6

0xCE90D1

0xD072DC

0xD0A5A6

0xD46D50

0xD867D9

0xD8B190

0xE00EDA

0xE4AA5D

0xE4C722

0xE86549

0xE8B748

0xE8BA70

0xEC3091

0xEC4476

0xECBD1D

0xECC882

0xF02572

0xF07F06

0xF40F1B

0xF44E05

0xF45FD4

0xF4ACC1

0xF4CFE2

0xF4DBE6

0xF80BCB

0xF866F2

0xF8C288

0xFC5B39

0xFCFBFB

All versions of NX-OS9.2(4) and later.

0x3C13CC

0x4CE176

0xDC774C

All versions of NX-OS10.1(2) and later.

Table 1.4. UCS FI OUI Database Additions

OUI

Software that includes OUI

0x0018BA

0x001B54

0x002255

0x0023AC

0x002498

0x0024F7

0x002651

0x002698

0x002A6A

0x00DEFB

0x04C5A4

0x108CCF

0x18EF63

0x1CDF0F

0x405539

0x68BDAB

0x8C604F

0x8CB64F

0xA8B1D4

0xB41489

0xC0626B

0xF02572

0xF866F2

Bundle software 2.x at 2.2(8j) and later.

Bundle software 3.x at 3.1(3b) and 3.1(3h) and later.

Bundle software 3.2(3d) and later.

0x002651

Bundle software 2.x at 2.2(8j) and later.

Bundle software 3.x at 3.1(3h) and later.

Bundle software 3.2(3d) and later.

0x0025B5

0xA44C11

Bundle software 3.x at 3.1(3h) and later.

Bundle software 3.2(3d) and later.

0x000831

0x003A9C

0xD0A5A6

Bundle software 3.x at 3.2.(3h) and later.

All versions of bundle software at 4.0(1x).

Bundle software at 4.0(2b) and later.

0x6400F1

Bundle software 3.3(5b) and later.

0x00EEAB

0x0476B0

0x24169D

0x34ED1B

0x3C510E

0x4CE175

0x4CE176

0x5CA62D

0x6C310E

0x70EA1A

0x7C210E

0x7CAD4F

0x9077EE

0xA0B439

0xA4530E

0xAC3A67

0xAC4A67

0xBC5A56

0xC014FE

0xC4F7D5

0xCC7F76

0xE41F7B

Bundle software 4.1(3a) and later.

0xAC7A56

Bundle software 4.1(2a) and later.

0x3C13CC

0x4CE176

0xDC774C

Bundle software 4.2(1d) and later.

Caveats

Cisco bug ID CSCut41864 - Port process crashes when interface > 64 used for NPV-NPIV connections

Cisco bug ID CSCuz52695 - UCS FI FC Uplinks Error Disable Immediately Connected NPV to MDS9396S

Cisco bug ID CSCvp95480 - 6454 FI - Restrict allowed WWxN addresses during UCSM Configuration (avoid multicast address)

Multiple bugs have been filed to document OUIs that were absent from switches' OUI libraries. They include:

MDS/UCS Bugs

Cisco bug ID CSCtr01652- Add N7K OUIs to liboui's list of recognized Cisco OUIs. MDS.

MDS/N5K/N6K Bugs

Cisco bug ID CSCty04686- Add MAC OUI "002a6a", "8c604f", "00defb" for 5k/UCS-FI

MDS Bugs

Cisco bug ID CSCta97375 - GLDN:SABRE: Sabre oui is not recognized as cisco oui 547FEE.

Cisco bug ID CSCur44218 - Add MiniUCS FI OUIs to liboui's list of recognized Cisco OUIs

Cisco bug ID CSCuz11494 - FC port channel does not come up with peer UCS OUI 00:c8:8b

Cisco bug ID CSCuu03717 - Add OUIs 00:3a:9c, 00:08:31, d0:a5:a6 to default OUI list

Cisco bug ID CSCuu26045 - Add MiniUCS FI OUI 0x74a02f to MDS list of recognized Cisco OUIs

Cisco bug ID CSCvx67383 - MDS Port-Channel towards ACI LEAF N9K with switch OUI 0x3C13CC (3C:13:CC) does not come up or trunk

Cisco bug ID CSCvx49403 - Port-Channel with Cisco switch OUI 0xDC774C does not come up or trunk.

Cisco bug ID CSCvr60193 - Add OUI C4:F7:D5 to the default OUI list

Cisco bug ID CSCvp19788- MDS 9000 Add OUI 70:EA:1A to OUI library

Cisco bug ID CSCvt28351 - Add six OUI's to the default OUI list

Cisco bug ID CSCvw14504 - Add OUI 0x4CE176 to OUI Database - Port-channels and trunking not working with OUI 0x4CE176

Cisco bug ID CSCvx67383 - MDS Port-Channel towards ACI LEAF N9K with switch OUI 0x3C13CC (3C:13:CC) does not come up or trunk

N5K/N6K Bugs

Cisco bug ID CSCvx72821 - Port-Channel with Cisco switch OUI 0xDC774C or 0x4CE176 or 0x3C13CC does not come up or trunk.

Cisco bug ID CSCvp54881 - Unable to create San Port Channel between UCS & Nexus 56128p

N9K Bugs

Cisco bug ID CSCvp21786 - Nexus 9000:::Unable to bring up F mode san-port-channel or trunking between MDS and N9K

Cisco bug ID CSCvx68507 - Port-Channel with Cisco switch OUI 0xDC774C or 0x4CE176 or 0x3C13CC does not come up or trunk.

UCS Bugs

Cisco bug ID CSCvk63036 - UCS Missing OUI 003a9c, 000831, d0a5a6 -- Prevents FC/FCoE Port-Channel From Forming

Cisco bug ID CSCvv70634 - UCS FI needs to add additional N9K OUI's

Cisco bug ID CSCvx68517 - Port-Channel with Cisco switch OUI 0xDC774C or 0x4CE176 or 0x3C13CC does not come up or trunk. Cisco bug ID