PDF(203.4 KB) View with Adobe Reader on a variety of devices
ePub(88.4 KB) View in various apps on iPhone, iPad, Android, Sony Reader, or Windows Phone
Mobi (Kindle)(97.3 KB) View on Kindle device or Kindle app on multiple devices
Updated:February 11, 2015
Bias-Free Language
The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.
Release notes are sometimes updated with new information about restrictions and caveats. See the following website for the most recent version of the Cisco NX-OS Release 11.0(3f) Release Notes for Cisco Nexus 9000 Series ACI-Mode Switches:
Cisco NX-OS Software for the Cisco Nexus 9000 Series is a data center, purpose-built, operating system designed with performance, resiliency, scalability, manageability, and programmability at its foundation. It provides a robust and comprehensive feature set that meets the requirements of virtualization and automation in data centers
Cisco NX-OS Release 11.0 works only on Cisco Nexus 9000 Series switches in ACI Mode.
See Table 2 for a list of modules that are supported on Cisco Nexus 9000 Series switches in ACI Mode.
Supported Hardware
Table 2 lists the hardware that the Cisco Nexus 9000 Series ACI Mode switches support.
Table 2 Supported Hardware
Hardware Type
Product ID
Description
Chassis
N9K-C9504
Cisco Nexus 9504 chassis with four slots
Chassis
N9K-C9508
Cisco Nexus 9508 chassis with 8 slots
Chassis component
N9K-C9508-FAN
Fan tray
Chassis component
N9k-PAC-3000W-B
Cisco Nexus 9500 3000W AC power supply, port side intake
Cisco Nexus 9300 48-port, 1/10 Gigabit Ethernet SFP+ and 6-port or 12-port, 40 Gigabit Ethernet QSFP switch
Top-of-rack (ToR) leaf switch
N9K-C9396TX
Cisco Nexus 9300 48-port, 1/10 Gbps Base-T and 6-port or 12-port, 40 Gigabit Ethernet QSFP switch
Top-of-rack (ToR) leaf switch power supply unit
UCSC-PSU-930WDC V01
Port side intake DC power supply compatible with all ToR leaf switches
Top-of-rack (ToR) leaf switch power supply unit
N9K-PAC-650W-B
650W AC Power supply, port side exhaust pluggable
Top-of-rack (ToR) leaf switch power supply unit
N9K-PAC-650W
650W AC Power supply, port side intake pluggable
Top-of-rack (ToR) leaf switch power supply unit
N9K-PAC-1200W-B
1200W AC Power supply, port side exhaust pluggable
Top-of-rack (ToR) leaf switch power supply unit
N9K-PAC-1200W
1200W AC Power supply, port side intake pluggable
Top-of-rack (ToR) leaf switch power supply unit
N9K-PUV-3000W-B
3000W AC Power supply, port side exhaust pluggable
Top-of-rack (ToR) leaf switch fan
NXA-FAN-30CFM-F
Port side exhaust fan
Top-of-rack (ToR) leaf switch fan
NXA-FAN-30CFM-B
Port side intake fan
Supported FEX Models
Table 3 lists the FEX models that the Cisco Nexus 9000 Series ACI Mode switches support. For more information on the FEX models, see Cisco Nexus 2000 Series Fabric Extenders Data Sheet.
Note FEX requires software version 5.x or later to be brought up successfully.
Table 3 Supported FEX Models
Product ID
Description
N2K-B22DELL-P
B22 FEX for Dell
N2K-B22IBM-P
B22 FEX for IBM
N2K-C2232PP-10GE
Cisco Nexus 2232PP Series 10GE Fabric Extender, 2 AC PS, 1 Fan Module (Standard Airflow/port side exhaust), 32x1/10GE (req SFP/SFP+) + 8x10GE (req SFP+), same as N2K-C2232PP
N2K-C2232TM-E-10GE
Cisco Nexus 2232TM-E Series 10GBASE-T Fabric Extender, 2PS, 1 Fan Module, 32x1/10GBase-T + 8x10GE Module (req SFP+), choice of airflow and power supply
N2K-C2248PQ-10GE
Cisco Nexus 2248PQ 10GE Fabric Extender, 2PS, 4 Fan Module, 48x1/10GE (req SFP/SFP+) + 4x40G QSFP+(req QSFP+), choice of airflow and power supply
N2K-C2248TP-1GE
Cisco Nexus 2248TP Series 1GE Fabric Extender, 2 AC PS, 1 Fan Module (Standard Airflow/port side exhaust), 48x100/1000Base-T + 4x10GE (req SFP+), same as N2K-C2248TP
N2K-C2248TP-E-1GE
Cisco Nexus 2248TP-E Series 1GE Fabric Extender, 2PS, 1 Fan Module, 48x100/1000Base-T + 4x10GE (req SFP+), 32MB buffer, choice of airflow and power supply
New and Changed Information
This section lists the new and changed features in Release 11.0(3f), and includes the following topics:
– N9K-PAC-650W-B - 650W AC Power supply, port side exhaust pluggable
– N9K-PAC-650W - 650W AC Power supply, port side intake pluggable
– N9K-PAC-1200W-B - 1200W AC Power supply, port side exhaust pluggable
– N9K-PAC-1200W - 1200W AC Power supply, port side intake pluggable
– UCSC-PSU-930WDC V01 - Port side intake DC power supply compatible with all ToR leaf switches
– N9K-PUV-3000W-B - 3000W AC Power supply, port side exhaust pluggable
Supported Fans:
– N9K-C9300-FAN3 - Port side exhaust fan
– N9K-C9300-FAN3-B - Port side intake fan
– NXA-FAN-30CFM-F - Port side exhaust fan
– NXA-FAN-30CFM-B - Port side intake fan
New Software Features
The Cisco Nexus 9000 Series ACI Mode Release 11.0(3f) supports the following new software:
Stretched Fabric - This feature allows for each leaf and all spines that participate in creating a fabric to be located up to 30 KMs apart and removes the restriction for every leaf to be connected to all spines.
For more information about the stretched fabric feature, see the KB: ACI Stretched Fabric Design knowledge base article:
For installation instructions, see the Cisco ACI Fabric Hardware Installation Guide.
Upgrade Instructions
Follow this procedure when upgrading from a 1.0(2x) release to a 1.0(3x) release:
1. Upgrade the APIC controller software image.
2. After all APICs in the cluster are successfully upgraded, upgrade all the switches in the fabric.
Note The switches may need to be rebooted after upgrading (See CSCut32029).
Downgrade Instructions
Follow this procedure when downgrading from 1.0(3x) release to a 1.0(2x) release:
1. Downgrade the APIC controller software image.
2. After all APICs in the cluster are successfully downgraded, downgrade all the switches in the fabric.
Note Switch models N9K-C9372PX, N9K-C9332PQ, and N9K-C9372TX are not supported for downgrading in the APIC 1.0(2x) or the Cisco Nexus 9000 11.0(2x) releases. If your fabric has those models, do not downgrade.
Compatibility Information
Cisco NX-OS Release 11.0(3f) supports the hardware and software listed on the ACI Ecosystem Compatibility List and the Cisco AVS, Release 4.2(1)SV2(2.3).
The breakout of 40G ports to 4x10G on the N9332PQ switch is not supported in ACI-Mode.
To connect the APIC (the controller cluster) to the ACI fabric, it is required to have a 10G interface on the ACI leaf. You cannot connect the APIC directly to the N9332PQ ACI Leaf.
Usage Guidelines
The current list of protocols that are allowed (and cannot be blocked through contracts) include the following. Some of the protocols have SrcPort/DstPort distinction.
– UDP DestPort 161: SNMP. These cannot be blocked through contracts. Creating an SNMP ClientGroup with a list of Client-IP Addresses restricts SNMP access to only those configured Client-IP Addresses. If no Client-IP address is configured, SNMP packets are allowed from anywhere.
This section lists the open caveats in Cisco NX-OS Release 11.0(3f). Click a Bug ID shown in Table 4 to access the Bug Search Tool and see additional information about the bug.
If the TOR 1RU system is configured with the RED fan (the reverse airflow), the air will flow from back to front. The temperature sensor in the back will be defined as an Inlet temperature sensor, and the temperature sensor in the front will be defined as an outlet temperature sensor.
If the TOR 1RU system is configured with the BLUE fan (normal airflow), the air will flow from front to back. The temperature sensor in the front will be defined as an Inlet temperature sensor, and the temperature sensor in the back will be defined as outlet temperature sensor.
From the airflow perspective, the Inlet sensor reading should always be less than the outlet sensor reading. However, in the TOR 1RU family, the front panel temperature sensor has some inaccurate readings due to the front panel utilization & configuration, which causes the Inlet temperature sensor reading to be very close, equal, or even greater than the outlet temperature reading.
When a traceroute is performed from a VM attached to a regular bridge domain and to a VM behind a border leaf, and both of these VMs are behind two different ToRs, the traceroute does not show the ToR with the border leaf.
Resolved Caveats
This section lists caveats that are resolved in Cisco NX-OS Release 11.0(3f). Click a Bug ID shown in Table 5 to access the Bug Search Tool and see additional information about the bug.
In case of multiple QoS Custom Policies (QoSCustomPol) or multiple operations on an existing QoS Custom Policy, the DSCP and Dot1p rules do not get configured correctly on ToRs.
The ToR's object store may or may not show an error.
Once the programming fails for DSCP/Dot1p Rules, further DSCP/DoT1p programming on ToR is not possible. It will also affect the contract configuration of new l3extOut/l2extOut (external networks) that requires DSCP remarking.
The static route to the DHCP provider's subnet is not deleted from the consumer's private network after deleting the contract and the DHCP label from the consumer Bridge Domain.
The new host has no network connectivity. The leaf switches keep pinning the IP address on the old MAC address and does not timeout the MAC address.
Known Behaviors
This section lists caveats that describe known behaviors in Cisco NX-OS Release 11.0(3f). Click a Bug ID shown in Table 6 to access the Bug Search Tool and see additional information about the bug.
When an IP moves from one MAC behind one ToR to another MAC behind another ToR, even though the VM sends a GARP packet, in ARP unicast mode, this GARP packet is not flooded. As a result, any other host with the original MAC to IP binding sending an L2 packet will send to the original ToR where the IP was in the beginning (based on MAC lookup), and the packet will be sent out on the old port (location). Without flooding the GARP packet in the network, all hosts will not update the MAC-to-IP binding.
When modifying the L2Unknown Unicast parameter on a Bridge Domain (BD), interfaces on externally connected devices may bounce. Additionally, the endpoint cache for the BD is flushed and all endpoints will have to be re-learned.
If an endpoint has multiple IPs, the endpoint will not be aged until all IPs go silent. If one of the IPs is reassigned to another server/host, fabric detects it as an IP move and forwarding will work as expected.
When removing the secondary IP on an external SVI interface, static routes defined in the VRF or context of the SVI are removed causing traffic to be looped in the fabric.
ARP does not reach hosts in the same endpoint group.
The Cisco Nexus 9508 ACI-mode switch supports warm (stateless) standby where the state is not synched between the active and the standby supervisor modules. For an online insertion and removal (OIR) or reload of the active supervisor module, the standby supervisor module becomes active, but all modules in the switch are reset because the switchover is stateless. In the output of the show system redundancy status command, warm standby indicates stateless mode.
When a recommissioned APIC controller rejoins the cluster, GUI and CLI commands can time out while the cluster expands to include the recommissioned APIC controller.
If connectivity to the APIC cluster is lost while a switch is being decommissioned, the decommissioned switch may not complete a clean reboot. In this case, the fabric administrator should manually complete a clean reboot of the decommissioned switch.
Before expanding the APIC cluster with a recommissioned controller, remove any decommissioned switches from the fabric by powering down and disconnecting them. Doing so will ensure that the recommissioned APIC controller will not attempt to discover and recommission the switch.
IGMP Snooping Known Behaviors:
Multicast router functionality is not supported when IGMP queries are received with VxLAN encapsulation.
IGMP Querier election across multiple Endpoint Groups (EPGs) or Layer 2 outsides (External Bridged Network) in a given Bridge Domain (BD) is not supported. Only one EPG or Layer 2 outside for a given BD should be extended to multiple multicast routers if any.
The rate of the number of IGMP reports sent to a leaf switch should be limited to 1000 reports per second.
Unknown IP multicast packets are flooded on ingress leaf switches and border leaf switches, unless “unknown multicast flooding” is set to “Optimized Flood” in a BD. This knob can be set to “Optimized Flood” only for a maximum of 50 BDs per leaf.
If “Optimized Flood” is enabled for more than the supported number of BDs on a leaf, follow these configuration steps to recover:
– Set “unknown multicast flooding” to “Flood” for all BDs mapped to a leaf.
– Set “unknown multicast flooding” to “Optimized Flood” on needed BDs.
Related Documentation
This section lists the product documentation for the Cisco ACI.
This document is to be used in conjunction with the documents listed in the
“Related Documentation” section.
Cisco and the Cisco logo are trademarks or registered trademarks of Cisco and/or its affiliates in the U.S. and other countries. To view a list of Cisco trademarks, go to this URL: www.cisco.com/go/trademarks. Third-party trademarks mentioned are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (1110R)
Any Internet Protocol (IP) addresses and phone numbers used in this document are not intended to be actual addresses and phone numbers. Any examples, command display output, network topology diagrams, and other figures included in the document are shown for illustrative purposes only. Any use of actual IP addresses or phone numbers in illustrative content is unintentional and coincidental.