Cisco Nexus 5000 NX-OS Quality of Service Configuration Guide, Release 5.1(3)N2(1)
Configuring Queuing and Flow Control
Downloads: This chapterpdf (PDF - 1.22MB) The complete bookPDF (PDF - 2.57MB) | Feedback

Configuring Queuing and Flow Control

Contents

Configuring Queuing and Flow Control

This chapter contains the following sections:

Information About Queues

Ingress Queuing Policies

You can associate an ingress policy map with an Ethernet interface to guarantee bandwidth for the specified traffic class or to specify a priority queue.

The ingress policy is applied in the adapter to all outgoing traffic that matches the specified CoS value.

When you configure an ingress policy for an interface, the switch sends the configuration data to the adapter. If the adapter does not support the DCBX protocol or the ingress policy type-length-value (TLV), the ingress policy configuration is ignored.

Egress Queuing Policies

You can associate an egress policy map with an Ethernet interface to guarantee the bandwidth for the specified traffic class or to configure the egress queues.

The bandwidth allocation limit applies to all traffic on the interface including any FCoE traffic.

Each Ethernet interface supports up to six queues, one for each system class. The queues have the following default configuration:

  • In addition to these queues, control traffic that is destined for the CPU uses strict priority queues. These queues are not accessible for user configuration.
  • FCoE traffic (traffic that maps to the FCoE system class) is assigned a queue. This queue uses weighted round-robin (WRR) scheduling with 50 percent of the bandwidth.
  • Standard Ethernet traffic in the default drop system class is assigned a queue. This queue uses WRR scheduling with 100 percent of the bandwidth.

If you add a system class, a queue is assigned to the class. You must reconfigure the bandwidth allocation on all affected interfaces. Bandwidth is not dedicated automatically to user-defined system classes.

You can configure a strict priority queue. This queue is serviced before all other queues except the control traffic queue (which carries control rather than data traffic).

Buffering and Queue Limits on the Cisco Nexus 5000 Platform

The following buffering limits exist for the Cisco Nexus 5000 Platform:

  • Maximum ingress port buffering: 320KB per port.
  • Maximum egress port buffering: 160KB per port.

The following default buffer allocations per port exist for the Cisco Nexus 5000 Platform:

Table 1 Cisco Nexus 5000 Platform Default Buffer Allocations Per Port
Traffic Class Ingress Buffer (KB)
Class-fcoe 76.8
User-defined no-drop class of service with an MTU less than 2240 76.8
User-defined no-drop class of service with an MTU greater than 2240 81.9
Tail drop class of service 20.4
Class-default All of the remaining buffer (243.2KB with the default QoS configuration)

The default buffer allocation varies depending on the type of class. For example, if you create a regular tail drop traffic class the default allocation is 22.7Kb, unless you specify a larger size using the queue-limit command.

To increase the buffer space available to a user-created qos-group, from a network-qos policy-map, use the queue-limit command.

All of the available buffer is allocated to the class-default. When you define a new qos-group, the required buffer for the new qos-group is taken from the class-default buffer.


Note


Each new class requires an additional 18.880KB, so the exact amount of buffer that is left in the class default is 243.2KB minus the buffer used by other qos-groups minus 18.880KB times the number of qos-groups.


The default QoS configuration for the Nexus 5000 platform creates the class-fcoe and class-default.

The show queuing interface command displays the configured qos-group and the ingress buffer allocated for each qos-group.

Buffering and Queue Limits on the Cisco Nexus Device

On the Nexus 5500 platform, the packet buffer per port is 640KB. The Nexus 5548P, Nexus 5548UP, and the Nexus 5596UP switch share the same buffer architecture. The Nexus 5500 platform implements Virtual Output Queueing (VOQ) and ingress buffer architecture with the majority of the buffer allocated at ingress. The architecture allows the switch to store packets at multiple ingress ports when there are multiple ports sending traffic to one egress port which causes congestion.

The following default buffer allocations per port exist for the Cisco Nexus 5500 Platform:

Table 2 Cisco Nexus 5500 Platform Default Buffer Allocations Per Port
Traffic Class Ingress Buffer (KB)
Class-fcoe 79.360
User-defined no-drop with an MTU less than 2240 79.360
User-defined no-drop class with an MTU greater than 2240 90.204
Tail drop traffic class 22.720
Class-default All of the remaining buffer (470 with default QoS configuration)

The default buffer allocation varies depending on the type of class. For example, if you create a regular tail drop traffic class the default allocation is 22.7KB, unless you specify a larger size using the queue-limit command.

To increase the ingress buffer space available to a user-created qos-group, from a network-qos policy-map, use the queue-limit command.

In addition to ingress buffer allocated for each user-created qos-group there is an additional 28.6KB buffer required at egress for each qos-group.

With the default QoS configuration, all of the available buffer (470KB) is allocated to the class-default. When you create a new qos-group, the buffer required for the new qos-group will be taken away from class-default. The amount of buffer that is left for class-default equals 470 minus the ingress buffer used by other qos-groups minus 28.6KB and times the number of qos-groups.


Note


Each new class requires an additional 28.6KB, so the exact amount of buffer that is left in the class default equals 478 minus the buffer used by other qos-groups minus 18.880KB times the number of qos-groups.


The default QoS policy for the Nexus 5500 platform doe not create class-fcoe and doesn't reserve buffer and qos-group for FCoE traffic.

The show queuing interface command can display the amount of ingress buffer allocated for each qos-group

Information About Flow Control

Link-Level Flow Control

IEEE 802.3x link-level flow control allows a congested receiver to communicate a transmitter at the other end of the link to pause its data transmission for a short period of time. The link-level flow control feature applies to all the traffic on the link.

The transmit and receive directions are separately configurable. By default, link-level flow control is disabled for both directions.

On the Cisco Nexus device, Ethernet interfaces do not automatically detect the link-level flow control capability. You must configure the capability explicitly on the Ethernet interfaces.

On each Ethernet interface, the switch can enable either priority flow control or link-level flow control (but not both).

Priority Flow Control

Priority flow control (PFC) allows you to apply pause functionality to specific classes of traffic on a link instead of all the traffic on the link. PFC applies pause functionality based on the IEEE 802.1p CoS value. When the switch enables PFC, it communicates to the adapter which CoS values to apply the pause.

Ethernet interfaces use PFC to provide lossless service to no-drop system classes. PFC implements pause frames on a per-class basis and uses the IEEE 802.1p CoS value to identify the classes that require lossless service.

In the switch, each system class has an associated IEEE 802.1p CoS value that is assigned by default or configured on the system class. If you enable PFC, the switch sends the no-drop CoS values to the adapter, which then applies PFC to these CoS values.

The default CoS value for the FCoE system class is 3. This value is configurable.

By default, the switch negotiates to enable the PFC capability. If the negotiation succeeds, PFC is enabled and link-level flow control remains disabled regardless of its configuration settings. If the PFC negotiation fails, you can either force PFC to be enabled on the interface or you can enable IEEE 802.x link-level flow control.

If you do not enable PFC on an interface, you can enable IEEE 802.3X link-level pause. By default, link-level pause is disabled.

Configuring Queuing

Configuring the Queue Limit for a Specified Fabric Extender

At the Fabric Extender configuration level, you can control the queue limit for a specified Fabric Extender for egress direction (from the network to the host). You can use a lower queue limit value on the Fabric Extender to prevent one blocked receiver from affecting traffic that is sent to other noncongested receivers ("head-of-line blocking"). A higher queue limit provides better burst absorption and less head-of-line blocking protection. You can use the no form of this command to allow the Fabric Extender to use all available hardware space.


Note


At the system level, you can set the queue limit for Fabric Extenders by using the fex queue-limit command. However, configuring the queue limit for a specific Fabric Extender will override the queue limit configuration set at the system level for that Fabric Extender.


You can specify the queue limit for the following Fabric Extenders:

  • Cisco Nexus 2148T Fabric Extender (48x1G 4x10G SFP+ Module)
  • Cisco Nexus 2224TP Fabric Extender (24x1G 2x10G SFP+ Module)
  • Cisco Nexus 2232P Fabric Extender (32x10G SFP+ 8x10G SFP+ Module)
  • Cisco Nexus 2248T Fabric Extender (48x1G 4x10G SFP+ Module)
  • Cisco Nexus N2248TP-E Fabric Extender (48x1G 4x10G Module)
Procedure
      Command or Action Purpose
    Step 1 switch# configure terminal 

    Enters global configuration mode.

     
    Step 2 switch(config)# fex fex-id
     

    Specifies the Fabric Extender and enters the Fabric Extender mode.

     
    Step 3 switch(config-fex)# hardware fex_card_type queue-limit queue-limit
     

    Configures the queue limit for the specified Fabric Extender. The queue limit is specified in bytes. The range is from 81920 to 652800 for a Cisco Nexus 2148T Fabric Extender and from 2560 to 652800 for all other supported Fabric Extenders.

     

    This example shows how to restore the default queue limit on a Cisco Nexus 2248T Fabric Extender:

    switch# configure terminal
    switch(config-if)# fex 101
    switch(config-fex)# hardware N2248T queue-limit 327680

    This example shows how to remove the queue limit that is set by default on a Cisco Nexus 2248T Fabric Extender:

    switch# configure terminal
    switch(config)# fex 101
    switch(config-fex)# no hardware N2248T queue-limit 327680

    Configuring No-Drop Buffer Thresholds

    You can configure the no-drop buffer threshold settings for 3000m lossless Ethernet.


    Note


    To achieve lossless Ethernet for both directions, the devices connected to the Cisco Nexus device must have the similar capability. The default buffer and threshold value for the no-drop can ensure lossless Ethernet for up to 300 meters.
    Procedure
        Command or Action Purpose
      Step 1 switch# configure terminal 

      Enters global configuration mode.

       
      Step 2 switch(config)# policy-map type network-qos policy-map name
       

      Enters policy-map network-qos class mode and identifies the policy map assigned to the type network-qos policy map.

       
      Step 3 switch(config-pmap-nq)# class type network-qos class-map name
       

      References an existing network QoS class map in a policy map and enters class mode.

       
      Step 4 switch(config-pmap-nq-c)# pause no-drop buffer-size buffer-size pause-threshold xoff-size resume-threshold xon-size
       

      Specifies the buffer threshold settings for pause and resume for 3000m lossless Ethernet:

      • buffer-size—Buffer size for ingress traffic, in bytes. Valid values are from 10240 to 490880.
        Note   

        On a Cisco Nexus 5500 Series device, you can configure a maximum buffer size of 152000 bytes.

      • pause-threshold—Specifies the buffer limit at which the port pauses the peer.
      • xoff-size—Buffer limit for pausing, in bytes. Valid values are 0 to 490880.
        Note   

        On a Cisco Nexus 5500 Series device, you can configure a maximum pause-threshold value of 103360 bytes.

      • resume-threshold—Specifies the buffer limit at which the port resumes the peer.
      • xon-size—Buffer limit at which to resume, in bytes. Valid values are 0 to 490880.
        Note   

        On a Cisco Nexus 5500 Series device, you can configure a maximum resume-threshold value of 83520 bytes.

       
      Step 5 switch(config-pmap-nq-c)# no pause no-drop buffer-size buffer-size pause-threshold xoff-size resume-threshold xon-size
       
      (Optional)

      Removes the buffer threshold settings for pause and resume for 3000m lossless Ethernet.

       
      Step 6 switch(config-pmap-nq-c)# exit
       

      Exits class mode.

       
      Step 7 switch(config-pmap-nq)# exit
       

      Exits policy-map network-qos mode.

       

      This example shows how to configure the no-drop buffer threshold for the Cisco Nexus device for 3000 meters.

      switch(config-pmap-nq)# policy-map type network-qos nqos_policy
      switch(config-pmap-nq)# class type network-qos nqos_class
      switch(config-pmap-nq-c)# pause no-drop buffer-size 152000 pause-threshold 103360 resume-threshold 83520 
      switch(config-pmap-nq-c)# exit
      switch(config-pmap-nq)# exit
      switch(config)# exit
      switch#
      

      Configuring the Buffer Threshold for the Cisco Nexus 2148T Fabric Extender

      In the Fabric Extender configuration mode, you can configure the buffer threshold for the Cisco Nexus 2148T Fabric Extender. The buffer threshold sets the consumption level of input buffers before an indication is sent to the egress queue to start observing the tail drop threshold. If the buffer usage is lower than the configured buffer threshold, the tail drop threshold is ignored.

      Procedure
          Command or Action Purpose
        Step 1 switch# configure terminal 

        Enters global configuration mode.

         
        Step 2 switch(config)# fex fex-id
         

        Specifies the Fabric Extender and enters the Fabric Extender mode.

         
        Step 3 switch(config-fex)# hardware N2148T buffer-threshold buffer limit
         

        Configures the buffer threshold for the Cisco Nexus 2148T Fabric Extender. The buffer threshold is specified in bytes. The range is from 81920 to 316160 for the Cisco Nexus 2148T Fabric Extender.

         

        This example shows how to restore the default buffer threshold on the Cisco Nexus 2148T Fabric Extender:

        switch# configure terminal
        switch(config)# fex 101
        switch(config-fex)# hardware N2148T buffer-threshold 163840

        This example shows how to remove the default buffer threshold on the Cisco Nexus 2148T Fabric Extender:

        switch# configure terminal
        switch(config)# fex 101
        switch(config-fex)# no hardware N2148T buffer-threshold

        Enabling Virtual Output Queuing Limits for Unicast Traffic on the Cisco Nexus Device

        You can enable the Virtual Output Queuing (VOQ) limit for unicast traffic. To alleviate congestion and blocking, use VOQ to prevent one blocked receiver from affecting traffic that is sent to other noncongested blocking receivers.

        Procedure
            Command or Action Purpose
          Step 1 switch# configure terminal
           

          Enters configuration mode.

           
          Step 2 switch(config)# hardware unicast voq-limit
           

          Enables the VOQ limit for unicast traffic. The default is disabled.

           
          Step 3 switch(config)# no hardware unicast voq-limit
           

          Disables the VOQ limit for unicast traffic.

           
          This example shows how to enable the VOQ limits for unicast packets on a switch:
          switch(config)# hardware unicast voq-limit
          switch(config)#
          

          Configuring Flow Control

          Link-Level Flow Control

          IEEE 802.3x link-level flow control allows a congested receiver to communicate a transmitter at the other end of the link to pause its data transmission for a short period of time. The link-level flow control feature applies to all the traffic on the link.

          The transmit and receive directions are separately configurable. By default, link-level flow control is disabled for both directions.

          On the Cisco Nexus device, Ethernet interfaces do not automatically detect the link-level flow control capability. You must configure the capability explicitly on the Ethernet interfaces.

          On each Ethernet interface, the switch can enable either priority flow control or link-level flow control (but not both).

          Configuring Priority Flow Control

          By default, Ethernet interfaces negotiate PFC with the network adapter using the DCBX protocol. When PFC is enabled, PFC is applied to traffic that matches the CoS value configured for the no-drop class.

          You can override the negotiation result by forcing the interface to enable PFC.

          Procedure
              Command or Action Purpose
            Step 1 switch# configure terminal 

            Enters global configuration mode.

             
            Step 2 switch(config)# interface type slot/port
             

            Specifies the interface to be changed.

             
            Step 3 switch(config-if)# priority-flow-control mode {auto | on}
             

            Sets PFC mode for the selected interface.

            Specifies auto to negotiate PFC capability. This is the default.

            Specifies on to force-enable PFC.

             
            Step 4 switch(config-if)# no priority-flow-control mode on
             
            (Optional)

            Disables the PFC setting for the selected interface.

             

            This example shows how to force-enable PFC on an interface:

            switch# configure terminal
            switch(config)# interface ethernet 1/2
            switch(config-if)# priority-flow-control mode on
            

            Configuring Link-Level Flow Control

            By default, LLC on Ethernet interfaces is disabled. You can enable LLC for the transmit and receive directions.

            Procedure
                Command or Action Purpose
              Step 1 switch# configure terminal 

              Enters global configuration mode.

               
              Step 2 switch(config)# interface type slot/port
               

              Specifies the interface to be changed.

               
              Step 3 switch(config-if)# flowcontrol [receive {on | off}] [transmit {on | off}]
               

              Enables LLC for the selected interface. Set receive and/or transmit on or off.

               
              Step 4 switch(config-if)# no flowcontrol [receive {on | off}] [transmit {on | off}]
               
              (Optional)

              Disables LLC for the selected interface.

               

              This example shows how to enable LLC on an interface:

              switch# configure terminal
              switch(config)# interface ethernet 1/2
              switch(config-if)# flowcontrol receive on transmit on
              

              Disabling Slow Port Pruning on Multicast Traffic on the Cisco Nexus 5500 Series Device

              You can disable slow port pruning on multicast packets.

              An interface on the Cisco Nexus 5500 Series device can become congested when it receives excessive multicast traffic or when the mixed unicast and multicast traffic rate exceeds the port bandwidth. When multiple interfaces receive the same multicast flow and one or more ports experience congestion, the slow port prunning feature allows the switch to drop only the multicast packets for the congested port. This feature is turned on by default. To turn the slow port pruning feature off, enter the hardware multicast disable-slow-port-pruning command.

              Procedure
                  Command or Action Purpose
                Step 1 switch# configure terminal  

                Enters configuration mode.

                 
                Step 2 switch(config)# hardware multicast disable-slow-port-pruning  

                Disables slow port pruning on multicast packets. The default is enabled.

                 
                Step 3 switch(config)# no hardware multicast disable-slow-port-pruning  

                Enables the slow port pruning feature.

                 
                This example shows how to disable slow port pruning on a Cisco Nexus 5548 switch:
                switch(config)# hardware multicast disable-slow-port-pruning
                switch(config)#
                
                

                Verifying the Queue and Flow Control Configurations

                Use one of the following commands to verify the configuration:

                Command

                Purpose

                show queuing interface [interface slot/\port]

                Displays the queue configuration and statistics.

                show interface flowcontrol [module numbef ]

                Displays the detailed listing of the flow control settings on all interfaces.

                show interface [interface slot/port] priority-flow-control [module number]

                Displays the priority flow control details for a specified interface.

                show wrr-queue cos-map [var]

                 

                running-config ipqos

                Displays information about the running configuration for QoS.

                startup-config ipqos

                Displays informationa bout the startup configuration for QoS.