N9K-C93180YC-FX supports long distance (up to 10 kilometers) on FCoE ISLs. The support is applicable on 10G, 25G, and 40G speeds. For line rate traffic without drops, the ingress buffer sizes and pause/resume thresholds need to be increased on long-distance ISLs. This can be achieved by applying custom long-distance FCoE policies to the ISL ports. The default FCoE-related system level network-qos and queuing policies allocate fixed ingress buffer sizes and pause/resume thresholds to all Ethernet ports. To facilitate the increase of ingress buffer allocation for long-distance ISLs, it may be required to decrease the ingress buffer allocation for a few Ethernet ports by using custom short distance FCoE policies.

Note	It is recommended to use FCoE long distance ISLs only for SAN traffic.

Note

Ingress buffer allocation for Ethernet ports that are not bound to VFC or for Ethernet ports bound to VFC (with short distance requirements, that is less than 100 meters) can be decreased by using the custom short distance FCoE policies. When policy is changed on ports running traffic, there will be momentary drop in traffic. If ingress buffer allocation failure happens for an Ethernet port, shut/no shut has to be executed on the port after the ingress buffer is made available for the Ethernet port to come up.

Configurations vary for different types of policies, that is, default system level policy and interface level custom policies for different speeds, as follows:

• Default System Level Policy for FCoE

  switch(config)# system qos 
  switch(config-sys-qos)# service-policy type network-qos default-fcoe-nq-policy
  switch(config-sys-qos)# service-policy type queuing input default-fcoe-in-que-policy
  switch(config-sys-qos)# service-policy type queuing output default-fcoe-out-policy
  

  The default settings for system level policy for FCoE are as follows:

  • Buffer-size - 104000

  • Pause-threshold - 20800

  • Resume-threshold - 19136

• Interface Level Custom Policies for Different Speeds

  Custom policies for long distance that need to be applied to Ethernet port/port channel bound to VFC/VFC-PO ISLs with long distance support are as follows:

  • Long Distance Policy for 10G ISLs

    switch(config)# policy-map type queuing ld_10G_fcoe_in_que_policy
    switch(config-pmap-que)# class type queuing c-in-q1
    switch(config-pmap-c-que)# bandwidth percent 90
    switch(config-pmap-c-que)# pause buffer-size 166400 pause-threshold 20800 resume-threshold 19136
    switch(config-pmap-c-que)# class type queuing c-in-q-default
    switch(config-pmap-c-que)# bandwidth percent 10
    switch(config-pmap-c-que)
    

    switch(config)# policy-map type queuing ld_10G_fcoe_in_que_policy
    switch(config-pmap-que)# class type queuing c-in-q1
    switch(config-pmap-c-que)# bandwidth percent 90
    switch(config-pmap-c-que)# pause buffer-size 291200 pause-threshold 145600 resume-threshold 143936
    switch(config-pmap-c-que)# class type queuing c-in-q-default
    switch(config-pmap-c-que)# bandwidth percent 10
    

  • Long Distance Policy for 25G ISLs

    switch(config)# policy-map type queuing ld_25G_fcoe_in_que_policy
    switch(config-pmap-que)# class type queuing c-in-q1
    switch(config-pmap-c-que)# bandwidth percent 90
    switch(config-pmap-c-que)# pause buffer-size 384800 pause-threshold 20800 resume-threshold 19136
    switch(config-pmap-c-que)# class type queuing c-in-q-default
    switch(config-pmap-c-que)# bandwidth percent 10
    switch(config-pmap-c-que)# 

    switch(config)# policy-map type queuing ld_25G_fcoe_in_que_policy
    switch(config-pmap-que)# class type queuing c-in-q1
    switch(config-pmap-c-que)# bandwidth percent 90
    switch(config-pmap-c-que)#  pause buffer-size 728000 pause-threshold 364000 resume-threshold 362336
    switch(config-pmap-c-que)# class type queuing c-in-q-default
    switch(config-pmap-c-que)# bandwidth percent 10
    switch(config-pmap-c-que)# 

  • Long Distance Policy for 40G ISLs

    switch(config)# policy-map type queuing ld_40G_fcoe_in_que_policy
    switch(config-pmap-que)# class type queuing c-in-q1
    switch(config-pmap-c-que)# bandwidth percent 90
    switch(config-pmap-c-que)# pause buffer-size 728000 pause-threshold 78208 resume-threshold 76544
    switch(config-pmap-c-que)# class type queuing c-in-q-default
    switch(config-pmap-c-que)# bandwidth percent 10
    switch(config-pmap-c-que)# 

    switch(config)# policy-map type queuing ld_40G_fcoe_in_que_policy
    switch(config-pmap-que)# class type queuing c-in-q1
    switch(config-pmap-c-que)# bandwidth percent 90
    switch(config-pmap-c-que)# pause buffer-size 1299584 pause-threshold 649792 resume-threshold 648128
    switch(config-pmap-c-que)# class type queuing c-in-q-default
    switch(config-pmap-c-que)# bandwidth percent 10
    switch(config-pmap-c-que)# 

Custom Policy for Ingress Buffer Size and Pause/Resume Thresholds

If sufficient buffers are not available to bring up long-distance ports, then fine tuning the buffers allocated to any 10G/25G Ethernet ports (with short distance requirements, that is, less than 100 meters) using default policies is required. If sufficient buffers are not available to bring up long-distance ports, a buffer allocation failure message appears. A sample buffer allocation failure message is as follows:

switch(config-if)# interface ethernet1/8
switch(config-if)# service-policy type queuing input ld_10G_fcoe_in_que_policy
switch(config-if)# no shutdown
2022 Oct 31 07:39:21 HW1 %$ VDC-1 %$ %ACLQOS-SLOT1-2-ACLQOS_FAILED: ACLQOS failure: Ingress buffer allocation failed for interface Ethernet1/8

Create a custom policy to free up the required buffers and apply it to the existing Ethernet ports or to Ethernet ports bound for VFC that is used for short distance connectivity.

switch(config)# policy-map type queuing 100m_fcoe_in_que_policy
switch(config-pmap-que)# class type queuing c-in-q1
switch(config-pmap-c-que)# bandwidth percent 50
switch(config-pmap-c-que)# pause buffer-size 41600 pause-threshold 20800 resume-threshold 19136
switch(config-pmap-c-que)# class type queuing c-in-q-default
switch(config-pmap-c-que)# bandwidth percent 50
switch(config-pmap-c-que)#

Required number of Ethernet Ports to Reduce Ingress Buffers on per Long Distance FCoE ISL

The following table displays the number of Ethernet ports needing an ingress buffer size reduction to accommodate a single long distance FCoE ISL of a given speed.

The following section includes configuration examples of policy applied to Ethernet interfaces for enabling 10G, 25G and 40G FCoE long distance ISLs.

switch(config)# interface ethernet 1/1     
switch(config-if)# service-policy type queuing input ld_10G_fcoe_in_que_policy 
switch(config-if)# 
 
switch(config)# interface ethernet 1/2     
switch(config-if)# service-policy type queuing input ld_25G_fcoe_in_que_policy 
switch(config-if)# 
 
switch(config)# interface ethernet 1/3     
switch(config-if)# service-policy type queuing input ld_40G_fcoe_in_que_policy 
switch(config-if)# 
 
switch(config)# interface ethernet 1/4     
switch(config-if)# service-policy type queuing input 100m_fcoe_in_que_policy 
switch(config-if)#