-
Beginning with Cisco NX-OS Release 9.3(3):
-
Total number of 16 GRE/IPIP tunnels are supported on Cisco Nexus 9200, 9300-EX/FX/FX2 switches, and 9500 switches with 9700-EX/FX
line cards.
-
Multiple IP-in-IP/GRE tunnel interfaces on a same Cisco Nexus Device can be sourced from or destined to the same IP Address
across different VRFs. This is supported on Cisco Nexus 9200 and 9300-EX/FX/FX2 platforms. This is not supported on Cisco
Nexus 9500 switches with 9300-GX line cards.
-
More than 1 and up to 16 IPIP Decap-any tunnels are supported - 1 decap-any tunnel per VRF. This is supported on Cisco Nexus
9200, and 9300-EX/FX/FX2 platforms.
-
VRF membership of the interface, where IPIP/GRE encapsulated packets are ingressing on the terminating node, should match
with the tunnel transport VRF for tunnel to correctly terminate the packets.
-
The IPIP/GRE packet coming on a non default VRF may get terminated by a tunnel in default VRF if the packet outer header matches
with the tunnel source and the tunnel destination.
-
Beginning with Cisco NX-OS Release 9.3(5), the following features are supported on N9K-C9316D-GX, N9K-C93600CD-GX and N9K-C9364C-GX
switches:
-
A total number of 16 GRE/IPIP tunnels.
-
Multiple IP-in-IP/GRE tunnel interfaces on a same Cisco Nexus device can be sourced from or destined to the same IP address
across different VRFs
-
More than 1 and upto 16 IPIP Decap-any tunnels are supported -- 1 decap-any tunnel per VRF.
-
Guidelines for source-direct and ipv6ipv6-decapsulate-any options for tunnels:
-
The source-direct command is supported on the Cisco Nexus 9500 platform switches with the Application Spine Engine (ASE) and the Leaf Spine
Engine (LSE).
Cisco Nexus 9500 platform switches with the Network Forwarding Engine (NFE) do not support the tunnel source direct command.
The tunnel source direct command with the tunnel mode ipv6ipv6 decapsulate-any command on the Cisco Nexus 9500 platform switches is only supported in the MPLS heavy routing template.
-
The IP tunnel supports the tunnel source CLI command with interface, IPv4 address, IPv6 address, or IPv4 prefix. You can configure IP-in-IP tunnel decapsulation on
directly connected IP addresses (for example, physical interface, port-channel, loopback, and SVI) using the new tunnel source direct CLI command. You can select the IP ECMP links when there are multiple IP links between the two switches. A single tunnel
interface can decapsulate the tunneled packets whose outer destination IP is any of the IPv4 or IPv6 address that is locally
configured and it is operationally Up in the switch.
-
Currently, tunnel mode ipip decapsulate-any is supported for decapsulating IPv4 payload over IPv4 transport (IPv4inIPv4 packets). tunnel mode ipv6ipv6 decapsulate-any command is introduced to support IPv6 payload over IPv6 transport (IPv6inIPv6 packets).
-
The tunnel source direct and tunnel mode ipv6ipv6 decapsulate-any CLI commands are not supported on Cisco Nexus 9500 platform switches with the Network Formation Engine (NFE).
-
The tunnel source direct CLI command is supported only when an administrator uses the IP-in-IP decapsulation to source route the packets through the
network. The source-direct tunnel is always operationally Up unless it is administratively shut down. The directly connected interfaces are identified using the show ip route direct CLI command.
-
The tunnel source direct CLI command is supported only on decapsulate-any tunnel modes, for example, tunnel mode ipip decapsulate-any and tunnel mode ipv6ipv6 decapsulate-any .
-
Auto-recovery for source-direct is not supported.
-
For ipv6ipv6 decapsulate-any, inter-VRF is not supported. The tunnel interface VRF (iVRF) and tunnel transport or forwarding
VRF (fVRF) must be the same. Only one decapsulate-any tunnel (irrespective of VRF) can be present in Cisco Nexus 9200, 9300-EX,
and 9300-FX platform switches; Cisco Nexus 9500 platform modular switches with EX and FX line cards.
-
To enable IPv6 on ipv6ipv6 decap-any tunnel interface, configure a valid IPv6 address or configure ipv6 address use-link-local-only under the tunnel interface.
-
See the following hardware limitations on the maximum sources that can be accommodated on a source direct tunnel and the related
behavior:
-
Source direct tunnel is now supported for Cisco Nexus 9000 Series switches with Network Forwarding Engine (NFE), Application
Spine Engine (ASE), and Leaf Spine Engine (LSE). Most of the limitations are only in case of scaled SIP (number of total IP/IPv6
addresses on the interfaces (L3, sub-interface, PC, PC-sub interfaces, loopback, SVI, and any secondary IP/IPv6 addresses.)
See the following sample use cases.
-
Use Case 1: Non-deterministic behavior of which SIP gets installed if the number of IP/IPv6 interface scale is more.
Both the switches have 512 entries for tunnel SIP. With tunnel source, direct any IP or IPv6 address w.r.t ipip or ipv6ipv6 decap any with tunnel source gets installed in the above table.
The insertion of these entries is on a first come first serve basis without any CLI command to control which interface IP
addresses get installed. If the system has more number of IP/IPv6 interfaces to be installed, the behavior is non-deterministic
(The behavior can change across reload with interface flaps.)
-
Use Case 2: The scale numbers are different in both switches and each has its own advantages and disadvantages.
IPv4 individual scale can be more (up to 512) in case of switches with NFE but it is shared with IPv6. In the switches with
ASE and LSE, the IPv4 individual scale can be 256 but it is not shared with IPv6.
Whenever the tunnel decap table gets filled, the TABLE_FULL error is displayed. If some entry gets deleted after the table
gets full, the table full error is cleared.
Table 1. Scale Numbers
|
Commands
|
Switches with NFE: Table size 512, v4 takes 1 entry, v6 takes 4 entries
|
Switches with ASE and LSE: Table size 512, v4 takes 1 entry, v6 takes 2 entries (paired index)
|
|
IPIP decap any with tunnel source direct
|
Shared between v4 and v6, v6 takes 4 entries
v4 + 4 *v6 =512
Maximum entries can be 512 with no v6 entries
|
Dedicated 256
|
|
IPv6IPv6 decap any with tunnel source direct
|
Shared between v4 and v6, v6 takes 4 entries
v4 + 4 *v6 =512
Maximum entries can be 128 with no v4 entries
|
Dedicated 128
|
-
Use Case 3: Auto-recovery is not supported.
If any entry does not get installed in the hardware due to exhaustion of above table, removal of an already installed IP/IPv6
from interfaces does not automatically trigger the addition of the failed SIP in the table though the table has space now.
You need to flap the tunnel interface or IP interface to get them installed.
However, if an entry does not get installed in the hardware due to a duplicate entry (if there was already a decap-any with one source present and now the source direct tunnel CLI command is configured, there is a duplicate entry for the prior source configured) that was taken care of by removing
the entry only when both the tunnels get deleted.
-
For Cisco Nexus 9000 Series switches with Network Forwarding Engine (NFE) and Application Spine Engine (ASE), the syslog is
different as the dedicated IPv4 and IPv6 decap entries are carved in the syslog. If the tunnel-decap-table is full, the user gets a syslog as follows:
2017 Apr 6 12:18:04 switch %$ VDC-1 %$ %IPFIB-2-FIB_HW_IPV4_TUNNEL_DECAP_TABLE_FULL: IPv4 tunnel decap hardware table full.
IP tunnel decapsulation may not work for some GRE/IPinIP traffic
2017 Apr 6 12:18:11 switch %$ VDC-1 %$ %IPFIB-2-FIB_HW_IPV6_TUNNEL_DECAP_TABLE_FULL: IPv6 tunnel decap hardware table full.
IP tunnel decapsulation may not work for some GRE/IPinIP traffic
If the table is full and once some entry gets deleted from the table (due to an interface being operationally down or removal
of IP address), the clear syslog for the table is displayed. Note that the deletion of the tunnel removes all the entries
that are added as part of that tunnel.
2017 Apr 5 13:29:25 switch %$ VDC-1 %$ %IPFIB-2-FIB_HW_IPV4_TUNNEL_DECAP_TABLE_FULL_CLRD: IPv4 tunnel decap hardware table full exception cleared
2017 Apr 4 19:41:22 switch %$ VDC-1 %$ %IPFIB-2-FIB_HW_IPV6_TUNNEL_DECAP_TABLE_FULL_CLRD: IPv6 tunnel decap hardware table full exception cleared
-
IP-in-IP tunnel decapsulation is supported on IPv6 enabled networks.
!
interface tunnel 1
ipv6 address use-link-local-only <<< enable IPv6
tunnel mode ipv6ipv6 decapsulate-any
tunnel source direct
description IPinIP Decapsulation Interface
mtu 1476
no shutdown
-
The show commands with the internal keyword are not supported.
-
Cisco NX-OS supports only the following protocols:
-
IPv4 passenger protocol.
-
GRE carrier protocol.
-
Cisco NX-OS supports the following maximum number of tunnels prior to Cisco NX-OS Release 9.3(3):
-
Beginning with Cisco NX-OS Release 9.3(3), the maximum number of supported GRE and IP-in-IP regular tunnels is 16.
-
IP tunnels do not support access control lists (ACLs) or QoS policies.
-
Cisco NX-OS supports the GRE header defined in IETF RFC 2784. Cisco NX-OS does not support tunnel keys and other options from
IETF RFC 1701.
-
Cisco NX-OS does not support GRE tunnel keepalives.
-
All unicast routing protocols are supported by IP tunnels.
-
The IP tunnel interface cannot be configured to be a span source or destination.
-
IP tunnels do not support PIM or other Multicast features and protocols.
-
The selection of GRE or IP-in-IP tunnel destination based on the PBR policy is not supported.
-
IP tunnels are supported only in the default system routing mode and not in other modes.
-
When configuring a tunnel interface to ipip mode , the maximum mtu value is9196.
When downgrading from NX-OS 9.2(1)to an earlier release, with a tunnel interface in ipip mode that has an mtu value of 9196, the mtu configuration is lost as a result of the downgrade operation. As a best practice,
adjust the mtu value to 9192 before commencing the downgrade to avoid losing the mtu configuration.
-
When configuring a tunnel interface to ipip mode , the default mtu value is1480.
When downgrading from NX-OS 9.2(1) or later release to an earlier release, with a tunnel interface in ipip mode with no explicit mtu configuration, the mtu value changes as a result of the downgrade operation from 1480 to 1476. As a
best practice, adjust the mtu value to 1476 before commencing the downgrade to avoid any changes to the mtu value.
When upgrading to NX-OS 9.2(1) or later release,, with a tunnel interface in ipip mode with no explicit mtu configuration, the mtu value changes as a result of the upgrade operation from 1476 to 1480. As a best
practice, adjust the mtu value to 1480 before commencing the upgrade to avoid any changes to the mtu value.
-
On Cisco Nexus 9200 Series switches, GRE packets that are received on an IP-in-IP tunnel are not dropped as expected and are
instead forwarded to the packet destination.
-
Tx packets originating from the switch, such as control pkts, are not included in Tx statistics.
-
Tunnel destinations that are reachable over another tunnel are not supported.
-
The consistency checker is not supported for routes over a tunnel.
-
Non-IP routing protocols, such as isis, are not supported over IP-in-IP tunnels.
-
RFC5549 is not be supported over tunnels.
-
BGP adjacency over tunnel is not supported in a scenario where the tunnel interface and tunnel source are in same VRF (example:
VRF-A) and tunnel destination is reachable with route-leak from opposite end (example: via VRF-B)
-
You can configure only 8 GRE tunnels per device.
-
GRE tunnels does not support RACLs.
-
GRE tunnel must belong to the same VRF as the underlying routing infrastructure. In other words, the tunnel use-vrf and vrf member values must always match on the same GRE tunnel.
-
GRE tunnels supports only limited traffic (ingress or egress) counters.
-
Layer 3 FEX interfaces not are allowed as tunnel source and/or destination
-
Double encapsulation is not allowed on GRE tunnels.
-
BFD is not supported on GRE tunnels.
-
On Cisco Nexus N9K-C9300-GX platforms, GRE/IPinIP tunnel interfaces cannot co-exist with Dot1Q tagged L2 bcast or 1Q tagged
L2/L3 mcast transit traffic. When you configure feature tunnel on Cisco Nexus N9300-GX platform, the following warning is displayed and you get a syslog message warning you. You should
not configure feature tunnel if you have Dot1Q tagged L2 bcast or 1Q tagged L2/L3 mcast transit traffic on the device.
N9300-GX(config)# feature tunnel
WARN:GRE/IPinIP cannot coexist with 1Q tagged L2 bcast or 1Q tagged L2/L3 mcast transit packets on this
platform
N9300-GX(config)#
N9300-GX(config)# show logging logfile
2019 Dec 12 00:41:08 N9300-GX %TUNNEL-2-TRAFFIC_WARNING: GRE/IPinIP cannot coexist with 1Q
tagged L2 bcast or 1Q tagged L2/L3 mcast transit packets on this platform
N9300-GX(config)#
-
The feature feature tunnel on the Cisco Nexus 9000 switches cannot co-exist with the VXLAN feature, feature nv overlay.
-
Cisco Nexus 9200, 9300-EX, 9300-FX, 9300-FX2 series switches and Cisco Nexus 9500 platform switches with 9700-EX/FX line cards
may not have multiple tunnel interfaces in a single VRF that are sourced from or destined to the same IP address. For example,
a device may not have tunnel 0 and tunnel 1 interfaces in the default VRF that are sourced from the same IP address or interface.
-
Cisco Nexus 9300-EX, 9300-FX, 9300-GX and Nexus 9500 platform switches in vPC can act as GRE Tunnel endpoints for their respective
tunnels. However, the tunnel destination can not be through a vPC.
Feedback