方形拓扑中采用CloudSec的多站点VXLAN故障排除
简介
本文档介绍以方形拓扑连接的边界网关之间的VXLAN多站点配置和CloudSec故障排除。
先决条件
要求
思科建议您熟悉以下主题：
- Nexus NXOS ©软件。
- VXLAN EVPN技术。
- BGP和OSPF路由协议。
使用的组件
本文档中的信息基于以下软件和硬件版本：
- Cisco Nexus 9000。
- NXOS 10.3(4a)版。
本文档中的信息都是基于特定实验室环境中的设备编写的。本文档中使用的所有设备最初均采用原始（默认）配置。如果您的网络处于活动状态，请确保您了解所有命令的潜在影响。
配置
网络图
方形拓扑中采用CloudSec的VXLAN MultiSite
拓扑详细信息
- 两个站点多站点VXLAN EVPN交换矩阵。
- 两个站点都配置了vPC边界网关。
- 终端托管在VLAN 1100中。
- 每个站点上的边界网关通过SVI接口Vlan3600彼此之间具有IPv4 iBGP邻居关系。
- 一个站点上的边界网关仅与另一个站点上的直接连接的边界网关具有eBGP IPv4邻居关系。
- 站点A的边界网关与站点B的边界网关具有eBGP L2VPN EVPN邻居关系。
寻址计划
表中的IP地址在配置期间使用：
| 站点 A | 站点 B | |||||||||||
| 设备角色 | 接口 ID | 物理接口IP | RID环路IP | NVE环路IP | MSITE-VIP | 备份SVI IP | 接口 ID | 物理接口IP | RID环路IP | NVE环路IP | MSITE-VIP | 备份SVI IP |
| 枝叶 | Eth1/52 | 192.168.1.1/30 | 192.168.2.1/32 | 192.168.3.1/32 | 不适用 | 不适用 | Eth1/36 | 192.168.11.1/30 | 192.168.12.1/32 | 192.168.13.1/32 | 不适用 | 不适用 |
| 主干 | Eth1/52 | 192.168.1.2/30 | 不适用 | Eth1/36 | 192.168.11.2/30 | 不适用 | ||||||
| Eth1/50 | 192.168.1.5/30 | 192.168.2.2/32 | 不适用 | 不适用 | 不适用 | Eth1/34 | 192.168.11.5/30 | 192.168.12.2/32 | 不适用 | 不适用 | 不适用 | |
| Eth1/51 | 192.168.1.9/30 | 不适用 | Eth1/35 | 192.168.11.9/30 | 不适用 | |||||||
| BGW-A | Eth1/51 | 192.168.1.6/30 | 192.168.2.3/32 | 192.168.3.2/32 | 192.168.100.1/32 | 192.168.4.1/30 | Eth1/4 | 192.168.11.6/30 | 192.168.12.3/32 | 192.168.13.2/32 | 192.168.200.1/32 | 192.168.14.1/30 |
| Eth1/48 | 10.12.10.1/30 | 192.168.3.254/32 | Eth1/3 | 10.12.10.2/30 | 192.168.13.254/32 | |||||||
| BGW-B | Eth1/51 | 192.168.1.10/30 | 192.168.2.4/32 | 192.168.3.3/32 | 192.168.100.1/32 | 192.168.4.2/30 | Eth1/4 | 192.168.11.10/30 | 192.168.12.4/32 | 192.168.13.3/32 | 192.168.200.1/32 | 192.168.14.2/30 |
| Eth1/48 | 10.12.10.5/30 | 192.168.3.254/32 | Eth1/3 | 10.12.10.6/30 | 192.168.13.254/32 |
配置
- 请注意，本指南中仅显示与多站点相关的配置。对于完整配置，您可以使用Cisco VXLAN官方文档指南Cisco Nexus 9000系列NX-OS VXLAN配置指南，版本10.3(x)
要启用CloudSec，dci-advertise-pip 命令必须配置在evpn多站点边界网关下：
| SA-BGW-A和SA-BGW-B | SB-BGW-A和SB-BGW-B |
|
|
BGP配置
此配置特定于站点。
| SA-BGW-A和SA-BGW-B | SB-BGW-A和SB-BGW-B |
|
|
- maximum-path命令允许从邻居接收多个eBGP L2VPN EVPN路径。
- additional-path命令指示BGP进程通告设备能够发送/接收其他路径
对于边界网关上的所有L3VNI VRF，还必须配置多路径：
| SA-BGW-A和SA-BGW-B | SB-BGW-A和SB-BGW-B |
|
|
隧道加密配置
所有边界网关上的此配置必须相同：
key chain CloudSec_Key_Chain1 tunnel-encryption
key 1000
key-octet-string Cl0udSec! cryptographic-algorithm AES_128_CMAC
feature tunnel-encryption
tunnel-encryption must-secure-policy
tunnel-encryption source-interface loopback0
tunnel-encryption policy CloudSec_Policy1此配置特定于站点。命令tunnel-encryption，只能应用于具有命令的接evpn multisite dci-tracking口。
| SA-BGW-A和SA-BGW-B | SB-BGW-A和SB-BGW-B |
|
|
启用隧道加密后，在向邻居通告路由时会将其他属性添加到本地环回，并且所有eBGP IPv4单播邻居必须看到此属性：
SA-BGW-A# show ip bgp 192.168.2.3
BGP routing table information for VRF default, address family IPv4 Unicast
BGP routing table entry for 192.168.2.3/32, version 1320
Paths: (2 available, best #1)
Flags: (0x000002) (high32 00000000) on xmit-list, is not in urib
Multipath: eBGP iBGP
Advertised path-id 1
Path type: local, path is valid, is best path, no labeled nexthop
AS-Path: NONE, path locally originated
0.0.0.0 (metric 0) from 0.0.0.0 (192.168.2.3)
Origin IGP, MED not set, localpref 100, weight 32768
Tunnel Encapsulation attribute: Length 152 !---This is a new attribute
Path type: redist, path is valid, not best reason: Locally originated, no labeled nexthop
AS-Path: NONE, path locally originated
0.0.0.0 (metric 0) from 0.0.0.0 (192.168.2.3)
Origin incomplete, MED 0, localpref 100, weight 32768
Path-id 1 advertised to peers:
10.12.10.2 192.168.4.2
SA-BGW-A# 对于路由类型2，还有新属性：
SA-BGW-A# show bgp l2vpn evpn 00ea.bd27.86ef
BGP routing table information for VRF default, address family L2VPN EVPN
Route Distinguisher: 65002:31100
BGP routing table entry for [2]:[0]:[0]:[48]:[00ea.bd27.86ef]:[0]:[0.0.0.0]/216, version 7092
Paths: (2 available, best #2)
Flags: (0x000202) (high32 00000000) on xmit-list, is not in l2rib/evpn, is not in HW
Multipath: eBGP iBGP
Path type: external, path is valid, not best reason: Router Id, multipath, no labeled nexthop
Imported to 1 destination(s)
Imported paths list: L2-31100
AS-Path: 65002 , path sourced external to AS
192.168.13.3 (metric 0) from 192.168.12.4 (192.168.12.4)
Origin IGP, MED 2000, localpref 100, weight 0
Received label 31100
Received path-id 1
Extcommunity: RT:65001:31100 ENCAP:8
ESI: 0300.0000.00fd.ea00.0309
Advertised path-id 1
Path type: external, path is valid, is best path, no labeled nexthop
Imported to 1 destination(s)
Imported paths list: L2-31100
AS-Path: 65002 , path sourced external to AS
192.168.13.2 (metric 0) from 192.168.12.3 (192.168.12.3)
Origin IGP, MED 2000, localpref 100, weight 0
Received label 31100
Received path-id 1
Extcommunity: RT:65001:31100 ENCAP:8
ESI: 0300.0000.00fd.ea00.0309
Path-id 1 not advertised to any peer
Route Distinguisher: 192.168.2.3:33867 (L2VNI 31100)
BGP routing table entry for [2]:[0]:[0]:[48]:[00ea.bd27.86ef]:[0]:[0.0.0.0]/216, version 7112
Paths: (2 available, best #1)
Flags: (0x000212) (high32 0x000400) on xmit-list, is in l2rib/evpn, is not in HW
Multipath: eBGP iBGP
Advertised path-id 1
Path type: external, path is valid, is best path, no labeled nexthop, in rib
Imported from 65002:31100:[2]:[0]:[0]:[48]:[00ea.bd27.86ef]:[0]:[0.0.0.0]/216
AS-Path: 65002 , path sourced external to AS
192.168.13.2 (metric 0) from 192.168.12.3 (192.168.12.3)
Origin IGP, MED 2000, localpref 100, weight 0
Received label 31100
Received path-id 1
Extcommunity: RT:65001:31100 ENCAP:8
ESI: 0300.0000.00fd.ea00.0309
Path type: external, path is valid, not best reason: Router Id, multipath, no labeled nexthop, in rib
Imported from 65002:31100:[2]:[0]:[0]:[48]:[00ea.bd27.86ef]:[0]:[0.0.0.0]/216
AS-Path: 65002 , path sourced external to AS
192.168.13.3 (metric 0) from 192.168.12.4 (192.168.12.4)
Origin IGP, MED 2000, localpref 100, weight 0
Received label 31100
Received path-id 1
Extcommunity: RT:65001:31100 ENCAP:8
ESI: 0300.0000.00fd.ea00.0309 !---Ethernet Segment Identifier (ESI) is also new attribute
Path-id 1 (dual) advertised to peers:
192.168.2.2
SA-BGW-A# 验证
启用cloudsec之前，最好检查设置是否运行良好，如果没有它：
SA-BGW-A(config)# show clock
Warning: No NTP peer/server configured. Time may be out of sync.
10:02:01.016 UTC Fri Jul 19 2024
Time source is NTP
SA-BGW-A(config)# show tunnel-encryption session
Tunnel-Encryption Peer Policy Keychain RxStatus TxStatus
------------------------ ---------------------------------------- ---------------------------------------- ----------------- -----------------
===============================================
SA-HOST-A# show clock
Warning: No NTP peer/server configured. Time may be out of sync.
10:02:21.592 UTC Fri Jul 19 2024
Time source is NTP
SA-HOST-A# ping 10.100.20.10 count unlimited interval 1
PING 10.100.20.10 (10.100.20.10): 56 data bytes
64 bytes from 10.100.20.10: icmp_seq=0 ttl=254 time=1.583 ms
64 bytes from 10.100.20.10: icmp_seq=1 ttl=254 time=10.407 ms
64 bytes from 10.100.20.10: icmp_seq=2 ttl=254 time=1.37 ms
64 bytes from 10.100.20.10: icmp_seq=3 ttl=254 time=1.489 ms
64 bytes from 10.100.20.10: icmp_seq=4 ttl=254 time=6.685 ms
64 bytes from 10.100.20.10: icmp_seq=5 ttl=254 time=1.547 ms
64 bytes from 10.100.20.10: icmp_seq=6 ttl=254 time=1.859 ms
64 bytes from 10.100.20.10: icmp_seq=7 ttl=254 time=5.219 ms
64 bytes from 10.100.20.10: icmp_seq=8 ttl=254 time=1.337 ms
64 bytes from 10.100.20.10: icmp_seq=9 ttl=254 time=3.528 ms
64 bytes from 10.100.20.10: icmp_seq=10 ttl=254 time=4.057 ms在cloudsec配置后，SA上的终端必须成功ping站点B上的终端。但在某些情况下，ping操作可能不成功。这取决于本地设备选择哪个cloudsec对等体来发送cloudsec加密流量。
SA-HOST-A# ping 10.100.20.10
PING 10.100.20.10 (10.100.20.10): 56 data bytes
Request 0 timed out
Request 1 timed out
Request 2 timed out
Request 3 timed out
Request 4 timed out
--- 10.100.20.10 ping statistics ---
5 packets transmitted, 0 packets received, 100.00% packet loss
SA-HOST-A# 故障排除
检查源终端上的本地ARP表：
SA-HOST-A# ping 10.100.20.10 count unlimited interval 1
Request 352 timed out
Request 353 timed out
Request 354 timed out
356 packets transmitted, 0 packets received, 100.00% packet loss
SA-HOST-A# clear ip arp delete-force
SA-HOST-A# show ip arp
Flags: * - Adjacencies learnt on non-active FHRP router
+ - Adjacencies synced via CFSoE
# - Adjacencies Throttled for Glean
CP - Added via L2RIB, Control plane Adjacencies
PS - Added via L2RIB, Peer Sync
RO - Re-Originated Peer Sync Entry
D - Static Adjacencies attached to down interface
IP ARP Table for context default
Total number of entries: 1
Address Age MAC Address Interface Flags
10.100.20.10 00:00:02 00ea.bd27.86ef Vlan1100
SA-HOST-A# 此输出证明，BUM流量正在通过，而控制平面正在工作。下一步是检查隧道加密状态：
SA-BGW-A# show tunnel-encryption session
Tunnel-Encryption Peer Policy Keychain RxStatus TxStatus
------------------------ ---------------------------------------- ---------------------------------------- ----------------- -----------------
192.168.13.2 CloudSec_Policy1 CloudSec_Key_Chain1 Secure (AN: 0) Secure (AN: 0)
192.168.13.3 CloudSec_Policy1 CloudSec_Key_Chain1 Secure (AN: 0) Secure (AN: 0)
SA-BGW-A# 此输出显示CloudSec会话已建立。下一步可以在SA-HOST-A上运行无限制的ping:
SA-HOST-A# ping 10.100.20.10 count unlimited interval 1从此，您必须检查站点A上的设备，查看流量是否到达此设备。您可以通过ELAM在站点A路径上的所有设备上完成此任务。将默认值从6更改为9可以基于内部报头进行匹配in-select 。您可在此链接上了解有关ELAM的更多信息：Nexus 9000云扩展ASIC(Tahoe)NX-OS ELAM。
SA-LEAF-A上的ELAM
在生产网络中存在多个SPINE设备。要了解流量发送到哪个主干，您必须首先在LEAF上使用ELAM。尽管使用in-select 9了此功能，但在连接到源的LEAF上，必须使用外部ipv4报头，因为到达此LEAF的流量未进行VXLAN加密。在真实网络中，很难捕获准确生成的数据包。在这种情况下，您可以运行具有特定长度的ping，并使用Pkt len报头来标识数据包。默认情况下，icmp数据包的长度为64字节。加上20字节的IP报头，总结起来为您提供84字节的PKT Len:
SA-LEAF-A# debug platform internal tah elam
SA-LEAF-A(TAH-elam)# trigger init in-select 9
Slot 1: param values: start asic 0, start slice 0, lu-a2d 1, in-select 9, out-select 0
SA-LEAF-A(TAH-elam-insel9)# set outer ipv4 src_ip 10.100.10.10 dst_ip 10.100.20.10
SA-LEAF-A(TAH-elam-insel9)# start
SA-LEAF-A(TAH-elam-insel9)# report
ELAM not triggered yet on slot - 1, asic - 0, slice - 0
SUGARBOWL ELAM REPORT SUMMARY
slot - 1, asic - 0, slice - 1
============================
Incoming Interface: Eth1/49
Src Idx : 0xc1, Src BD : 1100
Outgoing Interface Info: dmod 1, dpid 64 !---Note dpid value
Dst Idx : 0xcd, Dst BD : 1100
Packet Type: IPv4
Outer Dst IPv4 address: 10.100.20.10
Outer Src IPv4 address: 10.100.10.10
Ver = 4, DSCP = 0, Don't Fragment = 0
Proto = 1, TTL = 255, More Fragments = 0
Hdr len = 20, Pkt len = 84, Checksum = 0xb4ae !---64 byte + 20 byte IP header Pkt len = 84
Inner Payload
Type: CE
L4 Protocol : 1
L4 info not available
Drop Info:
----------
LUA:
LUB:
LUC:
LUD:
Final Drops:
SA-LEAF-A(TAH-elam-insel9)# show system internal ethpm info all | i i "dpid=64" !---Put dpid value here
IF_STATIC_INFO: port_name=Ethernet1/52,if_index:0x1a006600,ltl=5940,slot=0, nxos_port=204,dmod=1,dpid=64,unit=0,queue=65535,xbar_unitbmp=0x0,ns_pid=255,slice_num=1,port_on_slice=24,src_id=48
SA-LEAF-A(TAH-elam-insel9)# show cdp neighbors interface ethernet 1/52
Capability Codes: R - Router, T - Trans-Bridge, B - Source-Route-Bridge
S - Switch, H - Host, I - IGMP, r - Repeater,
V - VoIP-Phone, D - Remotely-Managed-Device,
s - Supports-STP-Dispute
Device-ID Local Intrfce Hldtme Capability Platform Port ID
SA-SPINE-A(FDO242210CS)
Eth1/52 130 R S s N9K-C93240YC-FX2 Eth1/52
Total entries displayed: 1
SA-LEAF-A(TAH-elam-insel9)# 从此输出中，您可以看到流量到达SA-LEAF-A并从接口Ethernet1/52转发出去，该接口从拓扑连接到SA-SPINE-A。
SA-SPINE-A上的ELAM
在SPINE上，Pkt Len值将更多，因为50字节的VXLAN报头也会添加。默认情况下，SPINE无法匹配没有或的内部vxlan-parse标feature nv overlay头。因此，必须在SPINEvxlan-parse enable上使用命令：
SA-SPINE-A(config-if)# debug platform internal tah elam
SA-SPINE-A(TAH-elam)# trigger init in-select 9
Slot 1: param values: start asic 0, start slice 0, lu-a2d 1, in-select 9, out-select 0
SA-SPINE-A(TAH-elam)# vxlan-parse enable
SA-SPINE-A(TAH-elam-insel9)# set inner ipv4 src_ip 10.100.10.10 dst_ip 10.100.20.10
SA-SPINE-A(TAH-elam-insel9)# start
SA-SPINE-A(TAH-elam-insel9)# report
ELAM not triggered yet on slot - 1, asic - 0, slice - 0
HEAVENLY ELAM REPORT SUMMARY
slot - 1, asic - 0, slice - 1
============================
Incoming Interface: Eth1/52
Src Idx : 0xcd, Src BD : 4153
Outgoing Interface Info: dmod 1, dpid 72
Dst Idx : 0xc5, Dst BD : 4151
Packet Type: IPv4
Outer Dst IPv4 address: 192.168.100.1
Outer Src IPv4 address: 192.168.3.1
Ver = 4, DSCP = 0, Don't Fragment = 0
Proto = 17, TTL = 255, More Fragments = 0
Hdr len = 20, Pkt len = 134, Checksum = 0x7d69 !---84 bytes + 50 bytes VXLAN header Pkt len = 134
Inner Payload
Type: IPv4
Inner Dst IPv4 address: 10.100.20.10
Inner Src IPv4 address: 10.100.10.10
L4 Protocol : 17
L4 info not available
Drop Info:
----------
LUA:
LUB:
LUC:
LUD:
Final Drops:
SA-SPINE-A(TAH-elam-insel9)# show system internal ethpm info all | i i "dpid=72"
IF_STATIC_INFO: port_name=Ethernet1/50,if_index:0x1a006200,ltl=5948,slot=0, nxos_port=196,dmod=1,dpid=72,unit=0,queue=65535,xbar_unitbmp=0x0,ns_pid=255,slice_num=1,port_on_slice=0,src_id=0
SA-SPINE-A(TAH-elam-insel9)# show cdp neighbors interface ethernet 1/50
Capability Codes: R - Router, T - Trans-Bridge, B - Source-Route-Bridge
S - Switch, H - Host, I - IGMP, r - Repeater,
V - VoIP-Phone, D - Remotely-Managed-Device,
s - Supports-STP-Dispute
Device-ID Local Intrfce Hldtme Capability Platform Port ID
SA-BGW-A(FDO242210CX)
Eth1/50 169 R S s N9K-C93240YC-FX2 Eth1/50
Total entries displayed: 1
SA-SPINE-A(TAH-elam-insel9)# SA-SPINE-A根据输出向SA-BGW-A发送流量。
SA-BGW-A上的ELAM
SA-BGW-A(TAH-elam-insel9)# set inner ipv4 src_ip 10.100.10.10 dst_ip 10.100.20.10
SA-BGW-A(TAH-elam-insel9)# start
SA-BGW-A(TAH-elam-insel9)# report
ELAM not triggered yet on slot - 1, asic - 0, slice - 0
HEAVENLY ELAM REPORT SUMMARY
slot - 1, asic - 0, slice - 1
============================
Incoming Interface: Eth1/50
Src Idx : 0xc5, Src BD : 1100
Outgoing Interface Info: dmod 1, dpid 48
Dst Idx : 0xbd, Dst BD : 1100
Packet Type: IPv4
Outer Dst IPv4 address: 192.168.100.1
Outer Src IPv4 address: 192.168.3.1
Ver = 4, DSCP = 0, Don't Fragment = 0
Proto = 17, TTL = 254, More Fragments = 0
Hdr len = 20, Pkt len = 134, Checksum = 0x7e69
Inner Payload
Type: IPv4
Inner Dst IPv4 address: 10.100.20.10
Inner Src IPv4 address: 10.100.10.10
L4 Protocol : 17
L4 info not available
Drop Info:
----------
LUA:
LUB:
LUC:
LUD:
Final Drops:
SA-BGW-A(TAH-elam-insel9)# show system internal ethpm info all | i i "dpid=48"
IF_STATIC_INFO: port_name=Ethernet1/48,if_index:0x1a005e00,ltl=5956,slot=0, nxos_port=188,dmod=1,dpid=48,unit=0,queue=65535,xbar_unitbmp=0x0,ns_pid=255,slice_num=0,port_on_slice=48,src_id=96
SA-BGW-A(TAH-elam-insel9)# show cdp neighbors interface ethernet 1/48
Capability Codes: R - Router, T - Trans-Bridge, B - Source-Route-Bridge
S - Switch, H - Host, I - IGMP, r - Repeater,
V - VoIP-Phone, D - Remotely-Managed-Device,
s - Supports-STP-Dispute
Device-ID Local Intrfce Hldtme Capability Platform Port ID
SB-BGW-A(FDO2452070B)
Eth1/48 122 R S s N9K-C93216TC-FX2 Eth1/3
Total entries displayed: 1
SA-BGW-A(TAH-elam-insel9)# 根据SA-BGW-A的输出，流量从Ethernet1/48流向SB-BGW-A。下一步是检查SB-BGW-A:
SB-BGW-A# debug platform internal tah elam
SB-BGW-A(TAH-elam)# trigger init in-select 9
Slot 1: param values: start asic 0, start slice 0, lu-a2d 1, in-select 9, out-select 0
SB-BGW-A(TAH-elam-insel9)# set inner ipv4 src_ip 10.100.10.10 dst_ip 10.100.20.10
SB-BGW-A(TAH-elam-insel9)# start
SB-BGW-A(TAH-elam-insel9)# report
ELAM not triggered yet on slot - 1, asic - 0, slice - 0
ELAM not triggered yet on slot - 1, asic - 0, slice - 1
!---Reset the previous filter and start again just in case if packet was not captured.
SB-BGW-A(TAH-elam-insel9)# reset
SB-BGW-A(TAH-elam-insel9)# set inner ipv4 src_ip 10.100.10.10 dst_ip 10.100.20.10
SB-BGW-A(TAH-elam-insel9)# start
SB-BGW-A(TAH-elam-insel9)# report
ELAM not triggered yet on slot - 1, asic - 0, slice - 0
ELAM not triggered yet on slot - 1, asic - 0, slice - 1
SB-BGW-A(TAH-elam-insel9)# 根据SB-BGW-A的输出，ELAM甚至没有被触发。这意味着SB-BGW-B正在接收数据包，无法正确解密和解析它们，或者根本无法接收它们。要了解cloudsec流量的情况，可以再次在SB-BGW-A上运行ELAM，但必须将触发过滤器设置为用于cloudsec的外部IP地址，因为无法查看cloudsec加密传输数据包的内部报头。从之前的输出中，您知道SA-BGW-A处理了流量，这意味着SA-BGW-A使用cloudsec加密流量。因此，您可以使用SA-BGW-A的NVE IP作为ELAM的触发过滤器。从上述输出中，VXLAN加密ICMP数据包长度为134字节。加上摘要中的32字节的cloudsec报头可提供166个字节：
SB-BGW-A(TAH-elam-insel9)# reset SB-BGW-A(TAH-elam-insel9)# set outer ipv4 src_ip 192.168.3.2 SB-BGW-A(TAH-elam-insel9)# start SB-BGW-A(TAH-elam-insel9)# report ELAM not triggered yet on slot - 1, asic - 0, slice - 0 HEAVENLY ELAM REPORT SUMMARY slot - 1, asic - 0, slice - 1 ============================ Incoming Interface: Eth1/3 Src Idx : 0x9, Src BD : 4108 Outgoing Interface Info: dmod 1, dpid 130 Dst Idx : 0xd, Dst BD : 4109 Packet Type: IPv4 Outer Dst IPv4 address: 192.168.13.3 !---NVE IP address of SB-BGW-B Outer Src IPv4 address: 192.168.3.2 Ver = 4, DSCP = 0, Don't Fragment = 0 Proto = 17, TTL = 254, More Fragments = 0 Hdr len = 20, Pkt len = 166, Checksum = 0xd546 !---134 byte VXLAN packet + 32 byte cloudsec header Pkt len = 166 Inner Payload Type: CE L4 Protocol : 17 L4 info not available Drop Info: ---------- LUA: LUB: LUC: LUD: Final Drops: !---To reach SB-BGW-B NVE IP traffic was sent out of Ethernet1/4 which is connected to SB-SPINE-A SB-BGW-A(TAH-elam-insel9)# show system internal ethpm info all | i i "dpid=130" IF_STATIC_INFO: port_name=Ethernet1/4,if_index:0x1a000600,ltl=6132,slot=0, nxos_port=12,dmod=1,dpid=130,unit=0,queue=65535,xbar_unitbmp=0x0,ns_pid=255,slice_num=1,port_on_slice=58,src_id=116SB-BGW-A(TAH-elam-insel9)# show cdp neighbors interface ethernet 1/4 Capability Codes: R - Router, T - Trans-Bridge, B - Source-Route-Bridge S - Switch, H - Host, I - IGMP, r - Repeater, V - VoIP-Phone, D - Remotely-Managed-Device, s - Supports-STP-Dispute Device-ID Local Intrfce Hldtme Capability Platform Port ID SB-SPINE-A(FDO22302CJ0) Eth1/4 131 R S s N9K-C9236C Eth1/34 Total entries displayed: 1 SB-BGW-A(TAH-elam-insel9)# show ip route 192.168.13.3 IP Route Table for VRF "default" '*' denotes best ucast next-hop '**' denotes best mcast next-hop '[x/y]' denotes [preference/metric] '%' in via output denotes VRF 192.168.13.3/32, ubest/mbest: 1/0 *via 192.168.11.5, Eth1/4, [110/6], 00:56:13, ospf-UNDERLAY, intra via 192.168.14.2, [200/0], 01:13:46, bgp-65002, internal, tag 65002 !---The device still have a route for SB-BGW-B NVE IP via SVI
SB-BGW-A(TAH-elam-insel9)# show ip route 192.168.14.2 IP Route Table for VRF "default" '*' denotes best ucast next-hop '**' denotes best mcast next-hop '[x/y]' denotes [preference/metric] '%' in via output denotes VRF 192.168.14.2/32, ubest/mbest: 1/0, attached *via 192.168.14.2, Vlan3600, [250/0], 01:15:05, am SB-BGW-A(TAH-elam-insel9)# show ip arp 192.168.14.2 Flags: * - Adjacencies learnt on non-active FHRP router + - Adjacencies synced via CFSoE # - Adjacencies Throttled for Glean CP - Added via L2RIB, Control plane Adjacencies PS - Added via L2RIB, Peer Sync RO - Re-Originated Peer Sync Entry D - Static Adjacencies attached to down interface IP ARP Table Total number of entries: 1 Address Age MAC Address Interface Flags 192.168.14.2 00:00:13 ecce.1324.c803 Vlan3600 SB-BGW-A(TAH-elam-insel9)# show mac address-table address ecce.1324.c803 Legend: * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC age - seconds since last seen,+ - primary entry using vPC Peer-Link, (T) - True, (F) - False, C - ControlPlane MAC, ~ - vsan, (NA)- Not Applicable VLAN MAC Address Type age Secure NTFY Ports ---------+-----------------+--------+---------+------+----+------------------ G 3600 ecce.1324.c803 static - F F vPC Peer-Link(R) SB-BGW-A(TAH-elam-insel9)#
从此输出中，您可以看到，根据路由表，cloudsec流量通过以太网接口1/4转发到SB-BGW-B。根据Cisco Nexus 9000系列NX-OS VXLAN配置指南，版本10.3(x)的指南和限制：
-
流向交换机的CloudSec流量必须通过DCI上行链路进入交换机。
根据同一指南的vPC Border Gateway Support for Cloudsec部分，如果vPC BGW获取对等vPC BGW的PIP地址并在DCI端通告，则来自两个vPC BGW的BGP路径属性将相同。因此，DCI中间节点最终可以从不拥有PIP地址的vPC BGW中选择路径。在此方案中，MCT链路用于来自远程站点的加密流量。但在本例中，使用指向SPINE的接口，尽管如此，BGW也通过BackUp SVI具有OSPF邻接关系。
SB-BGW-A(TAH-elam-insel9)# show ip ospf neighbors
OSPF Process ID UNDERLAY VRF default
Total number of neighbors: 2
Neighbor ID Pri State Up Time Address Interface
192.168.12.4 1 FULL/ - 01:33:11 192.168.14.2 Vlan3600
192.168.12.2 1 FULL/ - 01:33:12 192.168.11.5 Eth1/4
SB-BGW-A(TAH-elam-insel9)#问题的原因和修复
原因是SVI接口的OSPF开销。默认情况下，在NXOS上，自动成本参考带宽为40G。SVI接口的带宽为1Gbps，而物理接口的带宽为10Gbps:
SB-BGW-A(TAH-elam-insel9)# show ip ospf interface brief
OSPF Process ID UNDERLAY VRF default
Total number of interface: 5
Interface ID Area Cost State Neighbors Status
Vlan3600 3 0.0.0.0 40 P2P 1 up
Eth1/4 5 0.0.0.0 1 P2P 1 up 在这种情况下，对SVI成本进行管理更改可以解决此问题。必须在所有边界网关上完成调整。
SB-BGW-A(config)# int vlan 3600 SB-BGW-A(config-if)# ip ospf cost 1 SB-BGW-A(config-if)# sh ip route 192.168.13.3 IP Route Table for VRF "default" '*' denotes best ucast next-hop '**' denotes best mcast next-hop '[x/y]' denotes [preference/metric] '%' in via output denotes VRF 192.168.13.3/32, ubest/mbest: 1/0 *via 192.168.14.2, Vlan3600, [110/2], 00:00:08, ospf-UNDERLAY, intra via 192.168.14.2, [200/0], 01:34:07, bgp-65002, internal, tag 65002 SB-BGW-A(config-if)#
!---The ping is started to work immediatelyRequest 1204 timed out Request 1205 timed out Request 1206 timed out 64 bytes from 10.100.20.10: icmp_seq=1207 ttl=254 time=1.476 ms 64 bytes from 10.100.20.10: icmp_seq=1208 ttl=254 time=5.371 ms 64 bytes from 10.100.20.10: icmp_seq=1209 ttl=254 time=5.972 ms 64 bytes from 10.100.20.10: icmp_seq=1210 ttl=254 time=1.466 ms 64 bytes from 10.100.20.10: icmp_seq=1211 ttl=254 time=2.972 ms 64 bytes from 10.100.20.10: icmp_seq=1212 ttl=254 time=4.582 ms 64 bytes from 10.100.20.10: icmp_seq=1213 ttl=254 time=1.434 ms 64 bytes from 10.100.20.10: icmp_seq=1214 ttl=254 time=4.486 ms 64 bytes from 10.100.20.10: icmp_seq=1215 ttl=254 time=2.743 ms 64 bytes from 10.100.20.10: icmp_seq=1216 ttl=254 time=1.469 ms 64 bytes from 10.100.20.10: icmp_seq=1217 ttl=254 time=7.322 ms 64 bytes from 10.100.20.10: icmp_seq=1218 ttl=254 time=1.532 ms 64 bytes from 10.100.20.10: icmp_seq=1219 ttl=254 time=1.438 ms 64 bytes from 10.100.20.10: icmp_seq=1220 ttl=254 time=7.122 ms 64 bytes from 10.100.20.10: icmp_seq=1221 ttl=254 time=1.344 ms 64 bytes from 10.100.20.10: icmp_seq=1222 ttl=254 time=1.63 ms 64 bytes from 10.100.20.10: icmp_seq=1223 ttl=254 time=6.133 ms 64 bytes from 10.100.20.10: icmp_seq=1224 ttl=254 time=1.455 ms 64 bytes from 10.100.20.10: icmp_seq=1225 ttl=254 time=3.221 ms 64 bytes from 10.100.20.10: icmp_seq=1226 ttl=254 time=4.435 ms 64 bytes from 10.100.20.10: icmp_seq=1227 ttl=254 time=1.463 ms 64 bytes from 10.100.20.10: icmp_seq=1228 ttl=254 time=5.14 ms 64 bytes from 10.100.20.10: icmp_seq=1229 ttl=254 time=2.796 ms 64 bytes from 10.100.20.10: icmp_seq=1230 ttl=254 time=1.49 ms 64 bytes from 10.100.20.10: icmp_seq=1231 ttl=254 time=6.707 ms 64 bytes from 10.100.20.10: icmp_seq=1232 ttl=254 time=1.447 ms 64 bytes from 10.100.20.10: icmp_seq=1233 ttl=254 time=1.285 ms 64 bytes from 10.100.20.10: icmp_seq=1234 ttl=254 time=7.097 ms 64 bytes from 10.100.20.10: icmp_seq=1235 ttl=254 time=1.295 ms 64 bytes from 10.100.20.10: icmp_seq=1236 ttl=254 time=0.916 ms 64 bytes from 10.100.20.10: icmp_seq=1237 ttl=254 time=6.24 ms 64 bytes from 10.100.20.10: icmp_seq=1238 ttl=254 time=1.439 ms 64 bytes from 10.100.20.10: icmp_seq=1239 ttl=254 time=2.739 ms 64 bytes from 10.100.20.10: icmp_seq=1240 ttl=254 time=4.477 ms 64 bytes from 10.100.20.10: icmp_seq=1241 ttl=254 time=1.431 ms 64 bytes from 10.100.20.10: icmp_seq=1242 ttl=254 time=5.372 ms 64 bytes from 10.100.20.10: icmp_seq=1243 ttl=254 time=3.119 ms 64 bytes from 10.100.20.10: icmp_seq=1244 ttl=254 time=1.504 ms 64 bytes from 10.100.20.10: icmp_seq=1245 ttl=254 time=6.909 ms 64 bytes from 10.100.20.10: icmp_seq=1246 ttl=254 time=1.498 ms 64 bytes from 10.100.20.10: icmp_seq=1247 ttl=254 time=1.454 ms 64 bytes from 10.100.20.10: icmp_seq=1248 ttl=254 time=6.701 ms 64 bytes from 10.100.20.10: icmp_seq=1249 ttl=254 time=1.441 ms 64 bytes from 10.100.20.10: icmp_seq=1250 ttl=254 time=1.888 ms 64 bytes from 10.100.20.10: icmp_seq=1251 ttl=254 time=6.052 ms 64 bytes from 10.100.20.10: icmp_seq=1252 ttl=254 time=1.469 ms 64 bytes from 10.100.20.10: icmp_seq=1253 ttl=254 time=3.61 ms 64 bytes from 10.100.20.10: icmp_seq=1254 ttl=254 time=4.213 ms 64 bytes from 10.100.20.10: icmp_seq=1255 ttl=254 time=1.276 ms 64 bytes from 10.100.20.10: icmp_seq=1256 ttl=254 time=5.712 ms 64 bytes from 10.100.20.10: icmp_seq=1257 ttl=254 time=2.299 ms 64 bytes from 10.100.20.10: icmp_seq=1258 ttl=254 time=1.417 ms 64 bytes from 10.100.20.10: icmp_seq=1259 ttl=254 time=7.159 ms 64 bytes from 10.100.20.10: icmp_seq=1260 ttl=254 time=1.538 ms 64 bytes from 10.100.20.10: icmp_seq=1261 ttl=254 time=1.629 ms 64 bytes from 10.100.20.10: icmp_seq=1262 ttl=254 time=7.892 ms 64 bytes from 10.100.20.10: icmp_seq=1263 ttl=254 time=1.495 ms 64 bytes from 10.100.20.10: icmp_seq=1264 ttl=254 time=2.792 ms ^C --- 10.100.20.10 ping statistics --- 1265 packets transmitted, 58 packets received, 95.42% packet loss round-trip min/avg/max = 0.916/3.31/7.892 ms SA-HOST-A#
修订历史记录
| 版本 | 发布日期 | 备注 |
|---|---|---|
1.0 |
29-Jul-2024
|
初始版本 |
反馈