由于配置错误，对Ultra Packet Core和Nexus交换机之间的BGP抖动进行故障排除

简介

本文档介绍配置冗余BGP连接的Cisco Ultra Packet Core(UPC)和Nexus 9000交换机之间的边界网关协议(BGP)抖动的解决方案。

问题

当Cisco Ultra Packet Core和Nexus交换机之间的某个冗余接口抖动时，会触发BGP抖动。

条件

Ultra Packet Core(UPC)节点在单独的端口上连接到Nexus枝叶A和枝叶B。BGP IPv6对等体已建立，默认路由已安装在UPC节点上。图1显示了包含到枝叶交换机的冗余路径的高级网络图。

图1：网络图

配置

使用VLAN和接口绑定的UPC端口配置：

port ethernet 1/10
    no shutdown
    vlan 140
        no shutdown
        bind interface saegw_vlan140_1/10 saegw
#exit

#exit
port ethernet 1/11
    no shutdown
    vlan 141
        no shutdown
        bind interface saegw_vlan141_1/11 saegw
#exit
#exit
end

使用IP地址配置UPC接口：

interface saegw_vlan140_1/10
  ip address 10.11.11..8 255.255.255.0
  ipv6 address fd00:10:11:11::8/64 secondary
  bfd interval 300 min_rx 300 multiplier 3
#exit
interface saegw_vlan141_1/11
  ip address 10.11.12.8 255.255.255.0
  ipv6 address fd00:10:11:12::8/64 secondary
  bfd interval 300 min_rx 300 multiplier 3
#exit

UPC BGP配置：

router bgp 25949
  router-id 172.19.20.30
  maximum-paths ebgp 4
  neighbor 10.11.11..1 remote-as 25949
  neighbor 10.11.11..1 fall-over bfd 
  neighbor 10.11.12.1 remote-as 25949
  neighbor 10.11.12.1 fall-over bfd 
  neighbor fd00:10:11:11::1 remote-as 25949
  neighbor fd00:10:11:12::1 remote-as 25949
  address-family ipv4
    neighbor 10.11.11..1 route-map accept_default in
    neighbor 10.11.11..1 route-map gw-1-OUT out
    neighbor 10.11.12.1 route-map accept_default in
    neighbor 10.11.12.1 route-map gw-1-OUT out
    redistribute connected
#exit
address-family ipv6
  neighbor fd00:10:11:11::1 activate
  neighbor fd00:10:11:11::1 route-map accept_v6_default in
  neighbor fd00:10:11:11::1 route-map allow_service_ips_v6 out
  neighbor fd00:10:11:12::1 activate
  neighbor fd00:10:11:12::1 route-map accept_v6_default in
  neighbor fd00:10:11:12::1 route-map allow_service_ips_v6 out
  redistribute connected
#exit

ipv6 prefix-list name accept_v6_default_routes seq 10 permit ::/0
route-map accept_v6_default permit 10
  match ipv6 address prefix-list accept_v6_default_routes
#exit

Nexus 9000交换机配置：

Interface vlan140
ipv6 address fd00:10:11:11::1/64
no ipv6 redirects

interface vlan141
ipv6 address fd00:10:11:12::1/64
no ipv6 redirects

vrf upc
address-family ipv4 unicast
advertise l2vpn evpn
maximum-paths ibgp 2
address-family ipv6 unicast
advertise l2vpn evpn
maximum-paths ibgp 2
neighbor fd00:10:11:12::5
remote-as 25949
address-family ipv6 unicast
neighbor fd00:10:11:12::6
remote-as 25949
address-family ipv6 unicast
neighbor fd00:10:11:12::8
remote-as 25949
address-family ipv6 unicast

分析

最初，观察到一个UPC接口(fd00:10:11:12::8)和Nexus交换机（fd00:10:11:12::1属于vlan141）之间的正常BGP通信，包括TCP ACK消息：

2023-01-01 01:01:59.000000 fd00:10:11:12::8 -> fd00:10:11:12::1 TCP 35813 > bgp [ACK] Seq=250 Ack=8664 Win=31744 Len=0 TSV=2412344062 TSER=531234647
2023-01-01 01:01:59.000087 fd00:10:11:12::8 -> fd00:10:11:12::1 TCP 35813 > bgp [ACK] Seq=250 Ack=11520 Win=37376 Len=0 TSV=2412344062 TSER=531234647
2023-01-01 01:01:59.000162 fd00:10:11:12::8 -> fd00:10:11:12::1 TCP 35813 > bgp [ACK] Seq=250 Ack=14376 Win=43008 Len=0 TSV=241234062 TSER=531234647
2023-01-01 01:01:59.000281 fd00:10:11:12::8 -> fd00:10:11:12::1 TCP 35813 > bgp [ACK] Seq=250 Ack=17232 Win=49152 Len=0 TSV=2412344062 TSER=531234647
2023-01-01 01:01:59.000936 fd00:10:11:12::8 -> fd00:10:11:12::1 TCP 35813 > bgp [ACK] Seq=250 Ack=20663 Win=48640 Len=0 TSV=2412344063 TSER=531234647

当枝叶B接口向UPC发生故障时，日志中会出现错误行为，其中UPC（源：fd00:10:11:12::8）向属于不同VLAN(vlan140)的枝叶A发起新的BGP连接尝试。

2023-01-01 22:36:12.370117 fd00:10:11:12::8 -> fd00:10:11:11::1 TCP 41987 > bgp [SYN] Seq=0 Win=14400 Len=0 MSS=1440 TSV=2412347369 TSER=0 WS=9

在错误接口上发送的此类无效BGP SYN消息会导致BGP关闭。当Nexus通告其自身连接的路由，并且UPC获取通过BGP关闭的接口的路由时，UPC会尝试通过另一个接口使用不同/错误的传出IP进行连接。

解决方案

由于本文条件(Condition)部分中提到的配置，由于UPC从两个接口接收两个枝叶的已连接路由信息，因此当其中一个接口关闭时，UPC会尝试通过另一个接口与该枝叶通信。

要避免UPC从错误的接口发送BGP连接建立消息，请注意以下配置更改：

1. 在UPC配置中，添加 update-source 对于邻居。如果主接口关闭，此配置将阻止来自其他接口的BGP连接。例如，当saegw_vlan140_1/10(fd00:10:11:11::1/64)关闭时，节点不能将传出接口saegw_vlan141_1/11用于BGP对等体fd00:10:11:11::8。 
   下面是一个示例配置：
   

   neighbor fd00:10:11:11::1 update-source fd00:10:11:11::8
   neighbor fd00:10:11:12::1 update-source fd00:10:11:12::8

2. 在Nexus配置中，阻止来自错误接口的前缀。
   例如，我们拒绝通过邻居fd00:10:11:11::1的冗余枝叶的路由
   

   neighbor fd00:10:11:11::1
   update prefix list to deny fd00:10:11:12::8/64

3. 在Nexus交换机中，通过VXLAN从VTEP到外部节点的EBGP对等必须位于租户VRF中，并且必须使用 update-source 的 loopback 接口（通过VXLAN对等），如Cisco Nexus 9000配置指南中所建