Implement BGP EVPN DHCP Layer 2 Relay on Catalyst 9000 Series Switches

Available Languages

Download Options

PDF (556.4 KB)
View with Adobe Reader on a variety of devices
ePub (591.8 KB)
View in various apps on iPhone, iPad, Android, Sony Reader, or Windows Phone
Mobi (Kindle) (559.4 KB)
View on Kindle device or Kindle app on multiple devices

Updated:January 17, 2024

Document ID:221565

Bias-Free Language

The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.

Introduction

This document describes how to configure, verify, and troubleshoot the EVPN VxLAN DHCP L2 Relay feature.

Prerequisites

Requirements

This feature is used in any CGW type deployment where DHCP is used
If implementing Protected Segmentation please review these documents
- Implement BGP EVPN Routing Policy on Catalyst 9000 Series Switches
- Implement BGP EVPN Protected Overlay Segmentation on Catalyst 9000 Series Switches

Components Used

The information in this document is based on these software and hardware versions:

Catalyst 9300
Catalyst 9400
Catalyst 9500
Catalyst 9600
Cisco IOS® XE 17.12.1 and later versions

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Background Information

Document Details

This document can be used for any CGW deployment where DHCP needs to be relayed from a Leaf with no SVI toward the Central Gateway.

If you are not using protected segmentation use the section of the document where SVI is advertised into the fabric

If you are implementing protected segmentation this document is part 2 of 3 inter-related documents:

Document 1:Implement BGP EVPN Routing Policy on Catalyst 9000 Series Switches covers how to control the BGP BUM traffic in the Overlay, and must be configured first
Document 2:Implement BGP EVPN Protected Overlay Segmentation on Catalyst 9000 Series Switches builds upon the Overlay design and policy of document 1, describes the implementation of the 'protected' keyword.
Document 3: This document. Builds on top of the last two documents and describes the way DHCP relay is implemented with layer 2 only Leafs and CGW

L2 Relay Behavior

Relay	Snooping	Core Flood	Access Flood	IPv4
yes	yes	no	yes	Option 82 Suboption: (1) Agent Circuit ID (vni-mod-port) gets populated with dhcp snooping One can limit the access side with dhcp trust configuration * RECOMMENDED MODEL
yes	no	yes	yes	Option 82 Suboption: (1) Agent Circuit ID (vlan-mod-port) gets populated with dhcp snooping
no	yes	no	yes	Option 82 Suboption: (1) Agent Circuit ID (vni-mod-port) gets populated with dhcp snooping One can limit the access side with dhcp trust configuration
Relay	Snooping	Core Flood	Access Flood	IPv6
yes	yes	yes	yes	Option 82 Suboption: (1) Agent Circuit ID (vni-mod-port) gets populated with dhcp snooping One can limit the access side with dhcp trust configuration
yes	no	yes	yes	Option 82 Suboption: (1) Agent Circuit ID (vlan-mod-port) gets populated with dhcp snooping
no	yes	yes	yes	Option 82 Suboption: (1) Agent Circuit ID (vni-mod-port) gets populated with dhcp snooping One can limit the access side with dhcp trust configuration
no	no	yes	yes

Terminology

VRF	Virtual Routing Forwarding	Defines a layer 3 routing domain that be separated from other VRF and global IPv4/IPv6 routing domain
AF	Address Family	Defines which type prefixes and routing info BGP handles
AS	Autonomous System	A set of Internet routable IP prefixes that belong to a network or a collection of networks that are all managed, controlled and supervised by a single entity or organization
EVPN	Ethernet Virtual Private Network	Extension that allows BGP to transport Layer 2 MAC and Layer 3 IP information is EVPN and uses Multi-Protocol Border Gateway Protocol (MP-BGP) as the protocol to distribute reachability information that pertains to the VXLAN overlay network.
VXLAN	Virtual Extensible LAN (Local Area Network)	VXLAN is designed to overcome the inherent limitations of VLANs and STP. It is a proposed IETF standard [RFC 7348] to provide the same Ethernet Layer 2 network services as VLANs do, but with greater flexibility. Functionally, it is a MAC-in-UDP encapsulation protocol that runs as a virtual overlay on a Layer 3 underlay network.
CGW	Centralized Gateway	And implementation of EVPN where the gateway SVI are not on each leaf. Instead, all routing is done by a specific leaf using asymmetric IRB (Integrated Routing and Bridging)
DEF GW	Default Gateway	A BGP extended community attribute added to the MAC/IP prefix via the command "default-gateway advertise enable" under the 'l2vpn evpn' configuration section.
IMET (RT3)	Inclusive Multicast Ethernet Tag (Route)	Also called BGP type-3 route. This route type is used in EVPN to deliver BUM (broadcast / unknown unicast / multicast) traffic between VTEPs.
RT2	Route Type 2	BGP MAC or MAC/IP prefix that represents a host MAC or Gateway MAC-IP
EVPN Mgr	EVPN Manager	Central management component for various other components (example: learns from SISF and signals to L2RIB)
SISF	Switch Integrated Security Feature	An agnostic host tracking table that is used by EVPN to learn what local hosts are present on a Leaf
L2RIB	Layer 2 Routing Information Base	In intermediate component for managing interactions between BGP, EVPN Mgr, L2FIB
FED	Forwarding Engine Driver	Programs the ASIC (hardware) layer
MATM	Mac Address Table Manager	IOS MATM: software table which installs only local addresses and FED MATM: hardware table which installs local and remote addresses learned from control plane, and is part of the hardware forwarding plane

Configure (Standard CGW Deployment)

Network Diagram

Totally Isolated Diagram

Note: This section covers a standard CGW deployment without the use of the protected feature.

Debugs showing the DHCP DORA packet exchange are only shown in the Protected segment example

L2 VTEP (Leaf) Key Details

Request packet comes from client

Use the Default gw advertised CGW mac.
If more than one gw exists first gw mac would be used.
Convert outer broadcast MAC (client initiated: D and R in DORA) to unicast GW mac and forward to CGW

DHCP snooping adds: option 82 suboptions: circuit and RID

(RID is used by response pkt processing on CGW)

(Informs CGW its not local and to fabric relay back to L2VTEP)

Option: (82) Agent Information Option
        Length: 24
        Option 82 Suboption: (1) Agent Circuit ID
            Length: 12
            Agent Circuit ID: 010a00080000277501010000
        Option 82 Suboption: (2) Agent Remote ID
            Length: 8
          Agent Remote ID: 
          000682c7bf88700 <-- switch base mac 682c.7bf8.8700 (from 'show switch')

Response pkts received from CGW over vxlan tunnel.
Leaf Strips option 82.
Adds binding entries with client source interface. (vxlan-mod-port gives the client source interface).
Response packet forwarded to client.

L3 VTEP (CGW) Key Details

Enable DHCP SNOOPING
Enable DHCP RELAY in SVI
Request is received from L2VTEP, and is given to relay.
Relay adds other option 82 suboptions (gi, server override, and so on) and sends to DHCP server.
DHCP response from dhcp server first comes to RELAY component.
After RELAY strips off option 82 parameters (gi address, server override, and so on) the packet is passed to dhcp snooping component.
Snooping component checks the RID (Router ID) and if its not local doesn’t remove option 82 subopton 1 and 2.
Fabric relays (since RID is not local) packet is direct forwarded to remote client.
Uses client Mac and does bridge inject. Hardware does client mac lookup and forwards the packet with vxlan encap to the originating L2VTEP.

L2VTEP

Configure the evpn instance

Leaf-01#show run | beg l2vpn evpn instance 201
l2vpn evpn instance 201 vlan-based
 encapsulation vxlan
 replication-type ingress

Enable DHCP Snooping

Leaf-01#show run | sec dhcp snoop
ip dhcp snooping vlan 101,201
ip dhcp snooping

CGW

Configure the evpn instance

Border#sh run | s l2vpn evpn instance 201
l2vpn evpn instance 201 vlan-based
 encapsulation vxlan
 replication-type ingress
 default-gateway advertise enable  <-- Enable to add BGP DEF GW ext. community attribute

Note: The DEF GW attribute is critical for L2 Relay to know who to encapsulate & send the DHCP packet to.

Enable DHCP snooping

Border#sh run | s dhcp snoop
ip dhcp snooping vlan 101,201
ip dhcp snooping

Ensure the DHCP relay has the correct configuration to handle the additional options

Border#sh run int vl 201
Building configuration...

interface Vlan201
 mac-address 0000.beef.cafe
 vrf forwarding red
 ip dhcp relay information option vpn-id  <-- Ensure the vrf info is passed to the server
 ip dhcp relay source-interface Loopback0 <-- Sets the relay source interface to the loopback 
 ip address 10.1.201.1 255.255.255.0
 ip helper-address global 10.1.33.33      <-- In this scenario the DHCP server is in the global routing table

Verify (Standard CGW Deployment)

Gateway Prefix (Leaf)

Leaf-01#sh bgp l2vpn evpn route-type 2 0 0000.beef.cafe 10.1.201.1
BGP routing table entry for [2][172.16.255.3:201][0][48][0000BEEFCAFE][32][10.1.201.1]/24, version 896401
Paths: (1 available, best #1, table evi_201)  <-- In the EVI context for the segment
Not advertised to any peer
Refresh Epoch 3
Local, imported path from [2][172.16.255.6:201][0][48][0000BEEFCAFE][32][10.1.201.1]/24 (global)
  172.16.255.6 (metric 30) (via default) from 172.16.255.1 (172.16.255.1)
    Origin incomplete, metric 0, localpref 100, valid, internal, best
    EVPN ESI: 00000000000000000000, Label1 20101              <-- Correct segment ID
    Extended Community: RT:65001:201 ENCAP:8 EVPN DEF GW:0:0  <-- GW attribute added indicating this is GW prefix which L2 Relay uses
    Originator: 172.16.255.6, Cluster list: 172.16.255.1      <-- Learned from the Border (CGW)
    rx pathid: 0, tx pathid: 0x0
    Updated on Nov 14 2023 16:06:40 UTC

FED MATM (Leaf)

Leaf-01#show platform software fed switch active matm macTable vlan 201
VLAN   MAC                  Type  Seq#    EC_Bi  Flags  machandle           siHandle            riHandle            diHandle              *a_time  *e_time  ports                                                         Con

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
201    0006.f601.cd42        0x1   32436      0      0  0x71e058dc3368      0x71e058655018      0x0                 0x71e05877c888            300       69  GigabitEthernet1/0/1                                          Yes

201    0006.f601.cd01        0x1   32437      0      0  0x71e058dae308      0x71e058655018      0x0                 0x71e05877c888            300        9  GigabitEthernet1/0/1                                          Yes

201    0000.beef.cafe  0x5000001       0      0     64  0x71e059177138      0x71e058eeb418      0x71e058df81f8      0x0                         0       69  VTEP 172.16.255.6 adj_id 1371                                 No
 <--- The GW MAC shows learnt via the Border Leaf Loopback with the right flags

Total Mac number of addresses:: 3
Summary:
Total number of secure addresses:: 0
Total number of drop addresses:: 0
Total number of lisp local addresses:: 0
Total number of lisp remote addresses:: 1          <--- 
*a_time=aging_time(secs)  *e_time=total_elapsed_time(secs)
Type:
MAT_DYNAMIC_ADDR           0x1  MAT_STATIC_ADDR            0x2  MAT_CPU_ADDR               0x4  MAT_DISCARD_ADDR           0x8
MAT_ALL_VLANS             0x10  MAT_NO_FORWARD            0x20  MAT_IPMULT_ADDR           0x40  MAT_RESYNC                0x80
MAT_DO_NOT_AGE           0x100  MAT_SECURE_ADDR          0x200  MAT_NO_PORT              0x400  MAT_DROP_ADDR            0x800
MAT_DUP_ADDR            0x1000  MAT_NULL_DESTINATION    0x2000  MAT_DOT1X_ADDR          0x4000  MAT_ROUTER_ADDR         0x8000
MAT_WIRELESS_ADDR      0x10000  MAT_SECURE_CFG_ADDR    0x20000  MAT_OPQ_DATA_PRESENT   0x40000  MAT_WIRED_TUNNEL_ADDR  0x80000
MAT_DLR_ADDR          0x100000  MAT_MRP_ADDR          0x200000  MAT_MSRP_ADDR         0x400000  MAT_LISP_LOCAL_ADDR   0x800000
MAT_LISP_REMOTE_ADDR 0x1000000  MAT_VPLS_ADDR        0x2000000  MAT_LISP_GW_ADDR     0x4000000                                 <-- these 3 values added = 0x5000001 (note that 0x4000000 = GW type address

Local MAC (Leaf)

Leaf-01#show switch
Switch/Stack Mac Address : 682c.7bf8.8700 - Local Mac Address
Mac persistency wait time: Indefinite
                                             H/W   Current
Switch#   Role    Mac Address     Priority Version  State
-------------------------------------------------------------------------------------
*1       Active   682c.7bf8.8700     1      V01     Ready   <--- Use to validate the Agent ID in DHCP Option 82

DHCP Snooping (Leaf & CGW)

Leaf-01#show ip dhcp snooping
Switch DHCP snooping is enabled
Switch DHCP gleaning is disabled
DHCP snooping is configured on following VLANs:
101,201
DHCP snooping is operational on following VLANs:
101,201

Insertion of option 82 is enabled
circuit-id default format: vlan-mod-port
remote-id: 682c.7bf8.8700 (MAC)          <--- Leaf-01 adds the switch MAC to Option 82 to indicate to CGW the packet is remote

CGW#show ip dhcp snooping
Switch DHCP snooping is enabled
Switch DHCP gleaning is disabled
DHCP snooping is configured on following VLANs:
101,201
DHCP snooping is operational on following VLANs:
101,201

Configure (Partially Isolated Protected)

DHCP snooping on the Access Leaf relies on the default gateway route from CGW to learn the gateway MAC to forward DHCP packets to.

When using the Partially Isolated design with external gateway additional configurations are required on CGW to advertise the MAC-IP RT2 with the default gateway (DEF GW) attribute.

Note: Note: This section also works to describe a Totally Isolated Protected Segment implementation, which also uses a GW that is advertised into the fabric (versus GW outside the fabric).

Network Diagram

Partially Isolated Diagram

Expanded Partially Isolated logical flow

L2 VTEP (Leaf) Key Details

Request packet comes from client

Use the Default gw advertised CGW mac.
If more than one gw exists first gw mac would be used.
Convert outer broadcast MAC (client initiated: D and R in DORA) to unicast GW mac and forward to CGW

DHCP snooping adds: option 82 suboptions: circuit and RID

(RID is used by response pkt processing on CGW)

(Informs CGW its not local and to fabric relay back to L2VTEP)

Option: (82) Agent Information Option
        Length: 24
        Option 82 Suboption: (1) Agent Circuit ID
            Length: 12
            Agent Circuit ID: 010a00080000277501010000
        Option 82 Suboption: (2) Agent Remote ID
            Length: 8
          Agent Remote ID: 
          000682c7bf88700 <-- switch base mac 682c.7bf8.8700 (from 'show switch')

Response pkts received from CGW over vxlan tunnel.
Leaf Strips option 82.
Adds binding entries with client source interface. (vxlan-mod-port gives the client source interface).
Response packet forwarded to client.

L3 VTEP (CGW) Key Details

Enable DHCP SNOOPING
Enable DHCP RELAY in SVI
Request is received from L2VTEP, and is given to relay.
Relay adds other option 82 suboptions (gi, server override, and so on) and sends to DHCP server.
DHCP response from dhcp server first comes to RELAY component.
After RELAY strips off option 82 parameters (gi address, server override, and so on) the packet is passed to dhcp snooping component.
Snooping component checks the RID (Router ID) and if its not local doesn’t remove option 82 subopton 1 and 2.
Fabric relays (since RID is not local) packet is direct forwarded to remote client.
Uses client Mac and does bridge inject. Hardware does client mac lookup and forwards the packet with vxlan encap to the originating L2VTEP.

Steps required to support DHCP L2 Relay:

Enable ip local learning
Create a policy with gleaning disabled
Attach to external gateway evi/vlans
Add static entries into device tracking table for external gateway mac-ip
Create BGP route map to match RT2 MAC-IP prefixes and set the default gateway extended community
Apply route-map to BGP Route Reflector neighbors
Ensure the DHCP relay has the correct configuration to handle the additional option
Configure DHCP Snooping on fabric vlan and the External GW vlan

Note: No configuration changes are required on the access Leafs to support DHCP L2 Relay with external gateway.

CGW

Enable ip local learning

CGW#show running-config | beg l2vpn evpn instance 202
l2vpn evpn instance 202 vlan-based
 encapsulation vxlan
 replication-type ingress
 ip local-learning enable 
<-- to advertise RT-2 with default gateway EC, ip local-learning must be enabled on the CGW. 
    Use additional device-tracking policy shown in the next output to prevent MAC-IP binding flapping while still being able to advertise RT-2 with default gateway Ext Community
 multicast advertise enable
<--- There is no default-gateway advertise enable cli here, as the SVI (Vlan 2021) used by this segment does not exist in this Vlan and is not part of the fabric

Create a policy with gleaning disabled

device-tracking policy dt-no-glean <-- Configure device tracking policy to prevent MAC-IP flapping 
 security-level glean
 no protocol ndp
 no protocol dhcp6
 no protocol arp
 no protocol dhcp4

Attach to external gateway evi/vlans

CGW#show running-config | sec vlan config
vlan configuration 202
 member evpn-instance 202 vni 20201
 device-tracking attach-policy dt-no-glean <-- apply the new device tracking policy to the vlan configuration

Add static entries into device tracking table for external gateway mac-ip

device-tracking binding vlan 202 10.1.202.1 interface TwentyFiveGigE1/0/1 0000.beef.cafe
<-- All static entries in device tracking table should be for external gateway mac-ip’s. 
    If there is any other static entry in device tracking table, match ip/ipv6 configurations in route map with prefix lists (permit/deny) are required to attach default gateway extended community to external gateway mac-iproutes only.

Create BGP route map to match RT2 MAC-IP prefixes and set the default gateway extended community

route-map CGW_DEF_GW permit 10
 match evpn route-type 2-mac-ip <-- match RT2 type MAC-IP
 set extcommunity default-gw    <-- Set Default-gateway (DEF GW 0:0) extended community 
route-map CGW_DEF_GW permit 20

Apply route-map to BGP Route Reflector neighbors

CGW#sh run | sec router bgp
address-family l2vpn evpn
 neighbor 172.16.255.1 activate
 neighbor 172.16.255.1 send-community both
 neighbor 172.16.255.1 route-map CGW_DEF_GW out  <-- Sets the DEF GW Community when it advertises MAC-IP type RT2 to the RR
 neighbor 172.16.255.2 activate
 neighbor 172.16.255.2 send-community both
 neighbor 172.16.255.2 route-map CGW_DEF_GW out <-- Sets the DEF GW Community when it advertises MAC-IP type RT2 to the RR

Ensure the DHCP relay has the correct configuration to handle the additional options

CGW#show run int vl 2021
Building configuration...

Current configuration : 315 bytes
!
interface Vlan2021
 mac-address 0000.beef.cafe
 vrf forwarding pink
 ip dhcp relay information option vpn-id   <-- Ensure the vrf info is passed to the server
 ip dhcp relay source-interface Loopback0  <-- sets the relay source interface to the loopback 
 ip address 10.1.202.1 255.255.255.0
 ip helper-address global 10.1.33.33       <-- In this scenario the next hop to the DHCP server is in the global table
 no ip redirects
 ip local-proxy-arp
 ip route-cache same-interface
 no autostate

Configure DHCP Snooping on fabric vlans and the External GW vlan

Leaf01#sh run | s dhcp snoop
ip dhcp snooping vlan 202
ip dhcp snooping

CGW#sh run | s dhcp snoop
ip dhcp snooping vlan 202,2021  <-- snooping is required in both the fabric vlan and the external GW vlan 
ip dhcp snooping

Ensure that the uplink to the DHCP server is trusted on the CGW

CGW#sh run int tw 1/0/1

interface TwentyFiveGigE1/0/1
 switchport trunk allowed vlan 202
 switchport mode trunk
 ip dhcp snooping trust
end

CGW#sh run int tw 1/0/2

interface TwentyFiveGigE1/0/2
 switchport trunk allowed vlan 33,2021
 switchport mode trunk
 ip dhcp snooping trust
end

Note: Due to the way the server is placed on the Firewall device trust was configured on both links facing this device. In the zoomed in diagram you can see that the Offer arrives at both Tw1/0/1 & Tw1/0/2 in this design.

Verify (Partially Isolated Protected)

Gateway Prefix (Leaf)

Leaf01#show bgp l2vpn evpn route-type 2 0 0000.beef.cafe 10.1.202.1
BGP routing table entry for [2][172.16.254.3:202][0][48][0000BEEFCAFE][32][10.1.202.1]/24, version 341181
Paths: (1 available, best #1, table evi_202)
  Not advertised to any peer
  Refresh Epoch 2
  Local, imported path from [2][172.16.254.6:202][0][48][0000BEEFCAFE][32][10.1.202.1]/24 (global)
    172.16.254.6 (metric 3) (via default) from 172.16.255.1 (172.16.255.1)
      Origin incomplete, metric 0, localpref 100, valid, internal, best
      EVPN ESI: 00000000000000000000, Label1 20201
      Extended Community: RT:65001:202 ENCAP:8 EVPN DEF GW:0:0       <-- GW attribute added indicating this is GW prefix which L2 Relay uses
      Originator: 172.16.255.6, Cluster list: 172.16.255.1
      rx pathid: 0, tx pathid: 0x0
      Updated on Sep 19 2023 19:57:25 UTC

FED MATM (Leaf)

Confirm the Leaf has installed the CGW remote MAC in hardware

Leaf01#show platform software fed switch active matm macTable vlan 202
VLAN   MAC                  Type  Seq#    EC_Bi  Flags  machandle           siHandle            riHandle            diHandle              *a_time  *e_time  ports                                                         Con
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
202    0006.f601.cd01        0x1    1093      0      0  0x71e05918f138      0x71e05917a1a8      0x0                 0x71e05877c888            300        8  GigabitEthernet1/0/1                                          Yes

202    0006.f601.cd44        0x1   14309      0      0  0x71e058cdc208      0x71e05917a1a8      0x0                 0x71e05877c888            300      169  GigabitEthernet1/0/1                                          Yes

202    0000.beef.cafe  0x5000001       0      0     64  0x71e058ee5d88      0x71e059195f88      0x71e059171678      0x0                         0        1  VTEP 172.16.254.6 adj_id 894                                  No
  <--- The GW MAC shows learnt via the Border Leaf Loopback

Total Mac number of addresses:: 3
Summary:
Total number of secure addresses:: 0
Total number of drop addresses:: 0
Total number of lisp local addresses:: 0
Total number of lisp remote addresses:: 1
*a_time=aging_time(secs)  *e_time=total_elapsed_time(secs)
Type:
MAT_DYNAMIC_ADDR           0x1  MAT_STATIC_ADDR            0x2  MAT_CPU_ADDR               0x4  MAT_DISCARD_ADDR           0x8
MAT_ALL_VLANS             0x10  MAT_NO_FORWARD            0x20  MAT_IPMULT_ADDR           0x40  MAT_RESYNC                0x80
MAT_DO_NOT_AGE           0x100  MAT_SECURE_ADDR          0x200  MAT_NO_PORT              0x400  MAT_DROP_ADDR            0x800
MAT_DUP_ADDR            0x1000  MAT_NULL_DESTINATION    0x2000  MAT_DOT1X_ADDR          0x4000  MAT_ROUTER_ADDR         0x8000
MAT_WIRELESS_ADDR      0x10000  MAT_SECURE_CFG_ADDR    0x20000  MAT_OPQ_DATA_PRESENT   0x40000  MAT_WIRED_TUNNEL_ADDR  0x80000
MAT_DLR_ADDR          0x100000  MAT_MRP_ADDR          0x200000  MAT_MSRP_ADDR         0x400000  MAT_LISP_LOCAL_ADDR   0x800000
MAT_LISP_REMOTE_ADDR 0x1000000  MAT_VPLS_ADDR        0x2000000  MAT_LISP_GW_ADDR     0x4000000                                    <-- these 3 values added = 0x5000001 (note that 0x4000000 = GW type address

Local MAC (Leaf)

Leaf01#show switch
Switch/Stack Mac Address : 682c.7bf8.8700 - Local Mac Address
Mac persistency wait time: Indefinite
                                             H/W   Current
Switch#   Role    Mac Address     Priority Version  State
-------------------------------------------------------------------------------------
*1       Active   682c.7bf8.8700     1      V01     Ready  <-- this is the MAC that will be added to DHCP Agent Remote ID

DHCP Snooping (Leaf & CGW)

Confirm that DHCP snooping is enabled on the Leaf in the fabric vlan

Leaf01#show ip dhcp snooping
Switch DHCP snooping is enabled
Switch DHCP gleaning is disabled
DHCP snooping is configured on following VLANs:
202
DHCP snooping is operational on following VLANs:   <-- Snooping on in the Fabric Vlan 
202
<...snip...>

Insertion of option 82 is enabled
   circuit-id default format: vlan-mod-port
   remote-id: 682c.7bf8.8700 (MAC)            <--- Remote ID (RID) inserted by Leaf to DHCP packets
<...snip...>

Confirm that DHCP snooping is enabled on the CGW in the fabric and external gateway vlans

CGW#show ip dhcp snooping
Switch DHCP snooping is enabled
Switch DHCP gleaning is disabled
DHCP snooping is configured on following VLANs:
202,2021
DHCP snooping is operational on following VLANs:  <-- Snooping on in the Fabric and External GW Vlans 
202,2021
<...snip...>

DHCP snooping trust/rate is configured on the following Interfaces:

Interface                  Trusted    Allow option    Rate limit (pps)
-----------------------    -------    ------------    ----------------
TwentyFiveGigE1/0/1              yes        yes             unlimited  <-- Trust set on ports the OFFER arrives on
Interface                  Trusted    Allow option    Rate limit (pps)
-----------------------    -------    ------------    ----------------
  Custom circuit-ids:
TwentyFiveGigE1/0/2              yes        yes             unlimited  <-- Trust set on ports the OFFER arrives on
  Custom circuit-ids:

Confirm that DHCP snooping binding is created

Leaf01#show ip dhcp snooping binding
MacAddress          IpAddress        Lease(sec)  Type           VLAN  Interface
------------------  ---------------  ----------  -------------  ----  --------------------
00:06:F6:01:CD:43   10.1.202.10      34261       dhcp-snooping   202   GigabitEthernet1/0/1  <-- DHCP Snooping has created the binding
Total number of bindings: 1

Troubleshoot (Any CGW Type)

Debugs are helpful to show how the DHCP snooping and L2 Relay processes are handling DHCP packets.

Note: These debugs can be used for any type of deployment that uses CGW with DCHP L2 Relay.

DHCP Snooping Debugs (Leaf)

Debug Snooping to confirm packet processing

Leaf01#debug ip dhcp snooping packet
DHCP Snooping Packet debugging is on

Leaf01#show debugging
DHCP Snooping packet debugging is on

Start the host DHCP address attempt

For this document a shut/no shut of the SVI which is addressed via DHCP was performed to trigger the DORA exchange
For windows host you can do a ipconfig /release > ipconfig /renew

Collect the debugs from show logging or from the terminal window

DHCP DISCOVER

Discover is seen coming from the host facing port

*Sep 19 20:16:31.164: DHCP_SNOOPING: received new DHCP packet from input interface (GigabitEthernet1/0/1)  <-- host facing port

*Sep 19 20:16:31.177: DHCP_SNOOPING: process new DHCP packet, message type: DHCPDISCOVER, input interface: Gi1/0/1, MAC da: ffff.ffff.ffff, MAC sa: 0006.f601.cd43, IP da: 255.255.255.255, IP sa: 0.0.0.0, DHCP ciaddr: 0.0.0.0, DHCP yiaddr: 0.0.0.0, DHCP siaddr: 0.0.0.0, DHCP giaddr: 0.0.0.0, DHCP chaddr: 0006.f601.cd43, efp_id: 1412061046, vlan_id: 202, bootpflag:0x32768(Broadcast)
*Sep 19 20:16:31.177: DHCP_SNOOPING: add relay information option.
*Sep 19 20:16:31.177: DHCP_SNOOPING: Encoding opt82 CID in vlan-mod-port format         <-- Option 82 encoding 
*Sep 19 20:16:31.177: DHCP_SNOOPING:VxLAN : vlan_id 202 VNI 20201 mod 1 port 1
*Sep 19 20:16:31.177: DHCP_SNOOPING: Encoding opt82 RID in MAC address format           <-- Encoding the switch Remote ID (local MAC)
*Sep 19 20:16:31.177: DHCP_SNOOPING: binary dump of relay info option, length: 26 data:
0x52 0x18 0x1 0xC 0x1 0xA 0x0 0x8 0x0 0x0 0x4E 0xE9 0x1 0x1 0x0 0x0 0x2 0x8 0x0 0x6 0x68 0x2C 0x7B 0xF8 0x87 0x0  <-- the switch local MAC 682c.7bf8.8700 
*Sep 19 20:16:31.177: DHCPS BRIDGE PAK: vlan=202 platform_flags=1
*Sep 19 20:16:31.177: DHCP_SNOOPING: bridge packet get invalid mat entry: FFFF.FFFF.FFFF, packet is flooded to ingress VLAN: (202)
*Sep 19 20:16:31.177: DHCP_SNOOPING: L2RELAY: sent unicast packet to default gw: 0000.beef.cafe vlan 0 src intf GigabitEthernet1/0/1  <-- indicates the L2 Relay function has picked up the packet and sent it to the CGW

DHCP OFFER

Offer is seen arriving from the fabric Tunnel interface

*Sep 19 20:16:33.180: DHCP_SNOOPING: received new DHCP packet from input interface (Tunnel0)
*Sep 19 20:16:33.194: DHCP_SNOOPING: process new DHCP packet, message type: DHCPOFFER, input interface: Tu0, MAC da: 0006.f601.cd43, MAC sa: 0000.beef.cafe, IP da: 255.255.255.255, IP sa: 10.1.202.1, DHCP ciaddr: 0.0.0.0, DHCP yiaddr: 10.1.202.18, DHCP siaddr: 0.0.0.0, DHCP giaddr: 10.1.202.1, DHCP chaddr: 0006.f601.cd43, efp_id: 1412061046, vlan_id: 202, bootpflag:0x32768(Broadcast)
*Sep 19 20:16:33.194: DHCP_SNOOPING: binary dump of option 82, length: 26 data:
0x52 0x18 0x1 0xC 0x1 0xA 0x0 0x8 0x0 0x0 0x4E 0xE9 0x1 0x1 0x0 0x0 0x2 0x8 0x0 0x6 0x68 0x2C 0x7B 0xF8 0x87 0x0
*Sep 19 20:16:33.194: DHCP_SNOOPING: binary dump of extracted circuit id, length: 14 data:
0x1 0xC 0x1 0xA 0x0 0x8 0x0 0x0 0x4E 0xE9 0x1 0x1 0x0 0x0
*Sep 19 20:16:33.194: DHCP_SNOOPING: binary dump of extracted remote id, length: 10 data:
0x2 0x8 0x0 0x6 0x68 0x2C 0x7B 0xF8 0x87 0x0                                 <-- the switch local MAC 682c.7bf8.8700
*Sep 19 20:16:33.194:  actual_fmt_cid OPT82_FMT_CID_VXLAN_MOD_PORT_INTF global_opt82_fmt_rid OPT82_FMT_RID_DEFAULT_GLOBAL global_opt82_fmt_cid OPT82_FMT_CID_DEFAULT_GLOBAL cid: sub_option_length 12
*Sep 19 20:16:33.194: dhcp_snooping_platform_is_local_dhcp_packet: VXLAN-MOD-PORT opt82 vni 20201, vlan_id 202
*Sep 19 20:16:33.194: DHCP_SNOOPING: opt82 data indicates local packet      <-- switch found its own RID in Option 82 parameters 
*Sep 19 20:16:33.194: DHCP_SNOOPING: remove relay information option.
*Sep 19 20:16:33.194: DHCP_SNOOPING opt82_fmt_cid_intf OPT82_FMT_CID_VXLAN_MOD_PORT_INTF opt82_fmt_cid_global OPT82_FMT_CID_DEFAULT_GLOBAL cid: sub_option_length 12
*Sep 19 20:16:33.194: DHCP_SNOOPING: VxLAN vlan_id 202 VNI 20201 mod 1 port 1
*Sep 19 20:16:33.194: DHCP_SNOOPING: mod 1 port 1 idb Gi1/0/1 found for 0006.f601.cd43
*Sep 19 20:16:33.194: DHCP_SNOOPING: calling forward_dhcp_reply
*Sep 19 20:16:33.194: platform lookup dest vlan for input_if: Tunnel0, is  tunnel, if_output: NULL, if_output->vlan_id: 99999, pak->vlan_id: 202
*Sep 19 20:16:33.194: DHCP_SNOOPING opt82_fmt_cid_intf OPT82_FMT_CID_VXLAN_MOD_PORT_INTF opt82_fmt_cid_global OPT82_FMT_CID_DEFAULT_GLOBAL cid: sub_option_length 12
*Sep 19 20:16:33.194: DHCP_SNOOPING: VxLAN vlan_id 202 VNI 20201 mod 1 port 1
*Sep 19 20:16:33.194: DHCP_SNOOPING: mod 1 port 1 idb Gi1/0/1 found for 0006.f601.cd43
*Sep 19 20:16:33.194: DHCP_SNOOPING: vlan 202 after pvlan check
*Sep 19 20:16:33.207: DHCP_SNOOPING: direct forward dhcp replyto output port: GigabitEthernet1/0/1.  <-- sending packet to host port

DHCP REQUEST

Request is seen from the host facing port

*Sep 19 20:16:33.209: DHCP_SNOOPING: received new DHCP packet from input interface (GigabitEthernet1/0/1)
*Sep 19 20:16:33.222: DHCP_SNOOPING: process new DHCP packet, message type: DHCPREQUEST, input interface: Gi1/0/1, MAC da: ffff.ffff.ffff, MAC sa: 0006.f601.cd43, IP da: 255.255.255.255, IP sa: 0.0.0.0, DHCP ciaddr: 0.0.0.0, DHCP yiaddr: 0.0.0.0, DHCP siaddr: 0.0.0.0, DHCP giaddr: 0.0.0.0, DHCP chaddr: 0006.f601.cd43, efp_id: 1412061046, vlan_id: 202, bootpflag:0x32768(Broadcast)
*Sep 19 20:16:33.222: DHCP_SNOOPING: add relay information option.
*Sep 19 20:16:33.222: DHCP_SNOOPING: Encoding opt82 CID in vlan-mod-port format
*Sep 19 20:16:33.222: DHCP_SNOOPING:VxLAN : vlan_id 202 VNI 20201 mod 1 port 1
*Sep 19 20:16:33.222: DHCP_SNOOPING: Encoding opt82 RID in MAC address format
*Sep 19 20:16:33.222: DHCP_SNOOPING: binary dump of relay info option, length: 26 data:
0x52 0x18 0x1 0xC 0x1 0xA 0x0 0x8 0x0 0x0 0x4E 0xE9 0x1 0x1 0x0 0x0 0x2 0x8 0x0 0x6 0x68 0x2C 0x7B 0xF8 0x87 0x0
*Sep 19 20:16:33.222: DHCPS BRIDGE PAK: vlan=202 platform_flags=1
*Sep 19 20:16:33.222: DHCP_SNOOPING: bridge packet get invalid mat entry: FFFF.FFFF.FFFF, packet is flooded to ingress VLAN: (202)
*Sep 19 20:16:33.222: DHCP_SNOOPING: L2RELAY: sent unicast packet to default gw: 0000.beef.cafe vlan 0 src intf GigabitEthernet1/0/1

DHCP ACK

Ack is seen arriving from the fabric Tunnel interface

*Sep 19 20:16:33.225: DHCP_SNOOPING: received new DHCP packet from input interface (Tunnel0)
*Sep 19 20:16:33.238: DHCP_SNOOPING: process new DHCP packet, message type: DHCPACK, input interface: Tu0, MAC da: 0006.f601.cd43, MAC sa: 0000.beef.cafe, IP da: 255.255.255.255, IP sa: 10.1.202.1, DHCP ciaddr: 0.0.0.0, DHCP yiaddr: 10.1.202.10, DHCP siaddr: 0.0.0.0, DHCP giaddr: 10.1.202.1, DHCP chaddr: 0006.f601.cd43, efp_id: 1412061046, vlan_id: 202, bootpflag:0x32768(Broadcast)
*Sep 19 20:16:33.238: DHCP_SNOOPING: binary dump of option 82, length: 26 data:
0x52 0x18 0x1 0xC 0x1 0xA 0x0 0x8 0x0 0x0 0x4E 0xE9 0x1 0x1 0x0 0x0 0x2 0x8 0x0 0x6 0x68 0x2C 0x7B 0xF8 0x87 0x0
*Sep 19 20:16:33.239: DHCP_SNOOPING: binary dump of extracted circuit id, length: 14 data:
0x1 0xC 0x1 0xA 0x0 0x8 0x0 0x0 0x4E 0xE9 0x1 0x1 0x0 0x0
*Sep 19 20:16:33.239: DHCP_SNOOPING: binary dump of extracted remote id, length: 10 data:
0x2 0x8 0x0 0x6 0x68 0x2C 0x7B 0xF8 0x87 0x0
*Sep 19 20:16:33.239:  actual_fmt_cid OPT82_FMT_CID_VXLAN_MOD_PORT_INTF global_opt82_fmt_rid OPT82_FMT_RID_DEFAULT_GLOBAL global_opt82_fmt_cid OPT82_FMT_CID_DEFAULT_GLOBAL cid: sub_option_length 12
*Sep 19 20:16:33.239: dhcp_snooping_platform_is_local_dhcp_packet: VXLAN-MOD-PORT opt82 vni 20201, vlan_id 202
*Sep 19 20:16:33.239: DHCP_SNOOPING: opt82 data indicates local packet
*Sep 19 20:16:33.239: dhcp_snooping_platform_is_local_dhcp_packet: VXLAN-MOD-PORT opt82 vni 20201, vlan_id 202
*Sep 19 20:16:33.239: DHCP_SNOOPING: opt82 data indicates local packet
*Sep 19 20:16:33.239: DHCP_SNOOPING opt82_fmt_cid_intf OPT82_FMT_CID_VXLAN_MOD_PORT_INTF opt82_fmt_cid_global OPT82_FMT_CID_DEFAULT_GLOBAL cid: sub_option_length 12
*Sep 19 20:16:33.239: DHCP_SNOOPING: VxLAN vlan_id 202 VNI 20201 mod 1 port 1
*Sep 19 20:16:33.239: DHCP_SNOOPING: mod 1 port 1 idb Gi1/0/1 found for 0006.f601.cd43
*Sep 19 20:16:33.239: DHCP_SNOOPING: Reroute dhcp pak, message type: DHCPACK
*Sep 19 20:16:33.239: DHCP_SNOOPING: remove relay information option.
*Sep 19 20:16:33.239: DHCP_SNOOPING opt82_fmt_cid_intf OPT82_FMT_CID_VXLAN_MOD_PORT_INTF opt82_fmt_cid_global OPT82_FMT_CID_DEFAULT_GLOBAL cid: sub_option_length 12
*Sep 19 20:16:33.239: DHCP_SNOOPING: VxLAN vlan_id 202 VNI 20201 mod 1 port 1
*Sep 19 20:16:33.239: DHCP_SNOOPING: mod 1 port 1 idb Gi1/0/1 found for 0006.f601.cd43
*Sep 19 20:16:33.239: DHCP_SNOOPING: calling forward_dhcp_reply
*Sep 19 20:16:33.239: platform lookup dest vlan for input_if: Tunnel0, is  tunnel, if_output: NULL, if_output->vlan_id: 99999, pak->vlan_id: 202
*Sep 19 20:16:33.239: DHCP_SNOOPING opt82_fmt_cid_intf OPT82_FMT_CID_VXLAN_MOD_PORT_INTF opt82_fmt_cid_global OPT82_FMT_CID_DEFAULT_GLOBAL cid: sub_option_length 12
*Sep 19 20:16:33.239: DHCP_SNOOPING: VxLAN vlan_id 202 VNI 20201 mod 1 port 1
*Sep 19 20:16:33.239: DHCP_SNOOPING: mod 1 port 1 idb Gi1/0/1 found for 0006.f601.cd43
*Sep 19 20:16:33.239: DHCP_SNOOPING: vlan 202 after pvlan check
*Sep 19 20:16:33.252: DHCP_SNOOPING: direct forward dhcp replyto output port: GigabitEthernet1/0/1.

Note: These debugs are snipped. They produce a memory dump of the packet, but annotation of this part of the debug result is outside the scope of this document.

DHCP Snooping Debugs (CGW)

DHCP DISCOVER

Because of how the packet is sent out and received back on the CGW (hairpinned at the firewall) the debugs fire twice

Arriving from fabric on Tunnel interface & sent out Tw 1/0/1 toward Firewall in Fabric vlan 202

*Apr 16 14:37:43.890: DHCP_SNOOPING: received new DHCP packet from input interface (Tunnel0)  <-- Discover sent from Leaf01 arrives at Tunnel interface of CGW
*Apr 16 14:37:43.901: DHCP_SNOOPING: process new DHCP packet, message type: DHCPDISCOVER, input interface: Tu0, MAC da: 0000.beef.cafe, MAC sa: 0006.f601.cd43, IP da: 255.255.255.255, IP sa: 0.0.0.0, DHCP ciaddr: 0.0.0.0, DHCP yiaddr: 0.0.0.0, DHCP siaddr: 0.0.0.0, DHCP giaddr: 0.0.0.0, DHCP chaddr: 0006.f601.cd43, efp_id: 109663498, vlan_id: 202, bootpflag:0x32768(Broadcast)
*Apr 16 14:37:43.901: DHCPS BRIDGE PAK: vlan=202 platform_flags=1
*Apr 16 14:37:43.901: DHCP_SNOOPING: bridge packet send packet to port: TwentyFiveGigE1/0/1, pak_vlan 202. <-- Sent to Firewall interface where the CGW MAC is learned in Vlan 202

Arriving from Firewall on Tw 1/0/2 in Vlan 2021 to be sent to the SVI & helper to DHCP server

*Apr 16 14:37:43.901: DHCP_SNOOPING: received new DHCP packet from input interface (TwentyFiveGigE1/0/2) <-- Firewall sends discover back to switch on the Ext GW vlan 2021
*Apr 16 14:37:43.911: DHCP_SNOOPING: process new DHCP packet, message type: DHCPDISCOVER, input interface: Twe1/0/2, MAC da: 0000.beef.cafe, MAC sa: 0006.f601.cd43, IP da: 255.255.255.255, IP sa: 0.0.0.0, DHCP ciaddr: 0.0.0.0, DHCP yiaddr: 0.0.0.0, DHCP siaddr: 0.0.0.0, DHCP giaddr: 0.0.0.0, DHCP chaddr: 0006.f601.cd43, efp_id: 109663498, vlan_id: 2021, bootpflag:0x32768(Broadcast)
*Apr 16 14:37:43.911: DHCPS BRIDGE PAK: vlan=2021 platform_flags=1      <-- Vlan discover seen is now 2021 
*Apr 16 14:37:43.911: DHCP_SNOOPING: Packet destined to SVI Mac:0000.beef.cafe
*Apr 16 14:37:43.911: DHCP_SNOOPING: bridge packet send packet to cpu port: Vlan2021.  <-- Packet punted to CPU for handling by DHCP helper

DHCP OFFER

Arrives from DHCP server back to the SVI 2021 where the helper is configured and forwarded to firewall

*Apr 16 14:37:45.913: DHCP_SNOOPING: received new DHCP packet from input interface (Vlan2021)  <-- Arriving from the DHCP server to the SVI
*Apr 16 14:37:45.923: DHCP_SNOOPING: process new DHCP packet, message type: DHCPOFFER, input interface: Vl2021, MAC da: ffff.ffff.ffff, MAC sa: 0000.beef.cafe, IP da: 255.255.255.255, IP sa: 10.1.202.1, DHCP ciaddr: 0.0.0.0, DHCP yiaddr: 10.1.202.19, DHCP siaddr: 0.0.0.0, DHCP giaddr: 10.1.202.1, DHCP chaddr: 0006.f601.cd43, efp_id: 109663498, vlan_id: 2021, bootpflag:0x32768(Broadcast)
*Apr 16 14:37:45.923: DHCP_SNOOPING: binary dump of option 82, length: 26 data:
0x52 0x18 0x1 0xC 0x1 0xA 0x0 0x8 0x0 0x0 0x4E 0xE9 0x1 0x1 0x0 0x0 0x2 0x8 0x0 0x6 0x68 0x2C 0x7B 0xF8 0x87 0x0
*Apr 16 14:37:45.924: DHCP_SNOOPING: binary dump of extracted circuit id, length: 14 data:
0x1 0xC 0x1 0xA 0x0 0x8 0x0 0x0 0x4E 0xE9 0x1 0x1 0x0 0x0
*Apr 16 14:37:45.924: DHCP_SNOOPING: binary dump of extracted remote id, length: 10 data:
0x2 0x8 0x0 0x6 0x68 0x2C 0x7B 0xF8 0x87 0x0
*Apr 16 14:37:45.924: actual_fmt_cid OPT82_FMT_CID_VXLAN_MOD_PORT_INTF global_opt82_fmt_rid OPT82_FMT_RID_DEFAULT_GLOBAL global_opt82_fmt_cid OPT82_FMT_CID_DEFAULT_GLOBAL cid: sub_option_length 12
*Apr 16 14:37:45.924: dhcp_snooping_platform_is_local_dhcp_packet: VXLAN-MOD-PORT opt82 vni 20201, vlan_id 202
*Apr 16 14:37:45.924: DHCP_SNOOPING: opt82 data indicates not a local packet
*Apr 16 14:37:45.924: DHCP_SNOOPING: can't parse option 82 data of the message,it is either in wrong format or not inserted by local switch <-- This is expected even in working scenario (disregard it)
*Apr 16 14:37:45.924: DHCP_SNOOPING: calling forward_dhcp_reply
*Apr 16 14:37:45.924: platform lookup dest vlan for input_if: Vlan2021, is NOT tunnel, if_output: Vlan2021, if_output->vlan_id: 2021, pak->vlan_id: 2021
*Apr 16 14:37:45.924: DHCP_SNOOPING: vlan 2021 after pvlan check
*Apr 16 14:37:45.934: DHCP_SNOOPING: direct forward dhcp replyto output port: TwentyFiveGigE1/0/2. <-- sending back toward the Firewall

Arrives from Firewall in Fabric vlan and sent from CGW into fabric toward Leaf

*Apr 16 14:37:45.934: DHCP_SNOOPING: received new DHCP packet from input interface (TwentyFiveGigE1/0/1)
*Apr 16 14:37:45.944: DHCP_SNOOPING: process new DHCP packet, message type: DHCPOFFER, input interface: Twe1/0/1, MAC da: ffff.ffff.ffff, MAC sa: 0000.beef.cafe, IP da: 255.255.255.255, IP sa: 10.1.202.1, DHCP ciaddr: 0.0.0.0, DHCP yiaddr: 10.1.202.19, DHCP siaddr: 0.0.0.0, DHCP giaddr: 10.1.202.1, DHCP chaddr: 0006.f601.cd43, efp_id: 109663498, vlan_id: 202, bootpflag:0x32768(Broadcast)
*Apr 16 14:37:45.944: DHCP_SNOOPING: binary dump of option 82, length: 26 data:
0x52 0x18 0x1 0xC 0x1 0xA 0x0 0x8 0x0 0x0 0x4E 0xE9 0x1 0x1 0x0 0x0 0x2 0x8 0x0 0x6 0x68 0x2C 0x7B 0xF8 0x87 0x0
*Apr 16 14:37:45.944: DHCP_SNOOPING: binary dump of extracted circuit id, length: 14 data:
0x1 0xC 0x1 0xA 0x0 0x8 0x0 0x0 0x4E 0xE9 0x1 0x1 0x0 0x0
*Apr 16 14:37:45.944: DHCP_SNOOPING: binary dump of extracted remote id, length: 10 data:
0x2 0x8 0x0 0x6 0x68 0x2C 0x7B 0xF8 0x87 0x0
*Apr 16 14:37:45.944: actual_fmt_cid OPT82_FMT_CID_VXLAN_MOD_PORT_INTF global_opt82_fmt_rid OPT82_FMT_RID_DEFAULT_GLOBAL global_opt82_fmt_cid OPT82_FMT_CID_DEFAULT_GLOBAL cid: sub_option_length 12
*Apr 16 14:37:45.944: dhcp_snooping_platform_is_local_dhcp_packet: VXLAN-MOD-PORT opt82 vni 20201, vlan_id 202
*Apr 16 14:37:45.945: DHCP_SNOOPING: opt82 data indicates not a local packet
*Apr 16 14:37:45.945: DHCP_SNOOPING: EVPN enabled Ex GW:fabric relay can't parse option 82 data of the message,it is either in wrong format or not inserted by local switch
*Apr 16 14:37:45.945: DHCP_SNOOPING: client address lookup failed to locate client interface, retry lookup using packet mac DA: ffff.ffff.ffff
*Apr 16 14:37:45.945: DHCP_SNOOPING: lookup packet destination port failed to get mat entry for mac: 0006.f601.cd43 vlan_id 202
*Apr 16 14:37:45.945: DHCP_SNOOPING: L2RELAY: Ex GW unicast bridge packet to fabric: vlan id 202 from Twe1/0/1  <-- L2 RELAY function passed back to fabric

DHCP REQUEST

*Apr 16 14:37:45.967: DHCP_SNOOPING: received new DHCP packet from input interface (Tunnel0)
*Apr 16 14:37:45.978: DHCP_SNOOPING: process new DHCP packet, message type: DHCPREQUEST, input interface: Tu0, MAC da: 0000.beef.cafe, MAC sa: 0006.f601.cd43, IP da: 255.255.255.255, IP sa: 0.0.0.0, DHCP ciaddr: 0.0.0.0, DHCP yiaddr: 0.0.0.0, DHCP siaddr: 0.0.0.0, DHCP giaddr: 0.0.0.0, DHCP chaddr: 0006.f601.cd43, efp_id: 109663498, vlan_id: 202, bootpflag:0x32768(Broadcast)
*Apr 16 14:37:45.978: DHCPS BRIDGE PAK: vlan=202 platform_flags=1
*Apr 16 14:37:45.978: DHCP_SNOOPING: bridge packet send packet to port: TwentyFiveGigE1/0/1, pak_vlan 202. <-- Send toward Firewall

*Apr 16 14:37:45.978: DHCP_SNOOPING: received new DHCP packet from input interface (TwentyFiveGigE1/0/2) <-- Receive from Firewall 
*Apr 16 14:37:45.989: DHCP_SNOOPING: process new DHCP packet, message type: DHCPREQUEST, input interface: Twe1/0/2, MAC da: 0000.beef.cafe, MAC sa: 0006.f601.cd43, IP da: 255.255.255.255, IP sa: 0.0.0.0, DHCP ciaddr: 0.0.0.0, DHCP yiaddr: 0.0.0.0, DHCP siaddr: 0.0.0.0, DHCP giaddr: 0.0.0.0, DHCP chaddr: 0006.f601.cd43, efp_id: 109663498, vlan_id: 2021, bootpflag:0x32768(Broadcast)
*Apr 16 14:37:45.989: DHCPS BRIDGE PAK: vlan=2021 platform_flags=1
*Apr 16 14:37:45.989: DHCP_SNOOPING: Packet destined to SVI Mac:0000.beef.cafe
*Apr 16 14:37:45.989: DHCP_SNOOPING: bridge packet send packet to cpu port: Vlan2021.  <-- Punt to CPU / DHCP helper

DHCP ACK

*Apr 16 14:37:45.990: DHCP_SNOOPING: received new DHCP packet from input interface (Vlan2021)  <-- Packet back to SVI from DHCP server 
*Apr 16 14:37:46.000: DHCP_SNOOPING: process new DHCP packet, message type: DHCPACK, input interface: Vl2021, MAC da: ffff.ffff.ffff, MAC sa: 0000.beef.cafe, IP da: 255.255.255.255, IP sa: 10.1.202.1, DHCP ciaddr: 0.0.0.0, DHCP yiaddr: 10.1.202.19, DHCP siaddr: 0.0.0.0, DHCP giaddr: 10.1.202.1, DHCP chaddr: 0006.f601.cd43, efp_id: 109663498, vlan_id: 2021, bootpflag:0x32768(Broadcast)
*Apr 16 14:37:46.000: DHCP_SNOOPING: binary dump of option 82, length: 26 data:
0x52 0x18 0x1 0xC 0x1 0xA 0x0 0x8 0x0 0x0 0x4E 0xE9 0x1 0x1 0x0 0x0 0x2 0x8 0x0 0x6 0x68 0x2C 0x7B 0xF8 0x87 0x0
*Apr 16 14:37:46.000: DHCP_SNOOPING: binary dump of extracted circuit id, length: 14 data:
0x1 0xC 0x1 0xA 0x0 0x8 0x0 0x0 0x4E 0xE9 0x1 0x1 0x0 0x0
*Apr 16 14:37:46.000: DHCP_SNOOPING: binary dump of extracted remote id, length: 10 data:
0x2 0x8 0x0 0x6 0x68 0x2C 0x7B 0xF8 0x87 0x0
*Apr 16 14:37:46.001: actual_fmt_cid OPT82_FMT_CID_VXLAN_MOD_PORT_INTF global_opt82_fmt_rid OPT82_FMT_RID_DEFAULT_GLOBAL global_opt82_fmt_cid OPT82_FMT_CID_DEFAULT_GLOBAL cid: sub_option_length 12
*Apr 16 14:37:46.001: dhcp_snooping_platform_is_local_dhcp_packet: VXLAN-MOD-PORT opt82 vni 20201, vlan_id 202
*Apr 16 14:37:46.001: DHCP_SNOOPING: opt82 data indicates not a local packet   <-- found this is coming from Leaf01 RID 
*Apr 16 14:37:46.001: DHCP_SNOOPING: can't parse option 82 data of the message,it is either in wrong format or not inserted by local switch
*Apr 16 14:37:46.001: DHCP_SNOOPING: calling forward_dhcp_reply
*Apr 16 14:37:46.001: platform lookup dest vlan for input_if: Vlan2021, is NOT tunnel, if_output: Vlan2021, if_output->vlan_id: 2021, pak->vlan_id: 2021
*Apr 16 14:37:46.001: DHCP_SNOOPING: vlan 2021 after pvlan check
*Apr 16 14:37:46.011: DHCP_SNOOPING: direct forward dhcp replyto output port: TwentyFiveGigE1/0/2.  <-- Send to Firewall

*Apr 16 14:37:46.011: DHCP_SNOOPING: received new DHCP packet from input interface (TwentyFiveGigE1/0/1)  <-- Coming back in from Firewall
*Apr 16 14:37:46.022: DHCP_SNOOPING: process new DHCP packet, message type: DHCPACK, input interface: Twe1/0/1, MAC da: ffff.ffff.ffff, MAC sa: 0000.beef.cafe, IP da: 255.255.255.255, IP sa: 10.1.202.1, DHCP ciaddr: 0.0.0.0, DHCP yiaddr: 10.1.202.19, DHCP siaddr: 0.0.0.0, DHCP giaddr: 10.1.202.1, DHCP chaddr: 0006.f601.cd43, efp_id: 109663498, vlan_id: 202, bootpflag:0x32768(Broadcast)
*Apr 16 14:37:46.022: DHCP_SNOOPING: binary dump of option 82, length: 26 data:
0x52 0x18 0x1 0xC 0x1 0xA 0x0 0x8 0x0 0x0 0x4E 0xE9 0x1 0x1 0x0 0x0 0x2 0x8 0x0 0x6 0x68 0x2C 0x7B 0xF8 0x87 0x0
*Apr 16 14:37:46.022: DHCP_SNOOPING: binary dump of extracted circuit id, length: 14 data:
0x1 0xC 0x1 0xA 0x0 0x8 0x0 0x0 0x4E 0xE9 0x1 0x1 0x0 0x0
*Apr 16 14:37:46.022: DHCP_SNOOPING: binary dump of extracted remote id, length: 10 data:
0x2 0x8 0x0 0x6 0x68 0x2C 0x7B 0xF8 0x87 0x0
*Apr 16 14:37:46.022: actual_fmt_cid OPT82_FMT_CID_VXLAN_MOD_PORT_INTF global_opt82_fmt_rid OPT82_FMT_RID_DEFAULT_GLOBAL global_opt82_fmt_cid OPT82_FMT_CID_DEFAULT_GLOBAL cid: sub_option_length 12
*Apr 16 14:37:46.022: dhcp_snooping_platform_is_local_dhcp_packet: VXLAN-MOD-PORT opt82 vni 20201, vlan_id 202
*Apr 16 14:37:46.022: DHCP_SNOOPING: opt82 data indicates not a local packet
*Apr 16 14:37:46.022: DHCP_SNOOPING: EVPN enabled Ex GW:fabric relay can't parse option 82 data of the message,it is either in wrong format or not inserted by local switch
*Apr 16 14:37:46.022: DHCP_SNOOPING: client address lookup failed to locate client interface, retry lookup using packet mac DA: ffff.ffff.ffff
*Apr 16 14:37:46.022: DHCP_SNOOPING: lookup packet destination port failed to get mat entry for mac: 0006.f601.cd43 vlan_id 202
*Apr 16 14:37:46.022: DHCP_SNOOPING: can't find client's destination port, packet is assumed to be not from local switch, no binding update is needed.
*Apr 16 14:37:46.022: DHCP_SNOOPING: client address lookup failed to locate client interface, retry lookup using packet mac DA: ffff.ffff.ffff
*Apr 16 14:37:46.022: DHCP_SNOOPING: lookup packet destination port failed to get mat entry for mac: 0006.f601.cd43 vlan_id 202
*Apr 16 14:37:46.022: DHCP_SNOOPING: L2RELAY: Ex GW unicast bridge packet to fabric: vlan id 202 from Twe1/0/1  <-- Send packet back into fabric

Embedded Capture

Use EPC to confirm DHCP packet exchange & parameters are correct

This is shown from the perspective of the CGW, but the process can be repeated on Leaf to verify the packet exchange
This example shows the Discover as the process and analysis are the same for the other DHCP packets

Check the route to the Leaf Loopback

CGW#show ip route 172.16.254.3
Routing entry for 172.16.254.3/32
  Known via "ospf 1", distance 110, metric 3, type intra area
  Last update from 172.16.1.25 on TwentyFiveGigE1/0/47, 2w6d ago
  Routing Descriptor Blocks:
  * 172.16.1.29, from 172.16.255.3, 2w6d ago, via TwentyFiveGigE1/0/48
      Route metric is 3, traffic share count is 1
    172.16.1.25, from 172.16.255.3, 2w6d ago, via TwentyFiveGigE1/0/47
      Route metric is 3, traffic share count is 1

Configure capture to run on links facing the Leaf01

   monitor capture 1 interface TwentyFiveGigE1/0/47 BOTH
   monitor capture 1 interface TwentyFiveGigE1/0/48 BOTH
   monitor capture 1 match any
   monitor capture 1 buffer size 100
   monitor capture 1 limit pps 1000

Start the capture, trigger your host to request a DHCP IP address, stop the capture

monitor capture 1 start
  (have the host request dhcp ip)
monitor capture 1 stop

View the capture result starting with the DHCP Discover (Pay attention to the Transaction ID to confirm this is all the same DORA event)

CGW#show monitor cap 1 buff brief | i DHCP
   16  12.737135      0.0.0.0 -> 255.255.255.255 DHCP 434 DHCP Discover - Transaction ID 0x78b <-- Discover starts at Frame 16 with all same transaction ID
   18  14.740041   10.1.202.1 -> 255.255.255.255 DHCP 438 DHCP Offer    - Transaction ID 0x78b
   19  14.742741      0.0.0.0 -> 255.255.255.255 DHCP 452 DHCP Request  - Transaction ID 0x78b
   20  14.745646   10.1.202.1 -> 255.255.255.255 DHCP 438 DHCP ACK      - Transaction ID 0x78b

CGW#sh mon cap 1 buff detailed | b Frame 16

Frame 16: 434 bytes on wire (3472 bits), 434 bytes captured (3472 bits) on interface /tmp/epc_ws/wif_to_ts_pipe, id 0
    [Protocols in frame: eth:ethertype:ip:udp:vxlan:eth:ethertype:ip:udp:dhcp]
Ethernet II, Src: dc:77:4c:8a:6d:7f (dc:77:4c:8a:6d:7f), Dst: 10:f9:20:2e:9f:82 (10:f9:20:2e:9f:82) <-- Underlay Interface MACs
    Type: IPv4 (0x0800)
Internet Protocol Version 4, Src: 172.16.254.3, Dst: 172.16.254.6  
User Datagram Protocol, Src Port: 65281, Dst Port: 4789                  <-- VXLAN Port
Virtual eXtensible Local Area Network
    VXLAN Network Identifier (VNI): 20201                                <-- Correct VNI / Segment 
    Reserved: 0
Ethernet II, Src: 00:06:f6:01:cd:43 (00:06:f6:01:cd:43), Dst: 00:00:be:ef:ca:fe (00:00:be:ef:ca:fe) <-- Inner Packet destined to CGW MAC 
Internet Protocol Version 4, Src: 0.0.0.0, Dst: 255.255.255.255
User Datagram Protocol, Src Port: 68, Dst Port: 67                       <-- DHCP ports

Dynamic Host Configuration Protocol (Discover)                           <-- DHCP Discover Packet
    Client MAC address: 00:06:f6:01:cd:43 (00:06:f6:01:cd:43)
    Client hardware address padding: 00000000000000000000
    Server host name not given
    Boot file name not given
    Magic cookie: DHCP
    Option: (53) DHCP Message Type (Discover)
        Length: 1
        DHCP: Discover (1)
    Option: (57) Maximum DHCP Message Size
        Length: 2
        Maximum DHCP Message Size: 1152
    Option: (61) Client identifier
        Length: 27
        Type: 0
        Client Identifier: cisco-0006.f601.cd43-Vl202
    Option: (12) Host Name
        Length: 17
        Host Name: 9300-HOST-3750X-2
    Option: (55) Parameter Request List
        Length: 8
        Parameter Request List Item: (1) Subnet Mask
        Parameter Request List Item: (6) Domain Name Server
        Parameter Request List Item: (15) Domain Name
        Parameter Request List Item: (44) NetBIOS over TCP/IP Name Server
        Parameter Request List Item: (3) Router
        Parameter Request List Item: (33) Static Route
        Parameter Request List Item: (150) TFTP Server Address
        Parameter Request List Item: (43) Vendor-Specific Information
    Option: (60) Vendor class identifier
        Length: 8
        Vendor class identifier: ciscopnp
    Option: (82) Agent Information Option
        Length: 24
        Option 82 Suboption: (1) Agent Circuit ID
            Length: 12
            Agent Circuit ID: 010a000800004ee901010000
        Option 82 Suboption: (2) Agent Remote ID
            Length: 8
            Agent Remote ID: 0006682c7bf88700  <-- switch base mac 682c.7bf8.8700 (from 'show switch')
    Option: (255) End
        Option End: 255

Note: Capture tool can be used on any Leafs or CGW to determine the last point a part of the DHCP DORA exchange is suspected to be failing.

Verify snooping statistics for errors

Leaf01#show ip dhcp snooping statistics detail
 Packets Processed by DHCP Snooping                    = 1288
 Packets Dropped Because
   IDB not known                                       = 0
   Queue full                                          = 0
   Interface is in errdisabled                         = 0
   Rate limit exceeded                                 = 0
   Received on untrusted ports                         = 0
   Nonzero giaddr                                      = 0
   Source mac not equal to chaddr                      = 0
   No binding entry                                    = 0
   Insertion of opt82 fail                             = 0
   Unknown packet                                      = 0
   Interface Down                                      = 0
   Unknown output interface                            = 0
   Misdirected Packets                                 = 0
   Packets with Invalid Size                           = 0
   Packets with Invalid Option                         = 0
<-- Look for any drop counter that is actively incrementing when the issue is seen.

Verify punt path for DHCP Snooping

CoPP is the main component that drops packets in the punt path

Leaf01#show platform hardware switch active qos queue stats internal cpu policer

                         CPU Queue Statistics
============================================================================================
                                              (default) (set)     Queue        Queue
QId PlcIdx  Queue Name                Enabled   Rate     Rate      Drop(Bytes)  Drop(Frames)
--------------------------------------------------------------------------------------------
17   6      DHCP Snooping               Yes     400       400      0            0
  
              CPU Queue Policer Statistics
====================================================================
Policer    Policer Accept   Policer Accept  Policer Drop  Policer Drop
  Index         Bytes          Frames        Bytes          Frames
-------------------------------------------------------------------
6          472723           1288            0             0

Another very helpful command to locate where a possible packet flood is occurring is 'show platform software fed switch active punt rates interfaces"

This is very helpful to find a source interface where flooding is happening which is congesting the punt path and affecting legitimate CPU bound traffic

Leaf01#show platform software fed switch active punt rates interfaces
Punt Rate on Interfaces Statistics

Packets per second averaged over 10 seconds, 1 min and 5 mins
===========================================================================================
                             |           |  Recv |  Recv |  Recv |  Drop |  Drop  |  Drop   <-- Receive and drop rates for this port
 Interface Name              |  IF_ID    |  10s  |  1min |  5min |  10s  |  1min  |  5min
===========================================================================================
 GigabitEthernet1/0/1          0x0000000a       2       2       2       0       0       0   <-- the port and its IF-ID which can be used in the next command

-------------------------------------------------------------------------------------------

Leaf01#show platform software fed switch active punt rates interfaces 0xa  <-- From previous command (omit the leading zeros)
Punt Rate on Single Interfaces Statistics

Interface : GigabitEthernet1/0/1 [if_id: 0xA]

  Received                         Dropped
  --------                         -------
   Total          : 8032546          Total           : 0
   10 sec average : 2                10 sec average  : 0
    1 min average : 2                 1 min average  : 0
    5 min average : 2                 5 min average  : 0

Per CPUQ punt stats on the interface (rate averaged over 10s interval)
==========================================================================
 Q  |         Queue                 |  Recv   |  Recv  |  Drop  |  Drop  |
 no |         Name                  |  Total  |  Rate  |  Total |  Rate  |
==========================================================================
 17  CPU_Q_DHCP_SNOOPING                 1216        0        0        0
<...snip...>

DHCP Snooping Client Stats

Observe the DHCP message exchange using this command. This can be run on both Leaf or CGW to see the event trace

Leaf01#show platform dhcpsnooping client stats 0006.F601.CD43
DHCPSN: DHCP snooping server
DHCPD: DHCP protocol daemen
L2FWD: Transmit Packet to driver in L2 format
FWD: Transmit Packet to driver
(B): Dhcp message's response expected as 'B'roadcast
(U): Dhcp message's response expected as 'U'nicast
Packet Trace for client MAC 0006.F601.CD43:
Timestamp                Destination MAC  Destination Ip  VLAN  Message      Handler:Action
------------------------ ---------------- --------------- ----  ------------ ----------------
2023/09/28 14:53:59.866  FFFF.FFFF.FFFF   255.255.255.255 202  DHCPDISCOVER(B) PUNT:RECEIVED
2023/09/28 14:53:59.866  FFFF.FFFF.FFFF   255.255.255.255 202  DHCPDISCOVER(B) PUNT:TO_DHCPSN
2023/09/28 14:53:59.867  FFFF.FFFF.FFFF   255.255.255.255 202  DHCPDISCOVER(B) BRIDGE:RECEIVED
2023/09/28 14:53:59.867  0000.BEEF.CAFE   255.255.255.255 202  DHCPDISCOVER(B) L2INJECT:TO_FWD
2023/09/28 14:53:59.867  FFFF.FFFF.FFFF   255.255.255.255 202  DHCPDISCOVER(B) BRIDGE:TO_INJECT
2023/09/28 14:53:59.867  FFFF.FFFF.FFFF   255.255.255.255 202  DHCPDISCOVER(B) L2INJECT:TO_FWD
2023/09/28 14:54:01.871  0006.F601.CD43   255.255.255.255 202  DHCPOFFER(B)    PUNT:RECEIVED
2023/09/28 14:54:01.871  0006.F601.CD43   255.255.255.255 202  DHCPOFFER(B)    PUNT:TO_DHCPSN
2023/09/28 14:54:01.874  FFFF.FFFF.FFFF   255.255.255.255 202  DHCPREQUEST(B)  PUNT:RECEIVED
2023/09/28 14:54:01.874  FFFF.FFFF.FFFF   255.255.255.255 202  DHCPREQUEST(B)  PUNT:TO_DHCPSN
2023/09/28 14:54:01.874  FFFF.FFFF.FFFF   255.255.255.255 202  DHCPREQUEST(B)  BRIDGE:RECEIVED
2023/09/28 14:54:01.874  0000.BEEF.CAFE   255.255.255.255 202  DHCPREQUEST(B)  L2INJECT:TO_FWD
2023/09/28 14:54:01.874  FFFF.FFFF.FFFF   255.255.255.255 202  DHCPREQUEST(B)  BRIDGE:TO_INJECT
2023/09/28 14:54:01.874  FFFF.FFFF.FFFF   255.255.255.255 202  DHCPREQUEST(B)  L2INJECT:TO_FWD
2023/09/28 14:54:01.877  0006.F601.CD43   255.255.255.255 202  DHCPACK(B)      PUNT:RECEIVED
2023/09/28 14:54:01.877  0006.F601.CD43   255.255.255.255 202  DHCPACK(B)      PUNT:TO_DHCPSN

Additional Debugs

debug ip dhcp server packet detail
debug ip dhcp server packet
debug ip dhcp server events
debug ip dhcp snooping packet
debug dhcp detail

Caution: Use caution when running debugs!

Related Information

Revision History

Revision	Publish Date	Comments
1.0	17-Jan-2024	Initial Release

Contributed by Cisco Engineers

Ambrose Taylor

Was this Document Helpful?

Feedback

Contact Cisco

Open a Support Case
(Requires a Cisco Service Contract)

Implement BGP EVPN DHCP Layer 2 Relay on Catalyst 9000 Series Switches

Available Languages

Download Options

Bias-Free Language

Contents

Introduction

Prerequisites

Requirements

Components Used

Background Information

Document Details

L2 Relay Behavior

Terminology

Configure (Standard CGW Deployment)

Network Diagram

L2 VTEP (Leaf) Key Details

L3 VTEP (CGW) Key Details

L2VTEP

CGW

Verify (Standard CGW Deployment)

Gateway Prefix (Leaf)

FED MATM (Leaf)

Local MAC​ (Leaf)

DHCP Snooping (Leaf & CGW)

Configure (Partially Isolated Protected)

Network Diagram

L2 VTEP (Leaf) Key Details

L3 VTEP (CGW) Key Details

CGW

Verify (Partially Isolated Protected)

Gateway Prefix (Leaf)

FED MATM (Leaf)

Local MAC​ (Leaf)

DHCP Snooping (Leaf & CGW)

Troubleshoot (Any CGW Type)

DHCP Snooping Debugs (Leaf)

DHCP Snooping Debugs (CGW)

Embedded Capture

DHCP Snooping Client Stats

Additional Debugs

Related Information

Revision History

Contributed by Cisco Engineers

Was this Document Helpful?

Contact Cisco

This Document Applies to These Products

Local MAC (Leaf)

Local MAC (Leaf)