Tenant Routed Multicast (TRM) enables multicast forwarding on the VXLAN fabric that uses a BGP-based EVPN control plane. TRM provides multi-tenancy aware multicast forwarding between senders and receivers within the same or different subnet local or across VTEPs.

This feature brings the efficiency of multicast delivery to VXLAN overlays. It is based on the standards-based next generation control plane (ngMVPN) described in IETF RFC 6513, 6514. TRM enables the delivery of customer IP multicast traffic in a multitenant fabric, and thus in an efficient and resilient manner. The delivery of TRM improves Layer-3 overlay multicast functionality in our networks.

While BGP EVPN provides the control plane for unicast routing, ngMVPN provides scalable multicast routing functionality. It follows an “always route” approach where every edge device (VTEP) with distributed IP Anycast Gateway for unicast becomes a Designated Router (DR) for Multicast. Bridged multicast forwarding is only present on the edge-devices (VTEP) where IGMP snooping optimizes the multicast forwarding to interested receivers. Every other multicast traffic beyond local delivery is efficiently routed.

With TRM enabled, multicast forwarding in the underlay is leveraged to replicate VXLAN encapsulated routed multicast traffic. A Default Multicast Distribution Tree (Default-MDT) is built per-VRF. This is an addition to the existing multicast groups for Layer-2 VNI Broadcast, Unknown Unicast, and Layer-2 multicast replication group. The individual multicast group addresses in the overlay are mapped to the respective underlay multicast address for replication and transport. The advantage of using a BGP-based approach allows the VXLAN BGP EVPN fabric with TRM to operate as fully distributed Overlay Rendezvous-Point (RP), with the RP presence on every edge-device (VTEP).

A multicast-enabled data center fabric is typically part of an overall multicast network. Multicast sources, receivers, and multicast rendezvous points, might reside inside the data center but might also be inside the campus or externally reachable via the WAN. TRM allows a seamless integration with existing multicast networks. It can leverage multicast rendezvous points external to the fabric. Furthermore, TRM allows for tenant-aware external connectivity using Layer-3 physical interfaces or subinterfaces.

Beginning with Cisco NX-OS Release 10.2(1), Tenant Routed Multicast (TRM) supports IPv6 in the overlay.

Beginning with Cisco NX-OS Release 10.3(2)F, the Multicast Flow Path Visualization (FPV) for TRM Flows feature is supported for TRM L3 mode and underlay multicast along with the already supported multicast flows. This feature enables you to export all multicast states in a Cisco Nexus 9000 Series switch. This helps to have a complete and reliable traceability of the flow path from the source to a receiver. To enable Multicast Flow Path Data Export on Cisco Nexus 9000 Series switches, use the multicast flow-path export command.

Beginning with Cisco NX-OS Release 10.4(2)F, the support is provided for VXLAN with IPv6 Multicast in the Underlay. Hosts in the overlay can be IPv4 or IPv6. This requires IPv6 versions of the unicast routing protocols and using IPv6 multicast in the underlay (PIMv6). Any multi-destination overlay traffic (such as TRM, BUM) can use the IPv6 multicast underlay.

The above topology shows four leafs and two spines in a VXLAN EVPN fabric. The underlay is an IPv6 Multicast running PIMv6. RP is positioned in the spine with anycast RP.

Beginning with Cisco NX-OS Release 10.4(3)F, the combination of PIMv6 underlay on the fabric side and Ingress Replication (IPv6) on Data Center Interconnect (DCI) side is supported on Cisco Nexus 9300-FX/FX2/FX3/GX/GX2/H2R/H1 ToR switches and 9500 switches with X9716D-GX and X9736C-FX line cards.

Tenant Routed Multicast (TRM) has the following guidelines and limitations:

• Beginning with Cisco NX-OS Release 10.1(2), TRM Multisite with vPC BGW is supported.

• Beginning with Cisco NX-OS Release 10.2(1q)F, VXLAN TRM is supported on Cisco Nexus N9K-C9332D-GX2B platform switches.

• Beginning with Cisco NX-OS Release 10.2(3)F, VXLAN TRM is supported on Cisco Nexus 9364D-GX2A, and 9348D-GX2A platform switches.

• Beginning with Cisco NX-OS Release 10.4(1)F, VXLAN TRM is supported on Cisco Nexus 9332D-H2R switches.

• Beginning with Cisco NX-OS Release 10.4(2)F, VXLAN TRM is supported on Cisco Nexus 93400LD-H1 switches.

• Beginning with Cisco NX-OS Release 10.4(3)F, VXLAN TRM is supported on Cisco Nexus 9364C-H1 switches.

• With Tenant Routed Multicast enabled, FEX is not supported.

• If VXLAN TRM feature is enabled on a VTEP, it would stop to send IGMP messages to the VXLAN fabric.

• The Guidelines and Limitations for VXLAN also apply to TRM.

• With TRM enabled, SVI as a core link is not supported.

• If TRM is configured, ISSU is disruptive.

• TRM supports IPv4 and IPv6 multicast underlay.

• TRM supports overlay PIM ASM and PIM SSM only. PIM BiDir is not supported in the overlay.

• RP has to be configured either internal or external to the fabric.

• The internal RP must be configured on all TRM-enabled VTEPs including the border nodes.

• The external RP must be external to the border nodes.

• The RP must be configured within the VRF pointing to the external RP IP address (static RP). This ensures that unicast and multicast routing is enabled to reach the external RP in the given VRF.

• In a Transit Routing Multicast (TRM) deployment, the RP-on-stick model can sometimes lead to traffic drops if there is flapping on the Protocol Independent Multicast (PIM) enabled interface. Use the ip pim spt-switch-graceful command on the turnaround router that leads to the RP. This command allows for a graceful switch to the Shortest Path Tree (SPT) during flapping, which can minimize traffic drops.

• Replication of first packet is supported only on Cisco Nexus 9300 – EX, FX, FX2 family switches.

• Beginning with Cisco NX-OS Release 10.2(3)F, Replication of first packet is supported on the Cisco Nexus 9300-FX3 platform switches.

• TRM with Multi-Site is not supported on Cisco Nexus 9504-R platforms.

• TRM supports multiple border nodes. Reachability to an external RP/source via multiple border leaf switches is supported with ECMP and requires symmetric unicast routing.

• Both PIM and ip igmp snooping vxlan must be enabled on the L3 VNI's VLAN in a VXLAN vPC setup.

• For traffic streams with an internal source and external L3 receiver using an external RP, the external L3 receiver might send PIM S,G join requests to the internal source. Doing so triggers the recreation of S,G on the fabric FHR, and it can take up to 10 minutes for this S,G to be cleared.

• Beginning with Cisco NX-OS Release 10.3(1)F, the Real-time/flex statistics for TRM is supported on Cisco Nexus 9300-X Cloud Scale Switches.

• TRM supports vPC fabric peering leaf’s as well as vPC/Anycast BGW.

Layer 3 Tenant Routed Multicast (TRM) has the following configuration guidelines and limitations:

• When upgrading from Cisco NX-OS Release 9.3(3) to Cisco NX-OS Release 9.3(6), if you do not retain configurations of the TRM enabled VRFs from Cisco NX-OS Release 9.3(3), or if you create new VRFs after the upgrade, the auto-generation of ip multicast multipath s-g-hash next-hop-based CLI, when feature ngmvpn is enabled, will not happen. You must enable the CLI manually for each TRM enabled VRF.

• Layer 3 TRM is supported for Cisco Nexus 9200, 9300-EX, and 9300-FX/FX2/FX3/FXP and 9300-GX platform switches.

• Beginning with Cisco NX-OS Release 10.2(3)F, Layer 3 TRM is supported on the Cisco Nexus 9300-GX2 platform switches.

• Beginning with Cisco NX-OS Release 10.4(1)F, Layer 3 TRM is supported on the Cisco Nexus 9332D-H2R switches.

• Beginning with Cisco NX-OS Release 10.4(2)F, Layer 3 TRM is supported on the Cisco Nexus 93400LD-H1 switches.

• Beginning with Cisco NX-OS Release 10.4(3)F, Layer 3 TRM is supported on the Cisco Nexus 9364C-H1 switches.

• Beginning with Cisco NX-OS Release 9.3(7), Cisco Nexus N9K-C9316D-GX, N9K-C9364C-GX, and N9K-X9716D-GX platform switches support the combination of Layer 3 TRM and EVPN Multi-Site.

• Cisco Nexus 9300-GX platform switches do not support the combination of Layer 3 TRM and EVPN Multi-Site in Cisco NX-OS Release 9.3(5).

• Beginning with Cisco NX-OS Release 10.2(3)F, the combination of Layer 3 TRM and EVPN Multi-Site is supported on the Cisco Nexus 9300-GX2 platform switches.

• Beginning with Cisco NX-OS Release 10.4(1)F, the combination of Layer 3 TRM and EVPN Multi-Site is supported on the Cisco Nexus 9332D-H2R switches.

• Beginning with Cisco NX-OS Release 10.4(2)F, the combination of Layer 3 TRM and EVPN Multi-Site is supported on the Cisco Nexus 93400LD-H1 switches.

• Beginning with Cisco NX-OS Release 10.4(3)F, the combination of Layer 3 TRM and EVPN Multi-Site is supported on the Cisco Nexus 9364C-H1 switches.

• Beginning with Cisco NX-OS Release 9.3(3), the Cisco Nexus 9504 and 9508 platform switches with -R/RX line cards support TRM in Layer 3 mode. This feature is supported on IPv4 overlays only. Layer 2 mode and L2/L3 mixed mode are not supported.

  The Cisco Nexus 9504 and 9508 platform switches with -R/RX line cards can function as a border leaf for Layer 3 unicast traffic. For Anycast functionality, the RP can be internal, external, or RP everywhere.

• When configuring TRM VXLAN BGP EVPN, the following platforms are supported:

  • Cisco Nexus 9200, 9332C, 9364C, 9300-EX, and 9300-FX/FX2/FX3/FXP platform switches.

  • Cisco Nexus 9300-GX/GX2 platform switches.

  • Cisco Nexus 9300-H2R/H1 platform switches.

  • Cisco Nexus 9500 platform switches with 9700-EX line cards, 9700-FX line cards, or a combination of both line cards.

• Layer 3 TRM and VXLAN EVPN Multi-Site are supported on the same physical switch. For more information, see Configuring VXLAN EVPN Multi-Site.

• TRM Multi-Site functionality is not supported on Cisco Nexus 9504 platform switches with -R/RX line cards.

• If one or both VTEPs is a Cisco Nexus 9504 or 9508 platform switch with -R/RX line cards, the packet TTL is decremented twice, once for routing to the L3 VNI on the source leaf and once for forwarding from the destination L3 VNI to the destination VLAN on the destination leaf.

• TRM with vPC border leafs is supported only for Cisco Nexus 9200, 9300-EX, and 9300-FX/FX2/FX3/GX/GX2/H2R/H1 platform switches and Cisco Nexus 9500 platform switches with -EX/FX or -R/RX line cards. The advertise-pip and advertise virtual-rmac commands must be enabled on the border leafs to support this functionality. For configuration information, see the "Configuring VIP/PIP" section.

• To support any Layer 3 source behind one of the vPC peers, whether physical or virtual MCT, a physical link configured as VRF-lite is required between the vPC peers. This setup is necessary to accommodate a receiver located behind the vPC peer, especially if it is the sole receiver in the fabric. This requirement applies to all scenarios where the vPC functions as a BGW, border Leaf, or an internal Leaf.

  On the receiving vPC peer, the VRF-lite link must have a superior reachability metric to the L3 source compared to any other paths (iBGP or eBGP) to be selected as the RPF towards the L3 source. In this configuration, traffic will flow directly to the receiver without traversing the EVPN fabric.

• Well-known local scope multicast (224.0.0.0/24) is excluded from TRM and is bridged.

• When an interface NVE is brought down on the border leaf, the internal overlay RP per VRF must be brought down.

• Beginning with Cisco NX-OS Release 10.3(1)F, TRM support for the new L3VNI mode CLIs are provided on Cisco Nexus 9300-X Cloud Scale switches.

• Beginning Cisco NXOS release 10.2(1)F, TRM Flow Path Visualization is supported for flows within a single VXLAN EVPN site.

• Beginning Cisco NXOS Release 10.3(2)F, TRM Flow Path Visualization support has been extended to below traffic patterns on Cisco Nexus 9000 Series platform switches:

  • TRM Multisite DCI Multicast

  • TRM Multisite DCI IR

  • TRM Data MDT

  • TRM on Virtual MCT vPC

  • TRM using new L3VNI

  • BUM Traffic visibility is not supported.

• Beginning with Cisco NX-OS Release 10.4(3)F, the TRM Multi-Site Anycast BGW on Cisco Nexus 9808/9804 switches with Cisco Nexus X9836DM-A and X98900CD-A line cards support the following features:

  • TRMv4

  • Ingress Replication between DCI peers across the core

  • Multicast underlay for fabric peers.

  • Only new L3VNI mode is supported. However, the traditional L3VNI mode is not supported

  TRM Multi-Site Anycast BGW on Cisco Nexus 9808/9804 switches with Cisco Nexus X9836DM-A and X98900CD-A line cards do not support the following features:

  • TRMv6

  • Data MDT

  • Multicast underlay between DCI peers across the core is not supported.

Layer 2/Layer 3 Tenant Routed Multicast (TRM) has the following configuration guidelines and limitations:

• All TRM Layer 2/Layer 3 configured switches must be Anchor DR. This is because in TRM Layer 2/Layer 3, you can have switches configured with TRM Layer 2 mode that co-exist in the same topology. This mode is necessary if non-TRM and Layer 2 TRM mode edge devices (VTEPs) are present in the same topology. 


• Anchor DR is required to be an RP in the overlay.

• An extra loopback is required for anchor DRs.

• Non-TRM and Layer 2 TRM mode edge devices (VTEPs) require an IGMP snooping querier configured per multicast-enabled VLAN. Every non-TRM and Layer 2 TRM mode edge device (VTEP) requires this IGMP snooping querier configuration because in TRM multicast control-packets are not forwarded over VXLAN.

• The IP address for the IGMP snooping querier can be re-used on non-TRM and Layer 2 TRM mode edge devices (VTEPs).

• The IP address of the IGMP snooping querier in a VPC domain must be different on each VPC member device.

• When interface NVE is brought down on the border leaf, the internal overlay RP per VRF should be brought down.

• The NVE interface must be shut and unshut while configuring the ip multicast overlay-distributed-dr command.

• Beginning with Cisco NX-OS Release 9.2(1), TRM with vPC border leafs is supported. Advertise-PIP and Advertise Virtual-Rmac need to be enabled on border leafs to support with functionality. For configuring advertise-pip and advertise virtual-rmac, see the "Configuring VIP/PIP" section.

• Anchor DR is supported only on the following hardware platforms:

  • Cisco Nexus 9200, 9300-EX, and 9300-FX/FX2 platform switches

  • Cisco Nexus 9500 platform switches with 9700-EX line cards, 9700-FX line cards, or a combination of both line cards

• Beginning with Cisco NX-OS Release 10.2(3)F, Anchor DR is supported on the Cisco Nexus 9300-FX3 platform switches.

• Layer 2/Layer 3 Tenant Routed Multicast (TRM) is not supported on Cisco Nexus 9300-FX3/GX/GX2/H2R/H1 platform switches.

VXLAN EVPN and TRM with IPv6 Multicast Underlay has the following guidelines and limitations:

• Spine-based static RP is supported in underlay.

• Cisco Nexus 9300-FX, FX2, FX3, GX, GX2, H2R, and H1 ToR switches are supported as the leaf VTEP.

• Cisco Nexus X9716D-GX and X9736C-FX line cards are supported only on the spine (EoR).

• When an EoR is deployed as a spine node with Multicast Underlay (PIMv6) Any-Source Multicast (ASM), it in mandatory to configure non-default template using one of the following commands in global configuration mode:

  • system routing template-multicast-heavy

  • system routing template-multicast-ext-heavy

• OSPFv3, ISIS, eBGP underlay is supported.

• PIMv6 ASM (sparse mode) is supported in underlay.

• PIMv6 Anycast RP is supported in underlay as RP redundancy.

• Underlay IPv6 Multicast is supported.

• Underlay IPv6 Multicast is not supported on EOR platforms as a leaf.

• For overlay traffic, each Cisco Nexus 9000 leaf switch is an RP. External RP is also supported.

• EVPN TRMv4 and TRMv6 with IPv6 Multicast Underlay are supported on the Fabric.

• Fabric Peering and Multisite are not supported with IPv6 multicast underlay.

• The global mcast-group under NVE should not be configured as SSM range, and vice versa. If there is no explicit SSM configuration, then 232/8 is the default in data plane. hence 232.0.0.0/8 should not be configured as SSM and vice versea.

• GPO is not supported with IPv6 multicast underlay.

• For EVPN TRMv4 and TRMv6 with IPv6 Multicast Underlay, the TCAM region for ingress sup region must be carved to 768.

  • Check the ingress sup region using show hardware access-list tcam region command.

  • If the ingress sup region is not 768 or above, you must configure using the hardware access-list tcam region ing-sup 768 command.

    Note	If you get an error, “Aggregate ingress TCAM allocation failure” while configuring ing-sup as 768, you must borrow the amount from other TCAM regions.

  • Reload the device after this configuration.

For Tenant Routed Multicast, the following rendezvous point options are supported:

• Configuring a Rendezvous Point Inside the VXLAN Fabric

• Configuring an External Rendezvous Point

• Configuring RP Everywhere with PIM Anycast

• Configuring RP Everywhere with MSDP Peering

Configuring IPv6 multicast underlay in the VXLAN fabric involves the following configurations:

Beginning with Cisco NX-OS Release 10.3(1)F, the Real-time/flex statistics for TRM is supported for Overlay routes on Cisco Nexus 9300-X Cloud Scale Switches. Flex Stats is not supported for Underlay Routes

Note	VXLAN NVE VNI ingress and egress, NVE per-peer ingress and tunnel tx stats won't be supported.

In a VXLAN TRM setup, if you want mroute statistics for overlay mroutes you must configure the hardware profile multicast flex-stats-enable command in the default template. For more information on configuration, see Configuring Flex Stats for TRM.

The following CLIs will not be supported after the flex stats CLI is enabled:

• sh nve vni <vni_id>/<all> counters

• sh nve peers <peer-ip> interface nve 1 counters

• sh int tunnel <Tunnel interface number> counters

In the following examples, the sample configuration for the leaf, spine, and RP are shown:

• Leaf - Sample configuration of IPv6 multicast underlay:

  • NVE Configuration

    interface nve1
      no shutdown
      host-reachability protocol bgp
      source-interface loopback1
      member vni 10501
        mcast-group ff04::40
      member vni 50001 associate-vrf
        mcast-group ff10:0:0:1::1
    

  • PIMv6 Configuration

    feature pim6
    
    ipv6 pim rp-address 101:101:101:101::101 group-list ff00::/8
    
    interface loopback1
      ipv6 address 172:172:16:1::1/128
      ipv6 pim sparse-mode
    
    interface Ethernet1/27
      ipv6 address 27:50:1:1::1/64
      ospfv3 hello-interval 1
      ipv6 router ospfv3 v6u area 0.0.0.0
      ipv6 pim sparse-mode
      no shutdown
    

  • BGP Configuration

    router bgp 100
        router-id 172.16.1.1
        address-family ipv4 unicast
          maximum-paths 64
          maximum-paths ibgp 64
        address-family ipv6 unicast
          maximum-paths 64
          maximum-paths ibgp 64
        address-family ipv4 mvpn
        address-family l2vpn evpn
        neighbor 172:17:1:1::1
          remote-as 100
          update-source loopback0
          address-family ipv4 mvpn
            send-community
            send-community extended
          address-family ipv6 mvpn
            send-community
            send-community extended
          address-family l2vpn evpn
            send-community
         neighbor 172:17:2:2::1
           remote-as 100
           update-source loopback0
           address-family ipv4 mvpn
             send-community
             send-community extended
           address-family ipv6 mvpn
             send-community
             send-community extended
           address-family l2vpn evpn
             send-community
             send-community extended
         vrf VRF1
           reconnect-interval 1
           address-family ipv4 unicast
             network 150.1.1.1/32
             advertise l2vpn evpn
             redistribute hmm route-map hmmAdv
    
    evpn
      vni 10501 l2
        rd auto
        route-target import auto
        route-target export auto
    vrf context VRF1
      vni 50001
    rd auto
    address-family ipv4 unicast
       route-target both auto
       route-target both auto mvpn
       route-target both auto evpn
    address-family ipv6 unicast
       route-target both auto
       route-target both auto mvpn
       route-target both auto evpn
    
    Note: Incase of vPC leafs, you need to configure identical “mvpn vri id” on both the vPC nodes. For example:
    
    router bgp 100
      mvpn vri id 2001
    

    Note	MVPN VRI ID must be unique within the network or setup. That is, if the network has three different sets of vPC pairs, each pair must have a different VRI ID.

• Spine - sample configuration of IPv6 multicast underlay:

  • NVE Configuration

    nv overlay evpn

  • PIMv6 Configuration

    feature pim6
    
    ipv6 pim rp-address 101:101:101:101::101 group-list ff00::/8
    ipv6 pim anycast-rp 101:101:101:101::101 102:102:102:102::102
    ipv6 pim anycast-rp 101:101:101:101::101 103:103:103:103::103
    
    interface loopback101
      ipv6 address 101:101:101:101::101/128
      ipv6 router ospfv3 v6u area 0.0.0.0
      ipv6 pim sparse-mode
    
    interface loopback102
      ipv6 address 102:102:102:102::102/128
      ipv6 router ospfv3 v6u area 0.0.0.0
      ipv6 pim sparse-mode
    
    interface Ethernet1/50/1
      ipv6 address 27:50:1:1::2/64
      ipv6 pim sparse-mode
      no shutdown
    

  • BGP Configuration

    feature bgp
    
    router bgp 100
            router-id 172.16.40.1
             address-family ipv4 mvpn
            address-family ipv6 mvpn
            address-family l2vpn evpn
            neighbor 172:16:1:1::1
              remote-as 100
              update-source loopback0
              address-family ipv4 mvpn
                send-community
                send-community extended
                 route-reflector-client
              address-family ipv6 mvpn
               send-community
                send-community extended
                 route-reflector-client
              address-family l2vpn evpn
              send-community
                send-community extended
                 route-reflector-client