Contents

Introduction

Prerequisites

Requirements

Components Used

Topology

Why is L1 Service Graph needed in ACI?

About L1 Device

Active/Active L1 PBR

Configuration for L1 Service Graph

Verification for L1 Service Graph on APIC GUI

Verification for L1 Service Graph on APIC CLI

Traffic Validation

Introduction

This document describes how to configure and verify Active/Active L1 Service Graph in Application Centric Infrastructure (ACI).

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics:

- Understanding of how Layer 3 Service Graph works in ACI

- Understanding on how to configure Endpoint policy group, bridge domains and contract in ACI

Components Used

The information in this document is based on these software and hardware versions:

- APIC Version: 5.3(2a)

- Leaf H/W: N9K-C93180YC-FX , N9K-C93180YC-EX

- Leaf S/W: n9000-15.3(2a)

- Leaf Node 101, 102 , 103 ,104

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command

Topology

EPG1 and EPG2 configuration is not shown in this document, it must be connfigured before hand and endpoint must be learnt.

1. Validate EPG1 haș endpoint 192.168.132.100 learned (Node 101).

2. Validate EPG2 has endpoint 192.168.132.200 learned (Node 104).

Why is L1 Service Graph needed in ACI?

In Cisco ACI you can insert L4-L7 service device as L3/L2/L1. Layer 3 means that external device is capable of perfoming routing decision to forward traffic whereas Layer 2 means traffic to be solely forwarded on the basis of MAC address. In ACI you can insert an L2 device such as Intrusion Prevention System (IPS)/Transparent Firewall. Now think of a scenario where the device that you are going to redirect the traffic is not capable of taking any forwarding decision, so, in those cases you can deploy L1 Policy-Based Routing (PBR).

Traffic forwarding is same for L3 and L2 PBR cases, the only difference is in case of L3 PBR traffic is redirected to an IP address where as for L1/L2 PBR traffic is redirected to MAC address. These MAC addresses are statically binded to leaf interface for forwarding purpose. You are going to see more on this going further.

For more information on Active/Stanby or Active/Active L1/L2 PBR use cases, refer to the link; PBR White Paper.

About L1 Device

In this deployment model, no VLAN translation is performed on the service device, and both of its interfaces operate on the same VLAN. This approach is commonly known as inline mode or wire mode and is typically used for firewalls and Intrusion Prevention Systems (IPS). It is ideal when the service device is expected to perform security functions without participating in Layer 2 or Layer 3 forwarding.

Active/Active L1 PBR

From ACI release 5.0 onwards, deploying a service graph with L4-L7 devices in an active/active mode is supported. This is achieved by assigning a unique encap to each L4-L7 device interface (concrete interface) and leveraging automatic configuration of ACI of 'Flood in encap' on the hidden service EPG. This hidden service EPG is created by ACI in order to associate the L4-L7 device interface with the service bridge domain.

Administrators do not need to manually configure the hidden service EPG, as ACI automatically enables 'Flood in encap' during the service graph rendering process.

For L1 PBR active-active deployments, port local scope must be configured. This requires placing the consumer and provider cluster interfaces (connectors) of the L4-L7 device in separate physical domains, each with its own VLAN pool, while maintaining the same VLAN range across both domains.

Reference: PBR White Paper.

Configuration for L1 Service Graph

Unicast routing must be enabled, L2 unknown unicast must be set to Hardware proxy and no subnet is required for cons and prov bridge domains.

Step 1. Configure consumer bridge domain named as cons_bd1.

Step 2. Configure provider bridge domain named as prov-bd1.

Step 3. Configure IP Service Level Agreement (SLA) Policy with SLA type L2Ping.

Navigate to Tenant > Policies > Protocol > IP SLA > IP SLA Monitoring Policies, then right click and create policy.

Step 4. Configure L4/L7 device.

Navigate to Tenant > Services > Devices, then right click and create L4-L7 device.

Note: For L1 device, each cluster interface must be in different physical domains for port local scope.

Step 5. Configure L4-L7 Policy based redirect for inside and outside interface.

Navigate to Tenant > Policies > Protocol > L4-L7 Policy based redirect, then right click and create policy.

++ Policy name is inside

++ We have two L1 destinations, one for each L1 device

++ Policy name is outside

++ We have two L1 destinations, one for each L1 device

Note: Threshold down action must be same for both L4-L7 redirect policies.

Step 6. Configure Service graph template.

Navigate to Tenant > Services > Service Graph Template, then right click and create L4-L7 Service graph template.

Step 7. Create a contract.

Navigate to Tenant > Contract > Standard, then right click and create contract.

Step 8. Apply the contract as consumer and provider to EPG1 and EPG2 respectively.

Step 9. Apply L4-L7 service graph template.

Navigate to Tenant > Services > Service Graph Template, then right click PBR1 and apply L4-L7 service graph template.

++ Add consumer and provider EPG

++ Specify contract

++ click Next

++ Specify consumer connector details

++ Specify provider connector details

++ Click on finish

Verification for L1 Service Graph on APIC GUI

Step 1. Verify the device selection policy created after you apply the service graph template.

++ Verify consumer connector

++ Configure provider connector

Step 2. Verify Deployed graph instances, there you are going to see an instance, it must be in applied state.

++ Check for device interfaces and fucntion connectors where you are going to see PCTAG associated with service EPG's

Verification for L1 Service Graph on APIC CLI

Step 1. Verify if the service graph is applied on consumer and provider node along with health group status.

apic01# fabric 101,104 show service redir info

----------------------------------------------------------------

Node 101

----------------------------------------------------------------

=======================================================================================================================================

LEGEND

TL: Threshold(Low)  |  TH: Threshold(High) |  HP: HashProfile  |  HG: HealthGrp  | BAC: Backup-Dest |  TRA: Tracking  | RES: Resiliency

=======================================================================================================================================

List of Dest Groups

GrpID Name            destination                                                    HG-name                          BAC  operSt     operStQual      TL   TH   HP   TRAC RES

===== ====            ===========                                                    ==============                   ===  =======    ============    ===  ===  ===  ===  ===

10    destgrp-10      dest-[476f:9be9:5aab:4454:a5d6:8c9e:7017:61eb]-[vxlan-2490369] l1_pbr_tenant::HG2               N    enabled    no-oper-grp     51   100  sym  yes  no

dest-[8301:bb59:e940:4233:81c6:e007:437e:45f]-[vxlan-2490369]  l1_pbr_tenant::HG1               N

2     destgrp-2       dest-[2958:ddd3:6eda:4ede:8bb4:1b66:8b19:1eb4]-[vxlan-2490369] l1_pbr_tenant::HG1               N    enabled    no-oper-grp     51   100  sym  yes  no

dest-[d438:790d:6fdb:4485:bab7:197d:ef61:9a59]-[vxlan-2490369] l1_pbr_tenant::HG2               N

List of destinations

Name                                                           bdVnid          vMac                 vrf                       operSt     operStQual      HG-name

====                                                           ======          ====                 ====                      =====      =========       =======

dest-[8301:bb59:e940:4233:81c6:e007:437e:45f]-[vxlan-2490369]  vxlan-16252846  10:B3:D5:14:99:99    l1_pbr_tenant:l1_pbr_vrf  enabled    no-oper-dest    l1_pbr_tenant::HG1

dest-[476f:9be9:5aab:4454:a5d6:8c9e:7017:61eb]-[vxlan-2490369] vxlan-16252846  10:B3:D5:14:77:77    l1_pbr_tenant:l1_pbr_vrf  enabled    no-oper-dest    l1_pbr_tenant::HG2

dest-[2958:ddd3:6eda:4ede:8bb4:1b66:8b19:1eb4]-[vxlan-2490369] vxlan-15794150  10:B3:D5:14:66:66    l1_pbr_tenant:l1_pbr_vrf  enabled    no-oper-dest    l1_pbr_tenant::HG1

dest-[d438:790d:6fdb:4485:bab7:197d:ef61:9a59]-[vxlan-2490369] vxlan-15794150  10:B3:D5:14:77:77    l1_pbr_tenant:l1_pbr_vrf  enabled    no-oper-dest    l1_pbr_tenant::HG2

List of Health Groups

HG-Name                                    HG-OperSt  HG-Dest                                                                HG-Dest-OperSt

=======                                    =========  =======                                                                ==============

l1_pbr_tenant::HG1                         enabled    dest-[2958:ddd3:6eda:4ede:8bb4:1b66:8b19:1eb4]-[vxlan-2490369]]        up

dest-[8301:bb59:e940:4233:81c6:e007:437e:45f]-[vxlan-2490369]]         up

l1_pbr_tenant::HG2                         enabled    dest-[d438:790d:6fdb:4485:bab7:197d:ef61:9a59]-[vxlan-2490369]]        up

dest-[476f:9be9:5aab:4454:a5d6:8c9e:7017:61eb]-[vxlan-2490369]]        up

----------------------------------------------------------------

Node 104

----------------------------------------------------------------

=======================================================================================================================================

LEGEND

TL: Threshold(Low)  |  TH: Threshold(High) |  HP: HashProfile  |  HG: HealthGrp  | BAC: Backup-Dest |  TRA: Tracking  | RES: Resiliency

=======================================================================================================================================

List of Dest Groups

GrpID Name            destination                                                    HG-name                          BAC  operSt     operStQual      TL   TH   HP   TRAC RES

===== ====            ===========                                                    ==============                   ===  =======    ============    ===  ===  ===  ===  ===

3     destgrp-3       dest-[d438:790d:6fdb:4485:bab7:197d:ef61:9a59]-[vxlan-2490369] l1_pbr_tenant::HG2               N    enabled    no-oper-grp     51   100  sym  yes  no

dest-[2958:ddd3:6eda:4ede:8bb4:1b66:8b19:1eb4]-[vxlan-2490369] l1_pbr_tenant::HG1               N

4     destgrp-4       dest-[476f:9be9:5aab:4454:a5d6:8c9e:7017:61eb]-[vxlan-2490369] l1_pbr_tenant::HG2               N    enabled    no-oper-grp     51   100  sym  yes  no

dest-[8301:bb59:e940:4233:81c6:e007:437e:45f]-[vxlan-2490369]  l1_pbr_tenant::HG1               N

List of destinations

Name                                                           bdVnid          vMac                 vrf                       operSt     operStQual      HG-name

====                                                           ======          ====                 ====                      =====      =========       =======

dest-[2958:ddd3:6eda:4ede:8bb4:1b66:8b19:1eb4]-[vxlan-2490369] vxlan-15794150  10:B3:D5:14:66:66    l1_pbr_tenant:l1_pbr_vrf  enabled    no-oper-dest    l1_pbr_tenant::HG1

dest-[d438:790d:6fdb:4485:bab7:197d:ef61:9a59]-[vxlan-2490369] vxlan-15794150  10:B3:D5:14:77:77    l1_pbr_tenant:l1_pbr_vrf  enabled    no-oper-dest    l1_pbr_tenant::HG2

dest-[476f:9be9:5aab:4454:a5d6:8c9e:7017:61eb]-[vxlan-2490369] vxlan-16252846  10:B3:D5:14:77:77    l1_pbr_tenant:l1_pbr_vrf  enabled    no-oper-dest    l1_pbr_tenant::HG2

dest-[8301:bb59:e940:4233:81c6:e007:437e:45f]-[vxlan-2490369]  vxlan-16252846  10:B3:D5:14:99:99    l1_pbr_tenant:l1_pbr_vrf  enabled    no-oper-dest    l1_pbr_tenant::HG1

List of Health Groups

HG-Name                                    HG-OperSt  HG-Dest                                                                HG-Dest-OperSt

=======                                    =========  =======                                                                ==============

l1_pbr_tenant::HG1                         enabled    dest-[2958:ddd3:6eda:4ede:8bb4:1b66:8b19:1eb4]-[vxlan-2490369]]        up

dest-[8301:bb59:e940:4233:81c6:e007:437e:45f]-[vxlan-2490369]]         up

l1_pbr_tenant::HG2                         enabled    dest-[476f:9be9:5aab:4454:a5d6:8c9e:7017:61eb]-[vxlan-2490369]]        up

dest-[d438:790d:6fdb:4485:bab7:197d:ef61:9a59]-[vxlan-2490369]]        up

Step 2. Verify if the static MAC binding is created on service nodes (102 and 103).

apic01# fabric 102-103 show endpoint vrf l1_pbr_tenant:l1_pbr_vrf

----------------------------------------------------------------

Node 102

----------------------------------------------------------------

Legend:

S - static s - arp L - local O - peer-attached

V - vpc-attached a - local-aged p - peer-aged M - span

B - bounce H - vtep R - peer-attached-rl D - bounce-to-proxy

E - shared-service m - svc-mgr

+-----------------------------------+---------------+-----------------+--------------+-------------+

VLAN/ Encap MAC Address MAC Info/ Interface

Domain VLAN IP Address IP Info

+-----------------------------------+---------------+-----------------+--------------+-------------+

23/l1_pbr_tenant:l1_pbr_vrf vlan-1310 10b3.d514.7777 LS eth1/6

24/l1_pbr_tenant:l1_pbr_vrf vlan-1301 10b3.d514.9999 LS eth1/5

----------------------------------------------------------------

Node 103

----------------------------------------------------------------

Legend:

S - static s - arp L - local O - peer-attached

V - vpc-attached a - local-aged p - peer-aged M - span

B - bounce H - vtep R - peer-attached-rl D - bounce-to-proxy

E - shared-service m - svc-mgr

+-----------------------------------+---------------+-----------------+--------------+-------------+

VLAN/ Encap MAC Address MAC Info/ Interface

Domain VLAN IP Address IP Info

+-----------------------------------+---------------+-----------------+--------------+-------------+

40/l1_pbr_tenant:l1_pbr_vrf vlan-1310 10b3.d514.7777 LS eth1/6

1/l1_pbr_tenant:l1_pbr_vrf vlan-1301 10b3.d514.6666 LS eth1/5

Traffic Validation

1. 2000 ICMP ping packets are being generated from EP1 to EP2 which are going to be redirected to L1 device.

switch1# ping 192.168.132.200 vrf l1_pbr1 count 2000 >>>>> sending 2000 packets

64 bytes from 192.168.132.200: icmp_seq=464 ttl=251 time=0.859 ms

64 bytes from 192.168.132.200: icmp_seq=465 ttl=251 time=0.872 ms

64 bytes from 192.168.132.200: icmp_seq=466 ttl=251 time=0.844 ms

64 bytes from 192.168.132.200: icmp_seq=467 ttl=251 time=0.821 ms

64 bytes from 192.168.132.200: icmp_seq=468 ttl=251 time=0.814 ms

64 bytes from 192.168.132.200: icmp_seq=469 ttl=251 time=0.846 ms

64 bytes from 192.168.132.200: icmp_seq=470 ttl=251 time=0.863 ms

64 bytes from 192.168.132.200: icmp_seq=471 ttl=251 time=0.819 ms

64 bytes from 192.168.132.200: icmp_seq=472 ttl=251 time=0.802 ms

64 bytes from 192.168.132.200: icmp_seq=473 ttl=251 time=0.851 ms

64 bytes from 192.168.132.200: icmp_seq=474 ttl=251 time=0.815 ms

2. Validate interfaces counters on node 102 and 103 which are connected to L1 device.

apic01# fabric 102-103 show interface ethernet 1/5-6 | grep "Node\|Ethernet\|RX\|packets\|TX"

Node 102

Ethernet1/5 is up

Hardware: 100/1000/10000/25000/auto Ethernet, address: 10b3.d5c5.8f25 (bia 10b3.d5c5.8f25)

30 seconds input rate 0 bits/sec, 0 packets/sec

30 seconds output rate 64 bits/sec, 0 packets/sec

RX

2008 unicast packets 2 multicast packets 0 broadcast packets >>>>>2000 packets recieved from L1 device

2010 input packets 213180 bytes

0 jumbo packets 0 storm suppression bytes

TX

2009 unicast packets 1 multicast packets 0 broadcast packets >>>>> 2000 packets transmitted towards L1 device

2010 output packets 213003 bytes

0 jumbo packets

Ethernet1/6 is up

Hardware: 100/1000/10000/25000/auto Ethernet, address: 10b3.d5c5.8f26 (bia 10b3.d5c5.8f26)

30 seconds input rate 0 bits/sec, 0 packets/sec

30 seconds output rate 64 bits/sec, 0 packets/sec

RX

9 unicast packets 2 multicast packets 0 broadcast packets

11 input packets 1286 bytes

0 jumbo packets 0 storm suppression bytes

TX

9 unicast packets 1 multicast packets 0 broadcast packets

10 output packets 1003 bytes

0 jumbo packets

Node 103

Ethernet1/5 is up

Hardware: 100/1000/10000/25000/auto Ethernet, address: a453.0e75.9a85 (bia a453.0e75.9a85)

30 seconds input rate 0 bits/sec, 0 packets/sec

30 seconds output rate 64 bits/sec, 0 packets/sec

RX

2009 unicast packets 1 multicast packets 0 broadcast packets >>>>> 2000 packets recieved from L1 device

2010 input packets 213003 bytes

0 jumbo packets 0 storm suppression bytes

TX

2008 unicast packets 1 multicast packets 0 broadcast packets >>> 2000 packets transmitted towards L1 device

2009 output packets 212897 bytes

0 jumbo packets

Ethernet1/6 is up

Hardware: 100/1000/10000/25000/auto Ethernet, address: a453.0e75.9a86 (bia a453.0e75.9a86)

30 seconds input rate 0 bits/sec, 0 packets/sec

30 seconds output rate 64 bits/sec, 0 packets/sec

RX

9 unicast packets 1 multicast packets 0 broadcast packets

10 input packets 1003 bytes

0 jumbo packets 0 storm suppression bytes

TX

9 unicast packets 1 multicast packets 0 broadcast packets

10 output packets 1003 bytes

0 jumbo packets

Note: Interface counters were cleared on node 102 and 103 before the traffic was tested.