Introduction
This document explains how Catalyst 3850 switches implement security Access Control Lists (ACLs) in hardware and how security Ternary Content Addressable Memory (TCAM) is utilized among various types of ACLs.
Background Information
This list provides definitions for various types of ACLs:
- VLAN Access Control List (VACL) - A VACL is an ACL that is applied to a VLAN. It can only be applied to a VLAN and no other type of interface. The security boundary is to permit or deny traffic that moves between VLANs and permit or deny traffic within a VLAN. The VLAN ACL is supported in hardware, and has no effect on the performance.
- Port Access Control List (PACL) - A PACL is an ACL applied to a Layer 2 switchport interface. The security boundary is to permit or deny traffic within a VLAN. The PACL is supported in hardware and has no effect on the performance.
- Router ACL (RACL) - An RACL is an ACL that is applied to an interface that has a Layer 3 address assigned to it. It can be applied to any port that has an IP address such as routed interfaces, loopback interfaces, and VLAN interfaces. The security boundary is to permit or deny traffic that moves between subnets or networks. The RACL is supported in hardware, and has no effect on the performance.
- Group-based ACL (GACL) - GACL is group-based ACL defined in Object Groups for ACL.
Problem
On Catalyst 3850/3650 switches, input PACL and output PACL Access Control Entities (ACEs) are installed in two separate regions/banks. These regions/banks are called ACL TCAMs (TAQs). VACL input and output ACEs are stored in a single region (TAQ). Due to a Doppler hardware limitation, VACL cannot use both TAQs. Therefore, VACL/vlmap only have half the Value Mask Result (VMR) space available to security ACLs. These logs appear when any of these hardware limits are exceeded:
%ACL_ERRMSG-4-UNLOADED: 1 fed: Output IPv4 L3 ACL on interface Vl215
for label 19 on asic255 could not be programmed in hardware and traffic will be dropped.
%ACL_ERRMSG-4-UNLOADED: 1 fed: Output IPv4 L3 ACL on interface Vl216
for label 20 on asic255 could not be programmed in hardware and traffic will be dropped.
%ACL_ERRMSG-4-UNLOADED: 1 fed: Output IPv4 L3 ACL on interface Vl218
for label 22 on asic255 could not be programmed in hardware and traffic will be dropped.
However, Security ACE TCAM might not appear to be full when these logs appear.
Solution
It is incorrect to assume that one ACE always consumes one VMR. A given ACE can consume:
- 0 VMRs if it gets merged with a previous ACE.
- 1 VMR if VCU bits are available to handle the range.
- 3 VMRs if it gets expanded because no VCU bits are available.
The Catalyst 3850 Data Sheet suggests that 3,000 security ACL entries are supported. However, these rules define how these 3,000 ACEs can be configured:
- VACL/vlmaps support a total of 1.5K entries as they can use only one of the two TAQs.
- MAC VACL/vlmap needs three VMR/ACEs. This means 460 ACEs must be supported in each direction.
- IPv4 VACL/vlmap needs two VMR/ACEs. This means 690 ACEs must be supported in each direction.
- IPv4 PACL, RACL, and GACL need one VMR/ACE. This means 1,380 ACEs must be supported in each direction.
- MAC PACL, RACL, and GACL need two VMR/ACEs. This means 690 ACEs must be supported in each direction.
- IPv6 PACL, RACL, and GACL need two VMR/ACEs. This means 690 ACEs must be supported in each direction.
Troubleshooting Security ACL TCAM on Catalyst 3850 Switches
- Check security TCAM utilization:
Note: Even though the installed security ACEs are less than 3,072, one of the limits previously mentioned might have been reached. For example, if a customer has most of the RACLs applied in the input direction, they can use up 1,380 entries available for the inbound RACL. However, TCAM exhaustion logs can show up before all 3,072 entries are used.
3850#show platform tcam utilization asic all
CAM Utilization for ASIC# 0
Table Max Values Used Values
--------------------------------------------------------------------------------
Unicast MAC addresses 32768/512 85/22
Directly or indirectly connected routes 32768/7680 125/127
IGMP and Multicast groups 8192/512 0/16
QoS Access Control Entries 3072 68
Security Access Control Entries 3072 1648
Netflow ACEs 1024 15
Input Microflow policer ACEs 256 7
Output Microflow policer ACEs 256 7
Flow SPAN ACEs 256 13
Control Plane Entries 512 195
Policy Based Routing ACEs 1024 9
Tunnels 256 12
Input Security Associations 256 4
Output Security Associations and Policies 256 9
SGT_DGT 4096/512 0/0
CLIENT_LE 4096/64 0/0
INPUT_GROUP_LE 6144 0
OUTPUT_GROUP_LE 6144 0
- Check the hardware state of ACLs installed in the TCAM:
3850#show platform acl info acltype ?
all Acl type
ipv4 Acl type
ipv6 Acl type
mac Acl type
3850#show platform acl info acltype all
########################################################
######### ##################
######### ##################
######## Printing ACL Infos #################
######### ##################
########################################################
===================================
IPv4 ACL: Guest-ACL
aclinfo: 0x52c41030
ASIC255 Input L3 labels: 4
ipv4 Acl: Guest-ACL Version 16 Use Count 0 Clients 0x0
10 permit udp any 8 host 224.0.0.2 eq 1985
20 permit udp any 8 any eq bootps
30 permit ip 10.100.176.0 255.255.255.0 any
<snip>
3850#show platform acl info switch 1
########################################################
######### ##################
######### ##################
######## Printing ACL Infos #################
######### ##################
########################################################
===================================
IPv4 ACL: Guest-ACL
aclinfo: 0x52c41030
ASIC255 Input L3 labels: 4
ipv4 Acl: Guest-ACL Version 16 Use Count 0 Clients 0x0
10 permit udp any 8 host 224.0.0.2 eq 1985
20 permit udp any 8 any eq bootps
30 permit ip 10.100.176.0 255.255.255.0 any
<snip>
- Check acl-event logs whenever ACLs are installed/removed:
3850#show mgmt-infra trace messages acl-events switch 1
[04/22/15 21:35:34.877 UTC 3a8 5692] START Input IPv4 L3 label_id 22
asic3 num_les 1 old_unload 0x0, cur_unloaded 0x0, trid 236 num_vmrs 11
[04/22/15 21:35:34.877 UTC 3a9 5692] Trying L3 iif_id 0x104608000000100
input base FID 14
[04/22/15 21:35:34.878 UTC 3aa 5692] Input IPv4 L3 label_id 22 hwlabel
22 asic3 status 0x0 old_unloaded 0x0 cur_unloaded 0x0 trid 236
[04/22/15 21:35:35.939 UTC 3ab 5692] MAC: 0000.0000.0000
Adding Input IPv4 L3 acl [Postage-Printer] BO 0x1 to leinfo le_id 29on asic 255
[04/22/15 21:35:35.939 UTC 3ac 5692] MAC: 0000.0000.0000 Rsvd
label 0 --> New label 23, asic255
[04/22/15 21:35:35.939 UTC 3ad 5692] START Input IPv4 L3 label_id 23
asic3 num_les 1 old_unload 0x0, cur_unloaded 0x0, trid 237 num_vmrs 5
<snip>
- Print out the ACL Content Addressable Memory (CAM):
C3850-1#show platform acl cam
======== ACL TCAM (asic 0) =========
Printing entries for region ACL_CONTROL (135)
========================================================
TAQ-4 Index-0 Valid StartF-1 StartA-1 SkipF-0 SkipA-0:
Entry allocated in invalidated state
Mask1 00f00000:00000000:00000000:00000000:00000000:00000000:00000000:00000000
Key1 00400000:00000000:00000000:00000000:00000000:00000000:00000000:00000000
AD 90220000:2f000000
TAQ-4 Index-1 Valid StartF-0 StartA-0 SkipF-0 SkipA-0
Mask1 00f00000:0f000000:00000000:00000000:00000000:00000000:00000000:00000000
Key1 00400000:01000000:00000000:00000000:00000000:00000000:00000000:00000000
AD 00a00000:00000000
- Print out itemized ACL hit and drop counters:
C3850-1#show platform acl counters hardware switch 1
=========================================================
Ingress IPv4 Forward (280): 397555328725 frames
Ingress IPv4 PACL Drop (281): 147 frames
Ingress IPv4 VACL Drop (282): 0 frames
Ingress IPv4 RACL Drop (283): 0 frames
Ingress IPv4 GACL Drop (284): 0 frames
Ingress IPv4 RACL Drop and Log (292): 3567 frames
Ingress IPv4 PACL CPU (285): 0 frames
Ingress IPv4 VACL CPU (286): 0 frames
Ingress IPv4 RACL CPU (287): 0 frames
Ingress IPv4 GACL CPU (288): 0 frames
Feedback