Configuring Deduplication

This chapter describes how to configure deduplication on Cisco NX-OS devices.

This chapter contains the following sections:

Deduplication concept and models

Deduplication is a feature that

  • removes duplicate data to ensure only unique data packets are processed and stored

  • preventss unnecessary storage consumption by eliminating redundant copies of traffic, and

  • supports two operational models.

Due to the continuous rise in data flow, navigating in the networks and the data processed in different applications leads to data duplication both in source and target. For efficient data management, security, and storage you can use deduplication to remove duplicate data.

The deduplication feature eliminates the duplicate traffic which is received from traffic analysers or data storage tools. It identifies duplicate flows which are transferred in the Nexus Data Broker (NDB) switch.

Deduplication feature can be used to identify and eliminate duplicate data flows as they traverse data broker switches. By removing redundant copies of traffic—often generated by traffic analyzers or data storage tools—deduplication ensures that only unique data packets are processed and stored.

Deduplication supports two model types:

  • Inline deduplication Model (Interface Mode) : You can configure the inline model on existing NDB switches. This eliminates duplicate packets using interfaces. Initial interface packets are forwarded and the duplicate packets from other interfaces are dropped.

  • Deduplication on a Stick Model (VLAN Mode) : In this model, the packet flow is bound to a specific VLAN. The packet flow is permitted on the first VLAN, and duplicate flows on other VLANs are restricted. The packets are tagged with specific VLAN when the flow enters the data broker switches. Each packet is tagged with a unique VLAN, the packets with VLAN tag are forwarded to the deduplication switch, and duplicate flow is removed on that switch.

Guidelines and limitations for deduplication

This topic lists the classified categories of the guidelines and limitations for configuring deduplication.

Supported features

  • You can configure deduplication for inner packet flows only.

  • TCP and UDP flows are supported for flow deduplication.

  • Deduplication is supported for local SPAN and Optical TAP flows sessions.

  • Deduplication can be performed for VXLAN and GRE tunnel packets with a single encap.

  • Dynamic aging is supported only for 128k flows.

Unsupported features

  • Deduplication stick model (VLAN model) is not supported for tunnel termination, and Q-in-Q ports.

  • Deduplication is not supported for IPv6 and multicast flows.

Platform support

  • Deduplication is supported for Cisco Nexus 9300-FX2/FX3 and 9300-GX/GX2 platorm switches.

  • Deduplication is not supported for:

    • Cisco Nexus 9500 platorm switches

    • Cisco N9K-C9408 switch

ISSU

During ISSU, deduplication is disabled by default. Post ISSU all flows are cleared and refreshed.

Flows

  • You cannot delete duplication flow for short lived flows.

  • Flow packets from different VRFs with similar 5 tuples cannot be deleted. As deduplication is for the flows performed using 5-tuple of a packet. It cannot identify VRF packets.

  • For deduplication to function appropriately on tunnel traffic, ensure that you enable flow terminate.

Duplicate packets

You cannot configure deduplication for duplicate packets on the same interface such as ERSPAN. It may terminate more than a single ERSPAN session on an interface or SVI. This impacts the copy of same flow ends up on the same interface or VLAN. A switch cannot differentiate different ERSPAN sessions for deduplication.

Configure deduplication

Deduplication feature can be used to remove duplicate copies of the data flow when they are going through switches in Nexus Data Broker (NDB).

Procedure

Step 1

Enter global configuration mode using the configure terminal command.

Example:

switch# configure terminal
switch(config)#

Step 2

Enable flow-deduplication feature on a switch using the tap-aggregation flow-deduplication command.

Example:

switch(config)# tap-aggregation flow-deduplication
switch(config-dedup)#

Step 3

(Optional) Configure absolute timer for deduplication of packet flows using the absolute-timertime in minutes command.

The interval range is from 0 to 1440 minutes.

Example:

switch(config-dedup)# absolute-timer 10
switch(config-dedup)#

Step 4

(Optional) Configure dynamic timer for deduplication of packet flows using the dynamic-timertime in milli seconds command.

The interval range is from 0 to 300000 milliseconds.

Example:

switch(config-dedup)# dynamic-timer 2000
switch(config-dedup)#

Step 5

Configure deduplication on required mode using the mode {vlan|interface} command.

Note

 

Ensure to save configuration and reload the switch to configure deleting duplication on the switch.

Example:

switch(config-dedup)# mode interface
switch(config-dedup)#

Step 6

Clear deduplication on the required slot or module using the clear hardware deduplication statistics{slot|module in number} command.

The slot or module number range is from 1 to 30.

What to do next

Ensure that you reload switch after configuring deduplication, for the configuration changes to be effective.

Monitor commands for deduplication

Use the show commands listed in this section to monitor dedeplication. The examples display sample outputs.

  • show hardware deduplication summary 

    switch# show hardware deduplication summary
slot 1
=======
Deduplication		    : Enabled
Dedup Mode 		      : Interface
Dynamic timer                 : 200000 milliseconds
Absolute timer                : 5 minutes
Max Supported Flows           : 240K
Total number of learned flows : 240000
Total number of dropped bytes : 65698869600

  • show hardware deduplication detail 

    switch# show hardware deduplication detail
slot1
=========================================================================== 
			Dedup Flows
============================================================================
SourceIP 	Destination IP 	Ports(Src:Dst) Protocol    Interface     Learn-time
======================================================================================================================================================
192.0.2.10 	198.51.100.10 	3000 :3001 	6 	Eth1/1 	07/28/2023 11:47:09.532376
192.0.2.20 	198.51.100.11 	15000:15001	17      Eth1/1 	07/28/2023 11:47:09.532229
192.0.2.30 	198.51.100.12 	1841 :1842 	6 	Eth1/1 	07/28/2023 11:47:09.532340
192.0.2.40 	198.51.100.20 	2000 :2001 	6 	Eth1/1 	07/28/2023 11:47:09.532428
192.0.2.50 	198.51.100.21 	4000 :4001 	6 	Eth1/23	07/28/2023 11:47:09.532133

  • show hardware deduplication age-history 

    switch#show hardware deduplication age-history
slot 1
===========================================================================
				Dedup Flows
===========================================================================
Source   	Destination     Ports 	Protocol   Interface Timer 	  Learn-Time 			Aged-Time 
IP 		IP    	  (Src:Dst)		
===================================================================================================================================
192.0.2.50	203.0.113.20   4000:4001 	6 	Eth1/17 Dynamic   08/05/2023 2:24:49.26020      08/05/2023 12:33:29.21904
192.0.2.10	203.0.113.10   3000:3001       6 	Eth1/27 Dynamic   08/05/2023 12:24:49.126246    08/05/2023 12:33:29.21945
192.0.2.20	203.0.113.11  15000:15001      17	Eth1/5 Dynamic    08/05/2023 12:24:49.26070     08/05/2023 12:33:29.21957
192.0.2.40	203.0.113.21   2000 :2001      6 	Eth1/5 Dynamic    08/05/2023 12:24:49.26115     08/05/2023 12:33:29.21969
192.0.2.30	203.0.113.12   1841 :1842      6 	Eth1/17 Dynamic   08/05/2023 12:24:49.25949     08/05/2023 12:33:29.21979