Cisco Nexus 9300 Configuration Workflow for Cisco Nexus Data Broker, Release 3.2

1. OpenFlow OVA required:

 

Add openflow related configuration in the nexus switch
     For N9000 series nexus switch we need to add the following configuration-
conf t
      hardware access-list tcam region openflow 512

Copy the configuration in the nexus switches-
copy running-config startup
Reload the switch using :

reload

"feature lldp" command is required to be configured for centralized mode when ISL is present in between devices.

For additional information on configuring Cisco Nexus 9000 Series switches, see the following URL:

Cisco Nexus 9000 Series Switches

If Cisco Nexus Data Broker does not work after reloading of the device in Embedded mode, you have to run the ndb_launcher.py script from the ndb directory in the GitHub repository. Before launching the script, ensure that Cisco Nexus Data Broker is in activated mode. If it is in the deactivated mode, the script does not execute. You can use the show virtual-service list CLI command to display the status of Cisco Nexus Data Broker. Use python bootflash:ndb_launcher.py -v ndb command to execute the script. When the Python script is run, it creates an interfaces file in the embndb folder in the virtual machine.
The interfaces file contains the details of the interfaces and the management IP address and it also updates the launcher.sh file.

1. Installing and Activating embedded ova:

Follow the steps presented here to install and activate the Cisco embedded ova. It should be downloaded and available in the bootflash memory of each Cisco Nexus 3000 Series Switch.

N3K-123# virtual-service install name ndb package bootflash:ndb1000-sw-app-emb-k9-3.1.0.ova

Note: Installing package 'bootflash:/ndb1000-sw-app-emb-k9-3.1.0.ova' for virtual service 'ndb'. Once the install has finished, the VM may be activated. Use 'show virtual-service list' for progress.

N3K-123# show virtual-service list
Virtual Service List:
Name                    Status             Package Name
-----------------------------------------------------------------------
ndb                     Initializing       Not Available

N3K-123# show virtual-service list
Virtual Service List:
Name                    Status             Package Name
-----------------------------------------------------------------------
ndb                     Installed          ndb1000-sw-app-emb-k9-3.1.0.ova

N3K-123# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
N3K-123(config)# virtual-service ndb
N3K-123(config-virt-serv)# activate
Note: Activating virtual-service 'ndb', this might take a few minutes. Use 'show virtual-service list' for progress.

N3K-123(config-virt-serv)# show virtual-service list
Virtual Service List:
Name                    Status             Package Name
-----------------------------------------------------------------------
ndb                     Activating         ndb1000-sw-app-emb-k9-3.1.0.ova

N3K-123(config-virt-serv)# show virtual-service list
Virtual Service List:

Name                    Status             Package Name
-----------------------------------------------------------------------
ndb                     Activated          ndb1000-sw-app-emb-k9-3.1.0.ova

 

2. Installing and Activating Cisco Plug-in for OpenFlow(Openflow ova)

Follow the steps presented here to install and activate the Cisco Plug-in for OpenFlow. This example assumes that the OpenFlow open virtualization appliance (OVA) filename is ofa_mmemb-2.1.4-r2-nxos-SPA-k9.ova and that it is downloaded and available in the bootflash memory of each Cisco Nexus 3000 Series Switch.

N3K-123# virtual-service install name ofa package bootflash:ofa_mmemb-2.1.4-r2-nxos-SPA-k9.ova
Note: Installing package 'bootflash:/ofa_mmemb-2.1.4-r2-nxos-SPA-k9.ova' for virtual service 'ofa'. Once the install has finished, the VM may be activated. Use 'show virtual-service list' for progress.

N3K-123# show virtual-service list
Virtual Service List:
Name                    Status             Package Name
-----------------------------------------------------------------------
ndb                     Activated          ndb1000-sw-app-emb-k9-3.1.0.ova
ofa                     Initializing       Not Available

N3K-123# show virtual-service list
Virtual Service List:
Name                    Status             Package Name
-----------------------------------------------------------------------
ndb                     Activated          ndb1000-sw-app-emb-k9-3.1.0.ova
ofa                     Installing         ofa_mmemb-2.1.4-r2-nxos-SPA-k9.ova

N3K-123# show virtual-service list
Virtual Service List:
Name                    Status             Package Name
-----------------------------------------------------------------------
ndb                     Activated          ndb1000-sw-app-emb-k9-3.1.0.ova
ofa                     Installed          ofa_mmemb-2.1.4-r2-nxos-SPA-k9.ova

N3K-123# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
N3K-123(config)# virtual-service ofa
N3K-123(config-virt-serv)# activate
Note: Activating virtual-service 'ofa', this might take a few minutes. Use 'show virtual-service list' for progress.

N3K-123(config-virt-serv)# show virtual-service list

Virtual Service List:
Name                    Status             Package Name
-----------------------------------------------------------------------
ndb                     Activated          ndb1000-sw-app-emb-k9-3.1.0.ova
ofa                     Activating         ofa_mmemb-2.1.4-r2-nxos-SPA-k9.ova

N3K-123(config-virt-serv)# show virtual-service list
Virtual Service List:
Name                    Status             Package Name
-----------------------------------------------------------------------
ndb                     Activated          ndb1000-sw-app-emb-k9-3.1.0.ova
ofa                     Activated          ofa_mmemb-2.1.4-r2-nxos-SPA-k9.ova

 

Use the show virtual-service list command to verify that the service status is changed to “Activated.” This change process may take up to two minutes.

Add the controller to the switch with the following command-
N3K-123# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
N3K-123(config)# openflow
N3K-123(config-ofa)# switch 1
N3K-123(config-ofa-switch)# pipeline 201
N3K-123(config-ofa-switch)# controller ipv4 10.16.206.123 port 6653 vrf management security none
N3K-123(config-ofa-switch)# of-port interface ethernet1/1-54

Once all the above steps are performed switch will be converted to OF.

Use URL:  https://<DeviceIP>:8443 for logging into NDB GUI.

For additional information on configuring Cisco Nexus 9000 Series switches, see the following URL:

Cisco Nexus 9000 Series Switches

1. Enable nxapi feature with the following command-
conf t
  feature nxapi

2. Add nxapi configuration on the nexus switch-
For N9000 series nexus switch we need to add the following configuration-
conf t
 hardware access-list tcam region ifacl 256
 hardware access-list tcam region mac-ifacl 256
 hardware access-list tcam region ipv6-ifacl 256

No errors or warnings should be noticed when configuring the above three regions. These are the minimum regions to start and you can increase the number of regions. For more ACLs, set other TCAM regions in ingress direction to 0.
For example, set regions in ingress to zero:
 hardware access-list tcam region vacl 0
 hardware access-list tcam region racl 0
 hardware access-list tcam region qos 0

3. Copy the running configuration to the startup configuration:
copy running-config startup

4. Reload the switch to apply the updated configuration:
reload

The availability of space for a region depends on the TCAM chip on the switch. Refer to specific platform TCAM configuration guide for more information.

The feature lldp command is required to be configured for centralized mode when ISL is present in between devices.

For additional information on configuring Cisco Nexus 9000 Series switches, see the following URL:

Cisco Nexus 9000 Series Switches

If Cisco Nexus Data Broker does not work after reloading of the device in Embedded mode, you have to run the ndb_launcher.py script from the ndb directory in the GitHub repository. Before launching the script, ensure that Cisco Nexus Data Broker is in activated mode. If it is in the deactivated mode, the script does not execute. You can use the show virtual-service list CLI command to display the status of Cisco Nexus Data Broker. Use python bootflash:ndb_launcher.py -v ndb command to execute the script. When the Python script is run, it creates an interfaces file in the embndb folder in the virtual machine.
The interfaces file contains the details of the interfaces and the management IP address and it also updates the launcher.sh file.

Installing and Activating embedded ova:

Follow the steps presented here to install and activate the Cisco embedded ova. It should be downloaded and available in the bootflash memory of each Cisco Nexus 3000 Series Switch.

N3K-123# virtual-service install name ndb package bootflash:ndb1000-sw-app-emb-k9-3.1.0.ova
Note: Installing package 'bootflash:/ndb1000-sw-app-emb-k9-3.1.0.ova' for virtual service 'ndb'. Once the install has finished, the VM may be activated. Use 'show virtual-service list' for progress.

N3K-123# show virtual-service list
Virtual Service List
Name                    Status             Package Name
-----------------------------------------------------------------------
ndb                     Initializing       Not Available

N3K-123# show virtual-service list
Virtual Service List
Name                    Status             Package Name
-----------------------------------------------------------------------
ndb                     Installed          ndb1000-sw-app-emb-k9-3.1.0.ova

N3K-123# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
N3K-123(config)# virtual-service ndb
N3K-123(config-virt-serv)# activate
Note: Activating virtual-service 'ndb', this might take a few minutes. Use 'show virtual-service list' for progress.
N3K-123(config-virt-serv)# show virtual-service list
Virtual Service List:
Name                    Status             Package Name
-----------------------------------------------------------------------
ndb                     Activating         ndb1000-sw-app-emb-k9-3.1.0.ova

 

N3K-123(config-virt-serv)# show virtual-service list

Virtual Service List:

Name                    Status             Package Name
-----------------------------------------------------------------------
ndb                     Activated          ndb1000-sw-app-emb-k9-3.1.0.ova

 

Use URL:  https://<DeviceIP>:8443 for logging into NDB GUI.

 

For additional information on configuring Cisco Nexus 9000 Series switches, see the following URL:

Cisco Nexus 9000 Series Switches

1. Configure hardware configurations using:
conf t
hardware profile openflow
2. Copy the configuration in the nexus switches-
3. Copy the running-config startup and reload the switch using :
Reload
4. Install and Activate Cisco Plug-in for OpenFlow(Openflow ova):
N3K-123# virtual-service install name ofa package bootflash:ofa_mmemb-2.1.4-r2-nxos-SPA-k9.ova
Note: Installing package 'bootflash:/ofa_mmemb-2.1.4-r2-nxos-SPA-k9.ova' for virtual service 'ofa'.

Once the install has finished, the VM may be activated. Use 'show virtual-service list' for progress.
N3K-123# show virtual-service list

Virtual Service List:
Name         Status         Package Name
-----------------------------------------------------------------------
ndb            Activated       ndb1000-sw-app-emb-k9-3.1.0.ova
ofa             Initializing     Not Available
N3K-123# show virtual-service list

Virtual Service List:
Name         Status         Package Name
-----------------------------------------------------------------------
ndb            Activated      ndb1000-sw-app-emb-k9-3.1.0.ova
ofa             Installing     ofa_mmemb-2.1.4-r2-nxos-SPA-k9.ova
N3K-123# show virtual-service list

Virtual Service List:
Name         Status         Package Name
-----------------------------------------------------------------------
ndb            Activated     ndb1000-sw-app-emb-k9-3.1.0.ova
ofa             Installed     ofa_mmemb-2.1.4-r2-nxos-SPA-k9.ova

5. Add the controller to the switch
6. Enable nxapi feature with the following command-
conf t
feature nxapi
7. Also add the below hardware config:
hardware profile tcam region ifacl 1024 double-wide

 

For additional information on configuring Cisco Nexus 9000 Series switches, see the following URL:

Cisco Nexus 9000 Series Switches

Redirecting IPv4 traffic with filter A from two source ports to Monitor1
Redirecting IPv4 traffic with filter B from two source ports to Monitor2

 

3.1 Use Case for OpenFlow

1. Configure openflow in the device – Device should be present in the NDB UI
Device should have following in the interfaces:
  no lldp transmit
  spanning-tree bpdufilter enable
  mode openflow

2. Configure two edge port – After configuring Ethernet1/26 and Ethernet1/27 as edge span in NDB, the following is seen in the 3100 device in the corresponding interfaces
N3K-123# show running-config interface ethernet 1/26-27
 interface Ethernet1/26
  no lldp transmit
  spanning-tree bpdufilter enable
  mode openflow

interface Ethernet1/27
  no lldp transmit
  spanning-tree bpdufilter enable
  mode openflow

3. Configure two deliver port - After configuring Ethernet1/26 and Ethernet1/27 as edge span in NDB the following is seen in the 9000 device in the corresponding interfaces
N3K-123# show running-config interface ethernet 1/30, ethernet 1/48
 interface Ethernet1/30
  no lldp transmit
  spanning-tree bpdufilter enable
  mode openflow

interface Ethernet1/48
  no lldp transmit
  spanning-tree bpdufilter enable
  mode openflow

4. Configure simple IPv4 filters in NDB
a. filter A -- source IP(1.1.1.1) destination IP(1.1.1.2)
b. filter B -- source IP(2.2.2.1)destination IP(2.2.2.2)

5. Create a connections in NDB

CONNECTION A -- Source ports ethernet1/26 and ethernet1/27, filter B, monitor device with delivery port as ethernet1/30

After this connection created in NDB, the following flows will be present in the device:

N3K-123# show openflow switch 1 flows
Logical Switch Id: 1
Total flows: 3

Flow: 1
  Match:             ip,in_port=27,nw_src=2.2.2.1,nw_dst=2.2.2.2
  Actions:           output:30
  Priority:          500
  Table:             0
  Cookie:            0x0
  Duration:          10.317s
  Number of packets: 0
  Number of bytes:   0

Flow: 2
  Match:             ip,in_port=26,nw_src=2.2.2.1,nw_dst=2.2.2.2
  Actions:           output:30
  Priority:          500
  Table:             0
  Cookie:            0x0
  Duration:          10.439s
  Number of packets: 0
  Number of bytes:   0

Flow: 3
  Match:             any
  Actions:           drop
  Priority:          0
  Table:             0
  Cookie:            0x0
  Duration:          3433.97s
  Number of packets: 5492
  Number of bytes:   0
N3K-123#

 

CONNECTION B source ports ethernet1/26 and ethernet1/27, filter A, monitor device with delivery port as ethernet1/48

After this connection created in NDB, the following flows will be present in the device:

N3K-123# show openflow switch 1 flows
Logical Switch Id: 1
Total flows: 3

Flow: 1
  Match:             ip,in_port=27,nw_src=1.1.1.1,nw_dst=1.1.1.2
  Actions:           output:48
  Priority:          500
  Table:             0
  Cookie:            0x0
  Duration:          266.843s
  Number of packets: 0
  Number of bytes:   0

Flow: 2
  Match:             ip,in_port=26,nw_src=1.1.1.1,nw_dst=1.1.1.2
  Actions:           output:48
  Priority:          500
  Table:             0
  Cookie:            0x0
  Duration:          266.954s
  Number of packets: 0
  Number of bytes:   0

Flow: 3
  Match:             any
  Actions:           drop
  Priority:          0
  Table:             0
  Cookie:            0x0
  Duration:          4125.18s
  Number of packets: 6602
  Number of bytes:   0

N3K-123#

 

 

3.2 Use Case for NXAPI:

Configuring edge span ports with vlan 100 on Ethernet1/26 and vlan 200 on ethernet1/27 will configure the following in the device:
N3K-123# show running-config interface ethernet 1/26-27

interface Ethernet1/26
  ip port access-group ipacl-Ethernet126 in
  switchport mode dot1q-tunnel
  switchport access vlan 100
  mode tap-aggregation
  spanning-tree bpdufilter enable

interface Ethernet1/27
  ip port access-group ipacl-Ethernet127 in
  switchport mode dot1q-tunnel
  switchport access vlan 200
  mode tap-aggregation
  spanning-tree bpdufilter enable

Configuring monitor device on ports Ethernet1/29 and ethernet1/30 will configure the following in the device:

N3K-123# show running-config interface ethernet 1/29-30

interface Ethernet1/29
  ip port access-group ipacl-Ethernet129 in
  switchport mode trunk
  mode tap-aggregation

interface Ethernet1/30
  ip port access-group ipacl-Ethernet130 in
  switchport mode trunk
  mode tap-aggregation

Configure simple IPv4 filters in NDB
a. filter A -- source IP(1.1.1.1) destination IP(1.1.1.2)
b. filter B -- source IP(2.2.2.1)destination IP(2.2.2.2)

Create a connections in NDB

CONNECTION A -- Source ports ethernet1/26 and ethernet1/27, filter A, monitor device with delivery port as ethernet1/29

After this connection created in NDB, the following will be present in the device:

IP access list ipacl-Ethernet126
        statistics per-entry
        49500001 permit ip 1.1.1.1/32 1.1.1.2/32 redirect Ethernet1/29 [match=0]
        49993001 deny ip any any [match=0]
        49993002 deny ethertype 0x0 [match=0]
IP access list ipacl-Ethernet127
        statistics per-entry
        49500001 permit ip 1.1.1.1/32 1.1.1.2/32 redirect Ethernet1/29 [match=0]
        49993001 deny ip any any [match=0]
        49993002 deny ethertype 0x0 [match=0]

CONNECTION B -- Source ports ethernet1/26 and ethernet1/27, filter B, monitor device with delivery port as ethernet1/30

After this connection created in NDB, the following will be present in the device:

IP access list ipacl-Ethernet126
        statistics per-entry
        49500001 permit ip 2.2.2.1/32 2.2.2.2/32 redirect Ethernet1/30 [match=0]
        49993001 deny ip any any [match=0]
        49993002 deny ethertype 0x0 [match=0]
IP access list ipacl-Ethernet127
        statistics per-entry
        49500001 permit ip 2.2.2.1/32 2.2.2.2/32 redirect Ethernet1/30 [match=0]
        49993001 deny ip any any [match=0]

        49993002 deny ethertype 0x0 [match=0]

 

 

3.3 Use Case for Redirection (AUX mode)

1. Configure the device in Openflow mode and add NXAPI plugin by doing the following steps:

• Configure hardware configurations using:
• conf t
• hardware profile openflow
• Copy the configuration in the nexus switches-
• copy running-config startup
• Reload the switch using :
• Reload
• Install and Activate Cisco Plug-in for OpenFlow(Openflow ova)
• Add the controller to the switch
• Enable nxapi feature with the following command-
• conf t
• feature nxapi
• Also add the below hardware config:
• hardware profile tcam region ifacl 1024 double-wide

2. Device should have following in the interfaces:
  no lldp transmit
  spanning-tree bpdufilter enable
   mode openflow

3. Configure two production ports(bith having different VLAN), add a service node, and monitor port.

4. Configure simple IPv4 filter in NDB : filter A -- source IP(1.1.1.1) destination IP(1.1.1.2)

5. Create a Redirection in NDB by adding service node, production Ingress Eth1/19 and Production Egress Eth1/1 ports and monitor device Eth1/25. Select BOTH as the value for “Select monitor traffic”.

So the traffic from both production ingress and service node egress is directed to the monitor port.

Following flows are installed

Flow: 1
  Match:             ip,in_port=19,nw_src=1.1.1.1,nw_dst=2.2.2.1
  Actions:           mod_vlan_vid:44,output:1,output:25
  Priority:          50000
  Table:             0
  Cookie:            0x0
  Duration:          5.365s
  Number of packets: 0
  Number of bytes:   0

Flow: 2
  Match:             in_port=19
  Actions:           mod_vlan_vid:44,output:1
  Priority:          5
  Table:             0
  Cookie:            0x0
  Duration:          5.02s
  Number of packets: 0
  Number of bytes:   0

Flow: 3
  Match:             ip,in_port=19,dl_vlan=33,nw_src=1.1.1.1,nw_dst=2.2.2.1
  Actions:           mod_vlan_vid:44,output:25
  Priority:          50000
  Table:             0
  Cookie:            0x0
  Duration:          5.283s
  Number of packets: 0
  Number of bytes:   0

Flow: 4
  Match:             any
  Actions:           drop
  Priority:          0
  Table:             0
  Cookie:            0x0
  Duration:          45.694s
  Number of packets: 0
  Number of bytes:   0

Flow: 5
  Match:             ip,in_port=19
  Actions:           mod_vlan_vid:44,output:1
  Priority:          5
  Table:             0
  Cookie:            0x0
  Duration:          5.108s
  Number of packets: 0

  Number of bytes:   0

For additional information on configuring Cisco Nexus 9000 Series switches, see the following URL:

Cisco Nexus 9000 Series Switches