Chapter 1. New and Changed Information

New and Changed Information

Chapter 2. Overview

About OpenStack and Cisco ACI

About Hierarchical Port Binding

Chapter 3. OpFlex ML2 and GBP Deployment

Deploying OpFlex ML2 and GBP

Prerequisites

Deployment Overview

Staging the Software Archive on the Repo Server

Preparing the OpenStack Server

Updating the OpenStack Neutron Server

Installing and Configuring the OpFlex and Host Agents

Configuring the OpFlex Agent Port Using VXLAN Encapsulation

Configuring the OpFlex Agent Port Using VLAN Encapsulation

Starting and Enabling the Agent Services

Initializing the ACI Tenant

Chapter 4. Enabling Optimized Neutron Services

Enabling the Optimized Neutron Services

Optimized DHCP Services

Optimized Metadata Proxy

Chapter 5. Adding an OpenStack External Network

Adding an OpenStack External Network

Chapter 6. Adding an OpenStack External Network in Unified Mode

Adding an OpenStack External Network in Unified Mode

Appendix A. Reference Information

Deploying OpenStack on ACI using Physical Domain

Virtual Routing and Forwarding and Network Address Translation

Multiple Virtual Routing and Forwardings with Network Address Translation

Single Shared Virtual Routing and Forwarding Without Network Address Translation

Single Shared Virtual Routing and Forwarding with Network Address Translation

ACI Fabric Initialization Example

Manually Configure the Host vPC

Setting Up Auto-Configuration of Host Links

ACI External Routed Network Example

Network Constraints Template File

Troubleshooting the APIC OpenStack Plugin

Version Information

Configuring the OpFlex ML2 plugin to use Hierarchical Port Binding

Preparing the ACI

OpenStack Preparation and Integration

Example of the ml2_conf_cisco_apic.ini File

Sample of the ml2_conf.ini File

Configuring SR-IOV

Preface

Audience

Document Conventions

Related Documentation

Documentation Feedback

Obtaining Documentation and Submitting a Service Request

Copyright Page

Chapter 1. New and Changed Information

This chapter contains the following sections:

- New and Changed Information

New and Changed Information

The following table provides an overview of the significant changes to this guide up to this current release. The table does not provide an exhaustive list of all changes made to the guide or of the new features up to this release.

New Features and Changed Behavior in the Cisco ACI with OpenStack OpFlex Deployment Guide for Red Hat

Cisco APIC Release Version

Feature

Description

Where Documented

2.2(1)

--

Added Single Root I/O Virtualization (SR-IOV) support.

For more information see: Configuring SR-IOV.

2.2(1)

--

Added unified support for Neutron (ML2) and Group Base Policy (GBP).

For more information, see:

- About OpenStack and Cisco ACI

- Updating the OpenStack Neutron Server

- Adding an OpenStack External Network in Unified Mode

2.0(1m)

--

Added support to configure the OpFlex ML2 plugin to use Hierarchical Port Binding (HPB).

For more information, see About Hierarchical Port Binding and Configuring the OpFlex ML2 plugin to use Hierarchical Port Binding.

1.2(2g)

--

Added support for deploying OpenStack on ACI using physical domain.

For more information, see Deploying OpenStack on ACI using Physical Domain.

1.2(2g)

--

Added support for Liberty release of OpenStack.

--

1.2(2g)

ACI support outbound and inbound prefix-list and routemap based filtering

Introduced support for specify constraints on the subnets that may be created through OpenStack. These constraints enable you to disallow creation of certain subnets, or to make them public or private in APIC.

For more information, see Updating the OpenStack Neutron Server and Network Constraints Template File.

1.2(1i)

--

There are no significant changes to this document.

--

1.1(4e)

--

This guide was released.

--

Chapter 2. Overview

This chapter contains the following sections:

- About OpenStack and Cisco ACI

- About Hierarchical Port Binding

About OpenStack and Cisco ACI

Cisco Application Centric Infrastructure (ACI) is a comprehensive policy-based architecture that provides an intelligent, controller-based network switching fabric. This fabric is designed to be programmatically managed through an API interface that can be directly integrated into multiple orchestration, automation, and management tools, including OpenStack. Integrating ACI with OpenStack allows dynamic creation of networking constructs to be driven directly from OpenStack requirements, while providing additional visibility within the ACI Application Policy Infrastructure Controller (APIC) down to the level of the individual virtual machine (VM) instance.

OpenStack defines a flexible software architecture for creating cloud-computing environments. The reference software-based implementation of OpenStack allows for multiple Layer 2 transports including VLAN, GRE, and VXLAN. The Neutron project within OpenStack can also provide software based Layer-3 forwarding. When utilized with ACI, the ACI fabric provides an integrated Layer 2 and Layer 3 VXLAN-based overlay networking capability that can offload network encapsulation processing from the compute nodes onto the top-of-rack or ACI leaf switches. This architecture provides the flexibility of software overlay networking in conjunction with the performance and operational benefits of hardware-based networking.

Traditionally Cisco ACI OpenStack plugin could be deployed in either ML2 or GBP mode. In Modular Layer 2 (ML2) mode standard Neutron API is used to create networks. This is the traditional way of deploying VMs and services in OpenStack. In Group Based Policy (GBP) mode and a new API is provided to describe, create and deploy applications as policy groups without worrying about network-specific details. For more information, see OpenStack Group-Based Policy User Guide at: Click Here

In previous OpFlex plugin versions (referred to as Classical mode) it was necessary to decide at the time of deployment if the mode of the plugin will be Neutron/ML2 or GBP, and it was not possible to use both GBP and Neutron/ML2 APIs at the same time. Starting from OpFlex plugin version 2.2.1. It is possible to deploy the plugin in “Unified” mode. In unified mode it is possible to create application topologies using either Neutron or GBP API. Unified plugin mode also requires OpenStack release Mitaka or higher and ACI release 2.2(1) or higher.

This guide covers deployment of the OpFlex plugins in both Classical and Unified installation mode.

Note

While creating GBP groups in Unified mode, an (auto-ptg) group will also appear, these groups are for internal use and user interaction (attaching VM, adding members) is not supported.

About Hierarchical Port Binding

Hierarchical Port Binding (HPB) in Neutron allows software-defined networking (SDN) users to dynamically configure VLANs for a non-opflex node connected to ACI fabric. This node can be an OpenStack node, a bare metal server or a Layer 4 to Layer 7 service device. HPB allows you to use locally significant dynamically and create VLAN segments to connect endpoints to Neutron virtual networks.

The latest ACI OpFlex OpenStack Neutron plugin has support for HPB. For more information, see Configuring the OpFlex ML2 plugin to use Hierarchical Port Binding.

Chapter 3. OpFlex ML2 and GBP Deployment

This chapter contains the following sections:

- Deploying OpFlex ML2 and GBP

- Prerequisites

- Deployment Overview

- Staging the Software Archive on the Repo Server

- Preparing the OpenStack Server

- Updating the OpenStack Neutron Server

- Installing and Configuring the OpFlex and Host Agents

- Configuring the OpFlex Agent Port Using VXLAN Encapsulation

- Configuring the OpFlex Agent Port Using VLAN Encapsulation

- Starting and Enabling the Agent Services

- Initializing the ACI Tenant

Deploying OpFlex ML2 and GBP

This section describes how to installation and configuration the Cisco ACI OpenStack Plugin on a Red Hat OpenStack distribution.

These example steps were validated on Juno, Kilo, Liberty and Mitaka releases of OpenStack. OpenStack systems can vary widely in how they are installed. Therefore, the examples provided may be used as a basis to be adapted to the specifics of your installation.

Prerequisites

This section describes the prerequisites.

- Target audience—You must have working knowledge of Linux, Red Hat OpenStack distribution, ACI policy model and GUI-based APIC configuration. Also familiarity with OpenStack architecture and deployment.

- ACI fabric—ACI fabric installed and initialized with a minimum version of 1.1(4e) and 11.1(4e). For basic guidelines on initializing a new ACI fabric, see ACI Fabric Initialization Example. For communication between multiple leaf pairs, the fabric must have a BGP Route Reflector enabled to use an OpenStack external network.

- OpenStack—An installed version of Red Hat RDO/OSP Juno or Kilo release on RHEL/CentOS 7 or later.

In case of RedHat subscription, you need to enable rhel-7-server-rh-common-rpms andrhel-7-server-extras-rpms.

In case of CentOS, you need to enable EPEL.

For versions equal to Juno or earlier, you need to enable EPEL-testing repos.

In a Packstack on CentOS based installation, EPEL can be added using the CONFIG_USE_EPEL=y in the answer file.

For the Juno release, the system requires kernel version 3.10.0-229.14.1.el7.x86_64. If that is not an option you need to upgrade openvswitch kernel module with version later than 2.4.0.

The Kilo release and later, do not require this kernel version, as the upstream OVS or OpenvSwitch kernel module is used instead. This should be a clean install of OpenStack with no pre-existing networks or routers, since all network services will be provided by ACI.

- Repo Server—A RHEL or CentOS server accessible from the management network of the OpenStack servers is needed to house the repository for yum-based installation of the Cisco OpFlex software.

To check the current kernel version on your installed servers, enter the following command:

uname -a

If you need to change the kernel version, use the following syntax:

yum -y install <kernel version>

For example:

yum -y install kernel-3.10.0-229.14.1.el7.x86_64

Note

EPEL-testing repo is not required for the Kilo or higher versions of OpenStack.

Creation of the route reflector on the APIC can be completed through the web interface following the system documentation or by using the following command on the OpenStack controller once the ACI OpenStack Plugin software is installed:

apic route-reflector-create --apic-ip <ip address> \
--apic-password <password> --apic-username \
<username> --no-secure

Deployment Overview

This section describes the deployment overview.

With the prerequisites met, installation and configuration of the Cisco ACI OpenStack Plugin can begin. The following is a high-level overview of the installation process:

- Stage the OpFlex plugin software archive on the Repo Server

- OpenStack Server Prep (DHCP and LLDP, enable yum repo, NIC prep for ACI infrastructure VLAN)

- OpenStack Neutron Network Node(s) configuration

- OpFlex Agent installation and host configuration

- Choose one of the following:

- OpFlex Port configuration for VXLAN Encapsulation

- OpFlex Port configuration for VLAN Encapsulation

- Start agent services

- Initialize ACI OpFlex Tenant

During the installation, there is an option to choose either VLAN or VXLAN encapsulation between the OpenStack servers and the ACI leaf switches. Once on the leaf switch, all traffic will be encapsulated in VXLAN across the ACI fabric. The VXLAN encapsulation between server and leaf switch provides the possibility of a larger number of networks to be available to the OpenStack installation, but does require more processing on the compute nodes for packet traffic. VXLAN offload capabilities are available using certain Network Interface Cards to offset this impact. VLAN-mode encapsulation on the OpenStack servers provides lower overhead on the compute nodes for systems that do not require very large numbers of tenant networks.

Follow the instructions for either VXLAN or VLAN encapsulation or only, consistently. For the OpFlex Agent Configuration tasks, separate sections are provided for each encapsulation.

Once the OpFlex plugin is installed and configured, it will be operational for automated OpenStack Tenant networking. A separate section of this guide covers extending the OpenStack cloud to an external network using Source-NAT (sNAT) and Floating IP addresses.

Staging the Software Archive on the Repo Server

This section describes how to stage the software archive on the Repo Server.

Installation of the ACI OpenStack Plugin and associated software on RHEL or CentOS systems is accomplished through yum and RPM package management. The Repo Server acts as a central distribution point for consistent code versions out to all of the OpenStack server nodes in the system. This server can be any utility server in your environment, or in a pinch this function can be served by one of the OpenStack server nodes. The Repo Server is only accessed during installation and upgrade of the ACI OpenStack Plugin.

Note

Follow the guidelines of your Information Security. Your organization may already have place and policies for hosting package repositories. This example procedure is provided as a guideline.

Procedure

Step 1

Download the correct version and Cisco ACI OpenStack Plugin release archive file from Cisco's Download Software website to the Repo Server.

Step 2

Ensure the httpd service is installed and running, if needed install, start, and enable the service:

Example:

yum install httpd
systemctl start httpd
systemctl enable httpd

Step 3

Create a directory with a name you choose such as "opflex" under the /var/www/html directory on the Repo Server. Move the archive tar file to the new directory, un-tar the archive and create the repo for yum. Once created, ensure that the owner of all files to be served in the repo is set to the apache user using the chown command:

Example:

mv <release-archive-name> /var/www/html/opflex
cd /var/www/html/opflex
tar xvf <release-archive-name>
createrepo /var/www/html/opflex
cd ..
chown –R apache:apache opflex

Step 4

The yum repo should now be prepared to be used by OpenStack servers, ensure the servers have IP connectivity to the Repo Server on their management/SSH interface.

Preparing the OpenStack Server

This section describes how to prepare the OpenStack servers.

The OpenStack server nodes need to be prepared to properly interact with the ACI fabric for OpFlex. This includes DHCP configuration for the interface on the ACI Infrastructure VLAN, and LLDP communication. Also, the servers need to be set up to point to the yum repository on the Repo Server for the OpFlex software repo.

Note

Depending on deployment requirements, it may not be needed to extend the datapath to the network/neutron server (controller or network node). For example using optimized (distributed) DHCP and metadata eliminates the requirement to extend datapath to the controller/network server. There may be other services such as DNS needed that require datapath connectivity to the controller/neutron server. In case this is not required the setup of OpFlex interface and agents (neutron-opflex-agent and agent-ove) can be skipped on neutron/network servers. If you are planning to use optimized DHCP and metadata services, make sure to follow Enabling the Optimized Neutron Services. Before deploying OpenStack consider theses requirements: A network node will be needed when deploying with ACI and OpFlex, since most of the neutron extension will be implemented in ACI and/or will be distributed on the compute nodes.

Procedure

Step 1

The OpenStack Neutron and Compute servers require an ACI-connected network interface configured for 802.1Q VLAN tagged traffic matching the Infrastructure (infra) VLAN of the ACI fabric. This interface can be configured as a Linux sub-interface in the operating system, or the tagging function can be offloaded to a virtual NIC (vNIC), if using Cisco UCS servers with Cisco VIC cards.
Note:
If using a Cisco VIC card, disable the local LLDP functions on the VIC. For detailed information on configuring Cisco C-Series servers for VPC connections to ACI using Cisco VIC cards with VLAN tagging, see Manually Configure the Host vPC.

To use a Linux-level sub-interface for the infra VLAN, create an interface configuration file using the name of the parent physical or bond interface followed by a period and the number of the ACI infra VLAN. The default infra VLAN on an ACI fabric is 4093. For example, if the parent interface was named eth1, the configuration file for the sub-interface would be /etc/sysconfig/network-scripts/ifcfg-eth1.4093. Example contents for this configuration file are below:

PERSISTENT_DHCLIENT=1
DHCPRELEASE=1
DEVICE=eth1.4093
ONBOOT=yes
PEERDNS=yes
NM_CONTROLLED=no
MACADDR= <unique MAC address>
TYPE=Ethernet
BOOTPROTO=dhcp
VLAN=yes
ONPARENT=yes
MTU=1600

Step 2

Ensure that a unique MAC address for the sub-interface that is different from that of the parent interface is configured on the MACADDR= line of the configuration file. If this address is a duplicate of the parent interface it can cause issues with LLDP communication with the upstream switch. Also ensure this MAC address is unique on the VLAN.
You need to ensure that the parent interface is configured with MTU of 1600 otherwise the sub-interface will not have a higher MTU. Edit the ifcfg-eth1 file and add the following:

Example:

MTU=1600

Step 3

Bounce the interface:

Example:

ifdown eth1
ifup eth1

Note:
If a Cisco VIC card is used to provide a virtual interface for the OpFlex infra VLAN communication, a unique address is automatically generated by the CIMC and does not need to be added to the file.

Both the sub-interface and the parent interface require the MTU to be increased to allow for VXLAN headers to be added to packets. The line MTU=1600 in the example configuration provides for this, add the same line to the configuration file for the parent interface.

Step 4

The network interface on the infra VLAN will request a DHCP address from APIC Infrastructure network for OpFlex communication. The server needs to have a dhclient configuration for this interface to receive all of the correct DHCP options with the lease. If you need information on how to configure a VPC interface for the OpenStack servers, see Manually Configure the Host vPC.
Note:   The infra vlan interface in your environment may be a basic Linux-level sub-interface, such as "eth0.4093".

Create the /etc/dhcp/dhclient-eth0.4093.conf file with the following content and insert the MAC address of the Ethernet interface for each server on the first line of the file:

Example:

send dhcp-client-identifier 01:<mac-address of infra VLAN interface>;
request subnet-mask, domain-name, domain-name-servers, host-name;
send host-name <server-host-name>;

option rfc3442-classless-static-routes code 121 = array of unsigned integer 8;
option ms-classless-static-routes code 249 = array of unsigned integer 8;
option wpad code 252 = string;

also request rfc3442-classless-static-routes;
also request ms-classless-static-routes;
also request static-routes;
also request wpad;
also request ntp-servers;

Step 5

A multicast route needs to be applied specifically to the opflex infrastructure VLAN interface. This is done by creating /etc/sysconfig/network-scripts/route-eth0.4093 file with the following content:

Example:

ADDRESS0=224.0.0.0
NETMASK0=240.0.0.0
GATEWAY0=0.0.0.0
METRIC0=1000

Step 6

In order for the ACI fabric to be able to use dynamic discovery of the OpenStack nodes, a software LLDP stack is needed on the server. To install the LLDP package, execute the following commands:

Example:

cd /etc/yum.repos.d/
yum install wget
wget http://download.opensuse.org/repositories/home:vbernat/RHEL_7/home:vbernat.repo
yum install lldpd

Step 7

Start and enable the lldpd service to be reloaded after a server boot.
Note:
LLDP is used to automatically provision host links. You can skip installation of LLDP if automatic provisioning is not needed, in that case you can also skip installation of host/services agent. For details, see Manually Configure the Host vPC. LLDP also requires host, services agents and even though auto config is not required, you may chose to still use LLDP and host/services agents for debugging the host.

Example:

service lldpd start
chkconfig lldpd on

Note:
An alternate software LLDP stack may be used depending on your host operating system version, as long as the ACI fabric can dynamically discover the server nodes. If LLDP is not enabled on the compute nodes, you need to manually configure this in the ml2_conf_cisco_apic.ini. For example syntax, see Setting Up Auto-Configuration of Host Links.

Step 8

The OpenStack Network and Compute nodes need to have a pointer added in their yum configuration to allow them to pull the OpFlex software from the Repo Server. Create the /etc/yum.repos.d/opflex.repo file with the following content, substituting the IP address of your Repo Server in the lines with the baseurl statement:

Example:

[opflex]
name=opflex repo
baseurl=http://<ip address>/opflex
failovermethod=priority
enabled=1
gpgcheck=0

With this configuration in place, check to see if the repo is working properly and verify there are no errors:

Example:

yum makecache

Updating the OpenStack Neutron Server

This section describes how to update the OpenStack Neutron Servers.

The Neutron server(s) in the OpenStack system provide the primary interaction with the ACI fabric through the APIC for dynamic provisioning of networking for OpenStack Tenants. This section covers installation of the OpFlex Agent and Driver, along with specific configuration file edits required to enable APIC communication. The ACI OpenStack Plugin allows the ACI fabric to replace the Layer 3 forwarding functions typically provided by the OpenStack Neutron L3-Agent service. The service will no longer be used.

Procedure

Step 1

You must disable the service on the Neutron server(s) using the following commands:

Example:

systemctl stop neutron-l3-agent
systemctl disable neutron-l3-agent

Step 2

On the OpenStack controller node, install the neutron-opflex-agent, APIC API, and ML2/GBP driver along with required supporting modules. These packages are pulled from the EPEL repo, ensure EPEL is enabled on the node for successful installation. Supporting modules python-pip and python-pbr are also required as prerequisites:

Example:

yum install python-pip
yum install python-pbr

Step 3

If you are running Juno, ensure the epel-testing repo is enabled by modifying the /etc/yum.repos.d/epel-testing.repo file and making sure enabled=1 is set under the [epel-testing] section as shown below:

Example:

[epel-testing]
name=Extra Packages for Enterprise Linux 7 - Testing - $basearch
#baseurl=http://download.fedoraproject.org/pub/epel/testing/7/$basearch
mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=testing-epel7&arch=$basearch
failovermethod=priority
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7

Step 4

Install the opflex agent, apicapi, and ml2 driver:

Example:

yum install neutron-opflex-agent apicapi neutron-ml2-driver-apic

Step 5

In case of GBP based installation, you must install these additional packages:

- openstack-neutron-gbp

- python-gbpclient

- openstack-dashboard-gbp

- openstack-heat-gbp

Example:

yum install openstack-neutron-gbp python-gbpclient openstack-dashboard-gbp \
openstack-heat-gbp

Step 6

In case of Unified installation mode, you must install these additional packages:

- openstack-neutron-gbp

- python-gbpclient

- openstack-dashboard-gbp

- openstack-heat-gbp

- aci-integration-module

Example:

yum install openstack-neutron-gbp python-gbpclient openstack-dashboard-gbp \
openstack-heat-gbp aci-integration-module

Step 7

Once the installation completes, the /etc/neutron/neutron.conf file needs to be updated to point to the APIC for networking services. Edit the existing list of service plugins in the file and with:
For ML2:

Example:

service_plugins = cisco_apic_l3,metering,lbaas

For GBP:

Example:

service_plugins = group_policy,servicechain,apic_gbp_l3,metering

For Unified mode:

Example:

core_plugin = ml2plus
service_plugins = group_policy,servicechain,apic_aim_l3,metering

Note:
You need to be cautious not to remove services that do not clash with services required by this plugin. For example, if lbaas and/or metering service is enabled you need to make sure it stays enabled as shown in above example.

Step 8

For GBP based installation, you need to enable the heat plugin for GBP. This is done by making sure the plugin_dirs in the DEFAULT section of /etc/heat/heat.conf file includes the GBP path:

Example:

plugin_dirs = /usr/lib/python2.7/site-packages/gbpautomation/heat

Step 9

The ML2 configuration /etc/neutron/plugins/ml2/ml2_conf.ini file also requires the following changes to enable the correct mechanism driver for APIC, and add OpFlex as a new network type:

Example:

type_drivers = opflex,local,flat,vlan,gre,vxlan
tenant_network_types = opflex

For ML2:

Example:

mechanism_drivers = cisco_apic_ml2

For GBP:

Example:

mechanism_drivers = apic_gbp

For Unified:

Example:

mechanism_drivers = apic_aim
extension_drivers = apic_aim

Step 10

If using VXLAN Encapsulation, edit the /etc/neutron/plugins/ml2/ml2_conf.ini file and comment out the following line

Example:

# network_vlan_ranges =

Step 11

If using VLAN Encapsulation, edit the /etc/neutron/plugins/ml2/ml2_conf.ini file and add the range of the VLANS to the [ml2_type_vlan] section using the following line:

Example:

network_vlan_ranges = physnet1:1000:2000

Step 12

The keyword physnet1 comes from the bridge_mappings defined in the same section, make sure you also have a statement in the same section to define the bridge_mappings as the following:

Example:

bridge_mapppings = physnet1:br-prv

The range of VLANs defined in this file will be used by the ACI OpenStack Plugin to create a VLAN pool on the APIC, if apic_provision_infra is set to true in the ml2_conf_cisco_apic.ini file.

Step 13

Edit the /etc/neutron/dhcp_agent.ini file, change the dhcp_driver and verify other values:

Example:

dhcp_driver = apic_ml2.neutron.agent.linux.apic_dhcp.ApicDnsmasq
ovs_integration_bridge = br-int
enable_isolated_metadata = True

Ensure that the ovs_integration_bridge = br-int line is not commented out.

Step 14

The OpFlex agent-ovs component provides local DHCP lease delivery to VM instances on each compute node by default. There is a setting to control distributed behavior in the ml2_conf_cisco_apic.ini file; enable_optimized_dhcp which has a default setting of “True” if not overridden in the file. The neutron-dhcp-agent process is still required on the Neutron server(s) to handle IP address management and proper communication of dnsmasq processes to the agent-ovs DHCP function. Restart the neutron-dhcp-agent to ensure any configuration changes are applied.

Example:

systemctl restart neutron-dhcp-agent

Step 15

The ml2_conf_cisco_apic.ini file is the primary configuration file on the Neutron server(s) to customize the interaction of the ACI OpenStack Plugin with the ACI.
Using classical deployment mode:

APIC IP addresses, credentials, and default naming for objects in the ACI policy model are configured there. The following example file illustrates and explains the relevant settings that must be configured to match your ACI environment.

Note:
This example is designed for use with LLDP-based host discovery. If you would like to manually configure your host discovery, the entries are placed in the ml2_conf_cisco_apic.ini file as well. For details on manual configuration, see Setting Up Auto-Configuration of Host Links.

Edit the /etc/neutron/plugins/ml2/ml2_conf_cisco_apic.ini file:

Example:

[DEFAULT]
apic_system_id= <apic_system_id>

[opflex]
networks = ‘*’

[ml2_cisco_apic]

# Hostname:port list of APIC controllers
apic_hosts = <comma-separated list of APIC IP addresses>

# Username for the APIC controller
apic_username= <username with administrative access to APIC>

# Password for the APIC controller
apic_password= <password for apic_username>

# Whether use SSl for connecting to the APIC controller or not
apic_use_ssl = True

# How to map names to APIC: use_uuid or use_name.
apic_name_mapping = use_name

# Agent timers for State reporting and topology discovery
apic_agent_report_interval = 30
apic_agent_poll_interval = 2
apic_provision_infra = True
apic_provision_hostlinks = True
enable_optimized_dhcp = True
enable_optimized_metadata = True

where <apic_system_id> is:

- The unique name for identifying the OpenStack cluster.

- The name seen on the APIC in the ACI tenant for the OpenStack system and for automated creation of APIC objects by the driver.

The same apic_system_id you used in the /etc/neutron/plugins/ml2/ml2_conf_cisco_apic.ini file on the Neutron server(s).

- The prefix for all APIC domain, names, and profiles created and seen in the APIC GUI.

where apic_provision_infra is set to false if all policies are already configured in ACI to allow required communication on the port/VPC, including the AAP and required policies and pools. It should be set to true if all these policies are not pre-created and plugin is expected to create them. If pre-existing server connections are being used and already defined in the APIC, the VMM Domain created will also need to be manually associated to the AEP used when those connections were created.

where apic_provision_hostlinks = False is the manual server port provisioning.

where enable_optimized_dhcp = True is true by default.

where enable_optimized_metadata = True is the recommended string to distributed metadata.

Using Unified mode:

Edit the /etc/neutron/plugins/ml2/ml2_conf_cisco_apic.ini file:

Example:

[group_policy]
policy_drivers=aim_mapping
extension_drivers = aim_extension,proxy_group

[apic_aim_auth]
auth_plugin=v3password
auth_url=http://<CONTROLLER-IP or VIP>:35357/v3/
username=admin
password=<ADMIN_PASSWORD>
user_domain_name=default
project_domain_name=default
project_name=admin

[ml2_apic_aim]
enable_optimized_metadata = True

[group_policy_implicit_policy]
default_ip_pool=192.168.0.0/16

The Target Policy Group subnets will be carved out of the 192.168.0.0/16 address space.

Step 16

Using GBP in Classical mode, add the [group_policy] section to the ml2_conf_cisco_apic.ini file. The Target Policy Group subnets will be carved out of the 192.168.0.0/16 address space:

Example:

[group_policy]
policy_drivers=implicit_policy,apic
[group_policy_implicit_policy]
default_ip_pool=192.168.0.0/16

Step 17

Once the ml2_conf_cisco_apic.ini file has been edited, it needs to be added into the service definition for the OpenStack neutron-server service to be read for options when the service starts. Edit the /usr/lib/systemd/system/neutron-server.service file and add --config-file /etc/neutron/plugins/ml2/ml2_conf_cisco_apic.ini to the ExecStart line.

Example:

ExecStart=/usr/bin/neutron-server \
--config-file /usr/share/neutron/neutron-dist.conf \
--config-file /etc/neutron/neutron.conf \
--config-file /etc/neutron/plugin.ini
--config-file /etc/neutron/plugins/ml2/ml2_conf_cisco_apic.ini \
--log-file /var/log/neutron/server.log

Step 18

For GBP or Unified deployment, copy the policy.json file from /etc/group-based-policy/policy.d/policy.json to /etc/neutron.

Example:

cp /etc/group-based-policy/policy.d/policy.json /etc/neutron/policy.json

Note:
This overwrites the policy.json file in the /etc/neutron directory. If you have made any changes to this file you will need to merge the changes into the new policy.json file.

Step 19

(Optional) In release 1.2(2x), you can additionally specify constraints on the subnets that may be created through OpenStack. These constraints enable you to disallow creation of certain subnets, or to make them public or private in APIC.

1. Edit the [ml2_cisco_apic] section of the ml2_conf_cisco_apic.ini file to point to the constraints cisco_apic_network_constraints.ini file. Add the following line to the ml2_conf_cisco_apic.ini file:

Example:

[ml2_cisco_apic]
network_constraints_filename = /etc/neutron/plugins/ml2/cisco_apic_network_constraints.ini

2. Edit the network constraints /etc/neutron/plugins/ml2/cisco_apic_network_constraints.ini file to describe your constraints. For more information, see Network Constraints Template File.
You can modify the network constraints file any time later without needing to restart the Neutron server for the changes to take effect.

Note:
In deployments that have multiple Neutron controllers in high-availability mode, the constraints file should be identical on all controllers for correct operation (as is the case with other configuration files).

Step 20

For Classical mode, skip this step. For Unified mode, configure aim.

1. Enable the “aim” services:

Example:

systemctl enable aim-aid
systemctl enable aim-event-service-polling
systemctl enable aim-event-service-rpc

2. Edit the /etc/aim/aim.conf file:

Example:

[DEFAULT]
debug=True
rpc_backend=rabbit
control_exchange=neutron
default_log_levels=neutron.context=ERROR

[oslo_messaging_rabbit]
rabbit_host=<rabbit-mq-host>
rabbit_port=<rabbit-m1-port>
rabbit_hosts=<rabbit-mq-host>:<rabbit-m1-port>
rabbit_use_ssl=False
rabbit_userid=<username>
rabbit_password=<password>
rabbit_ha_queues=False

The above values can be obtained from the neutron.conf file. You can copy and paste the [oslo_messsaging_rabbit] section from the file.

Example:

[database]
connection=mysql+pymysql://neutron:<PASSWORD>@<CONTROLLER-IP>/neutron

The above value for connection can be obtained from the neutron.conf file. You can copy and paste the [database] section from the file.

Example:

[aim]
agent_down_time = 75
poll_config = False
aim_system_id = openstack_aid
max_operation_retry=5

[apic]
apic_hosts = <APIC-IP>
apic_username = admin
apic_password = <PASSWORD>
apic_use_ssl = True

3. Edit the /etc/aim/aimctl.conf file:

Example:

[DEFAULT]
apic_system_id = <apic_system_id>

[apic_vmdom:ostack]

[apic_switch:101]
compute01 = 1/10
compute02 = 1/11

[apic_switch:102]
compute02 = 1/10
compute04 = 1/11

4. Initialize the “aim” database:

Example:

aimctl db-migration upgrade head
aimctl config update

If you want to create ACI infra, run the following commands:

Example:

aimctl infra create
aimctl manager load-domains –enforce

5. Start the aim services:

Example:

systemctl start aim-aid
systemctl start aim-event-service-polling
systemctl start aim-event-service-rpc

Step 21

With the Neutron server service definition updated to read the config file, reload the systemd manager configuration and then restart the Neutron server using the following commands:

Example:

systemctl daemon-reload
service neutron-server restart

Installing and Configuring the OpFlex and Host Agents

This section describes how to install and configure the OpFlex and host agents.

Compute nodes require installation and configuration of the Neutron OpFlex Agent, and the OpFlex Agent that programs OVS (agent-ovs). You may also want to deploy these agents on the controller/network nodes if dataplane needs to be extended to the controller/network nodes. Refer to the note:

Note

Depending on deployment requirements, it may not be needed to extend the datapath to the network/neutron server (controller or network node). For example using optimized (distributed) DHCP and metadata eliminates the requirement to extend datapath to the controller/network server. There may be other services such as DNS needed that require datapath connectivity to the controller/neutron server. In case this is not required the setup of OpFlex interface and agents (neutron-opflex-agent and agent-ove) can be skipped on neutron/network servers. If you are planning to use optimized DHCP and metadata services, make sure to follow the Enabling the Optimized Neutron Services section. Before deploying OpenStack consider theses requirements: A network node will be needed when deploying with ACI and OpFlex, since most of the neutron extension will be implemented in ACI and/or will be distributed on the compute nodes.

Before You Begin

The Neutron nodes should already have the neutron-opflex-agent in place which was done in the Updating the OpenStack Neutron Server section.

Procedure

Step 1

Install the neutron-opflex-agent and agent-ovs from the yum repository. If optimized metadats is required, you should also install neutron-ml2-driver-apic package on compute nodes.

Example:

yum install neutron-opflex-agent
yum install agent-ovs
yum install neutron-ml2-driver-apic

Step 2

Ensure that the /etc/neutron/plugins/ml2/openvswitch_agent.ini file contains the following settings in the example below. If you are using a release older than liberty then use /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.in file instead.

Example:

[ovs]
enable_tunneling  =   False
integration_bridge =  br-int

Also ensure the configuration lines for tunnel_bridge, vxlan_udp_port and tunnel_types are deleted or commented out.

Step 3

Disable and stop the neutron-openvswitch-agent, enter the following commands:

Example:

systemctl disable neutron-openvswitch-agent
service neutron-openvswitch-agent stop

Step 4

If you are running Kilo or Higher, skip this step and proceed to step 5.
Note:
For Kilo running on CentOS, you need to make sure the Open vSwitch version is 2.4.0 or later. To check the version, run the following command:

Example:

# ovs-vsctl show

285c1281-1e63-49fd-a7c7-828e31b071b0
Bridge br-tun
Port br-tun
Interface br-tun
type: internal
Bridge br-int
fail_mode: secure
Port of-svc-ovsport
Interface of-svc-ovsport
Port "br-int_vxlan0"
Interface "br-int_vxlan0"
type: vxlan
options: {dst_port="8472", key=flow, remote_ip=flow}
Port br-int
Interface br-int
type: internal
ovs_version: "2.4.0"

If the ovs_version is prior to 2.4.0, then you must get the Open vSwitch package version 2.4.0 or later. One location to get the Open vSwitch package is from CentOS mirrors: Click Here

If you are running a prior version, the default Open vSwitch agent installed with OpenStack is not suitable for OpFlex configuration. You must install the modified Cisco-specific Open vSwitch package to interact properly with the ACI fabric, enter the following command:

Example:

yum install openvswitch-gbp

Note:
You may also need to update the openvswitch kernel module to match with the version of openvswitch userspace package.

Step 5

To ensure the system is cleanly running with the new configuration and OVS module up to this point. You may need to reboot your server.

Step 6

Once the servers are available with the right version of openvswitch. Login and change to the /etc/opflex-agent-ovs/conf.d directory.

Step 7

The agent-ovs service reads its configuration from the /etc/opflex-agent-ovs/opflex-agent-ovs.conf file, and the conf.d subdirectory allows granular override of specific settings in that file using smaller JSON-formatted files. Create a new /etc/opflex-agent-ovs/conf.d/10-opflex-connection.conf file with the example contents below:

Example:

{
"opflex": {
"domain": "comp/prov-OpenStack/ctrlr-[<apic_system_id>]-<apic_system_id>/sw-InsiemeLSOid",
"name": "<hostname of this system>",
"peers": [
{"hostname": "10.0.0.30", "port": "8009"}
],
"ssl": {
"mode": "encrypted"
}
}
}

where <apic_system_id> is:

- The unique name for identifying the OpenStack cluster.

- The name seen on the APIC in the ACI tenant for the OpenStack system and for automated creation of APIC objects by the driver.

The same apic_system_id you used in the /etc/neutron/plugins/ml2/ml2_conf_cisco_apic.ini file on the Neutron server(s).

- The prefix for all APIC domain, names, and profiles created and seen in the APIC GUI.

where <hostname of this system> is the Linux server hostname local to each of the OpenStack hosts.

Step 8

Save the new /etc/opflex-agent-ovs/conf.d/10-opflex-connection.conf file, ensuring not to alter any other format or bracketing of the JSON structure.

Step 9

The IP address next to hostname in this example is a fabric interface for OpFlex communication, if the ACI fabric was installed with default IP address pool for tunnel endpoints; 10.0.0.0/16. If this IP address pool was altered during the fabric install, change the addressing used here to match the fabric. SSH to a leaf switch and use the show ip interface command to identify the addresses in use on your fabric. The hostname address for the OpFlex peer is the anycast IP address assigned to the SVI of the infra VLAN on the leaf switches.

Step 10

If using VXLAN encapsulation between server and leaf switch, proceed to the next section, Configuring the OpFlex Agent Port Using VXLAN Encapsulation.
If using VLAN encapsulation, skip to the Configuring the OpFlex Agent Port Using VLAN Encapsulation section.

Configuring the OpFlex Agent Port Using VXLAN Encapsulation

This section describes how to configure the OpFlex agent port using VXLAN encapsulation,

Use this section, if you have chosen to use VXLAN encapsulation between the OpenStack servers and the ACI leaf switches.

Procedure

Step 1

The OpFlex configuration requires a second set of override values specific to the VXLAN configuration between the host and the Leaf switch. Create a new /etc/opflex-agent-ovs/conf.d/20-vxlan-aci-renderer.conf file using the example contents shown below:

Example:

{
"renderers": {
"stitched-mode": {
"ovs-bridge-name": "br-int",

"encap": {
"vxlan" : {
"encap-iface": "br-int_vxlan0",
"uplink-iface": "eth1.4093",
"uplink-vlan": 4093,
"remote-ip": "10.0.0.32",
"remote-port": 8472
}
},
"flowid-cache-dir": "/var/lib/opflex-agent-ovs/ids"
}
}
}

where eth1.4093 is the interface name and VLAN number for your OpFlex infra VLAN interface.

where 4093 is the VLAN number for your OpFlex infra VLAN interface.

Step 2

Save the new /etc/opflex-agent-ovs/conf.d/20-vxlan-aci-renderer.conf file, ensuring not to alter any other format or bracketing of the JSON structure.

Step 3

The IP address for remote-ip in the 20-vxlan-aci-renderer.conf file is a default fabric interfaces for OpFlex communication, if the ACI fabric was installed with default IP address pool for tunnel endpoints; 10.0.0.0/16. If this IP address pool was altered during the fabric install, change the addressing used here to match the fabric. SSH to a leaf switch and use the show vlan extended and show ip interface commands to identify the addresses in use on your fabric. The remote-ip address matches the anycast ip address assigned to interface Loopback 1023 on the leaf switches.

Step 4

In order to utilize VXLAN encapsulation between the OpenStack servers and the ACI Leaf switches, a VXLAN interface needs to be defined in OVS. This interface name should match the encap-iface setting in the opflex-agent-ovs.conf file, enter the following command:

Example:

ovs-vsctl add-port br-int br-int_vxlan0 -- set Interface br-int_vxlan0 type=vxlan
options:remote_ip=flow options:key=flow options:dst_port=8472

Step 5

Depending on the installation tool used to provision OpenStack, you may have extra ports and bridges configured in your OVS setup that are no longer required. For example, an OVS bridge called br-ex is commonly provisioned for an External network on Neutron nodes and will not be required. An interface bridge such as br-ethX is commonly provisioned by a VLAN-mode packstack installation for carrying VLAN traffic; its function is replaced by tenant network interface being added directly into br-int. You can use the ovs-vsctl commands del-br and del-port to remove unnecessary bridges or patch connections, until the simplified OVS configuration looks similar to the ovs-vsctl show output below:

Example:

Bridge br-int
fail_mode: secure
Port br-int
Interface br-int
type: internal
Port "br-int_vxlan0"
Interface "br-int_vxlan0"
type: vxlan
options: {dst_port="8472", key=flow, remote_ip=flow}
ovs_version: "2.4.1.gbp"

When VM instances are started on a compute node, the system will dynamically add OVS interfaces starting with qvo into br-int to link them to the individual Linux bridges used for attaching each VM. VM traffic transits br-int as programmed by agent-ovs out the tenant VXLAN interface to the ACI fabric.

Configuring the OpFlex Agent Port Using VLAN Encapsulation

This section describes how to configure the OpFlex agent port using VLAN encapsulation.

Use this section, if you have chosen to use VLAN encapsulation between the OpenStack servers and the ACI leaf switches.

Procedure

Step 1

The OpFlex configuration requires a second set of override values specific to the VLAN configuration between the host and the Leaf switch. Create a new /etc/opflex-agent-ovs/conf.d/20-vlan-aci-renderer.conf file using the example contents shown below:

Example:

{
"renderers": {
"stitched-mode": {
"ovs-bridge-name": "br-int",

"encap": {
"vlan" : {
"encap-iface": "<tenant-VLAN-trunk>"
}
},
"flowid-cache-dir": "/var/lib/opflex-agent-ovs/ids"
}
}
}

where <tenant-VLAN-trunk> is the interface name for your tenant VLAN trunk interface.

Step 2

Save the new /etc/opflex-agent-ovs/conf.d/20-vlan-aci-renderer.conf file, ensuring not to alter any other format or bracketing of the JSON structure.

Step 3

The interface for OpenStack tenant networking from the compute nodes is a physical interface supporting VLAN trunking. In some cases this will be the parent interface of the infra VLAN subinterface. In the case of a VPC and Cisco VIC based configuration is described in the Manually Configure the Host vPC section. This would be the separate main-bond interface where LACP traffic is transmitted. This interface name must match the encap-iface setting in the opflex-agent-ovs.conf file. Add the tenant VLAN trunk interface to the OVS bridge br-int, using the following command syntax:

Example:

ovs-vsctl add-port br-int <tenant-VLAN-trunk>

Step 4

Depending on the installation tool used to provision OpenStack, you may have extra ports and bridges configured in your OVS setup that are no longer required. For example, an OVS bridge called br-ex is commonly provisioned for an external network on Neutron nodes and will not be required. An interface bridge such as br-ethX is commonly provisioned by a VLAN-mode packstack installation for carrying VLAN traffic; its function is replaced by tenant network interface being added directly into br-int. You can use the ovs-vsctl del-br and ovs-vsctl del-port commands to remove unnecessary bridges or patch connections, until the simplified OVS configuration looks similar to the ovs-vsctl show output below:

Example:

Bridge br-int
fail_mode: secure
Port br-int
Interface br-int
type: internal
Port <tenant-VLAN-trunk>
Interface <tenant-VLAN-trunk>
ovs_version: "2.4.1"

When VM instances are started on a compute node, the system will dynamically add OVS interfaces starting with qvo into br-int to link them to the individual Linux bridges used for attaching each VM. VM traffic transits br-int as programmed by agent-ovs out the tenant VLAN interface to the ACI fabric.

Starting and Enabling the Agent Services

This section describes how to start and enable the agent services.

Procedure

Step 1

With the OpFlex configuration in place, start and enable the neutron-opflex-agent and agent-ovs services, enter the following commands:

Example:

service agent-ovs restart
service neutron-opflex-agent restart
systemctl enable agent-ovs
systemctl enable neutron-opflex-agent

Step 2

The APIC host agent is required to provide LLDP auto discovery of host-to-server connectivity between OpenStack servers and ACI leaf switches. The host agent listens to LLDP information coming from the ACI fabric to decode which leaf switch and physical port where each compute node is connected, and updates that information to the OpenStack controller. Execute the following to start the agent and ensure it is persistent:

Example:

service neutron-cisco-apic-host-agent restart
systemctl enable neutron-cisco-apic-host-agent

Step 3

Once all the services are running, you must verify that the interface for the OpFlex infra VLAN is in UP state, or bring up the interface using the ifup <interface-name> command.

Initializing the ACI Tenant

This section describes how to initialize the ACI tenant.

Procedure

Step 1

The ACI OpenStack Plugin software is now up and running and ready to provision tenant networks in OpenStack. The population of the APIC with OpenStack configuration will not begin until the first network segment is created in OpenStack with the ACI OpenStack Plugin active. Create a test Neutron network under the Admin Project in OpenStack to trigger initial creation of the ACI Tenant and VMM Domain in APIC.

Step 2

Once the network is created, Log in to the APIC GUI.
Note:
In Application Policy Infrastructure Controller (APIC) 1.2(1x) release, choose the Advanced mode when logging into the APIC GUI.

In the APIC 1.2(1x) release, Cisco recommends that you do not mix configuration modes (Advanced or Basic). When you make a configuration in either mode and change the configuration using the other mode, unintended changes can occur. For example, if you apply an interface policy to two ports using Advanced mode and then change the settings of one port using Basic mode, your changes might be applied to both ports.

1. On the menu bar, choose TENANTS to verify there is a new ACI Tenant created, named after your ACI OpenStack Plugin system name.

Step 3

On the menu bar, choose VM NETWORKING.

1. In the navigation pane, expand OpenStack and verify there is a VMM domain created for your system.

2. This VMM domain needs to be associated with the AEP that was referenced by the Interface Policy Groups when the OpenStack server connections were provisioned in APIC. On the menu bar, choose FABRIC > ACCESS POLICIES.

3. In the navigation pane, expand Global Policies > Attachable Access Entity Profiles and choose the AEP that is referenced by the Interface Policies > Policy Groups definitions for your OpenStack servers.

4. In the PROPERTIES pane, in the Domains field, click the "+" icon to add the OpenStack VMM Domain to the list of associated domains in the AEP.

5. Click Submit.

Step 4

With the ACI tenant initialized from OpenStack, you can verify the basic capabilities of your installation by spinning up a couple networks, VM instances, and a router in OpenStack and verifying expected connectivity. You can observe the EPG's and Bridge Domains being created dynamically under the ACI tenant within APIC, while orchestrating all of the activity through the OpenStack Horizon or CLI interfaces.
Note:
The apic_provision_infra=true setting in the /etc/neutron/plugins/ml2/ml2_conf_cisco_apic.ini file triggers the creation of the VMM domain, AEP, and a VLAN pool, if VLAN encapsulation is in use. It will also trigger creation of Interface and Switch level policy groups visible under FABRIC > ACCESS POLICIES in the APIC. If manually configured server hostlinks are in use the extra Interface and Switch policy groups starting with an underscore character will not be referenced. These groups can be left alone, or deleted.

Chapter 4. Enabling Optimized Neutron Services

This chapter contains the following sections:

- Enabling the Optimized Neutron Services

Enabling the Optimized Neutron Services

This section describes the configuration of distributed DHCP and Metadata Proxy functions.

The ACI OpenStack Plugin software stack can enable optimized functions for Local Layer 3, NAT, DHCP, and Metadata Proxy. The Local Layer 3 forwarding is inherent in the system. Therefore, does not need to be configured.

Distributed NAT services on the compute nodes are tied to enabling an External network in Neutron, see the Adding an OpenStack External Network section.

- Optimized DHCP Services

- Optimized Metadata Proxy

Optimized DHCP Services

There is a configuration line in the /etc/neutron/plugins/ml2/ml2_conf_cisco_apic.ini file called enable_optimized_dhcp, the default setting if not spelled out in the file is “True”. If left to the default or is set to “True”, the Discovery, Offer, Response, and Acknowledgement (DORA) functions that interact with the VM instances is kept local to each compute node.

The local agent-ovs service handles this interaction for each compute node. Address allocation is still handled on the Neutron server(s) by the neutron-dhcp agent and communicated to the agent-ovs instances over the Management network. If set to “False”, then the system reverts to a centralized DHCP function with all address allocation and DORA functions on the Neutron server(s).

Optimized Metadata Proxy

This section describes how to enable optimized metadata proxy.

Enabling the optimized metadata proxy requires a setting within the /etc/neutron/plugins/ml2/ml2_conf_cisco_apic.ini file called enable_optimized_metadata. The default setting is "False". Therefore, if it is not referenced in the file then traditional centralized metadata proxy over the tenant network will be used.

Procedure

Step 1

Edit the /etc/neutron/plugins/ml2/ml2_conf_cisco_apic.ini file and set enable_optimized_metadata=True.

Step 2

Make sure the /etc/neutron/metadata-agent.ini file exists and is correct, if it is not present you should copy this file from your controller.

Step 3

Make sure the neutron-ml2-driver-apic package is also installed on compute nodes.

Step 4

Disable the metadata agent on the neutron servers, and restart neutron-server, enter the following commands:

Example:

systemctl stop neutron-metadata-agent
systemctl disable neutron-metadata-agent
systemctl restart neutron-server

Step 5

Then on the compute nodes, the neutron-ml2-driver-apic package needs to be installed. Once this package is available and the opflex agent services are restarted, the system will begin to function in a distributed mode for metadata proxy, enter the following commands:

Example:

yum install neutron-ml2-driver-apic
systemctl restart neutron-opflex-agent
systemctl restart agent-ovs

Chapter 5. Adding an OpenStack External Network

This chapter contains the following sections:

- Adding an OpenStack External Network

Adding an OpenStack External Network

This section describes how to add an OpenStack External Network.

The OpFlex configuration for an External OpenStack network requires an External Routed Network or Layer 3 Out to exist in the APIC tenant or common configuration. This Layer 3 Out provides a path to an external routing entity for communication outside of the ACI fabric. There are many variations of Layer 3 Out configuration. OSPF, BGP, or static routing may be used. A routed interface, a vPC with an SVI, or a routed sub-interface may be used. The ACI OpenStack Plugin is able to interact with a pre-existing Layer 3 Out by name as a logical construct; it is up to the ACI system administrator to configure the appropriate routing settings for your environment. If a Layer 3 Out does not already exist, see the ACI External Routed Network Example section for the steps that may be used for a simple configuration or reference the Connecting Application Centric Infrastructure (ACI) to Outside Layer 2 and 3 Networks white paper: Click Here

Before You Begin

- Two separate IP subnets are required for SNAT and Floating IP communication on the external network. The upstream router(s) outside of the ACI fabric must be configured with IP routes for these two subnets, either through the routing protocol in use, or statically.

- Before completing this configuration, ensure you have an understanding of the IP subnet requirements to support Source NAT and Floating IP functions.

Procedure

Step 1

Configure the ACI OpenStack Plugin to communicate with the External Routed Network on the Neutron server(s), edit the /etc/neutron/plugins/ml2/ml2_conf_cisco_apic.ini file and add the following section:

Example:

[apic_external_network: <name of L3-Out> ]
preexisting=True
enable_nat=False
external_epg= <name of EPG>
host_pool_cidr= <ip of SNAT default gateway/prefix bits>

where <name of L3-Out> is the name of the External Routed Network defined in the tenant or common Networking folder in APIC.

where enable_nat=False disables NAT operation on this network. To use NAT operation, you can omit this parameter since the default is enable_nat=True.

It is possible to mix and match NAT and no NAT operation using multiple L3-Outs. Each L3-Out requires its own apic_external_network section and you will need to create a neutron provider network with same name.

where <name of EPG> is the name defined under the Networks folder, underneath the Layer 3 Out in APIC. This is referred to as an "EPG Network" when first created.

where <ip of SNAT default gateway/prefix bits> is the default gateway on the subnet used for SNAT, and should be identified with /prefix notation, for example 10.1.2.1/24. This is the address that the ACI OpenStack Plugin will use to add that subnet under the correct Bridge Domain in the APIC.

Step 2

On all the Neutron nodes, restart the neutron-server service for the new configuration settings to be active.

Step 3

Create the external network in OpenStack, the name of the network needs to match the name of apic_external_network value in the ml2_conf_cisco_apic.ini file. This should be done from the Neutron CLI to allow administrative creation of a network with the new type of “OpFlex.” After sourcing administrative credentials on the Neutron server, use the net-create command in Neutron to add the new network as external, and shared:

Example:

neutron net-create OS-L3Out --router:external --shared

Step 4

Add a subnet to the external network that was created by the initial the net-create command. The subnet added here will be used as the Floating IP range, and will also be added to the correct Bridge Domain in the APIC by the ACI OpenStack Plugin.

Step 5

Execute the neutron net-list command.
You will see a secondary network created by the driver named starting with “host-nat-network-for-internal-use”. This network allows OpenStack and ACI to properly process SNAT traffic for VM instances that do not have Floating IP addresses assigned. The OpFlex system will automatically assign a SNAT address from the host_pool_cidr subnet to each compute host in the OpenStack cluster.

Chapter 6. Adding an OpenStack External Network in Unified Mode

This chapter contains the following sections:

- Adding an OpenStack External Network in Unified Mode

Adding an OpenStack External Network in Unified Mode

For Unified mode, an L3Out on the APIC must be pre-created along with its endpoint group (EPG).

Note

As opposed to the Classical mode, there is no need to specify any L3Out or host_pool_cidr config parameters anywhere in the ostack controller configuration.

There is a change in the command for configuring external-network (new extensions have been added) in the OpenStack to correspond to the above pre-existing L3Out and its endpoint group (EPG). The configuration requires the DN of the EPG for the specific the L3Out in APIC. It can be done in two ways:

Procedure

Step 1

Go to the APIC and get the DN of the External Network using the API-Inspector:

Step 2

Use the aimctl manager command on the Openstack-Controller to get the DN as shown below:
Note:
See the example for L3Out: Datacenter-Out.

Example:

aimctl manager external-network-find
+---------------+----------------+------------+
| tenant_name   | l3out_name     | name       |
|---------------+----------------+------------|
| common        | Datacenter-Out | DcExtPol   |
| common        | Management-Out | MgmtExtPol |
| common        | default        | MgmtNet    |
+---------------+----------------+------------+

aimctl manager external-network-get common Datacenter-Out DcExtPol
+-------------------------+-------------------------------------------------+
|Property                 |Value                                            |
|-------------------------+--------------------------------------------------
| tenant_name             | common                                          |
| l3out_name              | Datacenter-Out                                  |
| name                    | DcExtPol                                        |
| display_name            |                                                 |
| nat_epg_dn              |                                                 |
| provided_contract_names | []                                              |
| consumed_contract_names | []                                              |
| monitored               | True                                            |
| dn                      | uni/tn-common/out-Datacenter-Out/instP-DcExtPol |
+---------------------------------------------------------------------------+

Now the DN from above can be used to create the external network.

For Distributed NAT:

Example:

neutron net-create Datacenter-Out --router:external True --shared --apic:distinguished_names \
type=dict ExternalNetwork=uni/tn-common/out-Datacenter-Out/instP-DcExtPol

For No NAT:

Example:

neutron net-create Datacenter-Out --router:external True --shared --apic:distinguished_names \
type=dict ExternalNetwork=uni/tn-common/out-Datacenter-Out/instP-DcExtPol --apic:nat_type ""

For Floating IPs, add a subnet to external network with the desired floating IP pool cidr.

For SNAT, add a subnet to the external network as below:

Example:

neutron subnet-create Datacenter-Out 10.104.21.0/24 --name ext-subnet --disable-dhcp --gateway \
50.50.50.1 --apic:snat_host_pool True

For GBP external connectivity, first create the external segment using the SNAT subnet:

Example:

gbp external-segment-create Datacenter-Out --subnet-id <SNAT Subnet>

Now this segment can be used as usual to create the network service policy.

Example:

gbp nat-pool-create nat-pool-0 --ip-pool 10.104.31.0/24 --external-segment Datacenter-Out \
gbp network-service-policy-create --network-service-params type=ip_pool,name=nat-pool-0,value=nat_pool \
net-svc-nat-0

Appendix A. Reference Information

This chapter contains the following sections:

- Deploying OpenStack on ACI using Physical Domain

- Virtual Routing and Forwarding and Network Address Translation

- ACI Fabric Initialization Example

- Manually Configure the Host vPC

- Setting Up Auto-Configuration of Host Links

- ACI External Routed Network Example

- Network Constraints Template File

- Troubleshooting the APIC OpenStack Plugin

- Version Information

- Configuring the OpFlex ML2 plugin to use Hierarchical Port Binding

- Configuring SR-IOV

Deploying OpenStack on ACI using Physical Domain

This section describes how to deploy OpenStack on ACI using physical domain.

Procedure

Step 1

In order for the ACI fabric to be able to use dynamic discovery of the OpenStack nodes, a software LLDP stack is needed on the server. On the OpenStack controller node, install the LLDP package, execute the following commands:

Example:

cd /etc/yum.repos.d/
yum install wget
wget http://download.opensuse.org/repositories/home:vbernat/RHEL_7/home:vbernat.repo
yum install lldpd

Step 2

On the OpenStack controller node, start and enable the lldpd service to be reloaded after a server boot.

Example:

service lldpd start
chkconfig lldpd on

Note:
An alternate software LLDP stack may be used depending on your host operating system version, as long as the ACI fabric can dynamically discover the server nodes. If LLDP is not enabled on the compute nodes, you need to manually configure this in the ml2_conf_cisco_apic.ini. For example syntax, see Setting Up Auto-Configuration of Host Links.

Step 3

The OpenStack Network and Compute nodes need to have a pointer added in their yum configuration to allow them to pull the OpFlex software from the Repo Server. On the OpenStack controller node, create the /etc/yum.repos.d/opflex.repo file with the following content, substituting the IP address of your Repo Server in the lines with the baseurl statement:

Example:

[opflex]
name=opflex repo
baseurl=http://<ip address>/opflex
failovermethod=priority
enabled=1
gpgcheck=0

With this configuration in place, check to see if the repo is working properly and verify there are no errors:

Example:

yum makecache

Step 4

On the OpenStack controller node, install the neutron-opflex-agent, APIC API, and ML2/GBP driver along with required supporting modules. These packages are pulled from the EPEL repo, ensure EPEL is enabled on the node for successful installation. Supporting modules python-pip and python-pbr are also required as prerequisites:

Example:

yum install python-pip
yum install python-pbr

Step 5

On the OpenStack controller node, if you are running Juno, ensure the epel-testing repo is enabled by modifying the /etc/yum.repos.d/epel-testing.repo file and make sure enabled=1 is set under the [epel-testing] section as shown below:

Example:

[epel-testing]
name=Extra Packages for Enterprise Linux 7 - Testing - $basearch
#baseurl=http://download.fedoraproject.org/pub/epel/testing/7/$basearch
mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=testing-epel7&arch=$basearch
failovermethod=priority
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7

Step 6

On the OpenStack controller node, install the opflex agent, apicapi, and ml2 driver:

Example:

yum install neutron-opflex-agent apicapi neutron-ml2-driver-apic

Step 7

In case of GBP based installation, on the OpenStack controller node, you must install these additional packages:

- openstack-neutron-gbp

- python-gbpclient

- openstack-dashboard-gbp

- openstack-heat-gbp

Example:

yum install openstack-neutron-gbp python-gbpclient openstack-dashboard-gbp \
openstack-heat-gbp

Step 8

Once the installation completes, the /etc/neutron/neutron.conf file needs to be updated to point to the APIC for networking services. On the OpenStack controller node, edit the existing list of service plugins in the file and with:
For ML2:

Example:

service_plugins = cisco_apic_l3,metering,lbaas

For GBP:

Example:

service_plugins = group_policy,servicechain,apic_gbp_l3,metering

Note:
You need to be cautious not to remove services that do not clash with services required by this plugin. For example, if lbaas and/or metering service is enabled, you need to make sure it stays enabled as shown in above example.

Step 9

For GBP based installation, on the OpenStack controller node, you need to enable the heat plugin for GBP. This is done by making sure the plugin_dirs in the DEFAULT section of /etc/heat/heat.conf file includes the GBP path:

Example:

plugin_dirs = /usr/lib/python2.7/site-packages/gbpautomation/heat

Step 10

On the OpenStack controller node, the ML2 configuration /etc/neutron/plugins/ml2/ml2_conf.ini file also requires the following changes to enable the correct mechanism driver for APIC, and add OpFlex as a new network type.
In case of GBP, use drivers: openvswitch,apic_gbp.

Example:

[ml2]
type_drivers = local,flat,vlan,gre,vxlan
tenant_network_types = vlan
mechanism_drivers = openvswitch,cisco_apic_ml2

[ml2_type_vlan]
network_vlan_ranges = physnet1:2500:3000

[securitygroup]
enable_security_group = True

Step 11

On the OpenStack controller node, edit the /etc/neutron/dhcp_agent.ini file, change the dhcp_driver and verify the other values:

Example:

dhcp_driver = apic_ml2.neutron.agent.linux.apic_dhcp.ApicDnsmasq
ovs_integration_bridge = br-int
enable_isolated_metadata = True

Step 12

On the OpenStack controller node, restart the dhcp agent:

Example:

service neutron-dhcp-agent restart

Step 13

On the OpenStack controller node, ensure that the /etc/neutron/plugins/ml2/openvswitch_agent.ini file contains the following settings in the example below.
If you are using a release older than Liberty, then use the /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.in file instead.

Example:

[ovs]
integration_bridge = br-int
local_ip = <Management IP of the server>
enable_tunneling = False
bridge_mappings = physnet1:br-eth

[agent]
polling_interval = 2
l2_population = False
arp_responder = False

[securitygroup]
enable_security_group = True
firewall_driver = neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver

Step 14

On the OpenStack controller node, create the br-eth bridge and add the uplink interface:

Example:

ovs-vsctl add-br br-eth
ovs-vsctl add-port br-eth <Name of the uplink interface>

Step 15

On the OpenStack controller node, restart the openvswitch agent:

Example:

service neutron-openvswitch-agent restart

Step 16

On the OpenStack controller node, edit the /etc/neutron/plugins/ml2/ml2_conf_cisco_apic.ini file:

Example:

[ml2_cisco_apic]

apic_hosts = 172.31.218.136
apic_username = admin
apic_password = cisco123
apic_use_ssl = True
apic_name_mapping = use_name
enable_optimized_metadata = False
enable_optimized_dhcp = False
enable_aci_routing = True
apic_arp_flooding = True
apic_provision_hostlinks = True
apic_clear_node_profiles = True
apic_provision_infra = True
use_vmm = False

[apic_switch:101]
bm1.sys.cisco.com = 1/19

[apic_switch:102]
bm2.sys.cisco.com = 1/19

[DEFAULT]
apic_system_id = liberty-perf

where <liberty-perf> is:

- The unique name for identifying the OpenStack cluster.

- The name seen on the APIC in the ACI tenant for the OpenStack system and for automated creation of APIC objects by the driver.

The same liberty-perf you used in the /etc/neutron/plugins/ml2/ml2_conf_cisco_apic.ini file on the Neutron server(s).

- The prefix for all APIC domain, names, and profiles created and seen in the APIC GUI.

Step 17

For GBP, on the OpenStack controller node, add the [group_policy] and [group_policy_implicit_policy] sections to the ml2_conf_cisco_apic.ini file. The subnets will be carved out of the 192.168.0.0/16 address space.

Example:

[group_policy]
policy_drivers=implicit_policy,apic

[group_policy_implicit_policy]
default_ip_pool=192.168.0.0/16

Step 18

Once the ml2_conf_cisco_apic.ini file has been edited, it needs to be added into the service definition for the OpenStack neutron-server service to be read for options when the service starts. On the OpenStack controller node, edit the /usr/lib/systemd/system/neutron-server.service file and add --config-file /etc/neutron/plugins/ml2/ml2_conf_cisco_apic.ini to the ExecStart line:

Example:

ExecStart=/usr/bin/neutron-server \
--config-file /usr/share/neutron/neutron-dist.conf \ --config-file /etc/neutron/neutron.conf \
--config-file /etc/neutron/plugin.ini
--config-file /etc/neutron/plugins/ml2/ml2_conf_cisco_apic.ini \ --log-file
/var/log/neutron/server.log

Step 19

On the OpenStack controller node, restart neutron server:

Example:

service neutron-server restart

Step 20

On the compute server, ensure that the /etc/neutron/plugins/ml2/openvswitch_agent.ini file contains the following settings in the example below.
If you are using a release older than Liberty, then use the /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.in file instead.

Example:

[ovs]
integration_bridge = br-int
local_ip = <Management IP of the server>
enable_tunneling = False
bridge_mappings = physnet1:br-eth

[agent]
polling_interval = 2
l2_population = False
arp_responder = False

[securitygroup]
enable_security_group = True
firewall_driver = neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver

Step 21

On the compute server, create the br-eth bridge and add the uplink interface:

Example:

ovs-vsctl add-br br-eth
ovs-vsctl add-port br-eth <Name of the uplink interface>

Step 22

On the compute server, restart the openvswitch agent:

Example:

service neutron-openvswitch-agent restart

Virtual Routing and Forwarding and Network Address Translation

You can deploy Cisco Application Centric Infrastructure (ACI) with virtual routing and forwarding (VRF) and network address translation (NAT) by using one of the following methods:

- If your configuration requires overlapping IP addresses, use one VRF per tenant.

For more information, see Multiple Virtual Routing and Forwardings with Network Address Translation.

- If your configuration does not require overlapping IP addresses, use a single shared VRF for the OpenStack cloud.

For more information, see Single Shared Virtual Routing and Forwarding Without Network Address Translation.

- If you plan to use floating IP addresses, you need NAT, with either multiple VRFs or a single shared VRF.

For more information, see Multiple Virtual Routing and Forwardings with Network Address Translation and Single Shared Virtual Routing and Forwarding with Network Address Translation.

- Multiple Virtual Routing and Forwardings with Network Address Translation

- Single Shared Virtual Routing and Forwarding Without Network Address Translation

- Single Shared Virtual Routing and Forwarding with Network Address Translation

Multiple Virtual Routing and Forwardings with Network Address Translation

Using multiple virtual routing and forwardings (VRFs) with network address translation (NAT) is the default behavior. In this scenario, the Cisco Application Centric Infrastructure (ACI) administrator creates an L3Out in the Common tenant that is attached to a VRF in the Common tenant. As OpenStack projects are created, additional Application Policy Infrastructure Controller (APIC) tenants are created and each has a dedicated private VRF and a shadow L3Out. The shadow L3out is used to represent the NAT function to the VRF in the Common tenant.

The following procedure provides an overview of configuring multiple VRFs with NAT.

Procedure

Step 1

Create an L3Out.

Step 2

Create an Internet VRF.

Step 3

Attach the L3Out to the Internet VRF.
Any additional VRFs that are required will be created by the system.

The configuration file for this scenario includes the following entries:

[ml2_cisco_apic]
per_tenant_context = True
[apic_external_DC-Out]
preexisting = True
enable_nat = True
external_epg=DC-Out-EPG
host_pool_cidr=1.2.3.1/24

Single Shared Virtual Routing and Forwarding Without Network Address Translation

You can use a single shared virtual routing and forwarding (VRF) in the Common tenant without using network address translation (NAT). The primary difference from the single VRF with NAT is that OpenStack networks are directly advertised over the L3Out.

Note

The bridge domains created by Neutron must be connected to the L3Out if you are using dynamic routing.

The following procedure provides an overview of configuring a single shared VRF without NAT.

Procedure

Step 1

Create an L3Out.

Step 2

Create a shared context VRF (Internet) and specify the VRF in the configuration file as shared_context_name.

Step 3

Attach the L3Out to the VRF.

The configuration file for this scenario includes the following entries:

[ml2_cisco_apic]
per_tenant_context = False
shared_context_name=my_shared_context

[apic_external_DC-Out]
preexisting = True
enable_nat = False
external_epg=DC-Out-EPG

Single Shared Virtual Routing and Forwarding with Network Address Translation

You can use a single virtual routing and forwarding (VRF) in the Common tenant rather than creating a separate VRF for each OpenStack project. Since you are using a single VRF, you cannot have overlapping IP addresses between OpenStack projects.

This configuration still employs network address translation (NAT). Only the subnets for floating IP addresses and SNAT are advertised externally, and all traffic going to and from the external network is translated by NAT.

The single VRF mode is triggered by the per_tenant_context parameter. You should also specify the name of the context to use, although by default the system creates one named "shared" if none exists.

The following procedure provides an overview of configuring a single shared VRF with NAT.

Procedure

Step 1

Create an L3Out.

Step 2

Create an Internet VRF.

Step 3

Attach the L3Out to the Internet VRF.

Step 4

Create a shared context VRF and specify the VRF in the configuration file as shared_context_name.

The configuration file for this scenario includes the following entries:

[ml2_cisco_apic]
per_tenant_context = False
shared_context_name=my_shared_context

[apic_external_DC-Out]
preexisting = True
enable_nat = True
external_epg=DC-Out-EPG
host_pool_cidr=1.2.3.1/24

ACI Fabric Initialization Example

This example solution is based on a basic Spine/Leaf switching fabric, installed with all defaults on the APIC configuration other than Fabric Name and controller IP addressing. Three APICs are used to form a highly available cluster. Each APIC is connected to one or more of the leaf switches in the fabric, it is best to use diverse leaf switches for connecting multiple APICs to provide higher availability of the controller services.

The switching system continues to forward traffic regardless of the presence of the APIC cluster. All configuration of the fabric is driven through the cluster so no configuration could be added, changed or deleted without APIC connectivity in place. To ensure that administrative control of the fabric is not dependent on the fabric itself, an Out-Of-Band (OOB) network connection is needed on each of the APICs as shown in Figure below:

APIC Cluster Connectivity

Procedure

Step 1

A good practice for setup of the ACI fabric is to make a note of the serial number of each of the switches in the fabric prior to discovering the fabric. Ideally, the console port of each of the switches is also connected to a terminal server so there is always administrative control regardless of the state of the ACI fabric. To recover the serial number when logged into a switch running an ACI software image, enter the show inventory command at the ACI switch CLI, noting the primary system serial number. This is the number that displays in the APIC during Fabric Discovery, allowing you to assign the correct name and node numbering in your scheme to the devices.

Step 2

To allow the APIC to discover and register the switches in the fabric, log in to the APIC GUI.

1. On the menu bar, choose FABRIC > INVENTORY.

2. In the Navigation pane, choose Fabric Membership.

3. In the Work pane, you should see an entry for the first switch discovered by the APIC.

4. Verify this is the expected first switch for the first APIC in the cluster based on serial number.

5. In the Work pane, choose the switch, right-click and choose Register Switch.

Note:
Assign logical numeric node ID’s and node names that make sense for future troubleshooting, and Virtual Port Channel (vPC) pairing plans. For example, Node ID’s 101/102 for the first two leafs, to be named leaf1/leaf2.

Step 3

Once the first leaf is discovered, the system will pass through that leaf to discover the spine switches, and then use the spine switches to discover remaining leaf switches. Register the additional nodes assigning logical node ID numbers and names according to the spine/leaf fabric layout.

Step 4

Confirm visually that the topology is discovered and physically connected as expected, perform the following actions:

1. On the menu bar, choose FABRIC > INVENTORY.

2. In the Navigation pane, choose Topology.
Discovered Spine/Leaf Topology

3. Once the fabric is discovered, choose Admin > Firmware and validate the firmware versions running on all APICs, and fabric nodes (switches). If needed, upgrade to current or consistent versions before beginning initial configuration.

Manually Configure the Host vPC

This section describes how to manually configure the host vPC.

Manual configuration of vPC connections for OpenStack servers allows the ACI administrator more control over naming of Fabric Access Policy objects. In addition, a single physical vPC can be used to carry multiple network types required by an OpenStack installation.

OpenStack hosts typically use separate physical or logical interfaces for Management/SSH and Tenant networking. In addition, separate interfaces may be allocated for API, storage, provider networks, or other specific purposes. The ACI OpenStack Plugin removes the requirement for a Neutron External network since SNAT and Floating IP access is handled through the ACI fabric and OpFlex with OVS.

The configuration examples in this section illustrate breaking out separate interfaces for Management/SSH traffic and Tenant traffic using 802.1Q VLAN headers. These separate interfaces are presented to the Linux operating system using the following elements:

- Cisco UCS C-Series standalone servers with Virtual Interface Card (VIC) 1225 network adapters.

- Virtual Network Interface Cards (vNICs) are used on the VIC adapters to handle the application of VLAN headers and present separate logical interfaces for each VLAN to the Linux operating system.

- Virtual Port Channel (vPC) is used on a pair of ACI leaf switches to present a highly available port-channel connection to the OpenStack server nodes.

- Bond interfaces are used in the Linux operating system to create a port-channel configuration to mate with the vPC configured on the ACI leaf switch pair.

A similar configuration can be accomplished on basic dual-port 10 Gigabit Ethernet adapters using VLAN sub-interface configuration at the Linux operating system level.

Procedure

Step 1

Access the Cisco Integrated Management Controller (CIMC) interface on each of the OpenStack servers, on the Server tab, perform the following actions:

1. Choose Inventory and navigate to the Cisco VIC Adapters tab.

2. Choose the General tab and choose Modify Adapter Properties.

3. In the Modify Adapter Properties dialog box, ensure the Enable FIP Mode and Enable LLDP boxes are unchecked for the adapter.

4. Click Save Changes.

Step 2

Choose the vNICs tab for the adapter card and perform the following actions:

1. By default, two vNICs named eth0 and eth1 exist in Trunk mode on a typical Cisco VIC. These two vNICs are allocated to separate physical uplinks. To add a vNIC which tags traffic for a specific VLAN, choose Properties.

2. In the Name field, enter the name for the new vNIC (eth4).

3. In the Uplink Port field, enter the uplink port (1).

4. In the VLAN Mode field, in the drop-down list, choose ACCESS.

5. In the Default VLAN field, choose the radio button and enter the VLAN number (168).

Note:
An Access mode vNIC presents an un-tagged virtual PCI interface to the operating system on the server, and tags packets with the Default VLAN on the switching fabric.

Step 3

To complete the configuration, add two vNICs in ACCESS mode on the ACI Infrastructure (infra) VLAN, and two interfaces in ACCESS mode on the VLAN to be used for management/SSH traffic. In this example, the ACI infra VLAN in use is the default setting of 4093, and the management/SSH VLAN is VLAN 168.

Step 4

The Linux operating system on the server will now find six virtual network interfaces after a reboot. In this example, three Linux bond interfaces will be created using these six interfaces to provide high availability over the vPC uplink:

- Main-bond—This interface is build from the original eth0/eth1 trunk mode vNICs on the VIC, and is used to transmit LACP to the upstream switch pair to bring up the vPC. If VLAN Encapsulation is being used between the OpenStack servers and ACI leaf switches, this will be the interface for OpenStack tenant VLAN networks.

- Ten-bond—Built from the two vNICs configured on the VIC carrying the ACI infra VLAN used for OpFlex communication. If VXLAN Encapsulation is being used between the OpenStack servers and ACI leaf switches, this interface will be used to carry OpenStack tenant VXLAN networks. (This interface is required in both VLAN and VXLAN mode operations for carrying OpFlex communication to the ACI fabric.)

- Mgt-bond—Built from the two vNICs carrying the management/ssh VLAN. If management traffic is being carried over the ACI fabric, this is the interface used for management and update of servers and installation of OpenStack.

On a Linux server running RHEL or CentOS, the network interface configurations are located in the /etc/sysconfig/network-scripts directory. Use the MAC Address table from the CIMC vNIC listing to resolve which vNICs are mapped to which VLAN. The ifcfg-<interface name> files for the eth0 and eth1 vNICs likely exist from the initial installation, files for the additional vNICs will need to be created.

Note:
The configuration examples shown in this guide are based on CentOS 7 servers running the “network” service, with the “NetworkManager” service disabled.

Main-bond configuration can be completed using interface files similar to those listed below, in the example configuration the two vNICs map to enp8s0 and enp9s0 in Linux. Change the interface names as needed to match your environment.

1. Example contents of ifcfg-main-bond:

Example:

DEVICE=main-bond
TYPE=Ethernet
ONBOOT=yes
BOOTPROTO=none
NM_CONTROLLED=no
USERCTL=no
BONDING_OPTS="mode=4"

Note:
The mode=4 bonding option allows the untagged interface to send LACP packets to synchronize the port channel with the vPC configuration that will be built on the leaf switches.

2. Example contents of physical main-bond member interface ifcfg-<iface-name>:

Example:

DEVICE=enp8s0
BOOTPROTO=none
ONBOOT=yes
MASTER=main-bond
SLAVE=yes
USERCTL=no
NM_CONTROLLED=no

where enp8s0 is where you enter your interface name.

3. Edit or create two of the above physical interface files for the two vNICs participating in the main-bond, and each of the bond examples.
The interface pair tagged for the Infrastructure VLAN needs to be provisioned to operate as a DHCP client. Therefore, carries a slightly different configuration. Also, the MTU of this interface must be set higher than default to provide space to carry VXLAN header traffic.

Example contents of ten-bond for OpFlex and Tenant traffic:

Example:

PERSISTENT_DHCLIENT=1
DHCPRELEASE=1
DEVICE=ten-bond
TYPE=Ethernet
ONBOOT=yes
BOOTPROTO=dhcp
NM_CONTROLLED=no
USERCTL=no
BONDING_OPTS="mode=2"
MTU=1600

Note:
The mode=2 bonding option allows the tagged interface to hash traffic between the two uplink interfaces while relying on the LACP transmission of main-bond to maintain the vPC.

4. Example contents of physical ten-bond member interface:

Example:

DEVICE=enp10s0
BOOTPROTO=none
ONBOOT=yes
MASTER=ten-bond
SLAVE=yes
USERCTL=no
NM_CONTROLLED=no
MTU=1600

where enp10s0 is where you enter your interface name

5. The interface pair used for vlan-tagged management traffic is slightly different as well, this bond interface has a static IP address assignment and does not require the increased MTU, and also utilizes the mode-2 hash.
Example contents of mgt-bond for management/SSH traffic:

Example:

DEVICE=mgt-bond
TYPE=Ethernet
DEFROUTE=yes
ONBOOT=yes
BOOTPROTO=none
NM_CONTROLLED=no
USERCTL=no
BONDING_OPTS="mode=2"
IPADDR=A.B.C.D
PREFIX=XX
GATEWAY=W.X.Y.Z

where yes is included if the interface is default route for the server.

where A.B.C.D is where you enter your IP address, prefix and gateway.

6. Example contents of physical mgt-bond member interface:

Example:

DEVICE=enp12s0
BOOTPROTO=none
ONBOOT=yes
MASTER=mgt-bond
SLAVE=yes
USERCTL=no
NM_CONTROLLED=no

Step 5

Once all of the bond interface configuration files has been created, and the physical interface configuration files edited/created to be slaves to the bond, ensure none of the physical interfaces are still mapped to an OVS bridge. If you need to remove an interface from an OVS bridge, use the ovs-vsctl del-port command. Once this interface creation is completed, corresponding vPC configuration must be added to the ACI fabric via APIC to support the LACP port channel.

Step 6

To manually provision the vPC configuration necessary to support the OpenStack servers, and match the LACP communication from the main-bond interface. The server vPC is a basic end-node vPC connection, the following steps are provided for reference of a working system's settings.
Fabric Access Policies configure a group of policy settings for the underlying switch fabric that can then be referenced later by Tenant Application Policies, perform the following actions:

1. On the APIC GUI, on the menu bar, choose FABRIC > ACCESS POLICIES.

2. In the Navigation pane, choose Pools.

3. Add a VLAN pool to be used for initial configuration of the OpenStack servers with management interfaces. For a statically configured EPG with VLANs mapped to OpenStack nodes, choose a Static Allocation for the pool.

4. On the menu bar, choose FABRIC > FABRIC POLICIES.

5. In the Navigation pane, choose Global Policies, create an Attachable Access Entity Profile. The profile will describe a group of interfaces with common access requirements. Assign a name to the AEP and check the box for Enable Infrastructure VLAN which allows OpFlex communication to the OpenStack server nodes.

6. On the menu bar, choose FABRIC > FABRIC POLICIES.

7. In the Navigation pane, choose Physical and External domains, create a physical domain for the OpenStack nodes management communication. Associate the physical domain with the VLAN Pool and AEP that was previously created.

Step 7

In the Navigation pane, expand Interface Policies > Policies, create the new interface policies to easily control Port Channel settings (LACP Active), CDP, and LLCP enable/disable status on interfaces. Defining a named policy for example CDP-Enable and CDP-disable options allows an administrator to choose these settings from other screens without updating the default for the entire system.

Step 8

In the Navigation pane, expand Interface Policies > Policies, create node profiles for TOR Switch connecting OpenStack servers, with Interface Selectors referencing the port ID they are connected to on the pair of leaf switches. The same port number on each switch will be used for a dual-homed server connection.

1. When specifying the Access Port Selector identity, pull down the Interface Policy Group menu and choose Create VPC Interface Policy Group. Utilize the CDP, LLDP, and Port Channel policies previously created, and also identify the AEP (OS-AEP-1).
For example, the interface profile has a selector defined for the OpenStack Controller and Compute-1 servers connecting to leaf switch 101.

Step 9

In the Navigation pane, expand Switch Policies > Profiles, create a new switch profile for each of the leaf switches.

1. In the Work pane, under Blocks, choose the leaf switch number.

2. On the first switch configured, from the drop-down list, choose Create Access Switch Policy Group and assign a name for a new policy group to be used for OpenStack Leaf switches.

3. Click Update.

Step 10

After assigning a switch to the profile, choose Next to move to the Associations screen and select the Port Profile previously created for the switch. The example name is OS-PProfile-101. Repeat this process for the second leaf switch in the VPC pair.

Step 11

Now the ports and switches to be used for the VPC have been defined, but the VPC relationship itself needs to be created and assigned a domain ID, if one does not already exist.

1. On the APIC GUI, on the menu bar, choose FABRIC > ACCESS POLICIES.

2. In the Navigation pane, choose Quick Start.

3. In the Work pane, choose the Configure an interface, PC, and VPC link.

4. Click the + icon under VPC Switch Pairs to define the new pairing.

5. Enter a numeric Domain ID for the vPC Domain, and choose the two switches to be associated as a VPC switch pair.

6. Click Save.
For example of VPC Domain 10 has been created between switch 101 and 102, with currently only the management VLAN supported from the VLAN pool in the physical domain. Later, once the ACI OpenStack Plugin has been configured you will associate the OpenStack VMM domain to the AEP which adds the virtual network encapsulation to the vPC.

Step 12

Once the vPC configurations are in place, the state of the port channels can be verified on the APIC, perform the following actions:

1. On the menu bar, choose FABRIC > INVENTORY.

2. In the Navigation pane, expand Pod > Leaf > Interfaces > VPC interfaces > port channel assigned on the given leaf for the vPC.

3. In the PROPERTIES pane, Ensure that the port channel is lacp-active, up, and connected.
If the port channel is not up, verify your physical connections, and that the main-bond interface configured on the Linux operating system for LACP is up and functioning.

Note:
The VLAN numbers displayed on the port channel are VLANs used internally to the system, not the VLAN tags used for the end-node encapsulation on the interfaces.

Step 13

The OpenStack servers now have the basic Fabric Access Policies configured in ACI to identify the vPC port attributes needed to connect servers. Traffic cannot yet flow across the fabric for server management purposes until an End Point Group (EPG) is defined under an ACI Tenant to define policy to allow the communication. To add the EPG, under Tenants on the APIC to the ACI Tenant to be used for the OpenStack system.
If this Tenant does not already exist, choose Add Tenant under the Tenants section to create the Tenant.

Note:
This Tenant name must match the apic_system_id defined later in the Driver configuration on the OpenStack controller or Network node.

Step 14

Within the Tenant, choose Application Profiles, and perform the following actions:

1. Add a profile name for OpenStack management traffic.

2. Within this profile, create an EPG, using a Bridge Domain that is applicable to your environment.
In this example configuration, you are using a simple Layer 2 Bridge Domain to link to a default gateway for management traffic that resides outside of the ACI fabric. The Layer 3 Hardware Proxy and Unicast routing is disabled for this Layer 2 transport domain.

Note:
If applicable to your environment, another option for OpenStack management traffic is to use a Layer 3 enabled Bridge Domain and an EPG that is later linked to a Layer 3 External Network connection via ACI contracts. There are many ways to design this connectivity; a Layer 2 configuration is used in this example for simplicity of the illustration.

Step 15

Once the EPG has been created, perform the following actions:

1. Highlight the Static Bindings folder and choose Deploy Static EPG under Actions to add the VPC interfaces to the EPG. In this example, you will have added a compute node, a controller and an external Layer 2 TOR connection to all communicate directly in tagged mode on management VLAN 168.

With the Management configuration in place, the OpenStack servers with Linux installed are ready for installation of the appropriate OpenStack distribution for your system. In this example configuration, a single interface is used for management/SSH and API traffic between OpenStack functions. With the Cisco VIC/vNIC approach illustrated in this section. More granularity can easily be applied for example to break out the API interface, or add a separate requirement such as a storage interface using a different VLAN tag.

Setting Up Auto-Configuration of Host Links

The Cisco ACI OpenStack Plugin software has the capability to automatically provision host links or vPC connections for OpenStack servers. Auto-Configuration is most useful when the management/SSH network used to install OpenStack is carried over a separate physical network interface on the server nodes. The auto-configuration needs to run in conjunction with an installed OpenStack system, and OpenStack generally requires at least the management/SSH/API interface to be already functional at installation time. Auto-configuration only configures interfaces for tenant networking, if the ports have not already been configured in APIC.

For more information, see Cisco Application Centric Infrastructure Fundamentals Guide.

Procedure

Step 1

The settings to enable automatic provisioning of host links reside in the /etc/neutron/plugins/ml2/ml2_conf_cisco_apic.ini file. The template file provided in the install package has some commented sections that provide the syntax to configure port definitions. Edit the /etc/neutron/plugins/ml2/ml2_conf_cisco_apic.ini file and change the apic_provision_hostlinks setting to True.

Step 2

Configure host definition blocks in the configuration /etc/neutron/plugins/ml2/ml2_conf_cisco_apic.ini file for each leaf switch with OpenStack nodes attached. The syntax for hosts with individual ports follows the outline in the configuration file:

Example:

# Specify your network topology.
# This section indicates how your compute nodes are connected to the fabric's
# switches and ports. The format is as follows:
#
# [apic_switch:<swich_id_from_the_apic>]
# <compute_host>,<compute_host> = <switchport_the_host(s)_are_connected_to>
#
# You can have multiple sections, one for each switch in your fabric that is
# participating in Openstack.
# An example, note you can list more than one host name per physical port if your
# topology has virtual elements:

[apic_switch:18]
ubuntu6 = 1/1
ubuntu7,ubuntu8 = 1/2

Or, if your servers are dual-homed with port channels to two leaf switches, first you must include the pairing of switches for VPC:

Example:

apic_vpc_pairs = 101:102,103:104

Step 3

Add port specifics according to the following example:

Example:

[apic_switch:101]
server1 = vpc-1-1/bundle-101-1-1-and-102-1-1
server2 = vpc-1-2/bundle-101-1-2-and-102-1-2
[apic_switch:102]
server1 = vpc-1-1/bundle-101-1-1-and-102-1-1
server2 = vpc-1-2/bundle-101-1-2-and-102-1-2

where bundle-101-1-1 represents the leaf switch 101 and port ethernet1/1.

Step 4

Once these configurations are present in the ml2_conf_cisco_apic.ini file, the ACI OpenStack Plugin validates that they are properly reflected in the APIC each time the neutron-server service restarts. If you have manually configured host ports in the file, it replaces the auto-discovery of hypervisor nodes into the VMM domain provided by LLDP. The LLDP approach provides more flexibility with dynamic discovery of nodes, particularly when server host links have already been defined in APIC and do not need to be provisioned by the ACI OpenStack Plugin.

ACI External Routed Network Example

An External Routed Network (L3-Out) in ACI is required to operate as a Neutron external network in OpenStack. A shared L3-Out can be configured under the Common tenant in APIC, or a private one can be added specifically to the APIC tenant allocated to the OpenStack Instance. This routed connection can be configured with static routing, or using a dynamic protocol such as OSPF or BGP depending on requirements of the environment.

There are many configuration options for an L3-Out in ACI. The following steps will provide an example of configuring a static-routed configuration on a single interface. This approach is useful for a test deployment of the ACI OpenStack Plugin with ACI. A production deployment must utilize an L3-Out configuration that is consistent with the rest of the specific ACI environment.

Before You Begin

The interface to be used for the L3-Out must already be configured with a basic Policy Group on APIC, under FABRIC > ACCESS POLICIES and associated with a leaf switch prior to utilizing it for an L3-Out.

Procedure

Step 1

Under either the ACI Common Tenant in the APIC, or the ACI Tenant associated with OpenStack in the APIC GUI, perform the following actions:

1. In the Navigation pane, expand Networking > External Routed Networks > Actions.

2. Choose Create Routed Outside to add a new L3-Out.

3. In the Work pane, enter a name for L3-Out (Example-L3-Out). This primary panel will be used to launch configuration of a Private Network, Domain, and add a node and interface to the L3-Out.

4. In the Private Network field, from the drop-down list, choose Create Private Network and enter the Private Network name.

5. In the External Routed Domain field, from the drop-down list, choose Create Layer 3 Domain, enter a name for the Layer-3 domain.
You can assign an existing AEP to the domain or create a new one depending on how you set up the interface to be used for the L3-Out. You can also create or choose an existing VLAN pool, if VLAN tagging will be required for the external connection.

6. With the Private Network and the External Routed Domain assigned, create a Node Profile, perform the following actions:

7. In the Nodes and Interface Protocol Profiles section, click the + icon.

8. Enter a name to the profile, this will then be used to associate specific nodes (leaf switches) and interfaces to the L3-Out.

9. In the Nodes field, click the + icon to add an ACI Leaf switch node to the profile. This Select Node pane will be used to define settings on the leaf switch used for the L3-Out.

10. In the Select Node pane, in the Node ID field, from the drop-down list, choose a valid Node ID from your ACI topology (topology/pod-1/node-101).

11. The Router ID field is used for OSPF and BGP protocol communication, this can be set to an address of your choosing for a static routed setup.

12. Click the + icon to add a route in the Static Routes, pointing a default route (0.0.0.0/0) to the Next Hop IP (192.168.100.1), which is the address assigned to the external router outside of the ACI fabric on the Link Subnet.

13. In the Interface Profiles section, click + icon to add an interface.

14. The Select Routed Interface pane, allows you to configure a routed interface, or a routed sub-interface, if VLAN tagging is in use. Or a routed SVI that would be used, if a VPC connection is used for the Layer 3 Out. The IP address and prefix assigned to the interface will be the ACI fabric address on the Link Subnet for the L3-Out. If configuring a routed interface, perform the following actions:

- In the Path field, enter the path (topology/pod-1/paths-101/pathep-[eth1/12]).

- In the IP Address field, enter the IP address (192.168.100.2/24).

- In the Secondary IP Addresses field, is blank.

- In the MAC Address field, enter the MAC address (00:22:BD:F8:19:FF).

- In the MTU (bytes) field, enter the MTU bytes (inherit).

- In the Target DSCP field, enter the target DSCP (unspecified).

Note:
If you are using a VPC connection for the L3-Out, assign separate physical IP addresses to each of the switches in the VPC pair, and then a shared secondary IP address to both. The shared address can be used as the static route destination for inbound traffic from the external router.

The completed Create Node Profile pane with a node and a specific interface assigned should have the following:

- Example-Leaf1 in the Name field.

- unspecified in the Target DSCP field.

- Node ID is topology/pod-1/node-101 and Static Routes is 0.0.0.0/0 in the Nodes field.

- Name is Example 1-12 and Interfaces is [eith1/2] in the INTERFACE PROFILES section.

At this point the Identity section of the Create Routed Outside pane should be completed.

Step 2

Click Next to move to the External EPG Networks pane, perform the following actions:

1. In the SUBNET section, click the + icon to add the external network.

2. Enter a name for the network that will be referenced as the external EPG in the ACI OpenStack Plugin configuration file and enter 0.0.0.0/0 as the subnet.
Note:
Once the External EPG Network is created, it will appear under the External Routed Network in the APIC in the Networks folder. When viewed in the Networks folder it will not be referred to as an EPG.

3. Click Finish to complete the configuration of the L3-Out.

Step 3

You can verify the configuration in the APIC using the constructs it creates under the External Routed Networks folder in the APIC tenant for OpenStack. To verify connectivity for the L3-Out itself, you can also use the following commands from the CLI of the connected leaf switch:

Example:

show vrf
show ip interface
iping –V <name of external vrf> <ip address of external router>

The show vrf command displays a vrf added to the leaf switch for the L3 Out.

The show ip interface command displays an interface carrying the IP address assigned.

You should also be able to ping inbound to the IP address assigned to ACI on the Link Subnet from the external router the address assigned in the step 1n.

Network Constraints Template File

Edit the /etc/neutron/plugins/ml2/cisco_apic_network_constraints.ini file and choose your values:

[DEFAULT]

# The subnet scope to use on APIC if no other constraint
# has been explicitly specified. Valid values are
# public, private or deny.
# public -> Subnet will be advertised externally
# private -> Subnet is private to VRF
# deny -> Disallow creation of subnet
# subnet_scope = public|private|deny

# Tenant (project)-specific constraints and network-specific
# constraints are described in sections of their own.
#
# A tenant section looks like:
# [tenant-name]
# ...
#
# A network section looks like:
# [tenant-name/network-name]
# ...
#
# Network-specific constraints, when specified, take preference over
# tenant-specific constraints.

# Both sections may have the following configuration keys:
# deny -> Comma-separated list of CIDRs. If the requested
#         subnet overlaps with a deny CIDR, then creation of
#         the subnet is disallowed.
# private -> Comma-separated list of CIDRs. If the requested
#           subnet is contained within a private CIDR, then
#           the subnet will be created with 'private' scope
#           (i.e. private to the corresponding VRF).
# public -> Comma-separated list of CIDRs. If the requested
#           subnet is contained within a public CIDR, then
#           the subnet will be created with 'public' scope
#           (i.e. advertised externally).
# default -> The scope to use if the subnet does not match
#            any of the explicitly specified CIDRs. Valid
#            values are public, private or deny.
#
# When deciding subnet scope, the order of preference is deny,
# private, public. Thus if the requested subnet is present in
# both private and public CIDRs, the scope used will be private.
#
# Example:
#
# [tenant1/network1]
# public = 10.10.10.0/24, 10.10.20.0/28
# deny = 30.10.0.0/16
# default = private
#
# [tenant1]
# private = 50.50.50.0/26
# default = deny

Troubleshooting the APIC OpenStack Plugin

The following checklist items can be used to help isolate and correct issues if the ACI OpenStack Plugin installation does not appear to be functioning properly:

- Verify the OpFlex interface and sub-interface is getting DHCP, and can ping the anycast IP addresses listed in the OpFlex configuration file.

- Verify neutron-l3-agent is disabled on Neutron servers.

- Verify Neutron-openvswitch-agent is disabled on compute nodes.

- Check APIC for Faults listed on the Faults tab associated with each EPG created for OpenStack, or check overall system faults on the APIC.

- Ensure neutron-server service restarts, if the controller/Neutron server has been rebooted. If it is not running after a reboot, restart manually.

- Check that neutron-opflex-agent and agent-ovs are running on each compute node. Restart both of these services, if system configuration has changed. If centralized metadata or dhcp are in use they are also required on the controllers.

- If the ml2_conf_cisco_apic.ini file settings have changed, ensure neutron-server service was restarted to read in the new configuration from the file.

- Verify the settings in the ml2_conf_cisco_apic.ini file against example provided. Ensure there are no missing items.

- Verify the settings in the opflex-agent-ovs.conf file additions in the conf.d file against the separate examples for VLAN and VXLAN based configuration. Ensure no bracketing or formatting has been altered in the file.

- Verify the VMM domain for OpenStack is associated to the AEP for server ports in APIC.

- Check to make sure the OpenStack tenant has been created in the APIC named for the system name in the ml2_conf_cisco_apic.ini file.

- Check neutron server and other associate process logging under the /var/log directory.

- Verify if endpoint files are being created under /var/lib/opflex-agent-ovs/endpoints directory for new VM instances. These files should be created by neutron-opflex-agent on the compute node.

- Check physical interface states for server uplinks on the servers, and on the APIC.

- Verify the correct openvswitch kernel module installed by running modinfo openvswitch. Version should be 2.4.1.gbp

- If using VXLAN mode, verify multicast route with "ip route”. Must contain an entry such as: "224.0.0.0/4 dev bond0.4093” for the infra VLAN subinterface.

Version Information

The configuration examples validated and captured in this guide were produced using the following hardware and software versions:

- Red Hat RDO Juno running on CentOS 7.1, Cirros QCOW2 test VM instances

- Cisco ACI/APIC version 1.1(4e) and 11.1(4e).

- Cisco Nexus 9504 ACI Spine switches

- Cisco Nexus 9396PX ACI Leaf switches

- OpenStack servers: UCS C220 M3S, Cisco VIC 1225

Configuring the OpFlex ML2 plugin to use Hierarchical Port Binding

This section describes how to configure the Opflex ML2 plugin to use Hierarchical Port Binding (HPB).

Note

There have been many changes in structure of the .ini file for the OpFlex ML2 plugin, which, in this document is referred to as ml2_conf_cisco_apic.ini and is located in /etc/neutron/plugins/ml2 directory. Below section describes the changes in the file.

For an example of the ml2_conf_cisco_apic.ini file, see Click Here .

- The [ml2_cisco_apic] main section has been replaced by the [apic] section. The parameters for this section are identical to the [ml2_cisco_apic] section.

- The [apic_vm_dom] section has been added to configure VMM domain names and their corresponding encapsulation type.

- The [apic_physdom] section is introduced to configure the phys_dom name to use in ACI.

- The [apic_physical_network] section is added to provide physnet name for HBP and also list nodes to be treated as HBP nodes as well as the encap type to use.

- Using the new config scheme it is now possible to configure a single node to use multiple physnets.

The below topology diagram and sample .ini files are provided for simplification. This example uses a cluster of OpFlex compute nodes and two HPB nodes, one node (physica01) running mainstream openvswitch plugin and other (balancer01) is a load-balancer with HBP support.

In the below example, nodes running default opflex will use a VMM_Domain name HBP-test with vxlan encapsulation as specified by the apic_system_id and the [vmm_dom:HBP-test] section. A Physical domain (phys_dom) “HBP” has been created on ACI with a corresponding vlan pool to use used for HPB as specified by [apic_physdom:HBP] section. This same vlan pool should also be configured in the ml2 configuration for example “network_vlan_ranges = physnet1:1600:1799”.

Note

For nodes using HPB, switch port mapping uses the [apic_switch] section.

Typical HBP Network Topology

Description:

- Compute and Controller - VXLAN/VLAN

- Load-Balanced and Physical - VLAN only

- Preparing the ACI

- OpenStack Preparation and Integration

- Example of the ml2_conf_cisco_apic.ini File

- Sample of the ml2_conf.ini File

Preparing the ACI

This section describes how to prepare the ACI.

Procedure

Step 1

Create the VLAN pool with the desired VLAN range. (In this example it is “1600-1799”.)

Step 2

Create a physical domain and associate this pool and take note of the name. (In this example it is “HPB”.)

Step 3

Create the attachable access entity profile (AEP) and attach the physical domain to it. If the AEP was already created during OpFlex installation, use the exiting AEP and attach the physical domain to it.

Step 4

Make sure the interface and switch profiles have been created for all leaf switch ports connected to the nodes in this cluster.

OpenStack Preparation and Integration

OpenStack preparation and integration is identical to the steps provided in the beginning of this guide excepts for the changes in the ml2_conf_cisco_apic.ini file described in this section. Since HBP local segments use VLAN, you also need to add the VLAN range to the ml2_conf.ini file.

For more information, see Click Here and Click Here .

Procedure

Example of the ml2_conf_cisco_apic.ini File

[apic]
apic_hosts = 10.10.210.2
apic_username = jl1321
apic_password = jltu123
apic_use_ssl = True
apic_name_mapping = use_name

integrated_topology_service=True
enable_aci_routing = True
vni_ranges = 11000:11100
apic_provision_hostlinks = false
enable_optimized_metadata = False
apic_clear_node_profiles = True
apic_provision_infra = false
apic_arp_flooding = True

[apic_switch:101]
physical01 = 1/13

[apic_switch:102]
balancer01 = 1/14

[DEFAULT]
apic_system_id = HBP-test

[opflex]
networks = *

[apic_vmdom:HPB-test]
encap_mode=vxlan

[apic_physdom:HPB]

[apic_physical_network:physnet1]
hosts=physical01,balancer01
segment_type=vlan

[apic_external_network:L3-Out]
preexisting = True
external_epg = L3-Out-EPG
host_pool_cidr = 10.100.2.1/24

Sample of the ml2_conf.ini File

[ml2]
type_drivers = opflex,local,flat,vlan,gre,vxlan
tenant_network_types = opflex
mechanism_drivers =openvswitch,cisco_apic_ml2
path_mtu = 0
[ml2_type_vlan]
network_vlan_ranges = physnet1:1600:1799

[securitygroup]
firewall_driver = neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver
enable_security_group = True

Configuring SR-IOV

This section describes configuring the Single Root I/O Virtualization (SR-IOV) ml2 mechanism agent to work with OpFlex plugin. This is accomplished by using Hierarchical Port Binding (HPB) and should work without any special modification to the configuration.

Here are the basic steps needed to configure OpFlex with SR-IOV:

1. Configure the driver to use HPB.

For more information, see Configuring the OpFlex ML2 plugin to use Hierarchical Port Binding

2. When configuring the ml2 mechanism the order of the listed mechanisms is important. Make sure the “sriovnicswitch” mechanism is listed before any OpFlex mechanism (apic_aim, cisco_apic_ml2 or apic_gbp). For example:

mechanism_drivers =sriovnicswitch,apic_aim

mechanism_drivers =sriovnicswitch,cisco_apic_ml2

3. You may need to specify switch port mapping for the leaf ports connected to the SR-IOV NICs using “[apic_switch]” section.

Procedure

Step 1

The sriov mechanism should be the first mechanism configured.

Step 2

Make sure the sriov agent is configured and running on nodes using SR-IOV as recommended by the OpenStack distribution. To verify, enter:

Example:

neutron agent-list

Step 3

Make sure you are using a supported SR-IOV NIC.

Step 4

The interface used for SR-IOV cannot be used by any other mechanism.

Preface

This preface includes the following sections:

- Audience

- Document Conventions

- Related Documentation

- Documentation Feedback

- Obtaining Documentation and Submitting a Service Request

Audience

This guide is intended primarily for data center administrators with responsibilities and expertise in one or more of the following:

- Virtual machine installation and administration

- Server administration

- Switch and network administration

Document Conventions

Command descriptions use the following conventions:

Convention

Description

bold

Bold text indicates the commands and keywords that you enter literally as shown.

Italic

Italic text indicates arguments for which the user supplies the values.

[x]

Square brackets enclose an optional element (keyword or argument).

[x | y]

Square brackets enclosing keywords or arguments separated by a vertical bar indicate an optional choice.

{x | y}

Braces enclosing keywords or arguments separated by a vertical bar indicate a required choice.

[x {y | z}]

Nested set of square brackets or braces indicate optional or required choices within optional or required elements. Braces and a vertical bar within square brackets indicate a required choice within an optional element.

variable

Indicates a variable for which you supply values, in context where italics cannot be used.

string

A nonquoted set of characters. Do not use quotation marks around the string or the string will include the quotation marks.

Examples use the following conventions:

Convention

Description

screen font

Terminal sessions and information the switch displays are in screen font.

boldface screen font

Information you must enter is in boldface screen font.

italic screen font

Arguments for which you supply values are in italic screen font.

< >

Nonprinting characters, such as passwords, are in angle brackets.

[ ]

Default responses to system prompts are in square brackets.

!, #

An exclamation point (!) or a pound sign (#) at the beginning of a line of code indicates a comment line.

This document uses the following conventions:

Note

Means reader take note. Notes contain helpful suggestions or references to material not covered in the manual.

Caution

Means reader be careful. In this situation, you might do something that could result in equipment damage or loss of data.

Warning

IMPORTANT SAFETY INSTRUCTIONS

This warning symbol means danger. You are in a situation that could cause bodily injury. Before you work on any equipment, be aware of the hazards involved with electrical circuitry and be familiar with standard practices for preventing accidents. Use the statement number provided at the end of each warning to locate its translation in the translated safety warnings that accompanied this device.

SAVE THESE INSTRUCTIONS

Related Documentation

Application Policy Infrastructure Controller (APIC) Documentation

Companion documents for APIC, Cisco APIC Getting Started Guide, Cisco APIC Basic Configuration Guide, Cisco APIC Layer 2 Networking Configuration Guide, Cisco APIC Layer 3 Networking Configuration Guide, Cisco APIC NX-OS Style Command-Line Interface Configuration Guide, Cisco APIC REST API Configuration Guide, Cisco APIC Layer 4 to Layer 7 Services Deployment Guide, and Cisco ACI Virtualization Guide are available at the following URL: Click Here

Cisco Application Centric Infrastructure (ACI) Documentation

The broader ACI documentation is available at the following URL: Click Here .

Cisco Application Centric Infrastructure (ACI) Simulator Documentation

The Cisco ACI Simulator documentation is available at Click Here .

Cisco Nexus 9000 Series Switches Documentation

The Cisco Nexus 9000 Series Switches documentation is available at Click Here .

Cisco Application Virtual Switch Documentation

The Cisco Application Virtual Switch (AVS) documentation is available at Click Here .

Cisco Application Centric Infrastructure (ACI) Integration with OpenStack Documentation

Cisco ACI integration with OpenStack documentation is available at Click Here .

Documentation Feedback

To provide technical feedback on this document, or to report an error or omission, please send your comments to apic-docfeedback@cisco.com. We appreciate your feedback.

Obtaining Documentation and Submitting a Service Request

For information on obtaining documentation, using the Cisco Bug Search Tool (BST), submitting a service request, and gathering additional information, see What's New in Cisco Product Documentation at: Click Here

Subscribe to What’s New in Cisco Product Documentation, which lists all new and revised Cisco technical documentation as an RSS feed and delivers content directly to your desktop using a reader application. The RSS feeds are a free service.

THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS IN THIS MANUAL ARE SUBJECT TO CHANGE WITHOUT NOTICE. ALL STATEMENTS, INFORMATION, AND RECOMMENDATIONS IN THIS MANUAL ARE BELIEVED TO BE ACCURATE BUT ARE PRESENTED WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. USERS MUST TAKE FULL RESPONSIBILITY FOR THEIR APPLICATION OF ANY PRODUCTS.

THE SOFTWARE LICENSE AND LIMITED WARRANTY FOR THE ACCOMPANYING PRODUCT ARE SET FORTH IN THE INFORMATION PACKET THAT SHIPPED WITH THE PRODUCT AND ARE INCORPORATED HEREIN BY THIS REFERENCE. IF YOU ARE UNABLE TO LOCATE THE SOFTWARE LICENSE OR LIMITED WARRANTY, CONTACT YOUR CISCO REPRESENTATIVE FOR A COPY.

The Cisco implementation of TCP header compression is an adaptation of a program developed by the University of California, Berkeley (UCB) as part of UCB's public domain version of the UNIX operating system. All rights reserved. Copyright © 1981, Regents of the University of California.

NOTWITHSTANDING ANY OTHER WARRANTY HEREIN, ALL DOCUMENT FILES AND SOFTWARE OF THESE SUPPLIERS ARE PROVIDED “AS IS" WITH ALL FAULTS. CISCO AND THE ABOVE-NAMED SUPPLIERS DISCLAIM ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING, WITHOUT LIMITATION, THOSE OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE.

IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THIS MANUAL, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

Any Internet Protocol (IP) addresses and phone numbers used in this document are not intended to be actual addresses and phone numbers. Any examples, command display output, network topology diagrams, and other figures included in the document are shown for illustrative purposes only. Any use of actual IP addresses or phone numbers in illustrative content is unintentional and coincidental.

Cisco and the Cisco logo are trademarks or registered trademarks of Cisco and/or its affiliates in the U.S. and other countries. To view a list of Cisco trademarks, go to this URL: Click Here . Third-party trademarks mentioned are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (1110R)