The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.
In Open SDN Controller, OpenFlow clusters work with datastore clusters to provide high availability (HA). So what is a cluster? In this context, a cluster is a collection of datastores that work together as if they are a single entity, even though they reside on different controller nodes.
Refer to the following topics for an overview of both OpenFlow clusters and the flow modification process, as well as troubleshooting information:
Open SDN Controller supports two versions of OpenFlow: versions 1.3 and 1.0. Let’s briefly cover the similarities and differences in how each version manages clusters.
In OpenFlow 1.3, each switch is connected to every controller node that belongs to a cluster. The switch assigns one of the following roles to each controller node:
Master—All synchronous and asynchronous messages are sent to the master controller node. This node has write privileges on the switch.
Slave—Only synchronous messages are sent to this controller node. Slave nodes only have read privileges on the switch.
Equal—When this role is assigned to a controller node, that node has the same privileges as the master node. By default, controller nodes are assigned the Equal role when they first connect to the switch.
Each datastore on the controller nodes is divided into smaller chunks known as shards, and one of these shards will act as the leader. For example, the inventory-operational-shard is present in the inventory datastore for all of a cluster’s nodes. One of these shards will act as the leader, with the other shards operating as followers. This is important because the node on which the inventory-operational-shard leader resides is assigned as the master node to the switch connected to the cluster.
Since OpenFlow 1.0 does not support roles, the switch that is connected to a cluster is only connected to one controller node at any given time (via the use of a floating/virtual IP address). In the event that the node connected to the switch goes down, the switch is automatically connected to another node which is then elected as the inventory-operational-shard leader.
Just like in OpenFlow 1.3, each datastore on the controller nodes is divided into shards, with one of these shards acting as the leader. This is important because the floating/virtual IP address for this cluster is configured to point to the node on which the inventory-operational shard leader resides (the master node).
There are two types of in-memory datastores: config and inventory. And on each of these datastores, four shards reside: default, inventory, toaster, and topology. Of note for flow modifications are the inventory shards in both the config and operational datastore. In the following topic, we’ll cover what happens when the inventory-config-shard leader and inventory-operational-shard leader reside on separate cluster nodes and a flow is modified.
Flows are first added to the inventory-config-shard leader.
The addFlow request is routed from the controller node that made the request to the node on which the inventory-config-shard leader resides (the master node).
When the flow is received by the inventory-config-shard leader, the leader replicates the flow and then commits it to the datastore.
A notification is generated when the flow is submitted to the datastore and an addFlow remote procedure call (RPC) is sent to the remote RPC connector.
All switch RPCs are registered only on the master node.
The Remote RPC connector locates the master node and forwards the addFlow RPC to it.
When the RPC component of the master node receives the addFlow RPC, it forwards the RPC to the OpenFlow plug-in, which in turn forwards the RPC to the switch.
In the background, the Statistics Manager regularly polls the switch by executing flow and other statistics RPCs against the switch. The Statistics Manager is enabled only on the master node.
The switch responds to these RPCs with notifications.
These notifications are sent only to the master node.
When these notifications are received, the Statistics Manager adds the flows to the inventory-operational datastore.
How do I determine the role of each controller node known to the switch?
For OVS switches, run the following command:
sudo ovs-vsctl list CONTROLLER
For other types of switches, refer to the switch’s documentation for the appropriate command.
The wrong role is assigned to a controller node.
Check the status of the last role change in the OF Role Service by running the broadcastRoleChange script. To do so, make a POST request, using the following URL:
https://token:$token@<controller-IP-address>/controller/restconf/operations/of-operational-status:get-operational-status
Note the following:
Before every RESTCONF request you make, you must first generate a security token. See Making RESTCONF Requests for more information.
If the request returns RUN, this indicates that the controller has submitted a role change request to the switch in order to assume the Master role.
If the request returns STANDBY, this indicates that the controller has submitted a role change request to the switch in order to assume the Slave role.
You need to make this request 3 times: once for each controller node in the cluster. The status for one node should be RUN and STANDBY for the other two. If this is not the case, check the log file for any related errors.
Navigate to the /opt/cisco/controller/bin/role_scripts directory and open the broadcastRoleChange.sh.log file:
If the last line of this file lists RUN, this indicates that the Openflowplugin Orchestration app received the role change request, executed the broadcastRoleChange script, and made a call to the OF Role Service.
If the broadcastRoleChange.sh.log file is not present, this indicates that the Openflowplugin Orchestration app never received the role change request. Open the log file and look for any related errors.
How do I determine a controller node’s role without accessing the switch?
Do one of the following:
In JConsole, identify the controller node on which the inventory-operational-shard leader resides. This is the Master controller node.
Open the broadcastRoleChange.sh.log file (located in the /opt/cisco/controller/bin/role_scripts directory) and check the last line:
I don't see certain flows programmed on the switch in either the inventory-operational database or OpenFlow Manager.
Check whether the flow was programmed to the switch.
Open Jconsole and connect to the Master controller node (on which the inventory-operational-shard leader resides).
Select
.The ID/name of the switch is displayed.
Select
and determine when the LastFlowStatsPollTime and LastFlowStatsNotificationReceived objects were last accessed.Wait 15–20 seconds and then refresh the Attributes screen.
If the attributes remain unchanged, this indicates that the flow’s statistics are not being updated. Check the logs for any related errors.
I have added a flow but it does not appear on the switch.
Before you complete the following procedure, we recommend that you review the "How do flow mods take place in the clustered environment" topic.
Check whether the flow is available in the switch’s inventory-config datastore by opening the following URL:
http://<controller-IP-address>/controller/restconf/config/opendaylight-inventory:nodes
where <controller-IP-address> is the IP address for any controller in the cluster.
Find the node on which the inventory-config-shard leader resides:
Select
to open the RemoteRpcBroker screen.In the findRpcByRoute section, enter the switch’s ID.
A popup window opens.
Look for the addFlow RPC:
If you cannot find it, look for RPC registration errors in the logs.
If you do find it, note its value:
If the value is local, this indicates that the inventory-operational-shard leader and inventory-config-shard leader reside on the same node. Verify this by selecting
and then checking the logs for any addFlow errors that have occurred on this node.If the value is an IP address or hostname, this indicates that the inventory-operational-shard leader resides on that device. Connect to that device via Jconsole and go back to Step 3 of this procedure.