In Open SDN Controller, OpenFlow clusters work with datastore clusters
to provide high availability (HA). So what is a cluster? In this context, a
cluster is a collection of datastores that work together as if they are a
single entity, even though they reside on different controller nodes.
Refer to the following topics for an overview of both OpenFlow clusters
and the flow modification process, as well as troubleshooting information:
Open SDN Controller supports two
versions of OpenFlow: versions 1.3 and 1.0. Let’s briefly cover the
similarities and differences in how each version manages clusters.
In OpenFlow 1.3, each switch is connected to every controller node
that belongs to a cluster. The switch assigns one of the following roles to
each controller node:
Master—All synchronous and asynchronous messages are sent
to the master controller node. This node has write privileges on the switch.
Slave—Only synchronous messages are sent to this controller
node. Slave nodes only have read privileges on the switch.
Equal—When this role is assigned to a controller node, that
node has the same privileges as the master node. By default, controller nodes
are assigned the Equal role when they first connect to the switch.
Each datastore on the controller nodes is divided into smaller chunks
known as shards, and one of these shards will act as the leader. For example,
the inventory-operational-shard is present in the inventory datastore for all
of a cluster’s nodes. One of these shards will act as the leader, with the
other shards operating as followers. This is important because the node on
which the inventory-operational-shard leader resides is assigned as the master
node to the switch connected to the cluster.
Since OpenFlow 1.0 does not support roles, the switch that is
connected to a cluster is only connected to one controller node at any given
time (via the use of a floating/virtual IP address). In the event that the node
connected to the switch goes down, the switch is automatically connected to
another node which is then elected as the inventory-operational-shard leader.
Just like in OpenFlow 1.3, each datastore on the controller nodes is
divided into shards, with one of these shards acting as the leader. This is
important because the floating/virtual IP address for this cluster is
configured to point to the node on which the inventory-operational shard leader
resides (the master node).
There are two types of in-memory
datastores: config and inventory. And on each of these datastores, four shards
reside: default, inventory, toaster, and topology. Of note for flow
modifications are the inventory shards in both the config and operational
datastore. In the following topic, we’ll cover what happens when the
inventory-config-shard leader and inventory-operational-shard leader reside on
separate cluster nodes and a flow is modified.
Flows are first added to the inventory-config-shard leader.
The addFlow request is routed from the controller node that made the
request to the node on which the inventory-config-shard leader resides (the
When the flow is received by the inventory-config-shard leader, the
leader replicates the flow and then commits it to the datastore.
A notification is generated when the flow is submitted to the
datastore and an addFlow remote procedure call (RPC) is sent to the remote RPC
All switch RPCs are registered only on the master node.
The Remote RPC connector locates the master node and forwards the
addFlow RPC to it.
When the RPC component of the master node receives the addFlow RPC,
it forwards the RPC to the OpenFlow plug-in, which in turn forwards the RPC to
In the background, the Statistics Manager regularly polls the switch
by executing flow and other statistics RPCs against the switch. The Statistics
Manager is enabled only on the master node.
The switch responds to these RPCs with notifications.
These notifications are sent only to the master node.
When these notifications are received, the Statistics Manager adds
the flows to the inventory-operational datastore.
FAQs and Troubleshooting
How do I determine the role of each controller node known to the
For OVS switches,
run the following command:
sudo ovs-vsctl list CONTROLLER
For other types of
switches, refer to the switch’s documentation for the appropriate command.
The wrong role is assigned to a controller node.
Check the status
of the last role change in the OF Role Service by running the
broadcastRoleChange script. To do so, make a POST request, using the following
RESTCONF request you make, you must first generate a security token. See
RESTCONF Requests for more information.
If the request returns
RUN, this indicates that the controller has submitted a
role change request to the switch in order to assume the Master role.
If the request returns
STANDBY, this indicates that the controller has
submitted a role change request to the switch in order to assume the Slave
You need to make this request
3 times: once for each controller node in the cluster. The status for one node
STANDBY for the other two. If this is not the case,
check the log file for any related errors.
Navigate to the
/opt/cisco/controller/bin/role_scripts directory and open the
If the last line of this file
RUN, this indicates that the Openflowplugin
Orchestration app received the role change request, executed the
broadcastRoleChange script, and made a call to the OF Role Service.
broadcastRoleChange.sh.log file is not present, this indicates that the
Openflowplugin Orchestration app never received the role change request. Open
the log file and look for any related errors.
How do I determine a controller node’s role without accessing
Do one of the
identify the controller node on which the inventory-operational-shard leader
resides. This is the Master controller node.
broadcastRoleChange.sh.log file (located in the
/opt/cisco/controller/bin/role_scripts directory) and check the last line:
RUN is listed, the controller is the Master
STANDBY is listed, the controller is a Slave
I don't see certain flows programmed on the switch in either the
inventory-operational database or OpenFlow Manager.
Check whether the
flow was programmed to the switch.
Open Jconsole and
connect to the Master controller node (on which the inventory-operational-shard
The ID/name of the
switch is displayed.
and determine when the
LastFlowStatsPollTime and LastFlowStatsNotificationReceived objects were last
Wait 15–20 seconds
and then refresh the Attributes screen.
If the attributes
remain unchanged, this indicates that the flow’s statistics are not being
updated. Check the logs for any related errors.
I have added a flow but it does not appear on the switch.
Before you complete
the following procedure, we recommend that you review the "How do flow mods
take place in the clustered environment" topic.
Check whether the
flow is available in the switch’s inventory-config datastore by opening the
<controller-IP-address> is the IP address for any
controller in the cluster.
If it is,
check the logs for any related errors.
If it isn’t, proceed to Step
2. Make sure to note the switch’s ID, which you can find in the following line
in the XML payload:
need this for Step 4.
Find the node on
which the inventory-config-shard leader resides:
JConsole and connect to a controller node.
and look for the Leader
the node tagged with the Leader attribute (unless you are already connected to
to open the RemoteRpcBroker
findRpcByRoute section, enter the switch’s ID.
A popup window
Look for the
cannot find it, look for RPC registration errors in the logs.
If you do
find it, note its value:
local, this indicates that the
inventory-operational-shard leader and inventory-config-shard leader reside on
the same node. Verify this by selecting
and then checking the logs for any addFlow errors that have occurred on this
value is an IP address or hostname, this indicates that the
inventory-operational-shard leader resides on that device. Connect to that
device via Jconsole and go back to Step 3 of this procedure.