Chapter 1. Overview

About the Multi-Node Setup

Primary Node

Service Nodes

Database Nodes

Minimum System Requirements for a Multi-Node Setup

Minimum System Requirements for a Small Multi-Node Setup

Minimum System Requirements for a Medium Multi-Node Setup

Minimum System Requirements for a Large Multi-Node Setup

Guidelines and Limitations for a Multi-Node Setup

Best Practices for a Multi-Node Setup

Upgrade of a Multi-Node Setup

Chapter 2. Configuring a Multi-Node Setup

Summary of Steps for Configuring a Multi-Node Setup

Creating the Inventory Database

Creating the Monitoring Database

Creating the Primary Node

Creating a Service Node

Setting URL Redirection from a Service Node to Primary Node

System Tasks

Creating a Node Pool

Creating a System Task Policy

Assigning a Node Pool to a System Task Policy

Creating a Service Node

Assigning a System Policy to a System Task

Executing System Tasks

Disabling or Enabling a System Task

Backing Up with a Multi-Node Setup

Restoring with a Multi-Node Setup

Removing a Service Node

Migrating a Standalone Appliance Database to a Multi-Node Setup

Chapter 3. Troubleshooting a Multi-Node High Availability Setup

Troubleshooting Primary Node Failures

Troubleshooting Monitoring Database Failures

Troubleshooting Inventory Database Failures

Troubleshooting Inventory Collection Performance Issues

Preface

Audience

Conventions

Related Documentation

Documentation Feedback

Obtaining Documentation and Submitting a Service Request

Copyright Page

Chapter 1. Overview

This chapter contains the following sections:

- About the Multi-Node Setup

- Minimum System Requirements for a Multi-Node Setup

- Guidelines and Limitations for a Multi-Node Setup

- Best Practices for a Multi-Node Setup

- Upgrade of a Multi-Node Setup

About the Multi-Node Setup

The multi-node setup is supported for Cisco UCS Director on a 64-bit operating system only. With a multi-node setup, you can scale Cisco UCS Director to support a larger number of VMs than is supported by a single installation of Cisco UCS Director. This setup includes the following nodes:

- One primary node

- One or more service nodes

- One monitoring database

- One inventory database

Multi-Node Topology

Note

For a multi-node setup, you have to install the license on the primary node only.

A multi-node setup improves scalability by offloading the processing of system tasks, such as inventory data collection, from the primary node to one or more service nodes. You can assign certain system tasks to one or more service nodes. The number of nodes determines how the processing of system tasks are scaled.

Node pools combine service nodes into groups and enable you to assign system tasks to more than one service node. This provides you with control over which system task is executed by which service node or group of service nodes. If you have multiple service nodes in a node pool and one of the service nodes is busy when a system task needs to be run, Cisco UCS Director uses a round-robin assignment to determine which service node should process that system task. if all service nodes are busy, you can have the primary node run the system task.

However, if you do not need that level of control over the system tasks, you can use the default task policy and add all of the service nodes to the default node pool. All system tasks are already associated with the default task policy, and the round-robin assignment will be used to determine which service node should process a system task.

If you want to have some critical tasks processed only by the primary node, you can assign those tasks to a local-run policy.

For more information about how to configure the primary node and service nodes, and how to assign system tasks, see the Cisco UCS Director Administration Guide.

- Primary Node

- Service Nodes

- Database Nodes

Primary Node

A multi-node setup can have only one primary node. This primary node contains the license for Cisco UCS Director.

The workflow engine is always on the primary node. The primary node also contains the configuration for the node pools and the service nodes, along with the list of which system tasks can be offloaded to the service nodes for processing.

Service Nodes

A multi-node setup can have one or more service nodes. The number of service nodes in a multi-node setup depends upon the number of devices and VMs you plan to configure and manage through Cisco UCS Director.

The service nodes processes any system tasks offloaded by the primary node. If the service nodes are not set up or reachable, the primary node will execute the system tasks.

Database Nodes

The inventory and monitoring databases are created from the Cisco UCS Director MySQL database. The data that Cisco UCS Director collects is divided between the two databases. The multi-node setup segregates the data collection which is historically very heavy on the database into a separate database.

Inventory Database

A multi-node setup can have only one inventory database. This database contains the following:

- Physical and virtual accounts and their related inventory data

- Data used in the normal operation of Cisco UCS Director for all supported features

Monitoring Database

A multi-node setup can have only one monitoring database. This database contains the data that Cisco UCS Director uses for historical computations, such as aggregations and trend reports.

The parameters of the monitoring database depend upon the number of devices and VMs you plan to configure and manage through Cisco UCS Director

Minimum System Requirements for a Multi-Node Setup

The minimum system requirements for a multi-node setup depends upon the number of VMs that need to be supported by Cisco UCS Director. We recommend deploying a Cisco UCS Director VM on a local datastore with a minimum of 25 Mbps I/O speed, or on an external datastore with a minimum of 50 Mbps I/O speed. The following table describes the number of VMs supported by each deployment size.

Deployment Size

Number of VMs Supported

Small

5,000 to 10,000 VMs

Medium

10,000 to 20,000 VMs

Large

20,000 to 50,000 VMs

- Minimum System Requirements for a Small Multi-Node Setup

- Minimum System Requirements for a Medium Multi-Node Setup

- Minimum System Requirements for a Large Multi-Node Setup

Minimum System Requirements for a Small Multi-Node Setup

The small multi-node setup supports from 5,000 to 10,000 VMs. We recommend that this deployment include the following nodes:

- One primary node

- Two service nodes

- One inventory database

- One monitoring database

Note

For optimal performance, reserve additional CPU and memory resources.

Minimum Requirements for each Primary Node and Service Node

Element

Minimum Supported Requirement

vCPU

4

Memory

16 GB

Hard disk

100 GB

Minimum Requirements for the Inventory Database

Element

Minimum Supported Requirement

vCPU

4

Memory

30 GB

Hard disk

100 GB (SSD Type Storage)

Minimum Requirements for the Monitoring Database

Element

Minimum Supported Requirement

vCPU

4

Memory

30 GB

Hard disk

100 GB (SSD Type Storage)

Minimum Memory Configuration for Cisco UCS Director Services on Primary and Service Nodes

Service

Recommended Configuration

File Location

Parameter

inframgr

8 GB

/opt/infra/bin/inframgr.env

MEMORY_MAX

Note

To modify the memory settings for the inframgr service, in the inframgr.env file, update the "MEMORY_MAX" parameter to the value you want. After changing this parameter, restart the service for the changes to take effect.

Minimum System Requirements for a Medium Multi-Node Setup

The medium multi-node setup supports between 10,000 and 20,000 VMs. We recommend that this deployment include the following nodes:

- One primary node

- Three service nodes

- One inventory database

- One monitoring database

Note

For optimal performance, reserve additional CPU and memory resources.

Minimum Requirements for each Primary Node and Service Node

Element

Minimum Supported Requirement

vCPU

8

Memory

30 GB

Hard disk

100 GB

Minimum Requirements for the Inventory Database

Element

Minimum Supported Requirement

vCPU

8

Memory

60 GB

Hard disk

100 GB (SSD type storage)

Minimum Requirements for the Monitoring Database

Element

Minimum Supported Requirement

vCPU

8

Memory

60 GB

Hard disk

100 GB (SSD type storage)

Minimum Memory Configuration for Cisco UCS Director Services on Primary and Service Nodes

Service

Recommended Configuration

File Location

Parameter

inframgr

12 GB

/opt/infra/bin/inframgr.env

MEMORY_MAX

Note

To modify the memory settings for the inframgr service, in the inframgr.env file, update the "MEMORY_MAX" parameter to the value you want. After changing this parameter, restart the service for the changes to take effect.

Minimum System Requirements for a Large Multi-Node Setup

The large multi-node setup supports between 20,000 and 50,000 VMs. We recommend that this deployment include the following nodes:

- One primary node

- Six service nodes

- One inventory database

- One monitoring database

Note

For optimal performance, reserve additional CPU and memory resources.

Minimum Requirements for each Primary Node and Service Node

Element

Minimum Supported Requirement

vCPU

8

Memory

60 GB

Hard disk

100 GB

Minimum Requirements for the Inventory Database

Element

Minimum Supported Requirement

vCPU

8

Memory

120 GB

Hard disk

200 GB (SSD type storage)

Minimum Requirements for the Monitoring Database

Element

Minimum Supported Requirement

vCPU

8

Memory

120 GB

Hard disk

600 GB (SSD type storage)

Minimum Memory Configuration for Cisco UCS Director Services on Primary and Service Nodes

Service

Recommended Configuration

File Location

Parameter

inframgr

24 GB

/opt/infra/bin/inframgr.env

MEMORY_MAX

Note

To modify the memory settings for the inframgr service, in the inframgr.env file, update the "MEMORY_MAX" parameter to the value you want. After changing this parameter, restart the service for the changes to take effect.

Guidelines and Limitations for a Multi-Node Setup

Before you configure a multi-node setup for Cisco UCS Director, consider the following:

- The multi-node setup is supported for Cisco UCS Director on a 64-bit operating system only.

- Your multi-node setup can have only one primary node.

- You must provide the UCSD Server primary node IP Address for the VMware OVF Deployment task to provision a VM using its OVF template.

- You must plan the location and IP addresses of your nodes carefully as you cannot reconfigure the types of most nodes later. You can only reconfigure a service node as a primary node. You cannot make any other changes to the type of node. For example, you cannot reconfigure a primary node as a service node or an inventory database node as a monitoring database node.

- You have to install the license on the primary node only.

- After you configure the nodes, the list of operations available in the shelladmin changes for the service nodes, the inventory database node, and the monitoring database node.

Best Practices for a Multi-Node Setup

Before you configure a multi-node setup for Cisco UCS Director, consider the following best practices:

- To maximize output and minimize network latency, we recommend that the primary node, service nodes, inventory database node, and monitoring database node reside on the same host.

- Network latency (average RTT) between the primary or service node and the physical, virtual compute, storage, and network infrastructures should be minimized. A lower average RTT results in increased overall performance.

- You can offload system tasks to available service nodes by associating the service nodes with a service node pool.

- You can reserve more CPU cycles (MHz) and memory than recommended for better performance at system load.

See Minimum System Requirements for a Multi-Node Setup.

- To re-start the services in the multi-node setup, perform the following steps:

Note

Before performing the following steps, make sure that the database service is up and running on the inventory as well as monitoring databases.

1. Stop services on the primary node.

2. Stop services on all the service nodes one by one.

3. Start services on all the service nodes one by one.

4. Start services on the primary node.

Upgrade of a Multi-Node Setup

For detailed information on upgrading a Cisco UCS Director multi-node setup, see the Cisco UCS Director Upgrade Guide.

Chapter 2. Configuring a Multi-Node Setup

This chapter contains the following sections:

- Summary of Steps for Configuring a Multi-Node Setup

- Creating the Inventory Database

- Creating the Monitoring Database

- Creating the Primary Node

- Creating a Service Node

- Setting URL Redirection from a Service Node to Primary Node

- System Tasks

- Backing Up with a Multi-Node Setup

- Restoring with a Multi-Node Setup

- Removing a Service Node

- Migrating a Standalone Appliance Database to a Multi-Node Setup

Summary of Steps for Configuring a Multi-Node Setup

This procedure provides a high-level summary of the steps involved in configuring a multi-node setup.

Note

Plan your multi-node setup carefully. You can only change a service node to a primary node. You cannot change any other type of node after you configure it. For example, you cannot reconfigure a primary node as a service node or an inventory database node as a monitoring database node.

Step 1

Deploy a Cisco UCS Director VM for each node in the multi-node setup.
For more information, see the relevant installation and upgrade guide.

For example, if your multi-node setup, includes a primary node, an inventory database, a monitoring database, and three service nodes, deploy six Cisco UCS Director VMs.

Step 2

In the Cisco UCS Director shelladmin, configure the nodes in the following order:

1. Creating the Inventory Database

2. Creating the Monitoring Database

3. Creating the Primary Node

4. Creating a Service Node

Note:
You must create and start the inventory database and the monitoring database nodes before you configure the primary node and any service nodes.

Step 3

Update the license file in the Cisco UCS Director node that will be the primary node.
You do not need to update the license file on any other node. See the "Updating the License" topic in Cisco UCS Director Installation on VMware vSphere.

Step 4

In Cisco UCS Director on the primary node, configure the system tasks as follows:

1. Create one or more node pools if you want to control the assignment of system tasks by service node, or accept the default node pool.
See Creating a Node Pool.

2. Create one or more system task policies if you want to control the assignment of system tasks by service node, or accept the default task policy.
See Creating a System Task Policy.

3. Configure the service nodes.

4. Configure the primary node.

5. Assign the system tasks to system policies if you want to control the assignment of system tasks by service node.
See Assigning a System Policy to a System Task.

For more information, see System Tasks.

Creating the Inventory Database

Step 1

Log on to the Cisco UCS Director shelladmin on the inventory database node.

Step 2

From the Cisco UCS Director Shell menu, choose Configure Multi-Node Setup and press Enter.

Step 3

From the menu, choose Configure Multi-Node Setup (Advanced Deployment) and press Enter.

Step 4

From the menu, choose Current Node to configure the current node.

Step 5

When prompted, press y to continue with the configuration.

Step 6

From the menu, choose the appropriate deployment type option: Small, Medium, or Large.

Step 7

From the menu, choose Configure as Inventory Database and press Enter.

Step 8

When prompted, press Enter to continue.

Step 9

To verify that the services for the inventory database are up and running, choose Display Services Status and press Enter.
You should see the following lines:

Service          State       PID         %CPU  %MEM    tELAPSED   #Threads
-----------      ------      ------      ---------------------------------
broker         ,  UP          9884       0.4    1.2    04:15:00       31
controller        UP         10025       0.2    1.2    04:14:14       73
eventmgr          UP         10184       1.3    5.4    04:13:29       85
idaccessmgr       UP         10337       0.7    5.7    04:13:23       85
inframgr          UP         10489      21.4   31.7    04:13:18      311
websock           UP         10565       0.0    0.0    04:13:12        1
tomcat            UP         10626       0.6   11.3    04:13:07       53
flashpolicyd      UP         10661       0.0    0.0    04:12:51        1
mysqld            UP          8686      86.8   15.6    04:15:48      167

Database      IP Address     State     Client        Connections
-----------   -----------   -------   ----------     -----------
infradb         127.0.0.1      UP       localhost        20

Disk          Size      Used     Available    %Use    Usage
-----------   -------   ------   ---------   -----   ----------
/dev/sda1      97G       45G       47G        49%     NORMAL
/dev/sda3     194M       35M      149M        20%     NORMAL

Press return to continue ...

After you return to the shelladmin, the menu options change to those available for an inventory database node.

Creating the Monitoring Database

Step 1

Log on to the Cisco UCS Director shelladmin on the monitoring database node.

Step 2

From the Cisco UCS Director Shell menu, choose Configure Multi-Node Setup and press Enter.

Step 3

From the menu, choose Configure Multi-Node Setup (Advanced Deployment) and press Enter.

Step 4

From the menu, choose Current Node to configure the current node.

Step 5

When prompted, press y to continue with the configuration.

Step 6

From the menu, choose the appropriate deployment type option: Small, Medium, or Large.

Step 7

From the menu, choose Configure as Monitoring Database and press Enter.

Step 8

When prompted, press Enter to continue.

Step 9

To verify that the services for the monitoring database are up and running, choose Display Services Status and press Enter.
You should see the following lines:

Service          State        PID       %CPU   %MEM    tELAPSED   #Threads
-----------      ------      ------     -----------------------------------
broker         ,  UP          9884       0.4    1.2    04:15:00       31
controller        UP         10025       0.2    1.2    04:14:14       73
eventmgr          UP         10184       1.3    5.4    04:13:29       85
idaccessmgr       UP         10337       0.7    5.7    04:13:23       85
inframgr          UP         10489      21.4   31.7    04:13:18      311
websock           UP         10565       0.0    0.0    04:13:12        1
tomcat            UP         10626       0.6   11.3    04:13:07       53
flashpolicyd      UP         10661       0.0    0.0    04:12:51        1
mysqld            UP          8686      86.8   15.6    04:15:48      167

Database      IP Address     State     Client        Connections
-----------   -----------   -------   ----------     -----------
infradb         127.0.0.1       UP       localhost        20

Disk          Size      Used     Available    %Use    Usage
-----------   -------   ------   ---------   -----   ----------
/dev/sda1      97G       45G       47G        49%     NORMAL
/dev/sda3     194M       35M      149M        20%     NORMAL

Press return to continue ...

After you return to the shelladmin, the menu options change to those available for a monitoring database node.

Creating the Primary Node

Before You Begin

The inventory and monitoring databases must be up and running before you create the primary node.

Note

Do not run any daemons on the primary database.

Step 1

Log on to the Cisco UCS Director shelladmin on the primary node.

Step 2

From the Cisco UCS Director Shell menu, choose Configure Multi-Node Setup and press Enter.

Step 3

From the menu, choose Configure Multi-Node Setup (Advanced Deployment) and press Enter.

Step 4

From the menu, choose Current Node to configure the current node.

Step 5

When prompted, press y to continue with the configuration.

Step 6

From the menu, choose the appropriate deployment type option: Small, Medium, or Large.

Step 7

From the menu, choose Configure as Primary Node and press Enter.

Step 8

At the Provide Inventory DB IP prompt, enter the IP address assigned to the Cisco UCS Director VM for the inventory database.
This step registers the VM as a primary node with the inventory database.

Step 9

At the Provide Monitoring DB IP prompt, enter the IP address assigned to the Cisco UCS Director VM for the monitoring database.
This step registers the VM as a primary node with the monitoring database.

Step 10

When prompted, press Enter to continue.

Step 11

To verify that the services for the primary node are up and running, choose Display Services Status and press Enter.
You should see the following lines:

Service          State        PID       %CPU   %MEM    tELAPSED   #Threads
-----------      ------      ------     -----------------------------------
broker         ,  UP          9884       0.4    1.2    04:15:00       31
controller        UP         10025       0.2    1.2    04:14:14       73
eventmgr          UP         10184       1.3    5.4    04:13:29       85
idaccessmgr       UP         10337       0.7    5.7    04:13:23       85
inframgr          UP         10489      21.4   31.7    04:13:18      311
websock           UP         10565       0.0    0.0    04:13:12        1
tomcat            UP         10626       0.6   11.3    04:13:07       53
flashpolicyd      UP         10661       0.0    0.0    04:12:51        1
mysqld            UP          8686      86.8   15.6    04:15:48      167

Database      IP Address          State       Client           Connections
-----------   -----------         -------  -------------       ------------
Inventory       172.29.109.134       UP       172.29.109.131     15
Monitoring      172.29.109.135       UP       172.29.109.131     12

Disk          Size         Used     Available      %Use    Usage
-----------   -------      ------   -----------    -----   ----------
/dev/sda1     477M         111M     341M           25%     NORMAL
/dev/sdb      50G          28G      20G            59%     NORMAL

Press return to continue ....

After you return to the shelladmin, the menu options change to those available for a primary node.

Creating a Service Node

Before You Begin

The inventory and monitoring databases must be up and running before you create the primary node.

Step 1

Log on to the Cisco UCS Director shelladmin on the service node.

Step 2

From the Cisco UCS Director Shell menu, choose Configure Multi-Node Setup and press Enter.

Step 3

From the menu, choose Configure Multi-Node Setup (Advanced Deployment) and press Enter.

Step 4

From the menu, choose Current Node to configure the current node.

Step 5

When prompted, press y to continue with the configuration.

Step 6

From the menu, choose the appropriate deployment type option: Small, Medium, or Large.

Step 7

From the menu, choose Configure as Service Node and press Enter.

Step 8

At the Provide Inventory DB IP prompt, enter the IP address assigned to the Cisco UCS Director VM for the inventory database.
This step registers the VM as a service node with the inventory database.

Step 9

At the Provide Monitoring DB IP prompt, enter the IP address assigned to the Cisco UCS Director VM for the monitoring database.
This step registers the VM as a service node with the monitoring database.

Step 10

When prompted, press Enter to continue.

Step 11

To verify that the services for the service node are up and running, choose Display Services Status and press Enter.
You should see the following lines:

Service          State        PID       %CPU   %MEM    tELAPSED   #Threads
-----------      ------      ------     -----------------------------------
broker         ,  UP          9884       0.4    1.2    04:15:00       31
controller        UP         10025       0.2    1.2    04:14:14       73
eventmgr          UP         10184       1.3    5.4    04:13:29       85
idaccessmgr       UP         10337       0.7    5.7    04:13:23       85
inframgr          UP         10489      21.4   31.7    04:13:18      311
websock           UP         10565       0.0    0.0    04:13:12        1
tomcat            UP         10626       0.6   11.3    04:13:07       53
flashpolicyd      UP         10661       0.0    0.0    04:12:51        1
mysqld            UP          8686      86.8   15.6    04:15:48      167

Database      IP Address            State      Client           Connections
-----------   -----------           -------  -------------      ------------
Inventory       172.29.109.134        UP       172.29.109.131        15
Monitoring      172.29.109.135        UP       172.29.109.131        12

Disk          Size         Used     Available      %Use    Usage
-----------   -------      ------   -----------    -----   ----------
/dev/sda1     477M         111M     341M           25%     NORMAL
/dev/sdb      50G          28G      20G            59%     NORMAL

Press return to continue ....

After you return to the shelladmin, the menu options change to those available for a service node.

Step 12

Repeat this procedure for every service node.

Setting URL Redirection from a Service Node to Primary Node

You can enable automatic redirection from a service node to a primary node. Whenever users try to log on to the Cisco UCS Director appliance from a service node, the automatic redirection takes the user to the appliance in the primary node. To enable automatic redirection, do the following in the Cisco UCS Director appliance on the primary node. Specifically, add the primary node IP Address in the Mail Setup pane.

Before You Begin

Configure the primary node and the service nodes before setting URL redirection from a service node to a primary node.

Step 1

On the menu bar, choose Administration > System.

Step 2

Choose the Mail Setup tab.

Step 3

In the Mail Setup pane, complete the following fields:

Name

Description

Outgoing Email Server (SMTP) field

The outgoing SMTP server address.

Outgoing SMTP Port field

The outgoing SMTP server port number.

Outgoing SMTP User field

The user ID.

Outgoing SMTP Password field

The user password.

Outgoing Email Sender Email Address field

The sender’s email address

Server IP address field

The IP address or DNS name of the primary node. The redirection from the service node happens to the IP address provided here.

Send Test Email check box

Check this check box to test the current email settings.

System Tasks

The System Tasks tab displays all the system tasks that are currently available in Cisco UCS Director. However, this list of system tasks is linked to the type of accounts that you have created in Cisco UCS Director. For example, if you have logged in for the first time, then only a set of general system-related tasks or VMware related tasks are visible on this page. When you add accounts, such as rack accounts or Cisco UCS Manager accounts, system tasks related to these accounts are populated on this page.

Following are the tasks that you can complete from the System Tasks page:

- View the available systems tasks—You can use the Expand and Collapse options to view all the system tasks that are available on this page. The tasks are categorized according to the accounts available in Cisco UCS Director. For example: Cisco UCS Tasks or NetApp Tasks.

- Disable and enable system tasks—In circumstances when there are multiple processes or tasks running on the appliance, you can choose to disable a system task. If you do so, then until such time that you manually enable it, the system task will not run. This will affect the data populated in other reports. For example, if you disable an inventory collection system task, then reports that require this data may not display accurate data. In this case, you will have to manually run an inventory collection process, or enable the system task.

For more information, see Disabling or Enabling a System Task.

In a single-node setup, where there is only one server, all system tasks will run on this server. In a multi-node setup, where there are multiple servers configured, all system tasks, by default, run on the primary server. However, you can specify system tasks to run on the secondary servers. Following are the recommended steps to perform this task:

1. Ensure that the secondary servers are available in Cisco UCS Director as nodes. If the servers are not available, then you must add the servers as nodes. For more information, see Creating a Service Node.

2. Create a node pool from the available servers. For more information, see Creating a Node Pool.

3. Create a system task policy, and associate it with a node policy. For more information, see Creating a System Task Policy.

4. Associate a node pool with the system task policy. For more information, see Assigning a Node Pool to a System Task Policy.

5. Select a system task, and associate it with a system-task policy. For more information, see Assigning a System Policy to a System Task.

- Creating a Node Pool

- Creating a System Task Policy

- Assigning a Node Pool to a System Task Policy

- Creating a Service Node

- Assigning a System Policy to a System Task

- Executing System Tasks

- Disabling or Enabling a System Task

Creating a Node Pool

Step 1

On the menu bar, choose Administration > System.

Step 2

Choose the Service Nodes tab.

Step 3

Click the Service Node Pools icon. The Service Node Pool dialog box displays.

Step 4

Click the + (plus) icon. The Add Entry to Service Node Pools dialog box displays.

Step 5

In the Name field, enter the node pool name.

Step 6

(Optional) In the Description field, enter a description of the node pool name.

Step 7

Click Submit. The node pool is created.

Creating a System Task Policy

As an administrator, you can choose to combine a few policies and create a system task policy, in addition to the default system task policy. You can group system tasks into a system task policy to later determine which system tasks are running on which node.

Step 1

On the menu bar, choose Administration > System.

Step 2

Choose the System Task Policy tab.

Step 3

Click the Add icon. The Add dialog box displays.

Step 4

In the Name field, enter the name that you gave the system task policy.

Step 5

(Optional) In the Description field, enter a description of the system task policy.

Step 6

From the Node Pool drop-down list, choose the node pool to which this system task policy belongs.

Step 7

Click Submit.
The selected node pool now belongs to the newly created system task policy.

Assigning a Node Pool to a System Task Policy

Step 1

On the menu bar, choose Administration > System.

Step 2

Choose the System Task Policy tab.

Step 3

Select an existing system task policy from the Name column and click the Edit icon. The Edit dialog box displays.
Note:
If the default system task policy is used, you can assign service nodes to this policy. See Creating a System Task Policy, if you want to configure a policy that is different from the default.

Step 4

From the Node Pool drop-down list, choose a node pool to which this System Task Policy belongs.

Step 5

Click Submit.
The selected node pool now belongs to the system task policy.

Creating a Service Node

Step 1

On the menu bar, choose Administration > System.

Step 2

Choose the Service Nodes tab.

Step 3

Click the Add (+)

Step 4

In the Service Node dialog box, complete the following fields:

Name

Description

Node Name field

The name of the service node.

Role field

You cannot edit this field. By default, this field displays Service as the role of this node.

Service Node Pool drop-down list

By default, the default-service-node-pool is displayed.

DNS Name field

Enter either the DNS name or IP address of the service node.
Note:   This field cannot use the Primary Node’s IP address. Ensure that a valid Service Node DNS name or IP address is entered.

Description field

The description of the of the service node.

Protocol drop-down list

Choose either http (default) or https.

Port field

The default TCP port for the Hypertext Transfer Protocol (HTTP) 80 is entered by default. Enter a different TCP port if necessary.

UserName field

The infraUser user name is entered by default.

The infraUser is a user account created by default. To find this user account on the menu bar, choose Administration > Users and Groups.

Choose the Login Users tab to find the infraUser user account in the Login Name column.

Note:   The InfraUser user name is not the default administrator user to login to the system.

Another user name can be added to this field. This user’s API key is used to authenticate with the Service Node.

Step 5

Click Submit.

Assigning a System Policy to a System Task

Step 1

On the menu bar, choose Administration > System.

Step 2

Choose the System Task tab.

Step 3

Choose a folder that contains system tasks. Click the folder arrow to expand its tasks.
Note:
128 system tasks are available.

Step 4

Choose the task and click the Manage Task icon.
The Manage Task dialog box appears.

Step 5

From the Task Execution drop-down list, choose Enable.

Step 6

From the System Task Policy drop-down list, choose a system policy.

Step 7

Click Submit.
The system task is assigned to the selected system policy.

Executing System Tasks

Cisco UCS Director includes a few system tasks that cannot be run remotely on a service node. Also, you can assign a system policy remotely from the local host or the primary node.

In addition, you can search and select a specific system task, and run it immediately in the system.

Step 1

On the menu bar, choose Administration > System.

Step 2

Choose the System Tasks tab.

Step 3

Choose a task from the list.

Step 4

Click Run Now.
The result of the executed system task is updated in the user interface.

Disabling or Enabling a System Task

Step 1

On the menu bar, choose Administration > System.

Step 2

Choose the System Task tab.

Step 3

Choose a folder that contains one or more system tasks. Click the folder arrow to expand its tasks.
Note:
128 system tasks are available.

Step 4

Choose the task and click the Manage Task icon.
The Manage Task dialog box appears.

Step 5

To disable a system task, from the Task Execution drop-down list, choose Disable.

Step 6

To enable a system task, from the Task Execution drop-down list, choose Enable.

Step 7

Click Submit.

Backing Up with a Multi-Node Setup

This procedure provides a high-level summary of the steps required to back up a multi-node setup.

The backup procedure requires you to first stop the Cisco services. This must be done before you work with the Monitoring and Inventory databases. Then you back up the databases, after which you re-start the services, starting with the Primary node.

Before You Begin

To back up the databases, you use an FTP server, for which you will need:

- An FTP server account and the IP address

- The FTP server's login credentials

- Backup file name(s)

SUMMARY STEPS

1.    Shut down the services for both the Primary and Service nodes. Use the, Stop Services Shell potion.

2.    Back up the Monitoring database. Use the Backup Database Shell option.

3.    Backup the Inventory database. Use the Backup Database Shell option.

4.    Start the Services (Service nodes first, followed by Primary node). Use the Start Services Shell option.

DETAILED STEPS

Step 1

Shut down the services for both the Primary and Service nodes. Use the, Stop Services Shell potion.

Step 2

Back up the Monitoring database. Use the Backup Database Shell option.

Step 3

Backup the Inventory database. Use the Backup Database Shell option.

Step 4

Start the Services (Service nodes first, followed by Primary node). Use the Start Services Shell option.

Restoring with a Multi-Node Setup

This procedure provides a high-level summary of the steps required to restore a multi-node setup.

The restore procedure requires you to first stop the Cisco services. This must be done before you work with the Monitoring and Inventory databases. Then you restore the databases, after which you re-start the services, starting with the Primary node.

Before You Begin

To restore the databases, you use an FTP server, for which you will need:

- An FTP server account and the IP address

- The FTP server's login credentials

- Backup file name(s)

SUMMARY STEPS

1.    Shut down the services for both the Primary and Service nodes. Use Shell option 3, Stop Services.

2.    Restore the Monitoring database. Use the Shell Restore Database option.

3.    Restore the Inventory database. Use the Shell Restore Database option.

4.    Start the Services (Service nodes first, followed by Primary node). Use Shell option 4, Start Services.

DETAILED STEPS

Step 1

Shut down the services for both the Primary and Service nodes. Use Shell option 3, Stop Services.

Step 2

Restore the Monitoring database. Use the Shell Restore Database option.

Step 3

Restore the Inventory database. Use the Shell Restore Database option.

Step 4

Start the Services (Service nodes first, followed by Primary node). Use Shell option 4, Start Services.

Removing a Service Node

This procedure provides a high-level summary of the steps required to remove a service node from a multi-node setup.

The removal procedure requires you to first stop the Cisco services. This must be done before you remove a service node from Monitoring and Inventory databases.

Step 1

Shut down the services for both the Primary node and the Service node(s). Use Shell option 3, Stop Services.

Step 2

Start the services on the Service node(s) that you want to be active and operational. Use Shell option 4, Start Services.

Step 3

Start the services on the Primary node. Use Shell option 4, Start Services.

Step 4

Using the Cisco UCS Director GUI, delete the Service node that you want to remove. This removes the respective Service node from the database.

Migrating a Standalone Appliance Database to a Multi-Node Setup

This procedure backs up and restores only the selected database tables required for each of the inventory database and the monitoring database.

Step 1

In the shelladmin, choose Login as Root to log in to the upgraded Cisco UCS Director.

Step 2

Back up the database tables required for the inventory database, as follows:

1. Navigate to the /opt/infra folder.

cd /opt/infra

2. From Infra, run the dbInfraBackupRestore.sh script.

# ./dbInfraBackupRestore.sh backup

This script takes a database backup with the tables required for the inventory database.

3. After the script has been executed, check for an output file in /tmp folder named infra_database_backup.tar.gz.

Step 3

Back up the database tables required for the monitoring database, as follows:

1. Navigate to the /opt/infra folder.

cd /opt/infra

2. From Infra, run the dbMonitoringBackupRestore.sh script.

# ./dbMonitoringBackupRestore.sh backup

This script takes a database backup with the tables required for the monitoring database.

3. After the script has been executed, check for an output file in /tmp folder named monitoring_database_backup.tar.gz.

Step 4

Deploy a Cisco UCS Director VM for each node in the multi-node setup.
For example, if your multi-node setup, includes a primary node, an inventory database, a monitoring database, and three service nodes, deploy six Cisco UCS Director VMs.

Step 5

In the shelladmin, configure the nodes in the following order:

1. Creating the Inventory Database

2. Creating the Monitoring Database

3. Creating the Primary Node

4. Creating a Service Node

Note:
You must create and start the inventory database and the monitoring database nodes before you configure the primary node and any service nodes.

Step 6

In the shelladmin, choose Stop services to stop the Cisco UCS Director services on the primary node and all service nodes.

Step 7

Restore the database to the inventory database, as follows:

1. Copy the infra_database_backup.tar.gz to the /tmp folder on the node that hosts the inventory database.

2. On the inventory database node, navigate to the /opt/infra folder.

cd /opt/infra

3. From Infra, run the dbInfraBackupRestore.sh script.

# ./dbInfraBackupRestore.sh restore

The script restores the database backup from the backup file in the /tmp folder .

Step 8

Restore the database to the monitoring database, as follows:

1. Copy the monitoring_database_backup.tar.gz to the /tmp folder on the node that hosts the monitoring database.

2. Navigate to the /opt/infra folder.

cd /opt/infra

3. From Infra, run the dbMonitoringBackupRestore.sh script.

# ./dbMonitoringBackupRestore.sh restore

The script restores the database backup from the backup file in the /tmp folder .

Step 9

Choose Start services on the primary node and all service nodes to start the Cisco UCS Director services.
After you log in to the primary node, all data from the database in the Cisco UCS Director standalone appliance should be available.

Chapter 3. Troubleshooting a Multi-Node High Availability Setup

This chapter contains the following sections:

- Troubleshooting Primary Node Failures

- Troubleshooting Monitoring Database Failures

- Troubleshooting Inventory Database Failures

- Troubleshooting Inventory Collection Performance Issues

Troubleshooting Primary Node Failures

Problem—The primary node fails.

Possible Cause—A primary node failure may be caused by file system corruption on an external datastore on which the Cisco UCS Director VM is running.

Recommended Solution—Log in to the Cisco UCS Director shelladmin to promote a service node to the primary node.

Step 1

In the Cisco UCS Director shelladmin, choose Configure Multi Node Setup (Advanced Deployment).
The following information displays:

Enter: [a/b/x]?

Step 2

Enter a and press the Enter key.
The following information displays:

Do you want to configure this node as Primary Node [y/n]?

Step 3

Enter y and press the Enter key.
The following information displays:

Configuring Primary Node
Stopping UCS Director Services
Select the IP version you want to configure [a) IPv4, b)IPv6] a/b:

Step 4

Enter a and press the Enter key.
The following information displays:

Provide Inventory DB IP:

Step 5

Enter the inventory database IP address and press the Enter key.
The following information displays:

Provide Monitoring DB IP:

Step 6

Enter the monitoring database IP address and press the Enter key.
The following information displays:

Disabling Database service at startup
Starting UCS Director Services
Configured Primary Node Successfully
In order for changes to take effect logout and login back
Do you want to logout [y/n]?

Step 7

Enter y and press the Enter key.

Troubleshooting Monitoring Database Failures

Problem—The master monitoring database fails.

Possible Cause—Database failures may be caused by:

- A mysqld crash, which may occur if the VM is powered off abruptly.

- A power failure on the node on which the Cisco UCS Director VM is running.

- File system corruption on an external datastore on which the Cisco UCS Director VM is running.

Recommended Solution—Fail over the database by stopping the infrastructure services on the primary node and the service node, replacing the IP address of the master monitoring VM with that of the backup monitoring VM, and restarting services.

Step 1

In the Cisco UCS Director shelladmin, choose Stop services to stop the Cisco UCS Director services on the primary node and all service nodes.

Step 2

Replace the IP address of the master monitoring VM with that of the backup monitoring VM in the following files:

- /opt/infra/inframgr/service.properties

- /opt/infra/eventmgr/service.properties

- /opt/infra/idaccessmgr/service.properties

Step 3

Start the application services on the primary node and the service node.

Troubleshooting Inventory Database Failures

Problem—The master inventory database fails.

Possible Cause—Database failures may be caused by:

- A mysqld crash, which may occur if the VM is powered off abruptly.

- A power failure on the node on which the Cisco UCS Director VM is running.

- File system corruption on an external datastore on which the Cisco UCS Director VM is running.

Recommended Solution—For a multi-node setup, fail over the database by stopping the infrastructure services on the primary node and the service node, replacing the IP address of the master inventory VM with that of the backup inventory VM, and restarting services.

Note

This solution is only applicable for a multi-node setup, and if you have a backup of the corrupted database node.

For a single-node setup, deploy a new Cisco UCS Director appliance and restore the database backup using the shelladmin.

Step 1

In the Cisco UCS Director shelladmin, choose Stop services to stop the Cisco UCS Director services on the primary node and all service nodes.

Step 2

Replace the IP address of the master inventory VM with that of the backup inventory VM in the following files:

- /opt/infra/inframgr/service.properties

- /opt/infra/eventmgr/service.properties

- /opt/infra/idaccessmgr/service.properties

Step 3

Start the application services on the primary node and the service node.

Troubleshooting Inventory Collection Performance Issues

Problem—In a Cisco UCS Director multi-node deployment you may experience performance issues when using a service node to perform inventory collection.

Possible Cause—The execution duration can be directly affected by the following factors:

- Poor disk I/O speed on the external datastore on which the Cisco UCS Director VM is deployed.

- Poor network latency in a multi-node setup.

- Insufficient CPU reservation in the hypervisor.

- Insufficient memory reservation in the hypervisor.

Recommended Solution—Cisco recommends that you deploy Cisco UCS Director VMs in either a local datastore with 25Mbps I/O speed, or an external datastore with 50Mbps I/O speed. Cisco recommends that you reserve at least 3000MHz CPU for the Cisco UCS Director VM in addition to the default number of vCPUs. To troubleshoot any inventory collection performance issues, determine the I/O speed for the local datastore or the external datastore, and determine the round-trip time (RTT) between the following nodes:

- Primary node and VMware vCenter

- Primary node and inventory node

- Service node and VMware vCenter

- Service node and inventory node

Step 1

Run the following command as the root user on the local datastore or the external datastore to determine the I/O speed:

dd if=/dev/zero of=/tmp/test1 bs=4096 count=262144 oflag=direct

Step 2

Check the average RTT between the nodes using the ping -cc 20 Peer IP or hostname command.

Example:

ping -c 20 192.0.2.253

rtt min/avg/max/mdev = 60.474/69.888/134.199/21.529 ms

An average RTT below 50 ms is good.

Step 3

If the average RTT value is above 100 ms, work with your network administrator to debug any possible network latency issues to reduce the RTT to under 100 ms.

Preface

- Audience

- Conventions

- Related Documentation

- Documentation Feedback

- Obtaining Documentation and Submitting a Service Request

Audience

This guide is intended primarily for data center administrators who use Cisco UCS Director and who have responsibilities and expertise in one or more of the following:

- Server administration

- Storage administration

- Network administration

- Network security

- Virtualization and virtual machines

Conventions

Text Type

Indication

GUI elements

GUI elements such as tab titles, area names, and field labels appear in this font.

Main titles such as window, dialog box, and wizard titles appear in this font.

Document titles

Document titles appear in this font.

TUI elements

In a Text-based User Interface, text the system displays appears in this font.

System output

Terminal sessions and information that the system displays appear in this font.

CLI commands

CLI command keywords appear in this font.

Variables in a CLI command appear in this font.

[ ]

Elements in square brackets are optional.

{x | y | z}

Required alternative keywords are grouped in braces and separated by vertical bars.

[x | y | z]

Optional alternative keywords are grouped in brackets and separated by vertical bars.

string

A nonquoted set of characters. Do not use quotation marks around the string or the string will include the quotation marks.

< >

Nonprinting characters such as passwords are in angle brackets.

[ ]

Default responses to system prompts are in square brackets.

!, #

An exclamation point (!) or a pound sign (#) at the beginning of a line of code indicates a comment line.

Note

Means reader take note. Notes contain helpful suggestions or references to material not covered in the document.

Caution

Means reader be careful. In this situation, you might perform an action that could result in equipment damage or loss of data.

Tip

Means the following information will help you solve a problem. The tips information might not be troubleshooting or even an action, but could be useful information, similar to a Timesaver.

Timesaver

Means the described action saves time. You can save time by performing the action described in the paragraph.

Warning

IMPORTANT SAFETY INSTRUCTIONS

This warning symbol means danger. You are in a situation that could cause bodily injury. Before you work on any equipment, be aware of the hazards involved with electrical circuitry and be familiar with standard practices for preventing accidents. Use the statement number provided at the end of each warning to locate its translation in the translated safety warnings that accompanied this device.

SAVE THESE INSTRUCTIONS

Related Documentation

Cisco UCS Director Documentation Roadmap

For a complete list of Cisco UCS Director documentation, see the Cisco UCS Director Documentation Roadmap available at the following URL: Click Here .

Cisco UCS Documentation Roadmaps

For a complete list of all B-Series documentation, see the Cisco UCS B-Series Servers Documentation Roadmap available at the following URL: Click Here .

For a complete list of all C-Series documentation, see the Cisco UCS C-Series Servers Documentation Roadmap available at the following URL: Click Here .

Note

The Cisco UCS B-Series Servers Documentation Roadmap includes links to documentation for Cisco UCS Manager and Cisco UCS Central. The Cisco UCS C-Series Servers Documentation Roadmap includes links to documentation for Cisco Integrated Management Controller.

Documentation Feedback

To provide technical feedback on this document, or to report an error or omission, please send your comments to ucs-director-docfeedback@cisco.com. We appreciate your feedback.

Obtaining Documentation and Submitting a Service Request

For information on obtaining documentation, using the Cisco Bug Search Tool (BST), submitting a service request, and gathering additional information, see What's New in Cisco Product Documentation.

To receive new and revised Cisco technical content directly to your desktop, you can subscribe to the What's New in Cisco Product Documentation RSS feed. RSS feeds are a free service.

THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS IN THIS MANUAL ARE SUBJECT TO CHANGE WITHOUT NOTICE. ALL STATEMENTS, INFORMATION, AND RECOMMENDATIONS IN THIS MANUAL ARE BELIEVED TO BE ACCURATE BUT ARE PRESENTED WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. USERS MUST TAKE FULL RESPONSIBILITY FOR THEIR APPLICATION OF ANY PRODUCTS.

THE SOFTWARE LICENSE AND LIMITED WARRANTY FOR THE ACCOMPANYING PRODUCT ARE SET FORTH IN THE INFORMATION PACKET THAT SHIPPED WITH THE PRODUCT AND ARE INCORPORATED HEREIN BY THIS REFERENCE. IF YOU ARE UNABLE TO LOCATE THE SOFTWARE LICENSE OR LIMITED WARRANTY, CONTACT YOUR CISCO REPRESENTATIVE FOR A COPY.

The Cisco implementation of TCP header compression is an adaptation of a program developed by the University of California, Berkeley (UCB) as part of UCB's public domain version of the UNIX operating system. All rights reserved. Copyright © 1981, Regents of the University of California.

NOTWITHSTANDING ANY OTHER WARRANTY HEREIN, ALL DOCUMENT FILES AND SOFTWARE OF THESE SUPPLIERS ARE PROVIDED “AS IS" WITH ALL FAULTS. CISCO AND THE ABOVE-NAMED SUPPLIERS DISCLAIM ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING, WITHOUT LIMITATION, THOSE OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE.

IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THIS MANUAL, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

Any Internet Protocol (IP) addresses and phone numbers used in this document are not intended to be actual addresses and phone numbers. Any examples, command display output, network topology diagrams, and other figures included in the document are shown for illustrative purposes only. Any use of actual IP addresses or phone numbers in illustrative content is unintentional and coincidental.

Cisco and the Cisco logo are trademarks or registered trademarks of Cisco and/or its affiliates in the U.S. and other countries. To view a list of Cisco trademarks, go to this URL: Click Here . Third-party trademarks mentioned are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (1110R)