Troubleshooting

This chapter contains the following sections:

Backing up the Monitoring Database in a Multi-Node Setup

Problem—You are unable to back up the monitoring database in a multi-node setup.

Recommended Solution—Edit the dbMonitoringBackupRestore.sh script.

    Step 1   Edit the /opt/infra/dbMonitoringBackupRestore.sh script using vi.
    Step 2   Remove the CHARGEBACK_HISTORY_ENTRY table name from the script.

    Pinging the Hostname and IP Address

    You can ping a hostname or IP address to test your connectivity by choosing the Ping Hostname/IP address option.

      Step 1   From the Cisco UCS Director Shell menu, choose the Ping Hostname/IP address option and press Enter.
      Step 2   Enter your IP address and press Enter.
      The following information is displayed: 
       Enter IP Address : 209.165.200.224
PING 209.165.200.224 (209.165.200.224) 56(84) bytes of data.
64 bytes from 209.165.200.224: icmp_seq=1 ttl=64 time=9.90 ms
64 bytes from 209.165.200.224: icmp_seq=2 ttl=64 time=0.316 ms
64 bytes from 209.165.200.224: icmp_seq=3 ttl=64 time=0.254 ms
64 bytes from 209.165.200.224: icmp_seq=4 ttl=64 time=0.198 ms
64 bytes from 209.165.200.224: icmp_seq=5 ttl=64 time=0.267 ms

--- 209.165.200.224 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 3999ms
rtt min/avg/max/mdev = 0.198/2.187/9.901/3.857 ms
Press return to continue ...
      Step 3   Press Enter to exit out of the operation.

      Viewing Tail Inframgr Logs

      This Shell lets enables you to see inframgr (Infrastructure Manager) log data, which are generated behind the scenes by use of the Unix tail command. When you are debugging, you can trace problems by using this log data. You use the Tail Inframgr Logs option to immediately tail the most recent inframgr logs. The results are displayed on your screen directly after you select this option.
        Step 1   From the Cisco UCS Director Shell menu, choose the Tail Inframgr Logs option and press Enter.
        Following are a few sample lines, typical of the results displayed immediately after use of the Tail Inframgr Logs option: 
          
2014-07-20 23:17:43,500 [pool-23-thread-17] 
INFO  getBestAgent(SystemTaskExecutor.java:308) 
- No Agent available for remoting SnapMirrorHistoryStatusSchedulerTask
2014-07-20 23:17:43,502 [pool-23-thread-17] 
INFO  updateStatus(SystemTaskStatusProvider.java:181) 
- Task: task.SnapMirrorHistoryStatusSchedulerTask changed state to OK
2014-07-20 23:17:43,562 [pool-23-thread-17] 
INFO  executeLocally(SystemTaskExecutor.java:133) 
- Executing task locally: SnapMirrorHistoryStatusSchedulerTask
2014-07-20 23:17:43,562 [pool-23-thread-17] 
INFO  getClusterLeaf(ClusterPersistenceUtil.java:81) 
- Leaf name LocalHost
2014-07-20 23:17:43,571 [pool-23-thread-17]
        Step 2   To exit from the log file display, type Ctrl+C, then press Enter.

        Collecting Logs from a Node

        The Collect Logs from a Node option lets you collect logs from the local node or from a remote node.

          Step 1   From the Cisco UCS Director Shell menu, choose the Collect Logs from a Node Status option.
          The following list of services appears: 
                   Cisco UCS Director Shell Menu
Node:Standalone | Version:6.0.0.0 | UpTime:  19:57:52 up 2 days, 14:23

1) Change ShellAdmin password
2) Display Services Status
3) Stop Services
4) Start Services
5) Stop Database
6) Start Database
7) Backup Database
8) Restore Database
9) Time Sync
10) Ping Hostname/IP Address
11) Show version
12) Generate Self-Signed Certificate and Certificate Signing Request
13) Import CA/Self-Signed Certificate
14) Configure Network Interface
15) Display Network Details
16) Enable Database for Cisco UCS Director Baremetal Agent
17) Add Cisco UCS Director Baremetal Agent Hostname/IP
18) Tail Inframgr logs
19) Apply Patch
20) Shutdown Appliance
21) Reboot Appliance
22) Manage Root Access
23) Login as Root
24) Configure Multi Node Setup (Advanced Deployment)
25) Clean-up Patch Files
26) Collect logs from a Node
27) Collect Diagnostics
28) Quit

SELECT>
          Step 2   Enter the Logs Collection option.

          • If you choose to collect logs from the current node, a response similar to the following appears: 

            Collecting all feature logs....
==========================================================
                   Collection of Logs
==========================================================
Moving logs from /opt/infra/broker to common/logs
Moving logs from /opt/infra/client to common/logs
Moving logs from /opt/infra/controller to common/logs
Moving logs from /opt/infra/eventmgr to common/logs
Moving logs from /opt/infra/idaccessmgr to common/logs
Moving logs from /opt/infra/inframgr to common/logs
Moving logs from /opt/infra/web_cloudmgr to common/logs

Logs archive path:  /opt/infra/common/logs-07-31-2014-08-36-48.tar
You can also view individual feature logs under /opt/infra/common/logs

Logs collection done for current node
Do you want to collect logs from another node?  [y/n]: Collect Logs from a Node
            Note   

            To collect logs from another node, the best practice is to return to the Shell menu, select the Collect Logs from a Node option again, and choose the Remote Node option.

          • If you choose to collect logs from a remote node, a response similar to the following appears: 

            Please enter the remote server IP/Hostname from where we collect logs:

            Follow the onscreen instructions to provide the address of the remote log, establish a secure connection, and provide the required login credentials for that remote node.

          Collecting Diagnostics

          The Collect Diagnostics option helps to collect logs from a Multi-Node setup and a Standalone setup for debugging purposes.

            Step 1   From the Cisco UCS Director Shell menu, choose Collect Diagnostics.
            The following information is displayed: 
                
Diagnostics Menu
==================
Options:
  1) Collect basic diagnostics
  2) Collect inframgr thread dump and heap dump
  3) Collect full diagnostics
  4) Exit
            Note   

            In a multi-node setup, only Collect basic diagnostics option is supported in inventory and monitoring nodes.

            Step 2   If you choose Collect basic diagnostics option, a response similar to the following appears: 
                
Type in option# : 1
Collecting basic diagnostics................ done
Creating diagnostics archive /opt/infra/diags/standalone_diags_basic_12-19-2017-05-25-39.tgz.... done
Press return to continue ...
            Step 3   If you choose Collect inframgr thread dump and heap dump option, a response similar to the following appears: 
                
Type in option# : 2
Pre-requisites:
1. Download JDK installer jdk-8u131-linux-x64.tar.gz from oracle.com JDK arvhive.
2. Copy the jdk-8u131-linux-x64.tar.gz under /opt/bin.
3. Install the JDK by running the following commands.
        cd /opt/bin
        tar -xzvf jdk-8u131-linux-x64.tar.gz

Enter JDK path if it's already installed (e.g. /opt/bin/jdk1.8.0_131): /root/jdk1.8.0_131/
Collecting infradump diagnostics. This operation may take several minutes to complete.
........ done
Creating diagnostics archive /opt/infra/diags/standalone_diags_infradump_12-19-2017-04-58-13.tgz......................................................... done
Press return to continue ...
            Step 4   If you choose Collect full diagnostics option, a response similar to the following appears: 
                
Type in option# : 3
Pre-requisites:
1. Download JDK installer jdk-8u131-linux-x64.tar.gz from oracle.com JDK arvhive.
2. Copy the jdk-8u131-linux-x64.tar.gz under /opt/bin.
3. Install the JDK by running the following commands.
        cd /opt/bin
        tar -xzvf jdk-8u131-linux-x64.tar.gz

Enter JDK path if it's already installed (e.g. /opt/bin/jdk1.8.0_131): /root/jdk1.8.0_131/
Collecting full diagnostics. This operation may take several minutes to complete.
.......................
Creating diagnostics archive /opt/infra/diags/standalone_diags_full_12-19-2017-05-00-38.tgz........................................................... done
Press return to continue ...
            Note   

            You can share dignostics information with the Cisco TAC team for troubleshooting purpose by raising a TAC case.

            Using Diagnostics Information

            User or TAC engineer can collect the basic diagnostics data using Collect basic diagnostics option in the shelladmin while reporting any issue. The diagnostics bundle contains the following diagnostics data that is used for troubleshooting the reported issues.

            • Summary file—Contains important and high level summary.

            • Diag file—Contains information such as version history with timestamp, average CPU utilization, infra services status, database status, and database size.

            • SummaryReport file—Contains summary report.

            • DiagOutput file—Contains detailed report.

            • UcsdExceptions file—Contains all exceptions found in the inframgr/logfile.txt.* and number of occurrences of each exception.

            • infra-env Directory—Contains the infra services configuration (<service>.env) files.

            • commands Directory—Contains the output of various system commands.

            • var-log-ucsd zip file—Contains the log files such as install.log, bootup.log, and services.log.

            Troubleshooting VMware Console Display Issues

            Problem—The VMware console does not display after an abrupt shutdown of the Cisco UCS Director VM from VMware vCenter.

            Possible Cause—Occasionally after Cisco UCS Director VM is powered on, the VMware console prompt gets stuck after the process restart and does not return to the shelladmin.

            Recommended Solution—After the VM is powered on, press Alt-F1 to refresh the VMware console.

            In the Cisco UCS Director VM prompt after the VM is powered on, press Alt-F1.

            The VMware console screen is refreshed.


            Enabling HTTP Access

            By default, HTTPS access mode is enabled during initial OVF installation and Cisco UCS Director upgrade. When HTTP is enabled, you can log in to Cisco UCS Director, using both HTTP and HTTPS modes. When HTTPS is enabled, you can log in to the Cisco UCS Director only using HTTPS mode. Even when you try to log in to Cisco UCS Director using HTTP mode, you will be redirected to HTTPS user interface only.

              Step 1   From the Cisco UCS Director Shell menu, choose the Enable HTTP/HTTPS option and press Enter.
              The following information is displayed: 
               HTTPS is currently enabled. Do you want to enable HTTP [y/n]? :
              Step 2   Enter y and press Enter. The following information is displayed: 
              UCS Director Services need to be stopped to proceed with the HTTP configuration. Do you want to continue [y/n]?
              Step 3   Enter y and press Enter. The Cisco services are restarted.

              Resetting MYSQL User Password in a Multi-Node Setup

                Step 1   From the Cisco UCS Director Shell menu, choose the Reset MYSQL User password option and press Enter. The following information is displayed: 
                This utility will restart the services after changing MYSQL user password, do you want to continue? [y/n]:
                Note   

                In a multi-node setup, ensure that the infra services are stopped in the primary and service nodes before executing the Reset MySQL User password option in DB nodes.

                Step 2   Enter y and press Enter. The following information is displayed: 
                Stopping the infra services...
The infra services are stopped.
Do you want to change the password for MYSQL 'admin' user? [y/n]:
                Step 3   Enter y and press Enter. The following information is displayed: 
                Current Password (Type in current password or press enter key to use password from the existing credentials file):

                This option is applicable only for the primary and service nodes in a multi-node setup.

                Step 4   Enter y and press Enter. The following information is displayed: 
                Do you want to generate random password for MYSQL 'admin' user? [y/n]:
                Step 5   Enter n and press Enter. The following information is displayed: 
                Specify the new password for MYSQL 'admin' user:
                Step 6   Enter a new MYSQL admin password and press Enter.
                Step 7   Enter your new MYSQL admin password and press Enter. The following information is displayed: 
                MYSQL user password is updated.
Checking if the database is running...yes.
Stopping the database...
The database is stopped.
Starting the database...
The database is started.
Copying credential files to BMA appliance...
Trying to get session to xxx.xxx.xxx.xxx ....
Trying to connect...
Successfully connected
Uloaded file:/opt/certs/mysql/dbkeys.key
Trying to get session to xxx.xxx.xxx.xxx...
Trying to connect...
Successfully connected
Uploaded file:/opt/certs/mysql/dbcreds.properties
Starting the infra services...
Press return to continue...
                Note   

                If a BMA appliance is associated with a Cisco UCS Director, the dbkeys and dbcreds files are copied to a specific location in the BMA appliance to establish successful connectivity to the Cisco UCS Director. After resetting the MySQL user password, you should restart the BMA services either from the Cisco UCS Director user interface or from the BMA appliance.

                Note   

                In a multi-node set up, if you want to reset the MySQL user password, you should execute the Reset MySQL User password option in all the nodes in the following sequence inventory, monitoring, primary, and service nodes.

                Resetting MYSQL User Password in a Standalone Setup

                  Step 1   From the Cisco UCS Director Shell menu, choose the Reset MYSQL User password option and press Enter. The following information is displayed: 
                  This utility will restart the services after changing MYSQL user password, do you want to continue? [y/n]:
                  Step 2   Enter y and press Enter. The following information is displayed: 
                  Stopping the infra services...
The infra services are stopped.
Do you want to change the password for MYSQL 'admin' user? [y/n]:
                  Step 3   Enter y and press Enter. The following information is displayed: 
                  Do you want to generate random password for MySQL 'admin' user? [y/n]:
                  Step 4   Enter y and press Enter. The following information is displayed: 
                  Generating Random Password....... done
Do you want to change the password for MySQL 'root' user? [y/n]:
                  Step 5   If you entered n, enter the new password for MySQL admin user and press Enter. The following information is displayed: 
                  Specify the new password for MySQL 'admin' user:
Confirm the new password for MySQL 'admin' user:
Password update takes few minutes. Please wait..... done
                  Step 6   Enter y and press Enter. The following information is displayed: 
                  Do you want to generate random password for MySQL 'root' user? [y/n]:
                  Step 7   Enter y and press Enter. The following information is displayed: 
                  Generating Random Password....... done
MySQL user password is updated.
Checking if the database is running... yes.
Stopping the database...
.............
The database is stopped.
Starting the database...
Checking if MySQL database is running... .UP
The database is started.

Starting the infra services...
                  Step 8   If you entered n, enter the new password for MySQL root user and press Enter. The following information is displayed: 
                  Specify the new password for MySQL 'root' user:
Confirm the new password for MySQL 'root' user:
Password update takes few minutes. Please wait..................... done


MySQL user password is updated.
Checking if the database is running... yes.
Stopping the database...
..
The database is stopped.
Starting the database...
Checking if MySQL database is running... UP
The database is started.

Starting the infra services...
                  Note   

                  After resetting the MySQL user password, you should restart the BMA services either from the Cisco UCS Director user interface or from the BMA appliance.