Upon completion of the pod management operations such as add-storage, the operator needs to ensure that any subsequent operation such as remove-storage on the same storage node is done after accounting for all of the devices and their corresponding OSDs have been marked in the persistent crush map as shown in the output of the ceph osd crush tree.

Execute the following command on the storage node where a remove-storage pod operation is performed, to get a list of all the devices configured for ceph osds:

[root@storage-3 ~]$ df | grep -oh ceph-[0-9]*
[root@storage-3 ~]$ df | grep -oh ceph-[0-9]*
ceph-1
ceph-5
ceph-7
ceph-10

Login to any of the controller nodes and run the following commands within the ceph mon container:

$ cephmon
$ ceph osd crush tree

From the json output, locate the storage node to be removed and ensure all of the devices listed for ceph osds have corresponding osd entries for them by running the following commands:

{
                "id": -3,
                "name": "storage-3",
                "type": "host",
                "type_id": 1,
                "items": [
                    {
                        "id": 1,
                        "name": "osd.1",
                        "type": "osd",
                        "type_id": 0,
                        "crush_weight": 1.091095,
                        "depth": 2
                    },
                    {
                        "id": 5,
                        "name": "osd.5",
                        "type": "osd",
                        "type_id": 0,
                        "crush_weight": 1.091095,
                        "depth": 2
                    },
                    {
                        "id": 7,
                        "name": "osd.7",
                        "type": "osd",
                        "type_id": 0,
                        "crush_weight": 1.091095,
                        "depth": 2
                    },
                    {
                        "id": 10,
                        "name": "osd.10",
                        "type": "osd",
                        "type_id": 0,
                        "crush_weight": 1.091095,
                        "depth": 2
                    }
                ]
            },

The following topics provide Cisco NFVI general troubleshooting procedures.

Container Download Problems

1. Check installer logs log file /var/log/mercury/mercury_buildorchestration.log for any build node orchestration failures including stuck "registry-Populate local registry". In some cases, the Docker container download from your management node might be slow.

2. Check the network connectivity between the management node and the remote registry in defaults.yaml on the management node (grep "^registry:" openstack-configs/defaults.yaml).

3. Verify valid remote registry credentials are defined in setup_data.yaml file.

4. A proxy server might be needed to pull the container images from remote registry. If a proxy is required, exclude all IP addresses for your setup including management node.

PXE Boot Problems

1. Check log file /var/log/mercury/mercury_baremetal_install.log and connect to failing node CIMC KVM console to find out more on PXE boot failure reason.

2. Ensure all validations (step 1) and hardware validations (step 3) pass.

3. Check the kickstart file used in setup_data.yaml for controller, compute, and storage nodes and that matches with the hardware of corresponding nodes.

4. Check the Cobbler web interface to see all the configured systems got populated:

   https://<management_node_ip>/cobbler_web/system/list username: 
   [cobbler password]: grep COBBLER ~/openstack-configs/secrets.yaml | awk -F":" '{print $2}'

5. Check that the gateway of management/provision network is not the same as that of management interface IP address of the management node.

6. Check L2/L3 network connectivity between the failing node and the management node. Also, check the VPC configuration and port-channel status.

7. Check that the actual PXE boot order is not different from the configured boot-order.

8. Check that PXE (DHCP/TFTP) packets arrive at the management node by performing tcpdump on management interface and looking for UDP 67 or UDP 69 port.

9. Check that HTTP request reaches management node by performing tcpdump on management node management interface on TCP 80 and TCP 443 port. Also check the Docker logs for HTTP request from the failing node: docker exec -it repo_mirror tail -f /var/log/httpd/access_log.

10. Verify all nodes are running on supported Cisco NFVI CIMC firmware or above.

11. Verify supported VIC cards are installed (UCS C-series 1225/1227; UCS B-series: 1240/1280/1340/1380).

12. There are times (especially on redeployment) when the overcloud nodes (controller or compute) for some reason do not get their boot order set correctly even though CIMC says the order is correct. In such instances step 4 of runner fails as PXE boot never completes in time. The workaround is to manually enter the boot menu from the KVM console of the affected nodes and make sure the correct order (NIC interface) is chosen for the PXE boot. Note: the solution is likely to clear the BIOS CMOS before PXE booting. Will confirm if this solves the problem to avoid waiting on the KVM console at the time of PXE booting

Cisco IMC Connection Problems during Bare Metal Installation

The likely cause is Cisco IMC has too many connections, so the installer cannot connect to it. Clear the connections by logging into your Cisco IMC, going into the Admin->Sessions tab and clearing the connections.

API VIP Connection Problems

Verify the active HAProxy container is running in one of the controller nodes. On that controller within the HAProxy container namespace verify the IP address is assigned to the API interface. Also, verify that your ToR and the network infrastructure connecting your ToR is provisioned with API network segment VLAN.

HAProxy Services Downtime after Initial Installation or HA Failover

The HAProxy web interface can be accessed on TCP port 1936

http://<external_lb_vip_address>:1936/
Username: haproxy
Password: <HAPROXY_PASSWORD> from secrets.yaml file

After initial installation, the HAProxy web interface might report several OpenStack services with downtime depending upon when that OpenStack service was installed after HAProxy install. The counters are not synchronized between HAProxy active and standby. After HA proxy failover, the downtime timers might change based on the uptime of new active HAproxy container.

Management Node Problems

See the Management Node Recovery Scenarios.

Service Commands

To identify all the services that are running, enter:

$ systemctl -a | grep docker | grep service
	On controller ignore status of:
docker-neutronlb 
On compute ignore status of:
docker-neutronlb, docker-keystone

To start a service on a host, enter:

$ systemctl start <service_name>

To stop a service on a host, enter:

$ systemctl stop <service_name>

To restart a service on a host, enter:

$ systemctl restart <service_name>

To check service status on a host, enter:

$ systemctl status <service_name>

The Cisco NFVI management node hosts the Cisco VIM Rest API service, Cobbler for PXE services, ELK for Logging/Kibana dashboard services and VMTP for cloud validation. Because the maintenance node currently does not have redundancy, understanding its points of failure and recovery scenarios is important. These are described in this topic.

The management node architecture includes a Cisco UCS C240 M4 server with dual CPU socket. It has a 1 Gbps on-board (LOM) NIC and a 10 Gbps Cisco VIC mLOM. HDDs are used in 8,16, or 24 disk configurations. The following figure shows a high level maintenance node hardware and software architecture.

Figure 1. Cisco NFVI Management Node Architecture

Different management node hardware or software failures can cause Cisco NFVI service disruptions and outages. Some failed services can be recovered through manual intervention. In cases where the system is operational during a failure, double faults might not be recoverable. The following table lists different management node failure scenarios and their recovery options.

Scenario 1: Failure of one or two active HDDs

The management node has either 8,16, or 24 HDDs. The HDDs are configured with RAID 6, which helps enable data redundancy and storage performance and overcomes any unforeseen HDD failures.

• When 8 HDDs are installed, 7 are active disks and one is spare disk.

• When 16 HDDs are installed, 14 are active disks and two are spare disks.

• When 24 HDDs are installed, 20 are active disks and four are spare disks.

With RAID 6 up to two simultaneous active HDD failures can occur. When an HDD fails, the system starts automatic recovery by moving the spare disk to active state and starts recovering and rebuilding the new active HDD. It takes approximately four hours to rebuild the new disk and move to synchronized state. During this operation, the system is completely functional and no impacts are seen. However, you must monitor the system to ensure that additional failures do not occur to enter into a double fault situation.

You can use the storcli commands to check the disk and RAID state as shown below:

Note	Make sure the node is running with hardware RAID by checking the storcli output and comparing to the one preceding.

[root@mgmt-node ~]# /opt/MegaRAID/storcli/storcli64 /c0 show

<…snip…>

TOPOLOGY:
========
-----------------------------------------------------------------------------
DG Arr Row EID:Slot DID Type  State BT       Size PDC  PI SED DS3  FSpace TR 
-----------------------------------------------------------------------------
 0 -   -   -        -   RAID6 Optl  N    4.087 TB dflt N  N   dflt N      N   
 0 0   -   -        -   RAID6 Optl  N    4.087 TB dflt N  N   dflt N      N       <== RAID 6 in optimal state
 0 0   0   252:1    1   DRIVE Onln  N  837.258 GB dflt N  N   dflt -      N  
 0 0   1   252:2    2   DRIVE Onln  N  837.258 GB dflt N  N   dflt -      N  
 0 0   2   252:3    3   DRIVE Onln  N  930.390 GB dflt N  N   dflt -      N  
 0 0   3   252:4    4   DRIVE Onln  N  930.390 GB dflt N  N   dflt -      N  
 0 0   4   252:5    5   DRIVE Onln  N  930.390 GB dflt N  N   dflt -      N  
 0 0   5   252:6    6   DRIVE Onln  N  930.390 GB dflt N  N   dflt -      N  
 0 0   6   252:7    7   DRIVE Onln  N  930.390 GB dflt N  N   dflt -      N  
 0 -   -   252:8    8   DRIVE DHS   -  930.390 GB -    -  -   -    -      N  

-----------------------------------------------------------------------------

<…snip…>

PD LIST:
=======
-------------------------------------------------------------------------
EID:Slt DID State DG       Size Intf Med SED PI SeSz Model            Sp 
-------------------------------------------------------------------------
252:1     1 Onln   0 837.258 GB SAS  HDD N   N  512B ST900MM0006      U   <== all disks functioning
252:2     2 Onln   0 837.258 GB SAS  HDD N   N  512B ST900MM0006      U  
252:3     3 Onln   0 930.390 GB SAS  HDD N   N  512B ST91000640SS     U  
252:4     4 Onln   0 930.390 GB SAS  HDD N   N  512B ST91000640SS     U  
252:5     5 Onln   0 930.390 GB SAS  HDD N   N  512B ST91000640SS     U  
252:6     6 Onln   0 930.390 GB SAS  HDD N   N  512B ST91000640SS     U  
252:7     7 Onln   0 930.390 GB SAS  HDD N   N  512B ST91000640SS     U  
252:8     8 DHS    0 930.390 GB SAS  HDD N   N  512B ST91000640SS     D  
-------------------------------------------------------------------------


[root@mgmt-node ~]# /opt/MegaRAID/storcli/storcli64 /c0 show

<…snip…>

TOPOLOGY :
========
-----------------------------------------------------------------------------
DG Arr Row EID:Slot DID Type  State BT       Size PDC  PI SED DS3  FSpace TR 
-----------------------------------------------------------------------------
 0 -   -   -        -   RAID6 Pdgd  N    4.087 TB dflt N  N   dflt N      N    <== RAID 6 in degraded state
 0 0   -   -        -   RAID6 Dgrd  N    4.087 TB dflt N  N   dflt N      N        
 0 0   0   252:8    8   DRIVE Rbld  Y  930.390 GB dflt N  N   dflt -      N  
 0 0   1   252:2    2   DRIVE Onln  N  837.258 GB dflt N  N   dflt -      N  
 0 0   2   252:3    3   DRIVE Onln  N  930.390 GB dflt N  N   dflt -      N  
 0 0   3   252:4    4   DRIVE Onln  N  930.390 GB dflt N  N   dflt -      N  
 0 0   4   252:5    5   DRIVE Onln  N  930.390 GB dflt N  N   dflt -      N  
 0 0   5   252:6    6   DRIVE Onln  N  930.390 GB dflt N  N   dflt -      N  
 0 0   6   252:7    7   DRIVE Onln  N  930.390 GB dflt N  N   dflt -      N  
-----------------------------------------------------------------------------

<…snip…>

PD LIST :
=======
-------------------------------------------------------------------------
EID:Slt DID State DG       Size Intf Med SED PI SeSz Model            Sp 
-------------------------------------------------------------------------
252:1     1 UGood -  837.258 GB SAS  HDD N   N  512B ST900MM0006      U    <== active disk in slot 1 disconnected from drive group 0 
252:2     2 Onln  0  837.258 GB SAS  HDD N   N  512B ST900MM0006      U        
252:3     3 Onln  0  930.390 GB SAS  HDD N   N  512B ST91000640SS     U        
252:4     4 Onln  0  930.390 GB SAS  HDD N   N  512B ST91000640SS     U  
252:5     5 Onln  0  930.390 GB SAS  HDD N   N  512B ST91000640SS     U  
252:6     6 Onln  0  930.390 GB SAS  HDD N   N  512B ST91000640SS     U  
252:7     7 Onln  0  930.390 GB SAS  HDD N   N  512B ST91000640SS     U  
252:8     8 Rbld  0  930.390 GB SAS  HDD N   N  512B ST91000640SS     U    <== spare disk in slot 8 joined drive group 0 and in rebuilding state
-------------------------------------------------------------------------      
                                                                           
                                                                               


[root@mgmt-node ~]# /opt/MegaRAID/storcli/storcli64 /c0/e252/s8 show rebuild
Controller = 0
Status = Success
Description = Show Drive Rebuild Status Succeeded.

------------------------------------------------------
Drive-ID    Progress% Status      Estimated Time Left 
------------------------------------------------------
/c0/e252/s8        20 In progress 2 Hours 28 Minutes     <== spare disk in slot 8 rebuild status
------------------------------------------------------   

To replace the failed disk and add it back as a spare:

[root@mgmt-node ~]# /opt/MegaRAID/storcli/storcli64 /c0/e252/s1 add hotsparedrive dg=0
Controller = 0
Status = Success
Description = Add Hot Spare Succeeded.


[root@mgmt-node ~]# /opt/MegaRAID/storcli/storcli64 /c0 show

<…snip…>

TOPOLOGY :
========
-----------------------------------------------------------------------------
DG Arr Row EID:Slot DID Type  State BT       Size PDC  PI SED DS3  FSpace TR 
-----------------------------------------------------------------------------
 0 -   -   -        -   RAID6 Pdgd  N    4.087 TB dflt N  N   dflt N      N  
 0 0   -   -        -   RAID6 Dgrd  N    4.087 TB dflt N  N   dflt N      N  
 0 0   0   252:8    8   DRIVE Rbld  Y  930.390 GB dflt N  N   dflt -      N  
 0 0   1   252:2    2   DRIVE Onln  N  837.258 GB dflt N  N   dflt -      N  
 0 0   2   252:3    3   DRIVE Onln  N  930.390 GB dflt N  N   dflt -      N  
 0 0   3   252:4    4   DRIVE Onln  N  930.390 GB dflt N  N   dflt -      N  
 0 0   4   252:5    5   DRIVE Onln  N  930.390 GB dflt N  N   dflt -      N  
 0 0   5   252:6    6   DRIVE Onln  N  930.390 GB dflt N  N   dflt -      N  
 0 0   6   252:7    7   DRIVE Onln  N  930.390 GB dflt N  N   dflt -      N  
 0 -   -   252:1    1   DRIVE DHS   -  837.258 GB -    -  -   -    -      N  
-----------------------------------------------------------------------------

<…snip…>

PD LIST :
=======
-------------------------------------------------------------------------
EID:Slt DID State DG       Size Intf Med SED PI SeSz Model            Sp 
-------------------------------------------------------------------------
252:1     1 DHS   0  837.258 GB SAS  HDD N   N  512B ST900MM0006      U    <== replacement disk added back as spare
252:2     2 Onln  0  837.258 GB SAS  HDD N   N  512B ST900MM0006      U 
252:3     3 Onln  0  930.390 GB SAS  HDD N   N  512B ST91000640SS     U  
252:4     4 Onln  0  930.390 GB SAS  HDD N   N  512B ST91000640SS     U  
252:5     5 Onln  0  930.390 GB SAS  HDD N   N  512B ST91000640SS     U  
252:6     6 Onln  0  930.390 GB SAS  HDD N   N  512B ST91000640SS     U  
252:7     7 Onln  0  930.390 GB SAS  HDD N   N  512B ST91000640SS     U  
252:8     8 Rbld  0  930.390 GB SAS  HDD N   N  512B ST91000640SS     U  
-------------------------------------------------------------------------

Scenario 2: Simultaneous failure of more than two active HDDs

If more than two HDD failures occur at the same time, the management node goes into an unrecoverable failure state because RAID 6 allows for recovery of up to two simultaneous HDD failures. To recover the management node, reinstall the operating system.

Scenario 3: Spare HDD failure

When the management node has 24 HDDs, four are designated as spares. Failure of any of the disks does not impact the RAID or system functionality. Cisco recommends replacing these disks when they fail (see the steps in Scenario 1) to serve as standby disks and so when an active disk fails, an auto-rebuild is triggered.

Scenario 4: Power outage/hard reboot

If a power outage or hard system reboot occurs, the system will boot up and come back to operational state. Services running on management node during down time will be disrupted. See the steps in Scenario 9 for the list of commands to check the services status after recovery.

Scenario 5: System reboot

If a graceful system reboot occurs, the system will boot up and come back to operational state. Services running on management node during down time will be disrupted. See the steps in Scenario 9 for the list of commands to check the services status after recovery.

Scenario 6: Docker daemon start failure

The management node runs the services using Docker containers. If the Docker daemon fails to come up, it causes services such as ELK, Cobbler and VMTP to go into down state. You can use the systemctl command to check the status of the Docker daemon, for example:

# systemctl status docker
docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2016-08-22 00:33:43 CEST; 21h ago
Docs: http://docs.docker.com
Main PID: 16728 (docker)

If the Docker daemon is in down state, use the systemctl restart docker command to restart the Docker service. Run the commands listed in Scenario 9 to verify that all the Docker services are active.

Scenario 7: Service container (Cobbler, ELK) start failure

As described in Scenario 8, all the services run as Docker containers on the management node. To find all services running as containers, use the docker ps –a command. If any services are in Exit state, use the systemctl command and grep for Docker to find the exact service name, for example:

# systemctl | grep docker- | awk '{print $1}'
  docker-cobbler-tftp.service
  docker-cobbler-web.service
  docker-cobbler.service
  docker-container-registry.service
  docker-elasticsearch.service
  docker-kibana.service
  docker-logstash.service
  docker-vmtp.service

If any services need restarting, use the systemctl command. For example, to restart a Kibana service:

# systemctl restart docker-kibana.service

Scenario 8: One link failure on the bond Interface

The management node is set up with two different networks: br_api and br_mgmt. The br_api interface is the external. It is used for accessing outside services such as the Cisco VIM REST API, Kibana and Cobbler. The br_mgmt interface is internal. It is used for provisioning and to provide management connectivity to all OpenStack nodes (control, compute and storage). Each network has two ports that are bonded to provide redundancy. If one port fails, the system will remain completely functional through the other port. If a port fails, check for physical network connectivity and/or remote switch configuration to debug the underlying cause of the link failure.

Scenario 9: Two link failures on the bond Interface

As described in Scenario 10, each network is configured with two ports. If both ports are down, the system is not reachable and management node services could be disrupted. After the ports are back up, the system is fully operational. Check the physical network connectivity and/or the remote switch configuration to debug the underlying link failure cause.

Scenario 10: REST API service failure

The management node runs the REST API service for Cisco VIM clients to reach the server. If the REST service is down, Cisco VIM clients cannot reach the server to trigger any server operations. However, with the exception of the REST service, other management node services remain operational.

To verify the management node REST services are fully operational, use the following command to check that the httpd and mercury-restapi services are in active and running state:

# systemctl status httpd
httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2016-08-22 00:22:10 CEST; 22h ago

# systemctl status mercury-restapi.service
mercury-restapi.service - Mercury Restapi
   Loaded: loaded (/usr/lib/systemd/system/mercury-restapi.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2016-08-22 00:20:18 CEST; 22h ago

A tool is also provided so that you can check the REST API server status and the location of the directory it is running from. To execute run the following command:

# cd installer-<tagid>/tools
#./restapi.py -a status
   Status of the REST API Server:  active (running) since Thu 2016-08-18 09:15:39 UTC; 9h ago
   REST API launch directory: /root/installer-<tagid>/

Confirm the server status is active and check that the restapi launch directory matches the directory where the installation was launched. The restapi tool also provides the options to launch, tear down, and reset password for the restapi server as shown below:

# ./restapi.py –h

usage: restapi.py [-h] --action ACTION [--yes] [--verbose]

REST API setup helper

optional arguments:
  -h, --help            show this help message and exit
  --action ACTION, -a ACTION
                        setup - Install and Start the REST API server.
                        teardown - Stop and Uninstall the REST API
 					 server.
                        restart - Restart the REST API server. 
                        regenerate-password - Regenerate the password for
 					 REST API server.
                        reset-password - Reset the REST API password with 
					 user given password.
                        status - Check the status of the REST API server
  --yes, -y             Skip the dialog. Yes to the action.
  --verbose, -v         Perform the action in verbose mode.

If the REST API server is not running, execute ciscovim to show the following error message:

# cd installer-<tagid>/
# ciscovim -setupfile  ~/Save/<setup_data.yaml> run

If the installer directory or the REST API state is not correct or points to an incorrect REST API launch directory, go to the installer-<tagid>/tools directory and execute:

# ./restapi.py –action setup

To confirm that the REST API server state and launch directory is correct run the following command:

# ./restapi.py –action status
 

Scenario 11: Graceful reboot with Cisco VIM Insight

Cisco VIM Insight runs as a container on the management node. After a graceful reboot of the management node, the VIM Insight and its associated database containers will come up. So there is no impact on recovery.

Scenario 12: Power outage or hard reboot with VIM Insight

The Cisco VIM Insight container will come up automatically following a power outage or hard reset of the management node.

Scenario 13: Cisco VIM Insight reinstallation

If the management node which is running the Cisco VIM Insight fails and cannot come up, you must uninstall and reinstall the Cisco VIM Insight. After the VM Insight container comes up, add the relevant bootstrap steps as listed in the install guide to register the pod. VIM Insight then automatically detects the installer status and reflects the current status appropriately.

# cd /root/installer-<tagid>/insight/ 
# ./bootstrap_insight.py –a uninstall –o standalone –f </root/insight_setup_data.yaml>

Scenario 14: VIM Insight Container reboot

On Reboot of the VIM Insight container, services will continue to work as it is.

Scenario 15: Intel (I350) 1Gbps LOM failure

The management node is set up with an Intel (I350) 1 Gbps LOM for API connectivity. Two 1 Gbps ports are bonded to provide connectivity redundancy. No operational impact occurs if one of these ports goes down. However, if both ports fail, or the LOM network adapter fails, the system cannot be reached through the API IP address. If this occurs you must replace the server because the LOM is connected to the system motherboard. To recover the management node with a new server, complete the following steps. Make sure the new management node hardware profile matches the existing server and the Cisco IMC IP address is assigned.

1. Shut down the existing management node.

2. Unplug the power from the existing and new management nodes.

3. Remove all HDDs from existing management node and install them in the same slots of the new management node.

4. Plug in the power to the new management node, but do not boot the node.

5. Verify the configured boot order is set to boot from local HDD.

6. Verify the Cisco NFVI management VLAN is configured on the Cisco VIC interfaces.

7. Boot the management node for the operating system to start.

   After the management node is up, the management node bond interface will be down due to the incorrect MAC address on the ifcfg files. It will point to old node network card MAC address.

8. Update the MAC address on the ifcfg files under /etc/sysconfig/network-scripts.

9. Reboot the management node.

   It will come up and be fully operational. All interfaces should be in an up state and be reachable.

10. Verify that Kibana and Cobbler dashboards are accessible.

11. Verify the Rest API services are up. See Scenario 15 for any recovery steps.

Scenario 16: Cisco VIC 1227 10Gbps mLOM failure

The management node is configured with a Cisco VIC 1227 dual port 10 Gbps mLOM adapter for connectivity to the other Cisco NFVI nodes. Two 10 Gbps ports are bonded to provide connectivity redundancy. If one of the 10 Gbps ports goes down, no operational impact occurs. However, if both Cisco VIC 10 Gbps ports fail, the system goes into an unreachable state on the management network. If this occurs, you must replace the VIC network adapters. Otherwise pod management and the Fluentd forwarding service will be disrupted. If you replace a Cisco VIC, update the management and provisioning VLAN for the VIC interfaces using Cisco IMC and update the MAC address in the interfaces under /etc/sysconfig/network-scripts interface configuration file.

Scenario 17: DIMM memory failure

The management node is set up with multiple DIMM memory across different slots. Failure of one or memory modules could cause the system to go into unstable state, depending on how many DIMM memory failures occur. DIMM memory failures are standard system failures like any other Linux system server. If a DIMM memory fails, replace the memory module(s) as soon as possible to keep the system in stable state.

Scenario 18: One CPU failure

Cisco NFVI management nodes have dual core Intel CPUs (CPU1 and CPU2). If one CPU fails, the system remains operational. However, always replace failed CPU modules immediately. CPU failures are standard system failures like any other Linux system server. If a CPU fails, replace it immediately to keep the system in stable state.

The Cisco NFVI Compute node hosts the OpenStack services to provide processing, network, and storage resources to run instances. The node architecture includes a Cisco UCS C220 M4 server with dual CPU socket, 10 Gbps Cisco VIC mLOM, and two HDDs in RAID 1 configuration.

Failure of one active HDD

With RAID 1, data are mirrored and can allow up to one active HDD failure. When a HDD fails, the node is still functional with no impacts. However, the data are no longer mirrored and losing another HDD will result in unrecoverable and operational downtime. The failed disk should be replaced soon as it takes approximately two hours to rebuild the new disk and move to synchronized state.

To check the disk and RAID state run the storcli commands as follows:

Note	Make sure that the node is running with hardware RAID by checking the storcli output and comparing to the one below. If hardware RAID is not found, refer to Cisco NFVI Admin Guide 1.0 for HDDs replacement or contact TAC.

[root@compute-node ~]# /opt/MegaRAID/storcli/storcli64 /c0 show

<…snip…>

TOPOLOGY :
========
-----------------------------------------------------------------------------
DG Arr Row EID:Slot DID Type  State BT       Size PDC  PI SED DS3  FSpace TR 
-----------------------------------------------------------------------------
 0 -   -   -        -   RAID1 Optl  N  837.258 GB dflt N  N   dflt N      N  <== RAID 1 in optimal state
 0 0   -   -        -   RAID1 Optl  N  837.258 GB dflt N  N   dflt N      N       
 0 0   0   252:2    9   DRIVE Onln  N  837.258 GB dflt N  N   dflt -      N  
 0 0   1   252:3    11  DRIVE Onln  N  837.258 GB dflt N  N   dflt -      N  
-----------------------------------------------------------------------------

<…snip…>

Physical Drives = 2

PD LIST :
=======
-------------------------------------------------------------------------
EID:Slt DID State DG       Size Intf Med SED PI SeSz Model            Sp 
-------------------------------------------------------------------------
252:2     9 Onln   0 837.258 GB SAS  HDD N   N  512B ST900MM0006      U   <== all disks functioning
252:3    11 Onln   0 837.258 GB SAS  HDD N   N  512B ST900MM0006      U  
-------------------------------------------------------------------------


[root@compute-node ~]# /opt/MegaRAID/storcli/storcli64 /c0 show

<…snip…>

TOPOLOGY :
========
-----------------------------------------------------------------------------
DG Arr Row EID:Slot DID Type  State BT       Size PDC  PI SED DS3  FSpace TR 
-----------------------------------------------------------------------------
 0 -   -   -        -   RAID1 Dgrd  N  837.258 GB dflt N  N   dflt N      N  <== RAID 1 in degraded state.
 0 0   -   -        -   RAID1 Dgrd  N  837.258 GB dflt N  N   dflt N      N        
 0 0   0   -        -   DRIVE Msng  -  837.258 GB -    -  -   -    -      N  
 0 0   1   252:3    11  DRIVE Onln  N  837.258 GB dflt N  N   dflt -      N  
-----------------------------------------------------------------------------

<…snip…>

PD LIST :
=======
-------------------------------------------------------------------------
EID:Slt DID State DG       Size Intf Med SED PI SeSz Model            Sp 
-------------------------------------------------------------------------
252:2     9 UGood -  837.258 GB SAS  HDD N   N  512B ST900MM0006      U   <== active disk in slot 2 disconnected from drive group 0
252:3    11 Onln  0  837.258 GB SAS  HDD N   N  512B ST900MM0006      U       
-------------------------------------------------------------------------     

To replace the failed disk and add it back as a spare run the following command:

[root@compute-node ~]# /opt/MegaRAID/storcli/storcli64 /c0/e252/s2 add hotsparedrive dg=0
Controller = 0
Status = Success
Description = Add Hot Spare Succeeded.


[root@compute-node ~]# /opt/MegaRAID/storcli/storcli64 /c0 show

<…snip…>

TOPOLOGY :
========
-----------------------------------------------------------------------------
DG Arr Row EID:Slot DID Type  State BT       Size PDC  PI SED DS3  FSpace TR 
-----------------------------------------------------------------------------
 0 -   -   -        -   RAID1 Dgrd  N  837.258 GB dflt N  N   dflt N      N  
 0 0   -   -        -   RAID1 Dgrd  N  837.258 GB dflt N  N   dflt N      N  
 0 0   0   252:2    9   DRIVE Rbld  Y  837.258 GB dflt N  N   dflt -      N  
 0 0   1   252:3    11  DRIVE Onln  N  837.258 GB dflt N  N   dflt -      N  
-----------------------------------------------------------------------------

<…snip…>

PD LIST :
=======
-------------------------------------------------------------------------
EID:Slt DID State DG       Size Intf Med SED PI SeSz Model            Sp 
-------------------------------------------------------------------------
252:2     9 Rbld   0 837.258 GB SAS  HDD N   N  512B ST900MM0006      U   <== replacement disk in slot 2 joined device group 0 and in rebuilding state
252:3    11 Onln   0 837.258 GB SAS  HDD N   N  512B ST900MM0006      U     
-------------------------------------------------------------------------     
                                                                             


[root@compute-node ~]# /opt/MegaRAID/storcli/storcli64 /c0/e252/s2 show rebuild
Controller = 0
Status = Success
Description = Show Drive Rebuild Status Succeeded.


------------------------------------------------------
Drive-ID    Progress% Status      Estimated Time Left 
------------------------------------------------------
/c0/e252/s2        10 In progress 1 Hours 9 Minutes    <== replacement disk in slot 2 rebuild status
------------------------------------------------------               

Cisco VIM includes a tech-support tool that you can use to gather Cisco VIM information to help solve issues working with Cisco Technical Support. The tech-support tool can be extended to execute custom scripts. It can be called after runner is executed at least once. The tech-support tool uses a configuration file that specifies what information to collect. The configuration file is located in the following location: /root/openstack-configs/tech-support/tech_support_cfg.yaml.

The tech-support tool keeps track of the point where the Cisco VIM installer has executed and collects the output of files or commands indicated by the configuration file. For example, if the installer fails at Step 3 (VALIDATION), the tech-support will provide information listed in the configuration file up to Step 3 (included). You can override this default behavior by adding the --stage option to the command.

The tech-support script is located at the management node /root/installer-{tag-id}/tech-support directory. To run it after the runner execution, enter the following command:

./tech-support/tech_support.py

The command creates a compressed tar file containing all the information it gathered. The file location is displayed in the console at the end of the execution. You do not need to execute the command with any options. However, if you want to override any default behavior, you can use the following options:

/tech_support.py --help
Usage: tech_support.py [options]
 
tech_support.py collects information about your cloud
Options:
  -h, --help            show this help message and exit
  --stage=STAGE         specify the stage where installer left off
  --config-file=CFG_FILE specify alternate configuration file
  --tmp-dir=TMP_DIR     specify alternate temporary directory
  --file-size=TAIL_SIZE specify max size (in KB) of each file collected

Where:

• stage—tells the tech-support at which state the installer left off. The possible values are: INPUT_VALIDATION, BUILDNODE_ORCHESTRATION, VALIDATION, BAREMETAL_INSTALL, COMMON_SETUP, CEPH, ORCHESTRATION or VMTP

• config-file—Provides the path for a specific configuration file. Make sure that your syntax is correct. Look at the default /root/tech-support/openstack-configs/tech_support_cfg.yaml file as an example on how to create a new config-file or modify the default file.

• tmp-dir—Provides the path to a temp directory tech-support can use to create the compressed tar file. The tech-support tool provides the infrastructure to execute standard Linux commands from packages included in the Cisco VIM installation. This infrastructure is extensible and you can add commands, files, or custom bash/Python scripts into the appropriate configuration file sections for the tool to collect the output of those commands/scripts. (See the README section at the beginning of the file for more details on how to do this.)

• file-size—Is an integer that specifies (in KB) the maximum file size that tech-support will capture and tail the file if needed. By default, this value is set to 10 MB. For example, if no file-size option is provided and the tech-support needs to collect /var/log/mercury/data.log and the data.log is more than 10 MB, tech-support will get the last 10 MB from /var/log/mercury/data.log.

Cisco VIM tech-support is a utility tool designed to collect the VIM pod logs which help users to debug the issues offline. The administrator uses the tech-support configuration files to provide the list of commands or configuration files. The tech support tool of the Cisco VIM gathers list of commands or configuration files for the offline diagnostic or debugging purposes.

By default the tech-support configuration file is located at the /root/openstack-configs/tech-support/tech_support_cfg.yaml file. Alternatively, you can use a different one by specifying the -config-file option. The syntax of this configuration file must be as follows:

The tech-support configuration file section is divided into eight sections which corresponds to each of the installer stages:

• INPUT_VALIDATION

• BUILDNODE_ORCHESTRATION

• VALIDATION

• BAREMETAL_INSTALL

• COMMON_SETUP

• CEPH

• ORCHESTRATION

• VMTP

Inside each of these eight sections, there are tags divided on hierarchical levels. At the first level, the tag indicates the host(s) or path on which the command(s) run and from where the file(s) can be collected. The possible tags are as follows:

• - HOSTS_MANAGEMENT: Run in Management node only

• - HOSTS_CONTROL: Run in all the Control nodes

• - HOSTS_COMPUTE: Run in all the Compute nodes

• - HOSTS_STORAGE: Run in all the Storage nodes

• - HOSTS_COMMON: Run in all the Compute and Control nodes

• - HOSTS_ALL: Run in all the Compute, Control and Storage nodes

Note	In any of these eight sections, if HOSTS tag is not specified then no information is collected for that stage.

For each of the hosts mentioned above there will be a second level tag which specifies where to run the command. The possible values of those tags are as follows:

• - SERVER_FILES: Path(s) to the file(s) that tech-support needs to collect.

• - SERVER_COMMANDS: Command(s) or script name(s) which need to be executed directly on the desired server. The command(s) need to be already included in the $PATH. For the scripts, please refer to the Custom Scripts paragraph below.

• - CONTAINERS: Indicates the tech-support tool that the command(s) need to be executed and the files to be gathered from inside a container. See the following steps for more specific information of what can be added in this section.

The tech-support runs the linux commands on the server (from packages included in RHEL7.3). Add the name of the commands under the SERVER_COMMANDS section of the configuration file to run the commands.

However, if the administrator wants to add a custom bash or python script to be executed in some set of servers in the cloud. In this case the user just needs to add the script into the custom-scripts directory on the current directory path (/root/openstack-configs/tech-support/) and add the script name into the corresponding SERVER_COMMANDS section.

The tech-support tool will scp the script(s) included in the custom-scripts directory into the appropriate cloud nodes where it will be executed (as# indicated in this config file) and capture the output (stdout and stderr) and add it to the collection of files collected by the tech-support tool. It is assumed that the scripts are self-standing and independent and needs no external input.

Following is an example of a custom tech-support configuration file. This is just an example of what information the tech-support tool will gather if given the following configuration file:

COMMON_SETUP:
  HOSTS_ALL:   # All compute, control and storage hosts
    SERVER_FILES:
      - /usr/lib/docker-storage-setup
    SERVER_COMMANDS:
      - docker info
      - my_script.sh
    CONTAINERS:
      all_containers:  #execute in all containers (even if they are in down state)
        CONTAINER_COMMANDS:
          - docker inspect
          - docker logs
      logstash:
        CONTAINER_FILES:
          - /var/log/
        CONTAINER_COMMANDS:
          - ls -l

Given this example of configuration, and assuming that the installer ended in at least the COMMON_SETUP state, the tech-support tool will run under all OpenStack nodes (Compute, Control and Storage) and it will:

• Gather (if exists) the contents of /usr/lib/docker-storage-setup file.

• Run docker info command and collect the output.

• Run my_script.sh and collect the output. The my_script.sh is an example of a bash script which the user previously added to the /root/openstack-configs/tech-support/custom-scripts directory.

• Collect the output of docker inspect and docker logs for all containers.

• Collect the files in /var/log inside the logstash container (if there is container with that name). This is equivalent to running the following command (where /tmp indicates a temporary location where the tech-support tool gathers all the information): docker cp logstash_{tag}:/var/log/ /tmp.

• Collect the output of the command docker exec logstash_{{tag}}: ls -l.

.