UltraM: VM Recovery after ESC Retry Expires

Available Languages

Download Options

PDF (26.4 KB)
View with Adobe Reader on a variety of devices
ePub (75.1 KB)
View in various apps on iPhone, iPad, Android, Sony Reader, or Windows Phone
Mobi (Kindle) (75.1 KB)
View on Kindle device or Kindle app on multiple devices

Updated:November 2, 2017

Document ID:212410

Bias-Free Language

The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.

Introduction

Problem

Solution

Introduction

This documents describes how to recover a Virtual Machine (VM) of Virtualized Packet Core (VPC) in Cisco Ultra Services Platform (UltraM) after the VM has been inaccessible for some time and Cisco Elastic Services Controller (ESC) tries to recover it.

Problem

Compute node is removed (or inaccessible) in UltraM setup. When ESC tries to recover the node, but it is unable to do so because the node was inaccessible. This scenario can be simulated when you leave power cables from a compute node. One way to simulate this scenario is to remove power cables from Unified Computing System (UCS) blade. After ESC fails to recover the VM, it goes into the "ERROR" state under OpenStack and stays “booting”.

In the example, the SF card 5 is on vnfd1-deployment_s6_0_e03f87f5-63b6-4053-8d0f-0c9df963162c:

[local]rcdn-ulram-lab# show card table

Slot         Card Type                               Oper State     SPOF  Attach

-----------  --------------------------------------  -------------  ----  ------

 1: CFC      Control Function Virtual Card           Standby        -          

 2: CFC      Control Function Virtual Card           Active         No         

 3: FC       1-Port Service Function Virtual Card    Active         No         

 4: FC       1-Port Service Function Virtual Card    Active         No         

 5: FC       1-Port Service Function Virtual Card    Booting        -          

 6: FC       1-Port Service Function Virtual Card    Active         No         

 7: FC       1-Port Service Function Virtual Card    Active         No         





[stack@ultram-ospd ~]$ nova list

+--------------------------------------+---------------------------------------------------------------+--------+------------+-------------+------------------------------------------------------------------------------------------------------------+

| ID                                   | Name                                                          | Status | Task State | Power State | Networks                                                                                                   |

+--------------------------------------+---------------------------------------------------------------+--------+------------+-------------+------------------------------------------------------------------------------------------------------------+

| beab0296-8cfa-4b63-8a05-a800637199f5 | Testcompanion                                                 | ACTIVE | -          | Running     | testcomp-gn=10.10.11.8; mgmt=172.16.181.18, 10.201.206.46; testcomp-sig=10.10.13.5; testcomp-gi=10.10.12.7 |

| 235f5591-9502-4ba3-a003-b254494d258b | auto-deploy-ISO-590-uas-0                                     | ACTIVE | -          | Running     | mgmt=172.16.181.11, 10.201.206.44                                                                          |

| 9450cb19-f073-476b-a750-9336b26e3c6a | auto-it-vnf-ISO-590-uas-0                                     | ACTIVE | -          | Running     | mgmt=172.16.181.8, 10.201.206.43                                                                           |

| d0d91636-951d-49db-a92b-b2a639f5db9d | autovnf1-uas-0                                                | ACTIVE | -          | Running     | orchestr=172.16.180.14; mgmt=172.16.181.13                                                                 |

| 901f30e2-e96e-4658-9e1e-39a45b5859c7 | autovnf1-uas-1                                                | ACTIVE | -          | Running     | orchestr=172.16.180.5; mgmt=172.16.181.12                                                                  |

| 9edb3a8d-a69b-4912-86f6-9d0b05d6210d | autovnf1-uas-2                                                | ACTIVE | -          | Running     | orchestr=172.16.180.16; mgmt=172.16.181.5                                                                  |

| 56ce362c-3494-4106-98e3-ba06e56ee4ed | ultram-vnfm1-ESC-0                                            | ACTIVE | -          | Running     | orchestr=172.16.180.9; mgmt=172.16.181.6, 10.201.206.55                                                    |

| bb687399-e1f9-44b2-a258-cfa29dcf178e | ultram-vnfm1-ESC-1                                            | ACTIVE | -          | Running     | orchestr=172.16.180.15; mgmt=172.16.181.7                                                                  |

| bfc4096c-4ff7-4b30-af3f-5bc3810b30e3 | ultram-vnfm1-em_ultram_0_9b5ccf05-c340-44da-9bca-f5af4689ea42 | ACTIVE | -          | Running     | orchestr=172.16.180.7; mgmt=172.16.181.14                                                                  |

| cf7ddc9e-5e6d-4e38-a606-9dc9d31c559d | ultram-vnfm1-em_ultram_0_c2533edd-8756-44fb-a8bf-98b9c10bfacd | ACTIVE | -          | Running     | orchestr=172.16.180.8; mgmt=172.16.181.15                                                                  |

| 592b5b3f-0b0b-4bc6-81e7-a8cc9a609594 | ultram-vnfm1-em_ultram_0_ce0c37a0-509e-45d1-9d00-464988e02730 | ACTIVE | -          | Running     | orchestr=172.16.180.6; mgmt=172.16.181.10                                                                  |

| 143baf4f-024a-47f1-969a-d4d79d89be14 | vnfd1-deployment_c1_0_84c5bc9e-9d80-4628-b88a-f8a0011b5d4b    | ACTIVE | -          | Running     | orchestr=172.16.180.26; ultram-vnfm1-di-internal1=192.168.1.13; mgmt=172.16.181.25                         |

| b74a0365-3be1-4bee-b1cc-e454d5b0cd11 | vnfd1-deployment_c2_0_66bac767-39fe-4972-b877-7826468a762e    | ACTIVE | -          | Running     | orchestr=172.16.180.10; ultram-vnfm1-di-internal1=192.168.1.5; mgmt=172.16.181.20, 10.201.206.45           |

| 59a02ec2-bed6-4ad8-81ff-e8a922742f7b | vnfd1-deployment_s3_0_f9f6b7a6-1458-4b22-b40f-33f8af3500b8    | ACTIVE | -          | Running     | ultram-vnfm1-service-network1=10.10.10.4; orchestr=172.16.180.17; ultram-vnfm1-di-internal1=192.168.1.6    |

| 52e9a2b0-cf2c-478d-baea-f4a5f3b7f327 | vnfd1-deployment_s4_0_8c78cfd9-57c5-4394-992a-c86393187dd0    | ACTIVE | -          | Running     | ultram-vnfm1-service-network1=10.10.10.11; orchestr=172.16.180.20; ultram-vnfm1-di-internal1=192.168.1.3   |

| bd7c6600-3e8f-4c09-a35c-89921bbf1b35 | vnfd1-deployment_s5_0_f1c48ea1-4a91-4098-86f6-48e172e23c83    | ACTIVE | -          | Running     | ultram-vnfm1-service-network1=10.10.10.12; orchestr=172.16.180.13; ultram-vnfm1-di-internal1=192.168.1.2   |

| 085baf6a-02bf-4190-ac38-bbb33350b941 | vnfd1-deployment_s6_0_e03f87f5-63b6-4053-8d0f-0c9df963162c    | ERROR  | -          | NOSTATE     |                                                                                                            |

| ea03767f-5dd9-43ed-8e9d-603590da2580 | vnfd1-deployment_s7_0_e887d8b1-7c98-4f60-b343-b0be7b387b32    | ACTIVE | -          | Running     | ultram-vnfm1-service-network1=10.10.10.10; orchestr=172.16.180.18; ultram-vnfm1-di-internal1=192.168.1.9   |

+--------------------------------------+---------------------------------------------------------------+--------+------------+-------------+------------------------------------------------------------------------------------------------------------+

After ESC tries to recover the failed VM, it marks the VM as fail instances in OpenStack and doesn't retry to recover from now on.

Here are logs from ESC where VM recovery fails:

15:11:04,617 11-Aug-2017 WARN  ===== SEND NOTIFICATION STARTS =====

15:11:04,617 11-Aug-2017 WARN  Type: VM_RECOVERY_INIT

15:11:04,617 11-Aug-2017 WARN  Status: SUCCESS

15:11:04,617 11-Aug-2017 WARN  Status Code: 200

15:11:04,617 11-Aug-2017 WARN  Status Msg: Recovery event for VM [vnfd1-deployment_s6_0_e03f87f5-63b6-4053-8d0f-0c9df963162c] triggered.

15:11:04,617 11-Aug-2017 WARN  Tenant: core

15:11:04,617 11-Aug-2017 WARN  Service ID: NULL

15:11:04,617 11-Aug-2017 WARN  Deployment ID: b41ad0ec-bc74-4bb3-85b6-7ef430074187

15:11:04,617 11-Aug-2017 WARN  Deployment name: vnfd1-deployment-1.0.0-1

15:11:04,617 11-Aug-2017 WARN  VM group name: s6

15:11:04,618 11-Aug-2017 WARN  VM Source:

15:11:04,618 11-Aug-2017 WARN      VM ID: 4d6b1b6f-6137-4e8e-b61c-66d5fb59ba0d

15:11:04,618 11-Aug-2017 WARN      Host ID: 20b7df6d083651eb04f1f014e8a4958ddf9c1654cb3ad9057adc7e73

15:11:04,618 11-Aug-2017 WARN      Host Name: ultram-rcdnlab-compute-4.localdomain

15:11:04,618 11-Aug-2017 WARN      [DEBUG-ONLY] VM IP: 10.10.10.9; 172.16.180.22; 192.168.1.12;

15:11:04,618 11-Aug-2017 WARN  =====  SEND NOTIFICATION ENDS  =====

15:16:38,019 11-Aug-2017 WARN 

15:16:38,020 11-Aug-2017 WARN  ===== SEND NOTIFICATION STARTS =====

15:16:38,020 11-Aug-2017 WARN  Type: VM_RECOVERY_REBOOT

15:16:38,020 11-Aug-2017 WARN  Status: FAILURE

15:16:38,020 11-Aug-2017 WARN  Status Code: 500

15:16:38,020 11-Aug-2017 WARN  Status Msg: VM [vnfd1-deployment_s6_0_e03f87f5-63b6-4053-8d0f-0c9df963162c] failed to be rebooted.

15:16:38,020 11-Aug-2017 WARN  Tenant: core

15:16:38,020 11-Aug-2017 WARN  Service ID: NULL

15:16:38,020 11-Aug-2017 WARN  Deployment ID: b41ad0ec-bc74-4bb3-85b6-7ef430074187

15:16:38,020 11-Aug-2017 WARN  Deployment name: vnfd1-deployment-1.0.0-1

15:16:38,020 11-Aug-2017 WARN  VM group name: s6

15:16:38,021 11-Aug-2017 WARN  VM Source:

15:16:38,021 11-Aug-2017 WARN      VM ID: 4d6b1b6f-6137-4e8e-b61c-66d5fb59ba0d

15:16:38,021 11-Aug-2017 WARN      Host ID: 20b7df6d083651eb04f1f014e8a4958ddf9c1654cb3ad9057adc7e73

15:16:38,021 11-Aug-2017 WARN      Host Name: ultram-rcdnlab-compute-4.localdomain

15:16:38,021 11-Aug-2017 WARN      [DEBUG-ONLY] VM IP: 10.10.10.9; 172.16.180.22; 192.168.1.12;

15:16:38,021 11-Aug-2017 WARN  =====  SEND NOTIFICATION ENDS  =====

15:16:48,286 11-Aug-2017 WARN 

15:16:48,286 11-Aug-2017 WARN  ===== SEND NOTIFICATION STARTS =====

15:16:48,286 11-Aug-2017 WARN  Type: VM_RECOVERY_UNDEPLOYED

15:16:48,286 11-Aug-2017 WARN  Status: SUCCESS

15:16:48,286 11-Aug-2017 WARN  Status Code: 204

15:16:48,286 11-Aug-2017 WARN  Status Msg: VM [vnfd1-deployment_s6_0_e03f87f5-63b6-4053-8d0f-0c9df963162c] has been undeployed.

15:16:48,286 11-Aug-2017 WARN  Tenant: core

15:16:48,286 11-Aug-2017 WARN  Service ID: NULL

15:16:48,286 11-Aug-2017 WARN  Deployment ID: b41ad0ec-bc74-4bb3-85b6-7ef430074187

15:16:48,286 11-Aug-2017 WARN  Deployment name: vnfd1-deployment-1.0.0-1

15:16:48,286 11-Aug-2017 WARN  VM group name: s6

15:16:48,286 11-Aug-2017 WARN  VM Source:

15:16:48,286 11-Aug-2017 WARN      VM ID: 4d6b1b6f-6137-4e8e-b61c-66d5fb59ba0d

15:16:48,286 11-Aug-2017 WARN      Host ID: 20b7df6d083651eb04f1f014e8a4958ddf9c1654cb3ad9057adc7e73

15:16:48,286 11-Aug-2017 WARN      Host Name: ultram-rcdnlab-compute-4.localdomain

15:16:48,287 11-Aug-2017 WARN      [DEBUG-ONLY] VM IP: 10.10.10.9; 172.16.180.22; 192.168.1.12;

15:16:48,287 11-Aug-2017 WARN  =====  SEND NOTIFICATION ENDS  =====

15:18:04,418 11-Aug-2017 WARN 

15:18:04,418 11-Aug-2017 WARN  ===== SEND NOTIFICATION STARTS =====

15:18:04,418 11-Aug-2017 WARN  Type: VM_RECOVERY_COMPLETE

15:18:04,418 11-Aug-2017 WARN  Status: FAILURE

15:18:04,418 11-Aug-2017 WARN  Status Code: 500

15:18:04,418 11-Aug-2017 WARN  Status Msg: Error deploying VM [vnfd1-deployment_s6_0_e03f87f5-63b6-4053-8d0f-0c9df963162c] as part of recovery workflow. VIM Driver: VM booted in ERROR state in Openstack: No valid host was found. There are not enough hosts available.

15:18:04,418 11-Aug-2017 WARN  Tenant: core

15:18:04,418 11-Aug-2017 WARN  Service ID: NULL

15:18:04,418 11-Aug-2017 WARN  Deployment ID: b41ad0ec-bc74-4bb3-85b6-7ef430074187

15:18:04,418 11-Aug-2017 WARN  Deployment name: vnfd1-deployment-1.0.0-1

15:18:04,418 11-Aug-2017 WARN  VM group name: s6

15:18:04,418 11-Aug-2017 WARN  VM Source:

15:18:04,418 11-Aug-2017 WARN      VM ID: 4d6b1b6f-6137-4e8e-b61c-66d5fb59ba0d

15:18:04,418 11-Aug-2017 WARN      Host ID: 20b7df6d083651eb04f1f014e8a4958ddf9c1654cb3ad9057adc7e73

15:18:04,418 11-Aug-2017 WARN      Host Name: ultram-rcdnlab-compute-4.localdomain

15:18:04,418 11-Aug-2017 WARN      [DEBUG-ONLY] VM IP: 10.10.10.9; 172.16.180.22; 192.168.1.12;

15:18:04,418 11-Aug-2017 WARN  =====  SEND NOTIFICATION ENDS  =====

Solution

1. Power up the computer with power on button and wait till hypervisor comes up:

[root@ultram-ospd ~]# su - stack

[stack@ultram-ospd ~]$ source stackrc 

[stack@ultram-ospd ~]$ nova hypervisor-list

+----+---------------------------------------+-------+---------+

| ID | Hypervisor hostname                   | State | Status  |

+----+---------------------------------------+-------+---------+

| 3  | ultram-rcdnlab-compute-10.localdomain | up    | enabled |

| 6  | ultram-rcdnlab-compute-5.localdomain  | up    | enabled |

| 9  | ultram-rcdnlab-compute-6.localdomain  | up    | enabled |

| 12 | ultram-rcdnlab-compute-3.localdomain  | up    | enabled |

| 15 | ultram-rcdnlab-compute-9.localdomain  | up    | enabled |

| 18 | ultram-rcdnlab-compute-1.localdomain  | up    | enabled |

| 21 | ultram-rcdnlab-compute-8.localdomain  | up    | enabled |

| 24 | ultram-rcdnlab-compute-4.localdomain  | down  | enabled |

| 27 | ultram-rcdnlab-compute-7.localdomain  | up    | enabled |

| 30 | ultram-rcdnlab-compute-2.localdomain  | up    | enabled |

| 33 | ultram-rcdnlab-compute-0.localdomain  | up    | enabled |

+----+---------------------------------------+-------+---------+





[stack@ultram-ospd ~]$ nova hypervisor-list

+----+---------------------------------------+-------+---------+

| ID | Hypervisor hostname                   | State | Status  |

+----+---------------------------------------+-------+---------+

| 3  | ultram-rcdnlab-compute-10.localdomain | up    | enabled |

| 6  | ultram-rcdnlab-compute-5.localdomain  | up    | enabled |

| 9  | ultram-rcdnlab-compute-6.localdomain  | up    | enabled |

| 12 | ultram-rcdnlab-compute-3.localdomain  | up    | enabled |

| 15 | ultram-rcdnlab-compute-9.localdomain  | up    | enabled |

| 18 | ultram-rcdnlab-compute-1.localdomain  | up    | enabled |

| 21 | ultram-rcdnlab-compute-8.localdomain  | up    | enabled |

| 24 | ultram-rcdnlab-compute-4.localdomain  | up    | enabled |

| 27 | ultram-rcdnlab-compute-7.localdomain  | up    | enabled |

| 30 | ultram-rcdnlab-compute-2.localdomain  | up    | enabled |

| 33 | ultram-rcdnlab-compute-0.localdomain  | up    | enabled |

+----+---------------------------------------+-------+---------+

2. Identify Instance ID in Nova List:

[root@ultram-ospd ~]# su - stack
[stack@ultram-ospd ~]$ source corerc 

[stack@ultram-ospd ~]$ nova list | grep ERROR

| 085baf6a-02bf-4190-ac38-bbb33350b941 | vnfd1-deployment_s6_0_e03f87f5-63b6-4053-8d0f-0c9df963162c    | ERROR  | -          | NOSTATE     |

3. Initiate manual recovery on ESC with the CLI and instance ID from previous step:

[admin@ultram-vnfm1-esc-0 ~]$ cd /opt/cisco/esc/esc-confd/esc-cli

[admin@ultram-vnfm1-esc-0 esc-cli]$ ./esc_nc_cli recovery-vm-action DO vnfd1-deployment_s6_0_e03f87f5-63b6-4053-8d0f-0c9df963162c

Recovery VM Action

/opt/cisco/esc/confd/bin/netconf-console --port=830 --host=127.0.0.1 --user=admin --privKeyFile=/home/admin/.ssh/confd_id_dsa --privKeyType=dsa --rpc=/tmp/esc_nc_cli.hZsdLQ2Mle

<?xml version="1.0" encoding="UTF-8"?>

<rpc-reply xmlns="urn:ietf:params:xml:ns:netconf:base:1.0" message-id="1">

  <ok/>

</rpc-reply>

4. Check logs, OpenStack Horizon to confirm that the instance has been recovered:

[admin@ultram-vnfm1-esc-0 ~]$ tail -f /var/log/esc/yangesc.log 



16:41:54,445 11-Aug-2017 INFO  ===== RECOVERY VM ACTION REQUEST RECEIVED =====
16:41:54,445 11-Aug-2017 INFO  Type: DO
16:41:54,445 11-Aug-2017 INFO  Recovery VM name: vnfd1-deployment_s6_0_e03f87f5-63b6-4053-8d0f-0c9df963162c
16:41:58,092 11-Aug-2017 INFO  =====  RECOVERY VM ACTION REQUEST ACCEPTED  =====
16:41:58,673 11-Aug-2017 WARN  
16:41:58,673 11-Aug-2017 WARN  ===== SEND NOTIFICATION STARTS =====
16:41:58,674 11-Aug-2017 WARN  Type: VM_RECOVERY_INIT
16:41:58,674 11-Aug-2017 WARN  Status: SUCCESS
16:41:58,674 11-Aug-2017 WARN  Status Code: 200
16:41:58,674 11-Aug-2017 WARN  Status Msg: Recovery event for VM [vnfd1-deployment_s6_0_e03f87f5-63b6-4053-8d0f-0c9df963162c] triggered.
16:41:58,674 11-Aug-2017 WARN  Tenant: core
16:41:58,674 11-Aug-2017 WARN  Service ID: NULL
16:41:58,674 11-Aug-2017 WARN  Deployment ID: b41ad0ec-bc74-4bb3-85b6-7ef430074187
16:41:58,674 11-Aug-2017 WARN  Deployment name: vnfd1-deployment-1.0.0-1
16:41:58,674 11-Aug-2017 WARN  VM group name: s6
16:41:58,674 11-Aug-2017 WARN  VM Source: 
16:41:58,674 11-Aug-2017 WARN      VM ID: 085baf6a-02bf-4190-ac38-bbb33350b941
16:41:58,674 11-Aug-2017 WARN      Host ID: 
16:41:58,674 11-Aug-2017 WARN      Host Name: 
16:41:58,674 11-Aug-2017 WARN      [DEBUG-ONLY] VM IP: 10.10.10.9; 172.16.180.22; 192.168.1.12; 
16:41:58,674 11-Aug-2017 WARN  =====  SEND NOTIFICATION ENDS  =====
16:42:19,794 11-Aug-2017 WARN  
16:42:19,794 11-Aug-2017 WARN  ===== SEND NOTIFICATION STARTS =====
16:42:19,794 11-Aug-2017 WARN  Type: VM_RECOVERY_REBOOT
16:42:19,794 11-Aug-2017 WARN  Status: FAILURE
16:42:19,794 11-Aug-2017 WARN  Status Code: 500
16:42:19,794 11-Aug-2017 WARN  Status Msg: VM [vnfd1-deployment_s6_0_e03f87f5-63b6-4053-8d0f-0c9df963162c] failed to be rebooted.
16:42:19,794 11-Aug-2017 WARN  Tenant: core
16:42:19,795 11-Aug-2017 WARN  Service ID: NULL
16:42:19,795 11-Aug-2017 WARN  Deployment ID: b41ad0ec-bc74-4bb3-85b6-7ef430074187
16:42:19,795 11-Aug-2017 WARN  Deployment name: vnfd1-deployment-1.0.0-1
16:42:19,795 11-Aug-2017 WARN  VM group name: s6
16:42:19,795 11-Aug-2017 WARN  VM Source: 
16:42:19,795 11-Aug-2017 WARN      VM ID: 085baf6a-02bf-4190-ac38-bbb33350b941
16:42:19,795 11-Aug-2017 WARN      Host ID: 
16:42:19,795 11-Aug-2017 WARN      Host Name: 
16:42:19,795 11-Aug-2017 WARN      [DEBUG-ONLY] VM IP: 10.10.10.9; 172.16.180.22; 192.168.1.12; 
16:42:19,795 11-Aug-2017 WARN  =====  SEND NOTIFICATION ENDS  =====
16:42:32,013 11-Aug-2017 WARN  
16:42:32,013 11-Aug-2017 WARN  ===== SEND NOTIFICATION STARTS =====
16:42:32,013 11-Aug-2017 WARN  Type: VM_RECOVERY_UNDEPLOYED
16:42:32,013 11-Aug-2017 WARN  Status: SUCCESS
16:42:32,013 11-Aug-2017 WARN  Status Code: 204
16:42:32,013 11-Aug-2017 WARN  Status Msg: VM [vnfd1-deployment_s6_0_e03f87f5-63b6-4053-8d0f-0c9df963162c] has been undeployed.
16:42:32,013 11-Aug-2017 WARN  Tenant: core
16:42:32,014 11-Aug-2017 WARN  Service ID: NULL
16:42:32,014 11-Aug-2017 WARN  Deployment ID: b41ad0ec-bc74-4bb3-85b6-7ef430074187
16:42:32,014 11-Aug-2017 WARN  Deployment name: vnfd1-deployment-1.0.0-1
16:42:32,014 11-Aug-2017 WARN  VM group name: s6
16:42:32,014 11-Aug-2017 WARN  VM Source: 
16:42:32,014 11-Aug-2017 WARN      VM ID: 085baf6a-02bf-4190-ac38-bbb33350b941
16:42:32,014 11-Aug-2017 WARN      Host ID: 
16:42:32,014 11-Aug-2017 WARN      Host Name: 
16:42:32,014 11-Aug-2017 WARN      [DEBUG-ONLY] VM IP: 10.10.10.9; 172.16.180.22; 192.168.1.12; 
16:42:32,014 11-Aug-2017 WARN  =====  SEND NOTIFICATION ENDS  =====
16:43:13,643 11-Aug-2017 WARN  
16:43:13,643 11-Aug-2017 WARN  ===== SEND NOTIFICATION STARTS =====
16:43:13,643 11-Aug-2017 WARN  Type: VM_RECOVERY_DEPLOYED
16:43:13,643 11-Aug-2017 WARN  Status: SUCCESS
16:43:13,643 11-Aug-2017 WARN  Status Code: 200
16:43:13,643 11-Aug-2017 WARN  Status Msg: VM [vnfd1-deployment_s6_0_e03f87f5-63b6-4053-8d0f-0c9df963162c] has been deployed as part of recovery.
16:43:13,643 11-Aug-2017 WARN  Tenant: core
16:43:13,643 11-Aug-2017 WARN  Service ID: NULL
16:43:13,643 11-Aug-2017 WARN  Deployment ID: b41ad0ec-bc74-4bb3-85b6-7ef430074187
16:43:13,643 11-Aug-2017 WARN  Deployment name: vnfd1-deployment-1.0.0-1
16:43:13,643 11-Aug-2017 WARN  VM group name: s6
16:43:13,643 11-Aug-2017 WARN  VM Source: 
16:43:13,643 11-Aug-2017 WARN      VM ID: 085baf6a-02bf-4190-ac38-bbb33350b941
16:43:13,643 11-Aug-2017 WARN      Host ID: 
16:43:13,643 11-Aug-2017 WARN      Host Name: 
16:43:13,643 11-Aug-2017 WARN      [DEBUG-ONLY] VM IP: 10.10.10.9; 172.16.180.22; 192.168.1.12; 
16:43:13,643 11-Aug-2017 WARN  VM Target: 
16:43:13,644 11-Aug-2017 WARN      VM ID: a313e8dc-3b0f-4b41-8648-f9b9419bc826
16:43:13,644 11-Aug-2017 WARN      Host ID: 20b7df6d083651eb04f1f014e8a4958ddf9c1654cb3ad9057adc7e73
16:43:13,644 11-Aug-2017 WARN      Host Name: ultram-rcdnlab-compute-4.localdomain
16:43:13,644 11-Aug-2017 WARN      [DEBUG-ONLY] VM IP: 10.10.10.9; 172.16.180.22; 192.168.1.12; 
16:43:13,644 11-Aug-2017 WARN  =====  SEND NOTIFICATION ENDS  =====
16:43:33,827 11-Aug-2017 WARN  
16:43:33,827 11-Aug-2017 WARN  ===== SEND NOTIFICATION STARTS =====
16:43:33,827 11-Aug-2017 WARN  Type: VM_RECOVERY_COMPLETE
16:43:33,827 11-Aug-2017 WARN  Status: SUCCESS
16:43:33,827 11-Aug-2017 WARN  Status Code: 200
16:43:33,827 11-Aug-2017 WARN  Status Msg: Recovery: Successfully recovered VM [vnfd1-deployment_s6_0_e03f87f5-63b6-4053-8d0f-0c9df963162c].
16:43:33,827 11-Aug-2017 WARN  Tenant: core
16:43:33,827 11-Aug-2017 WARN  Service ID: NULL
16:43:33,828 11-Aug-2017 WARN  Deployment ID: b41ad0ec-bc74-4bb3-85b6-7ef430074187
16:43:33,828 11-Aug-2017 WARN  Deployment name: vnfd1-deployment-1.0.0-1
16:43:33,828 11-Aug-2017 WARN  VM group name: s6
16:43:33,828 11-Aug-2017 WARN  VM Source: 
16:43:33,828 11-Aug-2017 WARN      VM ID: 085baf6a-02bf-4190-ac38-bbb33350b941
16:43:33,828 11-Aug-2017 WARN      Host ID: 
16:43:33,828 11-Aug-2017 WARN      Host Name: 
16:43:33,828 11-Aug-2017 WARN      [DEBUG-ONLY] VM IP: 10.10.10.9; 172.16.180.22; 192.168.1.12; 
16:43:33,828 11-Aug-2017 WARN  VM Target: 
16:43:33,828 11-Aug-2017 WARN      VM ID: a313e8dc-3b0f-4b41-8648-f9b9419bc826
16:43:33,828 11-Aug-2017 WARN      Host ID: 20b7df6d083651eb04f1f014e8a4958ddf9c1654cb3ad9057adc7e73
16:43:33,828 11-Aug-2017 WARN      Host Name: ultram-rcdnlab-compute-4.localdomain
16:43:33,828 11-Aug-2017 WARN      [DEBUG-ONLY] VM IP: 10.10.10.9; 172.16.180.22; 192.168.1.12; 
16:43:33,828 11-Aug-2017 WARN  =====  SEND NOTIFICATION ENDS  =====



[local]rcdn-ulram-lab# show card table 
Slot         Card Type                               Oper State     SPOF  Attach
-----------  --------------------------------------  -------------  ----  ------
 1: CFC      Control Function Virtual Card           Standby        -           
 2: CFC      Control Function Virtual Card           Active         No          
 3: FC       1-Port Service Function Virtual Card    Active         No          
 4: FC       1-Port Service Function Virtual Card    Active         No          
 5: FC       1-Port Service Function Virtual Card    Standby        -           
 6: FC       1-Port Service Function Virtual Card    Active         No          
 7: FC       1-Port Service Function Virtual Card    Active         No          
[local]rcdn-ulram-lab#

Contributed by Cisco Engineers

Dennis Lanov
Cisco TAC Engineer

UltraM: VM Recovery after ESC Retry Expires

Available Languages

Download Options

Bias-Free Language

Contents

Introduction

Problem

Solution

Contributed by Cisco Engineers

Was this Document Helpful?

Contact Cisco

This Document Applies to These Products