Cisco Elastic Services Controller monitors the
VM to detect any erroneous condition. ESC uses one of its monitoring methods to detect
actions on a VM, and passes this information to the rules service for processing. The
monitoring request comes from the northbound client along with VNF deployment requests.
There are two sections
in the datamodel xml file which define the events and rules: KPI and Rule.
Based on the monitors
and actions, rules are triggered.
In the example above,
an event is sent to check whether the VM is alive. The VM is pinged at regular
intervals, and based on the result VM_ALIVE event is sent to the rules engine
along with the details of the VM.
The rules engine
receives events from the monitoring engine. The rules engine can handle simple
to complex events. Based on the event received an action is triggered.
If the VM is not
alive, based on the event the actions defined in the <rule> section are
triggered. This can be found in the dep.xml datamodel.
<action>FALSE recover autohealing</action>
The rules section describes the actions to be
executed once a monitoring event has been detected. The dynamic mapping API drives the
rules based on keywords.
In the above example,
the following actions are performed based on the given condition:
Whether the event is pingable or not, the details are logged.
TRUE servicebooted.sh: The action identified by this keyword in the dynamic mapping API is triggered when the VM moves from a non-pingable to a pingable state. The serviceboot script informs ESC that the VM is alive allowing it to transition the VMs state.
autohealing: The action identified by this keyword will be triggered and the VM
will be recovered without the administrator's intervention.
Monitoring log files
for troubleshooting are available at
Monitoring the VM Network Status
When using ICMP ping monitoring, if ESC receives a VM Down event, the healing workflow attempts to recover the VM with the recovery policy. If there is an issue with the network interface or IP route from ESC to the VNF. For example, if the gateway is down, it might trigger the VM Down event incorrectly, which leads to an unnecessary recovery.
The check interface function does further scan to the network route by checking the health status of all the network interfaces and the operation state of the gateway. If there is any problem in the network environment, it assumes the VNF is alive.
The VM_NETWORK_STATE event is sent to northbound if ESC detects a network issue or if any existing issue is fixed (autohealing).
The following failure notification is sent to northbound:
16:13:15,567 14-Mar-2018 WARN ===== SEND NOTIFICATION STARTS =====
16:13:15,567 14-Mar-2018 WARN Type: VM_NETWORK_STATE
16:13:15,567 14-Mar-2018 WARN Status: FAILURE
16:13:15,567 14-Mar-2018 WARN Status Code: 500
16:13:15,567 14-Mar-2018 WARN Status Msg: Warning: VM [NG_G1_0_46fdcf70-f4ea-4289-ae79-08674e7d6f42] has a network problem: Network interface not healthy, please check.
16:13:15,567 14-Mar-2018 WARN Tenant: tenant2
16:13:15,567 14-Mar-2018 WARN Deployment ID: 455d2407-9dda-4203-95b0-724c4a651720
16:13:15,567 14-Mar-2018 WARN Deployment name: NG
16:13:15,567 14-Mar-2018 WARN VM group name: G1
16:13:15,567 14-Mar-2018 WARN VM Source:
16:13:15,567 14-Mar-2018 WARN VM ID: 4bee016a-6b30-43ff-a249-157a07d9b4db
16:13:15,567 14-Mar-2018 WARN VM Name: NG_G1_0_46fdcf70-f4ea-4289-ae79-08674e7d6f42
16:13:15,568 14-Mar-2018 WARN VM Name (Generated): NG_G1_0_46fdcf70-f4ea-4289-ae79-08674e7d6f42
16:13:15,568 14-Mar-2018 WARN VIM ID: default_openstack_vim
16:13:15,568 14-Mar-2018 WARN VIM Project: tenant2
16:13:15,568 14-Mar-2018 WARN VIM Project ID: 62afb63cd28647a7b526123cac1ba605
16:13:15,568 14-Mar-2018 WARN Host ID: b83004159a46c20bc8383927c2231067bb0c1905b4b4c28475653190
16:13:15,568 14-Mar-2018 WARN Host Name: my-ucs-50
16:13:15,568 14-Mar-2018 WARN ===== SEND NOTIFICATION ENDS =====
The following success notification is sent to northbound when the network problem is fixed.
16:13:19,141 14-Mar-2018 INFO ===== SEND NOTIFICATION STARTS =====
16:13:19,141 14-Mar-2018 INFO Type: VM_NETWORK_STATE
16:13:19,142 14-Mar-2018 INFO Status: SUCCESS
16:13:19,142 14-Mar-2018 INFO Status Code: 200
16:13:19,142 14-Mar-2018 INFO Status Msg: Network of VM [NG_G1_0_46fdcf70-f4ea-4289-ae79-08674e7d6f42] has been restored.
16:13:19,142 14-Mar-2018 INFO Tenant: tenant2
16:13:19,142 14-Mar-2018 INFO Deployment ID: 455d2407-9dda-4203-95b0-724c4a651720
16:13:19,142 14-Mar-2018 INFO Deployment name: NG
16:13:19,142 14-Mar-2018 INFO VM group name: G1
16:13:19,142 14-Mar-2018 INFO VM Source:
16:13:19,142 14-Mar-2018 INFO VM ID: 4bee016a-6b30-43ff-a249-157a07d9b4db
16:13:19,142 14-Mar-2018 INFO VM Name: NG_G1_0_46fdcf70-f4ea-4289-ae79-08674e7d6f42
16:13:19,142 14-Mar-2018 INFO VM Name (Generated): NG_G1_0_46fdcf70-f4ea-4289-ae79-08674e7d6f42
16:13:19,142 14-Mar-2018 INFO VIM ID: default_openstack_vim
16:13:19,143 14-Mar-2018 INFO VIM Project: tenant2
16:13:19,143 14-Mar-2018 INFO VIM Project ID: 62afb63cd28647a7b526123cac1ba605
16:13:19,143 14-Mar-2018 INFO Host ID: b83004159a46c20bc8383927c2231067bb0c1905b4b4c28475653190
16:13:19,143 14-Mar-2018 INFO Host Name: my-ucs-50
16:13:19,143 14-Mar-2018 INFO ===== SEND NOTIFICATION ENDS =====