Configure Alarm Email Notifications in vManage

Introduction

This document describes how to configure and send email notifications via vManage for events that take place in the network.

Prerequisites

Requirements

Cisco recommends that you have knowledge of vManage and ensure that your vManage version is 18.3.0 and above. 

Components Used

This document is not restricted to specific software and hardware versions.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Configure

These steps need to be configured in order to enable email notifications.

1. Edit the Email Notifications from Administration > Settings. 

2. Configure the Email Notifications from Monitor > Alarms. 

For Step 1., from vManage Dashboard, navigate to Administration > Settings > Email Notifications > Edit and configure the Enable Email Notifications section. Here is an example screenshot.

You can select the From address as per your choice, however, the domain name should match the mail server. As an example, <..username..>@cisco.com and the reply address is noreply@cisco.com. Because, if there is a reply, vManage does not capture it. This scenario is similar to the auto-generated emails that come with a no-reply address.

If the user is using GMail SMTP server, there has been a change with the way GMail integrates with the third party apps. Eg., vManage  is a third party app for GMail.   We need to make sure that two-step verification and app-password are enabled.   You can set this up in the Security tab under Manage your Google account.  When you enable SMTP Authentication under vManage settings, make sure to use the app-password.  

For Step 2., this includes sub-steps like Severity, Alarm Name, Email-list, and WebHook URL.

Sample screenshots:

Webhooks are used by an external system in order to notify the local system about a certain event or update. They are like API calls

in the opposite direction. HTTP POST can be sent from vManage to any service that listens for this. For example, when you set up a webhook in vManage that hits a "serverless" piece of code in AWS, it fires off an event to page a bunch of people in the organization. There are several online services that you can connect to do these things. Refer to https://testwebhooks.com/.

Another example is to create something in slack in order to receive vManage webhooks. Refer to https://api.slack.com/incoming-webhooks

Email Threshold

There is a threshold field in the Email Notifications page. Navigate to Monitor> Alarms> Email Notifications.

This field indicates how many emails you want to receive per minute. By default, a maximum of 5 emails per minute. When the emails go beyond that threshold, you receive the message as shown in the image. The emails are not sent for 5 minutes and then the threshold starts fresh.

The rest of the document captures the usage of "Email Alerts."

Logs

Check the vManage logs: /var/log/nms/vmanage-server.log

Verify

Verify via vManage-Dashboard. Navigate to Monitor > Audit Log as shown in the image. 

Check Email 

Troubleshoot

This section provides information you can use in order to troubleshoot your configuration.

Check Audit-Log:

Currently, there is an issue where vManage Audit-Log might say that email might have been sent, but in fact, the email is not received. You can verify this through the in/var/log/nms/vmanage-server.log file as shown in the image.

Not all email alerts get generated consistently:

Firstly, it depends on how many alarms are generated for the set of events. vManage tries to combine events to one alarm if they are related. If not, it generates multiple alarms at the same time. For each alarm generated and rule, there is an email. Email notification is tied to alarms, not events.

Secondly, if in the first rule, you see that there are multiple alarms, there is only 1 email sent for that rule match. If you want multiple emails for the individual event, the individual rule needs to be defined.

"Username and Password not accepted" in vmanage-server.logs:

As seen in the screenshot, you might see the error "Username and Password not accepted. Learn more at

535 5.7.8https://support.google.com/mail/?p=BadCredentials m92sm8305479qte.50 - gsmtp." Despite this, it might show on the vManage Audit Log that the email did indeed send.

By default, Gmail accounts are highly secure. When you use the Gmail SMTP from a non-Gmail tool, email is blocked. In order to test this, follow these steps: 

1. Log in to Gmail.
2. Access the URL ashttps://www.google.com/settings/security/lesssecureapps.
3. Select Turn on.

Email notifications can then be received on the email account specified. 

Failure to send email Notification(s):

In some cases, you might see that the generated email is denied by the mail server. This could be if the already provided account configuration might be incorrect or access is not granted. In the logs, you can see this message: SendAsDeniedException.

03-Dec-2018 15:46:37,177 CST ERROR [ts_vManage][EmailNotiUtil] (default task-84) |default| Sending email notification failed: 554 5.2.0 
STOREDRV.Submission.Exception:SendAsDeniedException.MapiExceptionSendAsDenied; Failed to process message due to a permanent exception
with message Cannot submit message.

Another use case is that the number of email messages users can send per day varies from SMTP Server to Server. You might see this log message in vManage:

"Sending email notification failed : com.sun.mail.smtp.SMTPSendFailedException: 550 5.4.5 Daily user sending quota exceeded."

For example, there is a limit if it is Gmail: https://support.google.com/a/answer/166852.

If you see either of the below  messages, collect a packet capture of the communication between the vManage and the mail server.  

1.  Verify that there is a resposne from the mail server.

"Sending email notification failed : javax.mail.MessagingException: Could not connect to SMTP host: mail.sino-bridge.com, port: 25, response: 550"

"Sending email notification failed : javax.mail.MessagingException: Could not convert socket to TLS;"

 If you see it fail after the certificate in a similar manner to the below capture,

    2. check the validity of the certificate on the mail server. 

3. Confirm if the server supports STARTTLS.   Currently, this is required for the mail server to work with vManage.  Enhancement CSCvv40941 is open for vManage to be able to support servers without STARTTLS

Other Validation Checks

Other Error Logs

 The email is sent from vManage IP (Public-IP of VPN0 Transport Interface).