Troubleshoot SNMP in QPS Release 5.5

Introduction

This document describes how to troubleshoot Simple Network Management Protocol (SNMP) framework architecture from a Quantum Policy Suite (QPS) perspective.

Prerequisites

Requirements

There are no specific requirements for this document.

Components Used

The information in this document is based on QPS Release 5.5 and later.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Background Information

SNMP Architecture

Each virtual machine (VM) runs an "snmpd" process. The alerts or traps generated by a VM in QPS are forwarded to a user Network Management System (NMS) through an active load balancer such as lbvip01 or lbvip02.

• lbvip01 sends SNMP traps to an external NMS server for external communication. A high QPS availability heartbeat ensures that lbvip01is always available, either on the load balancer VM lb01 or lb02.
• lbvip02 is used by all VMs (for example lb01x, qns0x, portal0x, and so forth) in order to send SNMP traps to an active load balancer for internal communication.
• lbvip01 and lbvip02 IP addresses on lb01and lb02 are located in the /etc/hosts file.
• 162 is the default UDP port used for SNMP trap messages.
• Broadhop is the default SNMP community string to access the framework.

SNMP Configuration Files

Four important QPS Release 5.5 SNMP configuration files or scripts reside on the load balancer VMs:

• /etc/snmp/snmpd.conf is used for IP address and port configuration. The "snmp"service polls the VM every minute for values that are configured in this file. The file sends version 2 traps to lbvip02 through the line: trap2sink lbvip02 broadhop.
• /etc/snmp/scripts/snmp_communities defines the SNMP community name to communicate with the external NMS system, for example, trap_community=Cisco.
• /etc/snmp/scripts/component_trap_conver transforms the trap to a Quantum Network Suite (QNS) trap, as defined in the BROADHOP-NOTIFICATION MIB, and sends it to the external user NMS IP address.
• /etc/snmp/snmptrapd.conf is the Net-SNMP notification receiver configuration file which uses one or more configuration files to control its operation and how received traps must be processed. The file sends the actual traps to the external NMS system through  lbvip0.

XML Files

The Policy and Changing Rules Function (PCRF) client virtual machines, such as pcrfclient01 and pcrfclient02, contain three XML files related to QPS Release 5.5 SNMP configuration:

• attributes.xml contains the mapping between an SNMP Object Identifier (OID) and the Java Management Extension (JMX) javabean or attribute that holds the value.
• managers.xml lists the remote hosts to receive the SNMP traps.
• notifications.xml lists the SNMP traps that can be sent and contains a mapping of the attributes to OIDs.

The three XML files are located in the /etc/broadhop/<servers>/snmp  directory. In order to find the correct directory:

1. Use the cat utility in order to locate the correct server file in the PCRF client virtual machine.

   cat /etc/broadhop/servers

   Example output:

   lb01=iomanager01
   lb02=iomanager02
   qns01=pcrf_A
   qns01=pcrf_A
   qns02=pcrf_A
   qns02=pcrf_A
   qns03=pcrf_B
   qns03=pcrf_B
   qns04=pcrf_B
   qns03=pcrf_B
   qns04=pcrf_B
   pcrfclient01=controlcenter
   pcrfclient01=pb
   pcrfclient02=controlcenter
   pcrfclient02=pb
   

2. After you view the file of the server, you can find the directory with the three XML files for a particular VM as follows:

   /etc/broadhop/pcrf_A/snmp
   /etc/broadhop/pcrf_B/snmp
   /etc/broadhop/pb/snmp
   /etc/broadhop/controlcenter/snmp
   /etc/broadhop/iomanager01/snmp
   /etc/broadhop/iomanager02/snmp
   

MIB Files

The MIBs used by SNMP is located on the respective VM in the /etc/snmp/mibs  directory.

• BROADHOP-QNS-MIB.mib defines retrievable statistics and Key Performance Indicators (KPIs).
• BROADHOP-NOTIFICATION-MIB.mib defines available notifications and traps.

Basic Steps to Troubleshoot SNMP

Perform these steps on the active load balancer virtual machine, either on lb01 or lb02.

1. Ensure that the snmpd process is functional on the respective VM with the command:

   service snmpd status

2. If the snmpd service has stopped, then start the service with the command:

   service snmpd start

3. Ensure that the iptables are turned off and check the status of UDP port 162 (if UDP port 162 is also used at the NMS).
4. Ensure that the Red Hat Package Manager (RPM) versions for SNMP are the same, with the command:

   rpm –qa | grep snmp
   

   Output Example with All Versions at 5.3.2.2:

   net-snmp-5.3.2.2-17.el5
   net-snmp-utils-5.3.2.2-17.el5
   net-snmp-libs-5.3.2.2-17.el5

   • net-snmp is a package required to deploy SNMP.
   • net-snmp-utils is a package that contains various utilities for use with NET-SNMP and is required to run utilities that manage the network with the SNMP protocol.
   • net-snmp-libs contains the runtime libraries for shared binaries and applications.
5. Enter the rpm-qi command to view detailed information on RPM.

   Command Example:

   rpm -qi net-snmp-5.3.2.2-17.el5
   rpm -qi net-snmp-utils-5.3.2.2-17.el5
   rpm -qi net-snmp-libs-5.3.2.2-17.el5

6. In order to ensure that QPS sends SNMP traps to the appropriate NMS, check to see that the external NMS IP is defined in the load balancer VM under /etc/hosts and /etc/snmp/scripts/ component_trap_convert with the command:

   grep 'corporate_nms_ip'

   Note: Where 'corporate_nms_ip' is the NMS IP, for example, 10.106.74.10.

7. Ensure that the snmpd.conf file contains the line rocommunity Broadhop with the command:

   cat /etc/snmp/snmpd.conf

   This line is required because all of the internal traps from the various QNS VMs to the active load balancer VM are sent over the default community name Broadhop.

8. Check to ensure that the load balancer and external NMS system use the same trap community name with the command:

   /etc/snmp/scripts/snmp_communities

   For example, ensure that the load balancer and external NMS system both use trap_community=Cisco .

9. Check the /var/log/snmp/trap log file to ensure that the traps from the respective QNS VMs have properly reached the active load balancer VM.

   Trap Log File Example:

   Jun 15 04:02:16 AIO-POD1 snmpd[3754]: Received SNMP packet(s) from UDP: [127.0.0.1]
   :56568
   Jun 15 04:02:16 AIO-POD1 snmpd[3754]: Connection from UDP: [127.0.0.1]:48071
   Jun 15 04:02:14 AIO-POD1 snmptrapd[3769]: 2014-06-15 04:02:14 AIO-POD1.cisco.com
   [10.106.74.34] (via UDP: [127.0.0.1]:39334) TRAP, SNMP v1, community broadhop       
    NET-SNMP-MIB::netSnmpNotificationPrefix Enterprise Specific Trap (NET-SNMP-AGENT-MIB
   ::nsNotifyRestart) Uptime: 62 days, 5:04:42.43

10. Capture the logs from /var/log/messages on the active load balancer and look for any error messages related to SNMP.