PAWS Management Administration Guide
Troubleshooting PAWS Management
Downloads: This chapterpdf (PDF - 508.0KB) The complete bookPDF (PDF - 2.72MB) | The complete bookePub (ePub - 721.0KB) | Feedback

Troubleshooting PAWS Management

Troubleshooting PAWS Management

SNMP setup

SNMP version 3 provides security features such as authentication (verifying that the request comes from a genuine source), privacy (encryption of data), authorization (verifying that the user allows the requested operation), and access control (verifying that the user has access to the objects requested). To prevent SNMP packets from being exposed on the network, you can configure encryption with SNMPv3.

This chapter, which describes how to configure SNMP v3 so the network management system can monitor Cisco PAWS-M, contains the following topics:

SNMP configuration checklist

The following table provides an overview of the steps for configuring SNMP:

Configuration steps Related Procedures and Topics

Step 1

Install and configure the SNMP NMS.

SNMP product documentation that supports the NMS

Step 2

In the CLI, verify that the system started the SNMP services, including:

  • SNMP Master Agent
  • Native Agent
  • System Application Agent
  • Cisco Syslog Agent
  • MIB2 Agent
  • Host Resources Agent

In the command line interface, enter the following command:

utils service list

Step 3

Configure the SNMP user.

SNMP users

Step 4

Configure the notification destination for traps or informs.

Step 5

Configure the system contact and location for the MIB2 system group.

MIB2 system group

Step 6

Configure trap settings for CISCO-SYSLOG-MIB.

Use these guidelines to configure CISCO-SYSLOG-MIB trap settings on your system:

  • Set clogsNotificationEnabled (1.3.6.1.4.1.9.9.41.1.1.2) to true by using the SNMP Set operation; for example, use the net-snmp set utility to set this OID to true from the linux command line using: snmpset -c <community string> -v2c <transmitter ipaddress> 1.3.6.1.4.1.9.9.41.1.1.2.0 i 1 You can also use any other SNMP management application for the SNMP Set operation.
  • Set clogMaxSeverity (1.3.6.1.4.1.9.9.41.1.1.3) value by using the SNMP Set operation; for example, use the net-snmp set utility to set this OID value from the linux command line using: snmpset -c public -v2c 1<transmitter ipaddress> 1.3.6.1.4.1.9.9.41.1.1.3.0 i <value> Enter a severity number for the <value> setting. Severity values increase as severity decreases. A value of 1 (Emergency) indicates highest severity, and a value of 8 (Debug) indicates lowest severity. Syslog agent ignores any messages greater than the value that you specify; for example, to trap all syslog messages, use a value of 8.

Step 7

Restart the SNMP Master Agent service. (Optional)

Tip    The system automatically restarts the SNMP Master Agent after you execute the utils snmp config commands.

At the command line, enter the following command:

utils service start SNMP Master Agent

Step 8

On the NMS, configure the Cisco PAWS-M trap parameters.

SNMP users

The following table shows the commands that you need to work with SNMP users on the Cisco PAWS-M platform:

Table 1 Trace CLI commands
Task Command

List the SNMP users.

utils snmp config user 3 list

Add an SNMP user.

utils snmp config user 3 add

The system prompts you for the parameters. See the SNMP users table for parameter names and descriptions.

Update an SNMP user.

utils snmp config user 3 update

The system prompts you for the parameters. See the SNMP users table for parameter names and descriptions.

Delete an SNMP user.

utils snmp config user 3 delete

The system prompts you for the parameters. See the SNMP users table for parameter names and descriptions.

SNMP user CLI parameters

The following table describes the SNMP user parameter settings for V3.

Table 2 SNMP User Parameter Settings for V3
Field Description

username

The name of the user for which you want to provide access. The name can contain up to 32 characters and can contain any combination of alphanumeric characters, hyphens (-), and underscore characters (_).

Tip    Enter users that you have already configured for the network management system (NMS).

authprotocol

Authentication protocol. To specify HMAC-SHA, enter SHA.

authpassphrase

Specifies the authentication protocol password. The password must contain at least 8 characters.

privprotocol

Specifies the privacy protocol, either AES128, AES192, or AES256

privpassphrase

Specifies the privacy protocol password. The password must contain at least 8 characters.

accessprivilege

Enter one of the following options for the access level:

  • ReadOnly—The user can only read the values of MIB objects.
  • ReadWrite—The user can read and write the values of MIB objects.
  • ReadWriteNotify—The user can read and write the values of MIB objects and send MIB object values for a trap and inform messages.
  • NotifyOnly—The user can only send MIB object values for trap and inform messages.
  • ReadNotifyOnly—The user can read values of MIB objects and also send the values for trap and inform messages.
  • None—The user cannot read, write, or send trap information.
Tip    To change the trap configuration parameters, you need to configure a user with NotifyOnly, ReadNotifyOnly, or ReadWriteNotify privileges.

ipaddress1

Specify an IP address from which to accept packets. The default specifies to accept packets from all hosts.

ipaddress2

Specify an IP address from which to accept packets. The default specifies to accept packets from all hosts.

SNMP trap notification destinations

An SNMP agent sends notifications to NMS in the form of traps or informs to identify important system events. Traps do not receive acknowledgments from the destination whereas informs do receive acknowledgments.

The following section applies to SNMP V3 notification destination configuration.

The following table shows the commands that you need to work with SNMP trap notification destinations on the PAWS-M platform:

Table 3 SNMP Trap Notification Destinations CLI commands
Task Command

List trap notification destinations.

utils snmp config trap 3 list

Add a v3 trap notification destination that is associated with a configured v3 username.

utils snmp config trap 3 add

The system prompts you for the parameters. See the SNMP trap notification destinations table for parameter names and descriptions.

Update a trap notification destination.

utils snmp config trap 3 update

The system prompts you for the parameters. See the SNMP trap notification destinations table for parameter names and descriptions.

Delete a trap notification destination.

utils snmp config trap 3 delete

The system prompts you for the parameters. See the SNMP trap notification destinations table for parameter names and descriptions.

Trap Notification Destination Parameter settings

The following table describes the trap notification destination parameter settings for V3.

Table 4 Trap Notification Destination Parameter Settings for V3
Field Description

ipaddress

The host IP address of the notification destination.

portno

The notification-receiving port number on the destination server.

oldportno

The notification-receiving port number on the destination server that is currently configured.

newportno

The notification-receiving port number on the destination server that you want to use when updating the trap notification destination.

username

Specifies the SNMP user associated to the notification destination.

SNMP inform notification destination

An SNMP agent sends notifications to NMS in the form of traps or informs to identify important system events. Traps do not receive acknowledgments from the destination whereas informs do receive events. Traps do not receive acknowledgments from the destination whereas informs do receive acknowledgments.

The following table describes the inform notification destination configuration settings for V3.

Table 5 SNMP inform notification destination CLI commands
Task Command

List inform notification destinations.

utils snmp config inform 3 list

Add a v3 inform notification destination.

utils snmp config inform 3 add

The system prompts you for the parameters. See SNMP inform notification destination for parameter names and descriptions.

Update an inform notification destination.

utils snmp config inform 3 update

The system prompts you for the parameters. See SNMP inform notification destination for parameter names and descriptions.

Delete an inform notification destination.

utils snmp config inform 3 delete

The system prompts you for the parameters. See SNMP inform notification destination for parameter names and descriptions.

Inform notification destination parameter settings

Table 6 Inform notification destination parameter settings for V3
Field Description

ipaddress

The host IP address of the notification destination.

portno

The notification-receiving port number on the destination server.

oldportno

The notification-receiving port number on the destination server that is currently configured.

newportno

The notification-receiving port number on the destination server that you want to use when updating the inform notification destination.

username

Specifies the SNMP user associated to the notification destination.

oldusername

Specifies the v3 username that is currently associated with the inform.

newusername

Specifies the v3 username that you want to associate with the inform.

deleteuserconf

Specifies confirmation for deleting the old user, either Y or N.

authprotocol

Authentication protocol. To specify HMAC-SHA, enter SHA.

authpassphrase

Specifies the authentication protocol password. The password must contain at least 8 characters.

privprotocol

Specifies the privacy protocol, either AES128, AES192, or AES256

privpassphrase

Specifies the privacy protocol password. The password must contain at least 8 characters.

accessprivilege

Enter one of the following options for the access level:

  • ReadWriteNotify—The user can read and write the values of MIB objects and send MIB object values for a trap and inform messages.
  • NotifyOnly—The user can only send MIB object values for trap and inform messages.
  • ReadNotifyOnly—The user can read values of MIB objects and also send the values for trap and inform messages.

engineId

Specifies the remote engine ID of the server to which to send inform messages.

MIB2 system group

You can use the CLI to configure the system contact and system location objects for the MIB-II system group. For example, you could enter Administrator, 555-121-6633, for the system contact and San Jose, Bldg 23, 2nd floor, for the system location.

The following table shows the commands that you need to work with MIB2 system groups on the Cisco IME server:

Table 7 MIB2 CLI commands
Task Command

List the MIB2 system group configuration.

utils snmp config mib2 list

Add a MIB2 system group.

utils snmp config mib2 add

The system prompts you for the parameters. See MIB2 system group for parameter names and descriptions.

Update a MIB2 system group.

utils snmp config mib2 update

The system prompts you for the parameters. See MIB2 system group for parameter names and descriptions.

Delete a MIB2 system group.

utils snmp config mib2 delete

The system prompts you for the parameters. See MIB2 system group for parameter names and descriptions.

MIB2 system group CLI parameters

Table 8 MIB2 system group CLI parameter settings
Field Description

Server

The server for which you want to configure contacts.

SysContact

Specifies a person to notify when problems occur.

SysLocation

Specifies the location of the person that is identified as the system contact.

SNMP Management Information Base (MIB)

SNMP allows access to Management Information Base (MIB), which is a collection of information that is organized hierarchically. MIBs comprise managed objects, which are identified by object identifiers. A MIB object, which contains specific characteristics of a managed device, comprises one or more object instances (variables).

The SNMP interface provides these Cisco Standard MIBs:

  • CISCO-CDP-MIB
  • CISCO-SYSLOG-MIB

The Simple Network Management Protocol (SNMP) extension agent resides in the server. The SNMP interface also provides these Industry Standard MIBs:

  • SYSAPPL-MIB
  • MIB-II (RFC 1213)
  • HOST-RESOURCES-MIB

Cisco PAWS-M SNMP Interface supports the following MIBs.

CISCO-CDP-MIB

Use the CDP subagent to read the Cisco Discovery Protocol MIB, CISCO-CDP-MIB. This MIB enables Cisco PAWS-M to advertise itself to other Cisco devices on the network.

The CDP subagent implements the CDP-MIB. The CDP-MIB contains the following objects:

  • cdpInterfaceIfIndex
  • cdpInterfaceMessageInterval
  • cdpInterfaceEnable
  • cdpInterfaceGroup
  • cdpInterfacePort
  • cdpGlobalRun
  • cdpGlobalMessageInterval
  • cdpGlobalHoldTime
  • cdpGlobalLastChange
  • cdpGobalDeviceId
  • cdpGlobalDeviceIdFormat
  • cdpGlobalDeviceIdFormatCpd

SYSAPPL-MIB

Use the System Application Agent to get information from the SYSAPPL-MIB, such as installed applications, application components, and processes that are running on the system.

System Application Agent supports the following object groups of SYSAPPL-MIB:

  • sysApplInstallPkg
  • sysApplRun
  • sysApplMap
  • sysApplInstallElmt
  • sysApplElmtRun

MIB-II

Use MIB2 agent to get information from MIB-II. The MIB2 agent provides access to variables that are defined in RFC 1213, such as interfaces, IP, and so on, and supports the following groups of objects:

  • system
  • interfaces
  • at
  • ip
  • icmp
  • tcp
  • udp
  • snmp

HOST-RESOURCES MIB

Use Host Resources Agent to get values from HOST-RESOURCES-MIB. The Host Resources Agent provides SNMP access to host information, such as storage resources, process tables, device information, and installed software base. The Host Resources Agent supports the following groups of objects:

  • hrSystem
  • hrStorage
  • hrDevice
  • hrSWRun
  • hrSWRunPerf
  • hrSWInstalled

CISCO-SYSLOG-MIB

Syslog tracks and logs all system messages, from informational through critical. With this MIB, network management applications can receive syslog messages as SNMP traps.

The Cisco Syslog Agent supports trap functionality with the following MIB objects:

  • clogNotificationsSent
  • clogNotificationsEnabled
  • clogMaxSeverity
  • clogMsgIgnores
  • clogMsgDrops

Troubleshooting installation

Use the following sections to troubleshoot problems that occur during installation of the Cisco PAWS-M:

Manage network errors during installation

During the installation process, the installation program verifies that the server can successfully connect to the network by using the network configuration that you enter. If the server cannot connect, a message appears; you get prompted to select one of the following options:

  • RETRY —The installation program tries to validate networking again. If validation fails again, the error dialog box appears again.
  • REVIEW (Check Install)—This option allows you to review and modify the networking configuration. When detected, the installation program returns to the network configuration windows. Networking is validated after you complete each networking window, so the message may appear multiple times.
  • HALT— The installation halts. You can copy the installation log files to a USB disk to aid troubleshooting of your network configuration.
  • IGNORE —The installation continues. The networking error gets logged. In some cases, the installation program validates networking multiple times, so this error dialog box may appear multiple times. If you choose to ignore network errors, the installation may fail.

Examine log files

If you encounter problems with the installation, you may be able to examine the install log files by entering the following commands in Command Line Interface.

To obtain a list of install log files from the command line, enter

CLI> file list install *

To view the log file from the command line, enter

CLI> file view install log_file

where:

log_file specifies a log file name having the format: install_log_YYYY-MM-DD.HH.MM.SS.log in which the date and time of the log file is the time at which the install or the upgrade was initiated on the system.

You can get more information about installation events by viewing or downloading the System History log. Refer to the following for more information: System History Log section in the Cisco Intercompany Media Engine Installation and Configuration Guide.

Troubleshooting PAWS-M components

Use the following sections to troubleshoot problems with Cisco PAWS-M components:

Troubleshoot PAWS-M services

The following table lists the troubleshooting steps to perform for Cisco PAWS-M.

Procedure
      Command or Action Purpose
    Step 1 Check to see that the Cisco Platform Manager service is running.  

    Services

     
    Step 2 Check for communication problems between the servers. 

    Configuration checklist for PAWS Management

     
    Step 3 If you have an issue with an upgrade task, check the server type in the server group. For Upgrade Tasks, publisher server and subscriber servers cannot be in the same group. 

    Manage server groups

    Failed COP file install or UC application server upgrade section below 
    Step 4 Check PAWS-M error messages. 

    Common PAWS-M errors section below

     
    Step 5 Check the Platform Manager Log Files. 

    PAWS-M Log files section below

    Traces

     

    Failed COP file install or UC application server upgrade

    Procedure
        Command or Action Purpose
      Step 1 If the UC application server with the failed upgrade is version 8.6.1, check the server inventory page. Be sure the correct publisher is administered in the publisher field. (This field will not appear if the server is version 8.6.2 or later.)   
      Step 2 If the upgrade (or COP file install) on a UC application server is not successful, obtain the install log files from the UC application server and check for errors.  

      An upgrade on a UC application server that is scheduled using PAWS-M is performed in the same manner as upgrades done using the UC application server's CLI or GUI interface. Therefore, all normal upgrade troubleshooting logs should be gathered from the UC Application server, to diagnose the problem.

       

      Common PAWS-M errors

      The following section contains common PAWS-M errors and possible reasons for those errors.

      "Could not contact server" or “Server not available”.

      • All connections between UC applications and PAWS-M must be established before you run a task.
      • All UC applications must be contacted at least once prior to the first PAWS-M task run.
      • SFTP or FTP and UC Applications must be routable from the PAWS-M; for example, use public IP if NATs are used. Likewise, the SFTP or FTP servers must be routable from the UC Applications.

      Go to the UC application server that can not be contacted, and verify that the "Platform Administrative Web Service" is running on the UC application server. See Configuration checklist for PAWS Management.

      PAWS-M Log files

      When you troubleshoot issues for PAWS-M, you can access the following log files on the PAWS-M platform at the following locations:

      • file get activelog tomcat/logs/platform-api/log4j/*— This log file includes information generated by the UC Applications. The same log file is also stored on the application server. Using this command, you can view:

        • SOAP messages from the Platform Manager to the UC App server - including inputs, results, errors, and messages

      • file get activelog tomcat/logs/pm/log4j/*— This log file includes information from the Platform Manager. Using this command, you can view:

        • REST traffic between the browser and the Platform Manager including inputs, results, errors, and messages
        • SOAP messages to the UC Applications including inputs, results, errors, and messages
        • Database access including updates, queries, and results
        • Task related events like scheduling, starting, and updating tasks
        • Background jobs like synchronization tasks

      • If you are troubleshooting communication between PAWS-M and an application server, you can also get these files from the UC application server: file get activelog tomcat/logs/platform-api/log4j/*— This log file includes information generated by the UC applications. The same log file is also stored on the application server. Using this command, you can view:

        • SOAP messages from the Platform Manager including inputs, results, errors, and messages
        • Calls to underlying OS components like the upgrade scripts

      Troubleshoot PAWS-M administrative interface

      The following table lists the troubleshooting steps to perform for the Administrative Interface.

      Procedure
          Command or Action Purpose
        Step 1 Check to see that the following services are running:
        • Cisco Tomcat
        • Cisco Platform Manager
         

        Services

         
        Step 2 Check the following log file: activelog tomcat/logs/pm/log4j 

        Troubleshoot PAWS-M services

        Trace management

        Traces

         

        Alarm management

        Alarms provide information on runtime status and the state of the system, so you can troubleshoot problems that are associated with your system; for example, to identify issues with the Disaster Recovery System. Alarm information, which includes an explanation and recommended action, also includes the application name, machine name, and so on, to help you perform troubleshooting.

        You configure the alarm interface to send alarm information to multiple locations, and each location can have its own alarm event level (from debug to emergency). Alarms can go to the Syslog Viewer (local syslog), Syslog file (remote syslog), SNMP traps, or to all destinations.

        When a service issues an alarm, the alarm interface sends the alarm information to the locations that you configure (and that are specified in the routing list in the alarm definition). The system can either forward the alarm information, as is the case with SNMP traps, or the system can write the alarm information to its final destination (such as a log file).

        As soon as you enter the CLI command, the system will prompt you for the required parameters. Enter the values to see the output.

        The following table shows the commands that you need to work with alarms on the Cisco PAWS-M platform:

        Table 9 Alarm CLI commands
        Task Command

        Display the alarm configuration for a specific service/list of all services

        show alarm

        Required Parameter:

        servicename—Name of the service. It can contain multiple words.

        Example:

        Enter the servicename as all to show the alarm configurations of all the services.

        Enter the servicename as Cisco Tomcat to show the alarm configuration of Cisco Tomcat service.

        Enable/Disable alarms for a particular destination

        set alarm status

        Required Parameters:

        status—enable or disable.

        servicename—Name of the service. It can contain multiple words.

        monitorname—SDI, SDL, Event_Log, or Sys_Log.

        Enable alarms for a remote Syslog server

        set alarm remotesyslogserver

        Required Parameters:

        servicename—Name of the service. It can contain multiple words.

        servername—Name of the remote syslog server.

        Set the event level for an alarm

        set alarm severity

        Required Parameters:

        servicename—Name of the service. It can contain multiple words.

        monitorname—SDI, SDL, Event_Log, or Sys_Log.

        severity equals one of the following:

        • Emergency—This level designates the system as unusable.
        • Alert—This level indicates that immediate action is needed.
        • Critical—The system detects a critical condition.
        • Error—This level signifies that an error condition exists.
        • Warning—This level indicates that a warning condition is detected.
        • Notice—This level designates a normal but significant condition.
        • Informational—This level designates information messages only.
        • Debug—This level designates detailed event information that Cisco TAC engineers use for debugging.

        Set alarm configuration to default values

        Tip    This option is available only for service names beginning with Cisco.

        set alarm default

        Required Parameters:

        servicename—Name of the service. It can contain multiple words.

        Trace management

        Traces assist you in troubleshooting issues with your application. You use the CLI to specify the level of information that you want traced as well the type of information that you want to be included in each log file. You can configure trace parameters for any service on the Cisco PAWS-M platform.

        After you have configured information that you want to include in the log files for the various services, you can collect and view log files through log collection. To do this, configure trace using set trace CLI command.

        You can configure the level of information that you want traced (debug level), what information you want to trace (trace fields), and information about the trace files (such as number of files per service, size of file, and time that the data is stored in the log files)

        Trace configuration

        You use the command line interface (CLI) to enable and disable tracing as well as to configure trace settings for specific services on the Cisco PAWS-M platform. As soon as you enter the CLI command, the system prompts you for the required parameters. For more information regarding trace collection, see the Trace management.

        The following table shows the commands that you need to work with traces on the Cisco PAWS-M platform:

        Table 10 Trace CLI commands
        Task Command

        Display the trace configuration for a specified service

        show trace

        Required Parameter:

        servicename—Name of the service. It can contain multiple words.

        Example:

        Enter the servicename as all to show the trace configurations of all the services.

        Enter the servicename as Cisco AMC Service to show the trace configuration of Cisco AMC service.

        Display the trace levels available for a specified service

        show tracelevels

        Required Parameter:

        servicename—Name of the service. It can contain multiple words.

        Enable/Disable trace for a specified service

        set trace status

        Required Parameters:

        status— enable or disable

        servicename—Name of the service. It can contain multiple words.

        Specify the debug trace level settings for a specified service

        set trace tracelevel

        Required Parameters:

        tracelevel—Use show tracelevels CLI command to find the trace levels for a given servicename.

        servicename—Name of the service. It can contain multiple words.

        Specify the maximum size of a trace files for a specific service from 1 to 10 megabytes

        set trace maxfilesize

        Required Parameters:

        servicename—Name of the service. It can contain multiple words.

        size—Maximum size of the trace files from 1 to 10 megabytes.

        Specify the maximum number of log files per service.

        The system automatically appends a sequence number to the file name to indicate which file it is; for example, cus299.txt. When the last file in the sequence is full, the trace data begins writing over the first file.

        set trace maxnumfiles

        Required Parameters:

        servicename—Name of the service. It can contain multiple words.

        filecount—Number of trace files from 1 to 10000.

        Set the user categories flag to the value provided, for a specified service.

        Tip    This option is available only for service names beginning with Cisco.

        set trace usercategories

        Required Parameters:

        flagnumber—Hexadecimal value from 0 to 7FFF. 7FFF means all the flags are enabled.

        servicename—Name of the service. It can contain multiple words.

        Set trace configuration to default values for a specified service.

        Tip    This option is available only for service names beginning with Cisco.

        set trace default

        Required Parameter:

        servicename—Name of the service. It can contain multiple words.

        Log collection

        You can collect log files by performing any of the following tasks:

        • To view the log files directly from the CLI, enter the following the CLI commands:
          • file list
          • file view
          • file search
        • To bundle the various log files and send them to the local SFTP directory, enter the CLI command file get. Then, use a SFTP client to obtain the .tar files and send to the team that troubleshoots.