AXP 1.6 Developer Guide
Logging and Troubleshooting

Table Of Contents

Logging and Troubleshooting

Troubleshooting Your Application

Troubleshooting the vi Editor

Logging

Syslog Server Logging

Integrating Log Files into the Host OS

Rsync

Updating System Libraries

System Performance Status Monitoring


Logging and Troubleshooting


Logging and Troubleshooting are described in the following sections:

Troubleshooting Your Application

Troubleshooting the vi Editor

Logging

Syslog Server Logging

Integrating Log Files into the Host OS

Rsync

System Performance Status Monitoring

Troubleshooting Your Application

For intermittent issues, Cisco AXP provides support for Linux commands such as vmstat and top in the shell so you can check if your application is using a lot of CPU capacity or it is using up a lot of RAM. Using up all the available RAM/ allocated CPU, may cause the application to function very slowly.

If you have configured your application for logging, the application log files are located in the application's virtual instance and contain messages written by your application. The log files contain error messages created by the system. The log file may contain evidence of why the virtual instance did not start and they are displayed one page at a time.

To debug your application perform the following basic debugging steps:


Step 1 Check that your application is running:

a. app-service appname

, where appname is the name of your application

b. show process

shows all processes sorted by process ID in ascending order.

Refer to the "Viewing Processes" section in Cisco AXP User Guide. If your application is part of a CLI plug-in application, the application must always be running.

Step 2 show state

If application health = "ALIVE", then your application is running, go to the end of the steps.

If the application state is not "online" go to Step 3.

If application state is "---" and your application is coded to provide its health status OR

if the application health is not = "ALIVE", then go to step 5.

Step 3 Perform the following steps:

a. exit

exits to the Cisco AXP prompt

b. Check the messages.log file or messages.log.prev file by entering one of the following commands:

show log name messages.log containing <application name> paged

show log name messages.log.prev containing <application name> paged

, where <application name> is the name of your application.


Note If the application writes log files to directory /var/log or has a link pointing to /var/log, view the log files by entering:
show log name /var/log/messages.log paged


Step 4 (Optional) For systems using the AIM platform, the log level of the host system does not include INFO messages. (The messages reduce the amount of disk space used to store log files.) The logs may not give enough information about why the application fails to start in the virtual instance. To see the INFO messages displayed on the console, perform the following steps:

router# configure terminal

Enter the configuration commands, one per line. End with CNTL/Z.

router(config)# log console info

no shutdown

When an attempt is made to start the virtual instance, INFO messages are printed to the console for debugging purposes.

Step 5 Do this step if the application health is not = "ALIVE", or
the application state is "---" and your application is coded to provide its health status.

Check the messages.log file or messages.log.prev file by entering one of the following commands:

show log name messages.log paged

show log name messages.log.prev paged

, where <application name> is the name of your application.

Step 6 trace all

turns on tracing

Step 7 Run your application

Step 8 show trace buffer

Step 9 On the router, enter the following command:

show interfaces interface


Troubleshooting the vi Editor

The vi editor does not behave as expected with some terminals. If this is the case, run vi with parameter:

-T ansi

Logging

For logging inside a virtual instance, the syslog daemon syslogd, adds tracing or logging information to the file /var/log/messages.log. To configure remote logging, see "Configuring Remote Logging" in Cisco AXP User Guide

Logging is described in the following sections:

Log File Rotation

Logging Levels

Logging Usage

Log File Rotation

The Cisco AXP syslog supports log file rotation of two files:

messages.log (primary log file)

messages.log.prev (backup log file)

When the messages.log reaches a limit specified by the limit log-file size command, the file's contents are moved to backup file messages.log.prev and a new messages.log file is started. The default log file size is 20 MB.

You can use the command show logs to display the contents of all log files, and the command show log name to display the contents of a log file in the /var/log directory.

Logging Levels

Cisco AXP supports the following log levels:

LOG_EMERG: system is unusable

LOG_ALERT: action must be taken immediately

LOG_CRIT: critical condition

LOG_ERR: error condition

LOG_WARNING: warning condition

LOG_NOTICE: normal, but significant, condition

LOG_INFO: informational message

LOG_DEBUG: debug-level message

Out of all the log levels that are available to the application, the CLI reduces these to a more manageable number of four log levels.

You can set your log threshold to one of the following values:

Errors: Events with LOG_ERR and lower severity are logged, including LOG_EMERG, LOG_ALERT, and LOG_CRIT.

Warning: Events with LOG_WARNING and lower severity are logged, including all error messages described in the error threshold.

Notice: Events with LOG_NOTICE and lower severity are logged, including all messages described in the warning threshold.

Info: Events with LOG_INFO and lower severity are logged, including all messages described in the notice threshold.

Logging Usage

Logging for the virtual instance is explained in the following sections:

C: Syslog

C Syslog: Example

Java: Log4j

Redirect logs to syslog: Example

Turn on logging: Example

Enable syslog appender: Example

Log messages go to file /var/log/messages.log in the virtual instance. After the log file reaches its specified limit (set by command limit log-file size), the contents of the log file are moved to a backup log file messages.log.prev. A new messages.log file starts to collect log messages. The default log file size for the log file and the backup log file is 5MB.

C: Syslog

For more information and examples on syslog APIs for C, see the GNU website.

C Syslog: Example

int main(int argc, char* argv[]){
	setlogmask (LOG_UPTO (LOG_DEBUG));
	openlog("myapp", LOG_CONS | LOG_PID | LOG_NDELAY, LOG_LOCAL1);
	syslog(LOG_DEBUG, "myapp_user_management.eval() processing a CLI message\n ");
}

Java: Log4j

For more information on the Log4j package regarding syslog, refer to syslog appender in the log4j documentation at: http://logging.apache.org/log4j/docs/documentation.html.

The logs for Cisco AXP APIs and libraries are redirected to stdout (via System.out.println).

To redirect logs to syslog, pipe the application's output to /bin/logger, as shown in the following example.

Redirect logs to syslog: Example

This example starts application MyAppMain, and uses a CLI plug-in. The stdout of the CLI plug-in and the output from System.out.println calls made by the application are redirected to syslog with a log level of LOG_INFO.

# java -cp 
./app_bin/myApp.jar:/cli_comm/:/usr/lib/java/localsocket.jar:/usr/lib/java/cli_distributio
n_vm.jar com.myApp.MyAppMain | /bin/logger -p info

Turn on logging: Example

Logger myLogger = Logger.getInstance("myapp");
myLogger.debug("myapp_user_management.eval() processing CLI message"); 

Enable syslog appender: Example

# To enable the syslog appender, set the root logger level to DEBUG and an appender to 
mySyslog.
log4j.rootLogger=DEBUG, mySyslog, myFile, myConsole

# appender definitions
log4j.appender.myConsole=org.apache.log4j.ConsoleAppender
log4j.appender.mySyslog=org.apache.log4j.net.SyslogAppender
log4j.appender.myFile = org.apache.log4j.RollingFileAppender

# myConsole config
log4j.appender.myConsole.layout=org.apache.log4j.PatternLayout
log4j.appender.myConsole.layout.ConversionPattern=%p %t %c - %m%n

# mySyslog config
log4j.appender.mySyslog.layout=org.apache.log4j.PatternLayout
log4j.appender.mySyslog.layout.ConversionPattern=%p %t %c - %m%n
log4j.appender.mySyslog.SyslogHost=localhost
log4j.appender.mySyslog.Facility=USER
log4j.appender.mySyslog.threshold=INFO

# myFile config
log4j.appender.myFile.File = example.log
log4j.appender.myFile.MaxFileSize = 100KB
log4j.appender.myFile.MaxBackupIndex=1
log4j.appender.myFile.layout=org.apache.log4j.PatternLayout
log4j.appender.myFile.layout.ConversionPattern=%p %t %c - %m%

Syslog Server Logging

To log messages to the host syslog server, use the syslog server feature. For example, messages can be logged from a workstation to the host syslog server.

Logging messages to the host syslog server for either C or Java code, is explained in the following sections.

C: Syslog Server

Java: Syslog Server

C: Syslog Server

In C programs, to direct log messages to the host syslog server, configure the syslog server config file, located at /etc/syslog.conf in the Cisco AXP Reference OS (AXPOS). This location name applies if the virtual instance's syslog server is syslogd. See the "Logging" section.

Redirect: Example

In the reference OS to redirect all log messages with facility= "local1", and priority value of "info" or higher, append the following line to the syslog server config file:

local1.info @myblade

Java: Syslog Server

In Java programs, to direct log messages to the host syslog server, edit the log4j.properties file in the log4j package (see the "Java: Log4j" section). For example,


log4j.rootLogger=DEBUG, mySyslog

# mySyslog config
log4j.appender.mySyslog.layout=org.apache.log4j.PatternLayout
log4j.appender.mySyslog.layout.ConversionPattern=%p %t %c - %m%n
log4j.appender.mySyslog.SyslogHost=myblade
/* Note the SyslogHost is "myblade" compared to "localhost" used by logging in the virtual 
instance */
log4j.appender.mySyslog.Facility=USER
log4j.appender.mySyslog.threshold=INFO

In addition to changing your application code, commands must be issued on the Cisco AXP service module to run the host syslog server:

blade# config terminal
(config)# syslog-server

To show the remote syslog server status:

blade# show syslog-server 
Syslog Server
Status:                 RUNNING

To configure the number of files for rotation and the file size:

blade# config terminal
(config)# syslog-server limit file-rotation 2 file-size 500

file-rotation has a value from 1 to 40, with a default of 10.

file-size has a value from 1 to 1000 MB, with a default of 20 MB.

Additional References

Configuring Log File Size Limits in the Cisco AXP User Guide.

Configuring System Log Levels in the Cisco AXP User Guide.

Configuring Remote Logging in the Cisco AXP User Guide.

Viewing Log and Core Files in the Cisco AXP User Guide.

Clearing Log and Core Files in the Cisco AXP User Guide.

Integrating Log Files into the Host OS

Files in /var/log directory of the guest OS are bundled using copy log bundle command and the files are copied to an external server.

To make sure that required log files are present in the /var/log directory, make log files in directory /yourdir available in the /var/log directory. When you bundle applications to be placed on an external server, create a log file in the /var/log directory in the prepackaged file directory using the touch command. Then use command ln -s to softlink the log file to the /yourdir directory.

Prepare Log Files for Bundling: Example

In this example an application called helloworld writes to a file called "helloworld.log".

The log file is located in directory "opt" and the purpose of this example is to export the log file with others in the "var/log" directory of the "build" directory.


Step 1 In directory build, create a directory var/log .

mkdir -p var/log

Step 2 In directory var/log, create a file helloworld.log .

cd build

touch var/log/helloworld.log

Step 3 Move to directory opt.

cd opt

Step 4 In directory opt create a symbolic link from source file helloworld.log in the directory var/log . The symbolic link is also named helloworld.log.

ln -s ../../var/log/helloworld.log helloworld.log


Note During packaging, the packaging tool and installer recreates the softlink created in Step 4.


Step 5 Package, install, and run your application.

The application writes to opt/helloworld.log and this log file is included with other log files when the files are bundled for export.


Rsync

rsync is a utility for synchronizing files between the virtual instance on the service module and the development workstation. rsync is dependent on the Open SSH Server which can be installed with the following RPMs: nspr, openssh-server, nss, tcp_wrappers and openssh.

During development you can move code from your development workstation to the virtual instance. Later, if you make changes to code on the virtual instance, you can synchronize with the code on the development workstation.

The Cisco AXP User Guide explains the prerequisite steps that must be completed before synchronization of files using the rsync utility. rsync is activated by using the sync command.

SUMMARY STEPS

1. app-service application-name

2. sync file url rsync:host_url direction [in|out] username username

3. exit

Rsync: Example

exec-appname> sync file url rsync://10.1.1.1/source/container direction in username john

In this example, the content of the virtual instance is overwritten by the directory /source/container located at the developer workstation (address 10.1.1.1). direction in specifies that content from the workstation is used as the master file.


Note If the development environment has a <source>/tmp directory on the workstation, and the rsync command is run, then files or directories in the <source>/tmp directory are not mounted in the /tmp directory of the virtual instance.



Note It is not recommended to synchronize from the workstation to the service module while the virtual instance is running—libraries such as libc may not work properly.


Updating System Libraries

To update system libraries on the service module, package the system libraries into your application and install the application.

If you need to update the libc library, also update libraries related to libc and libraries that are dependent upon libc. This ensures that there are no library version mismatches at runtime. For example, update associated libraries such as libpthread, libm, libdl, libresolv. To find out more about libraries related to libc, see the libc documentation: http://www.gnu.org/software/libc/.


System Performance Status Monitoring

To troubleshoot system performance issues, for example, disk performance, some tools are best used in the restricted shell rather than through the Cisco IOS CLI.

Status Monitoring Tools

vmstat reports information about processes, memory, paging, block IO, traps, and CPU activity. vmstat is available in Cisco AXP 1.0 and higher versions.

The following systtat commands are available from the restricted shell:

iostat—Reports CPU statistics and input/output statistics for devices, partitions and network file systems.

mpstat—Reports individual or combined processor-related statistics.

pidstat—Reports statistics for Linux tasks (processes) such as I/O, CPU, memory usage.