Table Of Contents
Logging and Troubleshooting
Troubleshooting Your Application
Troubleshooting the vi Editor
Logging
Syslog Server Logging
Integrating Log Files into the Host OS
Rsync
Updating System Libraries
System Performance Status Monitoring
Logging and Troubleshooting
Logging and Troubleshooting are described in the following sections:
•
Troubleshooting Your Application
•
Troubleshooting the vi Editor
•
Logging
•
Syslog Server Logging
•
Integrating Log Files into the Host OS
•
Rsync
•
System Performance Status Monitoring
Troubleshooting Your Application
For intermittent issues, Cisco AXP provides support for Linux commands such as vmstat and top in the shell so you can check if your application is using a lot of CPU capacity or it is using up a lot of RAM. Using up all the available RAM/ allocated CPU, may cause the application to function very slowly.
If you have configured your application for logging, the application log files are located in the application's virtual instance and contain messages written by your application. The log files contain error messages created by the system. The log file may contain evidence of why the virtual instance did not start and they are displayed one page at a time.
To debug your application perform the following basic debugging steps:
Step 1
Check that your application is running:
a.
app-service appname
, where appname is the name of your application
b.
show process
shows all processes sorted by process ID in ascending order.
Refer to the "Viewing Processes" section in Cisco AXP User Guide. If your application is part of a CLI plug-in application, the application must always be running.
Step 2
show state
If application health = "ALIVE", then your application is running, go to the end of the steps.
If the application state is not "online" go to Step 3.
If application state is "---" and your application is coded to provide its health status OR
if the application health is not = "ALIVE", then go to step 5.
Step 3
Perform the following steps:
a.
exit
exits to the Cisco AXP prompt
b.
Check the messages.log file or messages.log.prev file by entering one of the following commands:
–
show log name messages.log containing <application name> paged
–
show log name messages.log.prev containing <application name> paged
, where <application name> is the name of your application.
Note
If the application writes log files to directory /var/log or has a link pointing to /var/log, view the log files by entering:
show log name /var/log/messages.log paged
Step 4
(Optional) For systems using the AIM platform, the log level of the host system does not include INFO messages. (The messages reduce the amount of disk space used to store log files.) The logs may not give enough information about why the application fails to start in the virtual instance. To see the INFO messages displayed on the console, perform the following steps:
–
router# configure terminal
–
Enter the configuration commands, one per line. End with CNTL/Z.
–
router(config)# log console info
–
no shutdown
When an attempt is made to start the virtual instance, INFO messages are printed to the console for debugging purposes.
Step 5
Do this step if the application health is not = "ALIVE", or
the application state is "---" and your application is coded to provide its health status.
Check the messages.log file or messages.log.prev file by entering one of the following commands:
–
show log name messages.log paged
–
show log name messages.log.prev paged
, where <application name> is the name of your application.
Step 6
trace all
turns on tracing
Step 7
Run your application
Step 8
show trace buffer
Step 9
On the router, enter the following command:
show interfaces interface
Troubleshooting the vi Editor
The vi editor does not behave as expected with some terminals. If this is the case, run vi with parameter:
-T ansi
Logging
For logging inside a virtual instance, the syslog daemon syslogd, adds tracing or logging information to the file /var/log/messages.log. To configure remote logging, see "Configuring Remote Logging" in Cisco AXP User Guide
Logging is described in the following sections:
•
Log File Rotation
•
Logging Levels
•
Logging Usage
Log File Rotation
The Cisco AXP syslog supports log file rotation of two files:
•
messages.log (primary log file)
•
messages.log.prev (backup log file)
When the messages.log reaches a limit specified by the limit log-file size command, the file's contents are moved to backup file messages.log.prev and a new messages.log file is started. The default log file size is 20 MB.
You can use the command show logs to display the contents of all log files, and the command show log name to display the contents of a log file in the /var/log directory.
Logging Levels
Cisco AXP supports the following log levels:
•
LOG_EMERG: system is unusable
•
LOG_ALERT: action must be taken immediately
•
LOG_CRIT: critical condition
•
LOG_ERR: error condition
•
LOG_WARNING: warning condition
•
LOG_NOTICE: normal, but significant, condition
•
LOG_INFO: informational message
•
LOG_DEBUG: debug-level message
Out of all the log levels that are available to the application, the CLI reduces these to a more manageable number of four log levels.
You can set your log threshold to one of the following values:
•
Errors: Events with LOG_ERR and lower severity are logged, including LOG_EMERG, LOG_ALERT, and LOG_CRIT.
•
Warning: Events with LOG_WARNING and lower severity are logged, including all error messages described in the error threshold.
•
Notice: Events with LOG_NOTICE and lower severity are logged, including all messages described in the warning threshold.
•
Info: Events with LOG_INFO and lower severity are logged, including all messages described in the notice threshold.
Logging Usage
Logging for the virtual instance is explained in the following sections:
•
C: Syslog
•
C Syslog: Example
•
Java: Log4j
•
Redirect logs to syslog: Example
•
Turn on logging: Example
•
Enable syslog appender: Example
Log messages go to file /var/log/messages.log in the virtual instance. After the log file reaches its specified limit (set by command limit log-file size), the contents of the log file are moved to a backup log file messages.log.prev. A new messages.log file starts to collect log messages. The default log file size for the log file and the backup log file is 5MB.
C: Syslog
For more information and examples on syslog APIs for C, see the GNU website.
C Syslog: Example
int main(int argc, char* argv[]){
setlogmask (LOG_UPTO (LOG_DEBUG));
openlog("myapp", LOG_CONS | LOG_PID | LOG_NDELAY, LOG_LOCAL1);
syslog(LOG_DEBUG, "myapp_user_management.eval() processing a CLI message\n ");
Java: Log4j
For more information on the Log4j package regarding syslog, refer to syslog appender in the log4j documentation at: http://logging.apache.org/log4j/docs/documentation.html.
The logs for Cisco AXP APIs and libraries are redirected to stdout (via System.out.println).
To redirect logs to syslog, pipe the application's output to /bin/logger, as shown in the following example.
Redirect logs to syslog: Example
This example starts application MyAppMain, and uses a CLI plug-in. The stdout of the CLI plug-in and the output from System.out.println calls made by the application are redirected to syslog with a log level of LOG_INFO.
# java -cp
./app_bin/myApp.jar:/cli_comm/:/usr/lib/java/localsocket.jar:/usr/lib/java/cli_distributio
n_vm.jar com.myApp.MyAppMain | /bin/logger -p info
Turn on logging: Example
Logger myLogger = Logger.getInstance("myapp");
myLogger.debug("myapp_user_management.eval() processing CLI message");
Enable syslog appender: Example
# To enable the syslog appender, set the root logger level to DEBUG and an appender to
mySyslog.
log4j.rootLogger=DEBUG, mySyslog, myFile, myConsole
log4j.appender.myConsole=org.apache.log4j.ConsoleAppender
log4j.appender.mySyslog=org.apache.log4j.net.SyslogAppender
log4j.appender.myFile = org.apache.log4j.RollingFileAppender
log4j.appender.myConsole.layout=org.apache.log4j.PatternLayout
log4j.appender.myConsole.layout.ConversionPattern=%p %t %c - %m%n
log4j.appender.mySyslog.layout=org.apache.log4j.PatternLayout
log4j.appender.mySyslog.layout.ConversionPattern=%p %t %c - %m%n
log4j.appender.mySyslog.SyslogHost=localhost
log4j.appender.mySyslog.Facility=USER
log4j.appender.mySyslog.threshold=INFO
log4j.appender.myFile.File = example.log
log4j.appender.myFile.MaxFileSize = 100KB
log4j.appender.myFile.MaxBackupIndex=1
log4j.appender.myFile.layout=org.apache.log4j.PatternLayout
log4j.appender.myFile.layout.ConversionPattern=%p %t %c - %m%
Syslog Server Logging
To log messages to the host syslog server, use the syslog server feature. For example, messages can be logged from a workstation to the host syslog server.
Logging messages to the host syslog server for either C or Java code, is explained in the following sections.
•
C: Syslog Server
•
Java: Syslog Server
C: Syslog Server
In C programs, to direct log messages to the host syslog server, configure the syslog server config file, located at /etc/syslog.conf in the Cisco AXP Reference OS (AXPOS). This location name applies if the virtual instance's syslog server is syslogd. See the "Logging" section.
Redirect: Example
In the reference OS to redirect all log messages with facility= "local1", and priority value of "info" or higher, append the following line to the syslog server config file:
Java: Syslog Server
In Java programs, to direct log messages to the host syslog server, edit the log4j.properties file in the log4j package (see the "Java: Log4j" section). For example,
log4j.rootLogger=DEBUG, mySyslog
log4j.appender.mySyslog.layout=org.apache.log4j.PatternLayout
log4j.appender.mySyslog.layout.ConversionPattern=%p %t %c - %m%n
log4j.appender.mySyslog.SyslogHost=myblade
/* Note the SyslogHost is "myblade" compared to "localhost" used by logging in the virtual
instance */
log4j.appender.mySyslog.Facility=USER
log4j.appender.mySyslog.threshold=INFO
In addition to changing your application code, commands must be issued on the Cisco AXP service module to run the host syslog server:
To show the remote syslog server status:
blade# show syslog-server
To configure the number of files for rotation and the file size:
(config)# syslog-server limit file-rotation 2 file-size 500
file-rotation has a value from 1 to 40, with a default of 10.
file-size has a value from 1 to 1000 MB, with a default of 20 MB.
Additional References
•
Configuring Log File Size Limits in the Cisco AXP User Guide.
•
Configuring System Log Levels in the Cisco AXP User Guide.
•
Configuring Remote Logging in the Cisco AXP User Guide.
•
Viewing Log and Core Files in the Cisco AXP User Guide.
•
Clearing Log and Core Files in the Cisco AXP User Guide.
Integrating Log Files into the Host OS
Files in /var/log directory of the guest OS are bundled using copy log bundle command and the files are copied to an external server.
To make sure that required log files are present in the /var/log directory, make log files in directory /yourdir available in the /var/log directory. When you bundle applications to be placed on an external server, create a log file in the /var/log directory in the prepackaged file directory using the touch command. Then use command ln -s to softlink the log file to the /yourdir directory.
Prepare Log Files for Bundling: Example
In this example an application called helloworld writes to a file called "helloworld.log".
The log file is located in directory "opt" and the purpose of this example is to export the log file with others in the "var/log" directory of the "build" directory.
Step 1
In directory build, create a directory var/log .
Step 2
In directory var/log, create a file helloworld.log .
touch var/log/helloworld.log
Step 3
Move to directory opt.
Step 4
In directory opt create a symbolic link from source file helloworld.log in the directory var/log . The symbolic link is also named helloworld.log.
ln -s ../../var/log/helloworld.log helloworld.log
Note
During packaging, the packaging tool and installer recreates the softlink created in Step 4.
Step 5
Package, install, and run your application.
The application writes to opt/helloworld.log and this log file is included with other log files when the files are bundled for export.
Rsync
rsync is a utility for synchronizing files between the virtual instance on the service module and the development workstation. rsync is dependent on the Open SSH Server which can be installed with the following RPMs: nspr, openssh-server, nss, tcp_wrappers and openssh.
During development you can move code from your development workstation to the virtual instance. Later, if you make changes to code on the virtual instance, you can synchronize with the code on the development workstation.
The Cisco AXP User Guide explains the prerequisite steps that must be completed before synchronization of files using the rsync utility. rsync is activated by using the sync command.
SUMMARY STEPS
1.
app-service application-name
2.
sync file url rsync:host_url direction [in|out] username username
3.
exit
Rsync: Example
exec-appname> sync file url rsync://10.1.1.1/source/container direction in username john
In this example, the content of the virtual instance is overwritten by the directory /source/container located at the developer workstation (address 10.1.1.1). direction in specifies that content from the workstation is used as the master file.
Note
If the development environment has a <source>/tmp directory on the workstation, and the rsync command is run, then files or directories in the <source>/tmp directory are not mounted in the /tmp directory of the virtual instance.
Note
It is not recommended to synchronize from the workstation to the service module while the virtual instance is running—libraries such as libc may not work properly.
Updating System Libraries
To update system libraries on the service module, package the system libraries into your application and install the application.
If you need to update the libc library, also update libraries related to libc and libraries that are dependent upon libc. This ensures that there are no library version mismatches at runtime. For example, update associated libraries such as libpthread, libm, libdl, libresolv. To find out more about libraries related to libc, see the libc documentation: http://www.gnu.org/software/libc/.
System Performance Status Monitoring
To troubleshoot system performance issues, for example, disk performance, some tools are best used in the restricted shell rather than through the Cisco IOS CLI.
Status Monitoring Tools
vmstat reports information about processes, memory, paging, block IO, traps, and CPU activity. vmstat is available in Cisco AXP 1.0 and higher versions.
The following systtat commands are available from the restricted shell:
•
iostat—Reports CPU statistics and input/output statistics for devices, partitions and network file systems.
•
mpstat—Reports individual or combined processor-related statistics.
•
pidstat—Reports statistics for Linux tasks (processes) such as I/O, CPU, memory usage.