Cisco Application Velocity System User Guide (Software Version 5.0) - Appliance Administration [Cisco AVS 3100 Series Application Velocity System]

Table Of Contents

Appliance Administration

Starting and Stopping Software Components

Configuring the Node

Clock Synchronization

Disk Space Management

Diskmonitor.script Script Syntax

Diskmonitor.Script Contents

Core Files

Files Installed

Node Manager SSL Certificate

Replacing the Node Manager SSL Certificate

Importing the Node Manager SSL Certificate to the Management Console

Transparent Proxy Setup

Linux

Solaris

Appliance Administration

This chapter describes Cisco Application Velocity System (AVS) administrative details, and documents the important directories and files, in these sections:

•Starting and Stopping Software Components

•Configuring the Node

•Clock Synchronization

•Disk Space Management

•Core Files

•Files Installed

•Node Manager SSL Certificate

•Transparent Proxy Setup

Starting and Stopping Software Components

To manually start the application appliance Condenser, use the CLI command set condenser, like this:
velocity>set condenser start
Note The set condenser command applies only to the AVS 3120. On other hardware platforms, change to the $AVS_HOME/perfnode/bin directory and type the following command:
./fgnctl start

To start the application appliance Condenser in SSL mode, use the following command:
velocity>set condenser start ssl enable
Note The set condenser command applies only to the AVS 3120. On other hardware platforms, change to the $AVS_HOME/perfnode/bin directory and type the following command:
./fgnctl start ssl

To stop the application appliance Condenser, use the following command:
velocity>set condenser stop
Note The set condenser command applies only to the AVS 3120. On other hardware platforms, change to the $AVS_HOME/perfnode/bin directory and type the following command:
./fgnctl stop

To check if the Condenser server is running, look for a file named httpd.pid in the logs directory. The file contains the server's process id. The application appliance also writes out a standard Apache-style error log file, error_log, in the logs directory (or to a remote system, as configured by the set log-server command). Examine the tail of the error_log file for any errors. If there are none, the server started normally.

The SNMP Agent monitors the application appliance nodes and notifies the administrator if a node stops operating. The SNMP Agent can also provide statistics associated with an application appliance node via polling. The SNMP Agent supports third party network management applications using the SNMP protocol, such as HP OpenView, Sun NetManager, Sun Solstice Enterprise Manager, CA-Unicenter, IBM Tivoli, BMC Patrol, and others.

To start the SNMP Agent, change to the $AVS_HOME/perfnode/bin directory and type the following command:
./snmpctl start -d $AVS_HOME/perfnode
To stop the SNMP Agent, type the following command:
./snmpctl stop -d $AVS_HOME/perfnode
It is generally unnecessary to individually start and stop the Management Console, the Management Console database, and Node Manager components, since these are started automatically when the application appliance is rebooted. However the following commands can be used to manually start/stop these components:

•Management Console and database:
velocity>set console start|stop|restart
Note The set console command applies only to the AVS 3120 and AVS 3180 devices. On other hardware platforms, change to the $AVS_HOME/console/jboss*/bin directory and type the following command to start/stop the console:
./fgnconsolectl start|stop

And to start/stop the database, change to the $AVS_HOME/console/postgres/bin directory and type the following command:
./fgndbctl start|stop

The Management Console database must be started before the Management Console.

•Node Manager:
cd $AVS_HOME/perfnode/node_manager/bin;
./fgnnmctl start|stop
Note When stopping the Management Console, make sure that no users are logged in. Otherwise, the Management Console will not shut down correctly.

Configuring the Node

You will probably want to make changes to the application appliance node configuration file, fgn.conf, to suit your particular application. Rather than editing this file manually, you can use the Web Configurator feature of the Management Console to manage and edit the node configuration.

The Management Console lets you manage and edit the application appliance configuration file at both the cluster and node level. For details on using the Management Console to manage clusters of application appliance nodes, see the "Cluster Configuration" section; configuration of individual nodes is described in the "Configuring Individual Nodes" section.

For details on the configuration keywords, see "Configuration Reference."

Briefly, here is the procedure in the Management Console for making a typical configuration change, such as adding an Application Class to an application appliance node.

1. Use the Register Cluster command to create a new cluster (if there are no clusters yet).

2. Use the Register Node command to add the application appliance node to a cluster (if the application appliance node has not yet been added to a cluster).

3. Use the Import command to import the default configuration file from the newly installed application appliance node into the database.

4. Use the Application Class command to edit an existing Application Class, or create a new one by cloning an existing class or adding a new one. (To edit global parameters that are not within an Application Class, use the Global command.) Click Apply Changes to save your changes to the database.

5. Use the Publish command to write out the changed configuration file to the application appliance node.

6. Use the Cluster Control command to restart the node so that it reads the new configuration file.

Clock Synchronization

Many features of the application appliance depend on having the clocks synchronized between the application appliance and the origin servers. Be sure to synchronize all clocks after the software is installed. We recommend that you use NTP to do this. On the AVS 3120 and AVS 3180 devices, you can use the CLI command set ntp to configure an NTP server.

Disk Space Management

Note This section applies only to the Cisco AVS 3110 or older devices, including software-only products, that have been upgraded to AVS 5.0. On the AVS 3120 and AVS 3180 devices, disk space management is automatic and not configurable.

The application appliance automatically manages its disk space and deletes older log, cache, and database archive files as needed to maintain enough free disk space to operate.

The following files are involved in managing disk space:

•fgndiskmonitor: This executable monitors disk space usage.

•fgndirectorypruner: This executable moves or deletes unneeded directories.

•diskmonitor.script: This script configures the fgndiskmonitor activities.

•fdm.cron: This cron script runs the fgndiskmonitor program every 15 minutes.

Disk space management occurs in two stages. First, unneeded files are moved to the /usr/local/fineground/download directory, where the files remain accessible until total system disk usage reaches 50 GB. Then, the oldest files in the /download directory are deleted to bring the total disk usage down to 45 GB. Thus, in typical operation, disk usage for the /usr partition will vary from 45-50 GB.

Any files that you want to save, you should copy out of the /usr/local/fineground/download directory to another storage device on the network.

The fdm.cron script executes fgndiskmonitor every 15 minutes. The fgndiskmonitor program takes two file names (full paths) as arguments:

•the first is for writing informational messages to; it is set to /usr/local/fineground/appliance/logs/info.log

•the second is for writing errors to; it is set to /usr/local/fineground/appliance/logs/error.log

The following two sections describe topics concerning the diskmonitor.script file:

•Diskmonitor.script Script Syntax

•Diskmonitor.Script Contents

Diskmonitor.script Script Syntax

The fgndiskmonitor program reads each line in diskmonitor.script to determine what to do. Each script line has this format:
hour freq dir threshold cmd low high [args ...]
Essentially, this line sets a time at which monitoring is to occur, and a disk usage threshold (threshold) for a directory (dir), which, if exceeded, causes another program (cmd) to be executed; this other program (fgndirectorypruner) is passed low and high disk usage values and other arguments. It is expected to prune files from the high disk usage value down to the low disk usage value.

Table 3-1 describes each of these elements in detail.

Table 3-1 Diskmonitor Script Elements

Element

Description

hour

The integer hour of the day at which the disk monitoring service for a particular directory is to be started (using the 24-hour clock).

freq

The frequency in minutes (rounded to the nearest 15-minute multiple) with which the service is to be repeated.

If the current time of the day (measured in 15-minute blocks) matches the time specified in hour, and is a multiple of freq, then the disk monitoring service on dir is started.

dir

The directory to monitor for disk space usage.

threshold

The threshold disk space (in MB) that must be used by the filesystem of the directory dir, to trigger the execution of the cmd program. If the disk space used by the dir filesystem is less than threshold, the cmd program will not be executed. You can set threshold to zero to cause the cmd program to always run when monitoring occurs.

cmd

The full path name of a program that is to be executed if the disk usage threshold is exceeded. The cmd program is passed the following arguments: dir, low, high, informational log file name, error log file name, and zero or more optional arguments, args. The informational log and error log file names that are passed as arguments to the fgndiskmonitor program are passed on to the cmd program.

In the supplied diskmonitor.script, the cmd program is always fgndirectorypruner, which moves and/or deletes unneeded files. However, you can specify any other custom program that performs appropriate services and diagnostics.

low

The low disk usage level (in MB) for the monitored directory that the cmd program is allowed to prune disk space usage down to. When pruning is triggered, this value sets the directory disk usage level that should result after pruning is finished.

high

The high disk usage threshold (in MB) for the monitored directory that triggers the cmd program to prune files. When disk space usage for the directory exceeds the high level, the cmd program is expected to prune it down to the level specified by the low value. If disk space usage for the directory is lower than the high value, then the cmd program need not take action.

args

Optional additional arguments that are passed to the cmd program. Separate each argument from the next with a space.

The fgndirectorypruner program does the actual work of walking the specified directory and pruning files in an LRU order to reduce the disk space usage levels. It accepts the arguments dir, low, high, an informational log file name, an error log file name, and one or two additional arguments, args. The first additional argument is a keyword, delete or move, that specifies to delete or move pruned files. If it is move, it takes a second argument that is the target directory where files are to be moved. Files are moved and deleted in Least Recently Used (LRU) order.

The diskmonitor.script file can contain blank lines and comment lines (which begin with the # symbol) to improve readability.

Diskmonitor.Script Contents

This section describes the contents of the diskmonitor.script file that is supplied on the application appliance.

The first part of the file is general information. The work begins with the following line that monitors the appliance log file directory every minute (beginning at 12:00 am). When its size reaches 512 MB, least recently used files are moved to the /usr/local/fineground/download/var/log directory, until the directory disk usage is reduced to 256 MB.
#Move System Log files periodically
0 0 /var/log 0 /usr/local/fineground/appliance/bin/fgndirectorypruner
256 512 move /usr/local/fineground/download/var/log
The next line is commented out because it is no longer needed. The following line monitors the Node Manager log file directory every minute (beginning at 12:00 am) and when its size reaches 512 MB, least recently used files are moved to the /usr/local/fineground/download/perfnode/logs directory, until the directory disk usage is reduced to 256 MB.
#Move NodeManager Logs to download area
0 0 /usr/local/fineground/perfnode/node_manager/logs/ 0
/usr/local/fineground/appliance/bin/fgndirectorypruner 256 512 move
/usr/local/fineground/download/perfnode/logs
The following line monitors the Management Console database backup file directory every minute (beginning at 4:00 am) and when its size reaches 4096 MB, least recently used files are moved to the /usr/local/fineground/download/Database/backup directory, until the directory disk usage is reduced to 2048 MB.
#Move Database Backup Files to download area
4 0 /usr/local/fineground/console/postgres/backup 0
/usr/local/fineground/appliance/bin/fgndirectorypruner 2048 4096 move
/usr/local/fineground/download/Database/backup
The following line monitors the Postgres log file directory every minute (beginning at 4:00 am) and when its size reaches 4096 MB, least recently used files are moved to the /usr/local/fineground/download/postgres/log directory, until the directory disk usage is reduced to 2048 MB.
#Move Postgres Logs to download area
4 0 /usr/local/fineground/console/postgres/log/ 0 
/usr/local/fineground/appliance/bin/fgndirectorypruner 2048 4096 move 
/usr/local/fineground/download/postgres/log
The following line monitors the application appliance cache file directory every 60 minutes (beginning at 12:00 am) and when its size reaches 1280 MB, least recently used files are deleted, until the directory disk usage is reduced to 1024 MB.
#Delete Condenser Cache files periodically
0 60 /usr/local/fineground/perfnode/cache/DiskCache 0
/usr/local/fineground/appliance/bin/fgndirectorypruner 1024 1280 delete
The following line monitors another cache file directory every 60 minutes (beginning at 12:00 am) and when its size reaches 1280 MB, least recently used files are deleted, until the directory disk usage is reduced to 1024 MB.
#0 60 /usr/local/fineground/perfnode/cache/ausr 0 
/usr/local/fineground/appliance/bin/fgndirectorypruner 1024 1280 delete
The following line monitors the /download directory every 60 minutes (beginning at 12:00 am) and when its size reaches 60,000 MB, least recently used files are deleted, until the directory disk usage is reduced to 30,000 MB.
#Finally Delete files from Download area
0 60 /usr/local/fineground/download 0
/usr/local/fineground/appliance/bin/fgndirectorypruner 30000 60000 delete
The following line monitors the core directory every 60 minutes (beginning at 2:00 am) and when its size reaches 7,000 MB, least recently used files are deleted, until the directory disk usage is reduced to 4,000 MB.
#Manage the core area
#2 60 /usr/local/fineground/other-image/core/ 0 
/usr/local/fineground/appliance/bin/fgndirectorypruner 4000 7000 delete
Management Console log files are not handled in diskmonitor.script, but are instead handled in the fdm.cron script. The following line in that script monitors the Management Console log file directory every minute (beginning at 12:00 am) and when its size reaches 512 MB, least recently used files are moved to the /usr/local/fineground/download/console/logs directory, until the directory disk usage is reduced to 256 MB.
#Move Console Logs to download area
0  0 * * * root $HOME/bin/fgndirectorypruner 
/usr/local/fineground/console/jboss-3.0.1_tomcat-4.0.4/server/default/log/ 256 512   /dev/null /dev/null move 
/usr/local/fineground/download/console/logs "*.log*"
Core Files

The application appliance will store core files if the httpd process generates them. They are stored in the $AVS_HOME/perfnode/logs/coredump directory by default. You can change the location of this directory by using the CoreDumpDirectory keyword. Note that this core directory is managed by the directory pruner process, which will begin deleting core files once the total amount of disk space reaches 2 GB.

When a core file is generated, the following message is logged in the $AVS_HOME/perfnode/logs/error_log file:
[Wed Sep  8 13:39:40 2004] [notice] child pid 3790 exit signal Segmentation fault (11), 
possible coredump in $AVS_HOME/perfnode/logs/core
Files Installed

The product files are installed into the following directories:

•The Condenser and AppScope files are installed into several subdirectories in $AVS_HOME/perfnode

The $AVS_HOME/perfnode/logs directory contains log information from the Condenser acceleration software, Management Console, and file management processing.

•Management Console files are installed into several subdirectories in $AVS_HOME/console

•Application Velocity System files are installed in $AVS_HOME/appliance

•Files that you may want to back up are periodically moved to $AVS_HOME/download, and then eventually deleted. The files include logs and database backup files. Any files that you want to save, you should copy out of this directory to another storage device on the network. For more information on how the /download directory gets populated and managed, see the "Disk Space Management" section.

Note The previous statement applies only to the Cisco AVS 3110 or older devices, including software-only products, that have been upgraded to AVS 5.0.

Of particular interest are the subdirectories under $AVS_HOME/perfnode, which include the following:

•bin—for executable files

•conf—for configuration files

•logs—for log files

The following tables describe the important files in each of these subdirectories.

Table 3-2 bin Files

Filename

Description

fgncache

Cache pruning executable

fgnctl

Script that starts the server

httpd

Server executable

snmpctl

Script that starts the Master SNMP agent daemon as well as the SNMP agent daemon

Table 3-3 conf Files

Filename

Description

fgn.conf

Configuration file (for details on the configuration parameters found in this file, refer to "Configuration Reference")

fgnsnmpd.conf

SNMP agent daemon configuration

httpd.conf

HTTP server configuration file; it includes fgn.conf by reference

magentd.conf

Master SNMP agent daemon configuration

mimetypes.conf

Lists MIME types that are excluded from condensation. In other words, if the application appliance receives a response with one of these MIME types from the origin server, that response will not be condensed.

useragent.conf

Lists user agent identifiers of supported browsers

Table 3-4 log Files

Filename

Description

access_log

The optional Apache access_log file that contains an entry for every request to the server. For more details about this log file, see the "Access_log" section.

error_log

The standard Apache error_log file that contains server errors. For more details about this log file, see the "Error_log" section.

FgnStatLog

A link to the current FgnStatLog file.

FgnStatLog.nnn

The log file that contains an entry for every request to the server and is used by the Management Console for statistical analysis. For more details about this log file, see the "FgnStatLog" section.

httpd.pid

Contains the server process id when it is running.

Node Manager SSL Certificate

The node manager ships with a default SSL certificate. You can replace the certificate with your own SSL certificate.

Replacing the Node Manager SSL Certificate

To install a SSL certificate for the node manager, perform the following steps.

1. Copy the PEM-encoded X.509 certificate to the node manager configuration directory:
$AVS_HOME/perfnode/node_manager/conf/ssl.crt/server.crt

2. If the server private key is not combined with the certificate, copy it to this file:
$AVS_HOME/perfnode/node_manager/conf/ssl.crt/server.key

3. Restart the node manager by using the following commands:
cd $AVS_HOME/perfnode/node_manager/bin;
./fgnnmctl stop;
./fgnnmctl start
Importing the Node Manager SSL Certificate to the Management Console

If you use SSL certificates issued by certification authorities such as VeriSign Inc., or RSA Data Security Inc., no change is needed on the Management Console. Otherwise, you must import your certificates to the Management Console using the following steps.

1. Copy the PEM-encoded X.509 certificate (server.crt) to a temporary directory, such as /tmp, on the management station machine.

2. Change directory to the Management Console configuration directory and make the file node_manager_cacert writable as follows:
cd $AVS_HOME/console/jboss-3.0.1_tomcat-4.0.4/server/default/deploy/ 
condenser-mbeans.sar/META-INF
chmod +w ./node_manager_cacert
3. Make sure the java utility keytool is in the path.

4. Import the certificate as follows:
keytool -import -trustcacerts -alias mycertificate -file /tmp/server.crt -keystore 
./node_manager_cacert
5. When prompted for the keystore password, enter fineground and press the Enter key.

6. Type yes when prompted with the question:
Trust this certificate? [no]:
7. Restart the node manager and Management Console (see the "Starting and Stopping Software Components" section).

Transparent Proxy Setup

This section discusses how to set up a single application appliance as a transparent proxy. For multiple application appliance configuration, you must use a load balancer.

The ideal way to set up a single application appliance as a transparent proxy is to use a layer 4 web switch to redirect all port 80 traffic to the application appliance on the port it is listening on. Configure the load balancer to failover directly to the origin server in case the application appliance fails.

If a layer 4 switch is not available (for example, for testing), then the second best option is to change the DNS server configuration or client host file to point to the application appliance for the host name of interest, and configure the application appliance to listen on port 80 in the httpd.conf file. A destination mapping entry may also be necessary.

If configuring the application appliance to listen on port 80 is not acceptable, then you can use a tool such as ipchains or iptables on Linux, or ipfilter on Solaris, to set up the application appliance as a transparent proxy. Note that when using packet filtering tools such as these, performance may be degraded. For details on using these tools, see the following sections:

•Linux

•Solaris

Linux

Under Linux, to set up the application appliance as a transparent proxy for the origin servers, you use the native Linux transparent proxy capability. A transparent proxy takes all incoming traffic for the HTTP port and redirects it through the application appliance.

You use the Linux ipchains tool to configure a set of packet redirection rules in the Linux system. This enables the system to send all non-SSL and SSL traffic through the application appliance to the origin server, and act as a transparent proxy for port 80 (World Wide Web) for non-SSL traffic and for port 443 for SSL traffic.

Here is a sample ipchains rule set:
/sbin/ipchains --append input --protocol TCP --destination 127.0.0.1/32 www -j ACCEPT
/sbin/ipchains --append input --protocol TCP --destination any/0 www -j REDIRECT 8080
/sbin/ipchains --append input --protocol TCP --destination any/0 443 -j REDIRECT 8443
In the example above, all traffic arriving at port 80 is redirected to port 8080. The application appliance is set to listen on port 8080 for incoming HTTP traffic. And all traffic arriving at port 443 is redirected to port 8443.

Use the Linux ipchains tool to enable SSL pass-through for a single application appliance deployment. For a multiple application appliance deployment, enable SSL pass-through on the load balancer.

For RedHat Linux 7.3 and later releases, ipchains has been replaced by iptables. Here is an example shell script that shows how to use iptables to configure transparent proxying:
#!/bin/sh
# enable iptables
echo 1 > /proc/sys/net/ipv4/ip_forward 
# delete any existing rules
/sbin/iptables -t nat -F
# redirect any requests from Port 80 to Port 8080 (the application appliance http port)
/sbin/iptables -t nat -A PREROUTING -p tcp --dport 80 -j REDIRECT --to-ports 8080
# redirect any requests from Port 443 to Port 8443 (the application appliance https port)
/sbin/iptables -t nat -A PREROUTING -p tcp --dport 443 -j REDIRECT --to-ports 8443
# list the rules
/sbin/iptables -t nat -L
Solaris

Note This section applies only to upgraded software running on non-Cisco hardware platforms that use Solaris.

Under Solaris, to set up the application appliance as a transparent proxy for the origin servers, you use the Solaris ipfilter tool. (The ipfilter tool is not part of the Solaris distribution.) A transparent proxy takes all incoming traffic for the HTTP and HTTPS ports and redirects it through the Condenser.

You use the Solaris ipnat tool to configure a set of packet redirection rules in the Solaris system. This enables the system to send all non-SSL and SSL traffic through the application appliance to the origin server, and act as a transparent proxy for port 80 (World Wide Web) for non-SSL traffic and for port 443 for SSL traffic. Here is a sample rule set implemented in a shell script:
hostip=10.0.0.100
IPFRULES=/tmp/ipf.conf.$$
IPNATRULES=/tmp/ipnat.rules.$$
echo "rdr hme0 0.0.0.0/0 port 80 -> ${hostip} port 8080" > ${IPNATRULES} 
echo "rdr hme0 0.0.0.0/0 port 443 -> ${hostip} port 8443" >> ${IPNATRULES} 
echo "map hme0 ${hostip}/32 -> 0/32 portmap tcp/udp auto" >> ${IPNATRULES} 
/usr/sbin/ipnat -f ${IPNATRULES}
In the example above, all traffic arriving at port 80 is redirected to port 8080. The application appliance is set to listen on port 8080 for incoming HTTP traffic. And all traffic arriving at port 443 is redirected to port 8443.

Use the Solaris ipnat tool to enable SSL pass-through for a single application appliance deployment. For a multiple application appliance deployment, enable SSL pass-through on the load balancer.