Table Of Contents

Monitoring GSS Performance

Monitoring GSS and GSSM Status

Monitoring the Online Status of GSS Devices from the CLI

Monitoring the Status of Your GSS Network from the CLI

Monitoring the Status of the Boomerang Server on Your GSS

Monitoring the Status of the DNS Server on Your GSS

Monitoring the Status of Keepalives on Your GSS

Monitoring GSS Device Status from the Primary GSSM GUI

Monitoring GSSM Database Status

Monitoring the Database Status

Validating Database Records

Creating a Database Validation Report

Monitoring Global Load-Balancing Status

Monitoring Answer Hit Counts

Monitoring Answer Keepalive Statistics

Monitoring Answer Status

Monitoring DNS Rule Statistics

Monitoring Domain Statistics

Monitoring Source Address Statistics

Monitoring Global Statistics

Viewing Log Files

Understanding GSS Logging Levels

Viewing Device Logs from the CLI

Viewing the gss.log File from the CLI

Viewing Subsystem Log Files from the CLI

Rotating Existing Log Files from the CLI

Viewing System Logs from the Primary GSSM GUI

Viewing System Logs from the GUI

Purging System Log Messages from the GUI

System Log Messages

Monitoring GSS Performance

---

The GSS software features a number of tools for monitoring the status of your GSS devices and of global load balancing on your GSS network. These include CLI-based commands for determining the status of your GSSs, GSSMs (primary and standby), and the embedded GSS database. In addition, the primary GSSM GUI contains pages that display the status of global server load balancing activity. For example, tabulating answer and DNS rule hit counts.

This chapter contains the following major sections:

•Monitoring GSS and GSSM Status

•Monitoring GSSM Database Status

•Monitoring Global Load-Balancing Status

•Viewing Log Files

Monitoring GSS and GSSM Status

You can easily monitor the status of your GSSs and GSSMs from both the CLI and the GSSM GUI.

This section includes the following procedures:

•Monitoring the Online Status of GSS Devices from the CLI

•Monitoring the Status of Your GSS Network from the CLI

•Monitoring GSS Device Status from the Primary GSSM GUI

Monitoring the Online Status of GSS Devices from the CLI

Use the gss command to display the online status and resource usage of your GSS servers.

To monitor the status of a GSS device from the CLI:

1. Log on to the CLI following the instructions in Chapter 2, Setting Up Your GSS, the "Accessing the GSS CLI" section. The GSS CLI prompt appears.

2. Enable privileged EXEC mode. For example:

gssm1.yourdomain.com> enable

gssm1.yourdomain.com# 

3. Use the gss command to display the current running status of the GSS device that you have logged on to. For example:

gss1.yourdomain.com# gss status verbose

Cisco GSS - 1.1(0.0.1) - Development build GSSM - primary [Mon Sep 
15 13:16:38 UTC 2003]

Normal Operation [runmode = 5]

 %CPU  START  SERVER

  0.0  Jun17  Boomerang             

  0.0  Jun17  Config Agent          

  0.0  Jun17  Config Server         

  0.0  Jun17  DNS Server            

  0.0  Jun17  Database              

  0.0  Jun17  GUI Server            

  0.0  Jun17  Keepalive Engine      

  0.0  Jun17  Node Manager          

  0.0  Jun17  Syslog                

  0.0  Jun17  Web Server            

  ---    ---  SNMP                  [DISABLED]

Monitoring the Status of Your GSS Network from the CLI

Use the show statistics command to view the status of any request routing and load balancing component on your GSS devices, including answers, keepalives, domains, and DNS rules. Refer to the Cisco Global Site Selector Command Reference for detailed information about the show statistics command.

The following sections provide instructions about using and interpreting the output of the various show statistics command options.

•Monitoring the Status of the Boomerang Server on Your GSS

•Monitoring the Status of the DNS Server on Your GSS

•Monitoring the Status of Keepalives on Your GSS

---

Note If you specify the show statistics command after issuing either the gss start command or the reload command, the GSS device can take approximately one minute before the command can take affect and display the requested statistics.

---

Monitoring the Status of the Boomerang Server on Your GSS

The boomerang server is a server load-balancing component of the GSS that uses calculations of network delay provided by DNS races between content routing agents (CRAs) to determine which server is best able to respond to a given request.

Use the show statistics boomerang command option to view boomerang activity such as DNS races on your GSS device on a domain-by-domain or on a global basis. Refer to the Cisco Global Site Selector Command Reference for detailed information about the show statistics boomerang command.

To view DNS race statistics:

1. Log on to the CLI following the instructions in Chapter 2, Setting Up Your GSS, the "Accessing the GSS CLI" section. The GSS CLI prompt appears.

2. Enable privileged EXEC mode. For example:

gssm1.yourdomain.com> enable

gssm1.yourdomain.com# 

3. Use the show statistics boomerang command to display current boomerang server statistics for a particular domain, or across all domains managed by your GSS. For example:

gss1.yourdomain.com# show statistics boomerang global

Boomerang global statistics:

        Total races: 24

Monitoring the Status of the DNS Server on Your GSS

The DNS server component tracks all DNS-related traffic to and from your GSS device, including information about DNS queries received, responses sent, queries dropped and forwarded, and so on.

Using the show statistics dns command option, you can view DNS statistics with regard to your GSS request routing and server load-balancing components such as DNS rules, answers, answer groups, domains, domain lists, source addresses, and source address groups. Refer to the Cisco Global Site Selector Command Reference for detailed information about the show statistics dns command.

To view DNS statistics:

1. Log on to the CLI following the instructions in Chapter 2, Setting Up Your GSS, the "Accessing the GSS CLI" section. The GSS CLI prompt appears.

2. Enable privileged EXEC mode. For example:

gssm1.yourdomain.com> enable

gssm1.yourdomain.com# 

3. Use the show statistics dns command to display statistics from the domain name server (DNS) component of the GSS. For example:

gss1.yourdomain.com# show statistics dns answer

Answer          Type Total Hits  1-Min   5-Min   30-Min  4-Hr    

-----------------------------------------------------------------

192.168.1.80    VIP  0           0       0       0       0       

1.1.5.160       VIP  0           0       0       0       0       

192.168.1.24    VIP  0           0       0       0       0       

192.168.1.245   VIP  0           0       0       0       0 

Monitoring the Status of Keepalives on Your GSS

The keepalive engine on your GSS device monitors the online status of keepalive objects across your GSS network.

Using the show statistics keepalive command option, you can view statistics about the health of your GSS keepalives globally or by keepalive type. Refer to the Cisco Global Site Selector Command Reference for detailed information about the show statistics keepalive command.

To view keepalive statistics:

1. Log on to the CLI following the instructions in Chapter 2, Setting Up Your GSS, the "Accessing the GSS CLI" section. The GSS CLI prompt appears.

2. Enable privileged EXEC mode. For example:

gssm1.yourdomain.com> enable

gssm1.yourdomain.com#

3. Use the show statistics keepalive command to display current keepalive engine statistics for your GSS network. You can view statistics for all keepalive types on your network, or limit statistics to a particular keepalive type such as ICMP, HTTP HEAD, TCP, KAL-AP, or CRA.

For example:

gss1.yourdomain.com# show statistics keepalive tcp all

IP: 192.168.50.41  

Keepalive => 192.168.50.41

Destination Port: 80

Status: ONLINE

Packets Sent:                93188

Packets Received:            69891

Positive Probe:              23297

Negative Probe:                  0

Transitions:                     1

GID: 105 LID: 5

Monitoring GSS Device Status from the Primary GSSM GUI

To monitor the status of your GSS devices from the primary GSSM GUI:

1. From the primary GSSM GUI, click the Resources tab.

2. Click the Global Site Selectors navigation link. The Global Site Selector list page appears.

3. Click the Modify GSS icon for the GSS or GSSM that you wish to monitor. The device type (GSS or GSSM) appears in the Node Services column.

The Global Site Selectors details page appears, displaying configuration and status information about the device at the bottom of the page including:

–Status—Online status

–Version—Software version currently loaded on the device

–Node services—Current role of the device (GSS, primary or standby GSSM, or both)

–IP address—Network address of the device

–Hostname—Network host name of the device

–MAC—Machine address of the device

4. Click Cancel to return to the Global Site Selectors list page.

Monitoring GSSM Database Status

The GSS software includes a number of CLI commands that you can use to monitor the status of the GSSM database and its contents. This section includes the following procedures:

•Monitoring the Database Status

•Validating Database Records

•Creating a Database Validation Report

Monitoring the Database Status

To verify that the GSS database on the GSSM is functioning properly:

1. Log on to the CLI following the instructions in Chapter 2, Setting Up Your GSS, the "Accessing the GSS CLI" section. The GSS CLI prompt appears.

2. Enable privileged EXEC mode. For example:

gssm1.yourdomain.com> enable

gssm1.yourdomain.com# 

3. Use the gssm database status command to display the current running status of the GSS device that you have logged on to. For example:

gss1.yourdomain.com# gssm database status

GSSM database is running.

Validating Database Records

To validate the records in your GSSM database:

1. Log on to the CLI following the instructions in Chapter 2, Setting Up Your GSS, the "Accessing the GSS CLI" section. The GSS CLI prompt appears.

2. Enable privileged EXEC mode. For example:

gssm1.yourdomain.com> enable

gssm1.yourdomain.com# 

3. Use the gssm database validate command to validate the content of your GSSM database. For example:

gss1.yourdomain.com# gssm database validate

GSSM database passed validation.

Creating a Database Validation Report

Should you encounter problems while attempting to validate your GSSM database, you can generate a report, called validation.log, that details which database records failed validation.

To generate a database validation report:

1. Log on to the CLI, following the instructions in Chapter 2, Setting Up Your GSS, the "Accessing the GSS CLI" section. The GSS CLI prompt appears.

2. Enable privileged EXEC mode. For example:

gssm1.yourdomain.com> enable

gssm1.yourdomain.com# 

3. Use the gssm database report command to generate a validation report on the content of your GSSM database. For example:

gss1.yourdomain.com# gssm database report

GSSM database validation report written to validation.log.

4. Use the type command to view the contents of your validation report. For example:

gss1.yourdomain.com# type validation.log

validation.log

Start logging at Thu Aug 28 19:17:21 GMT+00:00 2003

- storeAdmin Validating ... Thu Aug 28 19:17:23 GMT+00:00 2003 -

- ObjectId  Object_Name.Field_Name   Description -

Validating FactoryInfo

Validating answerElement

Validating answerGroup

 70  answerGroup.OwnerId  Many-To-One List

Validating CachingConfig

Validating ClusterConfig

Validating CmdControl

Validating CmdPurgeRd

Validating CmdUpdate

Validating ConfigProperty

Validating Customer

Validating DistTree

Validating DnsRule

Validating DomainElement

Validating DomainGroup

Validating ENodeConfig

Validating ENodeStatus

Validating KeepAliveConfig

Validating KeepAlive

Validating Location

Validating OrderedanswerGroup

Validating Owner

Validating Region

Validating RequestHandler

Validating RoutedDomain

Validating RoutingConfig

Validating RrConfig

Validating RrStatus

Validating SNodeConfig

Validating SourceAddressElement

Validating SourceAddressGroup

Validating SpInfo

Validating SystemConfig

Validating UpdateInfo

Validating UserConfig

Validating VirtualCDN

Validating WlpanswerElement

Validating User Validations

End of file validation.log

Monitoring Global Load-Balancing Status

From the primary GSSM GUI, you can monitor the status of global load balancing on your GSS network using a variety of features that filter and condense GSS traffic and statistics.

This section includes the following procedures:

•Monitoring Answer Hit Counts

•Monitoring Answer Keepalive Statistics

•Monitoring Answer Status

•Monitoring DNS Rule Statistics

•Monitoring Domain Statistics

•Monitoring Source Address Statistics

•Monitoring Global Statistics

Monitoring Answer Hit Counts

The answer hit counts feature of the primary GSSM GUI provides you with an overview of your GSS answer resources and the number of times that user requests have been directed to each answer device. Looking at answer hit counts is one way to judge how well your GSS resources are being used in responding to user requests.

To view the number of hits recorded by each of your GSS answers:

1. From the primary GSSM GUI, click the Monitoring tab.

2. Click the Answers navigation link.

3. Click the Answer Hit Counts navigation link (located under the Contents table of contents).

4. . The Answer Hit Counts list page appears (Figure 10-1).

Figure 10-1 Answer Hit Counts List Page

Table 10-1 describes the fields on the Answer Hit Counts list page.

Table 10-1 Field Descriptions for Answer Hit Counts List Page

Field	Description
Answer	IP address of the answer device
Name	Name assigned to the answer using the primary GSSM GUI
Type	Type of answer: VIP (virtual IP address), NS (name server), or CRA (content routing agent)
Location	GSS network location into which the answer has been grouped
Name of the GSSM or GSS	Number of requests directed to the answer by each GSS device

5. Click the column header of any of the displayed columns to sort your answers by a particular property.

Monitoring Answer Keepalive Statistics

The answer keepalive statistics feature of the primary GSSM GUI provides you with an overview of the online status of your GSS answer resources. For each answer configured on your GSS, the answer keepalive statistics feature displays the number of keepalive probes that have been directed to that answer by the primary and the standby GSSM, as well as information about how that keepalive probe was handled. If a large number of keepalive probes are being rejected or are encountering transition conditions, the answer may be offline or may be having problems staying online.

To view the online status of each of your GSS answers:

1. From the primary GSSM GUI, click the Monitoring tab.

2. Click the Answers navigation link.

3. Click the Answer KeepAlive Statistics navigation link (located under the Contents table of contents). The Answer KeepAlive Statistics list page appears (Figure 10-2).

Figure 10-2 Answer Keepalive Statistics List Page

Table 10-2 describes the fields on the Answer KeepAlive Statistics list page.

Table 10-2 Field Descriptions for Answer Keepalive Statistics List Page

Field	Description
Answer	IP address of the answer device being probed
Type	Type of answer: VIP (virtual IP address), NS (name server), or CRA (content routing agent)
Name	Name assigned to the answer using the primary GSSM GUI
Keepalive	The address assigned to the remote device, CRA, or name server that the GSS is to forward requests
Method	The keepalive method used by the answer: VIP (virtual IP address), NS (name server), or CRA (content routing agent)
Location	GSS network location into which the answer has been grouped
Name of the GSSM or GSS	Number of keepalive probes directed to the answer by each GSS device, as well as a record of how those probes were handled. Statistics are presented in the following order: •Keepalive packets sent—Total number of keepalive probes sent to the answer by each GSS on the network •Keepalive packets received—Total number of keepalive probes returned from the answer •Keepalive positive probe count—Total number of keepalive probes received to which a positive (OK) response was returned •Keepalive negative probe count—Total number of keepalive probes received to which a negative response was returned •Keepalive transition count—Total number of keepalive probe transitions (for example, from the INIT to the ONLINE state) experienced by the keepalive

4. Click the column header of any of the displayed columns to sort your answers by a particular property.

Monitoring Answer Status

The answer status feature of the primary GSSM GUI provides you with an overview of your GSS answer resources and their online status. Answers can be sorted by IP address, name, type, location, or online status according to a particular device.

To view the status of your GSS answers:

1. From the primary GSSM GUI, click the Monitoring tab.

2. Click the Answers navigation link.

3. Click the Answer Status navigation link (located under the Contents table of contents). The Answer Status list page appears (Figure 10-3).

Figure 10-3 Answer Status List Page

Table 10-3 describes the fields on the Answer Status list page.

Table 10-3 Field Descriptions for Answer Status List Page

Field	Description
Answer	IP address of the answer device
Name	Name assigned to the answer using the primary GSSMGUI
Type	Type of answer: VIP (virtual IP address), NS (name server), or CRA (content routing agent)
Location	GSS network location into which the answer has been grouped
Name of the GSSM or GSS	Online status of the answer according to the named device

4. Click the column header of any of the displayed columns to sort your answers by a particular property.

Monitoring DNS Rule Statistics

The DNS rule statistics feature of the primary GSSM GUI provides you with an overview of your global load-balancing rules, as well as information about how many queries were processed by each rule and how many of those processed queries were successfully matched with answers.

To view the status of your DNS rules:

1. From the primary GSSM GUI, click the Monitoring tab.

2. Click the DNS Rules navigation link. The DNS Rule Statistics list page appears (Figure 10-4).

Figure 10-4 DNS Rule Statistics List Page

Table 10-4 describes the fields on the DNS Rule Statistics list page.

Table 10-4 Field Descriptions for DNS Rule Statistics List Page

Field	Description
Name	Name assigned to the answer using the primary GSSM.
Owner	GSS owner to which the DNS rule has been assigned.
Name of the GSSM or GSS	Total hit count and successful hit count for the DNS rule from the listed GSS device. Refer to the legend that appears below the listed DNS rules if you are confused about which number represents total hits and which represents successful requests served.

3. Click the column header of any of the displayed columns to sort your DNS rules by a particular property.

Monitoring Domain Statistics

The domain statistics feature of the primary GSSM GUI provides you with an overview of the hosted domains that your GSS is serving, as well as information about how many queries were directed to each domain by your DNS rules. The domain hit counts feature tracks the traffic directed to individual domains, not GSS domain lists, which may include one or more domains.

To view the status of your hosted domains:

1. From the primary GSSM GUI, click the Monitoring tab.

2. Click the Domains navigation link. The Domain Hit Counts list page appears (Figure 10-5).

Figure 10-5 Domain Hit Counts List Page

Table 10-5 describes the fields on the Domain Hit Counts list page.

Table 10-5 Field Descriptions for Domain Statistics List Page

Field	Description
Domain	DNS domains for which your GSS is responsible; these are the domains contained in your domain lists.
Name of the GSSM or GSS	Total number of requests for the listed domain from each GSS device

3. Click the column header of any of the displayed columns to sort the listed domains by a particular property.

Monitoring Source Address Statistics

The source address statistics feature of the primary GSSM GUI provides you with an overview of incoming requests received by each of your source addresses (that is, those addresses from which DNS queries to your GSS originate) from each of your GSS devices. The source address hit counts feature tracks requests from individual address blocks, not from GSS source address lists, which may contain one or more address blocks.

To view the statistics for your source address lists:

1. From the primary GSSM GUI, click the Monitoring tab.

2. Click the Source Addresses navigation link. The Source Address Lists Statistics list page appears (Figure 10-6).

Figure 10-6 Source Address List Statistics List Page

Table 10-6 describes the fields on the Source Address Lists Statistics list page.

Table 10-6 Field Descriptions for Source Address Statistics List Page

Field	Description
Source Address Block	Address or range of addresses from which DNS queries originate. Source address blocks make up GSS source address lists.
Name of the GSSM or GSS	Total number of requests received by the listed GSS device from each address or address block.

3. Click the column header of any of the displayed columns to sort the listed domains by a particular property.

Monitoring Global Statistics

The global statistics feature of the primary GSSM GUI provides you with an overview of your GSS network, providing average statistics for DNS requests received by each GSS device and keepalive messages sent to your answers, as well as the online status of each GSS device.

To view the status of your GSS network:

1. From the primary GSSM GUI, click the Monitoring tab.

2. Click the Global navigation link. The Global Statistics list page (Figure 10-7) appears.

Figure 10-7 Global Statistics List Page

Table 10-7 describes the fields on the Global Statistics list page.

Table 10-7 Field Descriptions for Global Statistics List Page

Field	Description
GSS Status	Online status of each GSS device
Unmatched DNS Queries	Total number of DNS queries received by each listed device for which no answer could be found
DNS Queries/sec	Average number of DNS queries received each second by each listed GSS device
Keepalive Probes/sec	Average number of keepalive probes received by each listed GSS device each second

3. Click the column header of any of the displayed columns to sort the listed domains by a particular property.

Viewing Log Files

The GSS maintains logged records for a wide range of GSS network activity in the gss.log file as well as through the system logs feature of the GSSM.

The following sections help you audit logged information about your GSS devices.

•Understanding GSS Logging Levels

•Viewing Device Logs from the CLI

•Viewing System Logs from the Primary GSSM GUI

Understanding GSS Logging Levels

The GSS employs eight separate logging levels to identify the wide range of critical and noncritical logged events that may occur on a GSS device. Table 10-8 lists these different logging levels and explains their meanings.

Table 10-8 GSS Logging Levels 

Level Number	Level Name	Description
0	Emergencies	The GSS has become unusable: for example, the device is shutting down and cannot be restarted, or it has experienced a hardware failure.
1	Alerts	The GSS requires immediate attention: for example, one of the GSS servers is not running.
2	Critical	The GSS has encountered a critical condition that requires attention: for example, being unable to connect to the primary GSSM and not having a configuration snapshot to use in the meantime.
3	Errors	The GSS has encountered an error condition that requires prompt attention but still enables the device to function: for example, running out of memory.
4	Warnings	The GSS has encountered an error condition that requires attention but is not interfering with the operation of the GSS device: for example, losing contact with the primary GSSM when a local configuration snapshot exists.
5	Notifications	The GSS has encountered a nonerror condition that should be brought to the administrator's attention: for example, a software upgrade.
6	Information	Messages at this level are normal operational messages for the GSS device, such as status or configuration changes.
7	Debug	Messages at this level (such as detailed information about DNS request or keepalive handling, specific code path tracking, and so on) are intended for use by technical support personnel.

Viewing Device Logs from the CLI

Each GSS device contains a variety of log files that retain records of both GSS-related activity and the functioning of various GSS subsystems. You can access these log files using the CLI to troubleshoot problems or better understand the behavior of a GSS device.

This section includes the following procedures:

•Viewing the gss.log File from the CLI

•Viewing Subsystem Log Files from the CLI

•Rotating Existing Log Files from the CLI

Viewing the gss.log File from the CLI

The gss.log file pulls together information that may be of use to customers, such as keepalive, availability, and load statistics for GSS devices. This log file can be viewed from the CLI using the show logs command.

Refer to the Cisco Global Site Selector Command Reference for a list of the various log files that are displayed using the show logs command.

---

Note The show logs command outputs all logged information to your terminal session. This output may be quite large and exceed the buffer size that you have set. If you wish to capture all logged information, adjust the size of your screen buffer. Otherwise, use the tail or follow options to limit the output of the file.

---

To view logged GSS messages in the gss.log file:

1. Log on to the CLI following the instructions in Chapter 2, Setting Up Your GSS, the "Accessing the GSS CLI" section. The GSS CLI prompt appears.

2. Enable privileged EXEC mode. For example:

gssm1.yourdomain.com> enable

gssm1.yourdomain.com# 

3. Use the show logs command to display logged information for the device on your terminal. For example:

gssm1.yourdomain.com# show logs

gss.log

Jul 14 21:42:01 gss-css2 KAL-7-KALAP[1240] KAL-AP (seq# 29410)=> Host 192.10.2.1

Jul 14 21:42:02 gss-css2 KAL-7-KALAP[1240] KAL-AP (seq# 29412)=> Host 192.10.4.1

Jul 14 21:42:02 gss-css2 KAL-7-KALAP[1240] Retrying IP [192.10.4.1] (Retry Count 
3)

Jul 14 21:42:07 gss-css2 KAL-7-KALAP[1240] Timeout: Found outstanding KAL 
[192.10.2.1]

Jul 14 21:42:07 gss-css2 KAL-7-KALAP[1240] KAL-AP (seq# 29411)=> Host 192.10.2.1

Jul 14 21:42:07 gss-css2 KAL-7-KALAP[1240] Retrying IP [192.10.2.1] (Retry Count 
1)

Jul 14 21:42:09 gss-css2 KAL-7-KALCRA[1240] rtt_task: waiting 10000 mseconds

Jul 14 21:42:12 gss-css2 KAL-7-KALAP[1240] KAL-AP (seq# 29412)=> Host 192.10.2.1

Jul 14 21:42:12 gss-css2 KAL-7-KALAP[1240] Retrying IP [192.10.2.1] (Retry Count 
2)

Jul 14 21:42:16 gss-css2 KAL-7-KALAP[1240] Sending circuit keepalive => 
[192.10.2.1]

Jul 14 21:42:16 gss-css2 KAL-7-KALAP[1240] Sending circuit keepalive => 
[192.10.3.1]

Jul 14 21:42:16 gss-css2 KAL-7-KALAP[1240] Sending circuit keepalive => 
[192.10.4.1]

Jul 14 21:42:16 gss-css2 KAL-7-KALAP[1240] Sending circuit keepalive => 
[192.10.6.1]

Jul 14 21:42:16 gss-css2 KAL-7-KALAP[1240] Sending circuit keepalive => 
[192.10.7.1]

Jul 14 21:42:16 gss-css2 KAL-7-KALAP[1240] Sending circuit keepalive => 
[192.10.8.1]

Jul 14 21:42:17 gss-css2 KAL-7-KALAP[1240] KAL-AP (seq# 29410)=> Host 192.10.3.1

Jul 14 21:42:17 gss-css2 KAL-7-KALAP[1240] KAL-AP (seq# 29413)=> Host 192.10.2.1

Jul 14 21:42:17 gss-css2 KAL-7-KALAP[1240] Retrying IP [192.10.2.1] (Retry Count 
3)

Jul 14 21:42:19 gss-css2 KAL-7-KALCRA[1240] rtt_task: waiting 10000 mseconds

Jul 14 21:42:22 gss-css2 KAL-7-KALAP[1240] Timeout: Found outstanding KAL 
[192.10.3.1]

Jul 14 21:42:22 gss-css2 KAL-7-KALAP[1240] KAL-AP (seq# 29411)=> Host 192.10.3.1

Jul 14 21:42:22 gss-css2 KAL-7-KALAP[1240] Retrying IP [192.10.3.1] (Retry Count 
1)

Jul 14 21:42:22 gss-css2 NMR-7-NODEMGR[1035] Checking process queue for defunct 
members.

Jul 14 21:42:27 gss-css2 KAL-7-KALAP[1240] KAL-AP (seq# 29412)=> Host 192.10.3.1

Jul 14 21:42:27 gss-css2 KAL-7-KALAP[1240] Retrying IP [192.10.3.1] (Retry Count 
2)

...

4. To limit the output of the show logs command, specify one of the following:

–Use the tail option of the show logs command to view just the last ten lines of logged information. For example:

gssm1.yourdomain.com# show logs tail

–Use the follow option of the show logs command to view data that is appended to the end of the log as it grows. For example:

gssm1.yourdomain.com# show logs follow

Viewing Subsystem Log Files from the CLI

In addition to the gss.log file, each GSS device maintains a number of additional log files that record subsystem-specific information (for example, the keepalive engine or DNS server component of the GSS). Although these log files are not generally associated with specific CLI commands as the gss.log file is, any of them can be viewed from the CLI using the type EXEC command.

---

Note Many GSS subsystem logs output all logged information to your terminal session. This output may be quite large and exceed the buffer size that you have set. If you wish to capture all logged information, adjust the size of your screen buffer. Otherwise, use the tail or follow options to limit the output of the file.

---

To view your GSS subsystem log files:

1. Log on to the CLI following the instructions in Chapter 2, Setting Up Your GSS, the "Accessing the GSS CLI" section. The GSS CLI prompt appears.

2. From privileged EXEC mode, navigate to the directory containing the log file or files that you wish to view. For example:

gssm1.yourdomain.com> cd ../sysout

gssm1.yourdomain.com> 

3. Use the type command to display the contents of the log file. For example:

gssm1.yourdomain.com> type dnsserver.log

dnsserver.log

Starting dnsserver: Mon Jul  1 13:52:50 UTC 2003 [(1221)]

2003-07-10 16:23:08 relog: Booting...

Starting dnsserver: Wed Jul 10 16:23:33 UTC 2003 [(1201)]

End of file dnsserver.log

]

4. Use the tail command to view just the last ten lines of the log file. For example:

gssm1.yourdomain.com# tail dnsserver.log

Rotating Existing Log Files from the CLI

You can force the GSS to restart its log files and save archive copies of all existing log files by using the rotate-logs command. This command forces the GSS to save archive copies of all existing log files in the $STATE directory and subdirectories and replaces them with fresh log files.

Existing log files are archived locally using the following naming convention:

logfile_name.log.number

where:

•logfile_name.log - Name of the archived log file (for example, gss.log or kale.log) .

•number - An incremented number representing the number of times the logs have been rotated (for example, .3). The number of the most recent rotated log file is .1. The maximum number of log files is 25 for the gss.log file, five for all other log files.

To rotate existing log files:

1. Log on to the CLI following the instructions in Chapter 2, Setting Up Your GSS, the "Accessing the GSS CLI" section. The GSS CLI prompt appears.

2. Enable privileged EXEC mode. For example:

gssm1.yourdomain.com> enable 

gssm1.yourdomain.com# 

3. Use the rotate-logs command to rotate existing log files. For example:

gssm1.yourdomain.com# rotate-logs

If you wish to clear all rotated log files in the $STATE directory and subdirectories, except for the active log files, include the delete-rotated-logs option. For example:

gssm1.yourdomain.com# rotate-logs delete-rotated-logs 

Viewing System Logs from the Primary GSSM GUI

From the primary GSSM GUI, you can view messages logged in the GSS system.log file. This log presents the logged information that is most likely of interest to GSS administrators. However, the system.log file presents only a subset of all logged information. See the "Viewing Subsystem Log Files from the CLI" section for information about viewing the entire contents of individual GSS log files.

This section includes the following procedures:

•Viewing System Logs from the GUI

•Purging System Log Messages from the GUI

•System Log Messages

Viewing System Logs from the GUI

To view the GSS system logs:

1. From the primary GSSM GUI, click the Tools tab.

2. Click the System Logs option. The System Log list page appears (Figure 10-8) displaying the following information:

–Time—Time in Universal Coordinated Time (UTC) at which the logged event occurred on the GSS device.

–Node type—Type of GSS node (GSS or GSSM) on which the logged event occurred.

–Node name—Name assigned to the GSS device using the primary GSSM.

–Module—GSS component logging the message. For example, server or storeAdmin.

–Severity— Severity of the logged message; system log messages are rated using one of four severity levels, as follows:

•Fatal—Indicates that the GSS or one of its components failed. Fatal errors are rare and are usually caused by exceptions from which it is impossible to recover, or by the failure of a GSS component to initialize properly.

•Warning—Indicates a noncritical error or unexpected condition.

•Info—Provides information about the normal operation of the GSS and its components.

•Debug—Provides very detailed information about the internal operations of the GSS or one of its components. Debug log messages are intended for use by Cisco support engineers in their efforts to troubleshoot a problem.

–Description—Text description that explains the event.

–Message—Information about any relevant conditions encountered while the event was being logged.

Figure 10-8 System Log List Page

3. Click the column header of any of the displayed columns (except for Severity or Description) to sort the listed domains by a particular property.

Purging System Log Messages from the GUI

You can instruct the GSS to purge system log messages from the GSSM database by using the gssm database purge-log-records CLI command. This option removes the system log messages appearing on the primary GSSM GUI, the System Log list page of the Tools navigation tab. You can instruct the GSS software to:

•Purge a quantity of system log messages from the database up to the last n records, where n equals the number of database records back from the last record to be retained when the database is purged.

•Purge system log messages covering a set time period up to n days before today, where n equals the number of days back from today to be retained when the database is purged.

To purge system log messages from the GSSM database:

1. Log on to the CLI following the instructions in Chapter 2, Setting Up Your GSS, the "Accessing the GSS CLI" section. The GSS CLI prompt appears.

2. Enable privileged EXEC mode. For example:

gssm1.yourdomain.com> enable 

gssm1.yourdomain.com# 

3. Use the gssm database purge-log-records command to purge system log messages.

For example, to purge all system log messages except for the last 3, enter:

gssm1.yourdomain.com# gssm database purge-log-records count 3

For example, to purge all system log messages except for those generated within the last 7 days, enter:

gssm1.yourdomain.com# gssm database purge-log-records days 7

4. From the primary GSSM GUI, click the Tools tab, then click the System Logs option. The System Log list page appears. Notice that system log message have been purged based on the criteria specified in the gssm database purge-log-records CLI command.

System Log Messages

Table 10-9 lists common GSS system messages that may be encountered in the System Log list page. Error messages are listed alphabetically, and each error message is accompanied by a brief description. Contact a Cisco technical support representative if you require more detailed information about the purpose of a message.

Table 10-9 System Log Messages 

System Log Message	Description
Deleted a Global Site Selector	The named GSS has been deleted from the primary GSSM
Error occurred while processing received data	An error occurred while the device was processing configuration updates from the primary GSSM. The affected device will attempt to recover automatically.
Failed store invalidation	The process of marking internally inconsistent database records has failed. Errors can be viewed in the validation log.
Failed store validation	The GSSM database has failed its internal consistency checks.
Multiple primary GSSMs detected	The system has detected multiple primary GSSMs operating concurrently.
Passed store invalidation	The process of marking internally inconsistent database records has been successfully completed.
Passed store validation	The GSSM database has passed its internal consistency checks.
Registered a new Global Site Selector	A new GSS is online and identified itself to the primary GSSM.
Registered a new standby GSSM	A new standby GSSM came online and identified itself to the primary GSSM.
Server is Shutting Down	The Cisco GSS software has been stopped from the CLI.
Server Started	The Cisco GSS software has been started from the CLI.
Standby GSSM database error	An error has occurred on the standby GSSM embedded database.
Started store invalidation	The process of marking internally inconsistent database records has begun.
Started store validation	An internal consistency check has begun for the GSSM database.
Store is corrupted	The GSS GSSM database has failed internal consistency checks.
x System Messages Dropped	The GSS device has dropped (did not report) a certain number of messages in an effort to throttle message traffic to the GSSM.
Unexpected GSSM activation timestamp warning	The primary GSSM has received a report from a GSS device with a GSSM activation time stamp that was not consistent with the primary GSSM's current time. The standby and primary GSSM may have clocks that are not synchronized.
User HTTP Password Change	A user has changed his or her password using the Change Password details page from the Tools tab.