Cisco CNS Network Registrar User's Guide, 6.1
Maintaining Servers
Downloads: This chapterpdf (PDF - 289.0KB) The complete bookPDF (PDF - 6.99MB) | Feedback

Maintaining Servers

Table Of Contents

Maintaining Servers

Controlling Servers

Logging Server Events

Logging Format and Settings

Log Files

Monitoring and Reporting Server Status

Displaying Health

Displaying Statistics

Displaying IP Address Usage

Displaying Related Servers

DNS Zone Distribution Servers

DHCP Failover Servers

Displaying Leases

Troubleshooting Servers

Immediate Troubleshooting Actions

Troubleshooting Server Failures

Troubleshooting and Optimizing the TFTP Server

Tracing TFTP Server Activity

Optimizing TFTP Message Logging

Enabling TFTP File Caching

Solaris and Linux Troubleshooting Tools

Using the TAC Tool


Maintaining Servers


This chapter explains how to administer and control your local and regional servers' operations through Cisco CNS Network Registrar's Web-based user interface (Web UI) and CLI (the nrcmd program).

Controlling Servers

You can control the Network Registrar servers as follows:

Start—Load the database and start the server

Stop—Stop the server

Reload—Stop and restart the server

Starting and stopping a server is self-explanatory. When you reload the server, Network Registrar performs three steps—stops the server, loads configuration data, and restarts the server. Only after you reload the server does it use your changes to the configuration.

In the Web UI, you can start, stop, and reload the protocol servers in one of two ways:

If you are a local CCM or regional cluster administrator, click Administration on the Primary Navigation bar, then Servers on the Secondary Navigation bar, to open the Manage Servers page (see Figure 6-1 for a local cluster example).

Figure 6-1 Manage Servers Page

The local and regional cluster Web UI access to server administration is identical, even though the available functions are different. As a regional administrator, you can check the state and health of the regional CCM server, server agent, and Router Interface Configuration (RIC) server. However, you cannot stop, start, reload, or view statistics for them.

At the regional cluster, click the Start icon () to start the server, the Stop icon () to stop it, or the Reload icon () to reload it.

If you are a local cluster zone administrator, click Zone on the Primary Navigation bar, then DNS Server on the Secondary Navigation bar, to open the Manage DNS Server page. Click the Start icon () to start the server, the Stop icon () to stop it, or the Reload icon () to reload it.

If you are a local cluster DHCP administrator, click DHCP on the Primary Navigation bar, then DHCP Server on the Secondary Navigation bar, to open the Manage DHCP Server page. Click the Start icon () to start the server, the Stop icon () to stop it, or the Reload icon () to reload it.

In the CLI for the local cluster:

To start the server, use the server type start command (or simply type start, such as dhcp start).

To stop the server, use the server type stop command (or simply type stop, such as dhcp stop). If stopping the server, it is advisable to save it first using the save command.

To reload the server, use the server type reload command (or simply type reload, such as dhcp reload). Network Registrar stops the server you chose, loads the configuration data, and then restarts the server.


Note The DNS and DHCP servers are enabled by default to start on a reboot. The TFTP server is not enabled by default to start on a reboot. You can change this using the [server] type enable or disable start-on-reboot command in the CLI.


Logging Server Events

When you start Network Registrar, it automatically starts logging Network Registrar system activity. Network Registrar maintains all the logs by default on:

Windows—install-path\logs

Solaris and Linux—install-path/logs (to view these logs, use the tail -f command)


Tip To avoid filling up the Windows Event Viewer and preventing Network Registrar from running, in the Event Log Settings, check the Overwrite Events as Needed box.


Logging Format and Settings

The server log entries include the following categories:

Activity—Logs the activity of your servers.

Info—Logs standard operations of the servers, such as starting up and shutting down.

Warning—Logs warnings, such as invalid packets, user miscommunication, or an error in a script while processing a request.

Error—Logs events that prevent the server from operating properly, such as out of memory, unable to acquire resources, or errors in configuration.

In the Web UI, you can affect which events to log. For example, to set the logging for the local cluster DNS and DHCP server:

DNS—Click Zone on the Primary Navigation bar, then DNS Server on the Secondary Navigation bar, to open the Manage DNS Server page. Click the name of the server to open the Edit DNS Server page. Expand the Logging attributes section of the page to view the log settings. Make changes to these settings as desired, click Modify Server, then reload the server.

DHCP—Click DHCP on the Primary Navigation bar, then DHCP Server on the Secondary Navigation bar, to open the Manage DHCP Server page. Click the name of the server to open the Edit DHCP Server page. Expand the Logging section (the first one on the page) to view the log settings. Make changes to these settings as desired, click Modify Server, then reload the server.

In the CLI, use the dns set log-settings, dhcp set log-settings, and tftp set log-settings commands for the respective servers.


Note Warnings and errors go to Syslog on Solaris or the Event Viewer on Windows. See the Caution. For a description of the log messages for each server module, see the install-path/docs/msgid/MessageIdIndex.html file.


Log Files

Table 6-1 describes the Network Registrar log files in the install-path/logs directory.

Table 6-1 Log Files in .../logs Directory 

Component
File in /logs Directory
Local/Regional
Content

Installation

install_cnr_log

Both

Log of installation process

Server agent

agent_server_1_log

Both

Log of server agent starts and stops

Port check

checkports_log

Both

Log of network ports

DNS server

name_dns_1_log

Local

Log of DNS activity

DHCP server

name_dhcp_1_log

Local

Log of DHCP activity

TFTP server

file_tftp_1_log
file_tftp_1_trace

Local

Log of TFTP activity

RIC server

ric_server_log

Regional

Log of RIC server activity

CCM database

config_ccm_1_log

Both

Log of CCM configuration, starts, stops

Web UI

cnrwebui_log

Both

Log of Web UI state

Tomcat/Web UI (in cnrwebui subdirectory)

catalina_log.date.txt
jsui_log.date.txt
localhost_access_log.date.txt

Both

Log of CCM database for Tomcat server and Web UI; new files created daily, so to monitor disk usage, periodically archive old log files


Each component can generate a number of log files, each with a preconfigured maximum size of 1 MB. The first log file name has the _log suffix. When this file reaches its maximum size, it gets the .01 version extension appended to its name and a new log file is created without the version extension. Each version extension is incremented by one for each new file created. When the files reach their configured maximum number, the oldest file is deleted and the next oldest assumes its name. The usual maximum number is four for the DNS, DHCP, and TFTP servers.

You can check the configured maximums for the DNS, DHCP, and TFTP servers using the CLI command [server] type serverLogs show, which shows the maximum number (nlogs) and size (logsize) of these protocol server log files. You can adjust these parameters using the [server] type serverLogs set nlogs=value and [server] type serverLogs set logsize=value commands. You cannot adjust these maximums for any of the other log files.


Note Some user commands can create User authentication entries in the Server Agent log because of separate connections to the cluster. Do not interpret these as a system security violation by another user.


Monitoring and Reporting Server Status

Monitoring the status of a server involves checking its:

Health

Statistics

Log messages

Address usage

Related servers (DNS and DHCP)

Leases (DHCP)

Displaying Health

You can monitor the state of your Network Registrar servers by displaying or reporting aspects of a server's health. The server's health determines how well it is running. The following items can decrement the server's health, so you should monitor their status periodically. For the:

Server agent (local and regional clusters)

CCM server (local and regional clusters)

DNS server (local cluster):

Configuration errors

Memory

Disk space usage

Inability to contact its root servers

DHCP server (local cluster):

Configuration errors

Memory

Disk space usage

Packet caching low

Options not fitting in the stated packet limit

No more leases available

TFTP server (local cluster):

Memory

Socket read or write error

Exceeding the overload threshold and dropping request packets

RIC server (regional cluster)


Tip Use the existence of any descending health values as a reminder to check the log files for the server.


In both the local and regional cluster Web UIs, on the Primary Navigation bar, click Administration., then Servers on the Secondary Navigation bar. Check the Manage Server s page for the state and health of each server (see Figure 6-1 for a DHCP server example).

In the local cluster CLI, use the [server] type getHealth command. The number 10 indicates the highest level of health, 0 that the server is not running.


Tip On Solaris or Linux, you can run the cnr_status command, in the install-path/usrbin/ directory, to see if your local cluster server is running. See the Network Registrar Installation Guide.


Displaying Statistics

To display server statistics, the server must be running. In the local cluster Web UI, go to the Manage DNS Server page or Manage DHCP Server page and click the Statistics icon (). On the Statistics for Server page (see Figure 6-2 for a DHCP example), click the name of the statistic attribute to get popup help for its description.

Figure 6-2 Statistics for Server Page

In the CLI, use the [server] type getStats command. The statistics for the DNS and DHCP servers are encoded in curly braces followed by sets of digits, as described in Table 6-2 for DNS, Table 6-3 for DHCP, and Table 6-4 for TFTP.

Table 6-2 DNS Statistics 

Position
Attribute
Description

{1}

id

Implementation ID (release and build information)

2

config-recurs

Recursion services—(1)available, (2)restricted, (3)unavailable

3

config-up-time

Process up time (seconds)

4

config-reset-time

Time since the last reset (seconds)

5

config-reset

Number of server resets

6

counter-auth-no-names

Number of queries returning "authoritative no such name" responses

7

counter-auth-no-data-resps

Number of queries returning "authoritative no such data" responses

8

counter-non-auth-datas

Number of queries answered nonauthoritatively (cached)

9

counter-non-auth-no-datas

Number of queries answered nonauthoritatively with no data

10

counter-referrals

Number of forwarded queries

11

counter-errors

Number of error responses

12

counter-rel-names

Number of requests received of names of only one label (relative names)

13

counter-req-refusals

Number of refused queries

14

counter-req-unparses

Number of unparsable requests

15

counter-other-errors

Number of aborted requests due to other errors


Table 6-3 DHCP Statistics 

Position
Attribute
Description

{1}

start-time

Date and time of last server reload

2

total-discovers

Number of DISCOVER packets received

3

total-requests

Number of REQUEST packets received

4

total-releases

Number of RELEASED packets received

5

total-offers

Number of OFFER packets sent

6

total-acks

Number of acknowledgement (ACK) packets sent

7

total-nacks

Number of negative acknowledgement (NACK) packets sent

8

total-declines

Number of DECLINE packets received


Table 6-4 TFTP Statistics in the CLI 

Position
Attribute
Description

{1}

id

Implementation ID (release and build information)

2

server-state

State of the server

3

server-start-time

Start date and time (in seconds)

4

server-reset-time

Reset date and time

5

server-time-since-start

Running time since last start

6

server-time-since-reset

Running time since last reset

7

total-packets-in-pool

Number of packets in the pool

8

total-packets-in-use

Number of packets the server is using

9

total-packets-received

Number of packets received since the last start or reload

10

total-packets-sent

Number of packets sent since the last start or reload

11

total-packets-drained

Number of packets read and discarded since the last start or reload

12

total-packets-dropped

Number of packets dropped since the last start or reload

13

total-packets-malformed

Number of packets received that were malformed since the last start or reload

14

total-read-requests

Number of packets read since the last start or reload

15

total-read-requests-completed

Number of read packets completed since the last start or reload

16

total-read-requests-refused

Number of read packets refused since the last start or reload

17

total-read-requests-ignored

Number of read packets ignored since the last start or reload

18

total-read-requests-timed-out

Number of read packets that timed out since the last start or reload

19

total-write-requests

Number of read packets that were write requests since the last start or reload

20

total-write-requests-completed

Number of write requests completed since the last start or reload

21

total-write-requests-refused

Number of write requests refused since the last start or reload

22

total-write-requests-ignored

Number of write requests ignored since the last start or reload

23

total-write-requests-timed-out

Number of write requests that timed out since the last start or reload

24

total-docsis-requests

Number of DOCSIS requests received since the last start or reload

25

total-docsis-requests-completed

Number of DOCSIS requests completed since the last start or reload

26

total-docsis-requests-refused

Number of DOCSIS requests refused since the last start or reload

27

total-docsis-requests-ignored

Number of DOCSIS requests ignored since the last start or reload

28

total-docsis-requests-timed-out

Number of DOCSIS requests that timed out since the last start or reload

29

read-requests-per-second

Number of read requests per second

30

write-requests-per-second

Number of write requests per second

31

docsis-requests-per-second

Number of DOCSIS requests per second


Displaying IP Address Usage

In the Web UI, you can look at the local or regional cluster's address space, or generate a subnet utilization or lease history report on the regional cluster, to determine IP address usage. These functions are available in both Web UIs by clicking Address Space on the Primary Navigation bar, if you have address space privileges on the local or regional cluster.

In the CLI, you can generate an IP address usage report using the report command. See the Network Registrar CLI Reference for additional options you can set.

Displaying Related Servers

Network Registrar displays the relationship among servers in a DNS zone distribution or a DHCP failover configuration.

DNS Zone Distribution Servers

A DNS zone distribution simplifies creating multiple zones that share the same secondary server attributes. In the Web UI, you can view and set the primary and secondary DNS servers in a zone distribution:

On the local cluster, click Zone on the Primary Navigation bar, then Zone Distribution on the Secondary Navigation bar. This opens the List Zone Distributions page. The local cluster allows only one zone distribution, the Default. Click this zone distribution name to open the Edit Zone Distribution page, which shows the authoritative and secondary servers in the zone distribution.

On the regional cluster, click DNS Configuration on the Primary Navigation bar, then Zone Distributions on the Secondary Navigation bar. This opens the List/Add Zone Distributions page. The regional cluster allows creating more than one zone distribution. Click the zone distribution name to open the Edit Zone Distribution page, which shows the primary, authoritative, and secondary servers in the zone distribution.

You cannot use the CLI to view or set zone distributions.

DHCP Failover Servers

Related servers in a DHCP failover pair relationship can show the following information:

Type—Main or backup DHCP server

Server name—DNS name of the server

IP address—Server's IP address in dotted octet format

Requests—Number of outstanding requests, or two dashes if not applicable

Communication status—OK or INTERRUPTED

Cluster state—Failover state of this DHCP server

Partner state—Failover state of its partner server

In the Web UI:

On the local cluster, on the Primary Navigation bar, click DHCP, then on the Secondary Navigation bar, click Failover. The List DHCP Failover Pairs page shows the main and backup servers in the failover relationship.

On the regional cluster, on the Primary Navigation bar, click DHCP Configuration, then on the Secondary Navigation bar, click Failover. The List DHCP Failover Pairs page shows the main and backup servers in the failover relationship.

In the CLI, use the dhcp getRelatedServers command to display the connection status between the main and partner DHCP servers. If there are no related servers, the output is simply "100 Ok."

Displaying Leases

After you create a scope, you can monitor lease activity and view lease attributes.

In the Web UI:

On the local cluster, on the Primary Navigation bar, click DHCP, then on the Secondary Navigation bar, click Scopes. Click a scope name on the List/Add DHCP Scopes page to open the Edit DHCP Scope page. Halfway down the page, click List Leases to open the List DHCP Lease for Scope page.

On the regional cluster, you can view the lease history. On the Primary Navigation bar, click Address Space, then on the Secondary Navigation bar, click Lease History. Set the query parameters, then click Query Lease History. (See the "Running Lease Histories" section.)

In the CLI, use the lease list command to view the properties of all the available leases.

Troubleshooting Servers

The following sections describe troubleshooting the DNS, DHCP, and TFTP server.

Immediate Troubleshooting Actions

When facing a problem, it is crucial not to cause further harm while isolating and fixing the initial problem. Here are things to do (or avoid doing) in particular:

Have 512 MB or more of memory and 2.5 GB or more of a data partition.

Do not reboot a cable modem termination system (CMTS).

Enable DHCP failover.

Do not reload, restart, or disrupt Network Registrar with failover resynchronization in progress.

Troubleshooting Server Failures

The server agent processes (nwreglocal and nwregregion) normally detect server failures and restart the server. You can usually recover from the failure and the server is not likely to fail again immediately after restarting. On rare occasions, the source of the server failure prevents the server from successfully restarting, and the server fails again as soon as it restarts. In such instances, perform the following steps:


Step 1 If the server takes a significantly long time to restart, stop and restart the server agent. On:

Windows:

net stop nwreglocal or nwregregion 
net start nwreglocal or nwregregion 

Solaris:

/etc/init.d/nwreglocal stop or nwregregion stop 
/etc/init.d/nwreglocal stop or nwregregion start 

Linux:

/etc/rc.d/init.d/nwreglocal stop or nwregregion stop 
/etc/rc.d/init.d/nwreglocal stop or nwregregion start 

Step 2 Keep a copy of all the log files. Log files are located in the install-path/logs directory on Solaris and Linux, and the install-path\logs folder on Windows. The log files often contain useful information that can help isolate the cause of a server failure.

Step 3 Use the TAC tool, as described in the "Using the TAC Tool" section, or save the core or user.dmp file, if one exists, depending on the operating system:

On Windows—The user.dmp file is located in the system directory, which varies depending on the Windows system. Search for this file and save a renamed copy.

On Solaris and Linux—The core file is located in the install-path. Save a renamed copy of this file that Network Registrar does not overwrite.

Step 4 On Windows, use the native event logging application to save the System and Application event logs to files. You can do this from the Event Viewer. These event logs often contain data that helps debug Network Registrar server problems. For a description of the log messages for each server module, see the install-path/docs/msgid/MessageIdIndex.html file.


Troubleshooting and Optimizing the TFTP Server

You can set certain attributes to troubleshoot and optimize TFTP server performance.

Tracing TFTP Server Activity

The TFTP server has two CLI commands that help create more output to logs and can be useful in troubleshooting, although this usually impacts performance. These commands set up server packet tracing. The tftp getTraceLevel command identifies the current trace level, which by default is 0, or no tracing. The tftp setTraceLevel command sets the packet tracing to a value between 0 and 4. The trace files are located in the /logs subdirectory of the installation directory. Windows tracing goes to the file_tftp_1_log file; Solaris and Linux tracing goes to the /var/nwreg2/{local | regional}/logs/file_tftp_1_log and file_tftp_1_trace files.

Here are the trace levels, with each higher level being cumulative:

0—Disables all server tracing (the default)

1—Displays all the log messages in the trace file

2—Displays the client's IP address and port number for all packets

3—Displays the packet header information

4—Displays the first 32 bytes of the packet


Note Setting and getting the trace level only works if the TFTP server is started. Turn on packet tracing only for debugging purposes, and then not for any extended time, for performance reasons.


Optimizing TFTP Message Logging

You can improve TFTP server performance by restricting logging and tracing. By default, the server logs error, warning, and informational messages to file_tftp_1_log files. You can set the log levels using a few TFTP server parameters:

Log level (use the log-level attribute)—Master controller of server logging, which defaults to, and is best left at, level 3 (logs all error, warning, and informational messages). As with packet tracing, the higher logging levels are cumulative. If set to 0, no server logging occurs.

Log settings (use the log-settings attribute)—This is the second level of logging control and takes only two values, default or no-success-messages. The default log setting does not alter the default of log level 3 (error, warning, and informational messages). However, you may want to disable writing success informational messages, and thereby improve server performance, by changing the log settings to no-success-messages.

Log file count and size (use the log-file-count attribute)—Sets how many log files to maintain and how large to allow them to get in the /logs directory. The default is to maintain a maximum of four files of one megabyte each.


Note Reload the TFTP server after changing these values.


Enabling TFTP File Caching

You can improve TFTP server performance significantly by enabling file caching on the server. You must do this explicitly, because it is disabled by default. You must also create and point to a file cache directory, and you can set the maximum size of this directory. Here are the steps:


Step 1 Determine where you want to store the TFTP cache files. This cache directory becomes a subdirectory of the TFTP home directory, which by default is install-path/data/tftp (on Solaris and Linux, it is /var/nwreg2/{local | regional}/data/tftp). If you want a different location, set the home-directory attribute.

Step 2 Change to the TFTP home directory and create the cache directory, such as CacheDir, in the home directory, using the mkdir Cachedir command. Note that Network Registrar ignores any files in any subdirectories of this cache directory.

Step 3 Use the file-cache-directory attribute to set up the TFTP server to point to the cache directory. You cannot use relative paths in the directory name, such as .../cachedir. If the directory does not exist, file caching cannot occur.

Step 4 Use the file-cache-max-memory-size attribute to set the maximum memory size, in bytes, of the cache. The default is 32 K bytes. Network Registrar loads all files into cache that cumulatively fit this memory size. If you set the value to 0, Network Registrar does not cache any data, even if you enable file caching.

Step 5 Copy all of the files you want cached into the cache directory, and not into any subdirectory. Because all files in this directory are loaded into cache, do not include large files.

Step 6 Enable the file-cache attribute to enable file caching, then reload the server. Network Registrar logs the name of each cached file, and skips any it cannot load. It reads in all files as binary data and translates them as the TFTP client requests. For example, if a client requests a file as NetASCII, the client receives the cached data in that form.

Step 7 Writing to cache is not allowed. If you need to update a cache file, overwrite it in the cache directory, then reload the server.


Solaris and Linux Troubleshooting Tools

You can also use the following commands on Solaris and Linux systems to troubleshoot Network Registrar. To:

See all Network Registrar processes:

ps -leaf | grep nwr 

Monitor system usage and performance:

top 
vmstat 

View login or bootup errors:

On Solaris—grep /var/adm/messages*

On Linux—grep /var/log/messages*

View the configured interfaces and other network data:

ifconfig -a 

Using the TAC Tool

There may be times when any amount of troubleshooting steps will not resolve your problem and you have to resort to contacting the Cisco Technical Assistance Center (TAC) for help. Network Registrar provides a tool so that you can easily assemble the server or system error information, and package this data for TAC support engineers. This eliminates having to manually assemble this information with TAC assistance. The resulting package from this tool provides the engineers enough data so that they can more quickly and easily diagnose the problem and provide a solution.

The cnr_tactool utility is available in the bin directory of the Windows, and usrbin directory of the UNIX or Linux, installation directories. Execute the cnr_tactool utility:

> cnr_tactool -N username -P password [-d output-directory]

The output directory is optional and normally is the temp directory of the installation directories (in the /var path on Solaris and Linux). If you do not supply the username and password on the command line, you are prompted for them:

> cnr_tactool 
username:
password:
[processing messages....] 

The tool generates a packaged tar file whose name includes the date and version. The tar file contains all the diagnostic files.