User Guide for Cisco Network Registrar, 7.0
Maintaining Servers and Databases
Downloads: This chapterpdf (PDF - 467.0KB) The complete bookPDF (PDF - 18.25MB) | Feedback

Maintaining Servers and Databases

Table Of Contents

Maintaining Servers and Databases

Managing Servers

Scheduling Recurring Tasks

Logging Server Events

Searching the Logs

Logging Format and Settings

Log Files

Change Logs and Tasks

Monitoring and Reporting Server Status

Server States

Displaying Health

Displaying Statistics

DNS Statistics

DHCP Statistics

TFTP Statistics

Displaying IP Address Usage

Displaying Related Servers

Monitoring Remote Servers Using Persistent Events

DNS Zone Distribution Servers

DHCP Failover Servers

Displaying Leases

Running Data Consistency Rules

Troubleshooting Servers

Immediate Troubleshooting Actions

Modifying the cnr.conf File

Troubleshooting Server Failures

Troubleshooting and Optimizing the TFTP Server

Tracing TFTP Server Activity

Optimizing TFTP Message Logging

Enabling TFTP File Caching

Solaris and Linux Troubleshooting Tools

Using the TAC Tool


Maintaining Servers and Databases


This chapter explains how to administer and control your local and regional server operations.

Managing Servers

If you are assigned the server-management subrole of the ccm-admin role, you can manage the Network Registrar servers as follows:

Start—Load the database and start the server.

Stop—Stop the server.

Reload—Stop and restart the server. (Note that you do not need to reload the server with each dynamic update. See Chapter 28, "Configuring DNS Update" for details.)

Check statistics—See the "Displaying Statistics" section.

View logs—See the "Searching the Logs" section.

Manage interfaces—See the specific protocol pages for how to manage server interfaces.

Starting and stopping a server is self-explanatory. When you reload the server, Network Registrar performs three steps—stops the server, loads configuration data, and restarts the server. Only after you reload the server does it use your changes to the configuration.


Note The DNS, DHCP, and SNMP servers are enabled by default to start on reboot. The TFTP server is not enabled by default to start on reboot. You can change this using [servertype enable or disable start-on-reboot in the CLI.


Local Basic or Advanced and Regional Web UI

You can manage the protocol servers in the following ways depending on if you are a:

Local or regional cluster administrator—Click Servers to open the Manage Servers page (see Figure 7-1 for a local cluster example).

Figure 7-1 Manage Servers Page (Local Basic)

The local and regional cluster web UI access to server administration is identical, even though the available functions are different. As a regional administrator, you can check the state and health of the regional CCM server, server agent, and Router Interface Configuration (RIC) server. However, you cannot stop, start, reload, or view statistics, logs, or interfaces for them.

At the local cluster, you can manage the DHCP, DNS, TFTP, and SNMP servers:

Click the Statistics icon () to view statistics for the server. (See the "Displaying Statistics" section.)

Click the Log icon () in the View Log column to view the log messages for the server. (See the "Logging Server Events" section.)

Click the Start icon () to start the server.

Click the Stop icon () to stop the server.

Click the Reload icon () to reload the server.

Local cluster DNS administrator—Click DNS, then DNS Server to open the Manage DNS Server page (see Figure 7-2).

Figure 7-2 Manage DNS Server Page (Local Basic)

Along with the Statistics (), Log (), Start (), Stop (), and Reload () functions, you can also perform other functions when you click the Run icon () in the Commands column to open the DNS Server Commands page (see Figure 7-3).

In Expert mode, you can also synchronize the CCM server from the DNS server. Click Sync CCM Server from DNS Server to open the Sync CCM Server from DNS Server page. Note the change sets to synchronize on this page, then click Run, followed by Return.

Figure 7-3 DNS Server Commands Page (Local Basic)

The server command functions are:

Flushing cache (see the "Flushing DNS Cache" section on page 17-15)—Click the Run icon (). If you want to flush cache older than a certain time, enter the time in the Time field next to Flush cache older than, then click the Run icon. This is the equivalent of dns flushCache in the CLI.

Forcing all zone transfers (see the "Enabling Zone Transfers" section on page 15-15)—Click the Run icon (). This is the equivalent of dns forceXfer secondary in the CLI.

Scavenging all zone transfers (see the "Scavenging Dynamic Records" section on page 28-18)—Click the Run icon (). This is the equivalent of dns scavenge in the CLI.

Removing nonauthoritative resource record sets (see the "Removing Cached Records" section on page 16-5)—Enter the name of the RR set in the Name field, then click the Run icon (). If you optionally want to remove nonauthoritative RR sets based on specific types or data, enter these values in the Type or Data field, respectively. This is the equivalent of dns removeCachedRR name type data in the CLI.

Local cluster DHCP administrator—Click DHCP, then DHCP Server to open the Manage DHCP Server page. Along with the Statistics (), Log (), Start (), Stop (), and Reload () functions, you can also perform other functions when you click the Run icon () in the Commands column to open the DHCP Server Commands page (see Figure 7-4).

Figure 7-4 DHCP Server Commands Page (Local Basic)

This page provides the Get Leases with Limitation ID feature, to find clients that are associated through a common limitation identifier (see the "Administering Option 82 Limitation" section on page 24-18). Enter at least the IP address of the currently active lease in the IP Address field, then click the Run icon (). You can also enter the limitation ID itself in the form nn:nn:nn or as a string ("nnnn"), in which case the IP address becomes the network in which to search. This function is the equivalent of dhcp limitationList ipaddress limitation-id show in the CLI.

CLI Commands

In the CLI, the regional cluster allows CCM server management only:

To start the server, use server type start (or simply type start; for example, dhcp start).

To stop the server, use server type stop (or simply type stop; for example, dhcp stop). If stopping the server, it is advisable to save it first using the save command.

To reload the server, use server type reload (or simply type reload; for example, dhcp reload). Network Registrar stops the server you chose, loads the configuration data, and then restarts the server.

To set or show attributes for the server, use [server] type set attribute=value or [server] type show. For example:

nrcmd> ccm set ipaddr=192.168.50.10 

See Also

Scheduling Recurring Tasks
Logging Server Events
Log Files
Change Logs and Tasks
Monitoring and Reporting Server Status
Running Data Consistency Rules
Troubleshooting Servers

Scheduling Recurring Tasks

In Basic and Advanced user mode in the local cluster web UI, you can schedule a number of recurring tasks. These tasks are:

Reloading the DHCP server.

Reloading the DNS server.

Synchronizing DHCP failover server pairs:

If in staged scope edit mode, reload the main DHCP server.

Synchronize the failover configuration to the backup DHCP server.

If in staged scope edit mode, reload the backup DHCP server.

Synchronizing High-Availability (HA) DNS server pairs:

If in staged scope edit mode, reload the main DNS server.

Synchronize the HA DNS configuration to the backup DNS server.

If in staged scope edit mode, reload the backup DNS server.

Synchronizing zone distribution maps:

If in staged scope edit mode, reload the main DNS server.

If in staged scope edit mode, reload the backup HA DNS server.

Synchronize the zone distribution maps.

If in staged scope edit mode, reload the secondary DNS server or servers.

Local Basic or Advanced Web UI

To set up one or more of these recurring server tasks:


Step 1 Click Servers, then Scheduler to open the List Scheduled Tasks page.

Step 2 Click Add Task to open the Add Scheduled Task page (see Figure 7-5).

Figure 7-5 Add Scheduled Task Page (Local Basic)

Step 3 Enter values in the appropriate fields:

a. Name of the scheduled task. This can be any identifying text string.

b. Description of the task.

c. Pull down from the available list of task types, which are:

dhcp-reload—Reloads the DHCP server

dns-reload—Reloads the DNS server

sync-dhcp-pair—Synchronizes the DHCP failover server pair

sync-dns-pair—Synchronizes the HA DNS failover server pair

sync-zd-map—Synchronizes zone distribution maps

sync-dns-update-map—Synchronizes DNS update maps

d. In Advanced mode, select to enable or disable the task (the preset value is enabled in both modes).

e. Indicate the time interval for the scheduled task, such as 60m or 4w2d.

f. Indicate any scheduled offset, or when you want the task scheduled at a fixed time of day, as an hour from 12:00 midnight. The scheduled interval must be less than 24h, and the offset must be less than the interval. For example, if you set the scheduled-interval to 4h and the scheduled-offset to 2, the task would occur at 2:00 A.M., 6:00 A.M., 10:00 A.M., 2:00 P.M., 6:00 P.M., and 10:00 P.M.

g. If you are synchronizing a DHCP failover pair, HA DNS pair, or zone distribution map, you can also choose the relevant object identifier associated with the task.

Step 4 Click Add New Task.

Step 5 If you click the name of the task on the List Scheduled Tasks page, on the Edit Scheduled Task page you can view (in the Task Status section) the last status or the list of last errors (if any) that occurred during the task execution. Click Run Now to run the task immediately.


Logging Server Events

When you start Network Registrar, it automatically starts logging Network Registrar system activity. Network Registrar maintains all the logs by default on:

Windowsinstall-path\logs

Solaris and Linuxinstall-path/logs (to view these logs, use the tail -f command)


Tip To avoid filling up the Windows Event Viewer and preventing Network Registrar from running, in the Event Log Settings, check the Overwrite Events as Needed box. If the events do fill up, save them to a file, then clear them from the Event Log.


Local Basic or Advanced and Regional Web UI

Server logging is available in the web UI when you open the Manage Servers page for a server (see the "Managing Servers" section), then click the Log icon () in the View Log column for the server. This opens the Log for Server page. The log is in chronological order with the page with the latest entries shown first. If you need to see earlier entries, click the left arrow at the top or bottom of the page.

See Also

Searching the Logs
Logging Format and Settings

Searching the Logs

The web UI provides a convenient way to search for entries in the activity and startup log files. You can locate specific message text, log message IDs, and message timestamps using a regular expression string entry. When you click the Log icon () in the View Log or View Startup column on the Manage Servers page (or one of the specific server pages), this opens a Log for Server page. In the text field next to the Search icon () at the top or bottom of the page, enter the search string in the regular expression syntax. (For example, entering name? searches for occurrences of the string "name" in the log file.)

Clicking the Search icon () opens a Log Search Result page in a separate browser window (see Figure 7-6 for an example). The page shows the log file, line number of the match, and the log number.

Figure 7-6 Log Search Result Page (Local)

Click the name of the log message, which opens the Log for Server page with the full message text. Change between Table and Text view by clicking the Log icon (). Click Close on the Log Search Result page to close the browser window.

Logging Format and Settings

The server log entries include the following categories:

Activity—Logs the activity of your servers.

Info—Logs standard operations of the servers, such as starting up and shutting down.

Warning—Logs warnings, such as invalid packets, user miscommunication, or an error in a script while processing a request.

Error—Logs events that prevent the server from operating properly, such as out of memory, unable to acquire resources, or errors in configuration.


Note Warnings and errors go to the Event Viewer on Windows (see the Tip). For a description of the log messages for each server module, see the install-path/docs/msgid/MessageIdIndex.html file.


Local Basic or Advanced and Regional Web UI

You can affect which events to log. For example, to set the logging for the local cluster DNS and DHCP server:

DNS—Click DNS, then DNS Server to open the Manage DNS Server page. Click the name of the server to open the Edit DNS Server page. Expand the Logging attributes section to view the log settings. Make changes to these settings as desired, click Modify Server, then reload the server. (See Table 17-2 on page 17-19 for the log settings to maximize DNS server performance.)

DHCP—Click DHCP, then DHCP Server to open the Manage DHCP Server page. Click the name of the server to open the Edit DHCP Server page. Expand the Logging section to view the log settings. Make changes to these settings as desired, click Modify Server, then reload the server. (See Table 23-2 on page 23-12 for the log settings to maximize DHCP server performance.)

CLI Commands

Use dns set log-settings, dhcp set log-settings, and tftp set log-settings for the respective servers.

Log Files

Table 7-1 describes the Network Registrar log files in the install-path/logs directory.

Table 7-1 Log Files in .../logs Directory 

Component
File in /logs Directory
Local/Regional
Logs

Installation

install_cnr_log

Both

Installation process

Upgrade

mcdupgrade_log

Both

Upgrade process

Server agent

agent_server_1_log

Both

Server agent starts and stops

Port check

checkports_log

Both

Network ports

DNS server

name_dns_1_log

Local

DNS activity

DHCP server

name_dhcp_1_log

Local

DHCP activity

TFTP server

file_tftp_1_log
file_tftp_1_trace

Local

TFTP activity

SNMP server

cnrsnmp_log

Local

SNMP activity

RIC server

ric_server_log

Both

RIC server activity

CCM database

config_ccm_1_log

Both

CCM configuration, starts, stops

Web UI

cnrwebui_log

Both

Web UI state

Tomcat/web UI (in cnrwebui subdirectory)

catalina_log.date.txt
jsui_log.date.txt
localhost_access_log.date.txt

Both

CCM database for Tomcat server and web UI (because new files are created daily, periodically archive old log files)


Each component can generate a number of log files, each with a preconfigured maximum size of 1 MB. The first log file name has the _log suffix. When this file reaches its maximum size, it gets the .01 version extension appended to its name and a new log file is created without the version extension. Each version extension is incremented by one for each new file created. When the files reach their configured maximum number, the oldest file is deleted and the next oldest assumes its name. The usual maximum number is four for the DNS, DHCP, and TFTP servers.


Note Some user commands can create User authentication entries in the Server Agent log because of separate connections to the cluster. Do not interpret these as a system security violation by another user.


CLI Commands

You can check the configured maximums for the DNS, DHCP, and TFTP servers using [servertype serverLogs show in the CLI, which shows the maximum number (nlogs) and size (logsize) of these protocol server log files. You can adjust these parameters using [servertype serverLogs set nlogs=value and [server] type serverLogs set logsize=value. You cannot adjust these maximums for any of the other log files.

Change Logs and Tasks

In the web UI, you can view the change logs and tasks associated with configurations you make.

Local Advanced and Regional Web UI

Click Administration, then Change Log, CCM Tasks, or MCD Tasks. To view the change log and tasks, you must be assigned the database subrole of the ccm-admin or regional-admin role:

The View Change Log page (see Figure 7-7) shows all the change logs, sorted by DBSN name. To get to the bottom of the list, click the right arrow at the bottom left of the page. Click the DBSN number of the change log entry to open a View Change Set page for it.

Figure 7-7 View Change Log Page (Local Advanced)

The List CCM Tasks page shows all the CCM tasks.

The List MCD Tasks page shows all the MCD tasks.

On the View Change Log page, you can filter the list, manually trim it, and save it to a file. You can filter the list by:

Start and end dates

Administrator who initiated the changes

Configuration object class

Specific object

Object identifier (ID), in the format OID-00:00:00:00:00:00:00:00

Server

Click Filter List or Clear Filter (to clear the filter that persists through the session). You can initiate a trim of the change log by setting how many days old you want the record to get before trimming it, by setting a number of days value in the "older than" field and clicking the Delete icon ().

To save the change log entries to a comma-separated values (CSV) file, click the Save icon ().

If a task is associated with a change log, it appears on the View Change Set page. You can click the task name to open the View CCM Task or View MCD Task page for it.

Monitoring and Reporting Server Status

Monitoring the status of a server involves checking its:

State

Health

Statistics

Log messages

Address usage

Related servers (DNS and DHCP)

Leases (DHCP)

See Also

Server StatesDisplaying Health
Displaying Statistics
Displaying IP Address Usage
Displaying Related Servers
Displaying Leases

Server States

All Network Registrar protocol servers (DNS, DHCP, SNMP, and TFTP) pass through a state machine consisting of the following states:

Loaded—First step after the server agent starts the server (transitional).

Initialized—Server was stopped or fails to configure.

Unconfigured—Server is not operational because of a configuration failure (transitional).

Stopped—Server was administratively stopped and is not running (transitional).

Running—Server is running successfully.

The two essential states are initialized and running, because the server transitions through the states so quickly that the other states are essentially invisible. Normally, when the server agent starts the server, it tells the server to be up. The server process starts, sets its state to loaded, then moves up to running. If you stop the server, it walks down the states to initialized, and if you restart, it moves up to running again. If it fails to configure for some reason, it drops back to initialized, as if you had stopped it.

There is also an exiting state that the server is in very briefly when the process is exiting. The user interface can also consider the server to be disabled, but this rarely occurs and only when there is no server process at all (the server agent was told not to start one).

Displaying Health

You can display aspects of the health of a server, or how well it is running. The following items can decrement the server health, so you should monitor their status periodically. For the:

Server agent (local and regional clusters)

CCM server (local and regional clusters)

DNS server (local cluster):

Configuration errors

Memory

Disk space usage

Inability to contact its root servers

DHCP server (local cluster):

Configuration errors

Memory

Disk space usage

Packet caching low

Options not fitting in the stated packet limit

No more leases available

TFTP server (local cluster):

Memory

Socket read or write error

Exceeding the overload threshold and dropping request packets

RIC server (regional cluster)


Tip Health values range from 10 (the highest level of health) to 0 (the server is not running). Use the existence of any descending health values as a reminder to check the log files for the server.

On Solaris or Linux, you can run the cnr_status command, in the install-path/usrbin/ directory, to see if your local cluster server is running. See the Installation Guide for Cisco Network Registrar.


Local Basic or Advanced and Regional Web UI

Click Administration, then Servers. Check the Manage Servers page for the state and health of each server (see Figure 7-1 for a DHCP server example).

CLI Commands

Use [server] type getHealth. The number 10 indicates the highest level of health, 0 that the server is not running.

Displaying Statistics

To display server statistics, the server must be running.

Local Basic or Advanced and Regional Web UI

Go to the Manage Servers page, then click the Statistics icon (), if available, in the Statistics column. On the Server Statistics page, click the name of the attribute to get popup help.

The DHCP and DNS statistics are each divided into two groups of statistics. The first group is for total statistics and the second group is for sample statistics. The total statistics are accumulated over time. The sample statistics occur during a configurable sample interval. The names of the two categories vary per server and per user interface, and are identified in Table 7-2.

Table 7-2 Server Statistics Categories 

Server
User Interface
Total Statistics (Command)
Sample Statistics (Command)

DHCP

Web UI

Total Statistics

Activity Summary

 

CLI

Total Counters since the start of the last DHCP server process
(dhcp getStats)

Sampled counters since the last sample interval
(dhcp getStats sample)

DNS

Web UI

Total Statistics

Sample Statistics

 

CLI

Total Counters since the start of the last server process
(dns get Stats)

Sampled counters since the last sample interval
(dns getStats sample)


To set up the sample counters, you must activate either the collect-sample-counters attribute for the server or a log-settings attribute value called activity-summary. You can also set a log-settings value for the sample interval for each server, which is preset to 5 minutes. The collect-sample-counters attribute is preset to true for the DNS server, but is preset to false for the DHCP server. For example, to enable the sample counters and set the interval for DHCP, set the following attributes for the DHCP server:

Enable collect-sample-counters (dhcp enable collect-sample-counters)

Set log-settings for activity-summary (dhcp set log-settings=activity-summary)

Set activity-summary-interval to 5m (dhcp set activity-summary-interval=5m)

CLI Commands

In the CLI, if you use [server] type getStats, the statistics are encoded in curly braces followed by sets of digits, as described in Table 7-3 for DNS, Table 7-4 for DHCP, and Table 7-5 for TFTP. The server type getStats all command is more verbose and identifies each statistic on a line by itself. Using the additional sample keyword shows the sample statistics only.

Reset the counters and total statistic by using dhcp resetStats or dns resetStats.

See Also

DNS Statistics
DHCP Statistics
TFTP Statistics

DNS Statistics

The DNS server statistics in the web UI appear on the DNS Server Statistics page, and you can click each statistic name to open a description for it.

The CLI dns getStats command has the following options:

dns getStats [performance | query | errors | security | maxcounters | ha | ipv6 | all] 
[total | sample]

The dns getStats all command is the most commonly used. The dns getStats command without this option returns the statistics in a single line of positional values in the following format (Table 7-3 shows how to read these values):

nrcmd> dns getStats 
100 Ok
{1} 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

Table 7-3 DNS Statistics 

Digit
Statistic
Description

{1}

id

Implementation ID (release and build information).

2

config-recurs

Recursion services—(1) available, (2) restricted, (3) unavailable.

3

config-up-time

Time (in seconds) elapsed since the last server startup.

4

config-reset-time

Time (in seconds) elapsed since the last server reset (restart).

5

config-reset

Status or action to reinitializes any name server state—If using the (2) reset action, reinitializes any persistent name server state; the following are read-only statuses: (1) other—server in some unknown state, (3) initializing, or (4) running.

6

counter-auth-ans

Number of queries answered authoritatively.

7

counter-auth-no-names

Number of queries returning authoritative no such name responses.

8

counter-auth-no-data-resps

Number of queries returning authoritative no such data (empty answer) responses.

9

counter-non-auth-datas

Number of queries answered nonauthoritatively (cached).

10

counter-non-auth-no-datas

Number of queries answered nonauthoritatively with no data.

11

counter-referrals

Number of queries forwarded to other servers.

12

counter-errors

Number of responses answered with errors (RCODE values other than 0 or 3).

13

counter-rel-names

Number of requests received for names of only one label (relative names).

14

counter-req-refusals

Number of refused queries.

15

counter-req-unparses

Number of unparsable requests.

16

counter-other-errors

Number of aborted requests due to other errors.

17

total-zones

Total number of configured zones.


DHCP Statistics

The DHCP server statistics in the web UI appear on the DHCP Server Statistics page, and you can click each statistic name to open a description for it.

The CLI dhcp getStats command has the following options:

dhcp getStats [[all | server [,] failover [,] dhcpv6] [total | sample]

The dhcp getStats all command is the most commonly used. The dhcp getStats command without this option returns the statistics in a single line of positional values in the following format (Table 7-4 shows how to read these values):

nrcmd> dhcp getStats 
100 Ok
{1} 2 3 4 5 6 7 8

Table 7-4 DHCP Statistics 

Digit
Statistic
Description

{1}

start-time-str

Date and time of last server reload, as a text string.

2

total-discovers

Number of DISCOVER packets received.

3

total-requests

Number of REQUEST packets received.

4

total-releases

Number of RELEASED packets received.

5

total-offers

Number of OFFER packets sent.

6

total-acks

Number of acknowledgement (ACK) packets sent.

7

total-naks

Number of negative acknowledgement (NAK) packets sent.

8

total-declines

Number of DECLINE packets received.


TFTP Statistics

The TFTP server statistics in the web UI appear on the TFTP Server Statistics page, and you can click each statistic name to open a description for it. Table 7-5 shows the TFTP statistics encoded as output to the generic tftp getStats command, in the format:

nrcmd> tftp getStats 
100 Ok
{1} 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 

Table 7-5 TFTP Statistics 

Digit
Attribute
Description

{1}

id

Implementation ID (release and build information).

2

server-state

State of the server (up or down).

3

server-time-since-start

Running time since last start.

4

server-time-since-reset

Running time since last reset.

5

total-packets-in-pool

Number of packets in the pool.

6

total-packets-in-use

Number of packets the server is using.

7

total-packets-received

Number of packets received since the last start or reload.

8

total-packets-sent

Number of packets sent since the last start or reload.

9

total-packets-drained

Number of packets read and discarded since the last start or reload.

10

total-packets-dropped

Number of packets dropped since the last start or reload.

11

total-packets-malformed

Number of packets received that were malformed since the last start or reload.

12

total-read-requests

Number of packets read since the last start or reload.

13

total-read-requests-completed

Number of read packets completed since the last start or reload.

14

total-read-requests-refused

Number of read packets refused since the last start or reload.

15

total-read-requests-ignored

Number of read packets ignored since the last start or reload.

16

total-read-requests-timed-out

Number of read packets that timed out since the last start or reload.

17

total-write-requests

Number of read packets that were write requests since the last start or reload.

18

total-write-requests-completed

Number of write requests completed since the last start or reload.

19

total-write-requests-refused

Number of write requests refused since the last start or reload.

20

total-write-requests-ignored

Number of write requests ignored since the last start or reload.

21

total-write-requests-timed-out

Number of write requests that timed out since the last start or reload.

22

total-docsis-requests

Number of DOCSIS requests received since the last start or reload.

23

total-docsis-requests-completed

Number of DOCSIS requests completed since the last start or reload.

24

total-docsis-requests-refused

Number of DOCSIS requests refused since the last start or reload.

25

total-docsis-requests-ignored

Number of DOCSIS requests ignored since the last start or reload.

26

total-docsis-requests-timed-out

Number of DOCSIS requests that timed out since the last start or reload.

27

read-requests-per-second

Number of read requests per second.

28

write-requests-per-second

Number of write requests per second.

29

docsis-requests-per-second

Number of DOCSIS requests per second.


Displaying IP Address Usage

Displaying IP address usage gives a good overview of how clients are currently assigned addresses.

Local Advanced and Regional Web UI

You can look at the local or regional cluster address space, or generate a subnet utilization or lease history report at the regional cluster, to determine IP address usage. These functions are available in both web UIs by clicking Address Space, if you have address space privileges at the local or regional cluster.

You can determine the current address space utilization by clicking the View icon () in the Current Usage column for the unified address space, address block, and subnet (see the "Viewing Address Utilization for Address Blocks, Subnets, and Scopes" section on page 9-12). You can also get the most current IP address utilization by querying the lease history (see the "Querying Leases" section on page 22-29). In the latter case, the regional CCM server references the appropriate DHCP server directly. To ensure this subnet-to-server mapping, you must update the regional address space view so that it is consistent with the relevant local cluster. Do this by pulling the replica address space, or reclaiming the subnet to push to the DHCP server (see the "Reclaiming Subnets" section on page 9-10). Also ensure that the particular DHCP server is running.

CLI Commands

You can generate an IP address usage report using the report command. The command has the following syntax:

report [column-separator=string | dhcpv4 | dhcp-only | dhcpv6 | file=outputfile | vpn=name 

The column-separator specifies the character string that separates the report columns (the preset value is the space character). If you want to include more than one space, precede them with the backslash (\) escape character (enclosed in quotation marks). You can specify DHCPv4 or DHCPv6 addresses (dhcp-only is the same as dhcpv4). Not specifying the VPN returns the addresses in the current VPN only.

Displaying Related Servers

Network Registrar displays the relationship among servers in a DNS zone distribution or a DHCP failover configuration. In the web UI, you can view a related servers page when you click the Related Servers icon () on various pages. You can use the display of related servers to diagnose and monitor misconfigured or unreachable servers.

See Also

Monitoring Remote Servers Using Persistent Events
DNS Zone Distribution Servers
DHCP Failover Servers

Monitoring Remote Servers Using Persistent Events

To service clients that require updates to DNS and LDAP related servers, the DHCP server uses a persistent event algorithm to ensure updates to related servers when a related server is temporarily unavailable. In addition, the algorithm prevents a misconfigured or offline DNS server from using up all the available update resources.

At startup, the DHCP server calculates the number of related servers in the configuration that require persistent events. A preconfigured Maximum Pending Events attribute (an Expert mode attribute that specifies the number of in-memory events that is preset to 40,000) is divided by the number of servers to obtain a limit on the number of events permitted for each remote server. This calculation covers related DNS and LDAP servers (DHCP failover does not use persistent storage for events). The DHCP server uses this calculation to issue log messages and take the action described in Table 7-6. The table shows a hypothetical case of a DHCP server with four related DNS servers each having a limit of 10K events.

Table 7-6 Persistent Event Algorithm 

Event Reached
DHCP Server Action

50% of the calculated per-server limit (Maximum Pending Events value divided by the number of total related servers); for example, 5K events on a related server out of a total of 40K maximum pending events

Issues an INFO log message every 2 minutes, as long as the limits are exceeded:

The queue of events for the name remote server at 
address has x events, and has reached the info 
limit of y/2 events out of an upper limit of y 
events per remote server. The remote server may be 
misconfigured, inoperative, or unreachable.

100% of the calculated per-server limit and less than 50% of the Maximum Pending Events value; for example, 10K events on a related server, with fewer than 10K total maximum pending events

Issues a WARNING log message every 2 minutes, as long as the limits are exceeded:

The queue of events for the name remote server at 
address has x events, has exceeded the limit of y 
events per remote server, but is below the limit of 
z total events in memory. The remote server may be 
misconfigured, inoperative, or unreachable.

100% of the calculated per-server limit and 50% or more of the Maximum Pending Events value; for example, 10K events on a related server, with 20K total maximum pending events

Issues an ERROR log message every 2 minutes, as long as the limits are exceeded:

The queue of events for the name remote server at 
address has x events, and has grown so large that 
the server cannot continue to queue new events to 
the remote server. The limit of y events per remote 
server and z/2 total events in memory has been 
reached. This and future updates to this server 
will be dropped. The current eventID n is being 
dropped.

The server drops the current triggering event and all subsequent events with that server.

100% of the Maximum Pending Events value; for example, 40K events across all related servers

Issues an ERROR log message:

The queue of pending events has grown so large that 
the server cannot continue to queue new events. The 
queue's size is z, and the limit is z.

The server drops all subsequent events with all related servers.


SNMP traps and DHCP server log messages also provide notification that a related server is unreachable.

DNS Zone Distribution Servers

A DNS zone distribution simplifies creating multiple zones that share the same secondary server attributes. You can view and set the primary and secondary DNS servers in a zone distribution.

Local Basic or Advanced Web UI

Click DNS, then Zone Distribution. This opens the List Zone Distributions page. The local cluster allows only one zone distribution, the default. Click this zone distribution name to open the Edit Zone Distribution page, which shows the authoritative and secondary servers in the zone distribution.

Regional Web UI

Click DNS, then Zone Distributions. This opens the List/Add Zone Distributions page (see Figure 15-7 on page 15-21). The regional cluster allows creating more than one zone distribution. Click the zone distribution name to open the Edit Zone Distribution page, which shows the primary, authoritative, and secondary servers in the zone distribution.

CLI Commands

Create a zone distribution using zone-dist name create primary-cluster, then view it using zone-dist list. For example:

nrcmd> zone-dist distr-1 create Boston-cluster 
nrcmd> zone-dist list 

DHCP Failover Servers

Related servers in a DHCP failover pair relationship can show the following information:

Type—Main or backup DHCP server.

Server name—DNS name of the server.

IP address—Server IP address in dotted octet format.

Requests—Number of outstanding requests, or two dashes if not applicable.

Communication status—OK or INTERRUPTED.

Cluster state—Failover state of this DHCP server.

Partner state—Failover state of its partner server.

For details on DHCP failover implementation, see Chapter 27, "Configuring DHCP Failover."

Local Basic or Advanced Web UI

Click DHCP, then Failover. The List DHCP Failover Pairs page shows the main and backup servers in the failover relationship.

CLI Commands

Use dhcp getRelatedServers to display the connection status between the main and partner DHCP servers. If there are no related servers, the output is simply 100 Ok.

Displaying Leases

After you create a scope, you can monitor lease activity and view lease attributes.

Local Basic or Advanced Web UI

Click DHCP, then Scopes or Prefixes (in Advanced mode). On the List/Add DHCP Scopes or List DHCPv6 Prefixes page, click the View icon () in the Leases column to open the List DHCP Leases for Scope or List DHCP Leases for Prefix page.

Regional Web UI

Click Address Space, then Lease History. Set the query parameters, then click Query Lease History. (See the "Querying Leases" section on page 22-29.)

Running Data Consistency Rules

Using consistency rules, you can check such data inconsistencies as overlapping address ranges and subnets. You can set data consistency rules at the regional and local clusters. See Table 7-7 and Table 7-8 for the consistency rules you can set in each case.

Table 7-7 Data Consistency Rule Settings at the Regional Cluster 

Regional Consistency Rule
Description

Attributes Consistency Rule

Ensures that policies and client-classes with the same names at replication have the same values.

Broadcast Address Rule

Ensures that the dynamic address range for a scope should not include a broadcast address.

Cable Helper Consistency Rule

Ensures that cable helpers and DHCP failover are consistent.

Client-Class Selection Match Tag Rule

Ensures that all client-classes have a scope-selection tag defined by some scope.

Ensure Scope for Subnet Rule

Ensures that a scope is present for every subnet.

Ensure Subnets for Scopes Rule

Ensures that a subnet is present for every scope.

IP Range Consistency Rule

Identifies any overlapping static or dynamic address ranges.

Owner Match Rule

Ensures that each subnet on the router has the same owner as the subinterface.

Router Subnets in Database Rule

Ensures that each subnet on the router has a corresponding subnet in the regional database.

Selection Tags Consistency Rule

Ensures that scope-selection tags on scopes match with one of those defined in the address types.

Subnet Consistency Rule

Identifies any overlapping subnets.

Utilization Collection Rule

Ensures that the DHCP server collection interval is greater than the regional server collection interval.


Table 7-8 Data Consistency Rule Settings at the Local Cluster 

Local Consistency Rule
Description

Broadcast Address Rule

Ensures that the dynamic address range for a scope should not include a broadcast address.

CNAME RR and Host Alias Consistency Rule

Ensures that CNAME records and host alias names are consistent.

Client-Class Selection Match Tag Rule

Ensures that all client-classes have a scope-selection tag defined by some scope.

Failover Pair Consistency Rule

Ensures that the CCM server and DHCP server have the same versions of the DHCP failover pair objects.

IP Range Consistency Rule

Identifies any overlapping static or dynamic address ranges.

Lame Delegation Rule

Ensures that no lame delegation exists for managed subzones.

Missing Hosts Consistency Rule

Identifies any missing hosts for PTR records.

Missing PTR Records Consistency Rule

Identifies any missing PTR records for hosts.

Owner Match Rule

Ensures that each subnet on the router has the same owner as the subinterface.

Router Subnets in Database Rule

Ensures that each subnet on the router has a corresponding subnet in the regional database.

Subnet Consistency Rule

Identifies any overlapping subnets.

Zone Reference Rule

Ensures the resolvability of DNS update policies, access control lists (ACLs), and encryption key objects referenced from zones.


Local Basic or Advanced and Regional Web UI


Step 1 Click Home, then Consistency Rules to open the List Consistency Rules page (see Figure 7-8 for a local cluster version of this page).

Step 2 Click check marks for each of the listed consistency rules that you want to apply (see Table 7-7 and Table 7-8).

Step 3 Click Run Rules. This opens the Consistency Rules Result page. If the CCM server finds violations, the page lists them for each rule.

Step 4 To view the network objects that caused the violations, click the + sign next to Detailed Objects under the violation. The result shows the object class name, object ID, sequence number, and the values impacted by the inconsistency.

Step 5 Click Return to return to the List Consistency Rules page.


Figure 7-8 List Consistency Rules Page (Local Basic)

Troubleshooting Servers

The following sections describe troubleshooting the configuration and the DNS, DHCP, and TFTP servers.

See Also

Immediate Troubleshooting Actions
Modifying the cnr.conf File
Troubleshooting Server Failures
Troubleshooting and Optimizing the TFTP Server
Solaris and Linux Troubleshooting Tools
Using the TAC Tool

Immediate Troubleshooting Actions

When facing a problem, it is crucial not to cause further harm while isolating and fixing the initial problem. Here are things to do (or avoid doing) in particular:

Have 512 MB or more of memory and 2.5 GB or more of a data partition.

Do not reboot a cable modem termination system (CMTS).

Enable DHCP failover.

Do not reload, restart, or disrupt Network Registrar with failover resynchronization in progress.

Modifying the cnr.conf File

Network Registrar uses the cnr.conf file for basic configuration parameters. This file is normally located in the install-path/conf directory. Network Registrar creates the file during installation and processes it line by line.

You can edit this file if configuration parameters change. Note that during normal operation, you would not want to change the values, especially sensitive ones such as the MCD key. However, certain conditions might require you to modify certain values, such as when you move the data files for disk space reasons.

The format of the cnr.conf file consists of parameter name-value pairs, one per line; for example, for a Windows local cluster installation:

cnr.rootdir=C:\\Program Files\\Network Registrar\\Local
cnr.ccm-port=1234
cnr.datadir=C:\\Program Files\\Network Registrar\\Local\\data
cnr.java-home=C:\\Program Files\\Java\\jre1.5.0_12
cnr.logdir=C:\\Program Files\\Network Registrar\\Local\\logs
cnr.https-port=8443
cnr.tempdir=C:\\Program Files\\Network Registrar\\Local\\temp
cnr.http-port=8080
cnr.ccm-mode=local
cnr.mcdpath=ccmsrv
cnr.backup-time=23:45
cnr.mcd-key=01A58EE87DD4EE78368805FD366F8505446FAD449D6C7A

Directory paths must be in the native syntax for the operating system. The format allows the use of colons (:) in directory paths, but not as name-value pair separators; it does not allow line continuation or embedded unicode characters. Other modifications to the file might include the location of the log directory (see the "Log Files" section) or the time mcdshadow backups should occur (see the "Setting Shadow Backup Times" section on page 8-3).

Troubleshooting Server Failures

The server agent processes (nwreglocal and nwregregion) normally detect server failures and restart the server. You can usually recover from the failure and the server is not likely to fail again immediately after restarting. On rare occasions, the source of the server failure prevents the server from successfully restarting, and the server fails again as soon as it restarts. In such instances, perform the following steps:


Step 1 If the server takes a significantly long time to restart, stop and restart the server agent. On:

Windows:

net stop nwreglocal or nwregregion 
net start nwreglocal or nwregregion 

Solaris:

/etc/init.d/nwreglocal stop or nwregregion stop 
/etc/init.d/nwreglocal stop or nwregregion start 

Linux:

/etc/rc.d/init.d/nwreglocal stop or nwregregion stop 
/etc/rc.d/init.d/nwreglocal stop or nwregregion start 

Step 2 Keep a copy of all the log files. Log files are located in the install-path/logs directory on Solaris and Linux, and the install-path\logs folder on Windows. The log files often contain useful information that can help isolate the cause of a server failure.

Step 3 Use the TAC tool, as described in the "Using the TAC Tool" section, or save the core or user.dmp file, if one exists, depending on the operating system:

Windows—The user.dmp file is located in the system directory, which varies depending on the Windows system. Search for this file and save a renamed copy.

Solaris and Linux—The core file is located in the install-path. Save a renamed copy of this file that Network Registrar does not overwrite.

Step 4 On Windows, use the native event logging application to save the System and Application event logs to files. You can do this from the Event Viewer. These event logs often contain data that helps debug Network Registrar server problems. For a description of the log messages for each server module, see the install-path/docs/msgid/MessageIdIndex.html file.


Troubleshooting and Optimizing the TFTP Server

You can set certain attributes to troubleshoot and optimize TFTP server performance.

See Also

Tracing TFTP Server Activity
Optimizing TFTP Message Logging
Enabling TFTP File Caching

Tracing TFTP Server Activity

To trace TFTP server activity, set the packet-trace-level attribute to a value of 1 through 4, depending on the level of verbosity you want the TFTP server to use to write messages to the trace file. The trace files are located in the /logs subdirectory of the installation directory. Windows tracing goes to the file_tftp_1_log file; Solaris and Linux tracing goes to the /var/nwreg2/{local | regional}/logs/file_tftp_1_log and file_tftp_1_trace files.

Here are the trace levels, with each higher level being cumulative:

0—Disables all server tracing (the default).

1—Displays all the log messages in the trace file.

2—Displays the client IP address and port number for all packets.

3—Displays the packet header information.

4—Displays the first 32 bytes of the packet.


Note Setting and getting the trace level only works if the TFTP server is started. Turn on packet tracing only for debugging purposes, and then not for any extended time, for performance reasons.


Optimizing TFTP Message Logging

You can improve TFTP server performance by restricting logging and tracing. By default, the server logs error, warning, and informational messages to file_tftp_1_log files. You can set the log levels using a few TFTP server parameters:

Log level (use the log-level attribute)—Master controller of server logging, which is preset to, and is best left at, level 3 (logs all error, warning, and informational messages). As with packet tracing, the higher logging levels are cumulative. If set to 0, no server logging occurs.

Log settings (use the log-settings attribute)—This is the second level of logging control and takes only two values, default or no-success-messages. The default log setting does not alter the default value of log level 3 (error, warning, and informational messages). However, you may want to disable writing success informational messages, and thereby improve server performance, by changing the log settings to no-success-messages.

Log file count and size (use the log-file-count attribute)—Sets how many log files to maintain and how large to allow them to get in the /logs directory. The default value is to maintain a maximum of four files of one megabyte each.


Note Reload the TFTP server after changing these values.


Enabling TFTP File Caching

You can improve TFTP server performance significantly by enabling file caching on the server. You must do this explicitly, because it is preset to disabled. You must also create and point to a file cache directory, and you can set the maximum size of this directory. Here are the steps:


Step 1 Determine where you want the TFTP cache files to go. This becomes a subdirectory of the TFTP home directory, which is preset to install-path/data/tftp (on Solaris and Linux, it is /var/nwreg2/{local | regional}/data/tftp). If you want a different location, set the home-directory attribute.

Step 2 Change to the TFTP home directory and create the cache directory, such as CacheDir, in the home directory, using the mkdir Cachedir command. Note that Network Registrar ignores any files in any subdirectories of this cache directory.

Step 3 Use the file-cache-directory attribute to set up the TFTP server to point to the cache directory. You cannot use absolute path or relative path in the directory name. The file-cache-directory name is either appended to the path given in the home-directory or the default home directory path (if you do not specify one).

Step 4 Use the file-cache-max-memory-size attribute to set the maximum memory size, in bytes, of the cache. The preset value is 32 KB. Network Registrar loads all files into cache that cumulatively fit this memory size. If you set the value to 0, Network Registrar does not cache any data, even if you enable file caching.

Step 5 Copy all of the files you want cached into the cache directory, and not into any subdirectory. Because all files in this directory are loaded into cache, do not include large files.

Step 6 Enable the file-cache attribute to enable file caching, then reload the server. Network Registrar logs the name of each cached file, and skips any it cannot load. It reads in all files as binary data and translates them as the TFTP client requests. For example, if a client requests a file as NetASCII, the client receives the cached data in that form.

Step 7 Writing to cache is not allowed. If you need to update a cache file, overwrite it in the cache directory, then reload the server.


Solaris and Linux Troubleshooting Tools

You can also use the following commands on Solaris and Linux systems to troubleshoot Network Registrar. To:

See all Network Registrar processes:

ps -leaf | grep nwr 

Monitor system usage and performance:

top 
vmstat 

View login or bootup errors:

On Solaris—grep /var/adm/messages*

On Linux—grep /var/log/messages*

View the configured interfaces and other network data:

ifconfig -a 

Using the TAC Tool

There may be times when any amount of troubleshooting steps will not resolve your problem and you have to resort to contacting the Cisco Technical Assistance Center (TAC) for help. Network Registrar provides a tool so that you can easily assemble the server or system error information, and package this data for TAC support engineers. This eliminates having to manually assemble this information with TAC assistance. The resulting package from this tool provides the engineers enough data so that they can more quickly and easily diagnose the problem and provide a solution.

The cnr_tactool utility is available in the bin directory of the Windows, and usrbin directory of the UNIX or Linux, installation directories. Execute the cnr_tactool utility:

> cnr_tactool -N username -P password [-d output-directory]

The output directory is optional and normally is the temp directory of the installation directories (in the /var path on Solaris and Linux). If you do not supply the username and password on the command line, you are prompted for them:

> cnr_tactool 
username:
password:
[processing messages....] 

The tool generates a packaged tar file whose name includes the date and version. The tar file contains all the diagnostic files.