Feedback
Table Of Contents
Maintaining Servers and Databases
Monitoring and Reporting Server Status
Immediate Troubleshooting Actions
Troubleshooting Server Failures
Troubleshooting and Optimizing the TFTP Server
Optimizing TFTP Message Logging
Solaris and Linux Troubleshooting Tools
Maintaining Servers and Databases
This chapter explains how to administer and control your local and regional servers' operations through Cisco CNS Network Registrar's Web UI and CLI (the nrcmd program).
Controlling Servers
If you are assigned the server-management subrole of the ccm-admin role, you can control the Network Registrar servers as follows:
•
Start—Load the database and start the server.
•
•
Starting and stopping a server is self-explanatory. When you reload the server, Network Registrar performs three steps—stops the server, loads configuration data, and restarts the server. Only after you reload the server does it use your changes to the configuration.
In the Web UI, you can manage the protocol servers in two ways:
•
Figure 6-1 Manage Servers Page (Local)
The local and regional cluster Web UI access to server administration is identical, even though the available functions are different. As a regional administrator, you can check the state and health of the regional CCM server, server agent, and Router Interface Configuration (RIC) server. However, you cannot stop, start, reload, or view statistics, logs, or interfaces for them.
At the local cluster, you can manage the DHCP, DNS, TFTP, and SNMP servers:
–
) to view statistics for the server.
–
) in the View Log column to view the log messages for the server.
–
) to start the server.
–
) to stop the server.
–
) to reload the server.
•
Figure 6-2 Manage DNS Server Page (Local)
Along with the Statistics (
) in the Commands column to open the DNS Server Commands page (see Figure 6-3).
Figure 6-3 DNS Server Commands Page (Local)
The server command functions are:
–
–
–
–
•
Along with the Statistics (
), Log (
Figure 6-4 DHCP Server Commands Page (Local)
This page provides the Get Leases with Limitation ID feature, to find clients that are associated through a common limitation identifier (see the "Administering Option 82 Limitation" section on page 23-16). Enter at least the IP address of the currently active lease in the IP Address field, then click the Run icon (
In the CLI (the regional cluster allows CCM server management only):
•
•
•
•
nrcmd> ccm set ipaddr=192.168.50.10
Note
Logging Server Events
When you start Network Registrar, it automatically starts logging Network Registrar system activity. Network Registrar maintains all the logs by default on:
•
•
Tip
Searching the Logs
The Web UI provides a convenient way to search for entries in the activity and startup log files. You can locate specific message text, log message IDs, and message timestamps using a regular expression string entry. When you click the Log icon (
) at the top of the page, enter the search string in the regular expression syntax. (For example, entering name? searches for zero or one occurrence of the string "name" in the log file.)
Clicking the Search icon opens a Log Search Result page in a separate browser window (see Figure 6-5 for an example). The page shows the log file, line number of the match, and the log number.
Figure 6-5 Log Search Result Page (Local)
Click the name of the log message, which opens the Log for Server page with the full message text. Change between Table and Text view by clicking the Log icon (
Logging Format and Settings
The server log entries include the following categories:
•
•
•
•
In the Web UI, you can affect which events to log. For example, to set the logging for the local cluster DNS and DHCP server:
•
•
In the CLI, use dns set log-settings, dhcp set log-settings, and tftp set log-settings for the respective servers.
Note
Log Files
Table 6-1 describes the Network Registrar log files in the install-path/logs directory.
Each component can generate a number of log files, each with a preconfigured maximum size of 1 MB. The first log file name has the _log suffix. When this file reaches its maximum size, it gets the .01 version extension appended to its name and a new log file is created without the version extension. Each version extension is incremented by one for each new file created. When the files reach their configured maximum number, the oldest file is deleted and the next oldest assumes its name. The usual maximum number is four for the DNS, DHCP, and TFTP servers.
You can check the configured maximums for the DNS, DHCP, and TFTP servers using [server] type serverLogs show in the CLI, which shows the maximum number (nlogs) and size (logsize) of these protocol server log files. You can adjust these parameters using [server] type serverLogs set nlogs=value and [server] type serverLogs set logsize=value. You cannot adjust these maximums for any of the other log files.
Note
Change Logs and Tasks
In the Web UI, you can view the change logs and tasks associated with configurations you make. Click Administration, and then Change Log, CCM Tasks, or MCD Tasks. To view the change log and tasks, you must be assigned the database subrole of the ccm-admin or regional-admin role. The resulting View Change Log page (see Figure 6-6) shows all the change logs, in reverse chronological order. Click the DBSN number of the change log entry to open a View Change Set page for it.
Figure 6-6 View Change Log Page (Local)
You can filter the list, manually trim it, and save it to a file. You can filter the list by:
•
•
•
•
Click Filter List or Cancel Filter (to cancel the filter that persists through the session). You can initiate a trim of the change log by setting how many days old you want the record to get before trimming it, by setting a value in the "older than" field and clicking the Delete icon (
). To save the change log entries to a comma-separated values (CSV) file, click the Save icon (
) in the left column.
If a task is associated with a change log, it appears on the View Change Set page. You can click the task name to open the View CCM Task or View MCD Task page for it.
Monitoring and Reporting Server Status
Monitoring the status of a server involves checking its:
•
•
•
•
•
•
•
Server States
All Network Registrar protocol servers (DNS, DHCP, SNMP, and TFTP) pass through a state machine consisting of the following states:
•
•
•
•
•
The two essential states are initialized and running, because the server transitions through the states so quickly that the other states are essentially invisible. Normally, when the server agent starts the server, it tells the server to be up. The server process starts, sets its state to loaded, then moves up to running. If you stop the server, it walks down the states to initialized, and if you restart, it moves up to running again. If it fails to configure for some reason, it drops back to initialized, as if you had stopped it.
There is also an exiting state that the server is in very briefly when the process is exiting. The user interface can also consider the server to be disabled, but this rarely occurs and only when there is no server process at all (the server agent was told not to start one).
Displaying Health
You can display aspects of a server's health, or how well it is running. The following items can decrement the server's health, so you should monitor their status periodically. For the:
•
•
•
–
–
–
–
•
–
–
–
–
–
–
•
–
–
–
•
Tip
In both the local and regional cluster Web UIs, click Administration, then Servers. Check the Manage Servers page for the state and health of each server (see Figure 6-1 for a DHCP server example).
In the local cluster CLI, use [server] type getHealth. The number 10 indicates the highest level of health, 0 that the server is not running.
Tip
Displaying Statistics
To display server statistics, the server must be running. In the local cluster Web UI, go to the Manage Servers page, then click the Statistics icon (
The DHCP and DNS statistics are each divided into two groups of statistics. The first group is for total statistics and the second group is for sample statistics. The total statistics are accumulated over time. The sample statistics occur during a configurable sample interval. The names of the two categories vary per server and per user interface, and are identified in Table 6-2.
To set up the sample counters, you must activate either the collect-sample-counters attribute for the server, or a log-setting attribute value called activity-summary. You can also set a log-setting value for the sample interval for each server, which defaults to 5 minutes. The collect-sample-counters attribute is set to true by default for the DNS server, but is set to false by default for the DHCP server. To enable the sample counters and set the interval for DHCP, set the following attributes:
nrcmd> dhcp enable collect-sample-countersor
nrcmd> dhcp set log-settings=activity-summaryThen:
nrcmd> dhcp set activity-summary-interval
Note
In the CLI, if you use [server] type getStats, the statistics are encoded in curly braces followed by sets of digits, as described in Table 6-3 for DNS, Table 6-4 for DHCP, and Table 6-5 for TFTP. The server type getStats all command is more verbose and identifies each statistic on a line by itself. As indicate in Table 6-3, using the additional sample keyword shows the sample statistics only.
Reset the counters using dhcp resetStats. For example:
nrcmd> dhcp resetStatsDNS Statistics
The DNS server statistics appear in Table 6-3. The table is organized by category, name and description of the statistic, and the digit position of the statistic in the output of the generic dns getStats command, in the format:
nrcmd> dns getStats100 Ok{1} 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
Table 6-3 DNS Statistics
General
Server Identifier (id)
Implementation ID (release and build information).
{1}
Recursive Service (config-recurs)
Recursion services—(1) available, (2) restricted, (3) unavailable.
2
Process Uptime (config-up-time)
Time (in seconds) elapsed since the last server startup.
3
Time Since Reset (config-reset-time)
Time (in seconds) elapsed since the last server reset (restart).
4
Server Status (config-reset)
Status or action to reinitializes any name server state—If using the (2) reset action, reinitializes any persistent name server state; the following are read-only statuses: (1) other—server in some unknown state, (2) initializing, or (3) running.
5
Authoritative Answers
(counter-auth-ans)Number of queries answered authoritatively.
6
Authoritative No Such Name Answers (counter-auth-no-names)
Number of queries returning authoritative no such name responses.
7
Authoritative No Such Data Answers (counter-auth-no-data-resps)
Number of queries returning authoritative no such name (empty answer) responses.
8
Nonauthoritative Answers (counter-non-auth-datas)
Number of queries answered nonauthoritatively (cached).
9
Nonauthoritative No Such Data Answers (counter-non-auth-no-datas)
Number of queries answered nonauthoritatively with no data.
10
Forwarded Queries (counter-referrals)
Number of queries forwarded to other servers.
11
Error Responses (counter-errors)
Number of responses answered with errors (RCODE values other than 0 or 3).
12
Single Label Name Queries (counter-rel-names)
Number of requests received for names of only one label (relative names).
13
Requests Refused (counter-req-refusals)
Number of refused queries.
14
Unparseable Requests (counter-req-unparses)
Number of unparsable requests.
15
Requests with Other Errors (counter-other-errors)
Number of aborted requests due to other errors.
16
Performance
updated-rrs
Number of resource records added and deleted, including updates, despite errors.
update-packets
Number of update packets processed.
ixfrs-out
Number of successful outbound incremental zone transfers (IXFRs).
ixfrs-in
Number of inbound IXFRs, including those resulting in full zone transfers.
ixfrs-full-resp
Number of outbound full zone transfers (AXFRs) in response to IXFR requests.
axfrs-out
Number of successful outbound AXFRs, including those counted in ixfrs-full-resp.
axfrs-in
Number of successful inbound AXFRs.
queries
Number of queries responded to, excluding updates.
xfrs-out-at-limit
Number of times that outbound transfers reached the concurrent limit.
xfrs-in-at-limit
Number of times that inbound transfers reached the concurrent limit.
notifies-out
Number of outbound notify packets.
notifies-in
Number of inbound notify packets.
Query
auth-answers
Numbered of authoritatively answered queries.
auth-no-names
Number of queries for which authoritative no such name answers occurred.
auth-no-data-responses
Number of queries for which authoritative no such name responses occurred.
nonauth-answers
Number of queries nonauthoritatively answered (from cache).
nonauth-no-data-responses
Number of queries nonauthoritatively answered, but with no data.
referrals
Number of requests referred to other servers.
relative-name-requests
Number of requests received for names that are only one label long.
refusals
Number of queries refused.
lame-delegations
Number of queries resulting in lame delegations (see the "Reporting Lame Delegation" section on page 16-11).
mem-cache-hits
Number of memory cache lookup hits.
mem-cache-misses
Number of memory cache lookup misses.
mem-cache-writes
Number of memory cache writes.
Security
rcvd-tsig-packets
Number of TSIG RRs processed (see the "Transaction Security" section on page 27-6).
detected-tsig-bad-time
Number of incoming packets with bad TSIG time stamps.
detected-tsig-bad-key
Number of incoming packets with invalid or unknown TSIG keys.
detected-tsig-bad-sig
Number of incoming packets with bad TSIG signatures.
rcvd-tsig-bad-time
Number of BADTIME errors after sending a TSIG.
rcvd-tsig-bad-key
Number of BADKEY errors after sending a TSIG.
rcvd-tsig-bad-sig
Number of BADSIG errors after sending a TSIG.
unauth-xfer-reqs
With zone transfer restrictions enabled, the number of ACL authorization failures.
unauth-update-reqs
Number of updates resulting in ACL authorization failures, or that target zones that do not support updates.
Error
update-errors
Number of updates resulting in errors or failures, including negative responses to prerequisite checks, but excluding TSIG responses.
ixfr-in-errors
Number of inbound IXFR errors, but excluding packet format errors.
ixfr-out-errors
Number of outbound IXFR errors, but excluding packet format errors.
axfr-in-errors
Number of inbound AXFR errors, but excluding packet format errors.
axfr-out-errors
Number of outbound AXFR errors, but excluding packet format errors.
sent-total-errors
Number of requests answered with errors (RCODE values other than 0, 3, 6, 7, and 8).
sent-format-errors
Number of unparsable requests received.
sent-other-errors
Number of requests aborted due to other local server errors.
sent-refusal-errors
Number of requests resulting in REFUSED responses.
rcvd-format-errors
Number of responses received with FORMERR status.
Max Counters
concurrent-xfrs-in
Maximum number of concurrent threads processing inbound transfers during the last sampling period.
concurrent-xfrs-out
Maximum number of concurrent threads processing outbound transfers during the last sampling period.
DHCP Statistics
The DHCP server statistics appear in Table 6-4. The table is organized by category, name and description of the statistic, and the digit position of the statistic in the output of the generic dhcp getStats command, in the format:
nrcmd> dhcp getStats100 Ok{1} 2 3 4 5 6 7 8
TFTP Statistics
The TFTP server statistics appear in Table 6-5. The table is organized by category, name and description of the statistic, and the digit position of the statistic in the output of the generic tftp getStats command, in the format:
nrcmd> tftp getStats100 Ok{1} 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
Displaying IP Address Usage
In the Web UI, you can look at the local or regional cluster's address space, or generate a subnet utilization or lease history report at the regional cluster, to determine IP address usage. These functions are available in both Web UIs by clicking Address Space, if you have address space privileges at the local or regional cluster.
You can determine the current address space utilization by clicking the View icon (
) in the Current Usage column for the unified address space, address block, and subnet (see the "Viewing Address Utilization for Address Blocks, Subnets, and Scopes" section on page 8-11). You can also get the most current IP address utilization by querying the lease history (see the "Querying Leases" section on page 21-21). In the latter case, the regional CCM server references the appropriate DHCP server directly. To ensure this subnet-to-server mapping, you must update the regional address space view so that it is consistent with the relevant local cluster. Do this by pulling the replica address space, or reclaiming the subnet to push to the DHCP server (see the "Reclaiming Subnets" section on page 8-9). Also ensure that the particular DHCP server is running.
In the CLI, you can generate an IP address usage report using the report command. See the Network Registrar CLI Reference for additional options you can set.
Displaying Related Servers
Network Registrar displays the relationship among servers in a DNS zone distribution or a DHCP failover configuration. In the Web UI, you can view a related servers page when you click the Related Servers icon (
) on various pages.
DNS Zone Distribution Servers
A DNS zone distribution simplifies creating multiple zones that share the same secondary server attributes. In the Web UI, you can view and set the primary and secondary DNS servers in a zone distribution:
•
•
In the CLI, create a zone distribution using zone-dist name create primary-cluster, then view it using zone-dist list. For example:
nrcmd> zone-dist distr-1 create Boston-clusternrcmd> zone-dist listDHCP Failover Servers
Related servers in a DHCP failover pair relationship can show the following information:
•
•
•
•
•
•
•
In the Web UI, click DHCP, then Failover. The List DHCP Failover Pairs page shows the main and backup servers in the failover relationship.
In the CLI, use dhcp getRelatedServers to display the connection status between the main and partner DHCP servers. If there are no related servers, the output is simply 100 Ok.
Displaying Leases
After you create a scope, you can monitor lease activity and view lease attributes.
In the Web UI:
•
•
In the CLI, use lease list to view the properties of all the available leases.
Troubleshooting Servers
The following sections describe troubleshooting the DNS, DHCP, and TFTP server.
Immediate Troubleshooting Actions
When facing a problem, it is crucial not to cause further harm while isolating and fixing the initial problem. Here are things to do (or avoid doing) in particular:
•
•
•
•
Troubleshooting Server Failures
The server agent processes (nwreglocal and nwregregion) normally detect server failures and restart the server. You can usually recover from the failure and the server is not likely to fail again immediately after restarting. On rare occasions, the source of the server failure prevents the server from successfully restarting, and the server fails again as soon as it restarts. In such instances, perform the following steps:
Step 1
•
net stop nwreglocal or nwregregionnet start nwreglocal or nwregregion•
/etc/init.d/nwreglocal stop or nwregregion stop/etc/init.d/nwreglocal stop or nwregregion start•
/etc/rc.d/init.d/nwreglocal stop or nwregregion stop/etc/rc.d/init.d/nwreglocal stop or nwregregion startStep 2
Step 3
•
•
Step 4
Troubleshooting and Optimizing the TFTP Server
You can set certain attributes to troubleshoot and optimize TFTP server performance.
Tracing TFTP Server Activity
The TFTP server has two CLI commands that help create more output to logs and can be useful in troubleshooting, although this usually impacts performance. These commands set up server packet tracing. The tftp getTraceLevel command identifies the current trace level, which by default is 0, or no tracing. The tftp setTraceLevel command sets the packet tracing to a value between 0 and 4. The trace files are located in the /logs subdirectory of the installation directory. Windows tracing goes to the file_tftp_1_log file; Solaris and Linux tracing goes to the /var/nwreg2/{local | regional}/logs/file_tftp_1_log and file_tftp_1_trace files.
Here are the trace levels, with each higher level being cumulative:
•
•
•
•
•
Note
Optimizing TFTP Message Logging
You can improve TFTP server performance by restricting logging and tracing. By default, the server logs error, warning, and informational messages to file_tftp_1_log files. You can set the log levels using a few TFTP server parameters:
•
•
•
Note
Enabling TFTP File Caching
You can improve TFTP server performance significantly by enabling file caching on the server. You must do this explicitly, because it is disabled by default. You must also create and point to a file cache directory, and you can set the maximum size of this directory. Here are the steps:
Step 1
Step 2
Step 3
Step 4
Step 5
Step 6
Step 7
Solaris and Linux Troubleshooting Tools
You can also use the following commands on Solaris and Linux systems to troubleshoot Network Registrar. To:
•
ps -leaf | grep nwr•
topvmstat•
–
–
•
ifconfig -aUsing the TAC Tool
There may be times when any amount of troubleshooting steps will not resolve your problem and you have to resort to contacting the Cisco Technical Assistance Center (TAC) for help. Network Registrar provides a tool so that you can easily assemble the server or system error information, and package this data for TAC support engineers. This eliminates having to manually assemble this information with TAC assistance. The resulting package from this tool provides the engineers enough data so that they can more quickly and easily diagnose the problem and provide a solution.
The cnr_tactool utility is available in the bin directory of the Windows, and usrbin directory of the UNIX or Linux, installation directories. Execute the cnr_tactool utility:
> cnr_tactool -N username -P password [-d output-directory]The output directory is optional and normally is the temp directory of the installation directories (in the /var path on Solaris and Linux). If you do not supply the username and password on the command line, you are prompted for them:
> cnr_tactoolusername:password:[processing messages....]The tool generates a packaged tar file whose name includes the date and version. The tar file contains all the diagnostic files.
