Table Of Contents
Administering Network Registrar
Administering Clusters
Adding Clusters
Connecting to a Cluster
Viewing the Connectivity of a Cluster
Disconnecting from a Cluster
Removing a Cluster
Using Session Assert Commands for Data Management
Administering Users
Adding an Administrator
Adding a Password Without Exposing It
Changing the Administrator Password
Listing and Deleting Network Administrators
Exiting Network Registrar
Controlling Servers
Starting Servers
Stopping Servers
Reloading Servers
Logging Server Events
Event Logging Format
Log Files
Monitoring and Reporting Server Status
Adding Servers to the Server Status Monitor
Removing Servers from the Server Status Monitor
Displaying the Server's Health
Displaying Server Statistics
Displaying IP Address Usage
Displaying Related DHCP Servers
Displaying Leases
Backing up the Network Registrar Databases
Using Third Party Backup Programs With mcdshadow
Before Performing a Backup or Recovery
Backup Strategy
Setting the Shadow Backup Time
Performing a Manual Backup
Database Recovery Strategy
Backing up and Recovering MCD Data
Checking MCD Database Integrity
Recovering MCD Data
MCD Data Files
Backing up CNRDB Data
Recovering CNRDB Data
Virus Scanning While Running Network Registrar
Troubleshooting Servers and Databases
Immediate Troubleshooting Actions
Troubleshooting Server Failures
Troubleshooting and Optimizing the TFTP Server
Tracing TFTP Server Activity
Optimizing TFTP Message Logging
Enabling File Caching
Using the mcdadmin Tool
Using the keybuild Tool
Using the dbcheck Tool
Solaris and Linux Troubleshooting Tools
Administering Network Registrar
This chapter explains how to administer and control your servers' operations through Network Registrar's CLI (the nrcmd program) and GUI (the ntwkreg program).
Table 4-1 lists the major Network Registrar server administration tasks and the sections where you can find procedures on how to accomplish these tasks.
Also, see Chapter 3, "Network Registrar User Interfaces."
Administering Clusters
A Network Registrar cluster consists of the following:
•
Network Registrar DNS, DHCP, and TFTP servers. These servers often run on the same physical machine. In this case, cluster refers to the physical machine.
•
MCD and CNRDB persistent stores that contain configuration and state data for the DNS and DHCP servers.
•
AIC Server Agent that starts and stops the servers and provides a standard control interface to them.
Adding Clusters
Each cluster requires a username and password combination that controls access to the cluster.
•
Enter localhost as the cluster name if the cluster is on the same machine as the user interface.
•
If the cluster is on a different machine than the user interface, ensure that the cluster hostname does not include the "at" (@) symbol. This is because Network Registrar concatenates the name of the cluster with the name of the server in each cluster with the symbol.
Using the CLI
When you invoke the CLI, you specify the cluster and connect to it at the same time. See the "Invoking the CLI" section on page 3-1. In interactive mode, you would use the following command:
> nrcmd -C cluster -N username -P password
Using the GUI
In the Server Manager window (Figure 3-1 on page 3-6), click Add on the toolbar, or right-click the List of Clusters object and click Add Cluster. This opens the Add Cluster dialog box (Figure 4-1). See also the "Starting the GUI" section on page 3-6.
Figure 4-1 Add Cluster Dialog Box
Add the hostname of the cluster (localhost if it is the same machine as the GUI). If you want to connect to the cluster immediately, check the "Connect to this cluster once added" box. If the cluster does not exist on the network, you cannot connect to it.
Connecting to a Cluster
After you add a cluster to Network Registrar, you must connect to it before you can configure or administer it. If you try to connect to a cluster that someone else is using, Network Registrar warns you that the cluster is locked and tells you who is holding the lock. The format of the warning message is:
username@machine-name.process-id
•
On Solaris, the machine-name is where the GUI executable resides, which can be different than the machine displaying the GUI. On Windows, the GUI always runs and displays on the same machine. Linux does not currently support the GUI.
•
The process-id identifies the unique process on machine-name in which the CLI or GUI is running.
Tip
If someone else is using the cluster, disconnect from it. If you want to connect to a locked cluster, contact the person who is currently connected and request that he or she disconnect from it. You can override the lock using the force-lock command, but you should do so only if you know that no one else is editing the cluster; for example, if the other system crashed while the cluster was connected.
Using the CLI
Use the nrcmd -C command to connect to a cluster. For example, to connect to the cluster named mycluster, enter the following command from a command line prompt on Windows:
C:\Program Files\Network Registrar\bin> nrcmd -C mycluster
Tip
You can include the -r option on the command line to indicate read-only access to the cluster.
If you are connecting to a cluster for the first time, you also need to add a license key. Network Registrar licensing controls your ability to configure your servers. Every copy of Network Registrar requires a license. The license key is located on the back of the software CD-ROM case.
•
If you have a permanent license, you will not see the license dialog box again.
•
If you have an evaluation copy of Network Registrar, you have a license that will expire.
•
If you have an invalid or missing license key, you cannot configure or manage the Network Registrar servers. However, the servers will continue to function normally.
Add the license key using the license set key command.
nrcmd> license set key=license-key
Once entered, you can confirm your settings.
nrcmd> license get expiration
expiration = Thu Dec 03 19:00:00 2037
Using the GUI
Step 1
From the Admin menu, click Clusters.
Step 2
Click the cluster in the Clusters dialog box.
Step 3
Click Connect.
Step 4
If necessary, complete the Login for Cluster dialog box with the username and password, and whether you want read-only access to the cluster. If read-only, the Server Manager window indicates it after the cluster name. Note that read-only access provides limited functionality.
Step 5
Click Close in the Clusters dialog box.
Viewing the Connectivity of a Cluster
To see if a cluster is connected, you can view its state.
Using the CLI
Invoking the CLI successfully means that you are connected to a cluster. To show the cluster to which you are connected, use the session show command to show all the attributes, or use the session get command to confirm each attribute separately.
current-namespace = global
Using the GUI
In the Server Manager window (Figure 3-1 on page 3-6), from the Admin menu, click Clusters. The State column indicates the state of the cluster—Connected or Disconnected. Then, click Close.
Disconnecting from a Cluster
When you disconnect from a cluster, you can no longer configure or administer it from that workstation. Another user can then administer the cluster.
Using the CLI
Use the exit command to disconnect from a cluster.
Using the GUI
In the Server Manager window (Figure 3-1 on page 3-6), right-click the cluster from which you want to disconnect. Then, click Disconnect. If this takes you to the Clusters dialog box, click Close.
Timesaver
Right-click on an object to select from the floating menu instead of using the menu bar.
Removing a Cluster
When you remove a cluster, the user interface no longer knows about it, and its name does not appear in the Server Manager.
Using the GUI
Step 1
In the Server Manager window (Figure 3-1 on page 3-6), right-click the cluster you want to remove. Alternatively, from the Admin menu, click Clusters, then choose the cluster.
Step 2
Click Remove.
Step 3
Click Yes in the Network Registrar dialog box.
Step 4
If you are in the Clusters dialog box, click Close.
Using Session Assert Commands for Data Management
You can use the CLI session assert commands to simplify interactions with external data management processes. It also helps in writing multicommand batch scripts that stop processing if an asserted precondition fails. You generally use these commands with the default session format of script. See the session show command in the "Viewing the Connectivity of a Cluster" section. If the assertion passes, you get a "100 Ok" message. If it fails, you get a "107 Assertion Failed xxx.dbsn (minor-serial-number) = value" message and the CLI exits.
Using a CLI Command File (Batch Script)
The session assert locked command exits the CLI if it cannot lock the session. The following is a sample command file that performs batch operations requiring a lock.
session set default-format=script
commands-that-require-a-lock
For example, the session assert dhcp.dbsn command exits the CLI session if the minor serial number of the DHCP server does not match (==) or does not exceed (!=) the value given. The minor serial number is incremented with each configuration change. Get its value using the dhcp get dbsn command. The following is a sample script for modifying a DHCP server based on configuration version 1234.
session set default-format=script
session assert dhcp.dbsn == 1234
scope scope1 create 192.168.1.0 255.255.255.0
scope scope1 addRange 192.168.1.10 192.168.1.200
The following is a sample script for listing DHCP configuration changes made since version 1234.
session set default-format=script
session assert dhcp.dbsn != 1234
Administering Users
You can add administrators and change their passwords for the cluster.
Adding an Administrator
You can add administrators responsible for the clusters.
Using the CLI
Use the admin name create command to create an administrator and associated password. Also, see the "Adding a Password Without Exposing It" section. To show the administrator and password (as asterisks), use the admin name [show] or admin name get password command.
nrcmd> admin bob create password=xyz
nrcmd> admin bob get password
Using the GUI
Step 1
From the Admin menu, click Add Administrator.
Step 2
Enter the administrator's username. You can choose any text string for the administrator's name.
Step 3
Enter the administrator's password. Network Registrar uses the password to authenticate the names. If you create an administrator without a password, Network Registrar cannot authenticate the name and denies that user access to the cluster.
Step 4
Enter the password a second time.
Step 5
Choose the clusters that the administrator can access. You can only choose clusters that were added. See the "Adding Clusters" section. You do not have to be connected to the cluster to add an administrator.
Step 6
Click Add.
Adding a Password Without Exposing It
You can add a password without exposing it on the screen.
Using the CLI
Create an administrator and omit the password. See the "Adding an Administrator" section. Then, use the admin name enterPassword command to enter a password to prevent echoing it on the screen. You are prompted to verify the password.
nrcmd> admin bob enterPassword
Changing the Administrator Password
You can change any administrator's password. You should do this for your security.
Using the CLI
Use the admin name set password command to change an existing password. This takes a plain text value. If you do not want to expose this password value when you enter it on the screen, see the "Adding a Password Without Exposing It" section.
nrcmd> admin bob set password=plaintext-password
Using the GUI
Step 1
From the Admin menu, click Change Administrator Password.
Step 2
Enter the administrator's current, old username and password.
Step 3
Enter the administrator's new password and re-enter it in the following field.
Step 4
Choose the cluster the administrator can access.
Step 5
Click OK.
Listing and Deleting Network Administrators
You can list the Network Registrar administrators and delete specific ones, if necessary.
Using the CLI
You can check the administrators created using the admin list command, or just their names using the admin listnames command, and delete a specific one using the admin name delete command.
Exiting Network Registrar
Exiting the Network Registrar user interface does not affect your network servers' or your hosts' ability to request leases or access the Internet.
Using the CLI
Use the exit command to exit Network Registrar. This writes all your unsaved changes to the database. If Network Registrar cannot save your changes, the same error code appears as for the save command.
Using the GUI
From the Admin menu, click Exit. Network Registrar may prompt you to save your changes.
Controlling Servers
You can control the Network Registrar servers as follows:
•
Start—Load the database and start the server
•
Stop—Stop the server
•
Reload—Stop and restart the server
Tip
You can get the Network Registrar software version of each of the servers using the server type get version command, where type is DNS, DHCP, or TFTP.
Starting Servers
This section describes how to start the Network Registrar servers.
Note
The DNS and DHCP servers are enabled by default to start on a reboot. The TFTP server is not enabled by default to start on a reboot. You can change this using the server type enable or disable start-on-reboot command in the CLI.
Using the CLI
Use the server type start command (or simply server-type start, such as dhcp start) to start the specified server.
Timesaver
Because the server command requires the server type keyword after it (dns, dhcp, or tftp), you can omit the server keyword and simply start the command with the server type keyword.
Using the GUI
In the Server Manager window, right-click the server you want to start, click Start, then OK.
Stopping Servers
This section describes how to stop the Network Registrar servers.
Using the CLI
Use the server type stop (or simply server-type stop, such as dhcp stop) command to stop the specified server. It is advisable to save the server first.
Using the GUI
In the Server Manager window, right-click the server you want to stop. Alternatively, choose the server, then click the Servers menu. Then click Stop, Yes to confirm, and OK.
Reloading Servers
When you reload the server, Network Registrar performs three steps—stops the server, loads configuration data, and restarts the server. Only after you issue the reload command does the server use your changes to the configuration.
Using the CLI
Use the server type reload (or simply server-type reload) command to reload the specified server. Network Registrar stops the server you chose, loads the configuration data, and restarts the server.
nrcmd> server dhcp reload
or
Using the GUI
In the Server Manager window, right-click the server you want to reload. Click Reload, then OK.
Logging Server Events
When you start Network Registrar, it automatically starts logging system activity. Network Registrar maintains all the logs by default:
•
On UNIX or Linux—/var/nwreg2/logs—To view these logs, use the tail -f command
•
On Windows—Program Files\Network Registrar\logs
Caution 
To avoid filling up the Windows Event Viewer, in the Event Log Settings, check the "Overwrite Events as Needed" box. Otherwise, you might fill up your disk with log messages and prevent Network Registrar from running. Also, preserve the UNIX or Linux core file, located by default in the /
install-path/nwreg2/ directory, by renaming it to prevent it from being overwritten.
Event Logging Format
The server log entries include the following categories:
•
Activity—Logs the activity of your servers.
•
Info—Logs standard operations of the servers, such as starting up and shutting down.
•
Warning—Logs warnings, such as invalid packets, user miscommunication, or an error in a script while processing a request.
•
Error—Logs events that prevent the server from operating properly, such as out of memory, unable to acquire resources, or errors in configuration.
Using the CLI
The DNS, DHCP, and TFTP servers have log settings that can severely restrict what is logged, and thereby improve server performance. These log settings are available using the dns set log-settings, dhcp set log-settings, and tftp set log-settings commands in the CLI, respectively. See the Network Registrar CLI Reference Guide for details.
Note
Warnings and errors go to Syslog on Solaris or the Event Viewer on Windows. See the "Caution".
Log Files
Table 4-2 describes the Network Registrar log files.
Table 4-2 Log Files
In this component...
|
This file...
|
Contains...
|
Server Agent
|
agent_server_log
|
Data about when the servers started and stopped. Located in the logs directory.
|
WIN32 GUI
|
aicwin32gui_log
|
GUI messages and only logs activity on the server PC, not from the remote GUI. Located in the logs directory.
|
MCD database
|
config_mcd_log
|
Management system configuration and start, stop, and GUI login. Located in the logs directory
|
CNRDB database
|
log.0000000001 through log.9999999999
|
CNRDB database log files. DHCP logs are located in the dhcp subdirectory, DNS logs in the dns subdirectory, of the data directory. Note that these files are not humanly readable.
|
DNS server
|
name_dns_1_log
|
Server state, DDNS updates, and zone transfers. Located in the logs directory.
|
DHCP server
|
name_dhcp_1_log
|
Server state, new leases, and lease renewal. Located in the logs directory.
|
TFTP server
|
file_tftp_1_log file_tftp_1_trace
|
Server logging and tracing files. If created, located in the tftp subdirectory of the logs directory.
|
Each component can generate a number of log files, each with a preconfigured maximum size of one megabyte. The first log file name has the _log suffix. When this file reaches its maximum size, it gets the .01 version extension appended to its name and a new log file is created without the version extension. Each version extension is incremented by one for each new file created. When the files reach their configured maximum number, the oldest file is deleted and the next oldest assumes its name. The usual maximum number is three for the DNS server, and four for the DHCP and TFTP servers.
Note
Some user commands can create "User authentication" entries in the Server Agent log because of separate connections to the cluster. These should not be interpreted as a violation of system security by another user.
Using the CLI
You can find out how the server maximums are set using the [server] type serverLogs show command and looking at the number (nlogs) and size (logsize) parameters. Follow this procedure.
Step 1
Stop the Network Registrar Server Agent.
•
On Windows:
net stop "AIC Server Agent 2.0"
•
On Solaris:
/etc/init.d/aicservagt stop
kill -9 pid pid ... any processes left running in this ps....
•
On Linux:
/etc/rc.d/init.d/aicservagt stop
kill -9 pid pid ... any processes left running in this ps....
Step 2
Run the CLI command for the server type.
nrcmd> dhcp serverLogs show
Step 3
Change these maximums, if needed, using the [server] type serverLogs nlogs=value logsize=value command. You can set the nlogs value from between 2 and 100.
nrcmd> dhcp serverlogs nlogs=6 logsize=200000
Step 4
Restart the Server Agent.
•
On Windows:
net start "AIC Server Agent 2.0"
•
On Solaris:
/etc/init.d/aicservagt start
•
On Linux:
/etc/rc.d/init.d/aicservagt start
Monitoring and Reporting Server Status
You can monitor the state of your Network Registrar servers by displaying or reporting aspects of a server's health. The following items can decrement the server's health, so you should monitor their status periodically:
•
For a DNS server:
–
Memory
–
Inability to contact its root servers
•
For a DHCP server:
–
Configuration errors
–
Memory
–
Packet caching low
–
Options not fitting in the stated packet limit
–
No more leases available
•
For a TFTP server:
–
Memory
–
Socket read or write error
–
Exceeding the overload threshold and dropping request packets
Tip
On a UNIX machine, you can run the aicstatus command in the /opt/nwreg2/usrbin/ directory to see if your server is running. See the Network Registrar Installation Guide for details.
Adding Servers to the Server Status Monitor
You can view a server's status by adding it to the Server Status Monitor in the GUI.
Using the GUI
The Status Monitor window (Figure 3-2 on page 3-6) is where you can place server icons to monitor their state. The icons change to reflect the server's current state. The traffic lights indicate the server's state—started is green, stopped is red. The bar to the right of the traffic light shows the server's health, which is a combination of server resources and network balance.
To add a server to the Status Monitor, right-click the server in the Server Manager window. (Alternatively, choose the server and click the Servers menu.) Then, click Add to Status Monitor. (Alternatively, drag the server icon to the Status Monitor window.)
You can add as many servers as you want to the Status Monitor window. They can be from any of the clusters to which you connected.
Tip
When Network Registrar cannot contact the server, you see the warning triangle and exclamation point, and the green or red color is muted. The warning can mean that the network is down, the server machine crashed, the server was stopped from the control panel, or the client lost its address (hence, its communication with the server).
Removing Servers from the Server Status Monitor
You can remove a server from the Server Status Monitor in the GUI. In the Status Monitor window (Figure 3-2 on page 3-6), right-click the server that you want to remove. The, click Remove.
Displaying the Server's Health
You can display a server's health (how well it is running) by using the [server] type getHealth command in the CLI. The number 10 indicates the highest level of health, 0 that the server is not running.
Tip
Use any descending health values as a reminder to check the log files for the particular server.
Using the CLI
Displaying Server Statistics
You can display statistics about a server.
Using the CLI
Use the [server] type getStats command to display the specified server's statistics. The statistics for the DNS and DHCP servers are encoded in curly braces followed by sets of digits, as described in Table 4-3 for DNS, Table 4-4 for DHCP, and Table 4-5 for TFTP.
{Cisco Systems, Inc. DNS Server, Release 5.5 nightly build #113902, Jan 25 2002
16:42:41} 3 82245 82236 4 0 0 0 0 0 0 0 0 0 0 0
Table 4-3 DNS Statistics
Position...
|
Means...
|
{1}
|
Implementation ID (release and build information)
|
2
|
Recursion services (3 in the example)
|
3
|
Server up time, in seconds (82245 in the example)
|
4
|
Server reset time, in seconds (82236 in the example)
|
5
|
Server reset number, as a frequency (4 in the example)
|
6
|
Requests authoritatively answered (0 in the example)
|
7
|
Requests answered with "no such name" (0 in the example)
|
8
|
Requests answered with "no such data" (0 in the example)
|
9
|
Requests nonauthoritatively answered (0 in the example)
|
10
|
Requests nonauthoritatively answered with "no such data" (0 in the example)
|
11
|
Requests referred to other servers (0 in the example)
|
12
|
Requests answered with errors (0 in the example)
|
13
|
Requests for names one label long (0 in the example)
|
14
|
Requests refused (0 in the example)
|
15
|
Requests received unparsable (0 in the example)
|
16
|
Requests aborted for other errors (0 in the example)
|
{Mon Jan 31 13:32:18 2000} 0 0 0 0 0 0 0
Table 4-4 DHCP Statistics
Position...
|
Means...
|
{1}
|
Server start date and time
|
2
|
DISCOVER packets (0 in the example)
|
3
|
REQUEST packets (0 in the example)
|
4
|
RELEASED packets (0 in the example)
|
5
|
OFFER packets (0 in the example)
|
6
|
ACK (acknowledgement) packets (0 in the example)
|
7
|
NACK (negative acknowledgement) packets (0 in the example)
|
8
|
DECLINE packets (0 in the example)
|
{Cisco Systems, Inc. TFTP Server, Release 5.5 nightly build #113902, Jan 25 2002
16:53:05} 4 7 6 512 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Table 4-5 TFTP Statistics
Position...
|
Means...
|
{1}
|
Implementation ID (release and build information)
|
2
|
Server state (4 in the example)
|
3
|
Time since reboot (7 in the example)
|
4
|
Time since reset (6 in the example)
|
5
|
Packets available (512 in the example)
|
6
|
Total packets received (0 in the example)
|
7
|
Total packets sent (0 in the example)
|
8
|
Total packets drained (0 in the example)
|
9
|
Total packets dropped (0 in the example)
|
10
|
Total packets malformed (0 in the example)
|
11
|
Total read requests (0 in the example)
|
12
|
Total read requests completed (0 in the example)
|
13
|
Total read requests refused (0 in the example)
|
14
|
Total read requests ignored (0 in the example)
|
15
|
Total read requests timed-out (0 in the example)
|
16
|
Total write requests (0 in the example)
|
17
|
Total write requests completed (0 in the example)
|
18
|
Total write requests refused (0 in the example)
|
19
|
Total write requests ignored (0 in the example)
|
20
|
Total write requests timed-out (0 in the example)
|
21
|
Total DOCSIS requests (0 in the example)
|
22
|
Total DOCSIS requests completed (0 in the example)
|
23
|
Total DOCSIS requests refused (0 in the example)
|
24
|
Total DOCSIS requests ignored (0 in the example)
|
25
|
Total DOCSIS requests timed-out (0 in the example)
|
26
|
Read requests per second (0 in the example)
|
27
|
Write requests per second (0 in the example)
|
28
|
DOCSIS requests per second (0 in the example)
|
Using the GUI
In the Server Manager window (Figure 3-1 on page 3-6), right-click the server whose statistics you want to view. Then, click Show Statistics. In the Statistics window, click Refresh to refresh the statistics.
Displaying IP Address Usage
You can generate an IP address usage report.
Using the CLI
Use the report command to display the IP address usage for specified servers. See the Network Registrar CLI Reference Guide for additional options you can set.
Displaying Related DHCP Servers
Network Registrar displays a report about related servers in a DHCP failover configuration that contains the following information:
•
Type—Main or backup DHCP server
•
Server name—DNS name of the server
•
IP address—Server's IP address in dotted octet format
•
Requests—Number of outstanding requests, or two dashes if not applicable
•
Communication status—OK or INTERRUPTED
•
Cluster state—Failover state of this DHCP server
•
Partner state—Failover state of its partner server
Using the CLI
Use the dhcp getRelatedServers command to display the connection status between the main and partner DHCP server. If there are no related servers, the output is simply "100 Ok." For details on the output from this command, see the "Monitoring Failover Server Status" section on page 11-6.
nrcmd> dhcp getRelatedServers
Using the GUI
In the Server Manager window, right-click the DHCP server for which you want to show the related server. Click Show related servers. This option is only available if you configured DHCP failover. See Chapter 11, "Configuring DHCP Failover." Network Registrar displays the Related Servers window. Network Registrar refreshes the window every ten seconds. If you want to refresh the window immediately, click Refresh.
Displaying Leases
After you establish a scope, you can monitor lease activity and view lease attributes.
Using the CLI
Use the lease list command to view the properties of all leases.
Using the GUI
In the Server Manager window, right-click the scope and click Properties. In the Scope Properties dialog box, click the Leases tab (Figure 4-2) and select the lease. Then, click Lease Properties. The properties of the lease you chose appear (Figure 4-3).
Figure 4-2 Leases Tab (Scope Properties)
Figure 4-3 Lease Properties Dialog Box (Scope Properties)
Backing up the Network Registrar Databases
Because the Network Registrar database does a variety of memory caching and may be active at any time, you cannot rely on system backups to protect the database. For one thing, there may be Network Registrar operations in progress during the backup, causing backup data inconsistency and an unusable replacement database.
For this purpose, Network Registrar provides a shadow backup facility, mcdshadow. Once a day, at a configurable time, Network Registrar suspends all activity to the database and takes a snapshot of the critical files. This snapshot is guaranteed to be a consistent view of the database and is preserved correctly on a system backup tape. Network Registrar backs up the DNS data even when you run the shadow backup on a secondary server. This backup is only a single generation backup. To maintain multiple backup versions, you must implement an archiving strategy.
Using Third Party Backup Programs With mcdshadow
You should avoid scheduling third party backup programs while mcdshadow is operating, with a margin of an hour in either direction. As described in the "Setting the Shadow Backup Time" section, the default shadow backup time is daily at 23:45.
Caution 
Configure third party backup programs to skip the Network Registrar operational database directories and files, and to back up only their shadow copies. Otherwise, your server might crash. The operational files are listed in the
"Backup Strategy" section. The shadow copies that you can back up are the db.bak, dhcp.bak, and dns.bak files in the .../data directory.
Before Performing a Backup or Recovery
Before undertaking a database backup or recovery, be sure to understand that the notation ".../data/db" in the following sections refers to directories in the Network Registrar product installation path, depending on the operating system:
•
On Windows: ".../data/db"—Means \Program Files\Network Registrar\data\db
•
On UNIX and Linux: ".../data/db"—Means /var/nwreg2/data/db
Network Registrar database utility programs mentioned in the following sections are located in the ".../bin" directory, which you run as its full path name:
•
On Windows: ".../bin/program"—Means \Program Files\Network Registrar\bin\program
•
On UNIX and Linux: ".../bin/program"—Means /opt/nwreg2/bin/program
Note
Use only the approved utilities for each type of database, as described in the following sections.
Backup Strategy
The backup strategy involves backing up three databases using the mcdshadow utility:
•
MCD database—...data/db
•
CNRDB database—...data/dhcp/ndb and data/dns/ndb
•
DNS zone checkpoint database—...data/dns/zchk
The most basic component of a backup strategy is the daily shadow backup. When problems occur with the operational database, you might need to try recovering based on the previous day's shadow backup. Therefore, you must recognize and correct any problems that prevent a successful backup. The most common problem is disk space exhaustion. To get a rough estimate of disk space requirements, sum up the sizes of the following MCD and CNRDB database files:
.../data/dhcp/ndb/dhcp.ndb
.../data/dhcp/ndb/logs/log.*
.../data/dns/ndb/log/log.*
A conservative approach is ten times the space consumed by these combined database files. System load, such as usage patterns, application mix, and the load on Network Registrar itself, may dictate that a much larger reserve of space be available.
You should regularly archive existing shadow backups (such as to tape, other disks, or other systems) to preserve them for possible future recovery purposes.
Caution 
There are different database technologies Network Registrar uses with distinct sets of utility programs, as described in the following sections. Using a utility on the wrong type of database can cause database corruption. Use only the utilities indicated. Also, never use the database utilities on the operational database, only on a copy.
Setting the Shadow Backup Time
You can set the time at which an automatic shadow backup should occur through a single entry in the system registry, at the following paths.
•
On Windows:
HKEY_LOCAL_MACHINE/SOFTWARE/American Internet/Network Registrar/2.0/DBShadowTime
•
On UNIX and Linux:
/opt/nwreg2/conf/aic.conf
The registry entry is a string that represents the time of day at which the shadow backup is scheduled to occur, in 24-hour HH:MM format. The default is 23:45. If you remove this registry entry or set it to an illegal value (for example, if it does not begin with a digit), you suppress the backups. The server is otherwise unaffected.
Performing a Manual Backup
You can also initiate a manual shadow backup with the mcdshadow utility. There are no command line arguments. Enter the mcdshadow command at the command prompt to perform the shadow backup. Because a full copy of the database is created, the backup may take a few minutes.
Database Recovery Strategy
The database recovery strategy involves manually restoring up to three Network Registrar databases using tools specific to each type of database. This is different from the backup procedure in which one mechanism backs up all the server data. The two database types are called MCD and CNRDB. The three databases are:
•
MCD Configuration database
•
DNS CNRDB database
•
DHCP CNRDB database
The MCD database generally store server configuation data. The two CNRDB databases are store state data for the DNS and DHCP servers.
Depending on what occurred to case the database recovery event, you may need to apply one of the following options to all or just one of the databases:
•
Restore the integrity of one or more of the current database files—For example, if the primary server has a hard disk failure and the Network Registrar data resides on a secondary disk, you can repair the current configuration and state databases once the system is back on line.
•
Restore one or more backed up databases—If the database is a state database, you can bring the state back up to dateusing the appropriate transaction logs.
•
Restore backed up system configuration and state databases—For example, if a system hard disk crashes and the Network Registrar database was on that disk, you can restore the database from a backup copy.
Each of the recovery options follows the same general approach. First, stop the Network Registrar server agent. Then, restore or repair the data. Finally, restart the server agent and monitor the server for errors.
After you are confident that you executed a sucessful database recovery, always manually execute the mcdshadow utility to make a backup of the current configuration and state.
Backing up and Recovering MCD Data
The operational database that the mcdshadow utility uses is in .../data/db and the shadow copy is in .../data/db.bak. The actual file names are mcddb.d01, mcddb.d02, and mcddb.d03.
The mcdshadow utility generates an SNMP trap if it finds inadequate disk space for the backup, provided that the trap is configured.
Checking MCD Database Integrity
You can use the dbcheck utility to check the integrity of the MCD database. Stop all Network Registrar servers. Go to the .../data/db directory for the MCD database. As a safety check, enter the command .../bin/dbcheck -a mcddb. This requires root privileges.
Caution 
For the MCD database, use only the
dbcheck and
keybuild utilities. Do not use the cnrdb_archive, cnrdb_recover, or cnrdb_check utility that applies to the CNRDB database. See the
"Troubleshooting Servers and Databases" section for details.
Network Registrar also maintains lock files in the /tmp directory on Solaris and Linux, and the \Temp directory on Windows. They are recreated during reboot, but are vital while a system is running. Any maintenance process (such as virus scanning and archiving) should exclude this temporary directory.
Recovering MCD Data
Use the shadow backup to recover MCD data, either because a system crash corrupted the regular working database or the disk on which it resides is corrupted.
Step 1
Stop all Network Registrar servers. Be sure that enough disk space is available for two copies of the database files, plus a 15% safety margin.
Step 2
Ensure that the following three files are in .../data/db.bak—mcddb.d01, mcddb.d02, and mcddb.d03.
Step 3
Copy these files to .../data/db. Do not move them, because you may need them again.
Step 4
Change the directory to .../data/db.
Step 5
To rebuild the key files, enter the command .../bin/keybuild mcddb.
Step 6
As a safety check, enter the command .../bin/dbcheck mcddb. This requires root privileges.
You should not have errors. If you do, ensure that:
•
The Network Registrar Server Agent (aicservagt) is stopped.
•
You are working in the .../data/db directory.
•
The mcddb.dbd file is present in the .../data/db directory.
MCD Data Files
You need the files listed in Table 4-6 for a fully functioning MCD database. As described earlier, only a small subset of these files are present in a shadow backup. You need at a minimum the mcddb.dbd, mcddb.d0x, and mcdschema.txt files to rebuild the database from a backup.
Table 4-6 MCD Data Files in .../data/db
This file...
|
Is or includes...
|
mcddb.dbd
|
Template file that describes the low level data schema for the MCD runtime library.
|
mcddb.k01-k03
|
Key files that contain the redundant data—Network Registrar does not back up these files because they can be completely rebuilt with the keybuild command.
|
mcddb.d01-d03
|
MCD data repository files.
|
mcdConfig.txt
|
Text file from which Network Registrar configures the initial install time database.
|
mcdschema.txt
|
Text file containing a version number of the schema contained in the mcddb.dbd file—Network Registrar does not try to open the database unless the number in this file matches a constant that is hard-coded in the libraries. If the result of the mcdshadow utility, which just copies the data files, is separated from its original mcdschema.txt, you cannot run Network Registrar.
|
vista.taf, tcf, tjf
|
Working files used by the MCD runtime to ensure transactional integrity.
|
Backing up CNRDB Data
In the case of the CNRDB database, the mcdshadow utility copies the database and all log files to a secondary directory in the directory tree of the installed Network Registrar product.
•
For DHCP—The operational database is in .../data/dhcp/ndb and the associated operational logs are in .../data/dhcp/ndb/logs. The database shadow copy is in .../data/dhcp.bak/ndb and the associated log shadow copy is in .../data/dhcp.bak/ndb/logs.
•
For DNS—The operational database is in .../data/dns/ndb, the changeset database (also updated during an mcdshadow backup) is in the .../data/dns/ndb/dns.ndb file, and the associated operational logs are in .../data/dns/ndb/log. The database shadow copy is in .../data/dns.bak/ndb and the associated log shadow copy is in .../data/dns.bak/ndb/logs. The zone checkpoint files are in the .../data/dns/zchk directory, and their shadow copies are in .../data/dns.bak/zchk.
The actual file naming convention is:
•
Database: dhcp.ndb and dns.ndb
•
Log files: log.0000000001 through log.9999999999—Typically only a small number of log files are present on a system. The specific file name extensions in use at a site vary over time as the database is used. These files are not humanly readable.
Recovering CNRDB Data
Always attempt recovery on a copy of the database file and associated log files, never on the operational files. This is a simple file copy operation, distinct from a shadow backup. Also, never attempt a recovery while Network Registrar is running.
Caution 
It is possible to damage the CNRDB database files without the damage being immediately obvious. Such damage could occur because of (a) inappropriately deleting log files; (b) mixing pre- and postrecovery database and log files; (c) attempting recovery of database files currently in use by an application; or (d) using tools intended for other databases, such as MCD database tools. For the CNRDB database, be sure to use only the
cnrdb_archive,
cnrdb_recover, or
cnrdb_check utilities. Do not use the keybuild or dbcheck utility that applies to the MCD database.
Use the cnrdb_recover utility, included in the Network Registrar product distribution, for database recovery. Use this tool with care. You should never use it directly on an operational database, or on files another application is concurrently accessing. On a successful database recovery, do not intermingle the recovered files (database file and log files) with files from another source, such as the operational database or shadow backups. Recovered database files acquire state information that make them incompatible with older database files.
Here is the procedure for CNRDB database recovery:
Step 1
Stop all Network Registrar servers. Be sure that enough disk space is available for two copies of the database files, plus a 15% safety margin.
Step 2
Create two temporary directories, outside the Network Registrar installation tree, for safety:
Step 3
Copy the database files using shell copy commands, in the following order.
a.
Copy the CNRDB operational database file to subdirectories of .../dbcopy/:
•
For DHCP—The .../data/dhcp/ndb/dhcp.ndb file to .../dbcopy/dhcp/
•
For DNS—The .../data/dns/ndb/dns.ndb file to .../dbcopy/dns/
b.
Copy the associated log files to subdirectories of .../dbcopy/:
•
For DHCP—The .../data/dhcp/ndb/logs/log.* files to .../dbcopy/dhcp
•
For DNS—The .../data/dns/ndb/log/log.* files to .../dbcopy/dns
c.
Double-check that the database file and all log files were copied correctly. The .../dbcopy area contains a safety copy of the database files. Do not allow these files to be modified in any way. Do not run any utilities or servers on these files.
Step 4
Copy the same files to subdirectories of the .../recover directory.
Step 5
Try to recover the current database files:
a.
Change to the appropriate subdirectory of .../recover and run the recovery utility program, cnrdb_recover -c -v. It is helpful to use the -v switch; otherwise, the utility displays no output in the absence of errors. In particular, this utility may not indicate that any log files are missing.
b.
Run the verification utility program for each of the servers.
–
For DHCP—cnrdb_verify dhcp.ndb
–
For DNS—cnrdb_verify dns.ndb
c.
Optionally, for additional confidence, run the cnrdb_archive utility program:
•
cnrdb_archive -l lists all log files
•
cnrdb_archive -s lists the database file
d.
In the absence of any error indication, replace the operational database file and log files, in the Network Registrar installation tree, with the files in .../recover. Be careful not to mix database files processed with cnrdb_recover with those that were not. To avoid this problem, delete the operational CNRDB file and log files from the Network Registrar installation tree. Verify again that all database files are in the .../dbcopy directory before deleting the operational files. In copying the recovered files, ensure that the database file and log files are copied to the appropriate and separate directories in the Network Registrar installation tree.
e.
Restart Network Registrar, including the servers.
Step 6
If there are any indications, such as server log messages or missing data, that database recovery was unsuccessful, you may need to base a recovery attempt on the current shadow backup (in the Network Registrar installation tree). The procedure is identical to that of Step 5, except that the current contents of .../dbcopy/ are not disturbed, and the source of the files copied to .../recover is .../data/dhcp.bak or .../data/dns.bak.
Once the files from the shadow backup are copied to .../recover, copy the operational log files from .../data/dhcp/ndb/logs or .../data/dns/ndb/log to .../recover. This may cause older files in.../recover to be overwritten by newer log files from the operational database. This is expected and typically occurs to at least one log file. Be careful, however, not to do the reverse by overwriting new log files from the operational database with older ones from the shadow backup.
If the attempted operation fails, try it again, this time using only the shadow backup files and not copying the operational log files. In this case, all data written to the database since the shadow backup was made is lost.
Step 7
If Step 6 should fail, perhaps because the current shadow backup is simply a copy of corrupted files, use the most recent previous shadow backup. (This illustrates the need to regularly archive shadow backups.) Unlike in Step 6, you cannot add operational log files to older shadow backup files. All data added to the database since the shadow backup was made will be lost.
Step 8
After a successful database recovery:
a.
As a safety measure, archive the contents of .../dbcopy/.
b.
Using the mcdshadow utility, initiate an immediate shadow backup and archive the files. See the "Performing a Manual Backup" section. This backup is the earliest, oldest backup from which you can attempt a future recovery. You cannot use older shadow backups.
Virus Scanning While Running Network Registrar
If you have virus scanning enabled on your system, it is best to configure it to exclude certain Network Registrar directories from being scanned. Including these directories might impede Network Registrar operation. Exclude from scanning the following directories and their subdirectories:
•
On Windows—\Program Files\Network Registrar\data and ...\logs
•
On UNIX and Linux—/var/nwreg2/data and .../logs
Troubleshooting Servers and Databases
The following sections describe troubleshooting the DNS, DHCP, and TFTP servers, as well as the Network Registrar databases.
Immediate Troubleshooting Actions
When facing a problem, it is crucial not to cause further harm while isolating and fixing the initial problem. Here are things to avoid doing in particular:
•
Do not have less than 512 MB of memory and less than 2.5 GB of a data partition.
•
Do not reboot a cable modem termination system (CMTS).
•
Do not disable DHCP failover.
•
Do not reload, restart, or disrupt Network Registrar with failover resynchronization in progress.
Troubleshooting Server Failures
The Server Agent process (aicservagt) normally detects server failures and restarts the server. You can usually recover from the failure and the server is not likely to fail again immediately after restarting. On rare occasions, the source of the server failure prevents the server from successfully restarting, causing the server to fail again as soon as it restarts. In such instances, perform the following steps:
Step 1
If the server takes a significantly long time to restart, stop and restart the Server Agent.
•
On Windows:
net stop "AIC Server Agent 2.0"
net start "AIC Server Agent 2.0"
•
On Solaris:
/etc/init.d/aicservagt stop
/etc/init.d/aicservagt start
•
On Linux:
/etc/rc.d/init.d/aicservagt stop
/etc/rc.d/init.d/aicservagt start
Step 2
Keep a copy of all the log files. By default, log files are located in the /var/nwreg2/logs directory on UNIX, and the C:\Program Files\Network Registrar\Logs folder on Windows. The log files often contain useful information that can help isolate the cause of a server failure.
Step 3
Save the core or user.dmp file, depending on the operating system:
•
On UNIX or Linux—The core file is located in the parent directory of the server executables, /opt/nwreg2. Save a renamed copy of this file that Network Registrar does not overwrite.
•
On Windows—The user.dmp file is located in the system directory, which varies depending on the Windows system. Search for this file and save a renamed copy.
Step 4
On Windows, use the native event logging application to save the System and Application event logs to files. You can do this from the Event Viewer. These event logs often contain data that helps debug Network Registrar server problems.
Troubleshooting and Optimizing the TFTP Server
You can set certain attributes to troubleshoot and optimize TFTP server performance.
Tracing TFTP Server Activity
The TFTP server has two commands that help create more output to logs and can be useful in troubleshooting, although this usually impacts performance. These commands set up server packet tracing. The tftp getTraceLevel command identifies the current trace level, which by default is 0, or no tracing. The tftp setTraceLevel command sets the packet tracing to a value between 0 and 4. The trace files are located in the data/tftp subdirectory of your installation directory. Windows tracing goes to the file_tftp_1_log files; Solaris tracing goes to the file_tftp_1_trace files.
nrcmd> tftp getTraceLevel
nrcmd> tftp setTraceLevel 4
The trace levels are as follows, with each higher level being cumulative:
•
0—Disables all server tracing (the default)
•
1—Displays all the log messages in the trace file
•
2—Displays the client's IP address and port number for all packets
•
3—Displays the packet header information
•
4—Displays the first 32 bytes of the packet
Note
Setting and getting the trace level will only work if the TFTP server is started. Turn on packet tracing only if the Cisco Technical Assistance Center (TAC) requests you to do so, and then not for any extended time, because tracing can have significant performance impact.
Optimizing TFTP Message Logging
You can improve TFTP server performance by restricting logging and tracing. By default, the server logs error, warning, and informational messages to file_tftp_1_log files. You can set the log levels using a few TFTP server parameters:
•
Log level—Master controller of server logging, which defaults to, and is best left at, level 3 (logs all error, warning, and informational messages). As with packet tracing, the higher logging levels are cumulative. If set to 0, no server logging occurs. To change it, you must reload the server.
nrcmd> tftp set log-level=3
•
Log settings—This is the second level of logging control and can take the value default or no-success-messages. Default logging is the default, which does not alter the default of log level 3 (error, warning, and informational messages). However, you may want to disable writing success informational messages, and thereby improve server performance, by changing the log settings to no-success-messages. Reload the server after changing this value.
nrcmd> tftp set log-settings=no-success-messages
•
Log file count and size—Sets how many log files to maintain and how large to allow them to get in the data/tftp directory. The default is to maintain a maximum of four files of one megabyte each. Reload the server after changing these values.
nrcmd> tftp set log-file-count=4
nrcmd> tftp set log-file-size=1
Enabling File Caching
You can improve TFTP server performance significantly by enabling file caching on the server. You must do this explicitly, because it is disabled by default. You must also create and point to a file cache directory, and you can set the maximum size of this directory. Here are the steps to follow:
Step 1
Determine where you want to store the TFTP cache files. This cache directory becomes a subdirectory of the TFTP home directory, which is by default .../data/tftp. On Windows, the home directory default is C:\Program Files\Network Registrar\data\tftp; on Solaris and Linux, it is /var/nwreg2/data/tftp.
If you want a different location, use the home-directory attribute.
nrcmd> tftp set home-directory=/tmp/tftp
Step 2
Change to the TFTP home directory and create the cache directory, such as CacheDir, in the home directory. Note that Network Registrar ignores any files in any subdirectories of this cache directory.
Step 3
Set up the TFTP server to point to the cache directory. You cannot use relative paths in the directory name, such as .../CacheDir. If the directory does not exist, file caching cannot occur.
nrcmd> tftp set file-cache-directory=CacheDir
Step 4
Set the maximum memory size, in bytes, of the cache. The default is 32 K bytes. Network Registrar loads all files into cache that cumulatively fit this memory size. If you set the value to 0, Network Registrar does not cache any data, even if you enable file caching.
nrcmd> tftp set file-cache-max-memory-size=32000
Step 5
Copy all of the files you want cached into the cache directory, and not into any subdirectory. Because all files in this directory are loaded into cache, do not include large files.
Step 6
Enable file caching and reload the server.
nrcmd> tftp enable file-cache
Network Registrar logs the name of each cached file, and skips any it cannot load. It reads in all files as binary data and translates them as the TFTP client requests. For example, if a client requests a file as NetASCII, the client receives the cached data in that form.
Step 7
Writing to cache is not allowed. If you need to update a cache file, overwrite it in the cache directory, then reload the server.
Using the mcdadmin Tool
The mcdadmin tool is a utility with which you can import and export the MCD database and diagnose Network Registrar conditions under Cisco guidance.
Caution 
Use this tool only under the guidance of the Cisco Technical Assistance Center. Casual use can inflict catastrophic damage to the Network Registrar configuration database.
Before using the mcdadmin tool, exit from the GUI or CLI. Then, find the tool at the following location:
•
On Windows, by default—C:\Program Files\Network Registrar\Bin\mcdamin.exe
•
On Solaris or Linux—/opt/nwreg2/usrbin/mcdadmin
Run the tool from the command shell. Here are some sample usages:
> mcdadmin -N admin -P changeme -c -o -i mcdconfig.txt -z m=3
> mcdadmin -N admin -P changeme -e mcdconfig.txt
-p /servers/name/dhcp/1/scopes/scopePC/ -x -z m=3
> mcdadmin -N admin -P changeme -r /servers/name/dhcp/1/scopes/scopePC/ -z m=3
Table 4-7 describes the qualifying options. The absolute path to an object in the MCD tree, dbpath, always begins and ends with a forward slash (/).
Table 4-7 mcdadmin Options
This mcdadmin option...
|
Does the following...
|
-c
|
Creates the MCD database.
> mcdadmin -N admin -P changeme -c -o -i mcdconfig.txt -z m=3
|
-e exportfile
|
Exports the configuration to the specified file, or a dash (-) to display it on the screen.
> mcdadmin -N admin -P changeme -e mcdconfig.txt
> mcdadmin -N admin -P changeme -e -
|
-i importfile
|
Imports the configuration from the specified file.
> mcdadmin -N admin -P changeme -o -i mcdconfig.txt -z m=3
Note Administrative account entries from the imported file supplement the existing accounts (assuming that the username does not exist in the MCD database), unless you include the -o option. In all cases, Network Registrar replaces the password in the MCD database with the value in the import file.
|
-N username
|
Specifies the username. If omitted, prompts for the username.
|
-o
|
When used with the -i (import) option, deletes and replaces the existing user accounts with those from the import file. (See the earlier Note.)
|
-P password
|
Specifies the password. If omitted, prompts for the password.
|
-p dbpath
|
When used with the -e (export) option, specifies the subset of the configuration database to export, as an absolute path.
> mcdadmin -N admin -P changeme -e scopepc.txt
-p /servers/name/dhcp/1/scopes/scopePC/ -x -z m=3
|
-r dbpath
|
Removes the specified path in the MCD database, not including subentries.
mcdadmin -r /servers/name/dhcp/1/scopes/scopePC/
|
-R dbpath
|
Recursively removes all entries below and including the specified path.
Caution  Using this option can destroy large portions of the MCD database that may require partial or total reconstruction
|
-v or -V
|
Prints version information.
|
-x
|
When used with the -e (export) option, exports data in hexadecimal format.
> mcdadmin -N admin -P changeme -e mcdconfig.txt -x
|
-z class=n
|
Sets the debugging class or classes to level n.
> mcdadmin -N admin -P changeme -c -o -i mcdconfig.txt -z m=3
|
Using the keybuild Tool
The keybuild tool is a utility you can use to rebuild the database key files, which contain redundant data with the MCD files.
•
On Windows—Click Start > Settings > Control Panel > Services, highlight AIC Server Agent, then click Stop. Go to the install-location\data\db folder and run the keybuild.exe file.
•
On Solaris—Stop the Server Agent. Go to the install-location/data/db folder. You need root privileges.
/etc/init.d/aicservagt stop
Using the dbcheck Tool
The dbcheck tool is a utility you can use to check the MCD integrity. You must have root privileges to run the dbcheck tool.
•
On Windows—Click Start > Settings > Control Panel > Services, highlight AIC Server Agent, then click Stop. Go to the install-location/data/db folder and run the dbcheck.exe file.
•
On Solaris—Stop the Server Agent. Go to the install-location/data/db folder. You need to have administrator privileges.
/etc/init.d/aicservagt stop
Solaris and Linux Troubleshooting Tools
You can also use the following commands on Solaris and Linux systems to troubleshoot Network Registrar:
•
To see all Network Registrar processes:
•
To monitor system usage and performance:
•
To view login or bootup errors:
–
On Solaris—grep /var/adm/messages*
–
On Linux—grep /var/log/messages*
•
To view the configured interfaces and other network data: