The following sections describe the major areas in the Cisco IP Solution Center installation in which troubleshooting might be necessary:
•Unable to Find the Hostname
•Moving a Repository or Renaming an Prime Fulfillment Server
•Multiple Prime Fulfillment Instances with the Same TIBCO Rendezvous Port
•Known Installation Issues
•Daylight Saving Time
•Error - DBSPAWN ERROR: -84
•Error - No VPNSC Host Entry in the Database, When Starting Prime Fulfillment
•Error - Could Not Connect to the Name Server, When Starting Prime Fulfillment
•Error - This Is Not a Database Server
•Error - Cannot Connect to the Data Store
Unable to Find the Hostname
Cannot find hostname.
Step 1 If you cannot find the hostname, check the /etc/nsswitch.conf file to determine how the hostname is resolved.
Step 2 Check the /etc/resolv.conf file to determine whether you have a DNS Server IP Address.
Step 3 If you have a DNS Server IP Address, enter ping <IP Address> to check whether it is reachable.
Step 4 If the DNS Server is reachable, use nslookup <machine name> to check if it is resolving the name properly.
Step 5 If it is not working properly, you need a system administrator to fix the DNS entry.
Step 6 If you are not using DNS, be sure there is an entry for your machine in the hosts file in the /etc directory.
Moving a Repository or Renaming an Prime Fulfillment Server
If you want to move an existing Repository to a new server with a new Prime Fulfillment installation or rename an existing Prime Fulfillment installation, your existing configuration must be updated. When renaming the Prime Fulfillment installation, the local configuration file needs to be modified. When moving an existing Repository to a new server, the server from which you are moving the Repository and the server to which you are moving the Repository must both be at the same version and patch levels. Otherwise, your Repository needs to be upgraded, as explained in Upgrading an Existing Installation to Prime Fulfillment 6.1. Both when moving an existing Repository and renaming an existing Prime Fulfillment installation, the changes must be inserted into the Repository.
Use the following steps:
Step 1 Stop Prime Fulfillment, using the following command:
Step 2 Update the $PRIMEF_HOME/runtime.properties file with the new values.
Step 3 Execute the command:
$PRIMEF_HOME/thirdparty/ant/bin/ant -f $PRIMEF_HOME/install.xml
Step 4 Start the database, using the following command:
Step 5 Incorporate the changes into the Repository by initializing the database, using the following command:
Multiple Prime Fulfillment Instances with the Same TIBCO Rendezvous Port
You might not see any error messages or a page might not appear, but you might see inconsistencies with events and tasks that you have just created.
You might have more than one Prime Fulfillment server on the same subnet of a LAN, in which case, multiple instances of the Prime Fulfillment server will have the same TIBCO Rendezvous port. To fix this problem, you must ensure that the TIBCO port has a unique value.
To change the value for the TIBCO port, follow these steps:
Step 1 From the terminal window where the WatchDog is running, stop the WatchDog with the following command:
./prime.sh stopwd -y
Step 2 Use a text editor to open the etc/install.cfg file.
Step 3 Change the TIBCO_PORT variable to the desired value.
The default value for the TIBCO_PORT variable is 7530.
Step 4 To update all the dependent files with the new TIBCO port value, run ant -f install.xml command.
Step 5 ./prime.sh startdb
Step 6 ./prime.sh initdb.sh
Step 7 ./prime.sh stopdb -y
Step 8 ps -e | grep rvrd
The returned result is the process id for the rvrd process.
Step 9 kill -9 <process id>
where: <process id> is the returned process from Step 8.
Step 10 rm -f $PRIMEF_HOME/tmp/rvrd.prime.store
Step 11 ./prime.sh rvrd -store $PRIMEF_HOME/tmp/rvrd.prime.store
Step 12 ./prime.sh startwd
Step 13 Run the following multiple line Java command:
java -classpath $VPNSC_HOME/resources/java/classes/common:\
com.cisco.vpnsc.install.RvrdCfg <tibco_port> <server> prime
<tibco_port> is the desired port specified in Step 3.
<server> is the server name, for example: server1.cisco.com.
Known Installation Issues
Known issues and solutions are as follows:
Out of disk space.
The error looks something like the following:
Prime Fulfillment 6.1 will be installed in /var/PrimeFulfillment
>tar:./shared/jre_1.3.1_solaris_sun_sparc/lib/rt.jar: HELP - extract
If you see an error like this, it is likely due to the server running out of disk space.
To verify what space is available, run the command df -k <install directory>.
See Chapter 1 "System Recommendations," for the disk space recommendations.
The Installation utility GUI never displays.
This problem should be accompanied with a Java stack dump.
Step 1 Run the following command to check for the $DISPLAY environment variable being set:
If you use the secure shell (ssh), then this will be set up and managed for you.
If you manually change the $DISPLAY environment variable in an SSH environment, the easiest recovery method is to log off and reestablish the SSH connection.
Step 2 To set the DISPLAY environment variable, do the following:
a. For the K or Bourne shell:
export DISPLAY=<machine name>:0.0
b. For the C shell:
setenv DISPLAY=<machine name>:0.0
Could not find temporary files.
If you receive an error that says the temporary file could not be created or found, it usually means the location used to write the temporary file is write-protected or out of disk space.
The two places that Prime Fulfillment uses for temporary files are /tmp and /var/tmp.
•Make sure both locations have write permission by doing a long list on the directories (ls -la). The directory should have wide open permissions: drwxrwxrwx.
•There is another temporary file problem that can arise, especially in cases where there have been previous aborted installation attempts—existing temp files might be left by previous installations. If this is the case, it is best to clean out all the files in the temp directories after aborted installation attempts.
Running ./prime install.sh fails.
Running ./prime install.sh can fail due to the following reasons:
1. You are not root.
Although it is possible to install as non-root if you have appropriate permissions in the target directory, this will still have problems since only root can write to /etc/init.d where the startup scripts reside. Therefore, it is easier to install as root.
2. You do not have enough disk space in the target directory. To find out the available disk space, issue the following command:
df -k <target directory>
3. You do not have enough disk space in the /tmp directory. Issue the command df -k /tmp to determine the available disk space for /tmp.
4. You do not have enough disk space in the /var/tmp directory. Issue the command df -k /var/tmp to determine the available disk space for /var/tmp.
5. The PATH and LD_LIBRARY_PATH environment variables are incorrect.
Make sure your PATH and LD_LIBRARY_PATH environment variables are correct.
export PATH LD_LIBRARY_PATH
a. Alternatively, start a clean root shell with this command:
env - ksh
b. Then issue a command like the following:
./install.sh /opt/PrimeFulfillment iscadm
Prime Fulfillment does not start on reboot.
Perform the following steps:
Step 1 Install Prime Fulfillment as the root user.
If you install as root, init.d has a script to start the Watchdog.
If you do not install as root, you do not get the startup on reboot feature.
Step 2 To become root, enter the following command:
Step 3 Get the prime.tmpl file from the installation media.
Step 4 Edit the following fields in prime.tmpl:
OWNER=_owner - replace _owner with the username whom owns prime
PRIMEF_HOME=_vpnsc_home - replace _vpnsc_home with the prime directory
Step 5 Rename prime.tmpl as prime and then enter the following commands:
mv prime /etc/init.d
chmod 744 /etc/init.d/prime
Step 6 Create the following symbolic links to prime:
a. cd /etc/rc1.d
ln -s /etc/init.d/prime K98ISC
b. cd to /etc/rc2.d
ln -s /etc/init.d/prime K98ISC
c. cd to /etc/rc3.d
ln -s /etc/init.d/prime S99ISC
Unable to create or delete IOS devices in the Cisco CNS IE2100 appliance repository when using Cisco CNS Configuration Engine 1.4 software with Prime Fulfillment.
Log into the Cisco CNS IE2100 appliance as root and modify the web.xml file located at /opt/CSCOcnsie/WEB-INF as follows.
Step 1 Locate the following entry:
Step 2 Immediately after the entry found in Step 1, insert the following lines:
Step 3 Locate the following entry:
Step 4 Immediately after the entry found in Step 3, insert the following lines:
Step 5 Reboot the Cisco CNS IE2100 appliance.
Not able to connect to the database.
Perform the following steps:
Step 1 Check that the following values are substituted correctly in the installation window:
•Oracle database server name
•Oracle port number
Step 2 If everything is correct, check that the server is reachable by entering:
ping <Oracle database server name>
Step 3 Issue the following to determine whether the database is running:
netstat -an | grep <oracle port number>
If no responses are found, your database is not running and you must restart, as explained in detail in the section, "Launching Oracle and Opening Your Database," in "Setting Up Oracle for Prime Fulfillment."
Unable to access Prime Fulfillment with your web browser.
Check the server status with the command ./prime.sh status.
If any server state is other than started, attempt to restart by entering the command,
./prime.sh wdclient restart <server name>. If this command does not succeed, enter the commands
./prime.sh stop and then ./prime.sh startwd.
Note The most common server not to start is the httpd server.
Daylight Saving Time
If Daylight Saving Time (DST) is not working correctly, perform the following steps:
Step 1 Go to the following URL to determine which patch is needed for your time zone:
Step 2 To download the Java Runtime Environment (JRE) patch, go to:
Step 3 Go to Prime Fulfillment home directory:
Step 4 Enter: ./prime.sh stop
Step 5 Follow this link to install the missing DST patch that you downloaded from Step 2:
Error - DBSPAWN ERROR: -84
The error: DBSPAWN ERROR: -84 is normally seen when the existing log files are not removed before loading a new repository.db file. The repository.log and sla.log files in the Repository/ directory must be deleted before initiating the ./prime.sh startdb command.
Error - No VPNSC Host Entry in the Database, When Starting Prime Fulfillment
To correct the error: No VPNSC Host Entry in the Database, run ./prime.sh initdb.sh in the following order:
Step 1 ./prime.sh stop
Ensure that no other Prime Fulfillment processes are running. To do this, you can enter: ps -ef | grep prime
Step 2 ./prime.sh startdb
Step 3 ./prime.sh initdb.sh
This step adds the host entry into the repository.
Step 4 ./prime.sh startwd
Error - Could Not Connect to the Name Server, When Starting Prime Fulfillment
The error: com.cisco.vpnsc.watchdog.WDRuntimeException: WD_108 :: Could not connect to the name server is normally seen when the domain name cannot be extracted from resolv.conf. The result is that the nameserver does not start, because it fools the system into thinking it is not a Master server.
To correct this error, you must have root privileges. As root, add the correct domain statement to the /etc/resolv.conf file for your server (not $PRIMEF_HOME/etc); for example, domain cisco.com.
Error - This Is Not a Database Server
The following error could occur after you install Prime Fulfillment:
<server name> ./prime.sh startdb Master database server is: This is not a database server. There is no need to start the database
Adding this host to the database ... com.cisco.cns.security.common.CannotConnectException: Cannot connect to the data store:Cannot connect to the data store. No valid connection to server type: com.cisco.cns.security.dataaccess
at com.cisco.cns.security.dataaccess.ConnectionPool.acquire (ConnectionPool.java:240)
Specifically, this could occur after issuing the command: ./install.bin <directory_where_PrimeFulfillment_is_to_be_installed> iscadm.
The error could be Domain Naming System (DNS) related. In the install.cfg file, <server name>.cisco.com needs to be changed to <server name> only. Then run applycfg.sh followed by ./prime.sh initdb.sh and ./prime.sh startwd.
Error - Cannot Connect to the Data Store
The primary reason for the error: Cannot Connect to the Data Store is DNS related. As root, make sure /etc/resolv.conf (not the $PRIMEF_HOME/etc directory) is correct for your server.
If you need more information, set the Security Policy Engine (SPE) logging to DEBUG and attempt to execute ./prime.sh initdb.sh. This provides more details. If an unknown host exception is created, double check the /etc/hosts file and the /etc/nsswitch.conf file. This controls the flow and sequence of the hostname lookup.
If DNS is not enabled or working, add the IP address to the following files: cns, vpnsc, and HA properties files, to use IP addresses instead of hostnames.
The cns properties files is located at $PRIMEF_HOME/etc/spe/cns.properties.
The vpnsc properties file is located at $PRIMEF_HOME/etc/vpnsc.properties.
The HA properties file is located at $PRIMEF_HOME/etc/HA.properties.
This explanation of Echo mode is specified in the following subsections:
•What is Echo Mode?
•Who Should Use Echo Mode and When Should It Be Used?
•How Should You Use Echo Mode?
What is Echo Mode?
Echo mode is a setting in Prime Fulfillment that is accessible through the Prime Fulfillment configuration window. Echo mode affects service provisioning. When you set Prime Fulfillment to run in echo mode, Prime Fulfillment performs service provisioning tasks without downloading the resulting commands to the physical hardware. The resulting service provisioning is stored only in the Repository, and no attempt is made to connect to the target devices.
Who Should Use Echo Mode and When Should It Be Used?
In a production environment, echo mode can be used to perform service provisioning on devices that are either temporarily offline or not yet commissioned. The service provisioning only occurs within the Prime Fulfillment Repository. When these devices become active, you can force the deployment of the previously provisioned services and Prime Fulfillment downloads the configurations to the devices.
Echo mode is a global configuration setting that affects the Service Provisioning for all users. Therefore, echo mode should be used with care. To enable echo mode, set the Dynamic Component Properties Library (DCPL) GTL/echo-mode to true (Administration > Control Center > Hosts, as explained in Appendix C, Property Settings of the Cisco Prime Fulfillment User Guide 6.1). When echo mode is enabled, no attempt is made to contact any devices and no attempt is made to audit the Service Request. This affects all Service Requests during the time period when echo mode is enabled.
How Should You Use Echo Mode?
Because echo mode affects all of Prime Fulfillment's provisioning, be sure that all provisioning requests that require device access are complete before turning on echo mode.
Turn on echo mode, as explained in the "Who Should Use Echo Mode and When Should It Be Used?" section.
Configure your Service Request as normal for the device that is not commissioned or is offline. Save and deploy the Service Request. No attempt is made to contact the device or audit the Service Request. The Service Request transitions into the Deployed state.
Now, you can disable echo mode, by changing the GTL/echo-mode property to false (see the "Who Should Use Echo Mode and When Should It Be Used?" section). From this point forward, all provisioning requests contact the devices and all provisioning requests are audited. You can now safely resume provisioning for all users.
After the device has been commissioned or brought back online, Force deploy the provisioning request for this device (see Chapter 3 in the Cisco Prime Fulfillment User Guide 6.1). This forces the provisioning request to go through the provisioning cycle and deploy the configlet onto the device.