Cisco IoT Field Network Director User Guide, Release 4.2.x

Bias-Free Language

The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.

Book Contents
Find Matches in This Book
Available Languages
Download Options

Book Title

Cisco IoT Field Network Director User Guide, Release 4.2.x

Chapter Title

Troubleshooting IoT FND

Results

Updated:
April 13, 2018

Chapter: Troubleshooting IoT FND

Troubleshooting IoT FND

This section describes how to troubleshoot common IoT FND issues.

■blank.gifTunnel Provisioning DHCP Configuration Issues

■blank.gifMesh Endpoint Registration Issues

■blank.gifRecovering an Expired Database Password

■blank.gifUnlocking the IoT FND Database Password

■blank.gifIoT FND Service Will Not Start

■blank.gifException in the server.log File on the IoT FND Server

■blank.gifResetting the root Password

■blank.gifSecond IoT FND Server Not Forming a Cluster

■blank.gifIoT FND Service Restarts Automatically

■blank.gifFAR Management Issues

■blank.gifMesh Endpoint Management Issues

Note: Always reference the release notes for your IoT FND version.

Tunnel Provisioning DHCP Configuration Issues

If there is a problem allocating an address, IoT FND logs a Tunnel Provisioning Failure event. The log entry includes details of the error.

To monitor the address allocation process:

■blank.gifCheck the IoT FND server.log file to determine if IoT FND is sending a DHCP request during tunnel provisioning.

■blank.gifCheck your DHCP server log file to determine if the DHCP request from IoT FND reached the DHCP server.

If requests are not reaching the server:

■blank.gifEnsure that the DHCP server address is correct on the Provisioning Settings page in IoT FND (Admin > System Management > Provisioning Settings).

■blank.gifCheck for network problems between IoT FND and the DHCP server.

If the DHCP server is receiving the request but not responding:

■blank.gifView the DHCP server log file, and ensure that the DHCP server is configured to support requests from the link address included in the DHCP requests. The link address is defined in the tunnel provisioning template.

■blank.gifEnsure that the DHCP server has not exhausted its address pool.

If the DHCP server is responding, but IoT FND is not processing the response:

■blank.gifEnsure that the lease time is infinite. Otherwise, IoT FND will not process the response.

■blank.gifView the DHCP server logs and IoT FND server logs for other errors.

Mesh Endpoint Registration Issues

To determine why MEs register with IoT FND, IoT FND collects the registration reason code from the MEs and logs events and the code with other relevant information as printed key value pairs to help diagnose registration issues.

Here is an example of a logged event:

?Event logged: Event(id=0, eventTime=1335304407477, eventSeverity=0, eventSource=cgmesh, eventMessage=Mesh node registered due to cold boot: [lastReg: 0, lastRegReason: 1], NetElement.id=10043, EventType.name=null, lat=1000.0, lng=1000.0, geoHash=null
 

Mesh Endpoint Registration Reason Codes lists reason codes for ME registration and corresponding event:

Table 1 Mesh Endpoint Registration Reason Codes
Registration Reason Code
Code
Event Type Name
Severity
Message
Description

REASON_UNKNOWN

0

unknownRegReason

INFO

Mesh node registered for unknown reason.

 

REASON_COLDSTART

1

coldBoot

INFO

Mesh node registered due to cold boot.

The message includes the new IP address of the ME.

REASON_ADMIN

2

manualReRegistration

INFO

Mesh node registered due to manual registration.

The endpoint received an NMSRedirectRequest without a URL field.

REASON_IP_CHANGE

3

rejoinedWithNewIP

INFO

Mesh node registered with new IP address.

The message includes the new IP address of the ME.

REASON_NMS_CHANGE

4

nmsAddrChange

INFO

Mesh node registered due to NMS address change.

The IoT FND IP address changed OUTSIDE of an NMSRedirect (a new DHCPv6 option value was received).

REASON_NMS_REDIRECT

5

manualNMSAddrChange

INFO

Mesh node registered due to manual NMS address change.

Endpoint received an NMSRedirect request.

REASON_NMS_ERROR

6

nmsError

INFO

Mesh node registered due to NMS error.

Endpoint received an error from IoT FND.

In addition to generating events when MEs register with IoT FND, IoT FND also generates events after receiving a WPAN change TLV WPANStatus.

Event logged: Event(id=0, eventTime=1335304407974, eventSeverity=0, eventSource=cgmesh, eventMessage=WPAN change due to migration to better PAN: [lastChanged: 0, astChangedReason: 4], NetElement.id=10044, EventType.name=null, lat=1000.0, lng=1000.0, geoHash=null)
 

Reasons for Mesh Endpoint WPAN Changes lists reasons for ME WPAN changes and the corresponding event.

Table 2 Reasons for Mesh Endpoint WPAN Changes

Registration Reason Code
Code
Event Name
Severity
Type
Description

IEEE154_PAN_LEAVE_UNKNOWN

-1

unknownWPANChange

MAJOR

WPAN change for unknown reason.

IEEE154_PAN_LEAVE_INIT

0

meshInit

N/A

No event is generated for this code.

IEEE154_PAN_LEAVE_SYNC_TIMEOUT

1

meshConnectivityLost

MAJOR

WPAN change due to mesh connectivity loss.

IEEE154_PAN_LEAVE_GTK_TIMEOUT

2

meshLinkKeyTimeout

MAJOR

WPAN change due to mesh link key timeout.

IEEE154_PAN_LEAVE_NO_DEF_ROUTE

3

defaultRouteLost

MAJOR

WPAN change for no default route.

IEEE154_PAN_LEAVE_OPTIMIZE

4

migratedToBetterPAN

MAJOR

WPAN change due to migration to better PAN.

For these events, the message includes the time elapsed since the ME left the network to when it rejoined. IoT FND displays the amount of time the ME was offline since the event was logged (for example, 4 hours 23 minutes ago).

Recovering an Expired Database Password

To recover from an expired password, run these commands:

su - oracle
 
sqlplus sys/cgmsDbaAccount@cgms as sysdba
alter user cgms_dev identified by test;
alter user cgms_dev identified by password;
exit;

Unlocking the IoT FND Database Password

If you enter an incorrect IoT FND Database password multiple times, Oracle locks your user account. Unlock your password using the Oracle software, as shown in this example:

# su - oracle
# sqlplus sys/<database_password>@cgms as sysdba
alter user cgms_dev account unlock;
exit;

IoT FND Service Will Not Start

If the IoT FND service does not start:

1.blank.gif Validate connectivity to the database:

a.blank.gif Log in as root on the IoT FND server.

b.blank.gif Enter the following at the command prompt:

service cgms status

c.blank.gif Verify the database server IP address and that IoT FND can connect to the database.

■blank.gifIf the IP address is incorrect or if IoT FND cannot access the database, run setupCgms.sh and enter the correct values.

d.blank.gif Run the service cgms status command and verify connectivity.

e.blank.gif Start IoT FND.

2.blank.gif Verify that the JRE version installed on the server is correct (see the System Requirements chapter).

3.blank.gif Verify that database migration was performed successfully.

Exception in the server.log File on the IoT FND Server

If the there is an exception in the server.log file indicating that IoT FND could not open the cgms_keystore file, then the cgms_keystore password stored in the cgms.properties file on the IoT FND server is incorrect.

The password for the cgms_keystore file is encrypted and stored in the /opt/cgms/server/cgms/conf/cgms.properties file.

To encrypt or decrypt the password, use the /opt/cgms/bin/ encryption_util.sh script.

Verify or update the password in the cgms.properties file, and if an update is required, restart IoT FND after modifying the password.

Resetting the root Password

If you forget the password of the IoT FND root user account, reset the password by running the /opt/cgms/bin/ password_admin.sh script.

Second IoT FND Server Not Forming a Cluster

Typically, discovery of nodes in a IoT FND cluster is automatic. As long as the IoT FND servers are on the same subnet, they form a cluster.

If you install a IoT FND server and it does not join the cluster:

1.blank.gif Verify that your servers are on the same subnet, can ping each other, and share the same cluster name.

2.blank.gif Check the status of all members by running the /opt/cgms/bin/print_cluster_view.sh script.

3.blank.gif Modify the cluster name, as follows:

a.blank.gif Change the value of the HA_PARTITION_NAME parameter on all IoT FND cluster nodes, and then restart them.

b.blank.gif Change the value of the UDP_MULTICAST_ADDR parameter (unique multicast address) to match on all nodes in the cluster.

c.blank.gif Change the value of the CLUSTER_BIND_ADDR parameter to the interface to which you want the NMS to bind.

4.blank.gif Verify that all the cluster nodes are configured to use NTP (see Configuring NTP Service)

5.blank.gif Check the /etc/hosts file and verify that the IP address is correctly mapped to the hostname of the local server.

IoT FND Service Restarts Automatically

When the IoT FND services are started, the watchdog script is invoked. The watchdog script checks the health of the IoT FND services. If the watchdog script detects an anomaly, it logs the conditions in the /opt/cgms/server/cgms/log/cgms_watchdog.log file

The watchdog script tries three times to determine if the anomaly condition improved. If not, it restarts the IoT FND services automatically, unless the database has become unreachable. If the database is not reachable, the watchdog stops the IoT FND services. Check the log files, including server.log, to determine what is causing the restarts.

Manually disable the watchdog process by running the /opt/cgms/bin/deinstall_cgms_watchdog.sh script on the IoT FND server as root.

FAR Management Issues

This section presents common issues with FAR management and possible resolutions.

Certificate Exception

If this exception appears in the server.log file stored on the IoT FND server when a FAR attempts to register with IoT FND, the cgms_keystore file does not contain the CA server certificates or the CA certificates that were imported into the cgms_keystore file are incorrect:

SSLException: Received fatal alert: unknown_ca
 

For information about how to import certificates into the cgms_keystore file, see “Generating and Installing Certificates in the Cisco IoT FND Installation Guide, 4.0.x and greater.

FAR Keeps Reloading and Does Not Switch to the Up State

When a FAR is continuously reloading every time it contacts IoT FND, it could be because the configuration pushed to the FAR by IoT FND is not being applied successfully.

Check the server.log file on the IoT FND server for clues on the cause of the configuration push failure. Sometimes, typos in the in the Field Area Router Tunnel Addition template cause this failure (IoT FND does not provide template validation).

Note: When a FAR registers with IoT FND, IoT FND queries the FAR with show commands. IoT FND then configures the FAR based on the configuration commands in the Field Area Router Tunnel Addition template.

Other reasons for continuous reloads may be:

■blank.gifA bad WAN link that drops packets and does not allow the registration to complete.

■blank.gifFirewall issues. Ensure that the firewall allows traffic in both directions and that traffic to and from the correct ports is allowed to pass.

Incorrect FAR State in IoT FND

In IoT FND, a FAR might appear in a Down state even though you can ping and trace the route to it without a problem.

IoT FND manages the FAR via the IoT-DM service running on the FAR. So even though the FAR is pingable and reachable, it is important to verify that the jetty server and call home features are enabled on the FAR:

'show run callhome' should have 'enable' in the config and 'sh jvm status'
 

Mesh Endpoint Management Issues

This section presents common issues with ME management and possible resolutions.

Mesh Endpoints Not Registering with IoT FND

Verify that the MEs have joined the FAR and are pingable from IoT FND over IPv6. If they are pingable, verify the following:

■blank.gifThe clock is in sync.

■blank.gifThe DHCP server used by the MEs is programmed with the correct IoT FND IP address.

■blank.gifThe MEs are running an image compatible with the current version of IoT FND.

■blank.gifIf HSM is used, HSM must be online and responding correctly.

Licensing Issues

This section presents common issues with license management and possible resolutions.

Device Import Failure

The importing of devices into IoT FND is dependent on the number of allotted IoT FND server licenses.

Verify that your IoT FND server has the adequate license count available for the number and type of devices being imported into the IoT FND database.

Only unique device EIDs are allowed in IoT FND. Check that no one else imported this device EID in to IoT FND or is currently trying to import the same device EID. Verify that no other user is simultaneously importing the same device into IoT FND.

License File Upload Failure

An expired license file will cause an error. Check the license file validity and expiration date.

Was this Document Helpful?

FeedbackFeedback

Contact Cisco