Troubleshooting IoT Services: IOx Application

How do I verify the IOx application is running on the AP?

Run the command: show iox applications

App State should be RUNNING to indicate if it is running. 

AP# show iox applications
Total Number of Apps : 1
--------------------------
App Name                     : cisco_dnas_ble_iox_app
   App Ip                    : 192.168.11.2
   App State                 : RUNNING
   App Token                 : 02fb3e98-ac02-4356-95ba-c43e8a1f4217
   App Protocol              : ble
   App Grpc Connection       : Up
   Rx Pkts From App          : 3878345
   Tx Pkts To App            : 6460
   Tx Pkts To Wlc            : 0
   Tx Data Pkts To DNASpaces : 3866864
   Tx Cfg Resp To DNASpaces  : 1
   Rx KeepAlive from App     : 11480
   Dropped Pkts              : 0
   App keepAlive Received On : Mar 24 05:56:49

How do I debug the IOx application installation failure?

  1. Ensure that the Network Time Protocol (NTP) server is synchronized with the Wireless Controller and APs in use.

  2. Cisco Spaces: Connector installs the IoX Application. Download the AP image bundle from Cisco Spaces to Connector. Next, use the Cisco Application Framework (CAF) to install the image and launch the application from Cisco Spaces, primarily utilizing the ioxclient tool. For more information, see What is ioxclient?.

  3. To examine the logs, you can either upload them to the Cisco Spaces or log into Cisco Spaces: Connector using SSH.

  4. Observe the following critical logs:

    • /opt/spaces-connector/runtime/logs/iot-services/server.log : Records the initiation and completion of requests. It indicates when the main installation begins and the parameters it uses.

    • /opt/spaces-connector/runtime/logs/iot-services/dnas_iox_app_manage.log: Provides detailed information on the installation process.

  5. To monitor the logs in real-time, do the following:

    • As a spacesadmin user, run the command, tail -F /opt/spaces-connector/runtime/logs/iot-services/server.log.

    • As a spacesadmin user, run the command, tail -F /opt/spaces-connector/runtime/logs/iot-services/dnas_iox_app_manage.log.

How do I verify the IoX Application AP bundle download from Cisco Spaces?

The IoX Application installation is done from the Cisco Spaces: Connector. The AP image bundle is downloaded from Cisco Spaces to Cisco Spaces: Connector. To verify if the IoX Application was downloaded accurately, you can check the log files. See How do I debug the IOx application installation failure?
If the logs suggest a problem with the download, you can attempt to manually download the image. To manually download the image, log into Cisco Spaces: Connector via SSH. As a spacesadmin user, use the wget command:
spacesadmin# wget "https://dnaspaces.io/api/edm/v1/device/iox-app/download?id=cisco_dnas_ble_iox_app&version=1.1.16"

How do I start an interactive shell session for the IOx application?

Run the command: connect iox application

This starts a shell which is running inside the IOx application container.

AP# connect iox application
/ #

How can I see the logs for the IOx application?

First, start an interactive shell using the show iox application command .

Then, run the command: tail -F /data/logs/dnas_ble.log

You can see the logs for the IOx application.

AP# tail -F /data/logs/dnas_ble.log
Tue Mar 24 06:55:21 2020 [INFO]: Starting DNA Spaces BLE IOx Application
Tue Mar 24 06:55:21 2020 [INFO]: Auth token file contents: db26a8ab-e800-4fe9-a128-80683ea17b12
Tue Mar 24 06:55:21 2020 [INFO]: Setting gRPC endpoint to: 1.1.7.101:57777
Tue Mar 24 06:55:21 2020 [INFO]: Auth with token: db26a8ab-e800-4fe9-a128-80683ea17b12
Tue Mar 24 06:55:21 2020 [INFO]: Attempt to connect to DNAS Channel
Tue Mar 24 06:55:21 2020 [INFO]: Starting to run metrics
Tue Mar 24 06:55:21 2020 [INFO]: Starting to run Channel Keepalive
Tue Mar 24 06:55:21 2020 [INFO]: Initialize DNAS Reader Channel
Tue Mar 24 06:55:21 2020 [INFO]: Start listener for messages
Tue Mar 24 06:55:21 2020 [INFO]: Running BLE scan thread

How do I monitor metrics in the IOx application?

First, start an interactive shell using the show iox application command .

Run the command: tail -F /data/logs/dnas_ble_metrics.log

This command begins watching the log file for IOx application metrics. Metrics are updated every 30 seconds in the log file.

Table 1. Monitor Metrics

Metrics Name

Metrics Description
Application Version The version number of the IOx application that is currently in use.
Start Time The timestamp of when the application was initialized on the AP.
Up Time The total time the application has been running since it was last started.
Total Physical Memory The total RAM allocated to the application's container.
Physical Memory Free The amount of RAM that remains unused in the application's container.
Physical Memory Used The amount of RAM that is currently being used by the application's container.
Total Physical Shared Memory The amount of memory shared amongst processes or containers.
Total Physical Buffer Memory The memory dedicated to buffering, which aids in optimizing I/O operations.
Total AP Percent CPU Used The percentage of the AP's CPU that is consumed by the application's container.
Process Virtual Memory The virtual memory used by the application's process.
Process Physical Memory The amount of physical RAM occupied by the application's process.
Process CPU Used The CPU usage of the application's process.
gRPC Reconnect Count The number of times a gRPC (remote procedure call) connection has been reestablished.
CAPWAP Restart Count The number of restarts of the Control And Provisioning of Wireless Access Points (CAPWAP) protocol connection.
Last CAPWAP Restart Time The timestamp marking the most recent CAPWAP connection restart.
BLE Device Open Count The number of instances a Bluetooth Low Energy (BLE) device connection has been established.
Last BLE Device Open Time The timestamp indicating the last occasion a BLE device was connected.
BLE Device Close Count A count of disconnections of a BLE device.
Last BLE Device Close Time The timestamp of the most recent closure of a BLE device connection.
Log Rotation Count The frequency with which the log file (dnas_ble.log) has been archived and a new log started.
Floor Beacon Scan Data Message Count The total count of BLE scan data messages since the application began.
Floor Beacon Scan Data Message Rate Per Second The average creation rate of BLE scan data messages per second.
Floor Beacon Scan Data Write Count The total number of BLE scan data packets transmitted since the start of the application.
Floor Beacon Scan Data Write Rate Per Second The transmission rate of BLE scan data packets per second.
Floor Beacon Scan Data Message Count Per Write The average count of BLE scan data messages included in each write operation.
Floor Beacon Scan Data Message Avg Write Time The average duration it takes to write a BLE scan data packet.
Floor Beacon Config Request Count The total number of floor beacon configuration requests since the application started.
Last Floor Beacon Config Request Time The timestamp of the most recent request for floor beacon configuration.
Floor Beacon Config Success Count The total number of successful floor beacon configuration requests.
Last Floor Beacon Config Success Time The timestamp indicating the completion of the most recent successful floor beacon configuration.
Floor Beacon Config Failure Count The count of floor beacon configuration requests that did not succeed.
Last Floor Beacon Config Failure Time The timestamp of the last unsuccessful floor beacon configuration request. 
AP# tail -F /data/logs/dnas_ble_metrics.log
Wed Oct 6 17:03:49 2021 [INFO]: Application Version: 1.2.5Wed Oct  6 17:03:49 2021 [INFO]: Start Time: Fri Sep 17 15:54:11 2021 Up Time:
        0019D:01H:09M:38S
Wed Oct 6 17:03:49 2021 [INFO]: Total Physical Memory: 1557 MBWed Oct 6 17:03:49 2021 [INFO]: Physical Memory Free: 786 MBWed Oct 6 17:03:49 2021 [INFO]: Physical Memory Used: 770 MBWed Oct 6 17:03:49 2021 [INFO]: Total Physical Shared Memory: 170 MBWed Oct 6 17:03:49 2021 [INFO]: Total Physical Buffer Memory: 0 MBWed Oct 6 17:03:49 2021 [INFO]: Total AP Percent CPU Used: 1.934973Wed Oct 6 17:03:49 2021 [INFO]: Process Virtual Memory: 108696 kBWed Oct 6 17:03:49 2021 [INFO]: Process Physical Memory: 8828 kBWed Oct 6 17:03:49 2021 [INFO]: Process CPU Used: 0.004167Wed Oct 6 17:03:49 2021 [INFO]: gRPC Reconnect Count: 0Wed Oct 6 17:03:49 2021 [INFO]: CAPWAP Restart Count: 1Wed Oct  6 17:03:49 2021 [INFO]: Last CAPWAP Restart Time: Fri Sep 17 15:54:11
      2021
Wed Oct 6 17:03:49 2021 [INFO]: BLE Device Open Count: 1Wed Oct  6 17:03:49 2021 [INFO]: Last BLE Device Open Time: Fri Sep 17 15:54:11
        2021
Wed Oct 6 17:03:49 2021 [INFO]: BLE Device Close Count: 1Wed Oct  6 17:03:49 2021 [INFO]: Last BLE Device Close Time: Sat Sep 18 05:48:12
        2021
Wed Oct 6 17:03:49 2021 [INFO]: Log Rotation Count: 0Wed Oct  6 17:03:49 2021 [INFO]: Floor Beacon Scan Data Message Count:
      10896160
Wed Oct  6 17:03:49 2021 [INFO]: Floor Beacon Scan Data Message Rate Per Second:
      00
Wed Oct  6 17:03:49 2021 [INFO]: Floor Beacon Scan Data Write Count:
      217955
Wed Oct  6 17:03:49 2021 [INFO]: Floor Beacon Scan Data Write Rate Per Second:
      00
Wed Oct  6 17:03:49 2021 [INFO]: Floor Beacon Scan Data Message Count Per Write:
      50
Wed Oct  6 17:03:49 2021 [INFO]: Floor Beacon Scan Data Message Avg Write Time
        (milliseconds): 12
Wed Oct 6 17:03:49 2021 [INFO]: Floor Beacon Config Request Count: 0Wed Oct  6 17:03:49 2021 [INFO]: Last Floor Beacon Config Request Time:
      None
Wed Oct 6 17:03:49 2021 [INFO]: Floor Beacon Config Success Count: 0Wed Oct  6 17:03:49 2021 [INFO]: Last Floor Beacon Config Success Time:
      None
Wed Oct 6 17:03:49 2021 [INFO]: Floor Beacon Config Failure Count: 0Wed Oct  6 17:03:49 2021 [INFO]: Last Floor Beacon Config Failure Time:
    None

How do I monitor BLE scans in the IoX Application?

  1. To monitor the IoX Application scan log file in real-time, execute the following command: 

    tail -F /data/logs/dnas_ble_scans.log

  2. This command will continuously display the log file's output as it updates with new scan information.

  3. The IoX Application scans update every 5 minutes, but they may occur more frequently if the scan table becomes full.

Table 2. Output Descriptions

Field

Description
Profile Beacon profile such as iBeacon, Eddystone URL, Eddystone UID, or Unknown.
MAC MAC address of the beacon scanned.
RSSI Last Received Signal Strength Indicator (RSSI) of the beacon detected.
Count Number of times the beacon was heard since the last scan values were dumped.
Interval Average interval between detections of the beacon.
Last-heard Time elapsed since the beacon was last detected based on the latest scan values. 

AP# tail -F /data/logs/dnas_ble_scans.log

Sat Sep 18 05:44:57 2021 [INFO]: Profile        MAC               RSSI  Count  Interval  Last-heard
Sat Sep 18 05:44:57 2021 [INFO]: iBeacon        00:00:00:00:00:0F 63    16           1S  0000D:00H:00M:01S
Sat Sep 18 05:44:57 2021 [INFO]: Kontakt Secure F1:01:AF:4E:8A:3B 55    1            0S  0000D:00H:00M:02S
Sat Sep 18 05:44:57 2021 [INFO]: Kontakt Telem  F1:01:AF:4E:8A:3B 55    1            0S  0000D:00H:00M:03S
Sat Sep 18 05:44:57 2021 [INFO]: iBeacon        F1:01:AF:4E:8A:3C 56    1            0S  0000D:00H:00M:01S
Sat Sep 18 05:44:57 2021 [INFO]: iBeacon        D1:03:15:95:D6:F3 77    1            0S  0000D:00H:00M:03S
Sat Sep 18 05:44:57 2021 [INFO]: Kontakt Secure DF:03:AB:CD:C2:DB 86    2            3S  0000D:00H:00M:01S
Sat Sep 18 05:44:57 2021 [INFO]: iBeacon        DF:03:AB:CD:C2:DC 76    2            2S  0000D:00H:00M:02S
Sat Sep 18 05:44:57 2021 [INFO]: Unknown        18:04:ED:04:1C:5F 62    7            1S  0000D:00H:00M:01S
Sat Sep 18 05:44:57 2021 [INFO]: Kontakt Secure C3:05:7E:BD:25:D4 81    1            0S  0000D:00H:00M:04S
Sat Sep 18 05:44:57 2021 [INFO]: iBeacon        C3:05:7E:BD:25:D5 85    3            1S  0000D:00H:00M:01S
Sat Sep 18 05:44:57 2021 [INFO]: iBeacon        CB:06:D8:B5:A7:97 86    1            0S  0000D:00H:00M:03S
Sat Sep 18 05:44:57 2021 [INFO]: iBeacon        D8:06:04:DE:80:59 88    1            0S  0000D:00H:00M:04S
Sat Sep 18 05:44:57 2021 [INFO]: Kontakt Secure FF:07:D0:2F:6A:AF 79    1            0S  0000D:00H:00M:02S
Sat Sep 18 05:44:57 2021 [INFO]: iBeacon        FF:07:D0:2F:6A:B0 79    3            1S  0000D:00H:00M:01S
Sat Sep 18 05:44:57 2021 [INFO]: Unknown        36:08:36:6C:DA:E8 81    5            1S  0000D:00H:00M:01S
Sat Sep 18 05:44:57 2021 [INFO]: Kontakt Secure C6:09:26:9D:4D:94 73    2            2S  0000D:00H:00M:01S
Sat Sep 18 05:44:57 2021 [INFO]: iBeacon        C6:09:26:9D:4D:95 73    1            0S  0000D:00H:00M:02S
Sat Sep 18 05:44:57 2021 [INFO]: iBeacon        C1:0A:21:02:A7:D8 77    3            1S  0000D:00H:00M:02S
Sat Sep 18 05:44:57 2021 [INFO]: Kontakt Secure FD:0C:9B:17:A2:22 88    1            0S  0000D:00H:00M:03S

What files exist in the IOx application?

The following log files are generated when the application is running and are located in the directory /data/logs

Table 3. Log Files
Log File Name Description
dnas_ble_scans.log Active log file recording data on BLE devices scanned by the application.
dnas_ble.log Active log file for debug messages, located in the temporary directory due to its high write frequency and to utilize the partition's I/O capabilities.
dnas_ble_metrics.log Active log file that records metric messages related to the IOx application's performance and operations.
dnas_ble_last_restart.log When the IOx application restarts, the current dnas_ble.log file is copied to this file to help troubleshoot the cause of the restart.
dnas_ble_metrics_last_restart.log When the IOx application restarts, dnas_ble_metrics.log is copied to this file to aid in diagnosing the reasons for the restart based on the metrics recorded before it occurred.
dnas_ble_scans_last_restart.log When the IOx application restarts, dnas_ble_scans.log is copied to this file to aid in diagnosing the reasons for the restart based on BLE scanning activity recorded prior to the restart.
dnas_ble_scans_1.log A rotated log file for BLE device scans. It is part of the log file management system that helps control file size by archiving older entries.
dnas_ble_metrics_1.log Rotated log file containing historical metric messages, also part of the log rotation strategy.
dnas_ble_1.log Rotated log file that includes debug messages for the application, ensuring older logs are archived for size management.
dnas_ble_stdout.log Log file capturing the standard output and error streams of the IOx application, which is useful for reviewing the application's console output and any error messages.

The following configuration files are generated when the application is running and are located in the directory /data/logs

Table 4. Configuration Files

Configuration File Name

Description

dnas_ble_config.json

Configuration settings for the BLE radio; these settings are used to reload the last configuration upon restart.

The following are binary files installed specifically for the IOx application. All the files are located in the directory: /var/dnas_ble

File Name Description
/dnas_ble_iox_app IOx application binary used to scan and configure floor beacons.
dnas_ble_iox_app_start.sh Script to start the application and restart it in case of failure.