Frequent Issues
-
Dashboard Log in Failure
Symptom
Dashboard Log in Fails.
Environment
Can occur in any of the QA or Production environment.
Possible Causes
1. Proxy server might be down.
2. WSO2_IS might be down.
3. Cerificate might have expired.
4. Any dependent service has stopped working.
Troubleshooting
a. Check token API, customer API , login API and accounts API.
I. https://<IP>:<port>/token
II. https://<APIM_IP>:<port>/t/<tenant.com>/cdp/v1/capabilities/customer
III. https://<APIM_IP>:<port>/t/<tenant.com>/cdp/v1/accounts/user/login
IV. https://<APIM_IP>:<port>/t/<tenant.com>/cdp/v1/accounts/
b. Easy way to check above 4 API status in browser is as follows:
-Open developer tool.
-Click on “Network” tab and
-Try to login again in dashboard, if any of the above 4 API is failing, click on response tab and check for the response.
c. If API response code is 404, check swagger in API publisher. Login to publisher with “tenantapiadmin” and click on edit API, bottom of the page, all API path are defined. Search for “login” and “customer” API.
d. If response code is 502, check for API redirection in proxy.
e. If response code is 500, check WSO2 APIM log, if error not found check UM log for login API.
f. In case of customer API failure, check KM log for error.
g. In case of Token API failure, check the IDM logs for error.
h. In case of Accounts API failure, check the UM logs for error.
i. If error shows SSL certificate issue, please check certificate expiry.
j. If certificate has not expired, check certificate shared between the components.
To check/verify above API, perform the following steps.
- Generate token for the same tenant API admin user using below API
Description : This API is used to generate the token for particular user.
URL : https://<IP>:<port>/token
Parameter:
client_id: <client_id>
client_secret: <client_secret>
username: <username@tenant.com>
password: <password>
grant_type: password
Header:
content-type:application/x-www-form-urlencoded
Method : POST
Response body:
{ "access_token": "f8871495-2ab8-37ac-b3bd-082c9b4a998c", "refresh_token": "da460cc7-6b22-3731-8ee1-a29f42da3f9d", "scope": "default", "token_type": "Bearer", "expires_in": 3278 }
- Call login API by providing the “access_token” in the below API.
Description : This API will return user details
Method : GET
URL : https://<APIM_IP>:<port>/t/<tenant.com>/cdp/v1/accounts/user/login
Header:
Authorization: Bearer <token>
Response body : { "id": "d163e7eb-ef00-43fb-a2e6-2226bcd4cc67", "userName": "newadmin", "tenant": "ust.com", "createdOn": "2018-03-06T11:31:53.000Z", "updatedOn": "2018-04-11T09:59:58.000Z", "state": "ACTIVE", "groupNames": [ "CDP_OPERATOR", "CDP_ADMIN", "admin" ] }
- Call login API by providing the “access_token” in below API
Description : This API used for getting the associated domain and locations.
URL : https://<APIM_IP>:<port>/t/<tenant.com>/cdp/v1/capabilities/customer
Header:
Authorization: Bearer <token>
Method: GET
Response Body: { "accountId": "5767e9e8-eede-4139-bf49-80f92cb670e4", "tenantId": "poland.com", "userName": "devop", "domains": [ { "domainName": "Parking", "permission": [ "read", "write" ] }, { "domainName": "Mobility", "permission": [ "read", "write" ] }, { "domainName": "Environment", "permission": [ "read", "write" ] }, { "domainName": "Lighting", "permission": [ "read", "write" ] } ], "locations": { "locationId": 10000, "locationName": "Root", "locationTypeId": 10000, "locationTypeName": "Root", "root": true, "childrenCount": 70, "children": [ { "locationId": 10005, "locationName": "update", "locationTypeId": 10010, "locationTypeName": "Country", "root": false, "childrenCount": 0, "tenantId": "poland.com" }, { "locationId": 10006, "locationName": "Canada1", "locationTypeId": 10010, "locationTypeName": "Country", "root": false, "childrenCount": 0, "tenantId": "poland.com" }, { "locationId": 10007, "locationName": "shimoga updated", "locationTypeId": 10010, "locationTypeName": "Country", "root": false, "childrenCount": 0, "tenantId": "poland.com" } ] } }
- If any API is not responding as expected, please check for the log in UM and KM service and if any error/exception, send it to development team.
Verification
Login into dashboard should be successful
Post Verification
If the issue is with any CKC component and not with configuration, please raise CDETS/BEMS/TAC.
1.Take the screenshot (in case of issues seen on the Dashboard)
2.Capture API request/response - in case of API issues
Raise CDETS with above details.
-
Dashboard Automatic Logout
Symptom
Dashboard automatically logs out every 1 min.
Environment
Can occur in any of the QA or Production environment.
Possible Causes
User details has not synced with Device engine from User Management.
Troubleshooting
a. Check Device Engine User sync.
b. Curl to find user in Device Engine (DE). curl -X POST \ https://DE_NB_HOSTNAME:PORT//fid-Authentication \ -H 'Content-Type: application/json' \ -H 'Postman-Token: 1c4acf5c-5414-477f-a56e-1441e7a1e1b1' \ -H 'cache-control: no-cache' \ -d '{ "Query": { "Find": { "User": { "sid": { "ne": "" } } } } } '
c. Find the user in the response. If user does not exist, please run user sync in DE and check if this resolves the problem If yes, post resolution and verification raise a CDETS/TAC.
d. If user exists, check location associated with that user.
e. Curl to find location of the user in DE curl -X POST \ https://COMPONENT_HOST:COMPONENT_PORT/fid-Authentication \ -H 'Content-Type: application/json' \ -H 'Postman-Token: 71705792-4018-4454-a19c-5dff71e0a958' \ -H 'cache-control: no-cache' \ -d '{ "Query": { "Find": { "UserLocationAssociation": { "userKeySid": { "ne": ""} } } } } '
f. Find the user id in response and check associated locations, if not found, follow step k.
g. If location found, check domain associated with user.
h. Curl to find domain in DE curl -X POST \ https://COMPONENT_HOST:COMPONENT_PORT/fid-Authentication \ -H 'AppKey: CDP-App' \ -H 'Content-Type: application/json' \ -H 'Postman-Token: d1bb0ef0-374d-4144-8bae-7a7a274aa058' \ -H 'SensorCustomerKey: 500900' \ -H 'UserKey: 501150' \ -H 'cache-control: no-cache' \ -d '{ "Query": { "Find": { "UserDomainAssociation": { "sid": { "ne": "" } } } } } '
i. Search user id and check domain association, if not found, please assign domain to the associated group in Key Management.
j. If domain is found, still not getting sensors, please contact DE team.
k. To sync location without sync script in DB, Go to Dashboard select user and update group or location, user, location and domain will automatically sync with DE.
Verification
Login into the Dashboard.
Wait for more than 2 min.
User should remian logged in.
Post Verification
If any CKC component issue other than configuration, please raise CDETS/BEMS/TAC.
-
All calls to Location API Fail
Symptom
All calls to Location APIs fail.
Environment
Can occur in any of the QA or Production environment.
Possible Causes
1. Proxy server might be down.
2. WSO2_IS might down.
3. Cerificate might have expired.
4. Core module might have got disconnected from the cluster.
5. Mongo DB might be down.
6. Location service might be down.
Troubleshooting
a. Check if Location service is running
b. If response code is 502, check proxy server redirections.
c. Log in to the dashboard application and go to the Platform health page to see the component status.
If the issue still persists, check for the logs in Location service under logs folder. If any error/exception is seen, contact the dev team.
Verification
Location API should work fine.
Post Verification
If any CKC component issue other than configuration, please raise CDETS/BEMS/TAC.
a) Take any screenshots (in case of issues seen on Dashboard)
b) Capture API request/response – in case of API issues.
Raise CDETS with above details.
-
Slow Resonse Time for Location API
Symptom
The response time of API increases with the increase in number of locations.
Environment
Can occur in any of the QA or Production environment.
Possible Causes
Memory issue in the machine where WSO2 and Core services are running.
Troubleshooting
a. If all location API is responding very slow, check for total count of locations, because if number of locations is more, API will take time to respond.
b.If location count is not more than 100, please check the location service machine RAM memory, if memory is less, than all location API will take time to respond.
Reach out to the dev team.
Verification
Location API should respond within 5 seconds .
Post Verification
If any CKC component issue other than configuration, please raise CDETS/BEMS/TAC.
a) Take backup of logs / errors in Location Management (/script/locations_log.log).
b) Take any screenshots (in case of issues seen on Dashboard)
b) Capture API request/response – in case of API issues.
Raise CDETS with above details.
-
Account API Response Time is Slow
Symptom
In dashboard all accounts/users do not load.
Environment
Can occur in any of the QA or Production environment.
Possible Causes
1. If more than 300 users exist in LDAP, it will make accounts API slow.
2. If RAM memory is less.
Troubleshooting
a. If accounts API is responding very slow, check for total count of accounts, because if accounts are more, API will take time to respond.
b. If accounts count is not more than 300, check the accounts service machine RAM memory, if memory is less, than all API will take time to respond.
Please contact the dev team.
Verification
If issue occurred because of RAM memory, after memory is freed, API should respond within 30 seconds.
Post Verification
If any CKC component issue other than configuration, please raise CDETS/BEMS/TAC.
a) Take backup of logs / errors in Location Management (/script/locations_log.log).
b) Take any screenshots (in case of issues seen on Dashboard)
b) Capture API request/response – in case of API issues.
Raise CDETS with above details.
-
API response code is HTTP 404
Symptom
API responds with response code 404 or not found or no matching resource found.
Environment
Can occur in any of the QA or Production environment.
Possible Causes
1. In WSO2 API manager, swagger file missed or wrong swagger file is updated.
2. Issue with redirections in proxy server.
Troubleshooting
a. Verify if API is published in WSO2 and subscribed.
b. If published and subscribed, check the API complete path in swagger as mentioned below
- Go to publisher, click on API edit button, on the first page, you can observe all API path.
c. If API path does not exist, check in the swagger file.
Swagger file path
WSO2_IS_5.7andAPIM_2.6_FRESH_SETUP\WSO2_configfile\Swagger\4.2_Swagger \Common_API
API CONTEXT - cdp/{version}/tenants/users
swagger_file - tenants_users.json
API CONTEXT - cdp/{version}/validate
swagger_file - validate.json
API CONTEXT - cdp/{version}/counts
swagger_file - counts.json
API CONTEXT - cdp/{version}/locations/hierarchy/users
swagger_file - locations_hierarchy_users.json
API CONTEXT - cdp/{version}/accounts/userlist
swagger_file - accounts_userList.json
API CONTEXT - cdp/{version}/changepassword
swagger_file - changepassword.json
API CONTEXT - cdp/{version}/forgotpassword
swagger_file - forgotpassword.json
API CONTEXT - cdp/{version}/activiti
swagger_file - activiti.json
API CONTEXT - cdp/{version}/token/verify
swagger_file - Token_verify.json
Below swagger file path
WSO2_IS_5.7andAPIM_2.6_FRESH_SETUP\WSO2_configfile\Swagger\4.2_Swagger \
/cdp/{version}
All_API_Swagger.json
Verification
API should work as expected.
Post Verification
If any CKC component issue other than configuration, please raise CDETS/BEMS/TAC.
a) Take backup of logs / errors in WSO2 APIM (/<APIM_HOME>/repository/logs/wso2carbon.log).).
b) Take any screenshots (in case of issues seen on Dashboard)
b) Capture API request/response – in case of API issues.
Raise CDETS with above details.
-
Call to any API Fails
Symptom
If API is responding with response code 404 or 500 or 502 or 503 or other http code other than 200 and 202
Environment
Can occur in any of the QA or Production environment.
Possible Causes
1. Proxy server is down.
2. Internal server error has occurred.
3. Core service is not running or not in cluster.
4. Time-Series Data Engine, or Real-Time/Device Engine services are down.
5. API is not published in WSO2 publisher
6.Not subscribed to published API in WSO2 store application.
Troubleshooting
a. Check if API is published in WSO2 and subscribed.
b.Check the redirection in required proxy servers.
c. If above steps are fine, check if all the dependent services are running, if any error/exception is found, send the logs to the dev team.
Verification
API should work as expected.
Post Verification
If any CKC component issue other than configuration, please raise CDETS/BEMS/TAC.
a) Take backup of logs / errors in WSO2 APIM (/<APIM_HOME>/repository/logs/wso2carbon.log).).
b) Take any screenshots (in case of issues seen on Dashboard)
b) Capture API request/response – in case of API issues.
Raise CDETS with above details.
-
Sensors Not Appearing on the Dashboard
Symptom
Sensors are not seen on the dashboard.
Environment
Can occur in any of the QA or Production environment.
Possible Causes
The logged in user's role has not been assigned any domain in the Key Management portal. URL for KM : https:<KM_IP>:<PORT>/sdp_km
Troubleshooting
a. Check assigned group for logged in user using below API.
API: /cdp/v1/accounts/{accountsId}
b. Login to KM as “devops” user and select each “Role” and check if required domain is assigned or not. If not assigned, assign required domain to get sensors on the dashboard.
c. If assigned, open developer tool and click on “Network” tab in browser and refresh the page. On refreshing the page, “Network” tab will show all called API, search for “customer” API (/t/<tenant_name>/cdp/v1/capabilities/customer), click on response tab and check for required domain name in response content.
d. Check if the DE API is working. if not, please contact the dev team.
Verification
On logging into the dashboard application, all the required sensors should be seen.
Post Verification
If any CKC component issue other than configuration, please raise CDETS/BEMS/TAC.
a) Take backup of logs / errors in WSO2 APIM (/<DEVICE_HOME>/logs/deviceengine.log)
b) Take any screenshots (in case of issues seen on Dashboard)
b) Capture API request/response – in case of API issues.
Raise CDETS with above details.
-
User Deletion Fails
Symptom
Delete or offboard user API (/cdp/v1/accounts/offboard/{accountId}) does not allow to delete the user, and shows message as below
{ "errorCode": "USER-0007", "message": "Please verify if this is the only user in its groups.", "description": "Error in deleting user: testdelete" }
Environment
Can occur in any of the QA or Production environment.
Possible Causes
If user is assigned to any group and that group has only one user, in this case, user cannot be deleted (Group cannot exist without any user, at least one user should exist in a group).
Troubleshooting
Please assign one more user to that group and delete the first user.
Verification
User should get deleted from the User Management.
Post Verification
If any CKC component issue other than configuration, please raise CDETS/BEMS/TAC.
-
Slow Response Time for All the APIs
Symptom
Dasboard login is very slow.
Environment
Can occur in any of the QA or Production environment.
Possible Causes
Processor consuming more than 90% in WSO2 or respective service.
Memory consuming 100% in WSO2 or respective service.
Troubleshooting
a. Check the APIM and IDM machine memory, if memory is enough, check memory of all core machine.
b. If memory is not an issue, verify the CPU consumption. If CPU consumption is more, then verify which process is has more CPU usage.
c. If memory and CPU is fine in all machines, check for environment issues.
d. Open the developer tool and click on “Network” tab in browser. find out the API which is taking the longest response time.
e. In order to validate the API, prepare a CURL and try to access from reverse proxy.
f. If API response is slow, change IP and port with API manager proxy and hit the API.
g. If API is slow, then change IP and Port of backend component. Here you may get error in response, but if response is slow, than check environment issues like RAM, CPU usage.
h. If API response is as expected, check for environment issue in any of the previous component.
Verification
API should respond within 5 seconds.
Post Verification
If any CKC component issue other than configuration, please raise CDETS/BEMS/TAC.
a) Take backup of logs / errors in WSO2 or respective service.
b) Take any screenshots (in case of issues seen on Dashboard)
b) Capture API request/response – in case of API issues.
Raise CDETS with above details.
-
Connectivity Issue with DB
Symptom
Login failure to dashboard / API not working.
Environment
Can occur in any of the QA or Production environment.
Possible Causes
1. DB service is down.
2. Port is closed.
3. Authentication issue.
4. Calling service IP not allowed to connect DB.
Troubleshooting
a. Please check that DB service is running and port is opened.
b. If DB service has stopped, please start service and open the port.
Verification
Failed API response should show proper response data.
Post Verification
If any CKC component issue other than configuration, please raise CDETS/BEMS/TAC.
a) Take backup of logs / errors in respective service.
b) Take any screenshot (in case of issues seen on Dashboard)
b) Capture API request/response – in case of API issues.
Raise CDETS with above details.
-
Intermittent Dashboard Login Issue
Symptom
Dashboard login works intermittently.
Environment
Can occur in any of the QA or Production environment.
Possible Causes
1. API response is slow.
2. Token API, customer API and login API taking long time to respond
Troubleshooting
a. If APIs are responding slowly, please refer Response Time of All APIs Is Slow.
b. If APIs are working one time and failing another time, try to call API with direct WSO2 APIM IP and PORT.
c. If issues persists, try to call API with core machine proxy once and without proxy once.
d. If API working fine in step c, try to call API with load balancer (in prod).
Verification
Login to Dashboard works everytime.
Post Verification
If any CKC component issue other than configuration, please raise CDETS/BEMS/TAC.
a) Take backup of logs / errors in respective service.
b) Take any screenshot (in case of issues seen on Dashboard)
b) Capture API request/response – in case of API issues.
Raise CDETS with above details.
-
Email Issue in New User Creation
Symptom
New user does not receive email to reset password.
Environment
Can occur in any of the QA or Production environment.
Possible Causes
1.Email notification is not enabled in WSO2 IS carbon portal. (https://<WSO2_IS_HOSTNAME>:<PORT>/carbon)
2. SMTP detail is wrong.
3. Email ID provided while onboarding is incorrect.
Troubleshooting
a. Login to IDM (carbon) with tenant admin user, click on main tab in left side vertical menu.
b. Go to Identity Providers -> Resident -> Account Management Policies -> Account Recovery.
c. Check if “Enable Notification Based Password Recovery” check box is enabled.
d. If not, enable to get mail on user creation.
e. If above steps are fine, check for WSO2 IDM configuration as below.
f. Open IS_HOME/repository/conf/axis2/axis2.xml for the following entry details. <transportSender name="mailto" class="org.apache.axis2.transport.mail.MailTransportSender"> <property key="mail.smtp.from"><user>@<host>.com</property> <property key="mail.smtp.user"><username@hostname.com></property> <property key="mail.smtp.password"><password></property> <property key="mail.smtp.host"><smtp_host></property> <property key="mail.smtp.port"><SMTP_PORT></property> <property key="mail.smtp.starttls.enable">true</property> <property key="mail.smtp.auth">true</property> </transportSender>
g. If above configuration is fine, open IS_HOME/repository/conf/output-event-adapters.xml and check smtp details <adapterConfig type="email"> <!-- Comment mail.smtp.user and mail.smtp.password properties to support connecting SMTP servers which use trust based authentication rather username/password authentication --> <property key="mail.smtp.from"><user>@<host>.com</property> <property key="mail.smtp.user"><username></property> <property key="mail.smtp.password"><password></property> <property key="mail.smtp.host"><smtp_host></property> <property key="mail.smtp.port"><SMTP_PORT></property> <property key="mail.smtp.starttls.enable">true</property> <property key="mail.smtp.auth">true</property> <!-- Thread Pool Related Properties --> <property key="minThread">8</property> <property key="maxThread">100</property> <property key="keepAliveTimeInMillis">20000</property> <property key="jobQueueSize">10000</property> </adapterConfig>
Verification
After onboarding, new user should get password reset mail.
Post Verification
If any CKC component issue other than configuration, please raise CDETS/BEMS/TAC.
a) Take backup of logs / errors in WSO2 service.
b) Take any screenshot (in case of issues seen on Dashboard)
b) Capture API request/response – in case of API issues.
Raise CDETS with above details.
-
Password Issue
Symptom
While changing the user password, system accepts any one of the last 10 passwords.
Environment
Can occur in any of the QA or Production environment.
Possible Causes
Password History Feature is not set.
Troubleshooting
a) Go to Identity Providers -> Resident -> Password Policies -> Password History.
b) Check if “Enable Password History Feature” check box is enabled.
c) Check if “Password History validation count” value is set as 10.
d) If not set, enable the same.
Verification
Change user password in the Dashboard application, system should not accept user’s last 10 password as the new password.
Post Verification
Not applicable as it is a configurational issue.
-
Mediation Policy Error
Symptom
API response is an XML showing Mediation Policy error
Environment
Can occur in any of the QA or Production environment.
Possible Causes
1. “Password change time” claim not added in wso2 IS carbon.
2. WSO2 IS server is down.
3. RBAC policy not added in wso2 IS carbon
Troubleshooting
If you find mediation policy error in API response, Please perform following steps:
• Check “Password change time” claim in IDM.
• Login to IDM to check claim details by tenant admin.
• Go to Claims -> List -> http://wso2.org/claims.
• Search for “Password change time” claim.
• Open claim and check “Mapping Attribute”
it should be “passwordTimestamp”.
• Also check “cdpMediationPolicy.xml” file under “<WSO2_APIM>/repository/tenants/<tenant_id(number)>/synapse-configs/default/sequences”.
If everything is fine, please check below points.
• Check WSO2 IS is up and working fine.
• Check whether RBAC policy is applied or not.
Verification
ALL tenant API should work fine with correct response.
Post Verification
If any CKC component issue other than configuration, please raise CDETS/BEMS/TAC.
a) Take backup of logs / errors in WSO2 APIM. (<WSO2_APIM_HOME>/repository/logs/wso2carbon.log)
b) Take any screenshot (in case of issues seen on Dashboard)
b) Capture API request/response – in case of API issues.
Raise CDETS with above details.
-
Core vert.x issue with error -1
Symptom
API response is very slow or responds with the following message "Please contact Administrator".
Environment
Can occur in any of the QA or Production environment.
Possible Causes
Services are out of the cluster.
Troubleshooting
Check Location service and Key management logs, if you find log as follows “handler not found”.
a) Take backup of logs / errors in Location(/script/logs/locations_log.log) and Key Management (/script/logs/ km_log.log)
b) Take any screenshots (in case of issues seen in Dashboard)
c) Take API request/response – in case of API issues
Restart the corresponding core.
Verification
All API should respond with proper data, without any error code.
Post Verification
If any CKC component issue other than configuration, please raise CDETS/BEMS/TAC.
a) Take backup of logs / errors in Location service (/script/logs/km_log.log)
b) Take any screenshot (in case of issues seen on Dashboard)
b) Capture API request/response – in case of API issues.
Raise CDETS with above details.
-
All Ports in DMZ Close on Machine Restart
Symptom
API stops working, WSO2 portal also stops working.
Environment
Can occur in any of the QA or Production environment.
Possible Causes
All Ports in DMZ have closed upon machine restart.
Troubleshooting
• If machine has been restarted, please open required ports in WSO2 IDM, APIM and in core, if not open.
• Port list:
o IDM port: 9445
o APIM port: 9443, 8243, 8280.
o PostgreSQL port: 5432 (default)
o Mongo port: 27017 (default)
o RabbitMQ port: 5672 (default)
o Core port: UM, LM, KM, Audit and activity port are mentioned in the corresponding property file, also open cluster port in each core property file.
Verification
All API should work fine.
Login to Dashboard should be successful.
Post Verification
If any CKC component issue other than configuration and port issue, please raise CDETS/BEMS/TAC.
-
Core API Failing Intermittently.
Symptom
Core API responding with unauthorized (401) response code intermittently.
Environment
Can occur in any of the QA or Production environment.
Possible Causes
Machine times are not in sync.
Troubleshooting
a. Verify the WSO2 Identity server machine time, WSO2 API manager machine time and OpenLdap machine time. All three-machine time should be same or in sync.
b. If any one of the machine time is not in sync, please sync its time with all the other machines.
c. Restart is not required for this change
Verification
All API should work fine.
Post Verification
If any CKC component issue other than configuration and time sync issue, please raise CDETS/BEMS/TAC.
-
Authorization Error Message from FND
Symptom
FND server is showing authorization error.
Environment
Can occur in any of the QA or Production environment.
Possible Causes
Necessary roles are not provided to the user in the CKC Network Service.
Troubleshooting
Assign "Northbound API" role to the user.
Verification
Authorization error should be resolved.
Post Verification
If any CKC component issue other than configuration and time sync issue, please raise CDETS/BEMS/TAC.