Introduction
This document will be an inclusive list of which log files should be referenced for various processes and troubleshooting of various issues.
Tip: f you aren't sure what log file to look at, you can do a long listing of all files the log directory and sort by time to see the last logs written to! Use the command ls -lah. For live troubleshooting, you may want to run ls -lahtr /var/log/springpath to see which log file was edited most recently at the bottom of the list.
root@SpringpathControllerG9ES5WGWDG:/var/log/springpath# ls -lah
total 433M
-rw-r----- 1 root adm 123M Sep 28 16:15 stat-storfs.log
-rw-r----- 1 root adm 63M Sep 28 16:14 debug-storfs.log
-rw-rw-rw- 1 root root 50M Sep 28 16:14 stNodeMgr.log
-rw-r----- 1 root adm 47M Sep 28 16:15 debug-cipmonitor.log
-rw-r----- 1 root adm 41M Sep 28 16:15 stat-scvmclient.log
-rw-rw-rw- 1 root root 22M Sep 28 16:14 stMgr.log
HyperFlex Installation
| Topic |
Log File |
Notes |
| Troubleshooting GUI |
/var/log/springpath/hxinstaller.log |
The log contains all of the API's fired from client to server and their payload/responses |
| Troubleshooting Backend |
/var/log/springpath/hxinstaller.log |
GO Lang Server |
| |
/var/log/springpath/ucs_logs/ucs_automation |
UCSM Failure |
| Host Related Issues |
C:\ProgramData\Cisco\HyperFlex\Install\Log
|
Hyper-V
|
| |
/var/log/hostd.log
|
ESXi
|
| |
/var/run/log/vmkernel.log
|
ESXi
|
| |
/var/run/log/scvmclient.log
|
ESXi Datastore mounts
|
| Deploy and Create/Expand Failures |
/var/log/tomact7/catalina.out
|
Tomcat logs. Tomcat versions may differ across versions
|
| |
/var/log/springpath/stDeploy.out
|
Thrift Failures
|
| |
/var/log/springpath/stBootstrapGuiBackend.log
|
REST Failures
|
| |
/var/log/springpath/springpath_validation.log |
Validation Services |
| ESXi Agent Manager |
C:\Program FIles\VMware\vCenter Server\vpxd\scripts |
ESXi Agent Manager user fails to log in to VC |
HyperFlex Upgrades
| Log File |
Notes |
| /var/log/springpath/stMgr.log |
Check on all controller VM's when cluster upgrade fails |
| /var/log/springpath/stUpgradeSvc.log |
All backend logs |
| /var/log/tomcat7/catalina.out |
REST Logs |
| /var/log/springpath/springpath_validation.log |
Validations |
HyperFlex Bootstrapping
| Log File |
Notes |
| /var/log/springpath/stMgr.log |
ESXi Cluster |
| /var/log/springpath/stUpgradeSvc.log |
Hyper-V Cluster
|
| |
|
HX Connect
| Log File |
Notes |
| /var/log/springpath/hxmanager.log |
GUI Logs
All URL requests are logged. To also enable response logging, edit:
/opt/springpath/storfs-mgmt-ui/config/config.json
LogHttpRequest and/or LogHttpResponse to "true"
restart hxmanager
**remember to turn off, else logging is too verbose
|
| |
|
| |
|
HX & Intersight
| Log FIle |
Notes |
| /opt/partner/cisco-hxdc-run/hxdc_current/bin/hxdp |
Intersight Binary |
| /opt/partner/cisco-hxdc-run/hxdc_current/version |
Current Version |
| /nfs/system/hx_device_connector/db/connector.db |
Cloud Connection Information |
| /var/log/springpath/hx_device_connector.log |
grep for "HX_PLAT" |
Network Logs
| Log File |
Notes |
| /var/log/springpath/ucs_logs |
Rack Unit Log |
Data Replication
| Log File |
Notes |
| /var/log/springpath/stDataSvcMgr.log |
|
| /var/log/springpathdebug-replsvc.log |
|
| /var/log/springpathdebug-repl-cipmonitor.log |
|
| /var/log/springpathreplicationNetworkConfig.log |
|
| /var/log/springpath/debug-storfs.log |
|
Stretch Cluster
| Log File |
Notes |
| /opt/springpath/config/stretch.tunes |
Install & Deploy |
HX Plugin
| Log File |
Notes |
| /var/log/vmware/vsphere-client/logs/vsphere_client_virgo.log |
vCenter Log |
| /var/log/spingpath/stMgr.log |
STCTL |
Audit Logs
| Log File |
Notes |
| /var/log/springpath/audit-rest.log |
REST |
| /var/log/vmware-vsphere-client/logs/vsphere_client_virg.log |
Plugin
|
| /var/log/shell.log |
Stcli
|
Core REST APIs / AAA
| Log File |
Notes |
| /var/log/nginx/ssl-access.log |
Nginx |
| /var/log/tomcat8/catalina.out |
Tomcat. Tomcat versions may change between HX versions |
| /var/log/springpath/stMgr.log |
StMgr |
| /var/log/springpath/hxSvcMgr.log |
HxSvcMgr |
| /var/log/springpath/hxHyperVSvcMgr.log |
HxHyperVSvcMgr |
| /var/log/springpath/audit-rest.log |
|
| /var/log/springpath/aaa.log |
Debug log of AAA subsystem |
ASUP
| Log File |
Notes |
| /var/log/springpath/asupcl.log |
|
| /etc/cron.d/callhome |
ASUP cron job |
Data at Rest Encryption
| Topic |
Log File |
Notes |
| |
/var/log/springpath/hxmanager.log |
Logs the calls that are made from the browser to the GUI proxy server that is forwarded to the REST server |
| |
/var/log/tomcat7/catalina.out |
Logs the REST server calls. Tomcat versions may change between HX versions |
| REST API's |
|
all calls are proxied through NGINX & hxmanager (go-server) to actual web-server (ex: tomcat) |
| |
/var/log/springpath/hxmanager.log |
HX Manager (go-server) |
|
|
/varlog/nginx/*.log |
Nginx Logs
|
| |
/etc/nginx/conf.d/springpath.conf |
Nginx conf file, look for routings
|
| |
/var/log/tomcat7/catalina.out |
Tomcat
|
| |
/var/log/tomcat7/* |
Tomcat
|
| |
grep SED /var/log/springpath/stMgr.log | cut -c 1-40 |
REST API Calls stMgr service APIs
|
| SED Server/PPP |
|
|
| |
/var/log/springpath/stevents.log |
Events that start with Encryption
|
| |
/var/log/springpath/encryption.log |
grep for enable-security, modify-security, disable-security, secure-drives, rescan-inventory, secure-erase
|
| SED Client |
|
|
| |
/usr/share/springpath/stevents.logs |
Reports current SED state (Capable, Enabled, Locked) |
| |
/var/log/syslog |
|
| Debugging UCSM Logs |
|
|
| |
DME |
-Verify that lstoragelocal or lstorageRemote sent as part of OobStorageConfig
-Search for "drive security" or "KMIP" to find errors logged in DME code
-If using remote policy and OOB CIMC Mgmt IP is configured, make sure ConfigMgmtIpRules stage ran
|
| |
BladeAG |
-Search logs after receiving "OobStorageConfigFSM" or "AdminOobStorageConfigFSM" stimulus
-Check if route addition failure is reported as part of ConfigMgmtIpRules
-Check for XML starting with "lock-operation" in case of Enable/Modify/Disable Unlock Operation
-Check for XML starting with "clear-secure-drive", "enable-security-on-jbod", "secure-drive-group", "clear-secure-foreign, config-drive"
NOTE: Last 2 XML checks for UCSM specific issues, all are for UCSM and certificate management issues
|
| CIMC |
|
|
| |
/obfl/obfl-log* |
|
| |
/var/log/messages* |
grepping for "storage" extracts the lines written by storaged and by mcserver involving storaged |
| |
/var/log/storaged_startup_log.txt |
-contains the first 5000 lines written by storaged and by mcserver involving storaged
-search for same keywords as with bladeAG
-during certificate create, search for "cert" to see if there is a further log indicating the reason for failure
|
Feedback