This document describes the logging capabilities available on the Nexus 7000.
Logflash is an 8 GB compact flash card (USB) mounted as "logflash:" filesystem for persistent storage of assorted logging information, such as syslog messages, debug output, core files, and Embedded Event Manager (EEM) information.
The filesystem structure might be damaged if the device is reset in a write operation. For example, if it was reloaded or power-cycled by the user or reset by an In Service Software Upgrade (ISSU), crashed, or so on.
N7K1# dir logflash: compact flash is either not present or not formatted
If the logflash filesystem is not mounted, try to manually mount it.
N7K1# mount logflash: Failed to mount logflash
If the logflash filesystem is still not mounted, try to eject/reinsert it.
Run the file system check.
N7K1# system health check logflash Unmount successful... Fix any file system errors ...done.
If all else fails, the logflash can be reformatted.
N7K1# format logflash: This command is going to erase the contents of logflash:. Do you want to continue? (yes/no) [n] y Notifying services to stop accessing the device... Formatting logflash: mke2fs 1.35 (28-Feb-2004) Formatting completed
NX-OS logging is not only robust, but is also saved as a file and thus is persistent across reloads.
Files are rotated (once they reach 10MB).
Non-default virtual device contexts (VDCs) and standby supervisor logs can be read (or copied to a remote location) from the default VDC.
Nexus# show clock 21:19:03.878 UTC Fri Jan 25 2013 Nexus# show ver | in uptime Kernel uptime is 16 day(s), 2 hour(s), 45 minute(s), 59 second(s)
Nexus# show file logflash://sup-active/log/messages 2008 Jan 1 14:05:54 %IDEHSD-2-MOUNT: logflash: online 2008 Jan 1 14:06:07 %MODULE-5-ACTIVE_SUP_OK: Supervisor 6 is active (serial: JAF1545BTGH) 2008 Jan 1 14:06:07 %PLATFORM-5-MOD_STATUS: Module 6 current-stat
Nexus# dir logflash://sup-standby/vdc_3/log/messages 219040 Jul 16 20:51:25 2012 vdc_3/log/messages
Only level 1-2 messages are printed to the console due to the baud rate. (An increase in the baud rate is an option to print messages above levels 1-2.)
The command show log nvram prints only level 1-2 messages.
The command show log logfile prints messages in the logging buffer-saved at /var/log/external/.
If the show log command does not show current logs or logging is stopped, then check the /var/log directory as shown here:
As seen above, /var/log is full. Then, check the actual contents of /var/log to see which file consumes the directory. It is possible, because of user defined debugs or core files, that the directory fills up and hence logging is stopped.
In the above example, libdt_helper.log consumes most of the space and hence there are issues with logging. Refer to Cisco bug ID CSCue98451.
Common Mistake About Logging
The "logging level <feature> <level>" does NOT cause that feature to print messages at that level. It actually tells the syslog function to "only" print messages for that feature to the logging buffer/server if they are of that level as "print threshold".
The example below shows a syslog that printed a message for Ethernet Port Manager (ETHPM), which has a default level of 5. Since the below message is a level 5, it meets the logging level "threshold".
Nexus(config)# int e 3/1 Nexus(config-if)# shut 2013 Jan 25 21:42:07 Nexus %ETHPORT-5-IF_DOWN_ADMIN_DOWN: Interface Ethernet3/1 is down (Administratively down)
If you change the level to 3, the messages are no longer printed. This can severely hinder the ability to troubleshoot.
Nexus(config-if)# logging level ethpm 3 Nexus(config)# int e 3/1 Nexus(config-if)# no shut Nexus(config-if)# sh log last 1 2013 Jan 25 21:42:07 Nexus %ETHPORT-5-IF_DOWN_ADMIN_DOWN: Interface Ethernet3/1 is down (Administratively down)
The accounting log for Authentication, Authorization, and Accounting (AAA) and local, allows us to see all the config commands run on the devices from any user.
Nexus# show accounting log Fri Mar 15 10:19:58 2013:type=update:id=console0:user=Ciscoadmin: cmd=configure terminal ; interface Ethernet1/1 (SUCCESS) Fri Mar 15 10:19:59 2013:type=update:id=console0:user=Ciscoadmin: cmd=configure terminal ; interface Ethernet1/1 ; shutdown (REDIRECT) Fri Mar 15 10:19:59 2013:type=update:id=console0:user=Ciscoadmin: cmd=configure terminal ; interface Ethernet1/1 ; shutdown (SUCCESS) Fri Mar 15 10:20:03 2013:type=update:id=console0:user=Ciscoadmin: cmd=configure terminal ; interface Ethernet1/1 ; no shutdown (REDIRECT) Fri Mar 15 10:20:03 2013:type=update:id=console0:user=Ciscoadmin: cmd=configure terminal ; interface Ethernet1/1 ; no shutdown (SUCCESS)
In Release 5.x and later (due to Cisco bug ID CSCtf04410), you can enable logging of "all" commands run on the device (not just the config commands) when you configure "terminal log-all".
N7K1(config)# terminal log-all N7K1(config)# show accounting log all Thu Mar 14 17:54:11 2013:type=update:id=console0:user=vbhutta: cmd=show system internal feature-mgr event-history errors (SUCCESS) Thu Mar 14 17:54:11 2013:type=stop:id=console0:user=Ciscoadmin:cmd= Thu Mar 14 17:54:11 2013:type=start:id=console0:user=Ciscoadmin:cmd= Thu Mar 14 17:54:11 2013:type=update:id=console0:user=Ciscoadmin: cmd=show system internal feature-mgr event-history msgs (SUCCESS)
Like syslogs, the accounting logs are also saved to a file, and thus are persistent after reloads.
N7K1# dir logflash://sup-active/vdc_1 130557 Jan 26 21:46:12 2013 accounting_log 250070 Jan 08 16:55:20 2013 accounting_log.1
If there is a supervisor failover, you can find the accounting log from the standby supervisor as well.
With NX-OS, there is "constant" logging of event-histories/debugs that run in the background by default (per VDC and per component).
No real impact to the CPU.
Configurable event-history log size.
N7K1(config)# ip adjmgr internal event-history errors size ? disabled Disabled *Default value is small large Large buffer medium Medium buffer small Small buffer
show run all | i event-history to see the configured size
Available for hardware and software components, as shown in these examples:
N7K1# show ip ospf internal event-history event
OSPF events for Process "ospf-1" 2013 Jan 23 17:45:06.518702 ospf 1 : : Got a URIB route notification message, xid 4294901878, count 3 2013 Jan 23 16:58:28.192141 ospf 1 : : Got a URIB route notification message, xid 4294901876, count 3 2013 Jan 23 16:35:47.630173 ospf 1 : : Got a URIB route notification message, xid 4294901874, count 3
Hardware Components at the Module Level
module-3# show hardware internal mac event-history info
1) At 596873 usecs after Tue Jan 22 17:06:52 2013 r2d2_fill_port_reset_info-275: Total ports 32 inst_affected 8 total_reset_time 4000 ms 2) At 577801 usecs after Tue Jan 22 17:06:52 2013 r2d2_fill_port_reset_info-187: Reset req. for Inband instance so choose all instances