Nexus 7000 Logging Capabilities

Introduction

This document describes the logging capabilities available on the Nexus 7000.

Logflash Capabilities

• Logflash is an 8 GB compact flash card (USB) mounted as "logflash:" filesystem for persistent storage of assorted logging information, such as syslog messages, debug output, core files, and Embedded Event Manager (EEM) information.

  SUP1

  

  SUP2

  

• The filesystem structure might be damaged if the device is reset in a write operation. For example, if it was reloaded or power-cycled by the user or reset by an In Service Software Upgrade (ISSU), crashed, or so on.

  N7K1# dir logflash:
  compact flash is either not present or not formatted

• If the logflash filesystem is not mounted, try to manually mount it.

  N7K1# mount logflash:
  Failed to mount logflash

• If the logflash filesystem is still not mounted, try to eject/reinsert it.
• Run the file system check.

  N7K1# system health check logflash
  Unmount successful...
  Fix any file system errors ...done.

• If all else fails, the logflash can be reformatted.

  N7K1# format logflash:
  This command is going to erase the contents of logflash:.
  Do you want to continue? (yes/no)  [n] y
  Notifying services to stop accessing the device...
  Formatting logflash:
  mke2fs 1.35 (28-Feb-2004)
  Formatting completed

• NX-OS logging is not only robust, but is also saved as a file and thus is persistent across reloads.
• Files are rotated (once they reach 10MB).
• Non-default virtual device contexts (VDCs) and standby supervisor logs can be read (or copied to a remote location) from the default VDC.

  Nexus# show clock
  21:19:03.878 UTC Fri Jan 25 2013
  Nexus# show ver | in uptime
  Kernel uptime is 16 day(s), 2 hour(s), 45 minute(s), 59 second(s)

  Nexus# show file logflash://sup-active/log/messages
  2008 Jan  1 14:05:54  %IDEHSD-2-MOUNT: logflash: online
  2008 Jan  1 14:06:07  %MODULE-5-ACTIVE_SUP_OK: Supervisor 6 is active 
  (serial: JAF1545BTGH)
  2008 Jan  1 14:06:07  %PLATFORM-5-MOD_STATUS: Module 6 current-stat
  
  Nexus# dir logflash://sup-standby/vdc_3/log/messages
       219040    Jul 16 20:51:25 2012  vdc_3/log/messages

• Only level 1-2 messages are printed to the console due to the baud rate. (An increase in the baud rate is an option to print messages above levels 1-2.)
• The command show log nvram prints only level 1-2 messages.
• The command show log logfile prints messages in the logging buffer-saved at /var/log/external/.
• If the show log command does not show current logs or logging is stopped, then check the /var/log directory as shown here:

  N7K1# show system internal flash
  Mount-on                  1K-blocks      Used   Available   Use%  Filesystem
  /                            409600     62432      347168     16   /dev/root
  /proc                             0         0           0      0   proc
  /sys                              0         0           0      0   none
  /isan                       1048576    366864      681712     35   none
  /var                          51200       544       50656      2   none
  /etc                           5120      1620        3500     32   none
  /nxos/tmp                     40960      1268       39692      4   none
  /var/log                      51200     51200           0    100   none
  /var/home                      5120        84        5036      2   none
  /var/tmp                     307200      2972      304228      1   none
  /var/sysmgr                 1572864        60     1572804      1   none
  /var/sysmgr/ftp              512000    108476      403524     22   none
  /var/sysmgr/srv_logs         102400         0      102400      0   none
  /var/sysmgr/ftp/debug_logs    10240         0       10240      0   none
  /dev/shm                    3145728    748672     2397056     24   none
  /volatile                    512000         0      512000      0   none
  /debug                         5120       108        5012      3   none
  /dev/mqueue                       0         0           0      0   none
  --------------------------------------SNIP---------------------------------

  As seen above, /var/log is full. Then, check the actual contents of /var/log to see which file consumes the directory. It is possible, because of user defined debugs or core files, that the directory fills up and hence logging is stopped.

  N7K1# sh system internal dir /var/log/external/
                                                                  ./         420
                                                                 ../         380
                                                         glbp.debug          231
                                                      libfipf.24944            0
                                                              vdc_4/          80
                                                      libfipf.24115            0
                                                              vdc_3/          80
                                                      libfipf.23207            0
                                                              vdc_2/          80
                                                   libdt_helper.log     51523584
                                                       libfipf.5582            0
                                                       libfipf.4797            0
                                                       libfipf.4717            0
                                                           messages       651264
                                                       startupdebug            0
                                              eobc_port_test_result            3
                                              mgmt_port_test_result            3
                                                    bootup_test.log        18634

  You can also use dir log: to view the contents.

  In the above example, libdt_helper.log consumes most of the space and hence there are issues with logging. Refer to Cisco bug ID CSCue98451.

Common Mistake About Logging

• The "logging level <feature> <level>" does NOT cause that feature to print messages at that level. It actually tells the syslog function to "only" print messages for that feature to the logging buffer/server if they are of that level as "print threshold".

  The example below shows a syslog that printed a message for Ethernet Port Manager (ETHPM), which has a default level of 5. Since the below message is a level 5, it meets the logging level "threshold".

  Nexus(config)# int e 3/1
  Nexus(config-if)# shut
  2013 Jan 25 21:42:07 Nexus %ETHPORT-5-IF_DOWN_ADMIN_DOWN:
   Interface Ethernet3/1 is down (Administratively down)

  If you change the level to 3, the messages are no longer printed. This can severely hinder the ability to troubleshoot.

  Nexus(config-if)# logging level ethpm 3
  Nexus(config)# int e 3/1
  Nexus(config-if)# no shut
  Nexus(config-if)# sh log last 1
  2013 Jan 25 21:42:07 Nexus %ETHPORT-5-IF_DOWN_ADMIN_DOWN:
   Interface Ethernet3/1 is down (Administratively down)

Accounting Log

• The accounting log for Authentication, Authorization, and Accounting (AAA) and local, allows us to see all the config commands run on the devices from any user. 
  

  Nexus# show accounting log
  Fri Mar 15 10:19:58 2013:type=update:id=console0:user=Ciscoadmin:
  cmd=configure terminal ; interface Ethernet1/1 (SUCCESS)
  Fri Mar 15 10:19:59 2013:type=update:id=console0:user=Ciscoadmin:
  cmd=configure terminal ; interface Ethernet1/1 ; shutdown (REDIRECT)
  Fri Mar 15 10:19:59 2013:type=update:id=console0:user=Ciscoadmin:
  cmd=configure terminal ; interface Ethernet1/1 ; shutdown (SUCCESS)
  Fri Mar 15 10:20:03 2013:type=update:id=console0:user=Ciscoadmin:
  cmd=configure terminal ; interface Ethernet1/1 ; no shutdown (REDIRECT)
  Fri Mar 15 10:20:03 2013:type=update:id=console0:user=Ciscoadmin:
  cmd=configure terminal ; interface Ethernet1/1 ; no shutdown (SUCCESS)

• In Release 5.x and later (due to Cisco bug ID CSCtf04410), you can enable logging of "all" commands run on the device (not just the config commands) when you configure "terminal log-all".

  N7K1(config)# terminal log-all
  N7K1(config)# show accounting log all
  Thu Mar 14 17:54:11 2013:type=update:id=console0:user=vbhutta:
  cmd=show system internal feature-mgr event-history errors (SUCCESS)
  Thu Mar 14 17:54:11 2013:type=stop:id=console0:user=Ciscoadmin:cmd=
  Thu Mar 14 17:54:11 2013:type=start:id=console0:user=Ciscoadmin:cmd=
  Thu Mar 14 17:54:11 2013:type=update:id=console0:user=Ciscoadmin:
  cmd=show system internal feature-mgr event-history msgs (SUCCESS)

• Like syslogs, the accounting logs are also saved to a file, and thus are persistent after reloads.

  N7K1# dir logflash://sup-active/vdc_1
       130557    Jan 26 21:46:12 2013  accounting_log
       250070    Jan 08 16:55:20 2013  accounting_log.1

• If there is a supervisor failover, you can find the accounting log from the standby supervisor as well.

Event-History

• With NX-OS, there is "constant" logging of event-histories/debugs that run in the background by default (per VDC and  per component).
• No real impact to the CPU.
• Configurable event-history log size.

  N7K1(config)# ip adjmgr internal event-history errors size ?
    disabled  Disabled
              *Default value is small
    large     Large buffer
    medium    Medium buffer
    small     Small buffer
  
  show run all | i event-history to see the configured size

• Available for hardware and software components, as shown in these examples:

  Software Components

  N7K1#  show ip ospf internal event-history event
  
  OSPF events for Process "ospf-1"
  2013 Jan 23 17:45:06.518702 ospf 1 [6219]: [6250]:
   Got a URIB route notification message, xid 4294901878, count 3
  2013 Jan 23 16:58:28.192141 ospf 1 [6219]: [6250]:
   Got a URIB route notification message, xid 4294901876, count 3
  2013 Jan 23 16:35:47.630173 ospf 1 [6219]: [6250]:
   Got a URIB route notification message, xid 4294901874, count 3

  Hardware Components at the Module Level

  module-3# show hardware internal mac event-history info
  
  1) At 596873 usecs after Tue Jan 22 17:06:52 2013
      r2d2_fill_port_reset_info-275: Total ports 32
      inst_affected 8 total_reset_time 4000 ms
  2) At 577801 usecs after Tue Jan 22 17:06:52 2013
      r2d2_fill_port_reset_info-187: Reset req. for
      Inband instance so choose all instances