Guest

Authentication Protocols

Nexus 7000 Series Switch Problem with Remote User Authentication via SSH with a TACACS account

Techzone Article content

Document ID: 116335

Updated: Sep 11, 2013

Contributed by Scott Laffer, Cisco TAC Engineer.

   Print

Introduction

This document provides the steps required in order to troubleshoot and confirm that a Cisco Nexus 7000 Series switch is affected by the known software defect Cisco bug ID CSCud02139.

Prerequisites

Requirements

There are no specific requirements for this document.

Components Used

The information in this document is based on these software and hardware versions:

  • Cisco Nexus 7000 Series Switches
  • Cisco Nexus Operating System (NX-OS) Versions 5.2(5) to 5.2(7) Inclusive
  • Cisco NX-OS Versions 6.0(1) to 6.1(3) Inclusive

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.

Problem

Symptoms

Users are unable to remotely log in to a Nexus 7000 Series switch Virtual Device Context (VDC) with TACACS authentication.

Additionally, these messages are seen in the logs:

n7k-vdc-1# show log last 200 | grep TACACS
2013 May 13 17:17:31 n7k-vdc-1 TACACS-3-TACACS_ERROR_MESSAGE All servers
failed to respond
2013 May 13 17:17:46 n7k-vdc-1 TACACS-3-TACACS_ERROR_MESSAGE All servers
failed to respond
2013 May 13 17:18:06 n7k-vdc-1 TACACS-3-TACACS_ERROR_MESSAGE All servers
failed to respond
2013 May 13 17:18:12 n7k-vdc-1 TACACS-3-TACACS_ERROR_MESSAGE All servers
failed to respond
2013 May 13 17:18:16 n7k-vdc-1 TACACS-3-TACACS_ERROR_MESSAGE All servers
failed to respond
2013 May 13 17:20:26 n7k-vdc-1 TACACS-3-TACACS_ERROR_MESSAGE All servers
failed to respond
2013 May 13 17:20:39 n7k-vdc-1 TACACS-3-TACACS_ERROR_MESSAGE All servers
failed to respond
2013 May 13 17:21:50 n7k-vdc-1 TACACS-3-TACACS_ERROR_MESSAGE All servers
failed to respond
2013 May 13 17:22:09 n7k-vdc-1 TACACS-3-TACACS_ERROR_MESSAGE All servers
failed to respond
n7k-vdc-1#

Conditions

This problem is encountered on Nexus 7000 Series switches that run Cisco NX-OS Versions between 5.2(5) and 5.2(7), as well as between 6.0.1 to 6.1(3).

The VDC must use TACACS authentication, like this example:

n7k-vdc-1# show run tacacs+

!Command: show running-config tacacs+
!Time: Mon May 13 17:20:57 2013

version 6.1(2)
feature tacacs+

ip tacacs source-interface mgmt0
tacacs-server timeout 30
tacacs-server host 192.0.2.9 key 7 "keypassword"
aaa group server tacacs+ default
server 192.0.2.9
use-vrf management


n7k-vdc-1# show run aaa

!Command: show running-config aaa
!Time: Mon May 13 17:21:30 2013

version 6.1(2)
aaa authentication login default group default
aaa authorization config-commands default group default
aaa authorization commands default group default
aaa accounting default group default
no aaa user default-role
aaa authentication login error-enable
tacacs-server directed-request

Troubleshoot

  1. Confirm the TACACS Server Status

    • Confirm that the Nexus 7000 Series switch is successfully able to ping the TACACS server via the correct Virtual Routing and Forwarding (VRF).
    • Confirm that the TACACS server still successfully authenticates users on other devices.
  2. Check the Authentication, Authorization, and Accounting (AAA) Process Error Logs

    Use this command in order to check the AAA process error logs:

    n7k-vdc-1# show system internal aaa event-history errors

    1) Event:E_DEBUG, length:54, at 786852 usecs after Mon May 13 17:22:09 2013
    [102] All Configured methods failed for default:default

    2) Event:E_DEBUG, length:53, at 786796 usecs after Mon May 13 17:22:09 2013
    [102] protocol TACACS failed with server group default

    3) Event:E_DEBUG, length:54, at 379206 usecs after Mon May 13 17:22:09 2013
    [102] All Configured methods failed for default:default

    4) Event:E_DEBUG, length:53, at 379172 usecs after Mon May 13 17:22:09 2013
    [102] protocol TACACS failed with server group default

    5) Event:E_DEBUG, length:54, at 89083 usecs after Mon May 13 17:21:51 2013
    [102] All Configured methods failed for default:default

    6) Event:E_DEBUG, length:53, at 89051 usecs after Mon May 13 17:21:51 2013
    [102] protocol TACACS failed with server group default


  3. Check the TACACS+ Process Error Logs

    Use this command in order to check the TACACS+ process error logs:

    n7k-vdc-1# show system internal tacacs+ event-history errors

    1) Event:E_DEBUG, length:88, at 786728 usecs after Mon May 13 17:22:09 2013
    [100] switch_tac_server: Unreachable servers case .setting error code for
    aaa session 0

    2) Event:E_DEBUG, length:77, at 786726 usecs after Mon May 13 17:22:09 2013
    [100] switch_tac_server: no more server in the server group for
    aaa session 0

    3) Event:E_DEBUG, length:103, at 786680 usecs after Mon May 13 17:22:09 2013
    [100] connect_tac_server: non blocking connect failed, switching server for
    aaa session id(0) rtvalue(3)

    4) Event:E_DEBUG, length:97, at 786677 usecs after Mon May 13 17:22:09 2013
    [100] non_blocking_connect(171): getaddrinfo(DNS cache fail) with retcode:-1
    for server:192.0.2.9

    5) Event:E_DEBUG, length:62, at 786337 usecs after Mon May 13 17:22:09 2013
    [100] tplus_encrypt(655):key is configured for this aaa session.

    6) Event:E_DEBUG, length:95, at 786287 usecs after Mon May 13 17:22:09 2013
    [100] tplus_make_acct_request(1343):Not calling the name-resolution routine
    as rem_addr is empty

    7) Event:E_DEBUG, length:63, at 786285 usecs after Mon May 13 17:22:09 2013
    [100] tplus_make_acct_request(1308):Accounting userdata:console0

    8) Event:E_DEBUG, length:63, at 786266 usecs after Mon May 13 17:22:09 2013
    [100] init_tplus_req_state_machine:Global source-interface mgmt0

    9) Event:E_DEBUG, length:48, at 785842 usecs after Mon May 13 17:22:09 2013
    [100] is_intf_up_with_valid_ip(1129):Port is up.

    10) Event:E_DEBUG, length:57, at 785812 usecs after Mon May 13 17:22:09 2013
    [100] is_intf_up_with_valid_ip(1126):Proper IOD is found.

    11) Event:E_DEBUG, length:52, at 785799 usecs after Mon May 13 17:22:09 2013
    [100] Exiting function: get_if_index_from_global_conf

    12) Event:E_DEBUG, length:66, at 785797 usecs after Mon May 13 17:22:09 2013
    [100] Function get_if_index_from_global_conf: found interface mgmt0

    13) Event:E_DEBUG, length:53, at 785783 usecs after Mon May 13 17:22:09 2013
    [100] Entering function: get_if_index_from_global_conf

    14) Event:E_DEBUG, length:68, at 785781 usecs after Mon May 13 17:22:09 2013
    [100] init_tplus_req_state_machine:Falling to globally configured one

    15) Event:E_DEBUG, length:79, at 785779 usecs after Mon May 13 17:22:09 2013
    [100] init_tplus_req_state_machine:No source-interface configured for this group


  4. Debug TACACS+ Authentication Requests

    • Turn on debugging for TACACS+ Authentication requests.
    • AAA debugging outputs these logs:

      n7k-vdc-1# debug tacacs+ aaa-request
      n7k-vdc-1# show logging logfile last 5
      2013 May 13 18:20:26.077572 tacacs: tplus_encrypt(655):key is configured
      for this aaa session.
      2013 May 13 18:20:26.077918 tacacs: non_blocking_connect(171): getaddrinfo
      DNS cache fail) with retcode:-1 for server:192.0.2.9
      2013 May 13 18:20:26.077938 tacacs: connect_tac_server: non blocking connect
      failed, switching server for aaa session id(0) rtvalue(3)
      2013 May 13 18:20:26.077978 tacacs: switch_tac_server: no more server in the
      server group for aaa session 0
      2013 May 13 18:20:26.077993 tacacs: switch_tac_server: Unreachable servers
      case .setting error code for aaa session 0


  5. Perform a Packet Capture on the TACACS Server

    A packet capture on the TACACS server shows that no packets arrive from the VDC.

  6. Perform a Ethanalyzer Capture on the Nexus 7000 Series Switch

    An Ethanalyzer capture shows that no packets egress towards the TACACS server.

  7. Check the Running Processes on the VDC

    The show proc cpu sort command shows 33 instances (32 defunct) of the TACACSD process running.
    n7k-vdc-1# show proc cpu sort | include tacacs
    1538 16 16 1014 0.0% tacacsd
    1855 16 10 1625 0.0% tacacsd
    2163 16 10 1678 0.0% tacacsd
    2339 15 23 676 0.0% tacacsd
    3820 15 10 1595 0.0% tacacsd
    3934 16 13 1272 0.0% tacacsd
    4416 25 8 3211 0.0% tacacsd
    4470 16 23 734 0.0% tacacsd
    5577 26 12 2191 0.0% tacacsd
    6592 969767 14589069 66 0.0% tacacs
    6934 16 13 1297 0.0% tacacsd
    8878 16 13 1252 0.0% tacacsd
    8979 16 12 1345 0.0% tacacsd
    10153 26 11 2453 0.0% tacacsd
    10202 15 8 1888 0.0% tacacsd
    10331 26 11 2368 0.0% tacacsd
    10482 16 14 1190 0.0% tacacsd
    14148 15 11 1433 0.0% tacacsd
    14385 14 10 1496 0.0% tacacsd
    14402 15 9 1775 0.0% tacacsd
    20678 16 9 1785 0.0% tacacsd
    20836 16 13 1246 0.0% tacacsd
    21257 15 13 1212 0.0% tacacsd
    21617 15 9 1749 0.0% tacacsd
    22159 15 12 1328 0.0% tacacsd
    23776 15 12 1320 0.0% tacacsd
    24017 25 9 2788 0.0% tacacsd
    29496 15 8 1990 0.0% tacacsd
    29972 15 11 1368 0.0% tacacsd
    30111 25 9 2847 0.0% tacacsd
    30204 15 9 1721 0.0% tacacsd
    30409 16 13 1254 0.0% tacacsd
    32410 15 8 1876 0.0% tacacsd


Solution

The VDC encounters the known software defect Cisco bug ID CSCud02139.

The TACACSD process spawns child processes that get stuck. This reaches a maximum of 32 processes, and it is unable to spawn any more in order to pass the authentication.

Confirmation

  1. Confirm that there are 33 instances of TACACSD. You can use the command show proc cpu sort | grep -c 'tacacsd' in order to count the instances.
  2. Perform an ethanalyzer capture, and confirm that the request does not leave the Nexus 7000 Series switch.
  3. Match the previous log messages.

Workarounds

There are three possibilities. Remove all of the TACACS configuration, and remove and readd the feature and the configuration. Another option is to perform a supervisor switchover. Or you can reload the VDC.

Resolved Versions

  • NX-OS Versions 5.2(9) and Later in the 5.2 Train
  • NX-OS Versions 6.1(3) and Later in the 6.1 Train

Related Information

Updated: Sep 11, 2013
Document ID: 116335