WAAS - Troubleshooting the CIFS AO

Chapter: Troubleshooting the CIFS AO

This article describes how to troubleshoot the CIFS AO.

Contents

• 1 CIFS AO Troubleshooting
  • 1.1 CIFS AO Logging
  • 1.2 Windows Print Accelerator Troubleshooting

CIFS AO Troubleshooting

The CIFS accelerator transparently optimizes CIFS traffic on ports 139 and 445.

You can verify the general AO configuration and status with the show accelerator and show license commands, as shown in Figure 1. The Enterprise license is required for CIFS accelerator operation.

Figure 1. Verifying Accelerator Status

Next, verify the status that is specific to the CIFS AO by using the show accelerator cifs command, as shown in Figure 2. You want to see that the CIFS AO is Enabled, Running, and Registered, and that the connection limit is displayed. If the Config State is Enabled but the Operational State is Shutdown, it indicates a licensing problem.

Figure 2. Verifying CIFS Accelerator Status

Use the show running-config command to verify that the CIFS traffic policy is properly configured. You want to see accelerate cifs for the WAFS application action and you want to see appropriate match conditions listed for the CIFS classifier, as follows:

WAE674# sh run | include CIFS

      classifier CIFS
    name WAFS classifier CIFS action optimize full accelerate cifs
WAE674# sh run | begin CIFS

...skipping
   classifier CIFS
      match dst port eq 139
      match dst port eq 445
   exit

Use the show statistics connection optimized cifs command to check that the WAAS device is establishing optimized CIFS connections. Verify that "TCDL" appears in the Accel column for a connection. A "C" indicates that the CIFS AO was used.

WAE674# sh stat conn opt cifs
Current Active Optimized Flows:                      3
   Current Active Optimized TCP Plus Flows:          3
   Current Active Optimized TCP Only Flows:          0
   Current Active Optimized TCP Preposition Flows:   1
Current Active Auto-Discovery Flows:                 0
Current Active Pass-Through Flows:                   0
Historical Flows:                                    100

D:DRE,L:LZ,T:TCP Optimization,
A:AOIM,C:CIFS,E:EPM,G:GENERIC,H:HTTP,M:MAPI,N:NFS,S:SSL,V:VIDEO

ConnID  Source IP:Port        Dest IP:Port          PeerID             Accel
1074    10.10.10.10:2704      10.10.100.100:445     00:14:5e:84:24:5f  TCDL       <------Look for "C"

If you see "TDL" in the Accel column, the connection was optimized by transport optimizations only and was not inspected by the CIFS AO. This situation can happen if the CIFS AO is disabled, the Enterprise license is not configured, or if the maximum connection limit is reached.

If you see a "G" instead of a "C" in the Accel column, then the connection was pushed down from the CIFS AO to the generic AO and was optimized with transport optimizations only. This situation can happen if the connection requires SMB2 or a digital signature and an error message is logged for it.

In version 4.1.3, the syslog has the following error message for digitally signed connections:

2009 Apr 25 13:42:08 wae java: %WAAS-CIFSAO-4-131230: (146708) Connection to test1.example.com will be handled by 
 generic optimization only, since test1.example.com requires digital signing.

In version 4.1.5 and later, check the CIFS internal error logs to see the reason why the connection was pushed down to the generic AO. In the cifs_err.log, look for this message for SMB2 connections:

2009-06-29 10:15:04,996  WARN (actona.cifs.netbios.IPacketerHandlerOrigCifs:139) Thread-2 -  Received SMBv2 packet 
 from host 10.56.64.205. Pushing down the connection.

In the cifs_err.log, look for this message for digitally signed connections:

2009-10-29 05:37:54,541  WARN (actona.rxFlow.cifs.requests.NegotiateRequest:359) lightRxFlowPool-4 - Request ID: 148/266 
 Connection to 10.56.78.167 will be handled by generic optimization only, since 10.56.78.167 requires digital signing.

To view similar information from the Central Manager, choose the WAE device, then choose Monitor > Optimization > Connections Statistics.

Figure 3. Connection Statistics Report

You can view the CIFS connection statistics by using the show statistics connection optimized cifs detail command as follows:

WAE674# sh stat connection optimized cifs detail
Connection Id:            1801
   Peer Id:                  00:14:5e:84:24:5f
   Connection Type:          EXTERNAL CLIENT
   Start Time:               Thu Jun 25 06:15:58 2009
   Source IP Address:        10.10.10.10
   Source Port Number:       3707
   Destination IP Address:   10.10.100.100
   Destination Port Number:  139
   Application Name:         WAFS                                      <-----Should see WAFS
   Classifier Name:          CIFS                                      <-----Should see CIFS
   Map Name:                 basic
   Directed Mode:            FALSE
   Preposition Flow:         FALSE
   Policy Details:
          Configured:        TCP_OPTIMIZE + DRE + LZ
             Derived:        TCP_OPTIMIZE + DRE + LZ
                Peer:        TCP_OPTIMIZE + DRE + LZ
          Negotiated:        TCP_OPTIMIZE + DRE + LZ
             Applied:        TCP_OPTIMIZE + DRE + LZ
   Accelerator Details:
               Configured:   CIFS                                      <-----Should see CIFS configured
                  Derived:   CIFS
                  Applied:   CIFS                                      <-----Should see CIFS applied
                     Hist:   None

                                               Original            Optimized
                                   -------------------- --------------------
   Bytes Read:                                   189314             10352510
   Bytes Written:                              91649704                28512

. . . 

Connection details:
Chunks: encoded 3,  decoded 49922,  anchor(forced) 0(1)
Total number of processed messges: 1820
 num_used_block per msg: 0.140659
Ack: msg 1609,  size   7066 B                                          
Encode bypass due to:                                                  
    last partial chunk: chunks: 1,  size:    142 B                     
    skipped frame header: messages: 138,  size:  27202 B               
Nacks: total 0                                                         
R-tx: total 0
Encode LZ latency:      0.060 ms per msg
Decode LZ latency:      0.071 ms per msg
Aggregation encode:  Retransmissions: 0                                   <-----Packets lost between peers
    level 0: chunks:        3  hits:        0 miss:        3
    level 1: chunks:        0  hits:        0 miss:        0
    level 2: chunks:        0  hits:        0 miss:        0
    level 3: chunks:        0  hits:        0 miss:        0
Aggregation decode: Collisions: 0
    level 0: chunks:   174093  hits:   128716 miss:        0
    level 1: chunks:        0  hits:        0 miss:        0
    level 2: chunks:        0  hits:        0 miss:        0
    level 3: chunks:        0  hits:        0 miss:        0
Aggregation stack memory usage: Sender:    452 B  Receiver:   9119 B
Noise filter: Chunks: 0, Bytes:      0 B
. . . 

If the Retransmissions counter increases, it means that packets are getting lost in the middle, between the two peer WAEs. This situation will result in lower throughput. You should investigate possible causes for packet lose in the network between the two peer WAEs.

You can view the CIFS request statistics by using the show statistics cifs requests command as follows:

Figure 4. Inspecting CIFS Request Statistics

CIFS AO Logging

The following log files are available for troubleshooting CIFS AO issues:

• Transaction log files: /local1/logs/tfo/working.log (and /local1/logs/tfo/tfo_log_*.txt)
• CIFS internal log file: /local1/errorlog/cifs/cifs_err.log
• Debug log files: /local1/errorlog/cifsao-errorlog.current (and cifsao-errorlog.*)

For easier debugging, you should first set up an ACL to restrict packets to one host.

WAE674(config)# ip access-list extended 150 permit tcp host 10.10.10.10 any
WAE674(config)# ip access-list extended 150 permit tcp any host 10.10.10.10

To enable transaction logging, use the transaction-logs configuration command as follows:

wae(config)# transaction-logs flow enable
wae(config)# transaction-logs flow access-list 150

You can view the end of a transaction log file by using the type-tail command as follows:

wae# type-tail tfo_log_10.10.11.230_20090715_130000.txt
:EXTERNAL CLIENT :00.14.5e.84.24.5f :basic :WAFS :CIFS :F :(DRE,LZ,TFO) (DRE,LZ,TFO) (DRE,LZ,TFO) (DRE,LZ,TFO) 
(DRE,LZ,TFO) :<None> :(CIFS) (CIFS) (CIFS) :<None> :<None>  :0 :180
Wed Jul 15 15:48:45 2009 :1725 :10.10.10.10 :2289 :10.10.100.100 :139 :OT :START :EXTERNAL CLIENT :00.14.5e.84.24.5f :basic :WAFS 
:CIFS :F :(DRE,LZ,TFO) (DRE,LZ,TFO) (DRE,LZ,TFO) (DRE,LZ,TFO) (DRE,LZ,TFO) :<None> :(CIFS) (CIFS) (CIFS) :<None> :<None> :0 :177
Wed Jul 15 15:48:55 2009 :1725 :10.10.10.10 :2289 :10.10.100.100 :139 :OT :END : EXTERNAL CLIENT :(CIFS) :0 :0 :159 :221

To set up and enable debug logging of the CIFS AO, use the following commands.

NOTE: Debug logging is CPU intensive and can generate a large amount of output. Use it judiciously and sparingly in a production environment.

You can enable detailed logging to the disk as follows:

WAE674(config)# logging disk enable 
WAE674(config)# logging disk priority detail

You can enable debug logging for connections in the ACL:

WAE674# debug connection access-list 150

The options for CIFS AO debugging are as follows:

WAE674# debug accelerator cifs ?
  all         enable all CIFS accelerator debugs
  shell       enable CIFS shell debugs

You can enable debug logging for CIFS connections and then display the end of the debug error log as follows:

WAE674# debug accelerator cifs all
WAE674# type-tail errorlog/cifsao-errorlog.current follow

Windows Print Accelerator Troubleshooting

The Windows print accelerator optimizes print traffic between clients and a Windows print server.

Troubleshooting the Windows print accelerator is similar to troubleshooting the CIFS AO. You can verify the general AO configuration and status with the show accelerator and show license commands, as shown in Figure 1. The CIFS accelerator must be enabled and the Enterprise license is required. Next, verify the status specific to the CIFS AO by using the show accelerator cifs command.

Use the show statistics windows-print requests command and verify that the "Documents spooled" and "Pages spooled" counters are incrementing, as follows:

WAE# sh stat windows-print requests
Statistics gathering period:  hours: 6 minutes: 4 seconds: 2 ms: 484
Documents spooled: 29                                                         <-----Should be incrementing
Pages spooled: 3168                                                           <-----Should be incrementing
Total commands: 61050
Remote commands: 849
ALL_COMMANDS total: 61050 remote: 849 async: 58719 avg local: 1.813ms avg remote: 177.466ms
. . .