Table Of Contents
Troubleshooting vPath and vServices
CLI Configuration
Event Log
Event Log Configuration Format
Viewing the Event Log Configuration
Viewing Event Logs
Event Log Configuration Persistence
Configuration and Restrictions
VNS Agent
Inspection Process
Service Path Process
Policy Engine Process
Restrictions
Using vPath Ping to Determine Connectivity
Troubleshooting the Cisco VSG and VEM Interactions When the Cisco VSG is on a VXLAN in a Service-Chain
Debugging the Traffic Flow Via a Service Chain
Troubleshooting the Service Chain by Excluding the Cisco VSG Node
VEM/vpath Configured Correctly on a VEthernet Interface for a ServiceChain
Cisco VSG on a VXLAN is not working
Troubleshooting vPath and vServices
This chapter includes the following sections:
•CLI Configuration
•Using vPath Ping to Determine Connectivity
•Troubleshooting the Cisco VSG and VEM Interactions When the Cisco VSG is on a VXLAN in a Service-Chain
See the Cisco Virtual Security Gateway for Nexus 1000V Series Switch Troubleshooting Guide, Release 4.2(1)VSG1(4.1) for more information on troubleshooting VSG.
CLI Configuration
This section includes the following topics:
•Event Log
•CLI Configuration
Event Log
This section includes the following topics:
•Event Log Configuration Format
•Viewing the Event Log Configuration
•Viewing Event Logs
•Event Log Configuration Persistence
Event Log Configuration Format
The configuration is displayed using this format:
[no] event-log inspect {{error | info} | {{ftp {error | info | warn | pkt_trace}} | {rsh
{error | info | pkt_trace}} | {tftp {error | info }}}} [terminal]
You can configure event logs for either the inspection process or one of its modules. For example, you can use the event-log inspect error terminal command to enable error events for the inspection process and to display these messages on the terminal where the command was entered.
Viewing the Event Log Configuration
You can display the event log configuration by using the show event-log all command. This example shows how to display the event logs for all the processes and their modules:
event-log inspect tftp error
event-log inspect rsh error
event-log inspect ftp error terminal
event-log policy_engine attr-mgr error
event-log service-path sp pkt-error terminal
Viewing Event Logs
Event logs are always logged in a process that is specific to the message buffer. Process logging in the event log buffer does not incur any overhead. In addition to using the show event-log command, you can display messages on a terminal where the event logs are enabled by using the terminal option, which is useful for reproducing a certain behavior.
The show command shows all the processes that are integrated with the event log Cisco VSG infrastructure. You can display inspection event logs using the show system internal event-log inspect command. The Cisco VSG event log infrastructure is a layer on top of the Cisco NX-OS event log infrastructure. Event logs can be redirected to a file and exported.
To display event logs on the terminal, use the terminal option while configuring the event. Different terminals can view different event logs. For example, use the event-log inspect ftp info terminal command to enable the information event logs for the inspection FTP module and to display the logs on the terminal. Use the event-log inspect rsh error terminal command to display only the error logs that are related to the RSH module. This command helps to debug various modules at the same time.
Event Log Configuration Persistence
You can save the event log configuration by using the event-log save config command. This command allows you to save all of the currently enabled event logs in a file. This file is read at the time of the module/process initialization with the event log infrastructure. The event log configuration that is relevant to the process is reapplied during initialization, which makes the event log configuration persistent across the process/system reboot. Some important things about the event log configuration are as follows:
•Terminal information is not reapplied for process or system restarts because that information might not be applicable.
•The event log configuration is independent of the other Cisco NX-OS configurations. The copy running-config startup-config and show running-config commands do not save and display the event log configuration.
•The event log configuration is specific to the individual system. In a high-availability setup, the configuration must be set up on both systems.
Configuration and Restrictions
Event logs CLIs are classified based on the process and its modules. This section descrubes event log commands.
This section includes the following topics:
•VNS Agent
•Inspection Process
•Service Path Process
•Using vPath Ping to Determine Connectivity
•Restrictions
VNS Agent
Virtual Network Service (VNS) agent-related event logs are maintained on the Virtual Supervisor Module (VSM), not on the Cisco VSG.
This section includes the following topics:
•Core Module
•VPath Module
•License Module
Core Module
Core events are those events that are related to port attach, port detach, Internet Protocol Database (IPDB), and to port-profile CLI.
This example shows how to enable/disable error messages for the vns_agent core module:
vsm# event-log vns-agent core-error [terminal] ----->enable messages to the terminal
vsm# no event-log vns-agent core-error [terminal] ----->disable messages to the terminal
This example shows how to enable/disable informational messages for the vns_agent core module:
vsm# event-log vns-agent core-info [terminal] ----->enable messages to the terminal
vsm# no event-log vns-agent core-info [terminal] ----->disable messages to the terminal
VPath Module
Because the vPath module works based on core-module events, you should always enable core module event logs before you enable the vPath module events.
This example shows how to enable/disable error messages for the vns_agent vPath module:
vsm# event-log vns-agent vpath-error [terminal] ----->enable messages to the terminal
vsm# no event-log vns-agent vpath-error [terminal] ----->disable messages to the terminal
This example shows how to enable/disable informational messages for the vns_agent vPath module:
vsm# event-log vns-agent vpath-info [terminal] ----->enable messages to the terminal
vsm# no event-log vns-agent vpath-info [terminal] ----->disable messages to the terminal
License Module
Because the license module works based on core-module events, you should always enable the core module event logs before enabling the license module.
This example shows how to enable/disable error messages for the vns_agent license module:
vsm# event-log vns-agent license-error [terminal] ----->enable messages to the terminal
vsm# no event-log vns-agent license-error [terminal] ----->disable messages to the
terminal
This example shows how to enable/disable informational messages for the vns_agent license module:
vsm# event-log vns-agent license-info [terminal] ----->enable messages to the terminal
vsm# no event-log vns-agent license-info [terminal] ----->disable messages to the terminal
Inspection Process
The inspection process uses event log commands for the inspection process and the File Transfer Protocol (FTP), Remote Shell (RSH), and Trivial File Transfer Protocol (TFTP) modules. These processes are all available on the Cisco VSG.
Use the event-log inspect error command to display configuration errors, process initialization errors, and so forth. This example shows how to enable/disable error messages for the inspection process:
vsg# event-log inspect error [terminal] ----->enable messages to the terminal
vsg# no event-log inspect error [terminal] ----->disable messages to the terminal
This example shows how to enable/disable informational messages for the inspection process:
vsg# event-log inspect info [terminal] ----->enable messages to the terminal
vsg# no event-log inspect info [terminal] ----->disable messages to the terminal
Use the event-log inspect ftp error command to display FTP packet processing errors. This example shows how to enable/disable error messages for the inspection FTP module:
vsg# event-log inspect ftp error [terminal] ----->enable messages to the terminal
vsg# no event-log inspect ftp error [terminal] ----->disable messages to the terminal
The command output is as follows:
Mon Oct 4 15:12:14 2010 ie_ftp: flow (->(ING), 6912), Bad ftp command.
Mon Oct 4 15:12:14 2010 ie_ftp: flow (->(ING), 6912), invalid PORT request / PASV reply.
This example shows how to enable/disable informational event log messages for the inspection FTP module:
vsg# event-log inspect ftp info [terminal] ----->enable messages to the terminal
vsg# no event-log inspect ftp info [terminal] ----->disable messages to the terminal
The command output is as follows:
Mon Oct 4 15:12:18 2010 ie_ftp: embryonic connection request (ip, port, proto, pfid, cid,
action, offload) = (192.168.1.20, 40074, tcp, 13569, 6912, 3,1).
Mon Oct 4 15:17:11 2010 ie_ftp: flow (<-(ING), 6912), more reply expected in cmd-reply.
This example shows how to enable/disable warning messages for the inspection FTP module:
vsg# event-log inspect ftp warn [terminal] ----->enable messages to the terminal
vsg# no event-log inspect ftp warn [terminal] ----->disable messages to the terminal
The command output is as follows:
Mon Oct 4 15:19:03 2010 ie_ftp: flow (<-(ING), 8192), ftp reply not terminated properly.
This example shows how to enable/disable packet trace messages for the inspection FTP module:
vsg# event-log inspect ftp pkt_trace [terminal] ----->enable messages to the terminal
vsg# no event-log inspect ftp pkt_trace [terminal] ----->disable messages to the terminal
The command output is as follows:
Mon Oct 4 15:31:46 2010 ie_ftp: flow (->(ING), 17152), flags(S:)
Mon Oct 4 15:31:54 2010 ie_ftp: flow (->(ING), 17152), cmd (USER)
This example shows how to enable/disable error messages for the inspection RSH module:
vsg# event-log inspect rsh error [terminal] ----->enable messages to the terminal
vsg# no event-log inspect rsh error [terminal] ----->disable messages to the terminal
This example shows how to enable/disable informational messages for the inspection RSH module:
vsg# event-log inspect rsh info [terminal] ----->enable messages to the terminal
vsg# no event-log inspect rsh info [terminal] ----->disable messages to the terminal
The command output is as follows:
Mon Oct 4 15:21:29 2010 ie_rsh: emryonic connection request (ip, port, proto, pfid, cid,
action, offload) = (192.168.1.10, 1021, tcp, 22529, 11264, 3, 1).
This example shows how to enable/disable packet trace messages for the inspection RSH module:
vsg# event-log inspect rsh pkt_trace [terminal] ----->enable messages to the terminal
vsg# no event-log inspect rsh pkt_trace [terminal] ----->disable messages to the terminal
The command output is as follows:
Mon Oct 4 15:25:26 2010 ie_rsh: flow (->(ING), 15872), rsh inspect action stop punt
This example shows how to enable/disable error messages for the inspection TFTP module:
vsg# event-log inspect tftp error [terminal] ----->enable messages to the terminal
vsg# no event-log inspect tftp error [terminal] ----->disable messages to the terminal
This example shows how to enable/disable informational messages for the inspection TFTP module:
vsg# event-log inspect tftp info [terminal] ----->enable messages to the terminal
vsg# no event-log inspect tftp info [terminal] ----->disable messages to the terminal
The command output is as follows:
Mon Oct 4 15:27:42 2010 ie_tftp: emryonic connection request (ip, port, proto, pfid, cid,
action, offload) = (192.168.1.10, 32771, udp, 33281, 16640, 3, 1)
Service Path Process
This section includes the following topics:
•Service Path Module
•Service Path Flow Manager
•AC Module
The service path process exposes event log output for the VSN service path, flow manager, AC infrastructure modules.
Service Path Module
The event-log service-path sp error command can display a failure to initialize the FE, and so forth. This example shows how to enable/disable error messages for the service path module:
vsg# event-log service-path sp error [terminal] ----->enable messages to the terminal
vsg# no event-log service-path sp error [terminal] ----->disable messages to the terminal
Use the event-log service-path sp info command to display FE initialization messages, control path messages, and so forth. This example shows how to enable/disable informational messages for the service path module:
vsg# event-log service-path sp info [terminal] ----->enable messages to the terminal
vsg# no event-log service-path sp info [terminal] ----->disable messages to the terminal
The event-log service-path sp pkt-error command can display failures to read or write a packet, a corrupted packet, and so forth.
This example shows how to enable/disable packet error messages for the service path module:
vsg# event-log service-path sp pkt-error [terminal] ----->enable messages to the terminal
vsg# no event-log service-path sp pkt-error [terminal] ----->disable messages to the
terminal
The event-log service-path sp pkt-info command can display the field description of a packet, where the packet arrived from or going to, decisions taken on the packet, and so forth.
This example shows how to enable/disable packet informational messages for the service path module:
vsg# event-log service-path sp pkt-info [terminal] ----->enable messages to the terminal
vsg# no event-log service-path sp pkt-info [terminal] ----->disable messages to the
terminal
The event-log service-path sp pkt-detail command can display the first few 100 bytes of the incoming packets.
This example shows how to enable/disable detailed packet messages for the service path module:
vsg# event-log service-path sp pkt-detail [terminal] ----->enable messages to the terminal
vsg# no event-log service-path sp pkt-detail [terminal] ----->disable messages to the
terminal
Service Path Flow Manager
This example shows how to enable/disable the packet messages for the service path flow manager module:
vsg# event-log service-path fm error [terminal] ----->enable messages to the terminal
vsg# no event-log service-path fm error [terminal] ----->disable messages to the terminal
AC Module
The event-log service-path ac error command can display failures to initialize the AC, timer, FD, pending queue, and so forth.
This example shows how to enable/disable error messages for the AC module:
vsg# event-log service-path ac error [terminal] ----->enable messages to the terminal
vsg# no event-log service-path ac error [terminal] ----->disable messages to the terminal
The event-log service-path ac info command can display AC initialization messages, control path messages, and so forth.
This example shows how to enable/disable informational messages for the AC module:
event-log service-path ac info [terminal] ----->enable messages to the terminal
no event-log service-path ac info [terminal] ----->disable messages to the terminal
Policy Engine Process
This section includes the following topic:
•Attribute Manager Module
Attribute Manager Module
This section describes the attribute manager-related errors.
You can use the event-log policy-engine attr-mgr error command to display the policy ID for a PE evaluation lookup based on the VNSP ID, IP address, zone name resolution, attribute fetched, and so forth.
This example shows how to enable/disable error messages for the attribute manager module:
vsg# event-log policy-engine attr-mgr error [terminal] ----->enable messages to the
terminal
vsg# no event-log policy-engine attr-mgr error [terminal] ----->disable messages to the
terminal
This example shows how to enable/disable informational messages for the attribute manager module:
vsg# event-log policy-engine attr-mgr info [terminal] ----->enable messages to the
terminal
vsg# no event-log policy-engine attr-mgr info [terminal] ----->disable messages to the
terminal
Restrictions
The event log configuration has the following restrictions:
•Terminal information is not reapplied in case of process restart/system restart bacause it may or may not be applicable.
•Event log configuration is independent of the other Cisco NX-OS configurations. The Cisco NX-OS copy running-config startup-config and show running-config commands do not save and display event log configuration.
•Event log configuration is specific to the individual system. In the high availability (HA) setup, this configuration must be done on both of the systems.
Using vPath Ping to Determine Connectivity
You can use the vpath ping command to determine the connectivity between the Cisco VSG and the VEM.
This example shows how to ping the Cisco VSG connections and if they are reachable:
VSM-1# ping vsn all src-module all
ping vsn 106.1.1.1 vlan 54 from module 3 5, seq=0 timeout=1-sec
module(usec) : 3(156) 5(160)
ping vsn 110.1.1.1 vlan 54 from module 3 5, seq=0 timeout=1-sec
module(failed) : 3(VSN ARP not resolved) 5(VSN ARP not resolved)
ping vsn 106.1.1.1 vlan 54 from module 3 5, seq=1 timeout=1-sec
module(usec) : 3(230) 5(151)
ping vsn 110.1.1.1 vlan 54 from module 3 5, seq=1 timeout=1-sec
module(failed) : 3(VSN ARP not resolved) 5(VSN ARP not resolved)
ping vsn 106.1.1.1 vlan 54 from module 3 5, seq=2 timeout=1-sec
module(usec) : 3(239) 5(131)
ping vsn 110.1.1.1 vlan 54 from module 3 5, seq=2 timeout=1-sec
module(failed) : 3(VSN ARP not resolved) 5(VSN ARP not resolved)
ping vsn 106.1.1.1 vlan 54 from module 3 5, seq=3 timeout=1-sec
module(usec) : 3(248) 5(153)
ping vsn 110.1.1.1 vlan 54 from module 3 5, seq=3 timeout=1-sec
module(failed) : 3(VSN ARP not resolved) 5(VSN ARP not resolved)
ping vsn 106.1.1.1 vlan 54 from module 3 5, seq=4 timeout=1-sec
module(usec) : 3(259) 5(126)
ping vsn 110.1.1.1 vlan 54 from module 3 5, seq=4 timeout=1-sec
module(failed) : 3(VSN ARP not resolved) 5(VSN ARP not resolved)
This example shows how to display VSN ping options:
all All VSNs associated to VMs
This example shows how to display VSN ping options for all source modules:
VSM-1# ping vsn all src-module ?
vpath-all All modules having VMs associated to VSNs
This example shows how to set up a ping for all source modules from a specified IP address:
VSM-1# ping vsn ip 10.1.1.60 src-module all
ping vsn 10.1.1.60 vlan 501 from module 4 5 7, seq=0 timeout=1-sec
module(usec) : 4(301) 5(236)
module(failed) : 7(VSN ARP not resolved)
ping vsn 10.1.1.60 vlan 501 from module 4 5 7, seq=1 timeout=1-sec
module(usec) : 4(241) 5(138) 7(270)
ping vsn 10.1.1.60 vlan 501 from module 4 5 7, seq=2 timeout=1-sec
module(usec) : 4(230) 5(155) 7(256)
ping vsn 10.1.1.60 vlan 501 from module 4 5 7, seq=3 timeout=1-sec
module(usec) : 4(250) 5(154) 7(284)
ping vsn 10.1.1.60 vlan 501 from module 4 5 7, seq=4 timeout=1-sec
module(usec) : 4(231) 5(170) 7(193)
This example shows to set up a ping for all vPath source modules for a specified IP address:
VSM-1# ping vsn ip 10.1.1.60 src-module vpath-all
ping vsn 10.1.1.60 vlan 501 from module 4 5, seq=0 timeout=1-sec
module(usec) : 4(223) 5(247)
ping vsn 10.1.1.60 vlan 501 from module 4 5, seq=1 timeout=1-sec
module(usec) : 4(206) 5(167)
ping vsn 10.1.1.60 vlan 501 from module 4 5, seq=2 timeout=1-sec
module(usec) : 4(241) 5(169)
This example shows how to set up a ping for all source modules of a specified IP address with a time-out and a count:
VSM-1# ping vsn ip 10.1.1.60 src-module all timeout 2 count 3
ping vsn 10.1.1.60 vlan 501 from module 4 5 7, seq=0 timeout=2-sec
module(usec) : 4(444) 5(238) 7(394)
ping vsn 10.1.1.60 vlan 501 from module 4 5 7, seq=1 timeout=2-sec
module(usec) : 4(259) 5(154) 7(225)
ping vsn 10.1.1.60 vlan 501 from module 4 5 7, seq=2 timeout=2-sec
module(usec) : 4(227) 5(184) 7(216)
Troubleshooting the Cisco VSG and VEM Interactions When the Cisco VSG is on a VXLAN in a Service-Chain
You can run a series of checks to ensure that interactions between the Cisco VSG and VEM are seamless.
Run the following verifications:
•Check if the Cisco VSG is alive by using the show vservice brief command from the VSM.
This example shows how to display the Cisco VSG configuration:
#License Information
Type In-Use
ID Name Type IP-Address Mode State Module
1 node_10.1.1.40_l3_fclose vsg 10.1.1.40 l3 Unreach 3,9,
3 node_10.1.1.40_501_fclose vsg 10.1.1.40 vxlan Alive 4,9,11,
5 node_10.1.1.45_502_fclose vsg 10.1.1.45 vxlan Unreach 9,
9 VASA1 asa 192.168.200.221 v-53 Alive 3,9,11,
13 VASA-vxlan-222 asa 192.168.200.222 vxlan Alive 4,9,11,
16 EL1 vsg 7.1.1.1 v-501 Unreach 4,
17 EL2 vsg 7.1.1.1 v-502 Unreach 3,
If a specific Cisco VSG is not alive (wherein `Unreach' or `??' is displayed), use the show vservice detail node_ipaddr node ip command for further analysis.
•Check if the Cisco VSG node definition has "adjacency l2 vxlan" in the output. For example:
vservice node node_10.1.1.40_501_fclose type vsg
adjacency l2 vxlan bridge-domain segment2
•Check the port profile attached to the VM. It should be pointing to either the vsevice node VSG directly or to a service path that contains the corresponding Cisco VSG.
•Check the port profile attached to the Cisco VSG data interface, which should be on a bridge domain. It must not have any org/vservice configuration. For example:
port-profile type vethernet segment-5001-nofw
switchport access bridge-domain segment2
Debugging the Traffic Flow Via a Service Chain
When configured, the service-chain functionality enables traffic to flow through the Cisco VSG and the Cisco ASA 1000V clould firewall. The Cisco VSG monitors the data packets and authorizes its flow from the VM to the destination ports. The VM and Cisco ASA 1000V are always in the same broadcast domain, that is, either a VLAN or a Virtual Extensible Local Area Network (VXLAN).
To debug the traffic flow via the service chain, follow these steps:
Step 1 Make sure that the VM's default gateway is set to the ASA 1000V inside interface and is reachable.
Step 2 On the VSM, ensure that the Cisco VSG and ASA 1000V are alive, which ensures that the vPath is able to reach the service nodes.
vsm# show vservice node brief
ID Name Type IP-Address Mode State Module
2 VSG vsg 192.168.10.1 v-140 Alive 3,4
6 ASA asa 3.3.3.1 v-200 Alive 3,4
Step 3 On the VSM, check a connection's status of action (SAct).
vsm# show vservice connection
Proto SrcIP[:Port] SAct DstIP[:Port] DAct Flags Bytes
icmp 192.168.10.15 Pp 192.168.11.15 882
In the SAct value Pp, the uppercase `P' indicates the action that is initiated by the Cisco VSG, while the lowercase `p' indicates the action that is deduced based on the returning traffic from the ASA V1000. If the SAct value is `rr,' it indicates that the traffic is redirecting to either the Cisco VSG or the ASA V1000 but no response is being received.
Step 4 On the VSM, verify that the service node version information (VVer) is `2' so that it works in the service-chain.
vsm# show vservice node detail
Type:asa IPAddr:192.168.10.1 Fail:open Vlan:140
3 Alive 00:50:56:a6:02:a5 2
4 Alive 00:50:56:a6:02:a5 2
Type:asa IPAddr:3.3.3.1 Fail:open Vlan:200
3 Alive 00:50:56:a6:02:6d 2
4 Alive 00:50:56:a6:02:6d 2
Troubleshooting the Service Chain by Excluding the Cisco VSG Node
The service-chain configuration has the Cisco VSG and ASA 1000V nodes in its service path for a given traffic flow. For debugging purposes, the Cisco VSG can be removed temporarily from the node configuration to isolate a problem. Thus, a user can verify the traffic flow with just the ASA 1000V. Later, the Cisco VSG can be added again to restore the original service-chain configuration using the two said service nodes.
VEM/vpath Configured Correctly on a VEthernet Interface for a ServiceChain
You can use the module vem vem-num execute vemcmd show vsn bindings command on the VSM to ensure that the bindings are correctly configured on the VEM for a service chain. Two entries appear for a single LTL—one of each service node must be displayed.
vsm# module vem 3 execute vemcmd show vsn bindings
VSG Services Enabled | VSG Licenses Available 2
ASA Services Enabled | ASA Licenses Available 2
LTL PATH VSN SWBD IP P-TYPE P-ID
60 6 2 3756 10.10.10.202 1 49
60 6 33 3770 172.31.2.11 2 52 >> two service node bindings for LTL
60 and 62
62 6 2 3756 10.10.10.202 1 49
62 6 33 3770 172.31.2.11 2 52
Cisco VSG on a VXLAN is not working
The Cisco VSG node can be configured with a VXLAN in the Layer 2 mode only. Make sure that the adjacency is correctly defined as Layer 2 and the bridge-domain configuration is valid. The show service node brief command can be used to check a service node's state with respect to the vPath.
This example shows a Cisco VSG node configuration for a VXLAN:
vservice node VSG-vxlan-33071 type vsg
adjacency l2 vxlan bridge-domain 33071
This example shows how to display the node status in a VXLAN:
vsm# show vservice node brief
ID Name Type IP-Address Mode State Module
35 VSG-vxlan-33071 vsg 20.20.20.182 vxlan Alive 3,4,5