The Linux kernel has iptables and ip6tables that are used to set up, maintain, and inspect the tables of IPv4 and IPv6 packet filter rules. The iptables and ip6tables consist of many predefined tables. Each table contains predefined chains and can also contain user-defined chains. These chains contain sets of rules and each of these rules specifies the match criteria for a packet. Predefined tables include raw, mangle, filter, and NAT. Predefined chains include INPUT, OUTPUT, FORWARD, PREROUTING, and POSTROUTING.

The Secure Workload agent programs a filter table that contains rules to allow or drop packets. The filter table consists of the predefined chains INPUT, OUTPUT, and FORWARD. Along with these, the agent adds custom TA chains to categorize and manage the policies from the controller. These TA chains contain Secure Workload rules that are derived from the policies along with rules that are generated by the agent. When the agent receives platform-independent rules, it parses and converts them into iptable, ip6table, or ipset rules and inserts these rules into TA defined chains in the filter table. After programming the firewall, the agent monitors the firewall for any rule or policy deviation and if so, reprograms the firewall. It keeps track of the policies that are programmed in the firewall and reports their stats periodically to the controller.

Here is an example to depict this behavior:

A typical policy in a platform-independent network policy message consists of:

source set id: “test-set-1”
destination set id: “test-set-2”
source ports: 20-30
destination ports: 40-50
ip protocol: TCP
action: ALLOW
. . .
set_id: “test-set-1”
     ip_addr: 1.2.0.0
     prefix_length: 16
     address_family: IPv4
set_id: “test-set-2”
     ip_addr: 3.4.0.0
     prefix_length: 16
     address_family: IPv4

Along with other information, the agent processes this policy and converts it into platform-specific ipset and iptables rule:

ipset rule:
Name: ta_f7b05c30ffa338fc063081060bf3
Type: hash:net
Header: family inet hashsize 1024 maxelem 65536
Size in memory: 16784
References: 1
Members:
1.2.0.0/16
Name: ta_1b97bc50b3374829e11a3e020859
Type: hash:net
Header: family inet hashsize 1024 maxelem 65536
Size in memory: 16784
References: 1
Members:
3.4.0.0/16
iptables rule:
TA_INPUT -p tcp -m set --match-set ta_f7b05c30ffa338fc063081060bf3 src -m set --match-
˓→set ta_1b97bc50b3374829e11a3e020859 dst -m multiport --sports 20:30 -m multiport --
˓→dports 40:50 -j ACCEPT

A native component on Windows, the Windows Firewall with Advanced Security, regulates network traffic that is based on the following types of settings:

• Rules that regulate inbound network traffic.

• Rules that regulate outbound network traffic.

• Override rules that is based on the authentication status of the source and destination of the network traffic.

• Rules that apply to IPsec traffic and to Windows services.

The Secure Workload Network Policy is programmed using inbound and outbound firewall rules.

On the Windows platform, the Secure Workload Network Policy is enforced as follows:

1. The platform-independent firewall rules from the Secure Workload Network policy are translated into Windows Firewall rules.

2. The rules are programmed in Windows Firewall.

3. The Windows Firewall enforces the rules.

4. The Windows Firewall and its ruleset are monitored. If a change is detected, the deviation is reported and the Secure Workload Network policy is reset in the Windows Firewall.

Windows Firewall groups the rules based on the network that the host is connected to. These rule groups are called Profiles and there are three such profiles:

• Domain Profile

• Private Profile

• Public Profile

The Secure Workload rules are programmed into all the profiles, but only rules in active profiles are continuously monitored.

The set of rules in the Windows Firewall is not ordered based on precedence. When multiple rules match a packet, the most restrictive of those rules take effect meaning that DENY rules take precedence over ALLOW rules. For more information, see the article on Microsoft TechNet.

1. ALLOW 1.2.3.30 tcp port 80
2. ALLOW 1.2.3.40 udp port 53
3. BLOCK 1.2.3.0/24 ip
4. ALLOW 1.2.0.0/16 ip
5. Catch-all: DROP ingress, ALLOW egress

When a packet headed for the host 1.2.3.30 TCP port 80 reaches the firewall, it matches all the rules, but the most restrictive of them all, Rule number 3, is the one that will be enforced and the packet will be dropped. This behavior is contrary to the expectation that the rules will be evaluated in order, Rule 1 is the rule that is enforced, and that the packet will be allowed.

This difference in behavior is expected in the Windows platform owing to the design of the Windows Firewall described above. This behavior can be observed in mixed-list policies with overlapping rules that have different rule actions.

For example,

1. ALLOW 1.2.3.30 tcp
2. BLOCK 1.2.3.0/24 tcp

Windows Advanced Firewall is considered as a stateful firewall, that is for certain protocols such as TCP, the firewall maintains internal state tracking to detect if a new packet hitting the firewall belongs to a known connection. Packets belonging to a known connection are allowed without the firewall rules having to be examined. A stateful firewall enables bidirectional communication without rules having to be established in the INBOUND and OUTBOUND tables.

For example, consider the following rule for a web server: Accept all TCP connections to port 443

The intention is to accept all TCP connections on port 443 to the server, and allow the server to communicate back to the clients. In this case, only one rule is inserted in the INBOUND table, allowing TCP connections on port 443. No rule is required to be inserted in the OUTBOUND table. Inserting a rule in the OUTBOUND table is implicitly done by the Windows Advanced Firewall.

Note	Stateful tracking applies only to protocols that establish and maintain explicit connections. For other protocols, both INBOUND and OUTBOUND rules must be programmed to enable bidirectional communication.

When enforcement is enabled, a given concrete rule is programmed as stateful when the protocol is TCP (the agent decides, based on the context, whether the rule is to be inserted in the INBOUND table or the OUTBOUND table). For other protocols (including ANY), both INBOUND and OUTBOUND rules are programmed.

Windows Filtering Platform (WFP) is a set of APIs provided by Microsoft to configure filters for processing network traffic. Network traffic processing filters are configured using kernel-level APIs and user level APIs. WFP filters can be configured at various layers, Network Layer, Transport Layer, Application Layer Enforcement (ALE). Secure Workload WFP filters are configured at the ALE layer, similar to Windows firewall rules. Each layer has several sublayers, ordered by weight, from highest to lowest. Within each sublayer, filters are ordered by weight, from highest to lowest. A network packet traverses through all the sublayers. At each sublayer, the network packet traverses through the matching filters that are based on weight, from highest to lowest and returns the action: Permit or Block. After passing through all the sublayers, the packet is processed based on the rule that Block action overrides Permit.

• Avoids Windows Firewall configuration dependencies.

• Overcomes GPO restrictions.

• Ensures ease of migration and policy reversion.

• Allows you to control policy ordering.

• Avoids strict block-first policy order of Windows Firewall.

• Reduces CPU overhead on policy update.

• Creates an efficient 1:1 policy rule filter.

• Ensures a faster single-step update.

When enforcement is configured to use WFP, Secure Workload filters override Windows Firewall rules.

In WFP mode, the agent configures the following WFP objects:

• Provider has a GUID and name, is used for filter management, and does not affect packet filtering

• Sublayer has a GUID, name, and weight. The Secure Workload sublayer is configured with higher weight than the Windows Advanced Firewall sublayer.

• Filter has name, GUID, ID, weight, layer ID, sublayer key, action (PERMIT/ BLOCK), and conditions. WFP filters are configured for Golder rules, Self Rules, and Policy Rules. The agent also configures the port scanning prevention filters. Secure Workload filters are configured with the FWPM_FILTER_FLAG_CLEAR_ACTION_RIGHT flag. This flag ensures that Secure Workload filters are not overridden by Microsoft Firewall rules. For each Secure Workload Network policy rule, one or more WFP filters are configured based on the direction (inbound or outbound) and protocol.

  Note	When port scan probes are sent to the host, the status of the corresponding flows is displayed as PERMITTED:REJECTED on the Policy Analysis page. Note that this is applicable to all agent versions up to 3.10.2.11.

For TCP inbound policy,

id: 14 , TCP Allow 10.195.210.184 Dir=In localport=3389

The WFP filters configured are:

Filter Name:                  Secure Workload Rule 14
------------------------------------------------------
EffectiveWeight:                       18446744073709551589
LayerKey:                              FWPM_LAYER_ALE_AUTH_LISTEN_V4
Action:                                Permit
Local Port:                            3389
Filter Name:                           Secure Workload Rule 14
------------------------------------------------------
EffectiveWeight:                       18446744073709551589
LayerKey:                              FWPM_LAYER_ALE_AUTH_RECV_ACCEPT_V4
Action:                                Permit
RemoteIP:                              10.195.210.184-10.195.210.184

Secure Workload agent configures Secure Workload Default Inbound and Secure Workload Default Outbound filters for inbound and outbound CATCH-ALL policy respectively.

• The agent does not monitor WAF rules or WAF profiles.

• The agent does not monitor firewall states.

• The agent does not require firewall state to be enabled.

• The agent does not conflict with GPO policies.

Agent enforcement in WFP mode supports mixed-list or grey list policies.

Consider the mixed-list (both allow and deny) policy example from the Enforcement Agent section:

1. ALLOW 1.2.3.30 tcp port 80-            wt1000
2. BLOCK 1.2.3.0/24 ip-                   wt998
3. ALLOW 1.2.0.0/16 ip-                   wt997
4. Catch-all: DROP ingress, ALLOW egress - wt996

When a packet headed for the host 1.2.3.30 tcp port 80 reaches the firewall, it matches filter 1 and is allowed. However, a packet that is headed for the host 1.2.3.10 is blocked because of filter 2. A packet that is headed for host 1.2.2.10 is allowed by filter 3.

Secure Workload WFP filters are configured at the ALE layer. Network traffic is filtered for socket connect(), listen(), and accept() operations. Network packets related to a L4 connection are not filtered after the connection is established.

You can view the configured Secure Workload WFP filters using c:\program files\tetration\tetenf.exe. Supported options:

• With administrative privileges, run cmd.exe.

• Run c:\program files\tetration\tetenf.exe -l -f <-verbose> <-output=outfile.txt>.

OR

• With administrative privileges, run cmd.exe.

• Run netsh wfp show filters.

• View the filters.xml file for configured Secure Workload filters.

You can delete the configured Secure Workload WFP filters using c:\program files\tetration\tetenf.exe. To avoid accidental deletions of filters, when you run the delete command, specify the token in <yyyymm> format, where yyyy is the current year and mm is the current month in the numerical form. For example, if today’s date is 01/21/2021, then the token is -token=202101

Supported options are:

• With administrative privileges, run cmd.exe.

• To delete all configured Secure Workload filters, run c:\program files\tetration\tetenf.exe -d -f -all - token=<yyyymm>

• To delete all configured Secure Workload WFP objects, run c:\program files\tetration\tetenf.exe -d -all -token=<yyyymm>

• To delete a Secure Workload WFP filter by name, run c:\program files\tetration\tetenf.exe -d -name=<WFP filter name> -token=<yyyymm>

• The Preserve Rules setting in Agent Config Profile has no effect when you set Enforcement Mode to WFP.

• Windows 2008 R2 does not support Windows OS based filtering policies.

• Network policy can be configured with a single user name whereas MS Firewall UI supports multiple users.

• While using the Windows OS-based policies, a consumer/ provider scope or filter should only contain Windows agents. Otherwise, non-Windows OSs (Linux, AIX) skip the policy and report a sync error in Enforcement Status.

• Avoid creating Windows OS filters with loose filtering criteria. Such criteria may open unwanted network ports.

• If OS filters are configured for consumer, then the policies are applicable only to consumer, similarly if it is configured for provider then it is applicable only to provider.

• Due to limited or no knowledge of the process context, user context or service context of the network flows, there will be discrepancy in the policy analysis if the policies have Windows OS-based filters.

If you use Windows OS-based filtering attributes, the following topics provide you with verification and troubleshooting information.

Cisco TAC can use this information as needed to troubleshoot such policies.

The IPFilter package on AIX is used to provide firewall services and is available on AIX as a kernel expansion pack.

It loads as a kernel extension module, /usr/lib/drivers/ipf. It includes ipf, ippool, ipfstat, ipmon, ipfs, and ipnat utilities that are used to program ipfilter rules and each of these rules specifies the match criteria for a packet. For more information, see the IPFilter pages in the AIX manual.

When enforcement is enabled, the agent uses IPFilter to program the IPv4 filter table that contains rules for allowing or dropping of IPv4 packets. The agent groups these rules to categorize and manage the policies using the controller. These rules include Secure Workload rules that are derived from the policies and rules that are generated by the agent.

When an agent receives platform-independent rules, it parses and converts them into ipfilter or ippool rules and inserts these rules into the filter table. After programming the firewall, the enforcement agent monitors the firewall for any rule or policy deviation and if so, reprograms the firewall. The agent keeps track of the policies that are programmed in the firewall and reports their status periodically to the controller.

A typical policy in a platform-independent network policy message consists of:

      source set id: "test-set-1"
      destination set id: "test-set-2"
      source ports: 20-30
      destination ports: 40-50
      ip protocol: TCP
      action: ALLOW
      ...
      set_id: "test-set-1"
         ip_addr: 1.2.0.0
         prefix_length: 16
         address_family: IPv4
      set_id: "test-set-2"
         ip_addr: 5.6.0.0
         prefix_length: 16
         address_family: IPv4

Along with other information, the agent processes the policy and converts it into platform-specific ippool and ipfilter rule:

    table role = ipf type = tree number = 51400
    { 1.2.0.0/16; };

    table role = ipf type = tree number = 75966
    { 5.6.0.0/16; };

pass in quick proto tcp from pool/51400 port 20:30 to pool/75966 port 40:50 flags S/SA group TA_INPUT
pass out quick proto tcp from pool/75966 port 40:50 to pool/51400 port 20:30 flags A/A group TA_OUTPUT

IPv6 enforcement is not supported.

1. Root scope owners have access only to create a Configuration Profile and Configuration Intent specification.

2. As a Root Scope owner, you can create configuration profiles that are associated with only owned scopes and impose these configuration profiles on agents.

   Note	Under the Agent Configuration Profile, you can now view the number of intents using the configuration profile before you edit the profile.

   

3. Site administrators have access to all the components in the Agent Configuration page that includes specifying interface configuration intents, remote virtual routing and forwarding configurations.

Network flow information is the summarization of all packets flowing through the system. There are two modes of capturing flow information: Detailed and Conversation. By default, the Conversation mode is used to capture the flow information. The captured flows are exported to a collector and the exported information includes:

• Flow identifier: Uniquely identify the network flow. It includes the general information such as: IP protocol, source and destination IP, and layer 4 ports.

• IP Information: Contains information that is seen in the IP header, such as: TTL, IP flags, Packet ID, IP options, and Fragmentation flags.

• TCP Information: Contains information that is seen in the TCP header, such as: sequence number, Ack number, TCP options, Rcvd windows size.

• Flow Information: Statistics of the flow (such as total packets, total bytes, TCP flags statistics, packet length statistics, and socket statistics), interface index from which the flow was observed, start time and end time of flow.

• In a K8s environment, the agent captures network flows from pods and hosts, and then correlates the flows and reports as related flows. This is qualified with the following CNIs:

  • Calico

  • Flannel

  • Weave

  • AKS/GKE/AWS VPC CNI

  • Openshift CNI

  • Cilium CNI

    Note	Network flows are captured from pods and hosts, however, the correlation of flows is not possible when Cilium CNI is used.

In Conversation mode, the agent exports only TCP flows that are bidirectional in nature along with other connectionless flows. Conversation mode is supported for Windows, AIX, and Linux platforms. For more information on Conversation mode, see Conversation Mode.

Note	In K8s environment, correlation of Pod or Host flows are not done in Conversation mode. In either of the modes, agents do not export the following flows: ARP/RARP conversations Agent’s flows to collectors

Machine info describes all the processes running on the host. In addition, it contains network information that is associated with the processes and the command used to launch the processes. Machine info is exported every minute and includes the following information:

• Process ID

• User ID: owner of the process

• Parent Process ID

• Command string used to launch the process

• Socket information: protocol (such as UDP or TCP), address type: IPv4 or IPv6, source and destination IP, source and destination port, TCP state, process’s start and end time, path to process binary

• Forensic information: for more information, see the section Compatibility.

Agent keeps track of various statistics, including system’s statistics and its own, such as:

• Agent’s start time and uptime

• Agent’s run time in user mode and kernel mode

• Number of packets received and dropped

• Number of successful and failed SSL connections

• Total flow packets and bytes

• Total exported flows and packets to collectors

• Agent’s memory and CPU usage

  {
    "AgentType":"ENFORCER",
    "Bios":"72EF1142-03A2-03BC-C2F8-F600567BA320",
    "CurrentVersion":"3.5.1.1.mrpm.build.win64-enforcer",
    "DesiredVersion":"",
    "HostName":"win2k12-production-db",
    "IP":"172.26.231.193",
    "Platform":"MSServer2012R2Standard"
  }

  {
    "AgentType":"SENSOR",
    "Bios":"72EF1142-03A2-03BC-C2F8-F600567BA320",
    "CurrentVersion":"3.5.1.1.mrpm.build.win64-sensor",
    "DesiredVersion":"",
    "HostName":"win2k12-production-db",
    "IP":"172.26.231.193",
    "Platform":"MSServer2012R2Standard"
  }

rpm -Uvh tet-sensor-1.101.2-1.el6-dev.x86_64.rpm

   error: cannot create transaction lock on /var/lib/rpm/.rpm.lock (Permission denied).

A: If you do not have the right privileges to install the agents, either switch to root or use sudo to install the agents.

Q: What happens when you run “sudo rpm -Uvh tet-sensor-1.0.0-121.1b1bb546.el6-dev.x86_64.rpm” and encounter the following error:

   Preparing...              ########################################### [100%]
   which: no lsb_release in (/sbin:/bin:/usr/sbin:/usr/bin:/usr/X11R6/bin)
   error: %pre(tet-sensor-site-1.0.0-121.1b1bb546.x86_64) scriptlet failed, exit status 1
   error:   install: %pre scriptlet failed (2), skipping tet-sensor-site-1.0.0-121.1b1bb546

A: The system does not satisfy the requirements to install the agents. In this particular case, lsb_release tool is not installed.

For more information, see the Software Agents Deployment Label section and install the required dependencies.

Q: What happens whn you run “sudo rpm -Uvh tet-sensor-1.0.0-121.1b1bb546.el6-dev.x86_64.rpm” and encounter the following error:

  Unsupported OS openSUSE project
  error: %pre(tet-sensor-1.101.1-1.x86_64) scriptlet failed, exit status 1
  error: tet-sensor-1.101.1-1.x86_64: install failed
  warning: %post(tet-sensor-site-1.101.1-1.x86_64) scriptlet failed, exit status 1

A: Your OS is not supported to run software agents (in this particular case, “openSUSE project” is a non- supported platform).

For more information, see the Software Agents Deployment Label section.

Q: After I have installed all the dependencies and run installation with proper privileges with no errors. How do I know the agents installation was successful?

A: After you have installed the agents, to verify if the installation, run the following command:

$ ps -ef | grep -e csw-agent -e tet-
root     14158     1  0 Apr03 ?        00:00:00 csw-agent
root     14160 14158  0 Apr03 ?        00:00:00 csw-agent watch_files
root     14161 14158  0 Apr03 ?        00:00:03 csw-agent check_conf
root     14162 14158  0 Apr03 ?        00:01:03 tet-sensor -f conf/.sensor_config
root     14163 14158  0 Apr03 ?        00:02:38 tet-main --sensoridfile=./sensor_id
root     14164 14158  0 Apr03 ?        00:00:22 tet-enforcer --logtostderr
tet-sen+ 14173 14164  0 Apr03 ?        00:00:21 tet-enforcer --logtostderr
tet-sen+ 14192 14162  0 Apr03 ?        00:07:23 tet-sensor -f conf/.sensor_config

You must see three entries of csw-agent and at least two entries of tet-sensor. If the services are not running, ensure that the following directories are available, else the installation has failed.

• /usr/local/tet for most Linux distributions

• /opt/cisco/tetration for AIX, Ubuntu

• /opt/cisco/secure-workload for Solaris, Debian

Q: When I run the PowerShell agent installer script, I get one of the following errors:

1. The underlying connection was closed: An unexpected error occurred on a receive.

2. The client and server cannot communicate, because they do not possess a common algorithm

[Net.ServicePointManager]::SecurityProtocol

[Net.ServicePointManager]::SecurityProtocol = [System.Net.SecurityProtocolType]’Ssl3,Tls,Tls11,Tls12’

This installation package could not be opened. Verify that the package exists and that you can access it, or contact the application vendor to verify that this is a valid Windows Installer package.

msiexec /i “TetrationAgentInstaller.msi” /l*v “msi_install.log” /norestart

Q: I have observed that Windows Sensor software fails to upgrade if underlying NIC is Nutanix VirtIO Network Driver.

A: There is an incompatibility issue between Npcap 0.9990 and Nutanix VirtIO Network Driver version earlier than 1.1.3 and Receive Segment Coalescing is enabled.

The resolution for this is to upgrade Nutanix VirtIO Network Driver to version 1.1.3 or later.

Q: I have installed windows sensor. The sensor doesnt seem to register and the sensor_id file contains the following: uuid-invalid-platform

set PATH=%PATH%;C:\Windows\System32\

Q: I am not receiving the network flows from Kubernetes Pods on Windows Nodes.

A: To verify if the required sessions are running to capture the flows from Kubernetes pods on Windows nodes, perform the following:

1. Run cmd.exe with administrative privileges.

2. Run the following command: logman query -ets

   Ensure that the following sessions are running:

   • CSW_MonNet: Captures network flows

   • CSW_MonHCS: Monitors creation of pods

   • CSW_MonNat: Monitors NATed flows

If the installer script fails during Kubernetes Daemonset Installation, there are a large number of possible reasons.

Q: Is the Docker Registry serving images reachable from nodes ?

A: Debug Direct or HTTPS Proxy issues with the cluster pulling images from Cisco Secure Workload cluster

Q: Is the container runtime complaining about SSL/TLS insecure errors ?

A: Verify that the Secure Workload HTTPS CA certificates are installed on all Kubernetes nodes in the appropriate location for the container runtime.

Q: Docker Registry authentication and authorization of image downloads failures ?

A: From each node, attempt to manually docker pull the images from the registry urls in the Daemonset spec using the Docker pull secrets from the secret created by the Helm Chart. If the manually image pull also fails, need to pull logs from the Secure Workload Cluster registryauth service to debug the issue further.

Q: Is the Kubernetes cluster hosted inside the Secure Workload appliance heathy ?

A: Check the service status page for the cluster to ensure all related services are healthy. Run the dstool snapshot from the explore page and retrieve the logs generated.

Q: Are the Docker Image Builder daemons running ?

A: Verify from the dstool logs that the build daemons are running.

Q: Are the jobs that build Docker images failing ?

A: Verify from the dstool logs that the images have not been built. Docker build pod logs can be used to debug errors during the buildkit builds. Enforcement Coordinator logs can also be used to debug the build failures further.

Q: Are the jobs creating Helm Charts failing ?

A: Verify from the dstool logs that the Helm Charts have not been built. Enforcement Coordinator logs will contain the output of the helm build jobs and can be used to debug the exact reason for the Helm Chart build job failures.

Q: Installation bash script was corrupt ?

A: Attempt to download the installation bash script again. The bash script contains binary data appended to it. If the bash script is edited in any way with a text editor or saved as a text file, special characters in the binary data may be mangled/modified by the text editor.

Q: Kubernetes cluster configuration – too many variants and flavors, we support classic K8s.

A: If the customer is running a variant of Kubernetes, there can be many failure modes at different stages of the deployment. Classify the failure stage - kubectl command run failure, helm command run failures, pod image download failures, pod privileged mode options rejected, pod image trust content signature failures, pod image security scan failures, pod binaries fail to run (architecture mismatch), pods run but the Secure Workload services fail to start, Secure Workload services start but have runtime errors due to unusual operating environment.

Q: Are the Kubernetes RBAC credentials failing ?

A: In order to run privileged daemonsets, we need admin privileges to the K8s cluster. Verify the the kubectl config file has its default context pointing towards the target cluster and admin-equivalent user for that cluster.

Q: Busybox image available or downloadable from all cluster nodes ?

A: Fix the connectivity issues and manually test that the busybox image can be downloaded. The exact version of busybox that is used in the pod spec must be available (pre-seeded) or downloadable on all cluster nodes.

Q: API Server and etcd errors or a general timeout during the install ?

A: Due to the instantiation of daemonset pods on all nodes in the Kubernetes cluster, the CPU/Disk/Network load on the cluster can spike suddenly. This is highly dependent on the customer specific installation details. Due to the overload, the installation process (images pulled on all nodes and written to disks) might take too long or overload the Kubernetes API server or the Secure Workload Docker Registry endpoint or, if configured, the proxy server temporarily. After a brief wait for image pulls on all nodes to complete and a reduction in CPU/Disk/Network load on the Kubernetes cluster nodes, retry the installation script again. API Server and etcd errors from the Kubernetes control plane indicate that the Kubernetes control plane nodes may be underprovisioned or affected by the sudden spike in activity.

Q: Secure Workload Agent experiencing runtime issues with its operations ?

A: Refer to the Linux Agent troubleshooting section if the pods are correctly deployed and the agent has started running but is experiencing runtime issues. The troubleshooting steps are the same once the Kubernetes deployment has successfully installed and started the pods.

Agent has stopped checking to the cluster services. This can happen due to several reasons:

• The host might have been down

• The network connectivity has been broken or blocked by firewall rules

• The agent service has been stopped

Agent upgrade has failed. This can be triggered by few cases such as:

• Not finding the package when the check in script attempts to download it - the upgrade package cannot be unpacked or the installer from the package cannot be verified.

• Installation process failing from an OS issue or dependency.

The current agent type mismatches desired agent type and the convert attempt has timed out. This issue can be caused by a communication issue when an agent does check_in to download the package, or wss service failed to push convert_commnad to the agent.

The ability to convert the agent from one type (such as deep visibility) to another type (such as enforcement) is not available by all agents. If an agent that is not capable to do the conversion is required to convert, the anomaly will be reported.

The current policy (NPC) version last reported by the agent does not match the current version generated on the cluster. This can be caused by a communications error between the agent and the cluster, the agent failing to enforce the policy with the local firewall, or the agent enforcement service not running.

If the Secure Workload Agent cannot open the pcap device to capture flows, you see errors in the Agent logs. A successfully opened Pcap device will report as follows:

I0609 15:25:52.354 24248 Started capture thread for device <device_name>
I0609 15:25:52.354 71912 Opening device {<device_id>}

I0610 03:24:22.354 16614 Opening device <device_name>
[2020/06/10 03:24:23:3524] NOTICE: lws_client_connect_2: <device_id>: address 172.29.
˓→136.139

Connectivity between the agent and the cluster is externally blocked therefore preventing flows and other system information from being delivered. This is caused by one or more configuration issues with network firewalls, SSL decryption services, or third party security agents on the host.

• If there are known firewalls or SSL decryption security devices between the agent and the cluster, make sure that communications to all Secure Workload collector and VIPs IP addresses are being permitted. For on-prem clusters, the list of collectors will be listed under Troubleshoot > Virtual Machines in the navigation bar at the left side of the Secure Workload web interface. Look for collectorDatamover-*. For Secure Workload cloud, all the IP addresses that need to be permitted will be listed in your Portal.

• To help identify if there is SSL decryption, openssl s_client can be used to make a connection and display the returned certificate. Any additional certificate added to the chain will be rejected by the Agent’s local CA. SSL Troubleshooting

  Note

  Typically, the service to update "flow export anomaly status” runs every 5 minutes. This duration may vary because the agents' status updates are being executed in small batches of 5000. Thus, when there are fewer agents in the cluster, the updates are faster. When there are larger number of agents, the updates can take a maximum of 70 minutes. After the initial sorting of agents records in the database, the cluster and agents become stable, and eventually, the update interval becomes lesser and more consistent.

In version 3.10.1.1, Secure Workload agent uses Extended Berkeley Packet Filter (eBPF) on the following distributions:

• Ubuntu 22/24

• Debian 11/12

• EL9 family

If the Secure Workload agent cannot use eBPF for flow capture, you see errors in the Agent logs.

A successful flow capture is reported for each network device as shown below:

I0604 21:29:20.357 833292 Opening device <device_name>
I0604 21:29:20.394 833292 Using eBPF TC to capture packets for <device_name>

• Run the command winver.exe

• Compare this release to what is listed here: Supported Platforms and Requirements

• Run cat /etc/os-release

• Compare this release to what is listed here: Supported Platforms and Requirements

• Run the command uname -a

• Note: The major and minor versions are reversed

  p7-ops2> # uname -a
  AIX p7-ops2 1 7 00F8AF944C00

• In this example, the first number after the host name is the minor and the second number is the major version, so AIX version 7.1. Compare this release to what is listed here: Supported Platforms and Requirements

• Npcap will sometimes not uninstall correctly if a process is currently using the Npcap libraries. To check for this run the following command:

  PS C:\Program Files\Npcap> .\NPFInstall.exe -check_dll
  WindowsSensor.exe, Wireshark.exe, dumpcap.exe
  

If you see processes listed, they must be stopped before the Npcap upgrade can continue. If no processes are using Npcap the above command will simply show <NULL>

• Check CA certificates installed on the system: Certificate Issues for NPCAP installer

• Check Windows Installer issues: Windows Installer Issues

• Verify no other user on the system is making changes to the network interfaces. This can cause a COM lock preventing NDIS driver binding.

Applicable to Windows 2016 Only

If you have a 3rd party LWF (Light Weight Filter) driver (e.g. netmon) or a teaming adapter is configured in your setup, and NPCAP is installed during agent deployment, you might experience

RDP is reconnected

NetBios service is restarted

Similar network connectivity issues

This is due to a BUG in Windows 2016 OS

Teaming NIC functionality is based on underneath Physical NICs (Intel, Broadcom, Realtek, MS virtual adapter etc) and Teaming driver configuration (switch based, loadbalancing or failover, algorithm to distribute the packets across multiple NICs).

Some NPCAP versions have compatibility issues with Teaming NICs, especially during binding to the underneath Teaming NICs.

NIC type : Intel(R) 82574L Gigabit Network Connection
Teaming Mode : Switch Independent
Load Balancing Mode: Address Hash
OS : Windows 2012 , Windows 2012 R2, Windows 2016, Windows 2019
NPCAP version: 1.55

Note	Windows 2008R2 does not support Microsoft supported NIC teaming.

The TetSensor service occasionally does not capture the network flows on cloned VMs when NPCAP service is running. This can happen when the agent is installed without the nostart flag using MSI installer or without goldenImage flag using PowerShell Installer on a VM template or golden image.

In this case, Secure Workload agent services start running on the VM template. NPCAP is installed and bound to the Network stack on the VM template. When a new VM is cloned from the VM template, NPCAP is not properly bound to the Network stack on the new cloned VM. As a result, NPCAP fails to capture the network flows.

It is observed that Network performance will be affected when Windows TetSensor service is running. Windows Tet- Sensor service (tetsen.exe) captures the network flows using NPCAP. NPCAP implementation to capture the network flows and the network flows to the tetsen.exe affects the network performance.

Compare the Network Performance after installing tetsensor, Client : Windows 2016

NPCAP 1.55

TetSensor Config : Conversation Mode with Enforcement mode WFP

Server : Windows 2016

NPCAP 1.55

TetSensor Config : Conversation Mode with Enforcement mode WFP

Run cmd : iperf3.exe -c <server_ip> -t 40

Network Performance with NPCAP 0.9990

Compare the Network Performance after installing tetsensor, Client : Windows 2016

NPCAP 0.9990

TetSensor Config : Conversation Mode with Enforcement mode WFP

Server : Windows 2016

NPCAP 0.9990

TetSensor Config : Conversation Mode with Enforcement mode WFP

Run cmd : iperf3.exe -c <server_ip> -t 40 .. table:: Network Performance with NPCAP 0.9990

class longtable

Note	Performance may vary based on Windows NPCAP version installed, Windows OS, and network Configuration.

OS may experience unknown performance or stability issues if the installed NPCAP version or NPCAP configuration is not supported by the Secure Workload Software.

Supported NPCAP Version: : 0.991 and 1.55

• WSS: Persistent socket connection over port 443 to the cluster

• Check in: A HTTPS call to the cluster every 15-20 minutes to check for current configurations, check for updates and to update the active state of the agent to the cluster. This also reports upgrade failures.

• Flow export: Persistent SSL connection over port 443 (TaaS) or 5640 (On-premise) to send flow metadata to the cluster

• Enforcement: Persistent SSL connection over port 443 (Taas) or 5660 (On-premise) to pull in enforcement policies and report enforcement state

The Teration UI will report either an inactive agent (no longer checking-in), no exported flows (on Agent Workload Profile page under Stats), or failed enforcement. Depending on the error, you can check different logs on the workload to help determine the source of the issue.

Tue 06/09/2020 17:25:25.08 check_conf_update: "curl did not return 200 code, it's 304,
˓→ exiting"
Tue 06/09/2020 17:25:25.08 check_conf_update: "error code after running check_conf_
˓→update = 2"

cfgserver.go:261] config server: StateConnected, wss://<config_server_ip>:443/wss/
˓→<sensor_id>/forensic, proxy:

collector.go:258] next collector: StateConnected, ssl://<collector_ip>>:5640

ssl_client.cpp:341] Successfully connected to EFE server

Secure Workload agents use TLS to secure the TCP connections to the Secure Workload Cloud SaaS servers. These connections are broken down into three distinctive channels.

• Agent -> Cisco Secure Workload SaaS control channel over port TCP/443 (TLS) (sensorVIP)

  This is a low volume control channel that allows the agent to register with Secure Workload and also handles configuration pushes and software upgrade notifications.

• Agent -> Cisco Secure Workload SaaS flow data over TCP/443 (TLS) (collector)

  Flow data is the extracted flow metadata information; the data will be sent to 1 set of 16 IP addresses at a time. The second set of IP addresses is for standby. This is around 1 – 5% of actual server traffic.

• Agent ->Cisco Secure Workload SaaS enforcement data over TCP/443 (TLS) (efe)

  The enforcement data channel is a low volume control channel that is used to push the policies to the sensors and also gather enforcement statistics.

The sensor validates the TLS certificate from the Secure Workload Cloud control, data and enforcement servers against a local CA that is installed with the agent. No other CAs are used, so any other certificate sent to the agent will result in a verification failure and the agent will not connect. This will result in the agent not registering, checking-in, sending flows or receiving enforcement policies.

As discussed in the previous section, it is important to configure your explicit or transparent web proxy to bypass SSL/TLS decryption for agent communications. If the bypass is not configured, these proxies might attempt to decrypt

SSL/TLS traffic by sending its own certificate to the agent. Because the agent only uses its local CA to validate the certificate, these proxy certificates will cause connection failures.

Symptoms include agent failing to register to the cluster, agent not checking-in, agent not sending flows, and/or agent not receiving enforcement configuration (if enforcement is enabled).

Note	Troubleshooting steps below are assuming default installation paths were used. Windows: C:Program FilesCisco Tetration Linux: /usr/local/tet. If you installed your agents in a different location, substitute that location in the instructions.

SSL/TLS Connection issues are reported in the agent logs. To verify if there are SSL errors in the logs, run the following commands for the associated issue being observed.

grep "NSS error" /usr/local/tet/log/check_conf_update.log

get-content "C:\Program Files\Cisco Tetration\logs\TetUpdate.exe.log" | select-
˓→string -pattern "curl failed SSL peer certificate"

grep "SSL connect error" /usr/local/tet/log/tet-sensor.log

get-content "C:\Program Files\Cisco Tetration\logs\WindowsSensor*.log" | select-
˓→string -pattern "Certificate verification error"

grep "Unable to validate the signing cert" /usr/local/tet/log/tet-enforcer.log

get-content "C:\Program Files\Cisco Tetration\logs\WindowsSensor*.log" | select-
˓→string -pattern "Handshake failed"

curl -v -x http://<proxy_address>:<port> https://<sensorVIP>:443

cd "C:\Program Files\Cisco Tetration"
.\curl.exe -kv -x http://<proxy_address>:<port> https://<sensorVIP>:443

openssl s_client -connect <sensorVIP from TaaS Portal>:443 -CAfile /usr/local/tet/
˓→cert/ca.cert

cd C:\Program Files\Cisco Tetration
.\openssl.exe s_client -connect <sensorVIP from TaaS Portal>:443 -CAfile cert\ca.cert

Verify return code: 0 (ok)

0 s:/C=US/ST=CA/L=San Jose/O=Cisco Systems, Inc./OU=Tetration, Insieme BU/CN=129.146.
˓→155.109
i:/C=US/ST=CA/L=San Jose/O=Cisco Systems, Inc./OU=Tetration Analytics/CN=Customer CA

[Net.ServicePointManager]::SecurityProtocol

Ssl3, Tls

[Net.ServicePointManager]::SecurityProtocol = [System.Net.SecurityProtocolType]'Ssl3,
˓→Tls,Tls11,Tls12'