Issue
Mac users experience intermittent failures when attempting CLI authentication to internal applications while connected to Cisco Secure Client VPN. The failures present as "host not found" errors during CLI authentication and when using commands such as curl. However, DNS resolution commands like nslookup and dig succeed. The issue occurs randomly and can be temporarily resolved by reconnecting the VPN, after which connectivity works for a short period before the problem reoccurs. Split-tunnel VPN is in use, and Cisco Umbrella is active. The problem does not occur when using Palo Alto GlobalProtect VPN.
- Error message: "host not found" on CLI authentication and
curl commands. - Error message: VPN client unable to successfully verify IP forwarding table modifications. Domain name server (DNS) resolution issue while connecting private resources
nslookup and dig commands succeed- Intermittent connectivity after reconnecting VPN
- Split-tunnel remote access VPN and Umbrella module enabled
- Issue reproducible only with Cisco Secure Client VPN on MacOS devices
Environment
- Product: Cisco Secure Client (CSC) with multiple modules
- Platform: Corporate Mac devices
- VPN Profile Configuration: Remote access VPN Profile - Bypass Secure access - Split-tunnel Mode and DNS mode selected as "Default DNS"
- DNS Filtering: Cisco Umbrella enabled
- Module Versions:
- Cloud Management v1.0.0.23
- AnyConnect VPN v5.1.13.177
- Umbrella v5.1.13.177
- DART v5.1.13.177
- Secure Firewall Posture v5.1.13.177
- Network Visibility Module v5.1.13.177
- Diagnostic Data: DART bundles collected for analysis
- Observed only on Cisco Secure Client VPN (not on Palo Alto GlobalProtect)
Resolution
- During debugging of the VPN profile (
naic.org) split-tunnel configuration and the AnyConnect VPN routing table on the client side, this behavior was observed:- Working scenario - When performing a
nslookup for the Vault non-prod local domains, DNS requests handled by the DNS servers configured within the VPN profile resolved correctly to 10.x addresses. Correspondingly, the routing table was updated with the resolved IP (for example, 10.59.130.193) under non-secured routes. - Non-working Scenario - However, when the same DNS requests were handled by the macOS system’s local DNS (192.168.x.x) configured on untun4 and en0 adapter instead of the DNS servers defined in the VPN profile, this behavior has been observed clearly from the packet capture while issue noticed.
- the private domains resolved to IP range of 34.x.x.x , which led to the connectivity issue. Wireshark capture helped to identify this underlying root cause of the issue.
- From a design and configuration standpoint, with a split-tunnel VPN profile setup, it is recommended to use split DNS rather than relying on local system DNS/default DNS.
- Additionally, the
us-east-eks-amazonaws.com entry has been added to ensure traffic for this EKS cluster is correctly steered through the remote tunnel interface. - Also discussed was that the RAVPN interface must take precedence over the Umbrella module and should not conflict with the
OrgInfo.json file containing the Umbrella Organization ID. - During our troubleshooting process, we have done fresh install of CSC client without Umbrella module, with that scenario we were not able to see the issue. I was able to review from Umbrella perspective as well, root domain naic.org configured in the internal domains list to bypass Umbrella which means that local domain resolutions is forwarded to macOS configured system DNS not intercepted by Umbrella DNS module at kernel level loopback interface.
This align with issue resolve when there is no Umbrella module in place. With proper VPN profile configuration including correct domains in the traffic steering rule and split DNS config, we shouldn’t be seeing the issue even with Umbrella model is ON.
The user confirmed that issue has been resolved after modifying the DNS mode to Split tunnel and edited the VPN profile configuration.
Cause
VPN profile - Bypass Secure Access - DNS mode supposed to set to Split Tunnel (most commonly seen options from a use case scenarios) and include all the private/internal application domains under split DNS configuration to resolve the issue.
Related Content
Feedback