Troubleshoot Infrastructure Client DNS on ASR 5000/5500
Available Languages
Contents
Introduction
This document describes how to troubleshoot issues related to Domain Name Service (DNS) infrastructure. This includes the various command line interfaces (CLI), DNS concepts, and additional data that possibly needs to be collected. Example output is provided as needed in order to better explain certain points.
Infrastructure DNS on Cisco Aggregation Services Router (ASR) 5000 / 5500 is responsible for the resolution of Fully Qualified Domain Names (FQDN) within the context where it is configured. This is usually in order to support various aspects of call control in the ingress context. Examples of this include:
- The resolution of any Diameter endpoint peers that are in FQDN format instead of IP address
- The resolution of Proxy Call Session Control Function (P-CSCF) FQDNs returned in Diameter S6b responses which are needed by the User Equipment (UE) in order to register with the IP Multimedia System (IIMS) core
- High Rate Packet Data Serving Gateway (HSGW) needs to make DNS Named Application Naming Authority Pointer (NAPTR) queries in order to get a list of Packet Data Network Gateways (PGW) to connect to (either new or handoff) and then make DNS AAAA queries in order to retrieve the IP address of the PGW Local Mobility Anchor (LMA) address in order to connect the call.
- Mobility Management Entity (MME) needs to make DNS NAPTR queries in order to get a list of Serving Gateways (SGW) / PGW pairs to which to connect with. This includes the creation of DNS AAAA/A queries in order to retrieve the IPs of those nodes.
Configuration
DNS is implemented as a client application very simply in the context where it is needed. Here is an example of such an implementation:
context ingress
ip name-servers 2001:5555:202:fffe:a0:e:0:3 2001:5555:203:fffe:c0:e:0:3
dns-client HSGW-DNS
bind address 2001:5555:200:1011:342:281::
resolver retransmission-interval 2
resolver number-of-retries 3
exit
exit
The minimum required in order to be configured is a service/bind address and a primary (and optionally secondary) DNS server address(es).
UDP vs. TCP
What makes DNS potentially more complicated is the transport layer. While DNS queries are normally UDP-based, NAPTR queries, based on the request, can end up TCP-based. The reason is that there is restriction in the size of responses with UDP which requires TCP in order to transmit the responses over multiple packets. The packet flow involves an initial request and then a response from the DNS server. This causes a re-request over TCP via a 0 payload response with the TrunCated (TC) flag set. This means that the client should retry as TCP/IP per RFC 5966. A typical TCP 3-way exchange ensues, followed by the request a second time. When are the sizes large enough to require this? For example, in the case of an HSGW, if the request is a handoff, UDP should be sufficient since there should only be one or a few (if multiple services returned) PGW FQDNs in order for the HSGW to connect. For new calls though, the list of all possible PGWs network-wide that can be returned might be long enough to require the TCP approach.
Here is an example response (from Wireshark) that requests TCP:
Frame 85: 143 bytes on wire (1144 bits), 143 bytes captured (1144 bits)
Ethernet II, Src: JuniperN_20:e7:f0 (64:87:88:20:e7:f0), Dst:
StarentN_02:b1:9d (00:05:47:02:b1:9d)
802.1Q Virtual LAN, PRI: 0, CFI: 0, ID: 2010
Internet Protocol Version 6, Src: 2001:5555:202:fffe:a0:e:0:3
(2001:5555:202:fffe:a0:e:0:3), Dst: 2001:5555:200:1011:304:281::
(2001:5555:200:1011:304:281::) User Datagram Protocol, Src Port: domain (53),
Dst Port: 35049 (35049)
Domain Name System (response)
[Request In: 81]
[Time: 0.088530870 seconds]
Transaction ID: 0x3b2b
Flags: 0x8780 Standard query response, No error
1... .... .... .... = Response: Message is a response
.000 0... .... .... = Opcode: Standard query (0)
.... .1.. .... .... = Authoritative: Server is an authority for domain
.... ..1. .... .... = Truncated: Message is truncated
.... ...1 .... .... = Recursion desired: Do query recursively
.... .... 1... .... = Recursion available: Server can do recursive queries
.... .... .0.. .... = Z: reserved (0)
.... .... ..0. .... = Answer authenticated: Answer/authority portion
was not authenticated by the server
.... .... ...0 .... = Non-authenticated data: Unacceptable
.... .... .... 0000 = Reply code: No error (0)
Questions: 1
Answer RRs: 0
Authority RRs: 0
Additional RRs: 1
Queries
APN1.apn.epc.mnc420.mcc300.3gppnetwork.org: type NAPTR, class IN
Name: APN1.apn.epc.mnc420.mcc300.3gppnetwork.org
Type: NAPTR (Naming authority pointer)
Class: IN (0x0001)
Additional records
Troubleshooting Commands
show dns-client statistics client <DNS Client Name>
This is the main command in order to troubleshoot DNS issues. Here are some highlights in order to run this command:
- It must be run in the context where the client is defined.
- Run it multiple times and note increases in appropriate statistics such as timeouts.
- Usage statistics count the actual call success/failures that are a result of being able / not being able to resolve DNS.
- DNS Resolver statistic failures count the number of timeouts, and other failures like connection refused. Timeouts can be due to TCP connection establishment issues.
- Configured threshold for DNS failures is captured with an SNMP trap (and alarm) ThreshDNSLookupFailure based on these statisticss. Example: threshold dns-lookup-failure 5 clear 5.
- If the primary fails, the secondary attempts about 2 seconds later (not configurable).
- Bulkstat variables in schemas CONTEXTSch1 and CONTEXTSch2 contain all the relevant DNS infrastructure variables related to this command. Examples for NAPTR query types include these and they are also applicable for AAAA and A type queries:
- dns-primary-ns-naptr-atmpts
- dns-primary-ns-naptr-fails
- dns-primary-ns-naptr-succs
- dns-secondary-ns-naptr-atmpts
- dns-secondary-ns-naptr-fails
- dns-secondary-ns-query-timeouts
In this example output, note the increase in NAPTR failures as reflected in usage and resolver statistics equally for both the primary and secondary servers (full outage):
[Ingress]HSGW> show dns-client statistics client HSGW-DNS
Monday June 02 00:26:29 UTC 2014
DNS Usage Statistics:
---------------------
Query Type Attempts Successes Failures
A 21802 0 21802
SRV 0 0 0
AAAA 3934082666 3934060659 21831
NAPTR 1393765619 1387607858 6156730
PTR 0 0 0
Total 1032902791 1026701221 6200363
DNS Cache Statistics:
---------------------
Total Cache Hits Cache Hits Not Found Hit Ratio
Lookups (Positive (Negative in Cache (Percentage)
Response) Response)
----------------------------------------------------------------------------
Central Cache: 94085256 89157603 6114 4921539 94.77%
Local Cache: 1032902770 926126458 20175 106756137 89.66%
DNS Resolver Statistics:
------------------------
Primary Name Server : 2001:5555:202:fffe:a0:e:0:3
Query Type Attempts Successes Failures
A 0 0 0
SRV 0 0 0
AAAA 66 64 2
NAPTR 746 37 709
PTR 0 0 0
Total Resolver Queries: 812
Successful Queries: 101
Query Timeouts: 705
Domain Not Found: 1
Connection Refused: 0
Other Failures: 5
Secondary Name Server : 2001:5555:203:fffe:c0:e:0:3
Query Type Attempts Successes Failures
A 0 0 0
SRV 0 0 0
AAAA 0 0 0
NAPTR 705 0 703
PTR 0 0 0
Total Resolver Queries: 705
Successful Queries: 0
Query Timeouts: 703
Domain Not Found: 0
Connection Refused: 0
Other Failures: 0
---------------------------------------------------
[Ingress]HSGW> show dns-client statistics client HSGW-DNS
Monday June 02 00:32:00 UTC 2014
DNS Usage Statistics:
---------------------
Query Type Attempts Successes Failures
A 21802 0 21802
SRV 0 0 0
AAAA 3934232613 3934210617 21831
NAPTR 1393923407 1387654707 6267989
PTR 0 0 0
Total 1033210526 1026898028 6320622
DNS Cache Statistics:
---------------------
Total Cache Hits Cache Hits Not Found Hit Ratio
Lookups (Positive (Negative in Cache (Percentage)
Response) Response)
----------------------------------------------------------------------------
Central Cache: 94120194 89157771 6114 4956309 94.73%
Local Cache: 1033210498 926323077 20175 106867246 89.66%
DNS Resolver Statistics:
------------------------
Primary Name Server : 2001:5555:202:fffe:a0:e:0:3
Query Type Attempts Successes Failures
A 0 0 0
SRV 0 0 0
AAAA 66 64 2
NAPTR 913 38 873
PTR 0 0 0
Total Resolver Queries: 979
Successful Queries: 102
Query Timeouts: 869
Domain Not Found: 1
Connection Refused: 0
Other Failures: 5
Secondary Name Server : 2001:5555:203:fffe:c0:e:0:3
Query Type Attempts Successes Failures
A 0 0 0
SRV 0 0 0
AAAA 0 0 0
NAPTR 869 0 869
PTR 0 0 0
Total Resolver Queries: 869
Successful Queries: 0
Query Timeouts: 869
Domain Not Found: 0
Connection Refused: 0
Other Failures: 0
show dns-client cache client <client name> [query-name <query-name>
[query-type <NAPTR | AAAA | A>] | [query-type <NAPTR | AAAA | A>]]
This command reports all of the saved responses in the cache (have not expired yet) for the various query types and include A, AAAA, and NAPTR. This gives the current state of the cache from which conclusions can be made as to whether there would be call failures based on missing entries:
Without any qualifiers the whole cache is displayed which can be more than needed for what you intend to troubleshoot. The cache entries have a time to live (TTL), so the entries returned are only applicable for as long as the respective TTLs also remain. The TTLs are likely to be different when you compare amongst all the entries, so that entries expire at different times. This is expected.
Choose a specific query type such as NAPTR and look for Application Point Name (APN) results or specific FQDN (handoffs) results. Things to look for include a specific APN this is missing, all APNs missing, or handoff results missing.
Example:
This output shows some entries from the cache for APN1 and APN2 which could be needed for new calls for those APNs. The actual list includes entries for every possible PGW for every possible APN, for both LTE (x-S5-gtp) and eHRPD (x-s2a-pmip) in the entire service provider's network. Only x-s2a-pmip is relevant here since this is an HSGW that needs to connect to a PGW over S2a connection. Note the same TTL (1307, 631) for entries with the same APN/PGW location that were returned from the DNS server at the same time, compared to a different TTL (1307 vs. 631) for entries that apply to a different APN/PGW location.
[Ingress]HSGW> show dns-client cache client HSGW-DNS
Monday June 02 00:26:59 UTC 2014
Query Name: so01.APN1.apn.epc.mnc485.mcc320.3gppnetwork.org
Query Type: NAPTR TTL: 1307 seconds
Answer:
Order: 100 Preference: 50000
Flags: a Service: x-3gpp-pgw:x-s5-gtp
Regular Expression:
Replacement: topon.lb1.pgw01.NYNY.sa008.so.node.epc.mnc485.mcc320.3gppnetwork.org
Query Name: so01.APN1.apn.epc.mnc485.mcc320.3gppnetwork.org
Query Type: NAPTR TTL: 1307 seconds
Answer:
Order: 100 Preference: 50000
Flags: a Service: x-3gpp-pgw:x-s2a-pmip
Regular Expression:
Replacement: topon.lb2.pgw01.NYNY.sa008.so.node.epc.mnc485.mcc320.3gppnetwork.org
Query Name: APN2.apn.epc.mnc485.mcc320.3gppnetwork.org
Query Type: NAPTR TTL: 631 seconds
Answer:
Order: 100 Preference: 50000
Flags: a Service: x-3gpp-pgw:x-s2a-pmip
Regular Expression:
Replacement: topon.lb2.pgw01.BOMA.sa001.mw.node.epc.mnc485.mcc320.3gppnetwork.org
Query Name: APN2.apn.epc.mnc485.mcc320.3gppnetwork.org
Query Type: NAPTR TTL: 631 seconds
Answer:
Order: 100 Preference: 50000
Flags: a Service: x-3gpp-pgw:x-s5-gtp
Regular Expression:
Replacement: topon.lb1.pgw01.BOMA.sa001.mw.node.epc.mnc485.mcc320.3gppnetwork.org
In this second output example are NAPTR entries needed for handoffs from Long Term Evolution (LTE) to eHRPD as demonstrated by the specific PGW FQDN location entry (pgw01.PHLA.xxxxxx). Similar to the previous output, the relevant entry that is used is the one with Service = x-s2a-pmip. Note the same TTL (515) for all these entries returned at the same time. The only difference is the Service. The AAAA entry resolves the s2a entry that represents the PGW LMA service address so that an ensuing Proxy MIPv6 request can be sent to the PGW in order to continue the call setup.
Query Name: pgw01.PHLA.sa004.mw.node.epc.mnc485.mcc320.3gppnetwork.org
Query Type: NAPTR TTL: 515 seconds
Answer:
Order: 100 Preference: 50000
Flags: a Service: x-3gpp-pgw:x-s2b-gtp
Regular Expression:
Replacement: topon.lb4.pgw01.PHLA.sa004.mw.node.epc.mnc485.mcc320.3gppnetwork.org
Query Name: pgw01.PHLA.sa004.mw.node.epc.mnc485.mcc320.3gppnetwork.org
Query Type: NAPTR TTL: 515 seconds
Answer:
Order: 100 Preference: 50000
Flags: a Service: x-3gpp-pgw:x-s2a-pmip
Regular Expression:
Replacement: topon.lb2.pgw01.PHLA.sa004.mw.node.epc.mnc485.mcc320.3gppnetwork.org
Query Name: pgw01.PHLA.sa004.mw.node.epc.mnc485.mcc320.3gppnetwork.org
Query Type: NAPTR TTL: 515 seconds
Answer:
Order: 100 Preference: 50000
Flags: a Service: x-3gpp-pgw:x-s5-gtp
Regular Expression:
Replacement: topon.lb1.pgw01.PHLA.sa004.mw.node.epc.mnc485.mcc320.3gppnetwork.org
Query Name: topon.lb2.pgw01.PHLA.sa004.mw.node.epc.mnc485.mcc320.3gppnetwork.org
Query Type: AAAA TTL: 646 seconds
Answer:
IPv6 Address: 2001:5555:200:1000:304:200::
dns-client query client-name <client name> query-type <NAPTR | AAAA> [query-name <query name>]
This is a manual test command that initiates the DSN client in order to immediately check the cache and report the answers if they are there. Otherwise it attempts the query and reports the results. Ensure the query string is properly spelled if complex:
- By default if just the query-name is specified, the client assumes a query type = A, so the query type is needed for NAPTR and AAAA requests.
- The results here are the same as would be returned if you query the cache with show dns-client cache. The exception is that if what is queried is not in the cache, the results have a fresh TTL. Whereas if already in the cache, the TTL has some value between what is returned in a new query and 0.
Example (same query as from previous output):
[Ingress]HSGW> dns-client query client-name HSGW-DNS query-type NAPTR
query-name pgw01.PHLA.sa004.mw.node.epc.mnc420.mcc300.3gppnetwork.org
Query Name: pgw01.PHLA.sa004.mw.node.epc.mnc420.mcc300.3gppnetwork.org
Query Type: NAPTR TTL: 188 seconds
Answer:
Order: 100 Preference: 50000
Flags: a Service: x-3gpp-pgw:x-s5-gtp
Regular Expression:
Replacement: topon.lb1.pgw01.PHLA.sa004.mw.node.epc.mnc420.mcc300.3gppnetwork.org
Query Name: pgw01.PHLA.sa004.mw.node.epc.mnc420.mcc300.3gppnetwork.org
Query Type: NAPTR TTL: 188 seconds
Answer:
Order: 100 Preference: 50000
Flags: a Service: x-3gpp-pgw:x-s2b-gtp
Regular Expression:
Replacement: topon.lb4.pgw01.PHLA.sa004.mw.node.epc.mnc420.mcc300.3gppnetwork.org
Query Name: pgw01.PHLA.sa004.mw.node.epc.mnc420.mcc300.3gppnetwork.org
Query Type: NAPTR TTL: 188 seconds
Answer:
Order: 100 Preference: 50000
Flags: a Service: x-3gpp-pgw:x-s2a-pmip
Regular Expression:
Replacement: topon.lb2.pgw01.PHLA.sa004.mw.node.epc.mnc420.mcc300.3gppnetwork.org
[Ingress]HSGW> dns-client query client-name HSGW-DNS query-type AAAA
query-name topon.lb2.pgw01.PHLA.sa004.mw.node.epc.mnc420.mcc300.3gppnetwork.org
Query Name: topon.lb2.pgw01.PHLA.sa004.mw.node.epc.mnc420.mcc300.3gppnetwork.org
Query Type: AAAA TTL: 117 seconds
Answer:
IPv6 Address: 2001:5555:200:1000:304:200::
This output shows an example of failures for a TCP-based query. You cannot tell that this is TCP based just from the query itself but knowledge of the fact that an APN-based NAPTR query response is too large for UDP.
[Ingress]HSW> dns-client query client-name HSGW-DNS query-type NAPTR
query-name APN1.apn.epc.mnc420.mcc300.3gppnetwork.org
Query Name: APN1.apn.epc.mnc420.mcc300.3gppnetwork.org
Query Type: NAPTR TTL: 0 seconds
Answer: -Negative Reply-
Failure Reason: DNS query timed out
...
[Ingress]HSGW> dns-client query client-name HSGW-DNS query-type NAPTR
query-name APN2.apn.epc.mnc420.mcc300.3gppnetwork.org
Query Name: APN2.apn.epc.mnc420.mcc300.3gppnetwork.org
Query Type: NAPTR TTL: 60 seconds
Answer: -Negative Reply-
Failure Reason: Connection Refused
Monitor Protocol (Option for DNS)
Monitor protocol reports all DNS infrastructure packet exchanges. Monitor subscriber (covered later) does not capture DNS packets even if the subscriber activity is what initiated a DNS exchange.
- Query ID is useful in order to match requests with responses.
But:
- In the case of a switch to TCP, the output does not indicate that fact (as output shows).
- The port numbers are not necessarily accurate in the output, for example port = 0.
- The system might combine multiple packets, such as APN queries, into a single packet on the wire which is not reflected at this level of output. This continues to show separate packets for each APN.
- Care must be taken with monitor protocol so as to not overload the system. Consult with Technical support before doing so.
<<<<OUTBOUND 00:58:57:284 Eventid:5957(3)
DNS PDU Tx
from : 2001:5555:200:1011:304:281:: : 52816
to : 2001:5555:202:fffe:a0:e:0:3 : 0
bytes : 73
Query ID : 17034
Type : Query
Question : NAPTR ? APN2.apn.epc.mnc420.mcc300.3gppnetwork.org.
Additional :
Name : .
Ext-RCODE : 0
Type : OPT
UDPsize : 4096
INBOUND>>>>> 00:58:57:469 Eventid:5956(3)
DNS PDU Rx
from : 2001:5555:202:fffe:a0:e:0:3 : 0
to : 2001:5555:200:1011:304:281:: : 0
bytes : 16738
Query ID : 17034
Type : Response
Authoritative Answer : Yes
Response code : Success
Question : NAPTR ? APN2.apn.epc.mnc420.mcc300.3gppnetwork.org.
Answer :
Name : APN2.apn.epc.mnc420.mcc300.3gppnetwork.org.
TTL : 1800
Type : NAPTR
Order : 100
Preference : 50000
Flags : a
Service : x-3gpp-pgw:x-s2a-pmip
Regexp :
Replacement : topon.lb2.pgw01.PHLA.sa001.we.node.epc.mnc420.
mcc300.3gppnetwork.org.
Name : APN2.apn.epc.mnc420.mcc300.3gppnetwork.org.
TTL : 1800
Type : NAPTR
Order : 100
Preference : 50000
Flags : a
Service : x-3gpp-pgw:x-s5-gtp
Regexp :
Replacement : topon.lb1.pgw01.PHLA.sa001.we.node.epc.
mnc420.mcc300.3gppnetwork.org
This example shows three APNs that ended up wrapped up into one packet, switched to TCP, 2 second timeouts for each APN, and finally retried to the secondary servers that also failed.
Primary Server: 2001:5555:202:fffe:a0:e:0:3
Secondary Server: 2001:5555:203:fffe:c0:e:0:3
<<<<OUTBOUND 13:03:08:056 Eventid:5957(3)
DNS PDU Tx
from : 2001:5555:200:1011:106:281:: : 35428
to : 2001:5555:202:fffe:a0:e:0:3 : 53
bytes : 78
Query ID : 23363
Type : Query
Opcode : Standard Query
Message Truncated : No
Recursion Desired : Yes
Authentication reqd. : No
Question count : 1
Additional count : 1
Question : NAPTR ? APN1.apn.epc.mnc420.mcc300.3gppnetwork.org.
Additional :
Name : .
Ext-RCODE : 0
EDNS Version : 0
Class : 4096
Data Length : 0
Type : OPT
UDPsize : 4096
Monday October 13 2014
<<<<OUTBOUND 13:03:08:057 Eventid:5957(3)
DNS PDU Tx
from : 2001:5555:200:1011:106:281:: : 60489
to : 2001:5555:202:fffe:a0:e:0:3 : 53
bytes : 73
Query ID : 48443
Type : Query
Opcode : Standard Query
Message Truncated : No
Recursion Desired : Yes
Authentication reqd. : No
Question count : 1
Additional count : 1
Question : NAPTR ? APN3.apn.epc.mnc420.mcc300.3gppnetwork.org.
Additional :
Name : .
Ext-RCODE : 0
EDNS Version : 0
Class : 4096
Data Length : 0
Type : OPT
UDPsize : 4096
Monday October 13 2014
<<<<OUTBOUND 13:03:08:057 Eventid:5957(3)
DNS PDU Tx
from : 2001:5555:200:1011:106:281:: : 34309
to : 2001:5555:202:fffe:a0:e:0:3 : 53
bytes : 73
Query ID : 51787
Type : Query
Opcode : Standard Query
Message Truncated : No
Recursion Desired : Yes
Authentication reqd. : No
Question count : 1
Additional count : 1
Question : NAPTR ? APN2.apn.epc.mnc420.mcc300.3gppnetwork.org.
Additional :
Name : .
Ext-RCODE : 0
EDNS Version : 0
Class : 4096
Data Length : 0
Type : OPT
UDPsize : 4096
Monday October 13 2014
INBOUND>>>> 13:03:08:064 Eventid:5956(3)
DNS PDU Rx
from : 2001:5555:202:fffe:a0:e:0:3 : 53
to : 2001:5555:200:1011:106:281:: : 35428
bytes : 78
Query ID : 23363
Type : Response
Opcode : Standard Query
Message Truncated : Yes
Recursion Desired : Yes
Recursion Available : Yes
Authenticated Answer : No
Authoritative Answer : Yes
Response code : Success
Question count : 1
Answer count : 0
Authoritative count : 0
Additional count : 1
Question : NAPTR ? APN1.apn.epc.mnc420.mcc300.3gppnetwork.org.
Additional :
Name : .
Ext-RCODE : 0
EDNS Version : 0
Class : 4096
Data Length : 0
Type : OPT
UDPsize : 4096
Monday October 13 2014
INBOUND>>>> 13:03:08:064 Eventid:5956(3)
DNS PDU Rx
from : 2001:5555:202:fffe:a0:e:0:3 : 53
to : 2001:5555:200:1011:106:281:: : 60489
bytes : 73
Query ID : 48443
Type : Response
Opcode : Standard Query
Message Truncated : Yes
Recursion Desired : Yes
Recursion Available : Yes
Authenticated Answer : No
Authoritative Answer : Yes
Response code : Success
Question count : 1
Answer count : 0
Authoritative count : 0
Additional count : 1
Question : NAPTR ? APN3.apn.epc.mnc420.mcc300.3gppnetwork.org.
Additional :
Name : .
Ext-RCODE : 0
EDNS Version : 0
Class : 4096
Data Length : 0
Type : OPT
UDPsize : 4096
Monday October 13 2014
INBOUND>>>> 13:03:08:069 Eventid:5956(3)
DNS PDU Rx
from : 2001:5555:202:fffe:a0:e:0:3 : 53
to : 2001:5555:200:1011:106:281:: : 34309
bytes : 73
Query ID : 51787
Type : Response
Opcode : Standard Query
Message Truncated : Yes
Recursion Desired : Yes
Recursion Available : Yes
Authenticated Answer : No
Authoritative Answer : Yes
Response code : Success
Question count : 1
Answer count : 0
Authoritative count : 0
Additional count : 1
Question : NAPTR ? APN2.apn.epc.mnc420.mcc300.3gppnetwork.org.
Additional :
Name : .
Ext-RCODE : 0
EDNS Version : 0
Class : 4096
Data Length : 0
Type : OPT
UDPsize : 4096
Monday October 13 2014
<<<<OUTBOUND 13:03:08:147 Eventid:5957(3)
DNS PDU Tx
from : 2001:5555:200:1011:106:281:: : 36524
to : 2001:5555:202:fffe:a0:e:0:3 : 0
bytes : 78
Query ID : 23363
Type : Query
Opcode : Standard Query
Message Truncated : No
Recursion Desired : Yes
Authentication reqd. : No
Question count : 1
Additional count : 1
Question : NAPTR ? APN1.apn.epc.mnc420.mcc300.3gppnetwork.org.
Additional :
Name : .
Ext-RCODE : 0
EDNS Version : 0
Class : 4096
Data Length : 0
Type : OPT
UDPsize : 4096
Monday October 13 2014
<<<<OUTBOUND 13:03:08:147 Eventid:5957(3)
DNS PDU Tx
from : 2001:5555:200:1011:106:281:: : 36524
to : 2001:5555:202:fffe:a0:e:0:3 : 0
bytes : 73
Query ID : 48443
Type : Query
Opcode : Standard Query
Message Truncated : No
Recursion Desired : Yes
Authentication reqd. : No
Question count : 1
Additional count : 1
Question : NAPTR ? APN3.apn.epc.mnc420.mcc300.3gppnetwork.org.
Additional :
Name : .
Ext-RCODE : 0
EDNS Version : 0
Class : 4096
Data Length : 0
Type : OPT
UDPsize : 4096
Monday October 13 2014
<<<<OUTBOUND 13:03:08:147 Eventid:5957(3)
DNS PDU Tx
from : 2001:5555:200:1011:106:281:: : 36524
to : 2001:5555:202:fffe:a0:e:0:3 : 0
bytes : 73
Query ID : 51787
Type : Query
Opcode : Standard Query
Message Truncated : No
Recursion Desired : Yes
Authentication reqd. : No
Question count : 1
Additional count : 1
Question : NAPTR ? APN2.apn.epc.mnc420.mcc300.3gppnetwork.org.
Additional :
Name : .
Ext-RCODE : 0
EDNS Version : 0
Class : 4096
Data Length : 0
Type : OPT
UDPsize : 4096
Monday October 13 2014
<<<<OUTBOUND 13:03:10:157 Eventid:5957(3)
DNS PDU Tx
from : 2001:5555:200:1011:106:281:: : 57041
to : 2001:5555:203:fffe:c0:e:0:3 : 0
bytes : 78
Query ID : 23363
Type : Query
Opcode : Standard Query
Message Truncated : No
Recursion Desired : Yes
Authentication reqd. : No
Question count : 1
Additional count : 1
Question : NAPTR ? APN1.apn.epc.mnc420.mcc300.3gppnetwork.org.
Additional :
Name : .
Ext-RCODE : 0
EDNS Version : 0
Class : 4096
Data Length : 0
Type : OPT
UDPsize : 4096
Monday October 13 2014
<<<<OUTBOUND 13:03:10:157 Eventid:5957(3)
DNS PDU Tx
from : 2001:5555:200:1011:106:281:: : 57041
to : 2001:5555:203:fffe:c0:e:0:3 : 0
bytes : 73
Query ID : 48443
Type : Query
Opcode : Standard Query
Message Truncated : No
Recursion Desired : Yes
Authentication reqd. : No
Question count : 1
Additional count : 1
Question : NAPTR ? APN3.apn.epc.mnc420.mcc300.3gppnetwork.org.
Additional :
Name : .
Ext-RCODE : 0
EDNS Version : 0
Class : 4096
Data Length : 0
Type : OPT
UDPsize : 4096
Monday October 13 2014
<<<<OUTBOUND 13:03:10:157 Eventid:5957(3)
DNS PDU Tx
from : 2001:5555:200:1011:106:281:: : 57041
to : 2001:5555:203:fffe:c0:e:0:3 : 0
bytes : 73
Query ID : 51787
Type : Query
Opcode : Standard Query
Message Truncated : No
Recursion Desired : Yes
Authentication reqd. : No
Question count : 1
Additional count : 1
Question : NAPTR ? APN2.apn.epc.mnc420.mcc300.3gppnetwork.org.
Additional :
Name : .
Ext-RCODE : 0
EDNS Version : 0
Class : 4096
Data Length : 0
Type : OPT
UDPsize : 4096
This image shows the three APNs wrapped into one packet #10. The original UDP requests in 1 - 3 are responded in 4, 5 and 7, and the TCP handshake consists of packets 6, 8, and 9. In this case the connection was immediately reset in packet 12 by the server after it initially ACK'd the TCP-based query via packet 11. These are the types of issues that you might need to troubleshoot:
Finally from the same capture, here is a successful query and response for a NAPTR query over UDP followed immediately by the required AAAA query and response in order to resolve the FQDN returned by the NAPTR query. This output matches the Wireshark trace saved as text:
Monday October 13 2014
<<<<OUTBOUND 13:03:11:535 Eventid:5957(3)
DNS PDU Tx
from : 2001:5555:200:1011:106:281:: : 38819
to : 2001:5555:202:fffe:a0:e:0:3 : 53
bytes : 87
Query ID : 55982
Type : Query
Opcode : Standard Query
Message Truncated : No
Recursion Desired : Yes
Authentication reqd. : No
Question count : 1
Additional count : 1
Question : NAPTR ? pgw02.PHLA.sa002.so.node.epc.mnc420.mcc300.3gppnetwork.org.
Additional :
Name : .
Ext-RCODE : 0
EDNS Version : 0
Class : 4096
Data Length : 0
Type : OPT
UDPsize : 4096
Monday October 13 2014
INBOUND>>>>> 13:03:11:543 Eventid:5956(3)
DNS PDU Rx
from : 2001:5555:202:fffe:a0:e:0:3 : 53
to : 2001:5555:200:1011:106:281:: : 38819
bytes : 307
Query ID : 55982
Type : Response
Opcode : Standard Query
Message Truncated : No
Recursion Desired : Yes
Recursion Available : Yes
Authenticated Answer : No
Authoritative Answer : Yes
Response code : Success
Question count : 1
Answer count : 2
Authoritative count : 0
Additional count : 1
Question : NAPTR ? pgw02.PHLA.sa002.so.node.epc.mnc420.
mcc300.3gppnetwork.org.
Answer :
Name : pgw02.PHLA.sa002.so.node.epc.mnc420.mcc300.3gppnetwork.org.
TTL : 1800
Class : IN
Data Length : 99
Type : NAPTR
Order : 100
Preference : 50000
Flags : a
Service : x-3gpp-pgw:x-s2a-pmip
Regexp :
Replacement : topon.lb2.pgw02.PHLA.sa002.so.node.epc.mnc420.
mcc300.3gppnetwork.org.
Name : pgw02.PHLA.sa002.so.node.epc.mnc420.mcc300.
3gppnetwork.org.
TTL : 1800
Class : IN
Data Length : 97
Type : NAPTR
Order : 100
Preference : 50000
Flags : a
Service : x-3gpp-pgw:x-s5-gtp
Regexp :
Replacement : topon.lb1.pgw02.PHLA.sa002.so.node.epc.mnc420.
mcc300.3gppnetwork.org.
Additional :
Name : .
Ext-RCODE : 0
EDNS Version : 0
Class : 4096
Data Length : 0
Type : OPT
UDPsize : 4096
Monday October 13 2014
<<<<OUTBOUND 13:03:11:543 Eventid:5957(3)
DNS PDU Tx
from : 2001:5555:200:1011:106:281:: : 50002
to : 2001:5555:202:fffe:a0:e:0:3 : 53
bytes : 97
Query ID : 1974
Type : Query
Opcode : Standard Query
Message Truncated : No
Recursion Desired : Yes
Authentication reqd. : No
Question count : 1
Additional count : 1
Question : AAAA? topon.lb2.pgw02.PHLA.sa002.so.node.epc.
mnc420.mcc300.3gppnetwork.org.
Additional :
Name : .
Ext-RCODE : 0
EDNS Version : 0
Class : 4096
Data Length : 0
Type : OPT
UDPsize : 4096
Monday October 13 2014
INBOUND>>>>> 13:03:11:551 Eventid:5956(3)
DNS PDU Rx
from : 2001:5555:202:fffe:a0:e:0:3 : 53
to : 2001:5555:200:1011:106:281:: : 50002
bytes : 125
Query ID : 1974
Type : Response
Opcode : Standard Query
Message Truncated : No
Recursion Desired : Yes
Recursion Available : Yes
Authenticated Answer : No
Authoritative Answer : Yes
Response code : Success
Question count : 1
Answer count : 1
Authoritative count : 0
Additional count : 1
Question : AAAA? topon.lb2.pgw02.PHLA.sa002.so.node.epc.
mnc420.mcc300.3gppnetwork.org.
Answer :
Name : topon.lb2.pgw02.PHLA.sa002.so.node.epc.mnc420.
mcc300.3gppnetwork.org.
TTL : 1800
Class : IN
Data Length : 16
Type : AAAA
Address : 2001:5555:200:1000:201:201::
Additional :
Name : .
Ext-RCODE : 0
EDNS Version : 0
Class : 4096
Data Length : 0
Type : OPT
UDPsize : 4096
Corresponding Wireshark trace:
Frame 25: 151 bytes on wire (1208 bits), 151 bytes captured (1208 bits)
Linux cooked capture
Internet Protocol Version 6, Src: HSGW, Dst: DNS_Server
User Datagram Protocol, Src Port: 38819 (38819), Dst Port: domain (53)
Domain Name System (query)
[Response In: 26]
Transaction ID: 0xdaae
Flags: 0x0100 Standard query
Questions: 1
Answer RRs: 0
Authority RRs: 0
Additional RRs: 1
Queries
pgw02.PHLA.sa002.so.node.epc.mnc420.mcc300.3gppnetwork.org: type NAPTR, class IN
Additional records
<Root>: type OPT
Name: <Root>
Type: OPT (EDNS0 option)
UDP payload size: 4096
Higher bits in extended RCODE: 0x0
EDNS0 version: 0
Z: 0x0
Data length: 0
Frame 26: 371 bytes on wire (2968 bits), 371 bytes captured (2968 bits)
Linux cooked capture
Internet Protocol Version 6, Src: DNS_Server, Dst: HSGW
User Datagram Protocol, Src Port: domain (53), Dst Port: 38819 (38819)
Domain Name System (response)
[Request In: 25]
[Time: 0.008125000 seconds]
Transaction ID: 0xdaae
Flags: 0x8580 Standard query response, No error
Questions: 1
Answer RRs: 2
Authority RRs: 0
Additional RRs: 1
Queries
pgw02.PHLA.sa002.so.node.epc.mnc420.mcc300.3gppnetwork.org: type NAPTR, class IN
Answers
pgw02.PHLA.sa002.so.node.epc.mnc420.mcc300.3gppnetwork.org:
type NAPTR, class IN, order 100, preference 50000, flags a
Name: pgw02.PHLA.sa002.so.node.epc.mnc420.mcc300.3gppnetwork.org
Type: NAPTR (Naming authority pointer)
Class: IN (0x0001)
Time to live: 30 minutes
Data length: 99
Order: 100
Preference: 50000
Flags length: 1
Flags: "a"
Service length: 21
Service: "x-3gpp-pgw:x-s2a-pmip"
Regex length: 0
Regex: ""
Replacement length: 70
Replacement: topon.lb2.pgw02.PHLA.sa002.so.node.epc.
mnc420.mcc300.3gppnetwork.org
pgw02.PHLA.sa002.so.node.epc.mnc420.mcc300.3gppnetwork.org:
type NAPTR, class IN, order 100, preference 50000, flags a
Name: pgw02.PHLA.sa002.so.node.epc.mnc420.mcc300.3gppnetwork.org
Type: NAPTR (Naming authority pointer)
Class: IN (0x0001)
Time to live: 30 minutes
Data length: 97
Order: 100
Preference: 50000
Flags length: 1
Flags: "a"
Service length: 19
Service: "x-3gpp-pgw:x-s5-gtp"
Regex length: 0
Regex: ""
Replacement length: 70
Replacement: topon.lb1.pgw02.PHLA.sa002.so.node.epc.
mnc420.mcc300.3gppnetwork.org
Additional records
<Root>: type OPT
Name: <Root>
Type: OPT (EDNS0 option)
UDP payload size: 4096
Higher bits in extended RCODE: 0x0
EDNS0 version: 0
Z: 0x0
Data length: 0
Frame 27: 161 bytes on wire (1288 bits), 161 bytes captured (1288 bits)
Linux cooked capture
Internet Protocol Version 6, Src: HSGW, Dst: DNS_Server
User Datagram Protocol, Src Port: 50002 (50002), Dst Port: domain (53)
Domain Name System (query)
[Response In: 28]
Transaction ID: 0x07b6
Flags: 0x0100 Standard query
Questions: 1
Answer RRs: 0
Authority RRs: 0
Additional RRs: 1
Queries
topon.lb2.pgw02.PHLA.sa002.so.node.epc.mnc420.mcc300.3gppnetwork.org:
type AAAA, class IN
Additional records
<Root>: type OPT
Name: <Root>
Type: OPT (EDNS0 option)
UDP payload size: 4096
Higher bits in extended RCODE: 0x0
EDNS0 version: 0
Z: 0x0
Data length: 0
Frame 28: 189 bytes on wire (1512 bits), 189 bytes captured (1512 bits)
Linux cooked capture
Internet Protocol Version 6, Src: DNS_Server , Dst: HSGW
User Datagram Protocol, Src Port: domain (53), Dst Port: 50002 (50002)
Domain Name System (response)
[Request In: 27]
[Time: 0.007622000 seconds]
Transaction ID: 0x07b6
Flags: 0x8580 Standard query response, No error
Questions: 1
Answer RRs: 1
Authority RRs: 0
Additional RRs: 1
Queries
topon.lb2.pgw02.PHLA.sa002.so.node.epc.mnc420.mcc300.3gppnetwork.org:
type AAAA, class IN
Answers
topon.lb2.pgw02.PHLA.sa002.so.node.epc.mnc420.mcc300.3gppnetwork.org:
type AAAA, class IN, addr 2001:5555:200:1000:201:201::
Name: topon.lb2.pgw02.PHLA.sa002.so.node.epc.mnc420.mcc300.3gppnetwork.org
Type: AAAA (IPv6 address)
Class: IN (0x0001)
Time to live: 30 minutes
Data length: 16
Addr: 2001:5555:200:1000:201:201::
Additional records
<Root>: type OPT
Name: <Root>
Type: OPT (EDNS0 option)
UDP payload size: 4096
Higher bits in extended RCODE: 0x0
EDNS0 version: 0
Z: 0x0
Data length: 0
Logs
A feature was added in version 12.2 where a high number of connection refused errors within a two minute period triggers a rebind of the IP address configured for the DNS client in outage scenarios. Example log entry is:
[vpn 5795 error] [1/0/30805 <vpnmgr:4> vpnmgr_msg.c:13773]
[context: Ingress, contextID: 4] [software internal system syslog]
Ingress: Rebinding DNS-CLIENT as connection refused errors
(<# of failures>) occuring continously
Example:
Jun 2 00:03:36 [10.142.250.226.171.216] evlogd: [local-60sec36.031]
[vpn 5450 error] [1/0/30805 <vpnmgr:4> vpnmgr_msg.c:13680] [context: Ingress,
contextID: 4] [software internal system syslog] Connection
refused for DNS query on QNAME:APN1.apn.epc.mnc420.mcc300.3gppnetwork.org
and QTYPE:NAPTR....... Many more of these logs
Jun 2 00:05:35 [10.142.250.226.171.216] evlogd: [local-60sec35.058]
[vpn 5450 error] [1/0/30805 <vpnmgr:4> vpnmgr_msg.c:13680]
[context: Ingress, contextID: 4] [software internal system syslog]
Connection refused for DNS query on QNAME:APN1.apn.epc.mnc420.mcc300.3gppnetwork.org
and QTYPE:NAPTR
Jun 2 00:05:35 [10.142.250.226.171.216] evlogd: [local-60sec35.058]
[vpn 5795 error] [1/0/30805 <vpnmgr:4> vpnmgr_msg.c:13773]
[context: Ingress, contextID: 4] [software internal system syslog]
Ingress: Rebinding DNS-CLIENT as connection refused errors (3132) occuring continously
Packet Capture
Some tricky DNS issues have been seen where a packet capture was necessary in order to determine what was sent and received back from the DNS server. Statistics and monitor protocol might not provide enough information.
- Technical Support has the ability to capture DNS packets with a TCP Dump facility and might suggest that approach as part of the troubleshooting process.
- The capture point can be important based on any firewalls that intervene and that can negotiate TCP/IP connections. Multiple capture points might be necessary to find the root cause of an issue.
- Use the Follow TCP stream menu option in Wireshark in order to filter specific TCP connections in order to more easily navigate multiple TCP streams in a large file.
Troubleshoot DNS as it Relates to Call Control
As mentioned earlier, DNS does not function by itself but it is an enabler or component of Call Control flows. For example, in the case of eHRPD, DNS is required at the point of the call when the PGW to connect to needs to be determined. If failure occurs at this point in the flow, then the appropriate Call Control statistics reflect this.
show hsgw-service statistics
Expect the "No PGW Available" counter to increase if DNS has failed. Because the call would fail before attempting to make a request to a PGW, the "show mag statistics" would not capture this (there would simply be no Binding Updates Sent counted for those events)
Example:
[Ingress]HSGW> show hsgw statistics all
Monday June 02 00:49:06 UTC 2014
Total PDNs Rejected Reason:
No PGW Available: 9549866
[Ingress]HSGW> show hsgw statistics all
Monday June 02 00:49:16 UTC 2014
No PGW Available: 9554113
Monitor Subscriber
Note that DNS packets themselves are NOT captured in monitor subscriber. Even though they do get triggered by individual subscriber activity, they work independent of a given subscriber and must be captured by monitor protocol as discussed earlier.
DNS infrastrucControl messages such as "No LMA address available for APN <APN Name> in subscriber profile, PDN connection failed" appear and a VSNCP Conf-Rej is sent to the subscriber with "Error-Code(6)=No-PDN-GW-Available(3)".
Example:
INBOUND>>>> 00:25:26:925 Eventid:25000(0) PPP Rx PDU (72) VSNCP 72:
Conf-Req(2), OUI=cf0002(3GPP2) , PDN-ID(1)=00, PDN-APN-Name(2)=\013APN1,
PDN-Type(3)=IPv4,IPv6(3), PDN-Address(4)=(Null), PCO(5)
{Protocol(0) = PPP(0),{IPCP
(1): Conf-Req(1), Pri-DNS=0.0.0.0, Sec-DNS=0.0.0.0},IPv6-DNS-Address(2)=Req,IP-Address-
Allocation-via-NAS-Signaling(3),}, Attach-Type(7)=Initial(1),
IPv4-Default-Router-Address(8)=0.0.0.0, Address-Allocation-Cause(9)=Null(0)
***CONTROL*** 00:25:27:054 Eventid:11813
No LMA address available for APN <APN1> in subscriber profile, PDN connection failed
Monday June 02 2014
<<<<OUTBOUND 00:25:27:054 Eventid:25001(0)
PPP Tx PDU (14)
VSNCP 14: Conf-Req(1), OUI=cf0002(3GPP2) , PDN-ID(1)=00
Monday June 02 2014
<<<<OUTBOUND 00:25:27:054 Eventid:25001(0)
PPP Tx PDU (52)
VSNCP 52: Conf-Rej(2), OUI=cf0002(3GPP2) , PDN-ID(1)=00, PDN-APN-Name(2)=\013APN1,
PDN-Type(3)=IPv4,IPv6(3), PDN-Address(4)=(Null), PCO(5){Protocol(0)
= PPP(0),}, Attach-Type(7)=Initial(1), IPv4-Default-Router-Address(8)=0.0.0.0,
Address-Allocation-Cause(9)=Null(0), Error-Code(6)=No-PDN-GW-Available(3)
Logs
Look for any error logs related to call control.
Example:
Jun 2 00:25:27 [10.142.250.226.171.216] evlogd: [local-60sec27.054]
[sessmgr 11813 error] [15/0/5827 <sessmgr:71> sessmgr_mag.c:3595]
[callid 14ec7ad1] [context: Ingress, contextID: 4] [software internal
system protocol-log syslog] No LMA address available for APN <APN1> in
subscriber profile, PDN connection failed
Related Information
Revision History
| Revision | Publish Date | Comments |
|---|---|---|
1.0 |
21-Apr-2016 |
Initial Release |
FeedbackContact Cisco
- Open a Support Case
- (Requires a Cisco Service Contract)