Troubleshoot Akamai-Served Content and Domains after the Switch to Umbrella

Introduction

This document describes how troubleshoot issues with Akamai-served content and domains after switching to Cisco Umbrella.

What is Akamai and how is it used?

Akamai is a Content Delivery Network (CDN) that stores and serves content for many content providers across the web. This includes streaming video, images, site content, scripts, advertising, and more. Akamai is one of many CDNs, such as Amazon CloudFront or Limelight Networks, which provide the backbone of many popular sites.

A CDN is used by a website because it has very high traffic, including major traffic spikes (like when there is a breaking news story). Rather than pay for their own hosting infrastructure to host their site (which needs to be powerful enough to serve peak times while also leaving it partly idle a large portion of the time), web administrators use CDNs so they only pay for the resources that they actually use. During slow times, content flows out using fewer resources and during peak times the CDN scales up its resources to serve everyone without missing a beat. The end result is a site that has much better uptime and saves money for the web administrator. Akamai is one of the very largest of this type of CDN.

To confirm whether or not a website is using Akamai, look for www.sitename.com pointing to subdomains as a CNAME such as edgesuite.net, akamai.net, edgekey.net, amakaiedge.net (generally all .net). Be sure to look up www.domain.com rather than domain.com to check if there is an Akamai CNAME.

Note: Some sites only use Akamai for images or video content within the site and only images reference a domain served by Akamai via CNAME.

Does using Cisco Umbrella affect Akamai content? Why am I getting a different IP returned from Umbrella compared to my ISP?

Using Cisco Umbrella does not affect Akamai content on its own unless any Akamai domains are blocked by one of your domain lists. Remember, any domain added is a wildcard automatically for all subdomains (eg: *.domain) so entering "akamai.net" into your block list breaks many sites across the web.

There can also be differences between the way your ISP's DNS servers and Cisco Umbrella handle Akamai traffic specifically. It can even be the case that the ISP's DNS results in different content being loaded. It is fairly common for the IP address returned by an ISP's DNS server to differ from the one received when querying Cisco, but that does not mean Cisco's results are incorrect. The opposite is true. While the results do differ, this is because Cisco/OpenDNS and Akamai participate in the Global Internet Speedup Project using EDNS Client Subnet (ECS). Continue reading this article to learn about how this affects the IP returned from the DNS request.

Does this only affect Akamai?

No. This can affect any CDN provider who utilizes ECS. Akamai is the most common occurrence.

What is ECS, and how does it make a difference?

ECS stands for EDNS Client Subnet, and it is part of the Global Internet Speedup Project designed to improve the functionality and speed of distributed websites (notably CDNs) across the globe.

Without ECS: DNS is requested from your current DNS server which then queries the Authoritative DNS server of the domain and returns an IP address of the server to connect to. This server is close to that of the DNS server queried and can be very far from your current location. The DNS server queried determines which region's content server is used to provide an answer. The end user can be very far from the server providing content resulting in slow speeds. For example: User A in Brazil queries the local DNS server set which happens to be a DNS server in Miami. The DNS server in Miami returns a reply IP address close to Miami for the domain. User A is frustrated that the download is so slow and coming from the USA.

With ECS: DNS to a recursive DNS server has the source subnet (typically a /24) appended to its DNS request to the authoritative DNS servers. The authoritative server for the domain responds with a custom answer to what it believes is the closest, best server to serve the end user's request. The user can query a DNS server anywhere and still get a local server to serve the content. For example: User A in Brazil queries the local DNS server set which happens to be a DNS server in Miami. The DNS server has ECS enabled in Miami and passes onto the authoritative DNS server for the domain that the user is coming from a subnet in Brazil. It then returns a reply IP address for a data center in Sao Paulo, Brazil for the domain. User A is happy that the DNS server in Miami supports ECS and has a fast download from the content provider.

What issues can that cause when using Cisco Umbrella?

Cisco Umbrella's global DNS network and Akamai are ECS enabled and therefore reply with the best server IP for your current egress subnet of DNS. This can become an issue when load balancing with certain ISPs or when ISPs have routing issues on their end. For example, one ISP can have an internal Akamai server and directs anyone using their ISP DNS servers to this local server. They then block the IPs of other Akamai servers to force everyone to use their local Akamai server to save on transit costs.

When switching to Cisco, we ask Akamai directly what the best server to use is for the end user subnet, and Akamai responds directly. Especially on load-balanced connections, Akamai can return a valid IP that the ISP does not correctly route to.

It is in this scenario where, after switching to Cisco Umbrella, Akamai-served content intermittently does not load. The most common manifestation is with news pages such as loading as a wireframe of HTML without the rich content. The most commonly seen issues are with the ISP Time Warner Cable (RoadRunner).

It is important to note that the IP returned by Cisco Umbrella for Akamai content is valid in these cases and the ISP is timing out as it is not making the connection. To confirm that it is indeed this type of issue, test the exact same IP on a device on a different type of network connection such as a cell phone on mobile data.

If you are not getting an IP address in response to a DNS request to Cisco Umbrella, this is a different issue. Contact support@umbrella.com for help. This case applies only when pages are not loading when Cisco Umbrella is responding with a valid Akamai IP address.

Who can help if this is happening?

Your ISP is the only party who is able to help resolve this issue. This is because the DNS request returns an IP address to which the ISP times out to while the rest of the world can connect to that very same IP. Cisco Umbrella has completed its part in returning a valid IP address in response to the DNS request. The ISP has not completed its part in routing data to this IP address. Your subscription to the ISP is to allow access to Internet resources and this is a failure to do so on their side.

The ISP is the only one able to troubleshoot routing between their network and its peers; Cisco Umbrella has no visibility and cannot remediate this routing issue. We can only show we are returning a valid IP and it cannot be routed to.

Example: Domain.com is a CNAME for a1234.a.akamaiedge.net which points to the IP 1.2.3.4. Connections to 1.2.3.4 time out. When running a traceroute, we get 8 steps in, past the ISP but then cease getting returns.

Example issue and solution: After speaking with the ISP, the issue was that the ISP's IP range does not have a return route from Akamai back to the user's IP within the ISP's space. The ISP presented two options: 1) Wait for Akamai to add a route back to this IP, or 2) Change IPs within the ISP's ranges to an IP that does have a return route from Akamai that works.

Need assistance in working with your ISP? We can help explain and provide evidence with confirmation that the IP returned from the Akamai A-record is valid from Akamai through our resolvers.

Troubleshooting Tips

Do you believe this issue affects you, but you are not sure? Here are some helpful hints for troubleshooting. These steps would be required for our support team to be able to assist you. Once this information is gathered, contact support@umbrella.com for assistance.

1. Try using nslookup queries for the domain(s) which is being problematic. This example uses www.foxnews.com, which is an Akamai-powered site at the time of writing.
   • nslookup www.foxnews.com
   • nslookup www.foxnews.com 8.8.8.8
   • nslookup www.foxnews.com 208.67.222.222
   • nslookup www.foxnews.com 208.67.220.220
2. Use traceroutes: OS dependent, this can be tracert (Windows) or traceroute (OS X). In this case, you are trying to trace to the FQDN as well as the IP addresses that you got in step 1.
   • tracert www.foxnews.com
   • tracert <resultant IP of nslookup www.foxnews.com>
   • tracert <resultant IP of nslookup www.foxnews.com 8.8.8.8>
   • tracert <resultant IP of nslookup www.foxnews.com 208.67.222.222>
3. Gather a list of the domains and IPs you are not able to reach, they can share common Akamai host infrastructure being blocked by your ISP.