Company Accelerates Application Performance Troubleshooting
PDF(144.1 KB) View with Adobe Reader on a variety of devices
Updated:December 23, 2013
Boxwood Technology implemented LiveAction to visualize application flow metrics gathered by Cisco AVC and Medianet.
Headquartered in Herndon, Virginia, Boxwood Technology provides job boards and other online career center services for associations. The company is endorsed by the American Society of Association Executives and is a charter member of the International Association of Employment Websites.
Boxwood employees in two locations depend on business applications housed in a collocation facility, including a ticketing system, customer databases, Microsoft SharePoint, accounting software, and an IT logging tool. Previously, if users reported slow application performance, Boxwood's systems administrators used the "show IP nbar" command to investigate. "But the command returned limited information about only a few protocols, and interpreting the data was time-consuming," says William Bordeau, systems administrator for Boxwood Technology. "To accelerate troubleshooting and provide a good user experience, we wanted deeper insight into end-to-end application performance, and an easy-to-use graphical user interface."
Complicating the challenge, Boxwood was preparing to replace a traditional phone system with an all-IP voice system. "We knew we needed the ability to visualize voice traffic from end to end, because voice quality affects customer satisfaction when they call for sales or support," Bordeau says. The existing, third-party NetFlow traffic analyzer would not work for voice traffic monitoring, because it relied on traceroute, which only shows traffic between routers, not between switches. This meant that a portion of the traffic flow for interoffice calls, which go through the Multiprotocol Label Switching (MPLS) cloud, would be hidden. To see the end-to-end flow, Boxwood needed a router that supported Mediatrace, a Medianet technology that discovers switches as well as routers. Mediatrace is a Cisco
® Medianet technology that follows flows hop-by-hop, collecting statistics across the flow path. It leverages another Medianet technology, Performance Monitor, to collect diagnostic data for mediatrace including packet loss, jitter, hop-by-hop latency, and response time.
Ease of use was critical. "Our staff is busy and needs a simple tool that displays actionable information without requiring them to jump through hoops," Bordeau says. "We wanted an easy-to-use visualization tool that would require less detective work."
Boxwood found a comprehensive solution to visualizing application flows by using services on the Cisco Integrated Services Router Generation 2 (ISR G2) in conjunction with LiveAction software from ActionPacked! Networks, a Cisco Developer Network partner:
• Cisco Application Visibility and Control (AVC) is a suite of services in Cisco network devices that provides application-level classification, monitoring and traffic control. It uses deep-packet inspection to identify more than 1000 applications, collecting performance statistics such as bandwidth use, latency, and response time. On a recent day, Cisco AVC reported that the top 10 types of traffic by volume included HTTP, Common Internet File System (CIFS), Exchange, Active Directory, Cisco WebEx®, Simple Network Management Protocol (SNMP), and YouTube.
• Mediatrace, a medianet technology supported in Cisco ISR G2 routers, monitors voice flows from point to point, across routers as well as switches. "With mediatrace, nothing is blocked from view, including voice traffic traversing our MPLS cloud," Bordeau says.
• LiveAction provides an intuitive GUI for filtering the information from Cisco AVC and Medianet and presenting it in an easy-to-understand, visual format. If an employee reports slow network performance, Bordeau can visualize current or historical network activity on the LiveAction interface, which color-codes traffic links by volume and protocol. This capability makes it easy to see the protocol responsible for the saturation, such as CIFS for file transfers, and the originating endpoint. "LiveAction is a single interface we can use with Cisco AVC and Medianet mediatrace to see congested areas, visualize flows from end-to-end, and apply changes that LiveAction pushes to our routers," Bordeau says. "The visual representation helps us pinpoint the location of a network incident that's affecting the user experience in just minutes, compared to hours or even days before we had the tools."
Successes to-date include quickly discovering the sources of network congestion, detecting and mitigating an attack against the company's web server farm, and identifying a misconfigured ACL before it caused voice quality problems.
Accelerated Troubleshooting of Network Performance Issues
Boxwood uses Cisco AVC and LiveAction primarily for diagnostics, to identify the cause of slow performance for TCP-based applications. "We use LiveAction on-demand anytime we need to troubleshoot or diagnose network performance issues," Bordeau says. "If a user reports slow application performance, I can apply a filter in LiveAction to look at metrics from that server. Being able to visualize network activity by application, not just the port, is very valuable for us." If LiveAction shows zero retransmissions for a large file transfer, Bordeau can narrow down the problem to the server, not the network. The combination of LiveAction and Cisco AVC also makes it easy to see if slow application performance is a result of client network delay or server network delay.
Boxwood experienced the value of Cisco AVC and LiveAction soon after implementation, when email traffic slowed abruptly. Cisco AVC reported congestion in an uplink to the MPLS cloud, clearly visible on the LiveAction interface. From within LiveAction, Bordeau built a filter to look for file transfers with a few clicks, quickly identifying the source. Further investigation revealed that an employee in the sales and marketing office was backing up large, offline mailboxes.
"Bandwidth issues are transient," Bordeau says. "If you can't diagnose the issue in five to ten minutes, the moment of opportunity has usually passed, especially if you can't look at historical data. LiveAction and Cisco AVC together give us the ability to look right into the network to pinpoint the source of bandwidth saturation in minutes."
Blocked Network Attack
LiveAction and Cisco AVC also helped Boxwood's IT team detect a Structured Query Language (SQL) injection attack on company web servers. The firewall at the facility reports the source country of all connection attempts. When Bordeau used the LiveAction to view historical data, he noticed multiple connections in a country where the company does not do business, always from 2 a.m. to 3 a.m. With this information, he was able to construct a filter that blocked the traffic.
Averted Voice Quality Problem by Discovering Misconfigured ACL
When preparing for the migration to unified communications, Bordeau connected an IP phone to the network for testing purposes. Although the phone worked, LiveAction showed that the connection was going across a different interface than expected. With further investigation, Bordeau discovered that an access control list (ACL) was helping to enable calls to travel over the public Internet instead of just the MPLS VPN. "Without mediatrace and LiveAction, we might never have discovered the error, allowing some calls to travel on a path without QoS," he says. "Discovering the problem helped us protect voice quality for our employees and customers."
Today, Boxwood uses LiveAction and Cisco AVC primarily for diagnostics. Later, the company might use the alerting feature in LiveAction to gain early awareness of issues before they affect the user experience.
Bordeau concludes, "We're excited to have the tools to collect detailed information about all application flows and view it from one easy-to-use interface. And visualizing the network is helping us provide a great quality of experience by identifying and resolving issues more quickly."