First American Corporation Implements Cisco QoS Policy Manager to Protect Web-Based Financial Application
First American Corporation, with annual revenues of $2.8 billion, is setting out to transform the real estate information services market by moving title search, insurance processes, and other mortgage information to the web for customers to access directly. Because of its aggressive commitment to innovation in e-business, PC Week magazine selected First American number two in its 1999 FAST@TRACK 100 list that recognizes technology innovators, chosen from more than 260,000 corporate information technology sites.
To ensure its own 1100 branch offices nationwide would enjoy rapid response times to the company's internal, web-based title insurance application--named FAST--First American implemented Cisco quality-of-service (QoS) features on the network, using Cisco QoS Policy Manager (QPM), see below. QoS provides strong protection of FAST traffic and have been a critical part of First American's move to centralize operations and provide a common application accessible to all of its nationwide branch offices.
The Cisco QPM is a graphical user interface (GUI)- and rules-based policy manager that allows administrators to define, administer, manage, and distribute QoS policies automatically. QPM, running on Microsoft Windows NT, supports a comprehensive list of Cisco routers, Catalyst switches and software releases; and that device information can be downloaded from CiscoWorks 2000. QPM is the industry's leading, award winning, directory-enabled policy manager supporting advanced features such as Common Open Policy Service (COPS), Resource Reservation Protocol (RSVP), and Voice. For more information on QPM, see http://www.cisco.com/warp/public/cc/pd/wr2k/qoppmn/prodlit/index.shtml
In 1998, First American Corporation set out to transform real estate information services by moving title search and insurance processes online. Over the last two years, the $2.8 billion Santa Ana, California, company has rolled out a series of Web sites that give mortgage lenders quick, online access to property files, tax information, credit reporting and other data that, until now, has been locked away on paper in file cabinets.
As part of its commitment to e-business, First American began to transform how its own branch offices accessed their mortgage-based information. For many years, First American operated in a decentralized manner--each region worked independently and built their own network infrastructures and applications.
With the growth of the web, First American saw the opportunity to centralize operations. This would allow it to dramatically cut costs incurred by each region by minimizing the IT resources required to develop applications and manage the infrastructure. First American IT personnel would develop the applications, manage and maintain them from a central location; branch offices would then only need to access the applications securely over the Web.
The initial application First American rolled out is named FAST, a title policy application. First American users access the application to conduct property searches and determine if properties have any encumbrances such as tax liens or undisclosed deed holders. Once this information has been uncovered, the FAST application generates a title policy based on the information.
The challenge First American faced in rolling out the application nationwide was region resistance. Because the regions were completely dependent on their previous home-grown application, many were unsure that they could rely on both the First American network and the FAST application. Therefore, First American had to convince the regions that its network had the performance and QoS capabilities needed to ensure each branch office had continuous and reliable access to the mission-critical FAST applications.
Santa Ana, the headquarters, contains the largest local-area network (LAN) and the centralized data center. All application servers are front ended by a Catalyst 6509, as shown in Figure 1. A Cisco 7206VXR and 7206 router running 12.0(7)T performs HQ routing functions. Desktop connectivity is provided through a two-layer hierarchy of Catalyst 6500 switches. Other location infrastructure equipment includes Cisco 4000, 3600, 2600, 2500, and 1600 series routers.
First American's nationwide network is Frame Relay based and extends across the country, also illustrated in Figure 1, through core locations in Santa Ana, Seattle, Sacramento, Chicago, New York, and Houston, among others. The network is structured in a three-layer hierarchy consisting of a core, distribution and access layer. This three-tier network architecture provides First American with significant redundancy as well as higher core router performance.
Figure 1: A Portion of First American's Network, Illustrating Core, Distribution, and Access Layers. Red Circles Show Interfaces Where Classification was Implemented, While the Green Circles Show Where Congestion Management Enforcement was Implemented.
The core layer, or backbone, consists of 16-7206 series routers running IOS 12.0(7)T and 2-7206VXR series routers running IOS 12.0(7)T. The 7206VXR are deployed at the Santa Ana headquarters and Sacramento site. The backbone consists of two separate partially meshed Frame Relay clouds. Each cloud provides connectivity to half of the backbone sites with alternate "backup" private virtual circuits (PVCs) between the neighboring sites on each cloud. The primary access links to each site are full T1 circuits with 1.536Kbps CIR. The alternate "backup" PVCs are full T1 circuits with 512Kbps CIR.
The distribution routers aggregate traffic from the branch offices and then route the traffic to one of the 16 core routers. The distribution layer consists of one 7507 router running IOS 11.1(28)CC, two 7206 routers running IOS 12.0(3.4)T and numerous 3640 routers running IOS 11.3(3a)T.
The access layer is made up of a large quantity of branch offices serviced by a variety of Cisco routers. The router platforms include 4000, 3640, 3620, 2600, 2500 and 2601 series routers running a variety of IOS ranging from 11.1.(12)AA to 11.3(10)T.
The FAST Web, application and database servers are housed at the data center at First American's Santa Ana headquarters. FAST servers are front ended by a Catalyst 6509 switch, which connects to a Cisco 7206VXR router that performs the major routing functions within the headquarters.
A major decision facing First American was to determine which of many rich QoS mechanisms provided by Cisco, would best protect FAST traffic without severely limiting background traffic flows. This decision process was greatly simplified by the work of Netigy (http://www.netigy.com), a premier architect of eBusiness-ready networks and a Cisco ecosystem partner.1 "We captured live data from First American's network and ran lab simulations to test the FAST response times under congested conditions using different QoS policies," said Kevin Adams, Principal Consultant, Netigy. "We found that Custom Queuing,2 after several tests, worked best with FAST traffic, as it improved the FAST response times without adversely affecting background traffic," said Adams. Custom queuing is a flexible traffic prioritization scheme that allocates a minimum bandwidth to specified types of traffic. You can create up to 16 of these custom queues. For custom queue interfaces, the device services the queues in a round-robin fashion, sending out packets from a queue until the byte count on the queue is met, then moving on to the next queue. This ensures that no queue gets starved, in comparison to priority queuing.
"While our goal was clearly the protection of FAST traffic, we still had background traffic that was important to our operation," says Ahmad Sidani, Network Manager, at First American. "Netigy's lab simulations allowed us to analyze performance using the different options. We are very pleased with the results."
As shown in Figure 2, response times of the FAST application varied, depending on the type of queuing option used. One option was Priority Queuing and it provided the fastest response time of 27.98 seconds across the test script. With Priority Queuing, the high-priority queue is always emptied before the medium-priority queue, and so on. As a result, traffic in lower-priority queues might not be forwarded in a timely manner or be forwarded at all. Priority Queuing is recommend for short-lived high priority flows, otherwise a low priority queue can be starved out and never be allowed to transmit if there is a limited amount of available bandwidth or if the transmission frequency of the critical traffic is too high.
But while Priority Queuing provided the best response times for FASTone of First American's key considerations--it severely impacted all background traffic, as shown in Figure 3. Using Priority Queuing, background response times shot up from just over 1.36 seconds during congested conditions to 3.71 seconds. This occurred because Priority Queuing virtually starves the lower priority queues as long as FAST traffic is present.
Netigy then simulated the traffic using Custom Queuing in different modes: one with FAST traffic guaranteed 80 percent of the bandwidth, and background traffic allocated the remaining 20 percent; and one with the weightings evenly matched at 50/50. First American also looked at Weighted Fair Queuing (WFQ). Custom Queuing, with the 80/20-percent, was selected for First American as it had the best compromise between improving response time for FAST, and maintaining an acceptable response time for the background traffic. When FAST packets reach the wide-area network (WAN) router they are placed into a separate queue that provides 80 percent of the bandwidth, providing extremely high throughput.
To classify or "color" the packets, First American used the source IP address to set the IP Precedence bits of FAST traffic to "4." This option was selected because all FAST applications run on dedicated servers. Other classification options include host name, Transmission Control Protocol (TCP)/User Datagram Protocol (UDP), or any other manually entered protocol port number, and URL.
QoS classification and coloring, for example, are now implemented on First American's access routers in Chicago and Riverside and on the Catalyst 6509 switches that front-end the FAST servers at Santa Ana headquarters (see Figure 1). Custom Queuing occurs on selected ports of the 7206 core routers in Chicago, Sacramento, and Santa Ana, and on the egress port of the Cisco 4000 access router in Riverside and the Cisco 7507 router in Santa Ana.
"We've been very impressed with the power of Cisco QoS features on congested links," says Ahmad."While some of our links are not currently at full capacity, the fact that QoS is now in place prepares us for the growth we're expecting. As more and more of our branch offices add traffic onto the WAN, the network will experience congestion; we now know that the FAST application will maintain strong response times. What's more, the QoS functionality reassures our regions that they can migrate to FAST and feel confident that the performance and reliability they need will be there."
Another decision facing First American was how to implement QoS without laboriously and manually, using the Command Line Interface (CLI), configuring each router and switch. First American decided to purchase the award-winning QoS Policy Manager from Cisco. "QPM made implementing new policies easy by automating the distribution of policies to all the routers on the network," says Ahmad. "This allows us to concentrate on building more effective policies, rather than on the tedious and error-prone work of how to implement them."
As QoS policies are distributed to network devices, they are converted, by QPM, into specific classification, queuing, limiting, and shaping configuration commands, reducing the complexity of configuring a mix of QoS features across a myriad of different devices, Cisco IOS versions, and Catalyst OS releases.
Figures 4 and 5 show the QPM GUI for the classification and enforcement policies, for First American.
Figure 4: Shows Classifying FAST Traffic Based Upon Source IP Address, then Marking or Coloring FAST Packets to IP Precedence=4
Figure 5: Shows Queuing of FAST Traffic, Using Custom Queuing Setting Bandwidth to 80-Percent for FAST, for Packets that are Marked with IP Precedence=4
1Netigy has dedicated QoS Policy Implementation practice, that involves a multi-step methodology (see http://www.netigy.com/expertise/thought/whitepapers/index.html
2. CBWFQ (Class Based Weighted Fair Queuing) is another good queuing option; and is supported on IOS 12.0(5) and above. CBWFQ allows the exact specification of the bandwidth amount by traffic class, configures up to 64 classes, permits specification of queue limits, and provides options for drop mechanisms. Custom Queuing was used at First American, as many of the devices on are IOS release 11.X. More detailed information on specific QoS techniques is available at