Understanding Performance Routing

Before optimizing traffic, PfR has to determine the traffic classes from the traffic flowing through the border routers. To optimize traffic routing, subsets of the total traffic must be identified, and these traffic subsets are named traffic classes. The list of traffic classes entries is named a Monitored Traffic Class (MTC) list. The entries in the MTC list can be profiled either by automatically learning the traffic flowing through the device or by manually configuring the traffic classes. Learned and configured traffic classes can both exist in the MTC list at the same time. The PfR profile phase includes both the learn mechanism and the configure mechanism. The overall structure of the PfR traffic class profile process and its component parts can be seen in the figure below.

Figure 1

PfR Traffic Class Profiling Process

The ultimate objective of this phase is to select a subset of traffic flowing through the network. This subset of traffic--the traffic classes in the MTC list--represents the classes of traffic that need to be routed based on the best performance path available.

PfR can automatically learn the traffic classes while monitoring the traffic flow through border routers. Although the goal is to optimize a subset of the traffic, you may not know all the exact parameters of this traffic and PfR provides a method to automatically learn the traffic and create traffic classes by populating the MTC list. Several features have been added to PfR since the original release to add functionality to the automatic traffic class learning process.

Within the automatic traffic class learning process there are now three components. One component describes the automatic learning of prefix-based traffic classes, the second component describes automatic learning of application-based traffic classes, and the third component describes the use of learn lists to categorize both prefix-based and application-based traffic classes. These three components are described in the following sections:

The PfR master controller can be configured, using NetFlow Top Talker functionality, to automatically learn prefixes based on the highest outbound throughput or the highest delay time. Throughput learning measures prefixes that generate the highest outbound traffic volume. Throughput prefixes are sorted from highest to lowest. Delay learning measures prefixes with the highest round-trip response time (RTT) to optimize these highest delay prefixes to try to reduce the RTT for these prefixes. Delay prefixes are sorted from the highest to the lowest delay time.

PfR can learn Layer 3 prefixes, and Layer 4 options such as protocol or port numbers can be added as filters to the prefix-based traffic class. The protocol and port numbers can be used to identify specific application traffic classes; protocol and port number parameters are monitored only within the context of a prefix and are not sent to the master controller database (MTC list). The prefix that carries the specific traffic is then monitored by the master controller. PfR application traffic class learning also supports Differentiated Services Code Point (DSCP) values in addition to protocol and port numbers, and these Layer 4 options are entered in the MTC list.

PfR supports a learn list configuration mode to simplify the learning of traffic classes. Learn lists are a way to categorize learned traffic classes. In each learn list, different criteria including prefixes, application definitions, filters, and aggregation parameters for learning traffic classes can be configured. A traffic class is automatically learned by PfR based on each learn list criteria, and each learn list is configured with a sequence number. The sequence number determines the order in which learn list criteria are applied. Learn lists allow different PfR policies to be applied to each learn list; in previous releases, the traffic classes could not be divided, and an PfR policy was applied to all the learned traffic classes.

Learn list configuration mode uses traffic-class commands to simplify the learning of traffic classes. Four types of traffic classes--to be automatically learned--can be profiled:

• Traffic classes based on destination prefixes
• Traffic classes representing custom application definitions using access lists
• Traffic classes based on a static application mapping name with optional prefix lists to define destination prefixes
• Traffic classes based on a NBAR application mapping name with optional prefix lists to define destination prefixes

Only one type of traffic-class command can be specified per learn list, and the throughput (PfR) and delay (PfR) commands are also mutually exclusive within a learn list.

PfR can be manually configured to create traffic classes for monitoring and subsequent optimizing. Automatic learning generally uses a default prefix length of /24 but manual configuration allows exact prefixes to be defined. Within the manual traffic class configuration process there are two components-- manually configuring prefix-based traffic classes and manually configuring application-based traffic classes, both of which are described in the following sections:

A prefix or range of prefixes can be selected for PfR monitoring by configuring an IP prefix list. The IP prefix list is then imported into the MTC list by configuring a match clause in a PfR map. A PfR map is similar to an IP route map. IP prefix lists are configured with the ip prefix-list command and PfR maps are configured with the pfr-map command in global configuration mode.

The prefix list syntax operates in a slightly different way with PfR than in regular routing. The ge keyword is not used and the le keyword is used by PfR to specify only an inclusive prefix. A prefix list can also be used to specify an exact prefix.

A master controller can monitor and control an exact prefix of any length including the default route. If an exact prefix is specified, PfR monitors only the exact prefix.

A master controller can monitor and control an inclusive prefix using thele keyword and the le-value argument set to 32. PfR monitors the configured prefix and any more specific prefixes (for example, configuring the 10.0.0.0/8 le 32 prefix would include the 10.1.0.0/16 and the 10.1.1.0/24 prefixes) over the same exit and records the information in the routing information base (RIB).

Note	Use the inclusive prefix option with caution in a typical PfR deployment because of the potential increase in the amount of prefixes being monitored and recorded.

An IP prefix list with a deny statement can be used to configure the master controller to exclude a prefix or prefix length for learned traffic classes. Deny prefix list sequences should be applied in the lowest PfR map sequences for best performance. The master controller can also be configured to tell border routers to filter out uninteresting traffic using an access list.

Note	IP prefix lists with deny statements can be applied only to learned traffic classes.

PfR supports the manual configuration of Layer 3 prefixes during the PfR profile phase. Application-aware routing for policy-based routing (PBR) is also supported. Application-aware routing allows the selection of traffic for specific applications based on values in the IP packet header, other than the Layer 3 destination address through a named extended IP access control list (ACL). Only named extended ACLs are supported. The extended ACL is configured with a permit statement and then referenced in a PfR map. The protocol and port numbers can be used to identify specific application traffic classes, but protocol and port number parameters are monitored only within the context of a prefix, and are not sent to the MTC list. Only the prefix that carries the specific application traffic is profiled by the master controller. With application-aware routing support, active monitoring of application traffic was supported. Passive monitoring of application traffic is also supported. Application traffic classes can be defined using DSCP values as well as protocol and port numbers. DSCP values, port numbers, and protocols in addition to prefixes, are all now stored in the MTC list.

Learn list configuration mode uses match traffic-class commands under PfR map configuration mode to simplify the configuration of traffic classes. Four types of traffic classes--to be manually configured--can be profiled:

• Traffic classes based on destination prefixes
• Traffic classes representing custom application definitions using access lists
• Traffic classes based on a static application mapping name and a prefix list to define destination prefixes
• Traffic classes based on NBAR application mapping name and a prefix list to define destination prefixes

Only one type of match traffic-class command can be specified per PfR map.

For a series of well-known applications, static ports have been defined and each application can be defined by entering a keyword. For more details about static application mapping, see the Static Application Mapping Using Performance Routing feature.

PfR supports the ability to profile an application-based traffic class using NBAR. NBAR is a classification engine that recognizes and classifies a wide variety of protocols and applications, including web-based and other difficult-to-classify applications and protocols that use dynamic TCP/UDP port assignments. PfR uses NBAR to recognize and classify a protocol or application, and the resulting traffic classes are added to the PfR application database to be passively and actively monitored. For more details about PfR application mapping using NBAR, see the Performance Routing with NBAR/CCE Application Recognition feature.

The PfR measure phase is the second step in the PfR performance loop and it follows the PfR profile phase where the traffic class entries fill the Monitored Traffic Class (MTC) list. The MTC list is now full of traffic class entries and PfR must measure the performance metrics of these traffic class entries. Monitoring is defined here as the act of measurement performed periodically over a set interval of time where the measurements are compared against a threshold. PfR measures the performance of traffic classes using active and passive monitoring techniques but it also measures, by default, the utilization of links. The master controller can be configured to monitor learned and configured traffic classes. The border routers collect passive monitoring and active monitoring statistics and then transmit this information to the master controller. The PfR measure phase is complete when each traffic class entry in the MTC list has associated performance metric measurements.

The overall structure of the PfR measure phase and its component parts can be seen in the figure below.

Figure 2

PfR Performance Measuring Process

PfR measures the performance of both traffic classes and links, but before monitoring a traffic class or link PfR checks the state of the traffic class or link. PfR uses a policy decision point (PDP) that operates according to a traffic class state transition diagram.

After determining the state of the traffic class or link, PfR may initiate one of the following performance measuring processes.

PfR uses three methods of traffic class performance measurement:

• Passive monitoring--measuring the performance metrics of traffic class entries while the traffic is flowing through the device using NetFlow functionality.
• Active monitoring--creating a stream of synthetic traffic replicating a traffic class as closely as possible and measuring the performance metrics of the synthetic traffic. The results of the performance metrics of the synthetic traffic are applied to the traffic class in the MTC list. Active monitoring uses integrated IP Service Level Agreements (IP SLAs) functionality.
• Both active and passive monitoring--combining both active and passive monitoring in order to generate a more complete picture of traffic flows within the network.

Fast failover monitoring mode is another variation of the combined active and passive monitoring modes. In fast failover monitoring mode, all exits are continuously probed using active monitoring and passive monitoring. When fast failover monitoring mode is enabled, the probe frequency can be set to a lower frequency than for other monitoring modes, to allow a faster failover capability.

No explicit NetFlow or IP SLAs configuration is required and support for NetFlow and IP SLAs is enabled automatically. You can use both active and passive monitoring methods for a traffic class.

After the master controller is defined and PfR functionality is enabled, the master controller uses both passive and active monitoring by default. All traffic classes are passively monitored using integrated NetFlow functionality. Out-of-policy traffic classes are actively monitored using IP SLA functionality. You can configure the master controller to use only passive monitoring, active monitoring, both passive and active monitoring, or fast failover monitoring. The main differences between the different modes can be seen in the table below.

Cisco IOS PfR uses NetFlow, an integrated technology in Cisco IOS software, to collect and aggregate passive monitoring statistics on a per traffic class basis. Passive monitoring is enabled along with active monitoring by default when an PfR managed network is created. Passive monitoring can also be enabled explicitly using the mode monitor passive command. Netflow is a flow-based monitoring and accounting system, and NetFlow support is enabled by default on the border routers when passive monitoring is enabled.

Passive monitoring uses only existing traffic; additional traffic is not generated. Border routers collect and report passive monitoring statistics to the master controller approximately once per minute. If traffic does not go over an external interface of a border router, no data is reported to the master controller. Threshold comparison is done at the master controller. Passive monitoring supports traffic classes defined by prefix, port, protocol, and DSCP value.

PfR uses passive monitoring to measure the following metrics for all the traffic classes:

• Delay--PfR measures the average delay of TCP flows for a given prefix. Delay is the measurement of the round-trip response time (RTT) between the transmission of a TCP synchronization message and receipt of the TCP acknowledgement.
• Packet loss--PfR measures packet loss by tracking TCP sequence numbers for each TCP flow. PfR estimates packet loss by tracking the highest TCP sequence number. If a subsequent packet is received with a lower sequence number, PfR increments the packet loss counter. Packet loss is measured in packets per million.
• Reachability--PfR measures reachability by tracking TCP synchronization messages that have been sent repeatedly without receiving a TCP acknowledgement.
• Throughput--PfR measures throughput by measuring the total number of bytes and packets for each traffic class for a given interval of time.

Note	Although all traffic classes are monitored, delay, loss, and reachability information is captured only for TCP traffic flows. Throughput statistics are captured for all non-TCP traffic flows.

DSCP values, port numbers, and protocols in addition to prefixes, are all sent from border routers to the master controller. Passive monitoring statistics are gathered and stored in a prefix history buffer that can hold a minimum of 60 minutes of information depending on whether the traffic flow is continuous. PfR uses this information to determine if the prefix is in-policy based on the default or user-defined policies. No alternative path analysis is performed as the traffic for a traffic class is flowing through one transit device in the network. If the traffic class goes OOP and only passive monitoring mode is enabled, the traffic class is moved to another point and the measurement repeated until a good or best exit is found. If the traffic class goes OOP and both passive and active monitoring modes are enabled, active probing is executed on all the exits and a best or good exit is selected.

If PfR passive monitoring techniques create too much overhead on a network device, or the performance metrics of a traffic class cannot be measured using the PfR passive monitoring mode, then PfR active monitoring techniques are performed. Active monitoring involves creating a stream of synthetic traffic that replicates a traffic class as closely as possible. The performance metrics of the synthetic traffic are measured and the results are applied to the traffic class entry in the MTC list. Active monitoring supports traffic classes defined by prefix, port, protocol, and DSCP value.

PfR uses active monitoring to measure the following metrics for all the traffic classes:

• Delay--PfR measures the average delay of TCP, UDP, and ICMP flows for a given prefix. Delay is the measurement of the round-trip response time (RTT) between the transmission of a TCP synchronization message and receipt of the TCP acknowledgement.
• Reachability--PfR measures reachability by tracking TCP synchronization messages that have been sent repeatedly without receiving a TCP acknowledgement.
• Jitter--Jitter means interpacket delay variance. PfR measures jitter by sending multiple packets to a target address and a specified target port number, and measuring the delay interval between packets arriving at the destination.
• MOS--Mean Opinion Score (MOS) is a standards-based method of measuring voice quality. Standards bodies like the ITU have derived two important recommendations: P.800 (MOS) and P.861 (Perceptual Speech Quality Measurement [PSQM]). P.800 is concerned with defining a method to derive a Mean Opinion Score of voice quality. MOS scores range between 1 representing the worst voice quality, and 5 representing the best voice quality. A MOS of 4 is considered "toll-quality" voice.

The creation of synthetic traffic in Cisco network devices is activated through the use of Cisco IOS IP SLA probes. PfR is integrated with IP SLAs functionality such that PfR will use IP SLA probes to actively monitor a traffic class. When active monitoring is enabled, the master controller commands the border routers to send active probes to set of target IP addresses. The border sends probe packets to no more than five target host addresses per traffic class, and transmits the probe results to the master controller for analysis.

Active probe monitoring periods are defined as short-term which consists of the last 5 probe results, and long-term which consists of the last 60 probe results.

Cisco IOS PfR can also be configured to combine both active and passive monitoring in order to generate a more complete picture of traffic flows within the network. There are some scenarios in which you may want to combine both PfR monitoring modes.

One example scenario is when you want to learn traffic classes and then monitor them passively, but you also want to determine the alternate path performance metrics in order to control the traffic classes. The alternate path performance metrics, in the absence of the actual traffic flowing through the alternate path in the network, can be measured using the active probes. PfR automates this process by learning traffic classes at five targets and probing through all the alternate paths using active probes.

Fast monitoring sets the active probes to continuously monitor all the exits (probe-all), and passive monitoring is enabled too. Fast failover monitoring can be used with all types of active probes: ICMP echo, Jitter, TCP connection, and UDP echo. When the mode monitor fast command is enabled, the probe frequency can be set to a lower frequency than for other monitoring modes, to allow a faster failover ability. Under fast monitoring with a lower probe frequency, route changes can be performed within 3 seconds of an out-of-policy situation. When an exit becomes OOP under fast monitoring, the select best exit is operational and the routes from the OOP exit are moved to the best in-policy exit. Fast monitoring is a very aggressive mode that incurs a lot of overhead with the continuous probing. We recommend that you use fast monitoring only for performance sensitive traffic. For example, a voice call is very sensitive to any performance problems or congested links, but the ability to detect and reroute the call within a few seconds can demonstrate the value of using fast monitoring mode.

Note	In fast monitoring mode, probe targets are learned as well as learned prefixes. To avoid triggering large numbers of probes in the network, use fast monitoring mode only for real time applications and critical applications with performance sensitive traffic.

The PfR Border Router Only feature introduced the ability to run PfR on some hardware platforms that do not support the master controller functions but do allow these platforms to operate as border routers with limited functionality. The master controller that communicates with the Cisco Catalyst 6500 series switch or a Cisco 7600 series router being used as a border router must be a router running Cisco IOS Release 12.4(6)T or a later release.

A special monitoring mode was introduced, mode monitor special, as an alternate syntax to mode monitor both for the hardware platforms that do not support the master controller functions. The special mode is set globally and cannot be configured using the command-line interface (CLI). On hardware platforms that do not support the master controller functions, PfR cannot determine performance characteristics such as delay, loss, or reachability from passive monitoring of TCP flows; PfR can only measure passive throughput. Throughput-based load balancing is still supported and active probing is enabled to accommodate the passive monitoring limitations and this allows active IP SLA measurements. The master controller automatically detects the limited capabilities of the Cisco Catalyst 6500 series switch or a Cisco 7600 series router and downgrades other border routers to capture only the throughput statistics for traffic classes. By ignoring other types of statistics, the master controller is presented with a uniform view of the border router functionality.

Note	The output from the show oer master prefix command lists an # next to the prefix with mode monitor special.

The PfR apply policy phase is the third step in the PfR performance loop following after the profile phase that identifies the traffic classes, and the measure phase where each traffic class entry in the MTC list is monitored to determine performance metric measurements. The apply policy phase compares the measured performance metrics against well-known or configured thresholds to determine if the traffic is meeting specified levels of service, or if some action is required. If the performance metric does not conform to the threshold, a decision is made by PfR to move the traffic class or exit into another state.

An PfR policy is a rule that defines an objective and contains the following attributes:

• A scope.
• An action.
• A triggering event or condition.

For example, a policy can be configured to maintain a delay of less than or equal to 100 milliseconds for packets sent to a specific traffic class entry. The scope is the network traffic sent to the specific traffic class entry, the action is a routing table change, and the triggering event is a measured delay of greater than 100 milliseconds for this traffic. The action may be not be executed until PfR is configured to control the traffic in the PfR control phase. By default, PfR runs in an observe mode during the profile, measure, and apply policy phases.

In the PfR apply policy phase you can configure and apply policies. Different types of PfR policies can be configured--see the figure below--and specific PfR parameters and options can be included within a policy. In this document, a parameter is a configurable element that can be fine-tuned, and an option is a configurable element that is either enabled or disabled. After an PfR policy is configured, the policy can be applied to learned traffic classes or configured traffic classes. PfR policies can be applied globally--to all the traffic classes--or to just a specific set of traffic classes.

Figure 4

PfR Apply Policy Phase Structure

In the figure above you can see that there are three types of PfR policies plus some operational options and parameters that can be configured. Use the following links to review more information about each policy type, parameter, or option:

After an PfR policy is configured, you can see from the figure above that a policy can be applied to learned traffic classes or configured traffic classes on a global basis for all traffic classes or for a specific set of traffic classes.

When configuring multiple policy parameters for traffic classes, it is possible to have multiple overlapping policies. To resolve the potential conflict of which policy to run, PfR uses its resolve function: a flexible mechanism that allows you to set the priority for most of the policy types.

When running an PfR policy that compares the traffic class performance metrics with default or configured thresholds, a traffic class may change state. PfR uses a policy decision point (PDP) that operates according to the traffic class state transition diagram shown in the figure below. The state transition diagram below contains the following states:

• Default--A traffic class is placed in the default state when it is not under PfR control. Traffic classes are placed in the default state when they are initially added to the central policy database, the MTC. A traffic class will transition into and out of the default state depending on performance measurements, timers, and policy configuration.
• Choose Exit--This is a temporary state in which the PDP compares the current state of the traffic class against its policy settings and chooses the optimal exit for the traffic class. PfR will try to keep a traffic class flowing through its current exit but, as in the default state, performance measurements, timers, and policy configurations can cause the master controller to place a traffic class in this state for the duration of the exit link selection process. The traffic class remains in the choose exit state until it is moved to the new exit.
• Holddown--A traffic class is placed in the holddown state when the master controller requests a border router to forward the traffic class to be monitored using probes. Measurements are collected for the selected traffic class until the holddown timer expires unless the exit used by this traffic class is declared unreachable. If the exit is unreachable, the traffic class transitions back to the choose exit state.

Figure 5

PfR Traffic Class State Transition Diagram

• In-Policy--After performance measurements are compared against default or user-defined policy settings and an exit selection is made, the traffic class enters an in-policy state. When a traffic class is in the in-policy state, the traffic class is forwarded through an exit that satisfies the default or user-defined settings. The master controller continues to monitor the traffic class, but no action is taken until the periodic timer expires, or an out-of-policy message is received from a measurement collector, when the traffic class transitions back to the choose exit state.

Note	When observe mode is running, a prefix goes into an in-policy state only if the exit selected for that prefix is the current exit.

• Out-of-Policy (OOP)--A traffic class is placed in this state when there are no exits through which to forward the traffic class that conform to default or user-defined policies. While the traffic class is in this state, the backoff timer controls exiting from this state. Each time the traffic class enters this state, the amount of time the traffic class spends in this state increases. The timer is reset for a traffic class when the traffic class enters an in-policy state. If all exit links are out-of-policy, the master controller may select the best available exit.

PfR traffic class performance policies are a set of rules that govern performance characteristics for traffic classes that can be network addresses (prefixes) or application criteria such as protocol, port number, or DSCP value. Network addresses can refer to individual endpoints within a network (e.g. 10.1.1.1/32) or to entire subnets (e.g. 10.0.0.0/8). The major performance characteristics that can be managed within an PfR policy are:

With the exception of reachability, none of these performance characteristics can be managed within the constructs of conventional routing protocol metrics. Cisco PfR extends the concept of reachability (beyond ensuring that a particular route exists in the routing table) by automatically verifying that the destination can be reached through the indicated path. Using Cisco PfR provides the network administrator with a new and powerful toolset for managing the flow of traffic.

PfR link policies are a set of rules that are applied against PfR-managed external link (an external link is an interface on a border router on the network edge). Link policies define the desired performance characteristics of the links. Instead of defining the performance of an individual traffic class entry that uses the link (as in traffic class performance policies), link policies are concerned with the performance of the link as a whole. Link policies can be applied to exit (egress) links and entrance (ingress) links. The following performance characteristics are managed by link policies:

• Traffic Load (Utilization)
• Range
• Cost

Traffic Load

A traffic load (also referred to as utilization) policy consists of an upper threshold on the amount of traffic that a specific link can carry. Cisco IOS PfR supports per traffic class load distribution. Every 20 seconds, by default, the border router reports the link utilization to the master controller, after an external interface is configured for a border router. Exit link and entrance link traffic load thresholds can be configured as an PfR policy. If the exit or entrance link utilization is above the configured threshold, or the default threshold of 75-percent, the exit or entrance link is in an OOP state and PfR starts the monitoring process to find an alternative link for the traffic class. The link utilization threshold can be manually configured either as an absolute value in kilobits per second (kbps) or as a percentage. A load utilization policy for an individual interface is configured on the master controller under the border router configuration.

Tip

When configuring load distribution, we recommend that you set the interface load calculation on external interfaces to 30-second intervals with the load-interval (PfR) interface configuration command. The default calculation interval is 300 seconds. The load calculation is configured under interface configuration mode on the border router. This configuration is not required, but it is recommended to allow Cisco IOS PfR to respond as quickly as possible to load distribution issues.

Range

A range policy is defined to maintain all links within a certain utilization range, relative to each other in order to ensure that the traffic load is distributed. For example, if a network has multiple exit links, and there is no financial reason to choose one link over another, the optimal choice is to provide an even load distribution across all links. The load-sharing provided by traditional routing protocols is not always evenly distributed, because the load-sharing is flow-based rather than performance- or policy-based. Cisco PfR range functionality allows you to configure PfR to maintain the traffic utilization on a set of links within a certain percentage range of each other. If the difference between the links becomes too great, PfR will attempt to bring the link back to an in-policy state by distributing traffic classes among the available links. The master controller sets the maximum range utilization to 20 percent for all PfR-managed links by default, but the utilization range can be configured using a maximum percentage value. Exit link and entrance link utilization ranges can be configured as a PfR policy.

Note	If you are configuring link grouping, configure the no max-range-utilization command because using a link utilization range is not compatible with using a preferred or fallback set of exit links configured for link grouping.

In the Performance Routing - Link Groups feature, the ability to define a group of exit links as a preferred set of links, or a fallback set of links for PfR to use when optimizing traffic classes specified in an PfR policy, was introduced. PfR currently selects the best link for a traffic class based on the preferences specified in a policy and the traffic class performance--using parameters such as reachability, delay, loss, jitter or MOS--on a path out of the specified link. Bandwidth utilization, cost, and the range of links can also be considered in selecting the best link. Link grouping introduces a method of specifying preferred links for one or more traffic classes in an PfR policy so that the traffic classes are routed through the best link from a list of preferred links, referred to as the primary link group. A fallback link group can also be specified in case there are no links in the primary group that satisfy the specified policy and performance requirements. If no primary group links are available, the traffic classes are routed through the best link from the fallback group. To identify the best exit, PfR probes links from both the primary and fallback groups.

Note	If you are configuring link grouping, configure the no max-range-utilization command because using a link utilization range is not compatible with using a preferred or fallback set of exit links configured for link grouping.

For more details about PfR link grouping, see the "Performance Routing Link Groups" module.

The ability to configure network security policies either to prevent unauthorized use of the network or to mitigate attacks inside and outside the network is provided by PfR. You can configure PfR to use black hole or sinkhole routing techniques to limit the impact of attacks against your network. Black hole routing refers to the process of forwarding packets to a null interface, meaning that the packets are dropped into a "black hole." Sinkhole routing directs packets to a next hop where the packets can be stored, analyzed, or dropped. Another term for sinkhole routing is honey-pot routing.

In addition to the specific types of PfR policies, there are some PfR policy operational parameters or options that can be configured. The operational parameters are timers and the operational options consist of different operational modes. For more details, see the following sections:

• PfR Timers Parameters
• PfR Mode Options

PfR policies can be applied to learned or configured traffic classes. PfR policies can be applied on a global basis when the policy is configured directly under PfR master controller configuration mode. All traffic classes inherit global policies. If, however, you want to apply a policy to a subset of the traffic classes, then a specific policy can be configured. A specific PfR policy applies only to the specific traffic classes that match a prefix list or access list. Specific policies inherit global policies unless the same policy is overwritten by the specific policy. PfR policies can apply to prefixes alone, or PfR policies can apply to traffic classes that define an application traffic class and may include prefixes, protocols, port numbers, and DSCP values. To apply specific policies to learned or configured traffic classes, PfR map configuration is used.

When configuring multiple policy criteria for a single traffic class entry, or a set of traffic classes, it is possible to have multiple overlapping policies. To resolve the potential conflict of which policy to run, PfR uses its resolve function: a flexible mechanism that allows you to set the priority for a PfR policy. Each policy is assigned a unique value, and the policy with the lowest value is selected as the highest priority policy. By default, PfR assigns the highest priority to delay policies, followed by utilization policies. Assigning a priority value to any policy will override the default settings. To configure the policy conflict resolution, use the resolve (PfR) command in PfR master controller configuration mode, or the set resolve (PfR) command in PfR map configuration mode.

Variance Setting for PfR Policy Conflict Resolution

When configuring PfR resolve settings, you can also set an allowable variance for the defined policy. Variance configures the average delay, as a percentage, that all traffic classes for one exit, or the specific policy traffic classes for an exit, can vary from the defined policy value and still be considered equivalent. For example, if the delay on the best exit link (best exit in terms of delay) for a traffic class entry is 80 milliseconds (ms) and a 10 percent variance is configured, then any other exit links with a delay between 80 and 88 ms for the same traffic class entry are considered equivalent to the best exit link.

To illustrate how variance is used by PfR consider three exit links with the following performance values for delay and jitter for a traffic class entry:

• Exit A--Delay is 80 ms, jitter is 3ms
• Exit B--Delay is 85 ms, jitter is 1ms
• Exit C--Delay is 100 ms, jitter is 5ms

The following PfR policy conflict resolution is configured and applied to the traffic class entry:

delay priority 1 variance 10
jitter priority 2 variance 10

PfR determines the best exit by looking at the policy with the lowest priority value, which in this example is the delay policy. Exit A has the lowest delay value, but Exit B has a delay value of 85 which is within a 10-percent variance of the delay value at Exit A. Exit A and Exit B can therefore be considered equal in terms of delay values. Exit C is now eliminated because the delay values are too high. The next priority policy is jitter, and Exit B has the lowest jitter value. PfR will select Exit B as the only best exit for the traffic class entry because Exit A has a jitter value that is not within 10-percent variance of the Exit B jitter value.

Note	Variance cannot be configured for cost or range policies.

After profiling the traffic classes during the PfR learn phase, measuring the performance metrics of the traffic classes during the measure phase, and using network policies to map the measured performance metrics of traffic class entries in the Monitored Traffic Class (MTC) list against well-known or configured thresholds to determine if the traffic is meeting specified levels of service in the policy phase, the next step in the PfR performance loop is the PfR enforce phase.

PfR, by default, operates in an observation mode and the documentation for the PfR learn, measure, and apply policy phases assumes that PfR is in the observe mode. In observe mode, the master controller monitors traffic classes and exit links based on default and user-defined policies and then reports the status of the network including out-of-policy (OOP) events and the decisions that should be made, but does not implement any changes. The PfR enforce phase operates in control mode, not observe mode, and control mode must be explicitly configured using the mode route control command. In control mode, the master controller coordinates information from the borders routers in the same way as observe mode, but commands are sent back to the border routers to alter routing in the PfR managed network to implement the policy decisions.

PfR initiates route changes when one of the following occurs:

• A traffic class goes OOP.
• An exit link goes OOP.
• The periodic timer expires and the select exit mode is configured as select best mode.

During the PfR enforce phase, the master controller continues to monitor in-policy traffic classes that conform to the desired performance characteristics, to ensure that they remain in-policy. Changes are only implemented for OOP traffic classes and exits in order to bring them in-policy. To achieve the desired level of performance in your network, you must be aware of the configuration options that can affect the policy decisions made by the master controller.

Another configuration issue to consider when deploying PfR is that if aggressive delay or loss policies are defined, and the exit links are also seriously over-subscribed, it is possible that PfR will find it impossible to bring a traffic class in-policy. In this case, the master controller will either choose the link that most closely conforms to the performance policy, even though the traffic class still remains OOP, or it will remove the prefix from PfR control. PfR is designed to allow you to make the best use of available bandwidth, but it does not solve the problem of over-subscribed bandwidth.

After PfR control mode is enabled, and configuration options are considered, the next step is to review the traffic class control techniques employed by PfR.

After the PfR master controller has determined that it needs to take some action involving an OOP traffic class or exit link, there are a number of techniques that can be used to alter the routing metrics, alter BGP attributes, or introduce policy-based routing using a route map to influence traffic to use a different link. If the traffic associated with the traffic class is defined only by a prefix then a traditional routing control mechanism such as introducing a BGP route or a static route can be deployed. This control is network wide after redistribution because a prefix introduced into the routing protocol with a better metric will attract traffic for that prefix towards a border router. If the traffic associated with the traffic class is defined by a prefix and other matching criteria for the packet (application traffic, for example), then traditional routing cannot be employed to control the application traffic. In this situation, the control becomes device specific and not network specific. This device specific control is implemented by PfR using policy-based routing (PBR) functionality. If the traffic in this scenario has to be routed out to a different device, the remote border router should be a single hop away or a tunnel interface that makes the remote border router look like a single hop.

The figure below shows the various traffic class control techniques grouped by exit or entrance link selection.

Figure 6

Controlling Traffic Class Techniques

Before introducing the exit link selection control techniques you need to understand one principle about load balancing with Performance Routing as it applies to exit selection. PfR does not treat a more specific route as a parent route unless you configure the more specific route as a default route.

When searching for a parent route, the software tries to find the most specific route that includes the specified prefix and verifies that it points to the expected exit. If there are two or more static routes that are more specific, each route is inspected for the expected exit. If the expected exit is found, the probe is created.

In the configuration where:

ip route 10.4.0.0 255.255.0.0 172.17.40.2
ip route 0.0.0.0 0.0.0.0 serial 6/0

Probes for prefix 10.4.1.0/24 and target 10.4.1.1 will not be created over the exit using serial interface 6/0 because the most specific route inclusive of 10.4.1.1 is the exit to 172.17.40.2. If you are looking to load balance the traffic over both exits, the answer is to create a default route of the more specific route. For example:

ip route 10.4.0.0 255.255.0.0 172.17.40.2
ip route 10.4.0.0 255.255.0.0 serial 6/0

Or

ip route 0.0.0.0 0.0.0.0 serial 6/0
ip route 0.0.0.0 0.0.0.0 172.17.40.2

In the modified configuration, two probes are created, one for the exit to 172.17.40.2 and one for the exit using serial interface 6/0.

To enforce an exit link selection, PfR offers the following methods:

BGP Control Techniques

PfR uses two BGP techniques to enforce the best exit path; injecting a BGP route, or modifying the BGP local preference attribute.

If the traffic associated with the traffic class is defined only by a prefix, the master controller can instruct a border router to inject a BGP route into the BGP table to influence traffic to use a different link. All PfR injected routes remain local to an autonomous system, and these injected routes are never shared with external BGP peers. As a safeguard to ensure this behavior, when PfR injects a BGP route, it will set the no-export community on it. This is done automatically, and does not require any user configuration. However, because these routes now have a special marking, some extra configuration is required to allow the information to be shared with internal BGP peers. For each iBGP peer, the send community configuration must be specified. Although the border routers know about the best exit for the injected route, it may also be necessary to redistribute this information further into the network.

PfR also uses BGP local preference to control traffic classes. BGP local preference (Local_Pref) is a discretionary attribute applied to a BGP prefix to specify the degree of preference for that route during route selection. The Local_Pref is a value applied to a BGP prefix, and a higher Local_Pref value causes a route to be preferred over an equivalent route. The master controller instructs one of the border routers to apply the Local_Pref attribute to a prefix or set of prefixes associated with a traffic class. The border router then shares the Local_Pref value with all of its internal BGP peers. Local_Pref is a locally significant value within an autonomous system, but it is never shared with external BGP peers. Once the iBGP reconvergence is complete, the router with the highest Local_Pref for the prefix will become the exit link from the network.

Note	If a local preference value of 5000 or higher has been configured for default BGP routing, you should configure a higher BGP local preference value in PfR using the mode (PfR) command.

The PfR BGP inbound optimization feature introduced the ability to influence inbound traffic. A network advertises reachability of its inside prefixes to the Internet using eBGP advertisements to its ISPs. If the same prefix is advertised to more than one ISP, then the network is multihoming. PfR BGP inbound optimization works best with multihomed networks, but it can also be used with a network that has multiple connections to the same ISP. To implement BGP inbound optimization, PfR manipulates eBGP advertisements to influence the best entrance selection for traffic bound for inside prefixes. The benefit of implementing the best entrance selection is limited to a network that has more than one ISP connection.

For more details about PfR entrance link selection control techniques, see the "BGP Inbound Optimization Using Performance Routing" module.

The last phase of the PfR performance loop is to verify that the actions taken during the PfR control phase control actually change the flow of traffic and that the performance of the traffic class or link does move to an in-policy state. PfR uses NetFlow to automatically verify the route control. The master controller expects a Netflow update for the traffic class from the new link interface and ignores Netflow updates from the previous path. If a Netflow update does not appear after two minutes, the master controller moves the traffic class into the default state. A traffic class is placed in the default state when it is not under PfR control.

In addition to the NetFlow verification used by PfR, there are two other methods you can use to verify that PfR has initiated changes in the network:

• Syslog report--The logging command can be configured to notify you of all the main PfR state changes, and a syslog report can be run to confirm that PfR changes have occurred. The master controller is expecting bidirectional traffic, and a syslog report delimited for the specified prefix associated with the traffic class can confirm this.
• PfR show commands--PfR show commands can be used to verify that network changes have occurred and that traffic classes are in-policy. Use the show pfr master prefix command to display the status of monitored prefixes. The output from this command includes information about the current exit interface, prefix delay, egress and ingress interface bandwidth, and path information sourced from a specified border router. Use the show pfr border routes command to display information about PfR controlled routes on a border router. This command can display information about BGP or static routes.