Use Local Congestion Mitigation (LCM) to Mitigate Network Congestion Locally

Note

Functionality described within this section is only available as part of the Advanced RTM license package.
Throughout this section, the navigation is documented as Traffic Engineering > Traffic Engineering. However, when using Crosswork Optimization Engine within the Crosswork Network Controller solution, the navigation is Traffic Engineering & Services > Traffic Engineering.

Local Congestion Mitigation Overview

Local Congestion Mitigation (LCM) searches for congestion on a configurable cadence (as opposed to a triggered event) and provides localized mitigation recommendations in surrounding interfaces (local interface-level optimization) within a domain. LCM computes the shortest paths for one or more tactical policies to divert the minimal amount of traffic on a congested interface to alternate paths with sufficient bandwidth. It attempts to keep as much of the traffic on the original IGP path. If the user approves, LCM performs the mitigation through the deployment of Tactical Traffic Engineering (TTE) SR policies. LCM will not modify paths of existing deployments of SR policies to mitigate congestion. With LCM, you are able to do the following:

Visually preview LCM recommendations on your network before you decide whether to commit the Tactical Traffic Engineering (TTE) SR policy deployment.
Enable LCM to deploy changes in the network automatically to address congestion and network failures based on LCM solution configurations. For more information, see the advanced configuration options (Auto Repair Solution and Adjacency Hop Type) in Configure LCM.

LCM allows for a wider applicability of the solution in various network topologies such as that involving multiple IGP areas due to its simpler path computation and limitation to specific network elements. Focusing on the problem locally within a domain eliminates the need for simulating edge-to-edge traffic flows in the network through a full traffic matrix and allows for better scaling of large networks. Also, LCM performs the collection of TTE SR policy and interface counters via SNMP and does not require the use of SR-TM.

TTE tunnel recommendations are listed in the LCM Operational Dashboard. From the dashboard, you can visually preview the TTE SR policy recommendations before deployment. TTE SR policy deployment to resolve congestion is not automated. You must approve and commit LCM recommended actions. LCM also recommends removal of previous TTE SR policies (instantiated by LCM) if they are no longer needed.

LCM Important Notes

Consider the following information when using LCM:

You must have the Advanced RTM license package to use LCM.
You cannot enable LCM if Bandwidth Optimization is enabled.
LCM supports domains with up to 2000 devices. A domain is an identifier assigned to an IGP process. Domains are learned from the network. The domain ID is taken from router configuration (link-state instance-id) that you use to advertise IGP with BGP-LS.
LCM recommended solutions use the resources within a single domain only.
LCM evaluates network utilization on a regular, configurable cadence of 5 minutes or more. The cadence is typically set to be greater than or equal to the SNMP traffic polling interval. The default cadence is 10 minutes.
LCM leverages ECMP across parallel TTE SR policies and assumes roughly equal splitting of traffic. The degree to which actual ECMP splitting adheres to this assumption depends on the presence of large elephant flows and the level traffic aggregation.
Traffic that can be optimized must not be carried on existing SR-TE policies.
When domain interfaces and links are removed (intentionally or unintentionally), the following occurs:
- As links go down (LINK_DOWN state), LCM configuration and the Domain UI card (see Configure LCM) will remain available until the links are aged out (after 4 hours). This behavior is intentional as it gives you time to recover domain interfaces and links if this was done by mistake.
- If you want to force domain removal before links age out, then you can remove links manually from the UI. The domain will remain in a "ready for deletion" status until the last link is removed.

LCM Platform Requirements

The following is a non-exhaustive list of high-level requirements for proper LCM operation:

Congestion Evaluation:

LCM requires traffic statistics from the following:
- SNMP interface traffic measurements
- SNMP headend SR-TE policy traffic measurements
Strict SID labels should be configured for SR.

Congestion Mitigation:

The headend device should support Equal Cost Multi-Path (ECMP) across multiple parallel SR-TE policies
The headend device must support PCE-initiated SR-TE policies with autoroute steering

Devices should be configured with force-sr-include to enable traffic steering into SR-TE policies with autoroute. For example:
```
segment-routing traffic-eng pcc profile <id> autoroute force-sr-include
```
See SR configuration documentation for your specific device to view descriptions and supported configuration commands (for example: Segment Routing Configuration Guide for Cisco ASR 9000 Series Routers)

Contact your Cisco sales representative for an exhaustive list of platform requirements.

BGP-LS Speaker Placement for Multiple AS Networks with a Dedicated IGP Instance Between ASBRs

To support interdomain latency-optimized SR policy path computation by an SR-PCE (or other use cases where egress peer engineering (EPE) is not supported), a dedicated IGP instance may be configured between autonomous system border routers (ASBRs) in different ASNs. In these cases, it is important to identify which ASBRs report the topology via BGP-LS for proper topology discovery.

In the following example, at least one ASBR in each AS participating in the dedicated inter-AS IGP (Domain 100) must have BGP-LS enabled to report the IGP between each ASBR. Each ASBR must report the domain with the same BGP-LS identifier.

Note

More than one ASBR per AS reporting the BGP-LS topology is also supported.

Figure 1. BGP-LS Session Reporting Domain 100

Previously Monitored Interfaces Missing After Upgrade

If a hostname change occurred in Crosswork Optimization Engine 2.0 and that system's backup is used to upgrade to the Crosswork Optimization Engine 3.0, any previously monitored LCM interface that belongs to the node with the updated hostname will not be migrated. Instead, these interfaces are dropped from the list of LCM monitored interfaces with a warning in the LCM logs. To workaround this, you can manually add the interfaces for LCM to monitor after the migration is complete and the system is stable.

LCM Calculation Workflow

This example walks you from congestion detection to the calculations LCM performs prior to recommending tactical tunnel deployment. With the release of Crosswork Optimization Engine 3.0, these calculations are done on a per domain basis which allows better scalability and faster calculation for larger networks.

Figure 2. LCM Configuration Workflow Example

Procedure

Step 1	LCM first analyzes the Optimization Engine Model (a realtime topology and traffic representation of the physical network) on a regular cadence.
Step 2	In this example, after a congestion check interval, LCM detects congestion when Node 2 utilization goes above the 70% utilization threshold.
Step 3	LCM calculates how much traffic is eligible to divert. LCM only diverts traffic that is not already routed on an existing SR policy (for example: unlabeled, IGP routed, or carried via FlexAlgo-0 SIDs). The traffic within an SR-TE policy will not be included in LCM calculation and will continue to travel over the original programmed path. Eligible traffic is computed by taking the interface traffic statistics that account for all traffic on the interface and subtracting the sum of traffic statistics for all SR-TE policies that flow over the interface. Total interface traffic – SR policy traffic = Eligible traffic that can be optimized This process must account for any ECMP splitting of SR policies to ensure the proper accounting of SR policy traffic. In this example, the total traffic on congested Node 2 is 800 Mbps. The total traffic of all SR policies routed over Node 2 is 500 Mbps. The total traffic that LCM can divert in this example is 300 Mbps: 800 Mbps – 500 Mbps = 300 Mbps
Step 4	LCM calculates the amount that must be sent over alternate paths by subtracting the threshold equivalent traffic from the total traffic on the interface. In this example, the amount to be diverted is 100Mbps: 800 Mbps – 700 Mbps (70% threshold) = 100 Mbps LCM must route 100 Mbps of 300 Mbps (eligible traffic) to another path. Note that if the Over-provisioning Factor (OPF) percentage is set to 10, then LCM must route 110 (100 Mbps x 1.10) of the eligible traffic. The OPF can be set in the Advanced tab within the LCM Configuration window. For more information, see Configure LCM.
Step 5	LCM determines how many TTE SR policies are needed and their paths. The ratio of how much LCM eligible traffic can stay on the shortest path to the amount that must be detoured, will determine the number of TTE SR policies that are needed on the shortest versus alternate paths, respectively. In this example, LCM needs to divert one-third of the total eligible traffic (100Mbps out of 300Mbps) away from the congested link. Assuming a perfect ECMP, LCM estimates that three tactical SR-TE policies are required to create this traffic split: one tactical SR-TE policy will take the diversion path and two tactical SR-TE policies will take the original path. There is sufficient capacity in the path between Node 2 and Node 4. Therefore, LCM recommends three TTE SR policies (each expected to route approximately 100Mbps ) to be deployed from Node 2 to Node 3 via SR-PCE: 2 TTE SR policies to take a direct path to Node 3 (200 Mbps) 1 TTE SR policy takes hop via Node 4 (100 Mbps) These recommendations will be listed in the LCM Operational Dashboard.
Step 6	Assuming you deploy these TTE SR policies, LCM continues to monitor the deployed TTE policies and will recommend modifications or deletions as needed in the LCM Operational Dashboard. TTE SR policy removal recommendations will occur if the mitigated interface would not be congested if these policies were removed (minus a hold margin). This helps to avoid unnecessary TTE SR policy churn throughout the LCM operation.

Mitigate Congestion on Local Interfaces Example

In this example, we will enable LCM and observe the congestion mitigation recommendations to deploy TTE SR policies when utilization on a device's interface surpasses a defined utilization threshold. We will preview the recommended TTE SR policies before committing them to mitigate the congestion. This example demonstrates the following workflow:

View uncongested topology.
Enable and configure LCM.
After LCM detects congestion, view LCM recommendations on the Operational Dashboard.
Preview the LCM TTE policies visually on the topology map.
Commit and deploy all LCM TTE policy recommendations to mitigate the congestion.
Verify that the LCM TTE policies have been deployed.

Note

The utilization thresholds used in this example are extremely low (12%) and are best used for lab environments.

The following image shows the topology that will be used for this example.

Procedure

Step 1

View initial topology and utilization prior to LCM configuration.

Click on the link between P4-NCS5501 and P1-ASR9k to view link details. Note that utilization on P4-NCS5501 is 11.57% .

Figure 4. Initial Utilization

Step 2

Enable LCM and configure the global and custom interface utilization thresholds.

From the main menu, choose Traffic Engineering > Local Congestion Mitigation > Domain-ID > > Configuration. In this example, the threshold is set at 12% and the Interfaces to Monitor > Selected Interfaces option is selected. For more information, see Configure LCM.

Figure 5. LCM Configuration Page
Click Commit Changes to save your configuration. After committing the configuration changes, LCM will display recommendations on the LCM Operational Dashboard if congestion occurs on any monitored interfaces. LCM will not commit or deploy new TTE policies automatically. Later, you will be able to preview the recommended TTE policies and decide whether or not to commit and deploy them onto your network.

If you selected Interfaces to Monitor > Selected Interfaces, go to the Link Management page (Traffic Engineering > Local Congestion Mitigation > Domain-ID > > Link Management) and upload the list of interfaces with custom utilization thresholds. Only the interfaces listed on this page will be monitored for congestion. See the following example.

Note

If Interfaces to Monitor > All Interfaces was selected then LCM would monitor all interfaces. This includes any individual thresholds that are imported to the Link Management page. The rest of the interfaces will be monitored using the global Utilization Threshold defined in the Configuration page.

Step 3

After some time, congestion occurs surpassing the configured LCM threshold. In this example, utilization surpassed 12%.

Step 4

View TTE SR policy recommendations in the LCM Operational Dashboard.

Navigate to Traffic Engineering > Local Congestion Mitigation. When congestion is detected a domain will have a timestamp of recommended actions.
(Optional) Click > Events tab to view the new event. You can also monitor this window to view LCM events as they occur. You should see events for LCM recommendations, commit actions, and any exceptions. Below is an example of LCM events:
Open the Operational Dashboard (Traffic Engineering > Local Congestion Mitigation > Domain-ID > > Operational Dashboard).

The dashboard shows that the utilization has surpassed 12% and is at 13.01%. In the Recommended Action column, there is a recommendation to deploy TTE policy solution sets (Create Set) to address the congestion on the interface. The Expected Util column shows the expected utilization of each of the interface if the recommended action is committed. For more information, see Monitor LCM Operations.

Figure 7. LCM Operational Dashboard
Before committing TTE policies, you can preview the deployment of each TTE policy solution set. Click in the Actions column and choose Preview. The window displays the node, interface, and the recommended action for each TTE policy. From the Preview window, you can select the individual TTE policies, and view different aspects and information as you would normally do in the topology map. The following figure shows the recommended TTE policies for the interface GigabitEthernet0/0/0/4. Note the Color IDs to later verify TTE policy deployment.

Figure 8. LCM TTE Deployment Preview

After you are done viewing the recommended TTE policies on the map, navigate back to the Operational Dashboard and click Commit All. The LCM Status column changes to Mitigating.

Note

All LCM recommendations per domain must be committed in order to mitigate congestion and produce the expected utilization as shown in the Operational Dashboard. The mitigating solution is based on all LCM recommendations being committed because of dependencies between solution sets.

Step 5

Validate TTE SR policy deployments.

Click Notifications icon > Events tab. Note which LCM events are listed in the Events window.

Note

Crosswork Optimization Engine will report network events that are detected based on the policies and features you have enabled. for example, if a link drop causes an SR-TE policy to go down or if LCM detects congestion. These alerts are reported in the UI and if desired can be fowarded to third party alerting/monitoring tools.

Return to the Operational Dashboard to see that the LCM state changes to Mitigated for all TTE policy solution sets.

Note that the LCM state change will take up to 2 times longer than the SNMP cadence.
Confirm the TTE policy deployment by viewing the topology map and the SR Policy table (Traffic Engineering > Traffic Engineering > SR-MPLS tab).
Select one of the new SR-TE policies and view the SR policy details (click and choose View Details).

Step 6

Remove the TTE SR policies upon LCM recommendation.

After some time, the deployed TTE SR policies might no longer be needed. This occurs if the utilization will continue to be under the threshold without the LCM-initiated TTE tunnels. In this case, LCM generates new recommended actions to delete the TTE SR policy sets. Click Commit All to remove the deployed TTE SR policies.
Click Commit All to remove the SR policies.
Confirm the removal by viewing the topology map and SR Policy table.

In this scenario we observed how to leverage LCM to alleviate traffic congestion in the network. LCM takes the manual tracking and calculation out of your hands but at the same time gives you control as to whether to implement the congestion mitigation recommendations, or not. You can preview the recommendations and see how the potential deployment will take effect in your network before you deploy them. As traffic changes, LCM tracks the deployed TTE SR-TE policies and decides whether or not they are still needed. If not, LCM recommends deleting them.

Configure LCM

To enable and configure LCM:

Before you begin

Please review LCM Important Notes.

Procedure

Step 1

From the main menu, choose Traffic Engineering > Local Congestion Mitigation > Domain-ID-card > > Configuration > Basic tab.

Example:

Step 2

Toggle the Enable switch to True.

Step 3

Enter the required information. Hover the mouse pointer over Field Help icon to view a description of each field.

The following list describes additional field information:

Utilization Threshold—Set the utilization percent at which LCM will consider an interface to be congested. This value applies to all interfaces, unless you specify thresholds to individual interfaces in the Link Management page.
Profile ID—This is a required configuration to enable traffic steering onto LCM policies. Autoroute (steers traffic into the tactical SR-TE policies LCM creates) is applied to SR-TE policies through the proper Profile ID option that is set here to align with the configuration on the PCC associating that Profile ID with autoroute feature.
Congestion Check Interval (seconds)—This value determines the interval at which LCM will evaluate the network for congestion. Under a steady state, when there are no recommendation commits, it uses this interval to re-evaluate the network to determine if changes are required to recommendations. For example, if the interval is set to 600 seconds (10 minutes), LCM will evaluate the network every 10 minutes for new congestion and determine whether a new recommendation or modifications to existing recommendations are needed. Examples of modifications can include removal or updates to individual policies that were previously recommended. Since network changes may take time for the information to stabilize and propagate to LCM, set the interval to no less than twice the SNMP collection cadence.
Interfaces to Monitor—By default, this is set to Selected Interfaces and you will need to add thresholds to individual interfaces by importing a CSV file in the Link Management page (Traffic Engineering > Local Congestion Mitigation > Domain-ID > > Link Management). Only interfaces defined in the Link Management page will be monitored. If set to All Interfaces, LCM will monitor the interfaces with custom thresholds that are uploaded in the Link Management page and the rest of the interfaces using the Utilization Threshold value configured on this page.
Advanced > Congestion Check Suspension Interval (seconds)—This interval determines the time to wait (after a Commit All is performed) before resuming congestion detection and mitigation. Since this interval should allow time for network model convergence, set the interval to no less than twice the SNMP collection cadence.
Advanced > Auto Repair Solution—If set to True, LCM will automatically delete any down, failed, or uncommitted LCM TTE policies. This option is mainly to address a failure in a policy.

If this option is disabled, and the Urgency status of the recommendation shown in the LCM Operational Dashboard is High, then the recommended solution is a candidate for the Auto Repair Solution. This means that a network failure will most likely occur if the solution is not deployed.

Advanced > Adjacency Hop Type—If set to Protected, LCM will create SR policies using protected adjacency SIDs. This allows for Topology-Independent Loop-Free Alternate (TI-LFA) to compute a path for any adjacency failures.

Note

This option should only be set to Protected if all nodes in the same IGP area as LCM is operating are strict SPF SID capable.

Advanced > Over-provisioning Factor (OPF)—This option helps address unequal ECMP traffic distribution (elephant flows). This value determines the percentage of how much extra traffic should be accounted for when computing a path for a by-pass policy. If LCM needs to divert x amount of traffic due to congestion, then it will search for a path that can support x * (1 + OPF) traffic. For more information, see LCM Calculation Workflow. The default value is 0.

Step 4

To save your configuration, click Commit Changes. If congestion occurs on any monitored interfaces, LCM will display recommendations (LCM will not automatically commit or deploy new TTE policies) on the LCM Operational Dashboard. You can then preview the recommended TTE policies and decide whether or not to commit and deploy them onto your network.

Add Individual Interface Thresholds

Networks have many different links (10G, 40G, 100G) that require different thresholds to be set. To assign specific threshold values for individual interfaces when using LCM or Bandwidth Optimization, do the following:

Procedure

Step 1	From the main menu, choose one of the following: Local Congestion Mitigation > Link Management Bandwidth Optimization > Link Management
Step 2	Click .
Step 3	Click the Download sample configuration file link.
Step 4	Click Cancel.
Step 5	Open and edit the configuration file (sampleLcmLinkManagement.csv) you just downloaded. Replace the sample text with your specific node, interface, and threshold information.
Step 6	Rename and save the file.
Step 7	Navigate back to the Link Management window.
Step 8	Click and navigate to the CSV file you just edited.
Step 9	Click Import.
Step 10	Confirm that the information appears correctly in the Link Management window.

Monitor LCM Operations

The LCM Operational Dashboard (Traffic Engineering > Local Congestion Mitigation > Domain-ID > > Operational Dashboard) shows congested interfaces as defined by the configured utilization threshold. For each interface, it lists details such as current utilization, recommended action, status, expected utilization after committing recommendations, and so on. Recommendations are listed as part of a set, and if deployed, all changes are committed. Urgency indicates the importance of recommendation deployment or action. Urgency values can be one of the following:

Low—Indicates that LCM instantiated policies can be removed because they are no longer needed or that no changes are required.
Medium—Indicates new or modified recommendations.
High—Indicates network failures and recommendations should be deployed. This is a candidate that can be addressed automatically if the Auto Repair Solution advanced option was enabled. See Configure LCM.

Hover the mouse pointer over Field Help icon to view a description of what type of information each column provides. From this dashboard, you can also preview ( > Preview) and deploy TTE policy recommendations.

In addition to the LCM Operational Dashboard, you can click Notifications icon > Events tab. Note which LCM events are listed in the Events window.

Cisco Crosswork Optimization Engine 3.0 User Guide

Bias-Free Language

Book Title

Cisco Crosswork Optimization Engine 3.0 User Guide

Chapter Title