Vulnerability in Cisco IOS with OSPF, MPLS VPN, and Supervisor 32, Supervisor 720, or Route Switch Processor 720

Vulnerable Products

Products Confirmed Not Vulnerable

Vulnerable Cisco devices, when configured for Multi Protocol Label Switching (MPLS) Virtual Private Networking (VPN) and Open Shortest Path First (OSPF) sham-link, can suffer from a blocked queue, memory leak, and/or restart of the device.

This vulnerability is documented in Cisco bug ID CSCsf12082 ( registered customers only) , and has been assigned CVE ID CVE-2008-0537.

The following combination of hardware and software configuration must be present for the device to be vulnerable:

• Cisco Catalyst Sup32, Sup720, or RSP720 is present
  
• MPLS VPN is configured
  
• OSPF sham-link is configured
  

In order to determine whether you are running this feature, use the show running-config command and search for the address-family vpnv4 and area sham-link router configuration commands. The following command displays all configuration lines that meet the following criteria:

• Begins with the word "router," OR
  
• Includes "address-family vpnv4," OR
  
• Includes "sham-link"
  

Router# show run | include ^router |address-family vpnv4|sham-link
router bgp 1
  address-family vpnv4
router ospf 1 vrf VRFNAME
 area 0 sham-link  192.168.1.1 192.168.100.1
Router# 

For customers that run versions of IOS that support the section modifier, an additional option is available to view the relevant sections of the running configuration:

Router# show run | section ^router
router bgp 1
[snip]
  address-family vpnv4
router ospf 1 vrf VRFNAME
 area 0 sham-link  192.168.1.1 192.168.100.1
[snip]

If certain packets are received by a device that meets the above requirements, the input queue of the interface that receives these packets can become blocked, which can prohibit additional traffic from entering the interface and cause a denial of service condition. In addition to a potential blocked interface queue, the device can also suffer a memory leak or restart. In the event of a memory leak, the device is unable to forward traffic once available memory is depleted.

For more information on MPLS VPNs, please reference the following document:

http://www.cisco.com/en/US/docs/net_mgmt/vpn_solutions_center/1.1/user/guide/VPN_UG1.html

For more information on OSPF sham-links, please reference the following document:

http://www.cisco.com/en/US/docs/ios/12_2t/12_2t8/feature/guide/ospfshmk.html

Identifying a Memory Leak

This vulnerability can manifest as a leak in the I/O memory pool. The following is an example of a system message that indicates an exhaustion of the I/O pool:

006029: Aug  10: %SYS-2-MALLOCFAIL: Memory allocation of 808 bytes failed from 0x41613238, alignment 32 
Pool: I/O Free: 176  Cause: Not enough free memory 
Alternate Pool: None  Free: 0  Cause: No Alternate pool 

Note that in the above output, the affected memory pool is Pool: I/O, and the cause is Cause: Not enough free memory. This output indicates that the I/O memory pool has been exhausted.

Additionally, a user with enable-level access can check the device through the show buffers command to identify buffer allocation failures.

Router#show buffers   
Buffer elements:   
496 in free list (500 max allowed)   
77298300 hits, 0 misses, 0 created   

Public buffer pools:   
Small buffers, 104 bytes (total 148654, permanent 1024, peak 148654 @ 1d12h):   
0 in free list (128 min, 2048 max allowed)   
24688031 hits, 4023203 misses, 0 trims, 147630 created   
3243434 failures (3182828 no memory) 

The above output shows that buffer allocation failed due to insufficient memory.

Identifying a Blocked Interface

A symptom of this type of blocked queue is the failure of control-plane protocols such as routing protocols (OSPF, Enhanced Interior Gateway Routing Protocol (EIGRP), Border Gateway Protocol (BGP), Intermediate System to Intermediate System (ISIS), etc.) and MPLS TDP/LDP to properly establish connections over an affected interface.

In order to identify a blocked input interface, issue the show interfaces command, and search for the Input Queue line. The size of the input queue can continue to increase. If the current size, which is 76 in the example below, is larger than the maximum size (75), the input queue is blocked.

It is possible that a device receives a high rate of traffic destined to the control plane, and the full queue is only a transient event. In order to verify if the interface is actually blocked, shut down the interface with the shutdown interface configuration command and examine the input queue. If the input queue does not display 0 packets, the interface is blocked.

Router#show interface ethernet 0/0
Ethernet0/0 is up, line protocol is up  
  Hardware is AmdP2, address is 0050.500e.f1e0 (bia 0050.500e.f1e0)   
  Internet address is 172.16.1.9/24
  MTU 1500 bytes, BW 10000 Kbit, DLY 1000 usec, rely 255/255, load 1/255
  Encapsulation ARPA, loopback not set, keepalive set (10 sec)
  ARP type: ARPA, ARP Timeout 04:00:00  
  Last input 00:00:41, output 00:00:07, output hang never
  Last clearing of "show interface" counters 00:07:18
  Input queue: 76/75/1091/0 (size/max/drops/flushes); Total output drops: 0  


!--- The 76/75 shows that this is blocked

Cisco has provided scores for the vulnerabilities in this advisory based on the Common Vulnerability Scoring System (CVSS). The CVSS scoring in this Security Advisory is done in accordance with CVSS Version 2.0.

CVSS is a standards-based scoring method that conveys vulnerability severity and helps determine urgency and priority of response.

Cisco has provided a base and temporal score. Customers can then compute environmental scores to assist in determining the impact of the vulnerability in individual networks.

Cisco has provided an FAQ to answer additional questions regarding CVSS at

http://www.cisco.com/web/about/security/intelligence/cvss-qandas.html .

Cisco has also provided a CVSS calculator to help compute the environmental impact for individual networks at

http://intellishield.cisco.com/security/alertmanager/cvss .

Exploitation of this vulnerability may result in a blocked interface input queue, memory leak, and/or restart of the device. Repeated exploitation of this vulnerability may result in an extended denial of service.

When considering software upgrades, also consult http://www.cisco.com/go/psirt and any subsequent advisories to determine exposure and a complete upgrade solution.

In all cases, customers should exercise caution to be certain the devices to be upgraded contain sufficient memory and that current hardware and software configurations will continue to be supported properly by the new release. If the information is not clear, contact the Cisco Technical Assistance Center (TAC) or your contracted maintenance provider for assistance.

Each row of the Cisco IOS software table (below) names a Cisco IOS release train. If a given release train is vulnerable, then the earliest possible releases that contain the fix (along with the anticipated date of availability for each, if applicable) are listed in the "First Fixed Release" column of the table. The "Recommended Release" column indicates the releases which have fixes for all the published vulnerabilities at the time of this Advisory. A device running a release in the given train that is earlier than the release in a specific column (less than the First Fixed Release) is known to be vulnerable. Cisco recommends upgrading to a release equal to or later than the release in the "Recommended Releases" column of the table.

Once a device interface queue has been exhausted, only a device restart can clear OSPF packets in the blocked queue.

Due to the manner in which these packets are processed, the queue block occurs prior to the OSPF MD5 check. The OSPF MD5 configuration does not protect a device from this vulnerability.

Increasing the Selective Packet Discard (SPD) Headroom

At the most basic level, the Selective Packet Discard (SPD) provides extended buffering for control plane traffic. Known as the SPD headroom, this additional queue depth is typically reserved for traffic with IP Precedence equal to 6 (such as BGP), the Connectionless Network Service (CLNS) based routing protocol Intermediate System-to-Intermediate System (IS-IS), OSPF, and Layer 2 keepalives.

Increasing the SPD headroom provides additional buffering for OSPF packets. In the event of a blocked queue, the SPD headroom can be increased to allow more control plane traffic buffer space.

More information on SPD can be found in the following white paper:

http://www.cisco.com/web/about/security/intelligence/spd.html

It is possible to expand the queue size to accommodate more packets, but packets can still accumulate until the expanded queue is exhausted. As a temporary workaround that allows traffic to continue to flow, the input hold queue can be increased. Any additional malformed packets still fill the queue, but increasing the input queue depth can extend the amount of time before the input queue fills and traffic ceases flowing. The following example demonstrates how to set the input queue size from the default of 75 to the maximum of 4096:

Router# configure terminal
Router(configure)# interface FastEthernet 0/0
Router(config-if)# hold-queue 4096 in

Removing OSPF Sham-Link Configuration

Because OSPF Sham-Link configuration is required for the vulnerability to be present, removing Sham-Link functionality eliminates exposure to this vulnerability. In order to remove the OSPF Sham-Link configuration from a device, the OSPF configuration must be changed on each interface where Sham-Link is configured.

For configuration information on OSPF Sham-Link, please consult the following document:

http://www.cisco.com/en/US/docs/ios/12_2t/12_2t8/feature/guide/ospfshmk.html

Cisco IOS Embedded Event Manager

Cisco IOS Embedded Event Manager (EEM) provides event detection and reaction capabilities on a Cisco IOS device. It is possible to detect blocked interface queues with an EEM policy. EEM can alert administrators of blocked interfaces with email, a syslog message, or a Simple Network Management Protocol (SNMP) trap.

A sample EEM policy that uses syslog to alert administrators of blocked interfaces is available at Cisco Beyond, an online community dedicated to EEM. A sample script is available at the following link:

http://forums.cisco.com/eforum/servlet/EEM?page=eem&fn=script&scriptId=981

More information about EEM is available from Cisco.com at the following link:

http://www.cisco.com/en/US/products/ps6815/products_ios_protocol_group_home.html

Cisco has released free software updates that address this vulnerability. Prior to deploying software, customers should consult their maintenance provider or check the software for feature set compatibility and known issues specific to their environment.

Customers may only install and expect support for the feature sets they have purchased. By installing, downloading, accessing or otherwise using such software upgrades, customers agree to be bound by the terms of Cisco's software license terms found at http://www.cisco.com/en/US/docs/general/warranty/English/EU1KEN_.html, or as otherwise set forth at Cisco.com Downloads at http://www.cisco.com/public/sw-center/sw-usingswc.shtml.

Do not contact psirt@cisco.com or security-alert@cisco.com for software upgrades.

Customers with Service Contracts

Customers using Third Party Support Organizations

Customers without Service Contracts

The Cisco PSIRT is not aware of any public announcements or malicious use of the vulnerability described in this advisory.

This vulnerability was reported to Cisco by a customer.

THIS DOCUMENT IS PROVIDED ON AN "AS IS" BASIS AND DOES NOT IMPLY ANY KIND OF GUARANTEE OR WARRANTY, INCLUDING THE WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR USE. YOUR USE OF THE INFORMATION ON THE DOCUMENT OR MATERIALS LINKED FROM THE DOCUMENT IS AT YOUR OWN RISK. CISCO RESERVES THE RIGHT TO CHANGE OR UPDATE THIS DOCUMENT AT ANY TIME.

A stand-alone copy or Paraphrase of the text of this document that omits the distribution URL in the following section is an uncontrolled copy, and may lack important information or contain factual errors.

This advisory is posted on Cisco's worldwide website at :

http://tools.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-20080326-queue

In addition to worldwide web posting, a text version of this notice is clear-signed with the Cisco PSIRT PGP key and is posted to the following e-mail and Usenet news recipients.

• cust-security-announce@cisco.com
  
• first-teams@first.org
  
• bugtraq@securityfocus.com
  
• vulnwatch@vulnwatch.org
  
• cisco@spot.colorado.edu
  
• cisco-nsp@puck.nether.net
  
• full-disclosure@lists.grok.org.uk
  
• comp.dcom.sys.cisco@newsgate.cisco.com
  

Future updates of this advisory, if any, will be placed on Cisco's worldwide website, but may or may not be actively announced on mailing lists or newsgroups. Users concerned about this problem are encouraged to check the above URL for any updates.

Complete information on reporting security vulnerabilities in Cisco products, obtaining assistance with security incidents, and registering to receive security information from Cisco, is available on Cisco's worldwide website at http://www.cisco.com/web/about/security/psirt/security_vulnerability_policy.html. This includes instructions for press inquiries regarding Cisco security notices. All Cisco security advisories are available at http://www.cisco.com/go/psirt.