IPS Testing


IPS Testing
      Replay Traffic
      Live Testing
Hybrid Approach


The detection ability of a device should be the primary focus of any IPS testing. Because evasion techniques are evolving every day, it is imperative that IPS devices have the ability to detect well-known exploits and new variants using advanced obfuscation techniques. Detection testing is usually performed using penetration testing tools or public exploits.

IPS Testing

A key factor in successful testing is the level of knowledge of the person conducting the test. For example, tools like Core Impact and Metasploit contain modules that scan for the vulnerable service before sending an attack; if the target system is not vulnerable, an attack may not be sent. To further complicate the issue, certain IPS devices may fire exploit specific signatures on this probe, regardless of the fact that nothing malicious has occurred. If this scenario is not recognized, the results of the test may be invalid. These testing processes apply as well to Layers 3, 4, and 7— a high level of knowledge and experience is crucial. Test results will be invalid unless confirmation exists that the exploit functions normally with the evasion techniques in place.

The goal of detection testing is to identify a device’s baseline detection and resistance to evasion. Proper testing consists of creating several test cases for the vulnerability being examined and vulnerable systems for test use. Traffic should be captured at each step to facilitate future testing and verification.

The first step is to choose a non-evasive, well-known version of the attack. Ensure that the attack functions correctly and exploits the vulnerable system without the IPS device monitoring the traffic. This step establishes the baseline attack (the simplest version of the attack). The second step involves changing the shell code used in the attack. This step helps confirm that the IPS is providing protection for the vulnerability as opposed to a specific variant of the attack. Once again, capture and test the attack against a vulnerable system without the IPS inline.

The third step is to add Layer 7 (Layers 3 and 4 if possible) obfuscation to the attack. Adding layers of obfuscation is the most important part of detection testing; the more complex the techniques, the better the test. You may want to create several obfuscation cases if the protocol allows several layers of obfuscation. The final step in testing is to combine all of the evasion and obfuscation methods into the attack.

These four steps allow insight into the type of coverage a device provides for a specific vulnerability, as well as its ability to inspect obfuscated traffic. By testing several attacks using the same protocol, the shortfalls of the IPS device’s ability to inspect the protocol will be illuminated. For example, if a vendor only detects the “out of the box” and nonfragmented variants of an Microsoft Remote Procedure Call (MSRPC) vulnerability, then that particular IPS may not have an engine capable of properly parsing fragmented (Layer 7) MSRPC traffic.

Depending on the protocol, several different application-level obfuscation methods may be used. Over HTTP, for example, the most common techniques are white space injection, header injection, Unicode encoding, and function obfuscation using javascript. If the vulnerability involves a malicious HTTP server page, chunked encoding and compression may also be used.

Figure 1 illustrates the Metasploit Project’s ie_xp_wmf exploit without obfuscation

Figure 1. Metasploit ie_xp_wmf Exploit—No Obfuscation

Metasploit ie_xp_wmf Exploit—No Obfuscation

Figure 2 illustrates the same exploit using the gzip, chunked encoding, and header injection obfuscation techniques.

Figure 2. Metasploit ie_xp_wmf Exploit—Multiple Obfuscation Techniques

Metasploit ie_xp_wmf Exploit—Multiple Obfuscation Techniques

The MSRPC protocol is also easily obfuscated. It is possible to obfuscate the bind attempt by sending several bind context ids in the initial bind request; however, an IPS device must monitor the traffic to see which binds were successful.

Figure 3 illustrates an example of bind obfuscation using multiple UUIDs.

Figure 3. Blind Obfuscation Using Mutiple UUIDs

Blind Obfuscation Using Mutiple UUIDs

It is also possible to connect to a non-vulnerable interface, have the bind accepted, and issue an alter context command to the server, switching the attack to the vulnerable interface. All of this can also be fragmented at the application level making detection extremely difficult. In some cases it is also possible to use MSRPC on top of SMB (all of the MSRPC evasions are still valid), allowing fragmentation on the SMB level, hiding the MSRPC headers.

Figure 4 illustrates an example of MSRPC fragmented by means of SMB.

Figure 4. MSRPC Fragmented on SMB Level

MSRPC Fragmented on SMB Levels

All of these examples can also be further obfuscated using standard ip evasion and obfuscation techniques. If a device can handle all of these techniques, it is properly inspecting the entire protocol, providing much more protection than a device that cannot inspect the protocol.

Replay Traffic

Many of the currently available IPS testing tools use replay traffic. The primary testing issue when using replay tools is the quality of the captured traffic. Sample issues, such as incomplete streams, packets with bad checksums, and out of order packets, can all create problems during testing. These issues can easily be fixed with open source tools similar to the netdude tool.

The most common problem occurs when capture does not include a sample of a real exploit. Capture samples are often created by vulnerability scanners that scan the services and look at banners or version strings. Normally these tools do not send any malicious traffic and should not be used when testing an IPS. Additionally, you must also ensure that the pcap file contains a copy of the entire attack. Depending on how a device engine is designed, some signatures may fire on partial attacks that do not properly represent the attack in a real world environment in that they would not exploit a vulnerable system.

Issues can occur when replaying samples at speeds other than that of the actual speed. If you replay traffic at different speed than the captured speed, some detection and protection algorithms may not work correctly. Threshold-based or behavioral algorithms need to see the traffic at the same speed it was originally sent. In addition, TCP uses timestamps for Protection against Wrapped Sequence Numbers (PAWS) and accelerating the traffic may cause time dilation to become a problem. Depending on the tool, the inability to react as a real network stack to changes in the network may result in unreliable testing.

If a replay tool can retransmit missing packets, it may retransmit the original packet, unlike a real network host stack. Retransmission can increase congestion on the network and cause a positive feedback loop. The pass or fail criteria of the tool must also be considered. Tomahawk, for example, replays packets and expects to receive the packets unmodified and in the same order as sent. If an IPS device does active TCP/IP normalization, it may reorder IP fragments and TCP segments to protect against evasions. Depending on the tool, this may be incorrectly reported as a failure or a block.

Live Testing

The most reliable form of testing for coverage is live testing. A baseline test case with a vulnerable machine should be setup and attacked with a working exploit. Verify that the exploit has the desired effect (a shell opened back to the attacker, or a crash of the service, for example), then reset the vulnerable machine to the pre-exploited state. Next, place the IPS under test between the attacker and the vulnerable host and attempt the exploit again. This process should be repeated for each of detection testing steps.

Vmware (or another virtual system that supports snapshots) is an excellent tool for hosting the vulnerable machines. If the exploit’s desired action occurs and the traffic is passed though a properly configured IPS device, then the attack was not detected. The problem with this approach is that it can become very time consuming when evaluating more than one IPS device.

Hybrid Approach

A fast and reliable approach for IPS testing is a hybrid of these two methods. By creating pcap file for each attack without the IPS device monitoring the traffic, we have created perfect samples for replay testing. Care must be taken to ensure you capture the entire attack and that the attack was successful. It is recommended to use a closed network so that the traffic sample is as clean as possible. Next, replay the traffic samples against the IPS device and note any that were not detected or blocked. You should see an inverse correlation between detection and obfuscation. All of the attacks that were not detected need to be manually verified using live testing. Manual verification eliminates any potential problem introduced by using a traffic sample and replay tool and provides most of replay testing’s time-saving benefits.


Craig Williams ( 
Software Engineer
Security Research and Operations

This document is part of Cisco Security Research & Operations.

This document is provided on an "as is" basis and does not imply any kind of guarantee or warranty, including the warranties of merchantability or fitness for a particular use. Your use of the information on the document or materials linked from the document is at your own risk. Cisco reserves the right to change or update this document at any time.

Back to Top