Introduction
The Relationship of Bandwidth and Packet Forwarding Rate
Other Performance Metric Relationships
Conclusions
Acknowledgments
References
Introduction
Building and operating an IP network requires an in-depth understanding of both the infrastructure and the performance of devices that are used within the network, including how packets are handled by each network device. Network engineers most often refer to the performance of network devices by using the speed of the interfaces expressed in bits per second (b/s). For example, a network device may be described as having a performance of 10 gigabits per second (Gb/s). Although this is useful and important information, expressing performance in terms of b/s alone does not adequately cover other important network device performance metrics.
Depending on the type of network device, additional performance metrics might be required to fully describe how the device will perform. This is particularly important when high-touch features are configured and the device is under a high network load. Metrics that are expressed in terms such as packets per second (p/s), connections per second (c/s), transactions per second (t/s), and maximum concurrent connections (mcc) also provide information that can be vital to a more complete understanding of the device performance characteristics.
For example, routers and switches are generally considered to be stateless devices because they forward each packet independently. Thus, metrics such as b/s and p/s may be sufficient to describe the performance of these devices. However, devices like firewalls, intrusion prevention systems, and load balancers, which create and maintain state tables to forward packets, require additional metrics, such as c/s and mcc, to fully and accurately describe their performance. Mathematical relationships can be defined, either directly or indirectly, between bandwidth and these additional metrics. The remainder of this document discusses these relationships, in addition to describing the metrics themselves and their importance to understanding network device performance.
Note: Additional metrics beyond those described in this document may be highly relevant for specific devices under specific operating conditions. Knowing interface, queue, and processing delays; delay variability; latency; and other forwarding-related metrics can be critical, especially in delay-sensitive voice and video traffic environments. System performance may vary depending on configured features, traffic type (for example, unicast versus multicast), and load. When designing networks, administrators should consider all factors that are pertinent to their specific environments.
The Relationship of Bandwidth and Packet Forwarding Rate
Network devices receive and forward packets through physical interfaces that employ Layer 2 technologies, such as Ethernet and Packet Over SONET (POS) framing. The description for these network links always includes bandwidth that is expressed in terms of b/s. By performing simple mathematical manipulations, it is possible to determine the potential range of p/s, or more correctly, frames per second (f/s) that a network link can support.
For example, the very common 1-Gb/s Ethernet interface is capable of transmitting up to 1,000,000,000 b/s. To determine p/s, first convert bits to bytes. (There are eight bits in one byte.) Then consider how many bytes exist in each packet. The size of the packet does not have to be a fixed value, but administrators can bound the problem by recognizing that there are both minimum and maximum packet sizes. The minimum size is based on both the IP-defined minimum IP packet size and the Layer 2-defined minimum frame size. The maximum IP packet size is based on the link maximum transmission unit (MTU) for the Layer 2 technology. Based on these factors, and using Ethernet as an example, the following two calculations can be considered:
- Maximum Frame Rate (Minimum Frame Size)
- Maximum Throughput (Maximum Frame Size)
The maximum Ethernet frame rate is achieved by a single transmitting node that does not suffer any collisions when Ethernet frames are at their smallest size. The minimum Ethernet frame payload is 46 bytes (based on the slot time of Ethernet), which yields a frame that consists of 72 bytes (see Table 1) plus a 12-byte inter-frame gap, for a total Minimum Frame size of 84 bytes.
The maximum Ethernet throughput is achieved by a single transmitting node that does not suffer any collisions when the Ethernet frames are at their maximum size. The maximum Ethernet frame payload is 1500 bytes (not considering Jumbo frames), which yields a frame that consists of 1526 bytes (see Table 1) plus a 12-byte inter-frame gap, for a total Maximum Frame size of 1538 bytes. (This calculation provides the lower bound on frame rate.)
Table 1. Maximum Frame Rate and Throughput Calculations For a 1-Gb/s Ethernet Link
| Frame Part | Minimum Frame Size | Maximum Frame Size |
|---|---|---|
| Inter Frame Gap (9.6 ms) | 12 bytes | 12 bytes |
| MAC Preamble (+ SFD) | 8 bytes | 8 bytes |
| MAC Destination Address | 6 bytes | 6 bytes |
| MAC Source Address | 6 bytes | 6 bytes |
| MAC Type (or length) | 2 bytes | 2 bytes |
| Payload (Network PDU) | 46 bytes | 1,500 bytes |
| Check Sequence (CRC) | 4 bytes | 4 bytes |
| Total Frame Physical Size | 84 bytes | 1, 538 bytes |
[1,000,000,000 b/s / (84 B * 8 b/B)] == 1,488,096 f/s (maximum rate)
[1,000,000,000 b/s / (1,538 B * 8 b/B)] == 81,274 f/s (minimum rate)
Using the computed maximum and minimum frame rate values above of 1,488,096 f/s and 81,274 f/s for a 1 Gb/s Ethernet link, the computed maximum and minimum frame rate values of 1,488,096 f/s and 81,274 f/s for a 1-Gb/s Ethernet link can be plotted as shown in Figure 1. Figure 1 also displays other common link speeds, such as 10 Mb/s, 100 Mb/s, and 10 Gb/s Ethernet, for comparison purposes. A constant rate line is also shown at 100 Kp/s. Whether hardware- or software-based, network devices have a maximum rate at which they can forward packets. Thus, graphing this maximum forwarding rate can provide an indication of the equivalent bandwidths that a device may be capable of handling for various packet sizes.
Figure 1. Relationship Between b/s and p/s For Various Packet Sizes and Ethernet Link Speeds Ranging From 10 Mb/s To 10 Gb/s
As Figure 1 demonstrates, the packet forwarding rate (p/s) for each type of Ethernet link varies depending on packet size. For example, a 1-Gb/s Ethernet interface can deliver anywhere between 81,274 and 1,488,096 p/s. By comparison, a 10-Gb/s Ethernet interface can deliver packets at 10 times these rates– between 812,740 and 14,880,960 p/s. That is, a 10x change in bandwidth (b/s) has the same 10X change in packet rate (p/s).
Other Performance Metric Relationships
It should now be clear then that in addition to bandwidth (b/s), a metric such as p/s is also quite important in order to fully understand network performance. In literature, the interfaces of a network device are said to operate at line rate when the device is capable of forwarding packets, regardless of size. Thus, even for the smallest packets (highest packet rate), the network device will perform its functions. For example, because one of the newest Cisco routers, the Cisco ASR 1000 Series Router, is capable of forwarding packets at up to 16 Mp/s with services enabled, it can support the processing of the equivalent of 10 Gb/s of traffic at line rate, with services, even for small packets. (See the References section of this document for information about the forwarding rates of other Cisco devices.)
However, p/s is not the only important metric for network devices. For routers, p/s is one of the most important parameters, but different metrics must also be considered for other devices, such as firewalls, load balances, intrusion prevention systems, NAT translation devices, and other stateful systems. By their nature, stateful devices create and manage unique information on each connection. For example, new TCP connections are recognized by the arrival of a TCP SYN packet, and the state information that is maintained by the device typically includes the source IP Address, destination IP Address, source port, destination port, and protocol (TCP). This information is often referred to as the five-tuple for the connection. Once this stateful connection information is cached, the device is free to perform the functions for which it was designed (for example, firewalling, load balancing, etc.).
With this context, several different metrics can be described:
- Connections Per Second
- Maximum Concurrent Connections
- Transactions Per Second
Connections per second (c/s) refers to the rate at which a device can establish state parameters for new connections. As previously noted, a stateful device must create and manage connection information on all unique IP streams that transit the device. Typically, the device must handle the first packet of a new connection differently than all subsequent packets so that the device can establish the state parameters for the new connection. Because this process is specialized, it usually occurs in the software process of the devices, as opposed to the normal hardware-based forwarding process. The rate at which a device can establish state for new connections is related to factors such as processor (CPU) speed, memory speed, architecture, TCP/IP stack efficiency, etc. In terms of packet handling performance, the rate at which a device can handle packets when establishing state parameters is usually a small percentage of the rate at which the same device can forward packets in hardware once the state parameters are established. As an example, the Cisco Application Control Engine (ACE) for the Cisco Catalyst 6500 Series Switch and Cisco 7600 Series Router is listed as providing 16 Gb/s and 6.5 Mp/s of throughput per module. The Cisco ACE is also listed as supporting 325,000 new connections per second (c/s).
Maximum concurrent connections (mcc) refers to the total number of sessions (connections) about which a device can maintain state simultaneously. This value is mainly related to the amount of memory that is dedicated to this task. However, even though memory is inexpensive, adding memory to support more concurrent connections makes little sense when the c/s rate is low. For example, if some device had sufficient memory to handle one million concurrent connections but the device had a maximum connection setup rate of 10 c/s, it would take more than 100,000 seconds (or 28 hours) to fully utilize all of this state. Thus, devices with low c/s rates typically also have a low mcc values as well. Continuing with the Cisco ACE example, the Cisco ACE supports four million concurrent connections, which is compatible with its new connection rate of 325,000 c/s. At the maximum c/s rate, the maximum number of concurrent connections could be established in 12 seconds. (Additional information, such as the life-span of individual connections is highly important in designing systems requiring state.)
Transactions per second (t/s) refers to the number of complete actions of a particular type that can be performed per second. The t/s measurement refers to more than just the processing of a single packet or even the setup of a new connection; it refers to the completion of a full cycle of a specific action. In database design, t/s is a common metric, and it refers to the number of database transactions that are performed per second. In networking, certain devices use this metric to describe the application of certain complex process to the packets to comprise a full conversation. For example, the Cisco ACE XML Gateway, which helps secure the deployment of XML applications and web services by processing XML messages and enforcing XML schema, is listed as delivering industry-leading performance exceeding 30,000 t/s.
Each of these metrics may be relevant to a certain type of network device. However, when appropriate, these metrics are essential for properly assessing network device performance and properly designing network architecture that can support the required services.
For example, assume a new firewall service is being designed to use 1Gbps Ethernet interfaces and that the normal traffic profile for which this service is being designed for 5,000 new connections per second of legitimate customer traffic. If these criteria are the sole metrics used for determining the size and scaling of the firewall services, the final design may conclude that one particular type or size of firewall could handle the load. However, the previous mathematic example showed that this device can potentially be expected to handle see up to 1.4 Mp/s of ingress traffic (assuming small packets). Considering that all this traffic could be new connection attempts, such as in a flash-crowd or distributed denial of service (DDoS) event, it is clear that the final design should be selecting a suitable device that also takes this metric into consideration. Alternatively, this metric could also be translated into additional design choices for defining this service. Options that may be considered could include increasing the capacity of the deployed firewall service, or deploying additional upstream mechanisms, such as rate-limiting or Clean-Pipes services, to protect the firewall deployment. Although the notion that in some circumstances a firewall may need its own protection seems odd, it is often required in high-bandwidth environments. In fact, all stateful devices must be evaluated for unanticipated loads and conditions.
Conclusions
The importance of fully understanding the performance range of each network device is essential for creating and operating reliable and available networks and services. Additional metrics beyond link speed (b/s), such as packets per second (p/s), connections per second (c/s), transactions per second (t/s), and maximum concurrent connections (mcc), are crucial to developing and deploying network services. These considerations are especially critical when considering the deployment of stateful devices in high-bandwidth networks, because this scenario can add constraints on device capacity and network architectures and requires protection approaches to the overall design and operation space.
Acknowledgements
Gregg Schudel (gschudel@cisco.com)
Network Consulting Engineer
Gregg Schudel (CCIE No. 9591) joined Cisco in 2000 as a Consulting System Engineer supporting the United States Service Provider organization. Now part of the Security Intelligence Engineering organization at Cisco, Gregg focuses on IP core network security architectures and technology for Interexchange Carriers, Mobile Carriers, and Web Services Providers.
Additional material created by Security Intelligence Engineering can be found at the Security Intelligence Best Practices section of Cisco Security Intelligence Operations.
References
Router Security Strategies: Securing IP Network Traffic Planes
http://www.ciscopress.com/bookstore/product.asp?isbn=1587053365
Portable Product Sheets – Routing Performance
//www.cisco.com/web/partners/downloads/765/tools/quickreference/routerperformance.pdf
//www.cisco.com/web/partners/tools/quickreference/index.html
Cisco Application Control Engine (ACE) Module for Cisco Catalyst 6500 Series Switches and Cisco 7600 Series Routers
www.cisco.com/en/US/prod/collateral/modules/ps2706/ps6906/product_data_sheet0900aecd8045861b.pdf
Cisco ACE XML Gateway
//www.cisco.com/en/US/prod/collateral/contnetw/ps5719/ps7314/product_data_sheet0900aecd8060acd2.html
Clean Pipes DDos Protection
//www.cisco.com/en/US/netsol/ns697/networking_solutions_solution.html
Cisco Security Intelligence Operations - Security Intelligence Best Practices
http://www.cisco.com/security/bestpractices
This document is part of Cisco Security Research & Operations.
This document is provided on an "as is" basis and does not imply any kind of guarantee or warranty, including the warranties of merchantability or fitness for a particular use. Your use of the information on the document or materials linked from the document is at your own risk. Cisco reserves the right to change or update this document at any time.
This document is part of the Cisco Security portal. Cisco provides the official information contained on the Cisco Security portal in English only.
This document is provided on an “as is” basis and does not imply any kind of guarantee or warranty, including the warranties of merchantability or fitness for a particular use. Your use of the information in the document or materials linked from the document is at your own risk. Cisco reserves the right to change or update this document without notice at any time.