In the last few years, the service provider industry has adopted the Ethernet technology as a viable method of providing both point-to-point and multipoint services. Major drivers for this emergence of this familiar technology has been the scalability and flexibility delivered for broadband economically. Additionally, the abundance of new fiber deployment to business areas has made Ethernet the logical choice for deployment. By utilizing Ethernet switching equipment to exploit this fiber, and by enabling service ubiquity by incorporating Ethernet over SONET as well as ATM interworking, there is new value to many different types of enterprise customers, along with opportunity for SPs to sell services to those enterprises from a converged infrastructure.
This paper is the first in a series of documents based on the continuing work at Cisco Systems® focused on service providers' markets. The paper is addressing networking architecture for deployment of a Metro Ethernet Network for business customers. It offers architectural roles and considerations that drive the optimum deployment model for providing services.
Metro Ethernet Architecture
Metro Ethernet Islands
This chapter describes the architecture, functional components, network topologies, hardware, and software configurations identified in a Metro Ethernet architecture.
The functional components represent the individual layers that form the Metro Ethernet architecture. These architectural layers break down into the following categories:
• Layer 2 Ethernet access layer
• Layer 2 Ethernet aggregation layer
• Layer 2 Ethernet/MPLS edge layer - Multiprotocol Label Switching (MPLS) demarcation between Layer 2 access and Layer 3 core
• MPLS core layer
The platforms within the network layers operate in the following roles1:
• User-facing provider edge (U-PE) within the access layer
• Provider edge aggregation (PE-AGG) within the aggregation layer2
• Network-facing provider edge (N-PE) within the edge layer
• Provider core within the core layer
In some cases, a given Metro Ethernet deployment might not contain all of these layers. In fact, the architectural functions can be merged into a single layer and platform. For example, various combinations of network technologies and topologies can be formed to deliver Ethernet services without passing through the MPLS core network. In this case, the core layer is not needed and, quite possibly, neither is the N-PE or PE-AGG layer, either. In this context, these access network technology and topology combinations can be viewed as separate and independent from the interconnecting core network and are hence referred to as Metro Ethernet islands (or Ethernet access domains [EADs]). Note that because Ethernet services can be offered solely within these islands, they are by definition Metro Ethernet networks.3
Within an island, or EAD, access topologies can be deployed as either star (hub and spoke) or rings (Gigabit Ethernet or SONET/SDH). The decision to adopt a particular access topology is primarily set by fiber availability.
Figure 1 provides examples of islands that include Gigabit Ethernet hub and spoke, Gigabit Ethernet rings, and Ethernet over SONET rings.
Figure 1
Metro Ethernet Islands
Furthermore, it is useful to describe Metro Ethernet services according to their scope or reach in the network, that is, whether services are provisioned within or between islands:
• Services whose endpoints reside within one EAD are referred as Intra-EAD services.
• Services whose endpoints reside among two or more EADs are referred to as Inter-EAD services.
Figure 2 illustrates Intra-EAD and Inter-EAD services.
Figure 2
Intra-EAD and Inter-EAD Services
Architectural Roles
The service provider network is organized into multiple functional layers, each of which is characterized by a specific architectural role. These architectural roles are described in the following sections, and Figure 3 illustrates the mapping to the Metro Ethernet architecture.
Figure 3
Metro Ethernet Architecture
User Provider-Edge Role
The device with the user provider-edge (U-PE) role acts as the demarcation point between the customer and the service provider network. Typically, it is a Layer 2 device located in the access layer at the customer premises or outside plant but owned and managed by the service provider. Some service providers install this equipment in basements and wiring closets, and others install platforms with U-PE functions in the central office. As the entry point into the service provider network, the U-PE should be secure.
The primary U-PE functions are:
• Aggregate multiple customers at the access layer
• Define the Ethernet services by providing the appropriate User-Network Interface (UNI) characteristics, for example, 802.1Q tunneling (Q-in-Q) and 802.1Q trunking
• Segregate customer traffic with the use of unique service provider VLAN IDs per service
• Help ensure that bandwidth meets the required service-level agreement (SLA) by applying traffic classification, policing, marking, and queuing
• Enforce the service and admission control policies of the network
• Offer traffic multiplexing and congestion management
Provider-Edge Aggregation Role
The device with the provider-edge aggregation (PE-AGG) role resides in the aggregation network layer, an intermediate and optional layer between access and edge devices. The aggregation layer provides a way to scale the number of access devices connected to the edge platforms4.
The primary PE-AGG functions are:
• Efficient aggregation of traffic
• Traffic multiplexing and congestion management
• Local switching for Ethernet services
These devices are usually connected in redundant fashion so that the access devices can be dual-homed to the redundant aggregation platforms to improve resiliency. Being a Layer 2 network layer, redundancy and fault recovery is handled by a Spanning Tree protocol, specifically IEEE 802.1w/1s. Note that this document does not intend to be a Spanning Tree tutorial. Refer to the following URLs for detailed information about the protocol: http://www.cisco.com/warp/public/473/146.html and http://www.cisco.com/warp/public/473/147.html.
Network Provider-Edge Role
The device in the network provider-edge (N-PE) role acts as the demarcation point between the Layer 2 protocols in the EAD and the Layer 3 domain (if in the presence of an MPLS core). Therefore, the edge device interfaces directly to the access (or aggregation) network and, if required, to the MPLS core. At the edge, service provider VLANs are mapped to Ethernet-over-MPLS (EoMPLS) virtual circuits for point-to-point services or to Virtual Private LAN Service (VPLS) instances for multipoint-to-multipoint services.
The main N-PE functions are:
• MPLS and IP services gateway
• VPLS and Virtual Private Wire Service (VPWS) definitions
• Layer 2 VPN service interworking gateway
• Layer 3 VPN service layer
• Local switching for Ethernet services
• MAC addresses learning for Layer 2 multipoint VPNs
• Sophisticated traffic and congestion management
• Load balancing across equal-cost multipath (ECMP) links
• Redundancy mechanisms for EADs with two or more N-PEs
When the network has an MPLS core, the provider core router is the device that performs the MPLS label switching. Here, the service provider backbone or core network consists of multiple routers acting in the provider core and N-PE roles, which connect in a partial or full-mesh configuration and provide for end-to-end MPLS connectivity. The main functions of the provider core role are to:
• Offer reliable backbone connectivity
• Interconnect edge devices using an MPLS-based control plane
• Provide sophisticated traffic-management mechanisms
• Balance loads across multiple ECMP links
Mapping Network Layers to Roles
Each architectural layer consists of components and platforms that can play single or multiple roles. Table 1 summarizes the various combinations of roles that can exist within the architectural layers. For example, a platform in the edge layer performs its main N-PE role, but in addition, it could also have PE-AGG and U-PE responsibilities. This happens, for example, when a customer requires an interface directly to the N-PE. In Table 1, the green cells (indicated by an asterisk) identify valid role combinations.
Table 1. Mapping Network Layers to Roles
Layer
U-PE
PE-AGG
N-PE
Provider Core
Access
*
Aggregation
*
*
Edge
*
*
*
Core
*
Mapping Platforms to Roles
Metro Ethernet architecture considers specific platforms for each of the roles and summarizes, respectively, the recommended product-to-network layer mappings and product-to-role mappings (see Table 2). The green cells (indicated by an asterisk) identify valid configurations, and the red cells highlight configurations not included in the Metro Ethernet architecture.
Table 2. Mapping Metro Ethernet Platform to Network Layers
Table 3 associates each Metro Ethernet platform to the roles it can assume in the network. For example, the column labeled "U-PE and PE-AGG" refers to those platforms that can interface directly to customer equipment as well as to aggregate multiple access platforms. Similar considerations apply for the other columns.
Note: The Metro Ethernet solution remains independent about the selection of the devices playing the MPLS provider core role. Although the solution includes provider core routers in the core, the emphasis centers on the access, aggregation, and core edge layers (U-PE to N-PE). Further characterization of provider core-capable devices is out of the scope of Metro Ethernet.
Table 3. Mapping Metro Ethernet Platforms to Roles
This section describes the hardware and software components for the Metro Ethernet network topologies. Table 4 summarizes the topologies by platform and network layer. Each row in the table indicates a combination of products that can be used for building a specific topology.
This section describes in greater detail the topologies included under the Metro Ethernet effort. The following hardware recommendations should be considered an initial baseline. However, because they do not include traditional components, a complete evaluation of the end-to-end network must be performed.
The component tables in the following sections are organized by network roles and layers. For each network role, the hardware components split into systemwide (supervisor engines, for example) and line-card-specific sections. For example, the aggregation layer column labeled "PE-AGG, U-PE" lists the hardware that is common to the entire platform plus details of line cards facing the access and the edge layers.
Switched Hub-and-Spoke Topology
Figure 4
Switched Hub-and-Spoke Topology
The main characteristic of switched hub-and-spoke topology is the interconnection model between access and aggregation/edge equipment following a star fiber topology (Figure 4). The UNI (through the U-PE role) can attach directly to the access, aggregation, or edge platforms. In this configuration, the access device is typically located in building basements or in business parks and not in the service provider's central office.
The aggregation devices are optional and would typically be connected in redundant fashion to provide hardware and data path diversity. A Spanning Tree protocol (IEEE 802.1w/1s or Cisco Per VLAN Rapid Spanning Tree Plus [PVRST+]) is implemented in the Layer 2 access network to prevent Layer 2 loops in a meshed topology.
Finally, point-to-point and multipoint-to-multipoint VLAN-based (802.1Q trunk UNI) and port-based (802.1Q tunnel UNI) services can be offered concurrently in this network infrastructure.
Table 5 highlights the benefits and limitations of this topology.
Table 5. Benefits and Limitations of Switched Hub-and-Spoke Topology
Benefits
Limitations
Uses dedicated links from U-PE to PE-AGG/N-PE, and therefore access bandwidth is not shared among access devices.
It requires dedicated fiber connectivity between spoke/access and hub nodes. This topology requires more fiber than an equivalent ring topology. Therefore the cost of installing or leasing the fiber must be considered in these implementations.
Note: A logical hub-and-spoke topology can be overlaid over a physical ring layout with the use of coarse wavelength division multiplexing (CWDM) or dense wavelength division multiplexing (DWDM). This eliminates the need for new fiber but adds the cost of the optical equipment.
Subsecond Spanning Tree convergence on redundant access network when using IEEE 802.1W (Rapid Spanning Tree Protocol) or Cisco PVRST+.
The number of required interfaces at the aggregation/edge node is directly proportional to the number of access nodes. This might be an issue on platforms with low Gigabit Ethernet density. In these situations, an intermediate aggregation layer helps scale the deployment and reduces the number of interfaces required at the edge platform.
Note: Dual-home connections between the customer edge devices and the service provider's network are not shown in Table 6. For more information on this topic, please contact your Cisco account representative.
Table 6 lists the hardware components in Metro Ethernet to deploy this architecture.
1. This module replaced the OSM-ATM-OC12+ recommendation in Metro Ethernet 3.0 because of shaper inaccuracy issues discovered with the ATM OSM line card.
Note: For distributed mode operation, the Cisco 7600/Cisco Catalyst 6500 series Distributed Feature Card (DFC) should match the supervisor engine PFC (PFC3 BXL and DFC 3BXL, for example).
Topology 1a applies to deployments with a star fiber layout and small access devices aggregating multiple customers outside the central office.
Topology 1b applies to deployments with an installed base of Cisco Catalyst 3550 Series in which the organization wants to migrate from a Cisco 7600/Cisco Catalyst 6500 series Supervisor Engine 2 to a Supervisor Engine 720 PFC 3B/3BXL.
Network Interface Device to Hub-and-Spoke Topology
Figure 5
NID to Hub-and-Spoke Topology
In concept, the network interface device (NID) to hub-and-spoke topology (Figure 5) is identical to the previously described Switched Hub-and-Spoke Topology. However, in this scenario, the access switch is deployed at the service provider's central office or point of presence (POP), whereas in the previous scenario it is deployed in basements and business parks. As a consequence, access equipment has different requirements for port density, line card redundancy, and standards compliance.
In the topology depicted in Figure 5, the customer equipment connects to an access or an edge device through an NID). The NID is typically deployed at the customer site, and it provides for the fiber loop termination and fiber-to-copper translation.
As mentioned previously, aggregation equipment is optional and would typically connect in a redundant fashion to provide hardware and data path diversity. A Spanning Tree protocol (IEEE 802.1w/1s or Cisco PVRST+) is implemented in the Layer 2 access to prevent Layer 2 loops in a meshed topology.
Point-to-point and multipoint-to-multipoint VLAN-based (802.1Q trunk UNI), and port-based (802.1Q tunnel UNI) services can be offered concurrently on this network infrastructure.
This topology is also used to highlight new functions at the access or aggregation layer, which would allow the service provider to offer value-added services on top of an existing port-based Ethernet Multipoint Service (EMS)5 The desired behavior includes the capability of identifying service frames based on the value of an outer VLAN ID (SP-VLAN) and an inner VLAN ID (CE-VLAN). Both of these values are used in combination as a unique identifier for mapping traffic for special handling, which could include mapping to an EoMPLS virtual circuit, VPLS virtual circuit, or MPLS VPN. Figure 6 illustrates this topology.
Figure 6
Service Gateway Functions
For more information on Bundling UNI Attribute, please contact your Cisco Systems account representative.
This topology shares the same benefits and limitations as the Switched Hub-and-Spoke Topology presented earlier.
Table 7 lists the hardware components covered in the tested Cisco Metro Ethernet solution required to deploy this architecture.
Table 7. NID to Hub-and-Spoke Component Part Numbers
U-PE
PE-AGG, U-PE, Gateway
N-PE
Customer Equipment-Facing Line Cards
Chassis/ Supervisor Engines
Aggregation-Facing Line Cards
Access-Facing Line Cards
Chassis/ Supervisor Engines
Edge-Facing Line Cards
Aggregation-Facing Line Cards
Chassis/ Supervisor Engines
Core-Facing Line Cards
2a
WS-X6748-GE-TX
Cisco Catalyst 6500 Series with: WS-SUP720-3B(XL) (CatOS)
WS-X6724-SFP
WS-X6748-SFP
WS-X6724-SFP
WS-X6748-SFP
WS-X6748-GE-TX
WS-F6700-DFC3BXL
Cisco 7600/Cisco Catalyst 6500 series with:
WS-SUP720-3B(XL)
WS-X6704-10GE
WS-X6724-SFP
WS-X6748-SFP
OSM-2+4GE-WAN
WS-F6700-DFC3BXL
4GE-SFP-LC
Cisco 12000 Series with:
PRP-2
ISE E3 POS/GE
2b
WS-X4448-GB-RJ45
WS-X4306-GB
WS-X4448-GB-SFP
Cisco Catalyst 4500 Series with:
WS-X4516
WS-X4306-GB
WS-X4448-GB-SFP
WS-X4448-GB-RJ45
WS-X4306-GB
WS-X4448-GB-SFP
Cisco Catalyst 4500 Series with:
WS-X4516
WS-X4306-GB
WS-X4448-GB-SFP
WS-X6724-SFP
WS-X6748-SFP
WS-X6748-GE-TX
WS-X6582-2PA
PA-A6-OC3MM
WS-F6700-DFC3BXL
Cisco 7600/Cisco Catalyst 6500 series with:
WS-SUP720-3B(XL)
OSM-2+4GE-WAN
OSM-1OC48-POS-SI+
2c
WS-X6148A-GE-TX
WS-X6148-FE-SFP
Cisco Catalyst 6500 Series with:
WS-SUP32-8GE (Cisco Catalyst OS)
WS-X6724-SFP
WS-X6748-SFP
WS-X6724-SFP
WS-X6748-SFP
WS-X6748-GE-TX
WS-F6700-DFC3BXL
Cisco 7600/Ciso Catalyst 6500 series with:
WS-SUP720-3B(XL)
WS-X6704-10GE
WS-X6724-SFP
WS-X6748-SFP
OSM-2+4GE-WAN
WS-F6700-DFC3BXL
4GE-SFP-LC
Cisco 12000 Series with:
PRP-2
ISE E3 POS/GE
Note: For distributed mode operation, the Cisco 6500/7600 series line card's DFC should match the supervisor engine's PFC (PFC 3BXL and DFC3 BXL, for example).
Topology 2a represents deployments using a scalable and fully redundant access device using the Cisco Catalyst 6500 Series Supervisor Engine 720 PFC 3B/3BXL. Observe also that the optional service gateway function could be performed at the PE-AGG layer with the presence of an OSM-2+4GEWAN (used for advanced Q-in-Q mapping and not for performing MPLS provider edge functions).
Topology 2b represents deployments in which the Cisco Catalyst 4500 Series is used as U-PE/PE-AGG.
Topology 2c represents a deployment scenario similar to 2a but is suited for a smaller central office access device using the Cisco Catalyst 6500 Series Supervisor Engine 32.
Switched Gigabit Ethernet Ring Topology
Figure 7
Switched Gigabit Ethernet Ring
The switched Gigabit Ethernet ring topology is characterized by the deployment of access equipment following a ring fiber layout (Figure 7). The UNI (through the U-PE) can attach directly to the access or edge devices.
Gigabit Ethernet rings can also aggregate to an optional aggregation device or directly to the edge platforms. A Spanning Tree protocol (IEEE 802.1w/1s or Cisco PVRST+) is implemented in the access to prevent Layer 2 loops in a meshed topology.
Point-to-point and multipoint-to-multipoint VLAN-based (802.1Q trunk UNI), and Port-based (802.1Q tunnel UNI) services can be concurrently offered on this network infrastructure.
Table 8 highlights the benefits and limitations of this topology.
Table 8. Benefits and Limitations of Gigabit Ethernet Ring Topology
Benefits
Limitations
Fits deployments in which fiber availability for a star topology is unfeasible, thus allowing for the interconnection of access devices over a ring layout.
Bandwidth in the ring is shared among the access nodes. Proper capacity planning and engineering or protocol operation must be considered in order to guarantee fair access to this common resource and to protect traffic with tighter SLAs.
The number of interfaces required at the aggregation or edge node is not directly proportional to the number of access nodes.
Sizing the number of nodes in the ring must be done in alignment with the expected traffic restoration times (jitter, latency) in case of ring failures.
Note: Dual-home connections between customer equipment and the service provider network are not considered in Figure 7. For more information about this topic, please contact your Cisco Systems account team.
Table 9 lists the hardware components covered in the tested Cisco Metro Ethernet solution required to deploy this architecture.
Table 9. Gigabit Ethernet Ring Component Product Numbers
U-PE
PE-AGG, U-PE
N-PE, U-PE (in CO)
Access-Facing Line Cards
Chassis/ Supervisor Engines
Edge-Facing Line Cards
Aggregation-Facing Line Cards
Chassis/Supervisor Engines
Core-Facing Line Cards
3a
ME-C3750-24TE
WS-X6724-SFP
WS-X6748-SFP
WS-X6748-GE-TX
WS-F6700-DFC3BXL
Cisco 7600/Cisco Catalyst 6500 series with:
WS-SUP720-3B(XL)
WS-X6724-SFP
WS-X6748-SFP
WS-X6704-10GE
WS-X6748-GE-TX
WS-F6700-DFC3BXL
WS-X6724-SFP
WS-X6748-SFP
WS-X6704-10GE
WS-X6748-GE-TX
WS-X6582-2PA
PA-A6-OC3MM
WS-F6700-DFC3BXL
Cisco 7600/Cisco Catalyst 6500 series with:
WS-SUP720-3B(XL)
OSM-2+4GE-WAN
OSM-1OC48-POS-SI+
3b
WS-C3550-12T
WS-C3550-12G
WS-C3550-24
WS-X4448-GB-RJ45
WS-X4306-GB
WS-X4448-GB-SFP
Cisco Catalyst 4500 Series with:
WS-X4516
WS-X4306-GB
WS-X4448-GB-SFP
WS-X6724-SFP
WS-X6748-SFP
WS-X6704-10GE
WS-X6748-GE-TX
WS-X6582-2PA
PA-A6-OC3MM
WS-F6700-DFC3BXL
Cisco 7600/Cisco Catalyst 6500 Series with:
WS-SUP720-3B(XL)
OSM-2+4GE-WAN
OSM-1OC48-POS-SI+
Note: For distributed mode operation, the Cisco 6500/7600 series line card's DFC should match the supervisor engine's PFC (PFC 3BXL and DFC 3BXL, for example).
Topology 3a applies to deployments with a ring fiber layout and small access devices aggregating multiple customers outside the central office.
Topology 3b represents a deployment with an installed base of Cisco Catalyst 3550 Series access rings in which the organization might consider a migration of the Cisco 7600/Cisco Catalyst 6500 series Supervisor Engine 2 to Supervisor Engine 720 PFC 3B/3BXL.
Integrated Optical Topology
Figure 8
Integrated Optical Topology
The access network in an integrated optical topology could be Ethernet over dark fiber or Ethernet over SONET/SDH while keeping the service characteristics consistent (Figure 8). The SONET/SDH network is provisioned with Resilient Packet Ring (RPR) for ring protection.
A Spanning Tree protocol (IEEE 802.1w/1s or Cisco PVRST+) is implemented in the Layer 2 Ethernet part of the network in order to prevent Layer 2 loops in a meshed topology, but it is not required over SONET/SDH because RPR takes care of Layer 2 redundancy.
Point-to-point and multipoint-to-multipoint VLAN-based (802.1Q trunk UNI), and port-based (802.1Q tunnel UNI) services can be offered concurrently in this network infrastructure. In addition to a switched Ethernet offering, the topology could also support Ethernet Private Line (EPL) services.
Note: Dual-home connections between customer equipment and the service provider's network are not illustrated in. For more information about this topic, please contact your Cisco Systems account representative.
Table 10 highlights the benefits and limitations of this topology.
Table 10. Benefits and Limitation of Integrated Optical Topology
Benefits
Limitations
Fits deployments in which fiber availability for a star topology is unfeasible, thus allowing for the interconnection of access devices over a ring layout.
Advanced data capabilities (Ethernet switching, RPR, and MPLS, for example) might not be present in existing add-drop multiplexer (ADM) and thus require a full upgrade.
The number of interfaces required at the aggregation or edge node is not directly proportional to the number of access nodes.
Cost of equivalent solution based on Ethernet switches and routers might be lower.
Meets 50-millisecond resiliency requirements typical of SONET/SDH networks
Allows a service provider to continue to offer (if desired) traditional time-division multiplexing (TDM) services
Table 11 lists the hardware components covered in the tested Metro Ethernet solution required to deploy this architecture.
Table 11. Integrated Optical Component Details
U-PE
PE-AGG, U-PE
N-PE
Customer Equipment-Facing Line Cards
Chassis/ Common Cards
Aggregation-Facing Line Cards
Access-Facing Line Cards
Chassis/ Common Cards
Edge-Facing Line Cards
Aggregation-Facing Line Cards
Chassis/ Supervisor Engines
Core-Facing Line Cards
4a
15454-ML100T-12
15454-ML1000-2
15454-TCC2
15454-XC-10G
15454-OC12xx
15454-OC48xx
15454-OC12xx
15454-OC48xx
15454-ML100T-12
15454-ML1000-2
15454-TCC2
15454-XC-10G
15454-ML1000-2
WS-X6724-SFP
WS-X6748-SFP
WS-X6748-GE-TX
WS-X6582-2PA
PA-A6-OC3MM
WS-F6700-DFC3BXL
Cisco 7600/Cisco Catalyst 6500 series with:
WS-SUP720-3B(XL)
OSM-2+4GE-WAN
OSM-1OC48-POS-SI+
4b
15310-CE-100T-8
15310-ML-100T-8
15310-CL
OC 12 LC
15454-OC12xx
15454-ML100T-12
15454-ML1000-2
15454-CE-100T-8
15454-TCC2
15454-XC-10G
15454-ML1000-2
WS-X6724-SFP
WS-X6748-SFP
WS-X6748-GE-TX
WS-X6582-2PA
PA-A6-OC3MM
WS-F6700-DFC3BXL
Cisco 7600/Cisco Catalyst 6500 series with: WS-SUP720-3B(XL)
OSM-2+4GE-WAN
OSM-1OC48-POS-SI+
Note: For distributed mode operation, the Cisco 7600/Cisco Catalyst 6500 series line card's DFC should match the supervisor engine's PFC (PFC 3BXL and DFC 3BXL, for example).
1This terminology is described in draft-ietf-l2vpn-12-framework-05.txt and RFC 4026.
2Note that PE-AGG is not named specifically in draft-ietf-l2vpn-12-framework-05.txt but it is acknowledged that a Layer 2 aggregation function can exist. (Section 3.4.3).
