Guest

Metro Ethernet Switching Solution for Service Providers

Enterprise Connections to Layer 2 Ethernet Services Design

Table Of Contents

Design Guide

Abstract

Introduction

Ethernet Services

Ethernet Wire Service

Ethernet Relay Service

Ethernet Multipoint Service

Ethernet Access Security Overview

Denial-of-Service Attacks

Spoofing Attacks

Hierarchically Structured EWAN Design

Hierarchy and Ethernet Service Selection

Containing VLAN, Subnet, and Spanning Tree Domains over the EWAN

Comparing Access Switches and Routers

Flexible Policing and Traffic Shaping

Address Structuring and Traffic Segmentation

Fault Isolation and Traffic Control

SP Value-Added, Service-Friendly Platform

Routers Are a Futureproof Solution

When Is It Acceptable to Use an Access Switch?

Appendix

Configurations

Network Configurations

Attaching to an EMS with Switches

Attaching to an EMS with Routers

Simple ERS with Routers

Large-Scale ERS with Routers

Hybrid EMS- ERS Configuration

Attaching to an EWS with Switches and Routers

Security Configurations

Network Element Level Security

VLAN and Other Layer 2 Security


Design Guide


Enterprise Connections to
Layer 2 Ethernet Services

Abstract

The purpose of this document is to aid in the design of enterprise WANs that connect via Layer 2 Ethernet services. Topics include architecture, security, and switching versus routing as they pertain to the different Ethernet service types. The appendix contains configuration examples for switches and routers connecting to different Ethernet services. The term "routing" will be used to indicate both Layer 3 routing and Layer 3 switching.


Note: Although Layer 3 design issues are discussed in this document, they are in relation to Layer 2 Ethernet service, not Ethernet service networks that operate on Layer 3. Cisco Systems already provides many design documents pertaining Layer 3 service offerings; see http://www.cisco.com/univercd/home/home.htm.


Introduction

In the last several years, significant advances in technology—including bandwidth, quality of service (QoS), multicast, and availability improvements—have taken place in the Ethernet LAN. In the LAN, Ethernet has clearly emerged as the dominant technology—due not only to its simplicity, cost advantages, and ubiquity but also because of its incremental speed advances. In the last four years, the industry has seen a jump from a shared 10-Mbps segment to switched 100 Mbps to switched 1 Gbps to multiple lambdas of 1 Gbps and now to 10 Gbps and multilambda 10 Gbps. Service providers looking to provide higher bandwidth, as well as enhanced services such as QoS, are now looking to Ethernet to allow them to scale the bandwidth offered to enterprise customers for WAN and MAN (metropolitan-area network) applications.

Until now, however, Ethernet MANs have remained relatively dormant. Metro service providers have relied mostly on their SONET/SDH infrastructures to provide data services. Although SONET/SDH is clearly well understood and works as specified, it is not optimized for data traffic. As the bandwidth demands of the WAN and LAN have increased, it has become necessary to match LAN and WAN capacity and transmission speeds in the MAN.

Because of the availability, cost, and speed advances in Ethernet, many service providers are looking to offer their customers metro Ethernet as a connectivity option. Many newly established Ethernet service providers (ESPs), are already offering their customers Ethernet connectivity. Many other incumbent local exchange carriers (ILECs); post, telegram, and telegraph operators (PTTs); and InterLATA exchange carriers (IXCs) are considering or offering Ethernet as a Layer 1 private line service, as a pure Layer 2 transport mechanism, or to provide Internet Protocol (IP) and Multiprotocol Label Switching (MPLS) VPN services to complement their existing SONET/SDH, Frame Relay, or ATM services.

Ethernet offers a viable alternative for increasing the capacity of the ATM and Frame Relay services. Many service providers are looking to limit their spending on their aging ATM and Frame Relay services while offering like services with superior functionality, scalability, and lifetime cost ownership.

Ethernet Services

Cisco and the Metro Ethernet Forum (MEF) endorse three main Layer 2 Ethernet service types. The names of the services differ, but their functionality is the same. They are as follows:

Ethernet Wire Service (EWS)

Ethernet Relay Service (ERS)

Ethernet Multipoint Service (EMS)

When discussing an Ethernet WAN (EWAN), the following terminology should be used (Figure 1):

CE (customer edge): The customer device connecting to the service provider

PE (provider edge): The service provider device connecting to the customer

UNI: The connection between the CE and PE

Multiplexed UNI: A UNI supporting multiple VLAN flows

Pseudowire: A term used to indicate an end-to-end path in a service provider network

Figure 1

EWAN Terms

Ethernet Wire Service

An Ethernet Wire Service is a service that emulates a point-to-point Ethernet segment (Figure 2). This is similar to Ethernet private line (EPL), a Layer 1 point-to-point service, except the provider edge operates at Layer 2 and typically runs over a Layer 2+ network. The EWS encapsulates all frames that are received on a particular UNI and transports these frames to a single-egress UNI without reference to the contents contained within the frame. The operation of this service means that an EWS can be used with VLAN-tagged frames. The VLAN tags are transparent to the EWS (bridge protocol data units [BPDUs])—with some exceptions. These exceptions include IEEE 802.1x, IEEE 802.2ad, and IEEE 802.3x, because these frames have local significance and it benefits both the customer and SP to terminate them locally.

Figure 2

EWS Example

Since the service provider simply accepts frames on an interface and transmits these without reference to the actual frame (other than verifying that the format and length are legal for the particular interface) the EWS is indifferent to VLAN tags that may be present within the customer Ethernet frames.

EWS subscribes to the concept of "all-to-one" bundling. That is, an EWS maps a port on one end to a point-to-point circuit and to a port on another end. EWS is a port-to-port service (Figure 3). Therefore, if a customer needs to connect a switch or router to n switches or routers it will need n ports and n pseudowires or logical circuits.

Figure 3

Nonservice Multiplexing Example: Each Destination (Left) Needs Its Own Port (Right)

One important point to consider is that, although the EWS broadly emulates an Ethernet Layer 1 connection, the service is provided across a shared infrastructure, and therefore it is unlikely that the full interface bandwidth will be, or needs to be, available at all times. EWS will typically be a sub-line rate service, where many users share a circuit somewhere in their transmission path. As a result, the cost will most likely be less than that of EPL. Unlike a Layer 1 EPL, the SP will need to implement QoS and traffic engineering to meet the specific objectives of a particular contract. However, if the customer's application requires a true wire rate transparent service, then an EPL service—delivered using optical transmission devices such as DWDM (dense wavelength division multiplexing), CDWM (coarse wavelength division multiplexing), or SONET/SDH—should be considered.

Ethernet Relay Service

Ethernet Relay Service is similar to EWS in that it offers point-to-point connectivity. The key differentiation between EWS and ERS is that an ERS uses a VLAN tag to multiplex several, non-same-destination pseudowires to one port. That is, unlike EPL and EWS, ERS is a "one-to-many," multiplexed service. Service multiplexing simply means that multiple pseudowires utilize a single access interface or UNI. These circuits can terminate within an L2VPN or on, for example, an Internet gateway. From the service user's perspective, this service multiplexing capability offers more efficient interface utilization, simplification of cable plant, and reduced maintenance costs associated with additional interfaces.

Using the same example as above, where a router connects to n other routers, the source router only needs one port for the service instead of n, as is the case with an EWS. The service need not be port-to-port, but can be logical-pseudowire-to-logical-pseudowire. In the case of an ERS, each circuit can terminate at a different remote location (Figure 4), whereas using EWS, all frames are mapped to a single circuit and therefore a single egress point.

Figure 4

ERS Service Multiplexing Example: One Port (Left) Can Be Used for All Destinations (Right)

Like Frame Relay, ERS allows a customer device to access multiple connections through a single physical port attached to the service provider network. The service offered by ERS can be thought of as being similar in concept to Frame Relay, in that a VLAN number is used as a virtual circuit identifier in a similar fashion to Frame Relay data link connection identifier (DLCI) or an ATM permanent virtual circuit (PVC). Unlike EWS, ERS does not forward BPDUs, because IEEE 802.1Q (VLAN tagging) only sends BPDUs on a default VLAN. In a hub-and-spoke network, only one spoke at most would receive BPDUs, thus breaking the spanning tree in the rest of the network. Therefore, an ERS does not transmit any BPDUs and runs routing protocols instead of Ethernet Spanning Tree. The routing protocols give the customer and provider greater flexibility, traffic determination characteristics, and value-added services.

Ethernet Multipoint Service

An Ethernet Multipoint Service (EMS) differs from EWS and ERS in that an EMS provides a multipoint connectivity model. It should be noted that an EMS service definition is still under review within the IETF Virtual Private LAN Service (VPLS) working group. Although EMS uses a multipoint model, it can forward unicast packets to single destinations; that is, it also supports point-to-point connections. To the end user, the network looks like a giant Ethernet switch where each customer has their own VLAN or broadcast domain, rather than end-to-end pseudowire link(s) (Figure 5).

Figure 5

EMS Example

An EMS does not map an interface or VLAN to a specific point-to-point pseudowire. Instead, it models the operation of a virtual Ethernet switch: EMS uses the customer's MAC address to forward frames to the correct egress UNI within the service provider's network. An EMS emulates the service attributes of an Ethernet switch and learns source MAC to interface associations, floods unknown broadcast and multicast frames, and (optionally) monitors the service user's spanning tree protocol. One important point to note is that although the service provider may utilize spanning tree within the transport network, there is no interaction with the service user's spanning tree.

This service works similar to an MPLS VPN, except it functions at Layer 2 instead of Layer 3. While a VPLS EMS is a viable solution, its scalability and QoS control are suspect compared to that of MPLS VPNs. In addition, it is much more difficult, and may be impossible, for the service provider to offer value-added Layer 3 services (this is discussed later in the document).

Finally, emulating LANs in the metro requires a lot of overhead. EMS and protocols run the risk of turning into ATM LAN Emulations (LANE), which have shown their overcomplexity and inability to scale.

Ethernet Access Security Overview

This section discusses security as it pertains to Layer 2 Ethernet as well as the myth that because an access device operates at Layer 2 it is much more secure than one that operates at Layer 3. This discussion assumes that the customer premises equipment (CPE) is an untrusted network element (NE). The CPE can be a switch or a router.

It is further assumed that the service provider network has been protected against "inside" attack, using procedures to secure access to network devices and that a number of measures have been taken to protect against external attacks against the service provider network. These precautions include hiding the IP core topology and deploying security measures such as packet filtering.

The two main threats to the access network are denial-of-service (DoS) attacks and spoofing attacks.

Denial-of-Service Attacks

DoS attacks are intended to bring a network to a state in which it can no longer carry legitimate users' data. Such attacks commonly take one of two forms: attacking network components or flooding the network with extraneous traffic. An attack is designed to cause a component to stop forwarding packets or to forward them improperly. Network attacks can take the form of a misconfiguration or the injection of a spurious update. A flood attack bombards a device with unroutable or unswitchable packets, causing its performance to degrade. A flood attack on a network is similar to a flood attack on an individual device, except that the flooded packets are usually broadcast.

DoS attacks include the following types:

Content-addressable memory (CAM) overflow: A CAM table is used to determine where to direct incoming frames depending on which port the incoming MAC address came from. When the CAM receives a frame with an unknown destination, the proper procedure is to flood frames within the acceptable Layer 2 domain (the proper VLAN). Hardware and software tools are available (some for free), that can flood a switch with MAC addresses. Once the CAM table limit is exceeded, switches behave differently depending on the brand of the switch.

DoS against Spanning Tree Protocol: Spanning Tree is not an authenticated protocol. A single host can disrupt the stability of an spanning tree topology by impersonating a bridge and sending BPDUs to an access network. When a link on a bridge port is turned up, spanning tree calculation is carried out on that port. The result of the calculation will be the transition of the port into forwarding or blocking state, depending on the position of the port in the network. With IEEE 802.1D, the calculation and transition period takes about 20 to 30 seconds. At this time, no user data is passing via the port. This process can be repeated over and over to permanently disrupt user services.

Dynamic Host Configuration Protocol (DHCP) DoS—One of main features of DHCP is its ability to assign end-station addresses. A threat common to both the client and the server is the DHCP resource DoS attack. This attack occurs when a hacker seizes all the remaining end-station addresses or exhausts the DHCP CPU with an enormous number of requests. The former attack captures all the resources, while the latter overburdens the DHCP engine itself.

DoS storms—A simple form of DoS is for a hacker to send a large number of frames that flood the network. These packets can include broadcast MAC addresses, multicast MAC addresses, nonexistent MAC addresses, and unknown-destination MAC addresses. In all cases, a switch would flood the frame on all ports. Traffic will be flooded within one VLAN, but since trunk ports usually contain many or all VLANs, a whole switch and its corresponding trunks can be affected.

Spoofing Attacks

The objective of DoS attacks is to make a device or network unusable—a state that will be quickly detected by a network's users and administrators. In contrast, spoofing uses a spurious update to cause packets to be routed to a host, from which an intruder may monitor the data in the packets. After examination, these packets are usually reswitched (routed) to their correct destinations. This is known as a "man in the middle" attack. The intruder may or may not have altered the contents of the packets, so these attacks are not always perceived by other network users. In an Ethernet environment different types of spoofing attacks are possible. Attackers can take advantage of the ability to spoof both IP and MAC addresses to carry out DoS attacks and avoid traceability, to hijack a service and avoid billing, or to eavesdrop on traffic.

Types of spoofing attacks include the following:

MAC address hijacking: In any large-scale network based on Ethernet technology with many users per IP subnet, a malicious host can potentially assign itself any IP address. There are two forms of s spoofing attack involving hijacked IP addresses: IP source spoofing and Address Resolution Protocol (ARP) spoofing.

IP source spoofing: Some users change their IP address to a static one (as opposed to automatic assignment by DHCP). This can happen due to ignorance or misconfiguration, or it can be used to hide an attack. Changing the source IP address enables device spoofing and anonymous DoS attacks, and it may enable the attacker to bypass access control lists (ACLs).

ARP spoofing: An attacker can send a gratuitous ARP packet—an ARP reply sent without first receiving an ARP request—with a spoofed source address, causing the default gateway or another host to learn about it and store it in its ARP table. The ARP protocol (RFC 826) will happily create an entry for any such malicious host without performing any type of authentication or filtering. This behavior results in vulnerability to spoofing attacks, and that lets the attacker receive frames intended for another user.

Hierarchically Structured EWAN Design

The development of Layer 2 switching in hardware several years ago led to network designs that emphasized Layer 2 switching. These designs are often characterized as "flat," because instead of relying on the logical, hierarchical structure and summarization provided by routers, they are most often based on the campuswide VLANs model, where a set of VLANs span the entirety of the network. This type of architecture favored the "departmental segmentation approach," in which, for example, Marketing and Engineering needed to exist on the same broadcast domain to avoid crossing "slow" routers. Since these departments could exist anywhere within the network, VLANs had to span the entire network. However, Layer 3+ switching provides exactly the same advantages as routing with the added performance boost from packet forwarding handled by specialized hardware. Adding Layer 3 switching in the distribution layer and backbone of the campus network segments the campus into smaller, more manageable pieces, as defined in several different ways. This approach also eliminates the need for networkwide VLANs, allowing for the design and implementation of a far more scalable architecture. This design strategy is directly applicable to large EWAN networks.

The foundation of the multilayer design is the building block, or module (Figure 6). Smaller networks will likely need only one module, while larger scale networks can use several. The basic building block comprises two layers, the access layer and the distribution layer". When scaling from a smaller to a larger network, a core layer is included as well. The access layer collects the users or sites and presents them to the network. The distribution layer collects the access links on a per-region basis. Intraregional traffic can be forwarded directly by the distribution layer. In the case where access sites are small branch or SOHO sites, the distribution layer provides services such as firewalling and policy management. The core layer is used for high-speed forwarding between regions or distribution points. Services such as firewalling, policy management, server farms, and Internet POP handoff occurs at core sites.

Figure 6

Hierarchical EWAN Network

Figure 6 shows seven sites: three access, two distribution, and two core network elements. In actuality, such an network would probably have 20 or more access nodes to support the two layers shown. In the case of smaller networks, the core layer and distribution layer can be collapsed into a single layer. Still smaller networks can be collapsed to the access layer only; here each access node would make up its own domain. Regardless of the size of the network and number of layers, the design principles are the same, even if the number of layers differs.

Access links are usually single links to the EWAN. However, multiple links running IP load balancing, Cisco EtherChannel® technology, or spanning tree can be used for redundancy. In addition, access links are usually not multihomed to more than one distribution point. However, they may be multihomed for high-availability sites. Core links are usually meshed based upon traffic load, and there should always be more than one path to and from each node, with no single point of failure.

The multilayer design model is highly deterministic and fault tolerant. It is also easier to troubleshoot than a flat network, and its scalability advantages are unrivaled. The modular building-block approach scales easily as new buildings or server farms are added to the enterprise. Intelligent Layer 3 routing protocols such as OSPF can handle load balancing and fast convergence across the backbone. Many value-added services in Cisco IOS® software, such as route summarization, DHCP relay, and intelligent multicast handling, are implemented in the Cisco Catalyst® multilayer switches at the distribution layer. Access policies are also implemented with access lists at the access, distribution, or distribution switches.

Hierarchy and Ethernet Service Selection

Simply stated, if the enterprise network requires a dual hierarchy (access, distribution, and core), then an ERS should always be used. If an EMS is used (Figure 7), the network turns into one large broadcast domain, where the access switches and routers will bypass the distribution routers and peer directly with the core routers. This will wreak havoc with traffic patterns, security, and any special services run within the enterprise.

Figure 7

EMS Will Cause Routers to Form Meshed Adjacencies, Ruining a Hierarchical Network

One way around this problem would be to have multiple instances of EMS between core and distribution routers and distribution and access routers. However, this is not recommended, because it would cost extra to support multiple instances of a single service. Also, the number of ports needed on the distribution routers will at least double, because they have to attach to both layers in the hierarchy. Therefore, if the network is large enough to need a dual-layer hierarchy, ERS is the service of choice, because it allows the circuits to connect between two specified and distinct points.

Containing VLAN, Subnet, and Spanning Tree Domains over the EWAN

A Layer 2 switched domain can be considered a failure domain because a misconfigured or malfunctioning workstation, server, or switch can negatively affect or disable the entire domain by flooding it with broadcasts or undesirable frames. A protocol malfunction (spanning-tree error or misconfiguration) can inhibit a large part of a network. Problems of this nature can be very difficult to localize, especially in a flat network.

The scope of a failure domain should therefore be reduced as much as possible. The best way to achieve this is by restricting its scope to a single EWAN link. In other words, only one unique VLAN should exist per EWAN link. Over the WAN, instead of traffic types being differentiated with VLAN IDs, traffic will be differentiated by class of service (CoS), type of service (ToS), or differentiated services code point (DSCP).

IP subnets come into play if routers are used. If all the network elements attaching to the EWAN are switches (not recommended) the IP structure of the enterprise is simply superimposed over the WAN. In a network where all the WAN network elements are routers, each access link should have a unique IP domain (prefix). At the distribution layer, these prefixes should be summarizable to a single prefix and in turn summarizable to the core. In a hybrid network where one or more layers are switches, unique and summarizable IP domains should be configured at the lowest layer where routers are present. By implementing a sensible IP addressing scheme, Layer 3 switches gain the ability to exchange summarized routing information, rather than having to learn a path to every host in the network. Summarization is the key to the scalability of routing protocols such as Open Shortest Path First (OSPF) and Border Gateway Protocol (BGP).

Containing VLANs and summarizing IP addresses is a good start. However, if the two are not synchronized, most of their benefits may be lost. All the EWAN IP subnets should map to the Layer 2 VLANs; therefore, if the IP subnet is the logical Layer 3 equivalent of the VLAN, at Layer 2 one VLAN = one subnet. The IP network address is defined at the Layer 3 switch where the Layer 2 switch domain terminates. This one-to-one mapping contains MAC-layer broadcasts, multicasts, and unknown unicasts flooded throughout the Layer 2 domain, such as those that could flood the EWAN and cripple an entire organization. In addition, intelligent, protocol-aware features of routers will further contain broadcast packets. Flooding of multicast traffic can be constrained to a community of interested ports by using Internet Group Management Protocol (IGMP), IGMP snooping, and Protocol Independent Multicast (PIM).

Another advantage of the one-to-one mapping of EWAN link-IP subnet-VLAN is that if spanning tree is used, the spanning tree domain is contained between each EWAN network element. Although Hot Standby Router Protocol (HSRP), IP load balancing, and EtherChannel technology provide better link utilization and failover times than spanning tree, when spanning tree is used for redundancy, the spanning tree domain is limited to the link group, not the entire network. Thus, when a link outage occurs it does not cause a topology change throughout the network and degrade service.

Finally, access policies are should be defined on the routers over the EWAN. A convenient way to define policy is with ACLs that apply to an IP subnet. Thus, a group of servers with similar access policies can be conveniently grouped together in the same IP subnet and the same VLAN. Other important services, such as DHCP are also defined on an IP-subnet basis.

Comparing Access Switches and Routers

An important question regarding Layer 2 EWAN services is, "Is it better to attach to the service with a switch or a router?" Unfortunately, there is no one correct answer. Clearly there are come CAPEX advantages when one looks at the price tag of a switch versus the price tag of a router. Switches are almost always less expensive than routers, and for some networks, cost is the main or only issue. In that case a switch will be used, regardless of any networking issues. However, in most cases a router is the better choice and will save the enterprise money in the long run, because they offer the following advantages:

Flexible policing and traffic shaping

Address structuring for traffic segmentation

Fault isolation and traffic control

A value-added service-friendly platform for service providers

A futureproof solution

Flexible Policing and Traffic Shaping

"Policing" is the ability to look at packets, compare them to a traffic contract, and either pass them, drop them, or mark them as nonconforming. It is a common misconception that policing alone provides complete traffic engineering—that is, that if the flow of packets is restricted into a network cloud, congestion will not occur. Although this may be true for a grossly underbooked, inefficient networks, it is not true for most "real" networks. When you simply restrict traffic into a cloud, important aspects—including traffic patterns, application-specific QoS issues, and time-of-day usage—must be considered. Even under the most thorough traffic analysis, many nondeterministic traffic patterns can still occur—especially with an EMS—any of which can cause a network element, or port, to congest and drop critical traffic. Since an EMS is a broadcast domain, its QoS characteristics are very unpredictable and can easily congest upon egress (Figure 8).

Figure 8

Policed, but Congested, Network

Even though policing in its own right does not constitute robust traffic engineering, it still plays a vital role in maintaining a congestion-free network. Although many switches can police, they do not have the same policing capabilities as routers. Many switches can police on a per-port basis, on IEEE 802.1P priority, and some can police on an IEEE 802.1Q VLAN. However, most routers can also do this, as well as police on IP ToS, DSCP, TCP port, UDP port, and IP address. Thus, with routers the granularity of policing can be based on IP level priorities, applications such as voice over IP (VoIP), and internal web applications, or even end stations, such as file servers or storage devices. This breadth of service enables the enterprise network to get the best use out of their expensive and critical wide-area infrastructure.

As stated earlier, policing at the edge cannot solve every problem, especially when you are trying to get every bit out of expensive WAN links. Traffic shaping adds another dimension to congestion avoidance and control. "Traffic shaping" is the ability of a router, under congestion or under traffic contract violation, to buffer and smooth traffic to an acceptable rate until the congestion or violation has abated. This feature is common on routers but seldom found on switches. Even if a switch can shape, it has the same limitations, compared to routers, as policing does. Figure 9 shows a 50-Mbps contract over a 100-Mbps link. Traffic is first policed to conform to 50 Mbps, and then the excess bandwidth is throttled, or shaped, so it does not have to be dropped. Most switches would simply drop the extra traffic at the policer.

Figure 9

Policing and Shaping

Address Structuring and Traffic Segmentation

With IP, each end station and router has a configurable address. Although some switches and network adapters allow you to customize the MAC address, this has only a few uses, because these addresses cannot be structured or summarized like IP addresses. Summarization allows large multiples of IP addresses, in a structured system, to be stored in memory as a single-entry summary, rather than individually. This reduces memory sizes, reduces address lookup times, aids in debugging, and reduces failure and recovery time because the device needs only to relearn a summary or a group of summaries rather than a complete list of addresses. Switches do not enjoy this luxury, and since many switches cannot learn addresses at line rate, after network failure, traffic is flooded while a switch tries to relearn its forwarding table. This only exacerbates the congestion and QoS problems associated with the failure.

Summaries can also be hierarchical. For example, a common scheme is to hierarchically vary summarizations as they relate to the access nodes, distribution layer, and core of a network. However, VLANs, like MAC addresses, cannot be hierarchically summarized. Even a VLAN tag-stacking scheme is a one-to-one mapping of customer to tag and is only used by an service provider. Thus, Ethernet cannot duplicating an IP or MPLS hierarchy. By using hierarchies, you can segment broadcast domains. This means that broadcast storms, intentional and unintentional, can be contained to small communities of interest that, under strain, do not affect the rest of the network.

Many protocols also help make Layer 3 multicasting even more efficient. Host-to-router protocols, such as IGMP, and router-to-router protocols such as PIM allow routers to create minimal tree multicast structures, ensuring that the multicast packets traverse only those links destined for valid destinations, rather than being broadcast (Figure 10). If this were done over a Layer 2 network, the multicast would be flooded throughout the Layer 2 domain (Figure 11). Furthermore, even when a switch performs IGMP and PIM snooping, there are issues regarding failure-recovery behavior and, more importantly, QoS, which limits these features' potential and predictability in large networks.)

Figure 10

Desired Effect When Multicasting

Figure 11

Adverse Effects of Multicasting over a Layer 2 Ethernet Switched Network

Fault Isolation and Traffic Control

Another important quality of traffic segmentation is fault isolation. Since traffic can be highly segmented, when issues arise they are constrained to smaller areas, allowing them to be located, and thus fixed, more quickly—lessening the mean time to repair (MTTR). Also aiding MTTR is the structured nature of IP and the vast array of tools that take advantage of this structure. Two common tools often taken for granted are ping and traceroute. These simple tools allow you to determine if a host or router is reachable at the network level, pointing to potential application-layer issues. Traceroute also lists the path that a packet takes as it traverses the network layer, pinpointing the beginning of the failure. In addition to these basic yet effective commands are the vast array of proprietary or management-based software tools available on management systems and protocol analyzers. In contrast, even simple tools like ping and traceroute have no counterparts in the switching world. In addition, as packets traverse from one Layer 2 boundary to another, they pass through routers, and at that point any Layer 2 packet trace loses its end-to-end significance, rendering it useless. The availability of these Layer 3 tools makes it easier to debug a network-not to mention that it reduces the number of expensive ($50,000 and up) protocol analyzers you need to buy.

Finally, routing protocols offer greater flexibility and control over to the path a packet takes through the network and how the topology reacts to change (Figure 12). Spanning Tree Protocol, the most commonly used Layer 2 topology protocol, has several shortcomings when it comes to large-scale networking. It is slow to react, needs to block links (and in most cases many links) rendering them useless, has trunking support protocols (IEEE 802.17AD) that cannot share loads over multiple-nonparallel links, and it cannot forward based on policies. On the other hand, link-state-based routing protocols can quickly react to and repair large-scale networks and can forward and balance loads over any number of links based on policies such as source, destination, route priority, and congested transit network. All of this allows you to use your network efficiently. Another shortcoming of large-scale flat Spanning Tree Protocol networks is that, to keep the network properly utilized and maintaining proper QoS the service provider has to reengineer the network every time a new customer is added. In contrast, routing protocols self-adapt as new users come online.

Figure 12

Inadequacies of Spanning Tree Protocol Versus Routing Protocols

SP Value-Added, Service-Friendly Platform

Another big advantage gained when choosing a router as an access device is that the service provider's ability to deploy value-added services grows. Simply put, routers are feature rich. They are so because all the important features reside above Layer 2, allowing the enterprise to outsource high-touch services and save money. These services range from security to voice management to storage integration. This allows an enterprise to streamline its network needs by combining extranet resources and sending them over one service and one WAN port.

A common example of such a service is outsourced firewalling—an enterprise's frontline defense, which authenticates and controls outside access. IP Security (IPSec) VPNs offer additional security by authenticating sources and encrypting data before it passes over WAN links. IPSec is commonly used by financial institutions and government authorities to protect their data. In addition, a routed access network allows the service provider or enterprise to deploy intrusion-detection software to detect and locate hackers.

Voice-related services include Survivable Remote Site Telephony (SRST) and IP Centrex. SRST detects failures (unreachable destinations) in the network, and then takes IP telephony calls and reroutes them to the public telephone network, rather than dropping the calls because the destination is no longer reachable. For those who do not wish to manage their own IP telephony system but still want the cost savings associated with an integrated voice and data network, service providers offer IP Centrex, a remote service that offers call-management features including voice mail.

Another exciting area that can be integrated is IP storage. IP enhances traditional storage networks by allowing storage area network (SAN) traffic to be multiplexed along with voice and data traffic. In addition, IP storage can apply IP structuring and VLAN concepts creating virtual storage communities that allow an enterprise to better utilize its facilities. These virtual storage communities can either be managed by the enterprise or outsourced to the service provider.

Routers Are a Futureproof Solution

Routers are the right technology for the future, not just today. Whereas switches cannot act as routers, most access enterprise routers can also act as full Layer 2 switches. More importantly, these routers are based on IP and they run IP protocols. The primary purpose of an IP routing protocol is to scale—IP routing protocols were developed because networks became too large to run without them.

In addition, for service providers to continue to operate, they must continue to develop value-added features that provide real benefits to their customers. This requires routers. In addition, many routers now are application aware—and even when they are not, application developers bind their applications to TCP and UDP ports and make it possible for their applications to write to ToS and DSCP priority fields. These hooks are not readily available to switches, and without them a service provider's ability to provide today's or future value-added services becomes difficult and expensive. Thus, today's routers are "futureproof" in that they will continue to scale and provide new and beneficial services to meet tomorrow's needs.

When Is It Acceptable to Use an Access Switch?

Given that switches typically cost less than a router, using a switch as an access node merits consideration, even though there are very inexpensive routers on the market. There are two cases in which it is acceptable to deploy a switch as an access node. The first occurs over a point-to-point link using EPL. In this case, the EWAN link appears to be another segment in a LAN, and EPL is secure in that it does not terminate on any extranet services (Figure 13). The second case occurs when dark fiber when is used, for the same reasons as with EPL. If you use an access switch with EPL, or EWS a hub-and-spoke topology should be implemented; the spoke nodes are switches that hub back to a router. Furthermore, hub-to-spoke traffic is switched, while spoke-to-spoke traffic is routed, for security and traffic segmentation reasons. This network design is commonly found in school districts and remote medical clinics or offices. You should never use a switch with an EMS because QoS, security, and traffic patterns are unpredictable.

Figure 13

Using Switches over EPL, Mapping Each Circuit to a Unique VLAN

Appendix

Configurations

This section has two main subsections: "Network Configurations" and "Security Configurations." The network section covers the configurations for the different network solutions. The security section discusses general network element and Layer 2 security configurations that should be in place when attaching to an EWAN. These configurations include general QoS and OSPF configurations. For an exhaustive list of configurations, consult the QoS and OSPF (Open Shortest Path First) design guides or the appropriate manual.

Network Configurations

This section first describes the solution parameters and then moves on to configure details. The following six solutions are covered:

Attaching to an EMS with switches

Attaching to an EMS with routers

Simple ERS with routers

Large-scale ERS with routers

Hybrid EMS- ERS service configuration

Attaching to an EWS with switches and routers

Attaching to an EMS with Switches

Figure 14 shows a network of three switches connected to an EMS. There are two service types: a high-priority gold service, and a low-priority best effort service. The switches will be configured to see the network as one common Layer 2 broadcast domain/VLAN.

Figure 14

Switches Connecting to an EMS

Keys to this implementation are as follows:

1. All WAN-facing ports or LAN ports with traffic destined for the WAN are in the same VLAN.

2. IGMP snooping is used to prune multicast flows.

Comments:

All switches use the same configuration, because EMS emulates a VLAN/Layer 2 broadcast domain. Of course, the IP addresses will differ at each switch.

Regarding QoS contracts, the distribution switch (switch 3) may need to support more traffic (more bits per second) than the other nodes, because it may be a headquarters. In that case, switch 3's QoS contracts would be larger than those of switches 1 and 2.

The WAN ports of VLAN 1 can be configured as IEEE 802.1Q VLAN tagged or untagged. This example demonstrates tagged VLANs. Other examples, using routers, show untagged VLANs.

switch1> enable
switch1# config t

Create VLAN 1, the WAN VLAN, and give it an IP address.

switch1(config)#int vlan 1
switch1(config-if)#ip address 10.1.1.1 255.255.0.0

Make Interface Gigabit Ethernet 0/11 an IEEE 802.1Q trunk.

switch1(config-if)#int gi0/11
switch1(config-if)# switchport trunk encapsulation dot1q
switch1(config-if)# switchport mode trunk
switch1(config-if)# exit

Create VLAN 1, the LAN VLAN, and give it the same ID as the WAN VLAN ID. Or, if a different VLAN ID is configured, the two must be bridged together.

switch1(config)# int vlan 1

Make Interface Gigabit Ethernet 0/1 an untagged access VLAN. Alternatively, it can be configured to be an IEEE 802.1Q trunked VLAN. This depends on the configuration of the attached LAN.

switch1(config-if)#int gi0/1
switch1(config-if)#switchport access vlan 1

Map CoS bits to DSCP bits.

switch1(config)# mls qos map cos-dscp

Create a class map for best effort traffic.

switch1(config)# class-map be

Make all packets with a DSCP value of 0 be matched to the best effort class.

switch1(config-cmap)# match-all be
switch1(config-cmap)# match ip dscp 0
switch1(config-cmap)# exit

Create a class map for gold service.

switch1(config)# class-map gold

Make all packets with a DSCP value of 3 be matched to the gold service class.

switch1(config-cmap)# match-all gold
switch1(config-cmap)#match ip dscp af31 af32 af33
switch1(config-cmap)# exit

Create a policy map for the WAN port, and police packets matched to class maps "be" and "gold".

switch1(config)# policy-map ewanpolicy
switch1(config-pmap)# class be
switch1(config-pmap-c)# trust dscp

Create a policer to police packets that match to class "be".

Police the flow at a sustained rate of 20 Mbps, with a maximum burst size of 500 kBps, and drop all nonconforming traffic.

switch1(config-pmap-c)# police 20000000 500000 exceed-action drop
switch1(config-pmap-c)# exit

Create a policer to police packets that match to class "gold".

Police the flow at a sustained rate of 10 Mbps, with a maximum burst size of 250 kBps, and downgrade all nonconforming traffic.

switch1(config-pmap)# class gold
switch1(config-pmap-c)# trust dscp
switch1(config-pmap-c)# police 10000000 250000 exceed-action policed-dsp-transmit
switch1(config-pmap-c)# exit
switch1(config-pmap)# exit

Map the policy "ewanpolicy" to the outgoing traffic on the WAN port.

switch1(config)# interface gi0/11
switch1(config-if)# service-policy output ewanpolicy
switch1(config-if)# exit

Turn on IGMP snooping on the WAN interface.

switch1(config)#int gi0/11
switch1(config-if)#ip igmp snooping

Save the configuration.

switch1(config)# write t

Attaching to an EMS with Routers

Figure 15 shows a network of three routers connected to an EMS. There are two service types: a high-priority gold service and a low-priority best effort service. The switches will be configured to see the network as one common Layer 2 broadcast domain/VLAN. However, the traffic flows between the two access switches will be forced to go through the distribution router.

Figure 15

Routers Connecting to an EMS

Keys to this implementation are as follows:

1. All WAN-facing ports are in the same VLAN.

2. All routers are in the same OSPF area.

3. All LAN ports have VLAN IDs that differ from the WAN VLANs' IDs.

4. IGMP smnooping and PIM are used to prune multicast flows.

Comments:

1. All routers use the same configuration, because EMS emulates a VLAN/Layer 2 broadcast domain. Of course, the IP addresses and other IP information will differ at each switch.

2. Regarding QoS contracts, the distribution router (router 6), may need to support more traffic (more bits per second) than the other nodes because it may be a headquarters. In that case router 6's QoS contracts would be larger than those of routers 4 and 5.

3. VLAN 2, the WAN VLAN, can be configured as IEEE 802.1Q VLAN tagged or untagged. This example demonstrates untagged VLANs. Other examples, using switches, show tagged VLANs.

4. In this example, a static route is in place at routers 4 and 5. It forces the traffic from routers 4 and 5 to transit via router 6, instead of going directly between routers 4 and 5. This may be desirable in hierarchical networks, because traffic for certain services or destinations may need to first pass through a central policy server or central firewall. In this case the policy server/firewall is located at router 6. Note: although the traffic from routers 4 and 5 pass through router 6, routers running OSPF will still form adjacencies because they discover each other via multicasting. Since the EMS is a Layer 2 broadcast domain, the OSPF multicasts will cause all the routers in the WAN network to create a fully connected mesh of adjacencies.

router4> enable
router4# config t

Create VLAN 2, the WAN VLAN, and give it an IP address.

router4(config)#int vlan 2
router4(config-if)#ip address 10.1.1.4 255.255.0.0

Give the Interface Gigabit Ethernet 0/11 an untagged VLAN.

router4(config-if)#int gi0/11
router4(config-if)# switchport access vlan 2
router4(config-if)# exit

Create VLAN 101, the LAN VLAN, and give it a VLAN ID different from the WAN VLAN ID.

router4(config)# int vlan 101

Make Interface Gigabit Ethernet 0/1 an IEEE 802.1Q trunked VLAN. Alternatively, it can be configured to be an untagged access VLAN. This depends on the configuration of the attached LAN.

router4(config-if)#int gi0/1
router4(config-if)# switchport trunk encapsulation dot1q
router4(config-if)# switchport mode trunk

Map CoS bits to DSCP bits.

router4(config)# mls qos map cos-dscp

Create a class map for best effort traffic.

router4(config)# class-map be

Make all packets with a DSCP value of 0 be matched to the "best effort" class.

router4(config-cmap)# match-all be
router4(config-cmap)# match ip dscp 0
router4(config-cmap)# exit

Create a class map for "gold" service.

router4(config)# class-map gold

Make all packets with a DSCP value of 3 be matched to the "gold" service class.

router4(config-cmap)# match-all gold
router4(config-cmap)#match ip dscp af31 af32 af33
router4(config-cmap)# exit

Create a policy map for the WAN port, and police packets matched to class maps "be" and "gold".

router4(config)# policy-map ewanpolicy
router4(config-pmap)# class be
router4(config-pmap-c)# trust dscp

Create a policer to police packets that match to class "be".

Police the flow at a sustained rate of 20 Mbps, with a maximum burst size of 500 kBps, and drop all nonconforming traffic.

router4(config-pmap-c)# police 20000000 500000 exceed-action drop
router4(config-pmap-c)# exit

Create a policer to police packets that match to class "gold".

Police the flow at a sustained rate of 10 Mbps, with a maximum burst size of 250 kBps, and downgrade all nonconforming traffic.

router4(config-pmap)# class gold
router4(config-pmap-c)# trust dscp
router4(config-pmap-c)# police 10000000 250000 exceed-action policed-dsp-transmit
router4(config-pmap-c)# exit
router4(config-pmap)# exit

Map the policy "ewanpolicy" to the outgoing traffic on the WAN port.

router4(config)# interface gi0/11
router4(config-if)# service-policy output ewanpolicy
router4(config-if)# exit

Create an OSPF instance, and give it an arbitrary process ID.

router4(config)# router ospf 10

Place the OSPF instance in Area 0, unless a backbone area 0 already exists somewhere else in the network. In that case, use some other, unused area.

router4(config-router)# network 10.1.0.0 0.0.255.255 area 0

Create the static route from router 4 (10.1.1.4) to router 5 (10.1.2.5) via router 6(10.1.31.6). If desired, a default route may be used instead.

router4(config)# ip route 10.1.2.0 10.1.3.6

Enable IP routing.

Router4(config)# ip routing

Enable multicast routing, PIM.

router4(config)# ip multicast-routing
router4(config)int gi0/11
router4(config-if)#ip pim sparse-dense
router4(config-if)# exit

Save the configuration.

router4(config)# write t

Simple ERS with Routers

Figure 16 shows a network of three routers connected to an ERS. Two point-to-point circuits run from the access routers to the distribution router. There are three service types: a voice service, a high-priority gold service, and a low-priority best effort service. The routers will be configured to see the network as a series of circuits, each with its own VLAN and IP subnet.

Figure 16

Routers Connecting to an ERS

Keys to this implementation are the following:

1. All WAN-facing ports of the access routers are in different VLANs. The distribution router, router 9, is a member of both WAN VLANs.

2. All WAN router ports are in the same OSPF area.

3. All LAN ports have VLAN IDs that differ from the WAN VLANs' IDs.

4. IGMP and PIM are used to prune multicast flows.

Comments:

All routers do not use the same configuration. Note: The prompt router n will change when different commands need to be entered on different routers. Those sections will be in blue bold italics. All other commands are common to all the routers.

Regarding QoS contracts, the distribution router (router 9) may need to support more traffic (more bits per second) than the other nodes, because it may be a headquarters. In that case router 9's QoS contracts would be larger than those of routers 7 and 8.

VLANs 7 and 8, the WAN VLANs, can be configured as untagged or as IEEE 802.1Q VLAN tagged. This example demonstrates tagged VLANs. Other examples, using switches, show tagged VLANs.

Router7> enable
router7# config t

Create VLAN 7, the WAN VLAN on router 7. Give it an IP address, and make it an 802.1Q trunked port.

router7(config)#int vlan 7
router7(config-if)#ip address 10.1.2.7 255.255.255.0
router7(config)#int gi 0/11
router7(config-if)# switchport trunk encapsulation dot1q

Create VLAN 8, the WAN VLAN on router 8. Give it an IP address, and make it an 802.1Q trunked port.

router8(config)#int vlan 8
router8(config-if)#ip address 10.1.3.8 255.255.255.0
router8(config)#int gi 0/11
router8(config-if)# switchport trunk encapsulation dot1q

Create VLANs 7 and 8, the WAN VLAN on router 9, and give them IP addresses that map the proper subnet to the proper VLAN.

router9(config)#int vlan 7
router9(config-if)#ip address 10.1.2.9 255.255.255.0
router9(config-if)#ip exit
router9(config)#int vlan 8
router9(config-if)#ip address 10.1.3.9 255.255.255.0
router9(config-if)#ip exit
router9(config)#int gi 0/11
router9(config-if)# switchport trunk encapsulation dot1q
router9(config)#siwtchport trunk allowed vlan 7 8
router7(config-if)# exit

Create VLAN 102, the LAN VLAN, and give it a VLAN ID different from the WAN VLAN IDs.

router7(config)# int vlan 102

Make Interface Gigabit Ethernet 0/1 an IEEE 802.1Q trunked VLAN. Alternatively, it can be configured to be an untagged access VLAN. This depends on the configuration of the attached LAN.

router7(config-if)#int gi0/1
router7(config-if)# switchport trunk encapsulation dot1q
router7(config-if)# switchport mode trunk

Map CoS bits to DSCP bits.

router7(config)# mls qos map cos-dscp

Create a class map for best effort traffic.

router7(config)# class-map be

Make all packets with a DSCP value of 0 be matched to the "best effort" class.

router7(config-cmap)# match-all be
router7(config-cmap)# match ip dscp 0
router7(config-cmap)# exit

Create a class map for "gold" service.

router7(config)# class-map gold

Make all packets with a DSCP value of 3 be matched to the "voice" services class.

router7(config-cmap)# match-all gold
router7(config-cmap)#match ip dscp af31 af32 af33
router7(config-cmap)# exit
router7(config)# class-map voice

Make all packets with a DSCP value of "ef" be matched to the "voice" services class ("ef" is the DSCP value for IP telephony; you could also match on RTP values).

router7(config-cmap)# match-all voice
router7(config-cmap)#match ip dscp ef
router7(config-cmap)# exit

Create a policy map for the WAN port, and police packets matched to class maps "be" and "gold".

router7(config)# policy-map ewanpolicy
router7(config-pmap)# class be
router7(config-pmap-c)# trust dscp

Create a policer to police packets that match to class "be".

Police the flow at a sustained rate of 20 Mbps, with a maximum burst size of 500 kBps, and drop all nonconforming traffic.

router7(config-pmap-c)# police 20000000 500000 exceed-action drop
router7(config-pmap-c)# exit

Create a policer to police packets that match to class "gold".

Police the flow at a sustained rate of 10 Mbps, with a maximum burst size of 250 kBps, and downgrade all nonconforming traffic.

router7(config-pmap)# class gold
router7(config-pmap-c)# trust dscp
router7(config-pmap-c)# police 10000000 250000 exceed-action policed-dsp-transmit
router7(config-pmap-c)# exit

Create a policer to police packets that match to class "voice".

Police the flow at a sustained rate of 5 Mbps, with a maximum burst size of 50 kBps, and drop all nonconforming traffic.

router7(config-pmap)# class voice
router7(config-pmap-c)# trust dscp
router7(config-pmap-c)# police 5000000 50000 exceed-action drop
router7(config-pmap-c)# exit
router7(config-pmap)# exit

Map the policy "ewanpolicy" to the outgoing traffic on the WAN port.

router7(config)# interface gi0/11
router7(config-if)# service-policy output ewanpolicy
router7(config-if)# exit

Create an OSPF instance, and give it an arbitrary process ID.

router7(config)# router ospf 10

Place the OSPF instance in area 0, unless a backbone area 0 already exists somewhere else in the network. In that case, use some other, unused area.

router7(config-router)# network 10.1.0.0 0.0.255.255 area 0

Enable IP routing.

Router7(config)# ip routing

Enable multicast routing, PIM.

router7(config)# ip multicast-routing
router7(config)int gi0/11
router7(config-if)#ip pim sparse-dense
router7(config-if)# exit

Save the configuration.

router7(config)# write t

Large-Scale ERS with Routers

Figure 17 shows a hierarchical network of four routers connected to an ERS. In the case of a large network that needs to be configured hierarchically, there would probably be two or more core routers, six or more distribution routers, and many access routers; this example is a simplified version. Two point-to-point circuits run from the access routers to the distribution router, and one runs from the distribution router to the core router. There are three service types: a voice service, a high-priority gold service, and a low-priority best effort service. The routers will be configured to see the network as a series of circuits, each with its own VLAN and IP subnet.

Figure 17

Hierarchically Routed Network Connecting to an ERS

Keys to this implementation are as follows:

1. All WAN-facing ports of the access routers are in different VLANs. The distribution router, router 9, is a member of all three VLANs.

2. The core router is a member of area 0. The access routers are members area 1. And the distribution router is a member of both areas.

3. All LAN ports have VLAN IDs that differ from the WAN VLANs' IDs.

4. IGMP and PIM are used to prune multicast flows.

Comments:

All routers do not use the same configuration. Note: the prompt router n will change when different commands need to be entered on different routers. Those sections will be in blue bold italics. All other commands are common to all the routers.

Regarding QoS contracts, the distribution and core routers may need to support more traffic (more bits per second) than the other nodes, because they may be a headquarters. In that case, routers 12 and 13's QoS contracts would be larger than those of routers 10 and 11.

VLANs 9, 10, and 11 (the WAN VLANs) can be configured as untagged or as IEEE 802.1Q VLAN tagged. This example demonstrates tagged VLANs. Other examples, using switches, show tagged VLANs.

Router10> enable
router10# config t

Create VLAN 9, the WAN VLAN on router 10. Give it an IP address and make it an 802.1Q trunked port.

router10(config)#int vlan 9
router10(config-if)#ip address 10.1.2.10 255.255.255.0
router10(config)#int gi 0/11
router10(config-if)# switchport trunk encapsulation dot1q

Create VLAN 10, the WAN VLAN on router 11. Give it an IP address, and make it an 802.1Q trunked port.

router11(config)#int vlan 10
router11(config-if)#ip address 10.1.3.11 255.255.255.0
router11(config)#int gi 0/11
router11(config-if)# switchport trunk encapsulation dot1q

Create VLAN 11, the WAN VLAN on router 13. Give it an IP address, and make it an 802.1Q trunked port.

router13(config)#int vlan 11
router13(config-if)#ip address 10.1.1.13 255.255.255.0
router13(config)#int gi 0/11
router13(config-if)# switchport trunk encapsulation dot1q

Create VLANs 9, 10, and 11, the WAN VLAN on router 12. Give them IP addresses that map the proper subnet to the proper VLAN.

router12(config)#int vlan 9
router12(config-if)#ip address 10.1.1.13 255.255.255.0
router12(config-if)#ip exit
router12(config)#int vlan 10
router12(config-if)#ip address 10.1.1.13 255.255.255.0
router12(config-if)#ip exit
router12(config)#int vlan 11
router12(config-if)#ip address 10.1.1.13 255.255.255.0
router9(config)#int gi 0/11
router9(config-if)# switchport trunk encapsulation dot1q
router9(config)#siwtchport allowed vlan 9 10 11
router10(config-if)# exit

Create VLAN 103, the LAN VLAN, and give it a VLAN ID different from the WAN VLAN IDs.

router10(config)# int vlan 103

Make Interface Gigabit Ethernet 0/1 an IEEE 802.1Q trunked VLAN. Alternatively, it can be configured to be an untagged access VLAN. This depends on the configuration of the attached LAN.

router10(config-if)#int gi0/1
router10(config-if)# switchport trunk encapsulation dot1q
router10(config-if)# switchport mode trunk

Map CoS bits to DSCP bits.

router10(config)# mls qos map cos-dscp

Create a class map for "best effort" traffic.

router10(config)# class-map be

Make all packets with a DSCP value of 0 be matched to the "best effort" class.

router10(config-cmap)# match-all be
router10(config-cmap)# match ip dscp 0
router10(config-cmap)# exit

Create a class map for "gold" service.

router10(config)# class-map gold

Make all packets with a DSCP value of 3 be matched to the voice service class.

router10(config-cmap)# match-all gold
router10(config-cmap)#match ip dscp af31 af32 af33
router10(config-cmap)# exit
router10(config)# class-map voice

Make all packets with a DSCP value of "ef" be matched to the voice services class ("ef" is the DSCP value for IP telephony; you could also match on RTP values.).

router10(config-cmap)# match-all voice
router10(config-cmap)#match ip dscp ef
router10(config-cmap)# exit

Create a policy map for the WAN port, and police packets matched to class maps "be" and "gold".

router10(config)# policy-map ewanpolicy
router10(config-pmap)# class be
router10(config-pmap-c)# trust dscp

Create a policer to police packets that match to class "be".

Police the flow at a sustained rate of 20 Mbps, with a maximum burst size of 500 kBps, and drop all nonconforming traffic.

router10(config-pmap-c)# police 20000000 500000 exceed-action drop
router10(config-pmap-c)# exit

Create a policer to police packets that match to class "gold".

Police the flow at a sustained rate of 10 Mbps, with a maximum burst size of 250 kBps, and downgrade all nonconforming traffic.

router10(config-pmap)# class gold
router10(config-pmap-c)# trust dscp
router10(config-pmap-c)# police 10000000 250000 exceed-action policed-dsp-transmit
router10(config-pmap-c)# exit

Create a policer to police packets that match to class "voice".

Police the flow at a sustained rate of 5 Mbps, with a maximum burst size of 50 kBps, and drop all nonconforming traffic.

router10(config-pmap)# class voice
router10(config-pmap-c)# trust dscp
router10(config-pmap-c)# police 5000000 50000 exceed-action drop
router10(config-pmap-c)# exit
router10(config-pmap)# exit

Map the policy "ewanpolicy" to the outgoing traffic on the WAN port.

router10(config)# interface gi0/11
router10(config-if)# service-policy output ewanpolicy
router10(config-if)# exit

Create an OSPF instance, and give it an arbitrary process ID.

router10(config)# router ospf 10

Place the OSPF instance in area 1.

router10(config-router)# network 10.1.0.0 0.0.255.255 area 1

Create an OSPF instance, and give it an arbitrary process ID.

router11(config)# router ospf 10
router11(config-router)# network 10.1.0.0 0.0.255.255 area 1
router13(config)# router ospf 10
router13(config-router)# network 10.1.0.0 0.0.255.255 area 1

Create an OSPF instance, and give it an arbitrary process ID.

router12(config)# router ospf 1

Place the OSPF instances in areas 1 and 0, unless a backbone area 0 already exists somewhere else in the network. In that case, make sure that area 0 in this example is connected to the real-world area 0. If you will not be connecting example area 0 to real-world Area 0, then put all the routers in this example into area 1.

router12(config-router)# network 10.1.0.0 0.0.255.255 area 1
router12(config-router)#exit
router12(config)# router ospf 10
router12(config-router)# network 10.1.0.0 0.0.255.255 area 0

Enable IP routing.

Router10(config)# ip routing

Enable multicast routing, PIM.

router10(config)# ip multicast-routing
router10(config)int gi0/11
router10(config-if)#ip pim sparse-dense
router10(config-if)# exit

Save the configuration.

router10(config)# write t

Hybrid EMS- ERS Configuration

Figure 18 shows a network of two switches connected to a router via an EMS. The router has an additional connection via an ERS to the Internet. The router and switches see the EMS as a broadcast domain. The router sees the ERS as a point-to-point circuit with a different VLAN and IP subnet. The switches do not see the ERS and will send traffic to the router to get to the Internet. The router attaches to both the ERS and EMS through one physical port, although if you prefer, separate ports can be used. There are two service types: a high-priority gold service and a low-priority best effort service.

Figure 18

Combined EMS-ERS Solution

Keys to this implementation are as follows:

1. The distribution router, router 15, is a member of both VLANs, one connecting to the EMS and one to the ERS.

2. All EMS WAN-facing ports/circuits of the access routers are in the same VLAN.

3. The ERS and EMS VLAN IDs are different.

4. All LAN ports have VLAN IDs that differ from the WAN VLANs' IDs.

5. IGMP snooping and PIM are used to prune multicast flows.

Comments:

All switches use the same configuration, because EMS emulates a VLAN/Layer 2 broadcast domain. Of course, the IP addresses will differ at each switch.

The router configuration follows the switch configuration.

Regarding QoS contracts, the distribution router (router 15) may need to support more traffic (more bits per second) than the other nodes, because it may be a headquarters. In that case, router 15's QoS contracts would be larger than those of switches 13 and 14.

VLANs 12 and 13, the WAN VLANs, can be configured as untagged or as IEEE 802.1Q VLAN tagged. In this example VLANs 12 and 13 are tagged. Other examples show untagged VLANs.

b> enable
switch13# config t

Create VLAN 1, the WAN VLAN, and give it an IP address.

switch13(config)#int vlan 12
switch13(config-if)#ip address 10.1.2.13 255.255.255.0

Make Interface Gigabit Ethernet 0/11 an IEEE 802.1Q trunk.

switch13(config-if)#int gi0/11
switch13(config-if)# switchport trunk encapsulation dot1q
switch13(config-if)# switchport mode trunk
switch13(config-if)# exit

Create VLAN 104, the LAN VLAN, and give it a VLAN ID different from the WAN VLAN ID.

switch13(config)# int vlan 104

Make Interface Gigabit Ethernet 0/1 an untagged access VLAN. Alternatively, it can be configured to be an IEEE 802.1Q trunked VLAN. This depends on the configuration of the attached LAN.

switch13(config-if)#int gi0/1
switch13(config-if)#switchport access vlan 104

Map CoS bits to DSCP bits.

switch13(config)# mls qos map cos-dscp

Create a class map for best effort traffic.

switch13(config)# class-map be

Make all packets with a DSCP value of 0 be matched to the "best effort" class.

switch13(config-cmap)# match-all be
switch13(config-cmap)# match ip dscp 0
switch13(config-cmap)# exit

Create a class map for gold service.

switch13(config)# class-map gold

Make all packets with a DSCP value of 3 be matched to the "gold" service class.

switch13(config-cmap)# match-all gold
switch13(config-cmap)#match ip dscp af31 af32 af33
switch13(config-cmap)# exit

Create a policy map for the WAN port, and police packets matched to class maps "be" and "gold".

switch13(config)# policy-map ewanpolicy
switch13(config-pmap)# class be
switch13(config-pmap-c)# trust dscp

Create a policer to police packets that match to class "be".

Police the flow at a sustained rate of 20 Mbps, with a maximum burst size of 500 kBps, and drop all nonconforming traffic.

switch13(config-pmap-c)# police 20000000 500000 exceed-action drop
switch13(config-pmap-c)# exit

Create a policer to police packets that match to class "gold".

Police the flow at a sustained rate of 10 Mbps, with a maximum burst size of 250 kBps, and downgrade all nonconforming traffic.

switch13(config-pmap)# class gold
switch13(config-pmap-c)# trust dscp
switch13(config-pmap-c)# police 10000000 250000 exceed-action policed-dsp-transmit
switch13(config-pmap-c)# exit
switch13(config-pmap)# exit

Map the policy "ewanpolicy" to the outgoing traffic on the WAN port.

switch13(config)# interface gi0/11
switch13(config-if)# service-policy output ewanpolicy
switch13(config-if)# exit

Turn on IGMP snooping on the WAN interface.

switch13(config)#int gi0/11
switch13(config-if)#ip igmp snooping
Save the configuration.
switch13(config)# write t
\*******************\
router15> enable
router15# config t

Create VLANs 12 and 13 , the WAN VLANs on router 15, and give them the appropriate IP addresses. VLAN 12 attaches to the EMS, and VLAN 13 attaches to the ERS. Also create an 802.1Q tagged trunk.

router15(config)#int vlan 12
router15(config-if)#ip address 10.1.1.15 255.255.255.0
router15(config-if)#ip exit
router15(config)#int vlan 13
router15(config-if)#ip address 10.1.2.15 255.255.255.0
router15(config-if)#ip exit
router15(config)#int gi 0/11
router15(config-if)# switchport trunk encapsulation dot1q
router15(config)#siwtchport trunk allowed vlan 12 13
router15(config-if)# exit

Create VLAN 104, the LAN VLAN, and give it a VLAN ID different from the WAN VLAN ID

router15(config)# int vlan 104

Make Interface Gigabit Ethernet 0/1 an IEEE 802.1Q trunked VLAN. Alternatively, it can be configured to be an untagged access VLAN. This depends on the configuration of the attached LAN.

router15(config-if)#int gi0/1
router15(config-if)# switchport trunk encapsulation dot1q
router15(config-if)# switchport mode trunk

Map CoS bits to DSCP bits.

router15(config)# mls qos map cos-dscp

Create a class map for best effort traffic.

router15(config)# class-map be

Make all packets with a DSCP value of 0 be matched to the "best effort" class.

router15(config-cmap)# match-all be
router15(config-cmap)# match ip dscp 0
router15(config-cmap)# exit

Create a class map for gold service.

router15(config)# class-map gold

Make all packets with a DSCP value of 3 be matched to the "gold" service class.

router15(config-cmap)# match-all gold
router15(config-cmap)#match ip dscp af31 af32 af33
router15(config-cmap)# exit

Create a policy map for the WAN port, and police packets matched to class maps "be" and "gold".

router15(config)# policy-map ewanpolicy
router15(config-pmap)# class be
router15(config-pmap-c)# trust dscp

Create a policer to police packets that match to class "be".

Police the flow at a sustained rate of 20 Mbps, with a maximum burst size of 500 kBps, and drop all nonconforming traffic.

router15(config-pmap-c)# police 20000000 500000 exceed-action drop
router15(config-pmap-c)# exit

Create a policer to police packets that match to class "gold".

Police the flow at a sustained rate of 10 Mbps, with a maximum burst size of 250 kBps, and downgrade all nonconforming traffic.

router15(config-pmap)# class gold
router15(config-pmap-c)# trust dscp
router15(config-pmap-c)# police 10000000 250000 exceed-action policed-dsp-transmit
router15(config-pmap-c)# exit
router15(config-pmap)# exit

Map the policy "ewanpolicy" to the outgoing traffic on the WAN port.

router15(config)# interface gi0/11
router15(config-if)# service-policy output ewanpolicy
router15(config-if)# exit

In this example a default route is used in which 1.2.3.4 is the address of the next-hop router in the Internet/Extranet cloud. Routing protocols such as EIGRP (Enhanced Interior Gateway Routing Protocol), OSPF, or BGP may be used instead of the default route.

router15(config)# ip route 0.0.0.0 1.2.3.4

Enable IP routing.

router15(config)# ip routing

Enable multicasting routing, PIM.

router15(config)# ip multicast-routing
router15(config)int gi0/11
router15(config-if)#ip pim sparse-dense
router15(config-if)# exit

Save the configuration.

router15(config)# write t

Attaching to an EWS with Switches and Routers

Figure 19 shows a network of two switches connected to a router via an EWS.

Figure 19

Attaching to an EWS with Switches and a Router

Keys to this implementation are as follows:

1. The distribution router, router 18, is a member of both VLANs, one connecting to the EWS.

2. All EWS WAN-facing ports/circuits of the access switches are in the different VLAN.

3. All LAN ports have VLAN IDs that differ from the WAN VLANs' IDs.

4. IGMP snooping and PIM are used to prune multicast flows.

Comments:

5. Router 18 connects to the EWS via two different ports, because it is a port-to-port service, not just point-to-point.

6. All switches use the same configuration. Of course, the IP addresses will differ at each switch as well as the WAN VLAN ID.

7. The router configuration follows the switch configuration.

8. Regarding QoS contracts, the distribution router (router 18) may need to support more traffic (more bits per second) than the other nodes, because it may be a headquarters. In that case, router 18's QoS contracts would be larger than those of switches 16 and 17.

9. VLANs 14 and 15, the WAN VLANs, can be configured as untagged or IEEE 802.1Q VLAN tagged. In this example VLANs 14 and 15 are tagged. Other examples show untagged VLANs.

Switch16> enable
switch16# config t

Create VLAN 1, the WAN VLAN and give it an IP address.

switch16(config)#int vlan 14
switch16(config-if)#ip address 10.1.1.16 255.255.0.0

Make Interface Gigabit Ethernet 0/11 an IEEE 802.1Q trunk.

switch16(config-if)#int gi0/11
switch16(config-if)# switchport trunk encapsulation dot1q
switch16(config-if)# switchport mode trunk
switch16(config-if)# exit

Create VLAN 104, the LAN VLAN, and give it a VLAN ID different from the WAN VLAN ID.

switch16(config)# int vlan 104

Make Interface Gigabit Ethernet 0/1 an untagged access VLAN. Alternatively, it can be configured to be an IEEE 802.1Q trunked VLAN. This depends on the configuration of the attached LAN.

switch16(config-if)#int gi0/1
switch16(config-if)#switchport access vlan 100

Map CoS bits to DSCP bits.

switch16(config)# mls qos map cos-dscp

Create a class map for best effort traffic.

switch16(config)# class-map be

Make all packets with a DSCP value of 0 be matched to the "best effort" class.

switch16(config-cmap)# match-all be
switch16(config-cmap)# match ip dscp 0
switch16(config-cmap)# exit

Create a class map for gold service.

switch16(config)# class-map gold

Make all packets with a DSCP value of 3 be matched to the "gold" service class.

switch16(config-cmap)# match-all gold
switch16(config-cmap)#match ip dscp af31 af32 af33
switch16(config-cmap)# exit

Create a policy map for the WAN port, and police packets matched to class maps "be" and "gold".

switch16(config)# policy-map ewanpolicy
switch16(config-pmap)# class be
switch16(config-pmap-c)# trust dscp

Create a policer to police packets that match to class "be".

Police the flow at a sustained rate of 20 Mbps, with a maximum burst size of 500 kBps, and drop all non-conforming traffic.

switch16(config-pmap-c)# police 20000000 500000 exceed-action drop
switch16(config-pmap-c)# exit

Create a policer to police packets that match to class "gold".

Police the flow at a sustained rate of 10 Mbps, with a maximum burst size of 250 kBps, and downgrade all nonconforming traffic.

switch16(config-pmap)# class gold
switch16(config-pmap-c)# trust dscp
switch16(config-pmap-c)# police 10000000 250000 exceed-action policed-dsp-transmit
switch16(config-pmap-c)# exit
switch16(config-pmap)# exit

Map the policy "ewanpolicy" to the outgoing traffic on the WAN port.

switch16(config)# interface gi0/11
switch16(config-if)# service-policy output ewanpolicy
switch16(config-if)# exit

Turn on IGMP snooping on the WAN interface.

switch16(config)#int gi0/11
switch16(config-if)#ip igmp snooping

Save the configuration.

switch16(config)# write t
\*******************\
router18> enable
router18# config t

Create VLANs 14 and 15. Also create an 802.1Q tagged trunk.

router18(config)#int vlan 14
router18(config-if)#ip address 10.1.1.18 255.255.0.0
router18(config-if)#ip exit
router18(config)#int vlan 15
router18(config-if)#ip address 10.1.2.18 255.255.0.0
router18(config-if)#ip exit
router18(config)#int gi 0/11
router18(config-if)# switchport trunk encapsulation dot1q
router18(config)#siwtchport trunk allowed vlan 14
router18(config-if)# exit
router18(config)#int gi 0/10
router18(config-if)# switchport trunk encapsulation dot1q
router18(config)#siwtchport allowed vlan 15
router18(config-if)# exit

Create VLAN 100, the LAN VLAN, and give it a VLAN ID different from the WAN VLAN ID.

router18(config)# int vlan 100

Make Interface Gigabit Ethernet 0/1 an IEEE 802.1Q trunked VLAN. Alternatively, it can be configured to be an untagged access VLAN. This depends on the configuration of the attached LAN.

router18(config-if)#int gi0/1
router18(config-if)# switchport trunk encapsulation dot1q
router18(config-if)# switchport mode trunk

Map CoS bits to DSCP bits.

router18(config)# mls qos map cos-dscp

Create a class map for best effort traffic.

router18(config)# class-map be

Make all packets with a DSCP value of 0 be matched to the "best effort" class.

router18(config-cmap)# match-all be
router18(config-cmap)# match ip dscp 0
router18(config-cmap)# exit

Create a class map for gold service.

router18(config)# class-map gold

Make all packets with a DSCP value of 3 be matched to the "gold" service class.

router18(config-cmap)# match-all gold
router18(config-cmap)#match ip dscp af31 af32 af33
router18(config-cmap)# exit

Create a policy map for the WAN port, and police packets matched to class maps "be" and "gold".

router18(config)# policy-map ewanpolicy
router18(config-pmap)# class be
router18(config-pmap-c)# trust dscp

Create a policer to police packets that match to class "be".

Police the flow at a sustained rate of 20 Mbps, with a maximum burst size of 500 kBps, and drop all non-conforming traffic.

router18(config-pmap-c)# police 20000000 500000 exceed-action drop
router18(config-pmap-c)# exit

Create a policer to police packets that match to class "gold".

Police the flow at a sustained rate of 10 Mbps, with a maximum burst size of 250 kBps, and downgrade all nonconforming traffic.

router18(config-pmap)# class gold
router18(config-pmap-c)# trust dscp
router18(config-pmap-c)# police 10000000 250000 exceed-action policed-dsp-transmit
router18(config-pmap-c)# exit
router18(config-pmap)# exit

Map the policy "ewanpolicy" to the outgoing traffic on the WAN port.

router18(config)# interface gi0/11
router18(config-if)# service-policy output ewanpolicy
router18(config-if)# exit

Enable IP routing.

Router18(config)# ip routing

Enable multicasting routing, PIM.

router18(config)# ip multicast-routing
router18(config)int gi0/11
router18(config-if)#ip pim sparse-dense
router18(config-if)# exit

Save the configuration.

router18(config)# write t

Security Configurations

When connecting to Ethernet services, especially an EMS, security is of extra concern. This section describes ways to secure your Layer 2 systems as well as some general network element security issues.

Network Element Level Security

This section covers basic network element security. Passwords should always be configured to access the various running levels of any Cisco switch or router. Remember to use smart passwords with a mix of capital and lowercase letters and numbers. A password like e515!oF is a good password.

Basic Passwords

To set the password for the enable command to be "e5150A", enter the following commands:

host(config)# service password-encryption
host(config)# enable secret e5150A

To configure the password for the network element to be "blah34blaX" enter the following commands.

host(config)# line console 0
host(config-line)# password blah34blaX

Securing Telnet

To configure the password for all Telnet access ports to be "r3f56po", enter the following commands:

host(config)# line vty 0 15
host(config)# password r3f56po

This example restricts Telnet access to a particular IP address. Several parameters can be configured, as listed below. To restrict Telnet access, enter the following commands:

host(config)# ip access-list extended listname1
host(config-ext-nacl)# permit ?
<0-255> An IP protocol number
 ahp Authentication Header Protocol
 eigrp Cisco's EIGRP routing protocol
 esp Encapsulation Security Payload
 gre Cisco's GRE tunneling
 icmp Internet Control Message Protocol
 igmp Internet Gateway Message Protocol
 igrp Cisco's IGRP routing protocol
 ip Any Internet Protocol
 ipinip IP in IP tunneling
 nos KA9Q NOS compatible IP over IP tunneling
 ospf OSPF routing protocol
 pcp Payload Compression Protocol
 pim Protocol Independent Multicast
 tcp Transmission Control Protocol
udp User Datagram Protocol

Configure the Telnet access list.

host(config-ext-nacl)#permit tcp 10.0.1.0 0.0.0.255 any eq telnet
host(config-ext-nacl)# exit

To apply the list to all the incoming Telnet ports, perform the following:

host(config)# line vtty 0-15
host(config-line)# access-class listname1 in

Community Strings

Default community string names are well known. Delete the default ("public" and "private") community strings.

host(config)# no snmp-server community public
host(config)# no snmp-server community private

Create a read-only community string.

host(config)#snmp-server community newreadonlystring ro

Create a read/write community string.

host(config)#snmp-server community newreadwritestring rw

TACACS+ Authentication

To configure TACACS+ user authentication, enter the following commands. TACACS+ is enabled to the server with the IP address of 10.1.4.5 and the key of "12we45". The user has three attempts to log in correctly. Wait 10 seconds to receive a reply.

host(config)# tacacs-server extended
host(config)#tacacs-server host 10.1.4.5
host(config)#tacacs-server key 12we45
host(config)#tacacs-server attempts 3
host(config)#tacacs-server timeout 10

VLAN and Other Layer 2 Security

Port Security

Administer down all ports.

host(config)# interface gi0/2
host(config-if)# shutdown

802.1X authentication should be used on all host ports.

host(config)# int gi0/1
host(config-if)#dot1x port-control auto
host(config)# int gi0/1

Spanning Tree Security

To configure the port to not send or receive BPDUs on this port, enter the following command:

host(config-if)#spanning-tree bpdufilter enable

To configure the port to not accept BPDUs on a given interface, enter the following command:

host(config-if)#spanning-tree bpduguard enable

To disable instantaneous forwarding on a given interface, enter the following command:

host(config-if)#spanning-tree portfast disable

Make this the root bridge for VLAN 123.

host(config)#spanning-tree vlan 123 root

Given that this is the root bridge, any time another bridge announces itself on the specified interface as the root bridge, ignore it.

host(config-if)#spanning-tree guard root

Cisco Discovery Protocol Security

Cisco Discovery Protocol can reveal network information. You can disable it globally.

host(config)# no cdp run

Or you can disable it on a specific port. It should always be disabled on the WAN and host connected ports.

host(config)# int gi0/11
host(config-int)# no cdp enable

VLAN Trunking Security

To prevent trunk port spoofing attacks, disable trunk negotiation (signaling) on all ports. This prevents an attacker from mimicking a trunk port and receiving packets from all VLANs.

host(config)# int gi0/1
host(config-if)#switchport nonnegotiate
host(config-if)# no switchport host