navbar
White Papers

How to PDF acrobat

Table Of Contents

Application Note

Executive Summary

Service Provider Network Design without Dot1q Tunneling

Network Design with Dot1q Tunneling

Dot1q Tunneling Operation Details

Dot1q Tunneling Caveats

Spanning-Tree Protocol Guidelines

Spanning-Tree Interaction

Root Guard at the Ingress Switch

VLAN Hopping and the dot1q-all-tagged Feature

Redundancy Implementation with dot1q Tunneling

1.0 Sample Configurations

Configuration for service provider switches

Service provider 1's switch configuration

Service provider 2's switch configuration

Service provider 3's switch configuration

Summary


Application Note


Using Dot1q Tunneling (1q in 1q)
on Cisco Catalyst 6500 Series Switches
to Form Layer 2 VPNs

Executive Summary

The 802.1q tunneling feature allows the creation of secure virtual private networks (VPNs) by using the IEEE 802.1q protocol on the Catalyst 6500 Series switches. This feature enables service providers to segregate traffic from different customers in their infrastructure while significantly reducing the number of virtual LANs (VLANs) required to support VPNs. Multiple customer VLANs can be carried inside a single VLAN on the Catalyst 6500 Switch without losing their unique VLAN IDs. The number of VLANs required to support the dot1q tunnels in the service provider network can be reduced significantly. The VPNs deliver enterprise-scale connectivity deployed on a shared infrastructure with the same security, prioritization, reliability, and manageability enjoyed in a private network. This application note provides an overview of dot1q tunneling and its applicability in networks, especially service providers' networks. The document also includes a brief configuration guide and examples for dot1q tunneling.

Note: Dot1q tunneling is supported on all Catalyst 6500 supervisors and line cards except WAN OSMs starting with Catalyst OS Release 6.1.

Service Provider Network Design without Dot1q Tunneling

Service provider customers frequently have Layer 2 connectivity requirements relating to the total number of supported VLANs or the assignment of specific VLAN numbers. One problem is that the ranges of VLANs required by the customers may overlap (see Figure 1), a setup that may lead to inadvertently mixing different customers' traffic through the network infrastructure. Assigning a unique range of VLANs to each customer in the service provider network solves this problem, but this solution can easily consume the 4096 VLAN space supported in 802.1q.

Figure 1 Traffic from Corp B and Corp C is on overlapping VLANs.

Dot1q tunneling addresses this limitation by allowing service providers to assign a VLAN to each customer without losing the original customer VLAN IDs inside the tunnel.

Network Design with Dot1q Tunneling

An ideal scenario to support multiple customers in the service provider environment would be to have customers utilizing any range of VLAN numbers while the service provider forwards the traffic independent of those VLAN IDs. By assigning a unique VLAN to each customer, the identity of multiple VLAN IDs from the customer site will not be lost. This builds a Layer 2 VPN where traffic from different business customers is segregated inside the service provider core and is dot1q tagged with appropriate VLAN IDs. Dot1q tunneling is in essence a 1q-in-1q technique that expands the VLAN space by retagging the tagged packets entering the service provider infrastructure illustrated in Figure 2:

Figure 2 Point-to-Multipoint Dot1q Tunneling Design within a Service Provider Network

The interfaces on the customer sites are configured as 802.1q trunks (InterSwitch Link [ISL] is not supported), whereas the interfaces on service provider border switches are configured as special dot1q tunnel access ports. This special access port is configured with a unique VLAN that the service provider assigns to each customer. When this port receives the tunnel traffic from the neighboring device, it does not follow the usual procedure of stripping the 802.1q tags from the frame header. Instead, it leaves the 802.1q tags intact and puts all the received 802.1q traffic into the VLAN assigned to the tunnel port. As a result, traffic is double-tagged as it enters the service provider infrastructure.

The customer premises equipment (CPE) may be any Cisco switch that understands the 802.1q trunk protocol. The switches inside the service provider core need to be Catalyst 6500 switches running Catalyst OS 6.1 or later software because all the switches inside the core need to understand 802.1q-tunneled packets.

Dot1q Tunneling Operation Details

This section outlines the flow a frame takes as it goes through a dot1q tunnel from the CPE to ingress and egress on the service provider network. Figure 3 illustrates the process, and the steps are described further in the section.

Figure 3 Frame Passing through a dot1Q Tunnel

1. An Ethernet frame on VLAN 2 arrives on the service provider switch from a customer. The customer switch can be any Cisco switch capable of supporting dot1q trunks. If the customer chooses to have a Catalyst 6500 Switch, the software may also be Catalyst OS 5.x Release Software. Note that ISL trunks are not supported; the egress frame from CPE needs to have dot1q encapsulation as depicted:

DA

SA

Etype (8100)

Dot1Q Trunk Tag

Length/Etype

Data

FCS

(6B)

(6B)

(2B)

(2B)

(2B)

(0—1500 Bytes)

(4B)


2. b) The ingress Catalyst 6500 Switch on the service provider's network does not strip off any dot1q headers from the incoming frame, but depending on the trunk protocol adds additional encapsulation. In this example, VLAN 60 is being used for this customer on the service provider switch and that tag is added on. It is recommended that ISL trunks carry tunnel traffic inside the service provider infrastructure. Standard 802.1q trunks require very careful configuration because mistakes might direct tunnel traffic to a wrong port and thus the wrong customer. Below is the diagram of the frames leaving the Catalyst 6500 Switch. Additions to the frame are shown in a different color; with ISL trunk encapsulation, we get:

ISL

DA

SA

Etype (8100)

Dot1Q Trunk Tag

Length/Etype

Data

FCS

(26B)

(6B)

(6B)

(2B)

(2B)

(2B)

(0—1500 Bytes)

(4B)


With dot1q trunk encapsulation, we get:

DA

SA

Etype (8100)

Dot1Q Trunk Tag

Etype (8100)

Dot1Q Trunk Tag

Length/Etype

Data

FCS

(6B)

(6B)

(2B)

(2B)

(2B)

(2B)

(2B)

(0—1500 Bytes)

(4B)


Note: Two dot1q tags are leaving the ingress service provider switch.

3. The egress switch on the service provider infrastructure removes the 802.1q tag and the traffic is removed from the tunnel. In this case, the tag for VLAN 60 is removed but the original tag of VLAN 2 is retained:

DA

SA

Etype (8100)

Dot1Q Trunk Tag

Length/Etype

Data

FCS

(6B)

(6B)

(2B)

(2B)

(2B)

(0—1500 Bytes)

(4B)


Finally, the customer switch removes the original VLAN 2 tag and forwards the frame to the end host:

DA

SA

Length/Etype

Data

FCS

(6B)

(6B)

(2B)

(0—1500 Bytes)

(4B)


Because the tunnel traffic retains the 802.1q tag within the Catalyst switch and packets to the backplane are tagged (that is, with an extra four bytes after the source Media Access Control [MAC] address field), byte realignment occurs within the frame, and the IP header information can no longer be identified. The Layer 2 frame has what would otherwise be an invalid-length header, imposing the following restrictions:

The Layer 3 packet within the Layer 2 frame cannot be identified.

Layer 3 and higher parameters are not identifiable in tunnel traffic (for example, source/destination IP addresses) and thus cannot be routed or Layer 3 policed.

The switch can filter tunnel traffic using only Layer 2 parameters (VLANs, source/destination MAC addresses).

The switch can provide only MAC-layer quality of service (QoS) for tunnel traffic. Traffic inside the infrastructure will not enjoy the same kind of QoS as in the customer layer.

The 802.1q tunneling feature cannot be configured on ports to support private VLANs or IP phones. The VLAN Trunk Protocol (VTP) does not work with devices on an asymmetrical link, and VTP domains at the customer network and service provider network are different.

Asymmetrical links do not support the Dynamic Trunk Protocol (DTP) because only one end on the link is a trunk. Configuration on the 802.1q trunk port on the customer site requires the nonnegotiate dot1q trunking keyword.

Dot1q Tunneling Caveats

This section describes some restrictions and caveats that are associated with the use of dot1q tunneling.

Spanning-Tree Protocol Guidelines

The dot1q tunneling feature tunnels the customer's Layer 2 data packets and bridge protocol data units (BPDUs) through the service provider cloud. The customers need to run per-VLAN spanning tree (PVST+) on the ingress switch. The service provider infrastructure may run Multi-Instance Spanning-Tree Protocol (MISTP)-PVST+ spanning-tree mode or PVST+ mode. MISTP-PVST+ spanning-tree mode is a transition spanning-tree mode that allows the MISTP functionality on the Catalyst 6500 switches while continuing to communicate with other switches such as the Catalyst 3500/4000/5000 Series running PVST+ spanning-tree mode. MISTP mode cannot be used inside a service provider infrastructure because a switch using PVST+ mode connected with a switch running MISTP mode cannot see the BPDUs of the other switch, a condition that may cause loops in the network. MISTP-PVST+ spanning-tree mode conforms to the limits of the PVST+. For example, the amount of VLAN ports on the MISTP-PVST+ switches and PVST+ switches is exactly the same.

Spanning-Tree Interaction

Spanning-Tree Protocol (STP) is designed to remove loops in a Layer 2 switched network. STP was defined for a legacy LAN environment. With the switched networks, 802.1Q trunking defines the way a VLAN must be transported between switches. However, the STP for each VLAN is not fully defined by 802.1Q. At least one standard STP must be running (the STP of a native VLAN), but nothing is defined to transport the STP of tagged VLANs.

Adding the 802.1Q tunneling function makes the spanning-tree situation more complex. For example, the STP of a tunneled VLAN 2 for one customer must not interfere with the STP of a tunneled VLAN 2 for another customer. To resolve this issue, the 802.1Q tunnel function does not transport STP for tunneled VLANs. Only the native VLAN STP is used.

Not all the designs are valid when using the 802.1Q tunneling function because of the STP interactions. Layer 2 loops are not allowed between client sites interconnected by 802.1Q tunneling. The only design allowed with a loop is that in which the CPE switches of a client site are connected to the same Catalyst 6500 in the service provider core, as depicted in Figure 4.

Figure 4 Spanning-Tree Topology for Customers and Service Providers with dot1q Tunneling

Root Guard at the Ingress Switch

In switched networks, the root of the spanning tree is very important and must be protected when the service provider and customer networks are connected. The root-guard feature can be used for this purpose.

Spanning-tree calculations make use of bridge ID (BID) and path cost. A BID is a single, 8-byte field that is composed of two subfields shown below:

Bridge Priority

MAC Address

(2 bytes)

(6 bytes)


The default bridge priority is 32,768. The Catalyst switch elects a single root bridge by looking for the bridge with the lowest BID, and in the STP the lowest BID wins. The lower the bridge priority is, the more preferred it is for the spanning-tree root. If the customer switch in the dot1q tunneling scenario has a lower MAC address and bridge priority than the service provider switch, it will become the spanning-tree root and may get data from multiple customers, as shown in Figure 5.

Figure 5 Selection of an Incorrect Root Switch outside Service Provider Network

The root-guard feature can be enabled per port on the service provider site to restrict customer switches from becoming the root. First define a perimeter where you want the root switch to be confined, and then enable root guard on each port of this perimeter. The feature can be enabled on per-port basis, not per port per VLAN. If the client side tries to become the root, the dot1q tunnel access port will go into root-inconsistent state, as seen below.

cat6k-2> (enable) set spantree guard root 1/1
Rootguard on port 2/13 is enabled.
Warning!! Enabling rootguard may result in a topology change.
2001 Apr 20 12:31:20 pdt -07:00 %SPANTREE-2-ROOTGUARDBLOCK: Port 1/1 (agPort 13/38) tried to become 
non-designated in VLAN 2. Moved to root-inconsistent state
cat6k-2> (enable)

The feature verifies if the port has the designated role in STP. The root-guard port can also go to root-inconsistent state if a superior BPDU is received from a customer switch and the customer BPDU is dropped. The offending switch is disconnected. It cannot inject superior BPDUs inside the defined perimeter, and the service provider STP topology is not affected. If the port stops receiving superior BPDUs, it leaves the root-inconsistent state after a maximum age.

cat6k-4 (enable) show spantree guard 1/1
Port                     VLAN Port-State    Guard Type
------------------------ ---- ------------- ----------
4/13-14                  20   forwarding          root
cat6k-4 (enable)

Occasionally the service provider switch gets a superior BPDU from a client side and it immediately goes in root-inconsistent state. The following steps are needed to resolve this behavior:

1. Set the spanning-tree priority higher than the customer switch; the lower the number, the higher the priority for the customer VLAN; (for example, 20):

set spantree priority 200     20

2. Enable the root guard on the port:

set spantree guard root 1/1

BPDU filtering does not work with the 802.1Q tunneling access port. It is meant only for PortFast-enabled nontrunking ports. Control packets from customers' switches such as Cisco Discovery Protocol (CDP), Port Aggregation Protocol (PAgP), VTP, and GARP VLAN Registration Protocol (GVRP) are dropped at the edge switch and they are not tunneled. CDP should be turned off on the service provider switch.

VLAN Hopping and the dot1q-all-tagged Feature

Another feature similar to dot1q tunneling is the dot1q-all-tagged feature on Catalyst OS 6.1. This feature tags all egress packets on a dot1q trunk, including the packets on the native VLAN. It also ensures that all the ingress untagged packets are dropped on dot1q trunks. The feature is enabled by a global command and, if turned on, applies to all the 802.1q trunks on the switch. It can be a standalone feature and may be turned on even if the Catalyst switch does not have any tunnel ports.

The dot1q-all-tagged feature prevents the VLAN hopping (VLAN security) problem with dot1q tunneling and is 802.1q standards compliant. It also prevents malicious attacks from some customers trying to reach other customers. (See Figure 6.)

Figure 6 VLAN Hopping from Customer X to Customer Y

For example, Customer X may try to generate a frame with a dot1q tag to reach another Customer Y. Suppose Customer X is sending traffic on VLAN 1. The frame would have destination address, security association, and the dot1q tag with VLAN 1 information. Let's say that the service provider ingress switch is using VLAN 2 to carry all the Customer X traffic on the dot1q tunnel access port. The frame now gets assigned to VLAN 2. If VLAN 2 is the native VLAN on the dot1q trunk; then the frame will go out without any additional tag. The egress switch receiving the tagged frame will remove the tag and assign the frame to VLAN 1. The frame will go to Customer Y with VLAN 1. The frame jumped from Customer X to Customer Y. This problem can be avoided by turning on the dot1q-all-tagged feature at a service provider edge switch.

The dot1q-all-tagged feature avoids this VLAN hopping problem and requires tagging of all the frames on the normal dot1q trunk so that frames from one customer will reach only the same customer on the other end of the service provider's infrastructure. Another workaround to this problem is to make sure that the dot1Q trunk connected between the edge switches in the service provider network does not have the same native VLAN as the customer VLAN. It is also recommended that the service provider core switches run ISL trunks instead of dot1q trunks because this setup eliminates the VLAN hopping problem.

Redundancy Implementation with dot1q Tunneling

One customer cannot have redundant paths to two different edge switches in the service provider infrastructure. A customer may have redundant paths in an EtherChannel to the same edge switch in an Internet service provider's (ISP's) network. The customer's spanning tree must be PVST+, and the service provider may use either PVST+ or MISTP/PVST+.

Figure 7 Redundancy with dot1q Tunneling

The topology shown in Figure 7 is not supported because BPDUs are tunneled inside the service provider edge switch and may lead to loops in Customer Y's network. This topology was also covered in the spanning-tree interaction shown in Figure 4 of this white paper as a not-supported design. Note that point-to-multipoint topologies are supported and different customer sites may join different edge switches, as seen in Figure 2 of this document.

1.0 Sample Configurations

Figure 8 A Sample Dot1q tunneling Topology

Configuration for customer switches:

Customers are not required to run Catalyst 6.1 Software. Any Catalyst switch capable of dot1q trunking will work at customer sites.

1. Catalyst 6000-1 configuration:

DTP cannot be used between the customer and the service provider because both ends of the asymmetric link are not trunks. There is no need to match the native VLAN on the trunk and the service provider ingress dot1q tunnel port.

cat6k-1 (enable) set vtp mode transparent
cat6k-1 (enable) set vtp domain CorpX
cat6k-1 (enable) set trunk 3/1  nonegotiate dot1q 1-1005,1025-4094
cat6k-1 (enable) set trunk 3/2  nonegotiate dot1q 1-1005,1025-4094
cat6k-1 (enable) set port channel 3/1-2 mode desirable silent 
cat6k-1 (enable) set spantree mode pvst+

2. Catalyst 6000-2 configuration:

cat6k-2 (enable) set vtp mode transparent
cat6k-2 (enable) set vtp domain CorpX
cat6k-2 (enable) set trunk 3/1  nonegotiate dot1q 1-1005,1025-4094
cat6k-2 (enable) set trunk 3/2  nonegotiate dot1q 1-1005,1025-4094
cat6k-2 (enable) set port channel 3/1-2 mode desirable silent 
cat6k-2 (enable) set spantree mode pvst+

3. Catalyst 6000-3 configuration:

cat6k-3 (enable) set vtp mode transparent
cat6k-3 (enable) set vtp domain CorpX
cat6k-3 (enable) set trunk 3/1  nonegotiate dot1q 1-1005,1025-4094
cat6k-3 (enable) set trunk 3/2  nonegotiate dot1q 1-1005,1025-4094
cat6k-3 (enable) set port channel 3/1-2 mode desirable silent 
cat6k-3 (enable) set spantree mode pvst+

Configuration for service provider switches

Perform the following steps for each customer:

1. Assign a VLAN to each customer. Note that all sites for a customer need to have the same VLAN on all service provider switches. VLAN 10 will be used in this example. There is no need to match the native VLAN with the customer.

2. Configure the switch to accept larger-than-default Ethernet maximum-transmission-unit (MTU) size frames ("Jumbo" frames) for tunnels.

3. Activate the "set dot1q-all-tagged enable all" feature on all service provider switches with dot1q trunks to avoid VLAN hopping.

4. Activate "root guard" at the dot1q tunnel access to the port so that the root of the spanning tree remains inside the service provider network.

5. Disable CDP to avoid "peeping" among customers.

6. Use ISL trunks between the service provider switches.

Service provider 1's switch configuration

1. Configuring the customer's Site 1:

SP1 (enable) set dot1q-all-tagged enable
SP1 (enable) set vlan 10   3/1-2
SP1 (enable) set cdp disable  3/1-2
SP1 (enable) set spantree guard root 3/1-2
SP1 (enable) set port jumbo 3/1-2 enable
SP1 (enable) set port channel 3/1-2 mode desirable silent
SP1 (enable) set port dot1qtunnel 3/1-2 access

2. Configuring the trunking for service provider core switches:

SP1 (enable) set trunk 1/1 desirable isl
SP1 (enable) set trunk 1/2 desirable isl

The following optional QoS configuration can also be applied on the dot1q tunnel port. This limits the customer MAC-layer traffic from 100 Mbps to 70 Mbps. Policing or QoS on the IP layer is not supported.

1. Enable QoS.

SP1 (enable) set qos enable

2. Create a policer to limit traffic to 70 Mbps and a MAC-based access control list (ACL) to match incoming traffic.

SP1 (enable) set qos policer aggregate TEST rate 70000 burst 13 drop
SP1 (enable) set qos acl mac AclOne dscp 5 aggregate TEST any any 

A 13-kb burst is used to support large packet sizes (1518 * 8 = 12144) on this Fast Ethernet policer.

3. Commit the ACL.

SP1 (enable) commit qos acl all

4. Map the ACL to customer VLAN 10.

SP1 (enable) set qos acl map AclOne 10

5. Change the ingress port QoS from default port based to VLAN based.

SP1 (enable) set port qos 3/1-2 vlan-based

Service provider 2's switch configuration

1. Configuring the customer's Site 2:

SP2 (enable) set dot1q-all-tagged enable
SP2 (enable) set vlan 10  3/1-2
SP2 (enable) set cdp disable  3/1-2
SP2 (enable) set spantree guard root 3/1-2
SP2 (enable) set port jumbo 3/1-2 enable
SP2 (enable) set port channel 3/1-2 mode desirable silent
SP2 (enable) set port dot1qtunnel 3/1-2 access

2. Configuring the trunking for service provider core switches:

SP2 (enable) set trunk 1/1 desirable isl
SP2 (enable) set trunk 1/2 desirable isl

Service provider 3's switch configuration

1. Configuring the customer's Site 3:

SP3 (enable) set dot1q-all-tagged enable
SP3 (enable) set vlan 10  3/1-2
SP3 (enable) set cdp disable  3/1-2
SP3 (enable) set spantree guard root 3/1-2
SP3 (enable) set port jumbo 3/1-2 enable
SP3 (enable) set port channel 3/1-2 mode desirable silent
SP3 (enable) set port dot1qtunnel 3/1-2 access

2. Configuring the trunking for service provider core switches:

SP3 (enable) set trunk 1/1 desirable isl
SP3 (enable) set trunk 1/2 desirable isl

Summary

Dot1q tunneling enables service providers to segregate the traffic from different customers in their infrastructure while the customers may appear to be on the same VLANs. Asymmetric links are created by making one side 802.1q trunking and the other side a dot1q tunnel access port. The native VLANs should be selected with care, and the dot1q-all-tagged feature should be turned on for VLAN hopping. Customers may have tunnel redundancy with EtherChannel technology using MAC-based frame distribution. CDP should be disabled and STP root guard is recommended on ports connecting to the customer's networks.


Toolbar

Posted: Tue Mar 16 09:54:12 PST 2004
All contents are Copyright © 1992--2004 Cisco Systems, Inc. All rights reserved.
Important Notices and Privacy Statement.