Table Of Contents
Advanced Services Case Study on DLSw+ Ethernet Redundancy
The purpose of this document is to provide the different options available to customers who wish to implement Data-Link Switching Plus (DLSw+) Ethernet Redundancy. Because of the levels of redundancy built into customer environments (particularly in the campus networks) and the nature of switched Ethernet technology, this often creates an inherent problem when connecting SNA clients over DLSw+ with multiple paths to the host. SNA devices historically ran across Token Ring (via source-route bridging), and multiple paths were possible by means of the Routing Information Field (RIF). The RIF was basically a field in the Token Ring frame that plotted the "route" (bridge number, ring number) from a source to destination, providing a unique series of coordinates. Transparent bridging does not use a RIF, so it is impossible to tell whether a frame has already traversed a particular domain, creating an opportunity for loops.
For Cisco IOS" releases prior to 12.0, the recommendation for multiple paths over DLSw+ (over Ethernet) was to use a single primary peer with a secondary peer as backup. This meant that there would be only one active peer at any one time, preventing any looping conditions. This paper covers the solutions available in Release 12.1 and later.
Figure 1 shows a typical customer topology when deploying DLSw+ in a switched Ethernet environment.
Figure 1 Typical DLSw+ over Ethernet Topology
In this example, the remote site usually has two Cisco Catalyst" switches (for redundancy) connecting to two devices running Cisco IOS Software, which could be either multilayer switch feature card (MSFC) technology or external routers. Two front-end processors (FEPs) share the load across all the remote sites. Host connectivity in the data center could be provided by a FEP or Channel Interface Processor (CIP), and the medium could be Token Ring, Fiber Distributed Data Interface (FDDI), or even switched Ethernet, although DLSw+ Ethernet Redundancy would also need to be employed.
For the customer to perform guaranteed load balancing via DLSw+ Ethernet Redundancy, a major piece of work is required—that is, all the SNA clients may need to have their destination media access control (MAC) address altered or the FEP might need to undergo a MAC address change.
The following sections detail some design alternatives with associated caveats, which may lessen the pain, particularly if the customer has a strategy to migrate from SNA to IP and considers the use of DLSw+ Ethernet Redundancy as an interim phase.
DLSw+ Partial Mesh
A partial mesh is considered good practice in typical DLSw+ topologies because it restricts the risk of looping explorers and added complexity. This section describes the topology and configuration for this design and issues that may be encountered.
The topology is relatively simple: the DLSw+ Ethernet Redundancy routers (master/slave) are considered as the remote devices, whereas the central site routers locally bridge to the SNA host. There are single peer relationships on each of the DLSw+ pairings (see Figure 2).
Figure 2 DLSw+ Partial Mesh Design
In this instance the Ethernet Redundancy master router is at the remote site, and an initial test saw all circuits connect via the slave router, as shown in the following output:
The master router is identified by the Master keyword beside its MAC address. On the slave router, the following output can be seen:
The master/slave relationship is determined by the master-priority parameter, or by the lowest MAC address in the event of a tie. Note that election and participation is only possible when the DLSw+ Ethernet Redundancy routers are in the same multicast group.
The circuits connected to all DLSw+ routers in the multicast group can be viewed from the master, as follows:
In the previous output, there are five circuits connected to all the routers in the multicast group, and their state is negative. This means that the circuit is not attached to the master router, and the owner field identifies the MAC address of the router attached to these circuits.
If the same command is performed on the slave router, the following output can be seen:
The slave state and ownership identify that all the circuits are connected via this router. Note that if the master owned any of the circuits, they would not be seen by the slave router. All circuits connected to all routers in the same multicast group can be viewed from the master only.
The previous discussion has dealt with normal operation. This section covers tests performed in the Advanced Services laboratory. The first test performed was to identify how load balancing could be achieved, so the CAM table on the Catalyst 6500 Series switch was cleared (via the command clear cam dyn), and the DLSw+ circuits were reset. The following debug details the activity after the command was issued:
In this output, the Test frame (TEST_STN.Ind) can be seen entering the DLSw+ router on Ethernet2/0. The DLSw+ router checks its reachability cache and realizes that it is empty (ws_status = NO_CACHE_INFO) before sending to all available peers (Write to all peers ok). This operation was repeated for the remaining Test frames.
The next stage shows the return of the ICR (ICANREACH) frames to the slave router. As a result, it sent an IWANTIT message to the master in the form of the following message, which details the MAC address of the slave in the front of the source MAC (SMAC)/source service access point (SSAP) and destination MAC/destination SAP pairing (DMAC/DSAP):Jul 19 12:28:06.485: DLSW-ER:Et2/0:dm_action_j: Rcvd IW <- 0080.6973.0589 0000.6c04.0000:4 0000.6c04.0200:14
At this point, with the responses coming back from the remote DLSw+ peers, both DLSw+ Ethernet Redundancy routers started communicating with each other to agree on ownership of the circuits coming in:Jul 19 12:28:07.511: DLSW-ER:Et0/1:dm_action_k: Rcvd UG for 0000.6c04.0200:14 0000.6c04.0000:4: CSM: update local cache for mac 0000.6c04.0200, Ethernet2/0
Interestingly, the first ICR responses were coming via the slave router, in this case SSAP ID of 14. The CAM table for the switch had cached the entry of the DMAC address pointing to the port of the slave router. The slave router requested the circuit via the IW (IWANTIT) activity. However, as the dialogue continued, the master router was already in the process of sending explorer frames and started receiving responses also:Jul 19 12:28:07.511: CSM: Calling csm_to_core with SSP_START_NEWDL - dlsw_csm_er_handlerNDING: 0000.6c04.0000:4 0000.6c04.0200:10
As the master router received the responses to the explorers for the DMAC of the FEP, the cache in the CAM table on the switch was updated to reflect this. Subsequently, all new circuits destined for this remote MAC address were forwarded to the master router within the cache aging time (default 5 minutes of inactivity). The next series of messages determined the ownership of the circuits:
In the previous output there were two distinct differences. The first message (UG) for SSAP of 14 had the slave MAC address appended at the beginning of the message. All corresponding IWANTIT messages were local and were followed by a new circuit setup request (obviously, the slave router details will be on the slave router debug output).
Some further activity can be seen after the UGOTIT decisions had been made in the following output:Jul 19 12:28:08.557: DLSW-ER:Et2/0:dm_action_j: Rcvd IW <- 0080.6973.0589 0000.6c04.0000:4 0000.6c04.0200:8Jul 19 12:28:08.557: DLSW-ER:Et2/0:dm_action_j: Rcvd IW <- 0080.6973.0589 0000.6c04.0000:4 0000.6c04.0200:10Jul 19 12:28:08.557: DLSW-ER:Et2/0:dm_action_j: Rcvd IW <- 0080.6973.0589 0000.6c04.0000:4 0000.6c04.0200:CJul 19 12:28:08.557: DLSW-ER:Et2/0:dm_action_j: Rcvd IW <- 0080.6973.0589 0000.6c04.0000:4 0000.6c04.0200:4
An IWANTIT request had been sent from the slave to the master, which responded with a circuit taken (CT) message. This occurred because the new circuit for the SMAC/SSAP and DMAC/DSAP pairing had already been created on the master router, which prevented the duplicate circuit issue.
When all the circuits had been established, the following operational commands could be entered on the master router to check the status:
The output confirms the fact that the SSAP of 14 connected via the slave router and all subsequent circuits connected via the master.
The output from the switch CAM table was as follows:
The CDP adjacency is useful to identify which port connects to which router. When the Test frames received a response (destination is 0000.3620.0000), the CAM was updated to reflect the router on port 3/1 as follows:
To prove that all subsequent circuits would connect to the DLSw+ router that "owned" the entry in the CAM table, the DLSw+ circuits were cleared, and the output was displayed as follows:
As can be seen, all circuits attached via the master router. On the slave router the following activity was observed:
The master router advised the slave that the circuits had been taken (CG - Circuit-Got), and no other activity was seen during the Test requests and responses.
Because the first test was designed to examine redundancy and fault tolerance, in this test the master router was deliberately failed to see how fast the failover mechanism would perform. All the following output was taken from the slave router:
When the failure was detected-bearing in mind that the master and slave router set up a Logical Link Control, type 2 (LLC2) session, which is connection-oriented—the DLSw+ Ethernet Redundancy component changed the role of the slave to become the new master.
Approximately 5 minutes passed before any Test frames were sent from the DLSw+ router to its remote peers:
Subsequently, the reachability cache went through the usual routine of being repopulated and so on, and all circuits connected via this router.
The following configurations are samples of those used in the partial mesh design:
Although this topology simplifies the DLSw+ topology, there are some major caveats that a customer may consider unacceptable. The main problem with this setup is the fact that load balancing is virtually nonexistent, because it relies heavily on the cache entry from the switch CAM table. Another issue is the time taken for the CAM entry to timeout, which causes at least a 5-minute delay (taking default aging time into consideration) before any Test frames take the alternative path via the other DLSw+ router.
DLSw+ Full Mesh
The full mesh topology provides each DLSw+ router with a second pair, offering some potential for load balancing on a per-router basis.
Similar to the partial mesh topology, in the full mesh topology the DLSw+ Ethernet Redundancy routers are the remote routers, and the central site routers connect via a Token Ring or Ethernet hub to the SNA host (see Figure 3).
Figure 3 DLSw+ Full Mesh Design
The following section details a test that shows how the full mesh topology worked in terms of circuit setup, load balancing, and the DLSw+ Ethernet Redundancy behavior.
Note: For effective results, it was necessary to include the command dlsw timers explorer-wait-time 10.
Initially, we had to assume that the CAM table had already been populated and the router nsa-voip-3661-2 was connected to the port on the switch that the CAM points to for the FEP MAC address. The reachability cache contained two fresh entries (state is FOUND) for the DMAC address (0000.3620.0000 in noncanonical format):
The next stage was to initiate six SNA sessions on a router behind the switch, remembering that the CAM entry forces all traffic destined for the 3661-2 router:
The SNA session generator sent out six Test frames for each of the SSAPs, which can be seen in the debug (that is, SSAP 4,8,C,10,14,18).
The DLSw+ Ethernet Redundancy routers then communicated whether they could own the circuit, so some communication between the slave and master in the form of LLC2 traffic was performed. The IW indicated that the IWANTIT message was being received by the master. Because no associated MAC address was "tagged" onto this message (that is, no MAC address was present), one could assume that the master itself was requesting the circuit. The following output is a segment of debug from two of the requests (out of six):
The next action was for the DLSw+ master router to decide on the resources and agree on which router would own the circuit. No IWANTIT frames came from the slave router (because of the CAM entry pointing to the master). All circuits connected that way as follows:
If the slave router was offered a circuit, then in the UGOTIT message the slave router's MAC address was inserted at the beginning of the line before the SMAC address. When all the circuits were set up—that is, when CUR/ICR activity was complete—and the reachability cache had remained fresh for both paths (while using round-robin load balancing) to the DMAC address (via DLSw+), the following results were obtained:
This was tested several times and was successful in all attempts.
The following configurations are samples of those used in the full mesh design:
There are a few caveats with this configuration that need to be considered. The first point has to do with load balancing. Although load balancing is possible between the different FEP sites, it can be achieved only on one DLSw+ router at a time, because of the second caveat, which is the restriction of the CAM table to allow only one entry for the DMAC address being reachable via one port only. It is likely that in this configuration all circuits will establish on one of the DLSw+ routers, but with round-robin load balancing they will spread across both remote peers. Fault tolerance is also an issue, as described in the partial mesh section, because the Test frames will not be aware of the secondary router until the CAM entry for the "old" router has aged out. This is a configurable parameter and may add overhead onto the switch depending on the existing size of the CAM table.
Mapping SNA Resources
For accurate load balancing and appropriate fault tolerance in a switched Ethernet environment, Cisco recommends using DLSw+ Ethernet Redundancy where each resource targets a MAC address that appears as a logical MAC address on the DLSw+ Ethernet Redundancy router and is subsequently mapped to a real MAC address for the FEP. Each logical MAC address on the DLSw+ Ethernet Redundancy router is different, meaning that 50 percent of the clients will target MAC address 1, while the others will target MAC address 2. In addition, each peer knows about the resources that are attached to the other peer via the LLC2 session that communicates between both DLSw+ Ethernet Redundancy routers, and in the event of a failure, one of the routers can back up the other.
For the purpose of the testing, the simplest solution was offered, partial mesh instead of full mesh, as shown in Figure 4.
Figure 4 DLSw+ Mapping Resources Design
Before any of the tests were started, the master was checked and found to have perfect load balancing—that is, three LLC2 sessions targeted the master as primary, and the other three targeted the virtual MAC address of the slave:
At this point, the 3660-1 router had failed, and the master router was receiving the details from the slave advising that it had cleared the circuits that were previously attached (CG):Mar 1 20:00:15.003: DLSW-ER:Et2/0:dm_action_h: Rcvd CG <- 0000.6c06.0080 0000.6c04.0000:4 0000.6c04.0200:4Mar 1 20:00:15.007: DLSW-ER:Et2/0:dm_action_h: Rcvd CG <- 0000.6c06.0080 0000.6c04.0000:4 0000.6c04.0200:8Mar 1 20:00:15.007: DLSW-ER:Et2/0:dm_action_h: Rcvd CG <- 0000.6c06.0080 0000.6c04.0000:4 0000.6c04.0200:C
The virtual MAC address previously owned by the failed router was now owned by the master. Subsequently, any packets targeting this MAC address were replaced with the real host address 0000.6c04.0000:Mar 1 20:00:55.367: DLSW-ER:Swapping SMAC 0000.6c04.0000 with 4000.3660.0002 before sending on Et2/0
Now that the Master router "owned" both MAC addresses, the circuits previously connected to the slave router followed the normal process of connecting to the master router, as follows:
Finally, all the circuits connected and could be observed as follows:
The previous failed router finally recovered and the following information could be observed:
These statements show the election process that occurred between the master and slave routers confirming the use of LLC2 to maintain this relationship. The MA frame was basically an acknowledgement from the slave router advising that it had now become the master. When this operation was performed, a further series of messages was passed between the master and slave to identify which router would back up which MAC address. This action was performed in case there were multiple DLSw+ Ethernet Redundancy routers in the same multicast domain:Mar 1 20:02:41.464: DLSW-ER: Sending BACKMEUP_REQ 4000.3660.0002 --> 0000.6c04.0000 to neighbor 0000.6c06.0080 (61F61014)Mar 1 20:02:41.464: DLSW-ER:Et2/0:Rcvd BACKMEUP_REQ from 0000.6c06.0080 for map entry 4000.3660.0001 --> 0000.6c04.0000
Last, when all initial activity was complete, an admin-stop was placed on the circuits that should be owned on the recently recovered DLSw+ Ethernet Redundancy router:Mar 1 20:02:41.464: DLSW-ER:calling admin_stop for ckt(0000.6c04.0200(4) 0000.6c04.0000(4)) with lmac 4000.3660.0001Mar 1 20:02:41.468: DLSW-ER:calling admin_stop for ckt(0000.6c04.0200(8) 0000.6c04.0000(4)) with lmac 4000.3660.0001Mar 1 20:02:41.468: DLSW-ER:calling admin_stop for ckt(0000.6c04.0200(C) 0000.6c04.0000(4)) with lmac 4000.3660.0001
Unfortunately, this action caused another session disruption and forced all the circuits back onto the original DLSw+ Ethernet Redundancy target router:Mar 1 20:02:51.468: DLSW-ER: Sending BACKMEUP_ACK 4000.3660.0001 --> 0000.6c04.0000 to neighbor 0000.6c06.0080Mar 1 20:03:43.529: DLSW-ER:Et2/0:dm_action_j: Rcvd IW <- 0000.6c06.0080 0000.6c04.0000:4 0000.6c04.0200:4Mar 1 20:03:43.529: DLSW-ER:Et2/0:dm_action_j: Rcvd IW <- 0000.6c06.0080 0000.6c04.0000:4 0000.6c04.0200:8Mar 1 20:03:43.533: DLSW-ER:Et2/0:dm_action_j: Rcvd IW <- 0000.6c06.0080 0000.6c04.0000:4 0000.6c04.0200:C
Finally, the following output confirmed that all circuits were now available on their specified DLSw+ Ethernet Redundancy router:
The Catalyst 6500 Series switch showed how the CAM table behaved during the period of one DLSw+ Ethernet Redundancy router becoming unavailable. First, the switch was correctly populated, with 3660-0001 on port 3/1 and 3660-0002 on port 3/3:
Then the 3661-1 router was reloaded, and the CAM entries to this router were deleted (for both real and virtual addresses):VOIP-6509-2 (enable) sh cam dyn2002 Jul 30 06:17:37 %PAGP-5-PORTFROMSTP:Port 3/1 left bridge port 3/1
When the master DLSw+ Ethernet Redundancy router took ownership of the MAC address and started responding to the Test frames from the SNA generator, the CAM table was subsequently updated as follows:
Note that the virtual MAC addresses were in their bit-swapped format and would be 0000.3660.0001 and 0000.3660.0002, respectively.
The following configurations are samples of those used in the mapping resources design:
Although this configuration is preferred by Cisco, very few customers actually deploy it. The main reason is because it requires them to perform some widescale changes. For example, when two DLSw+ Ethernet Redundancy routers are in operation, 50 percent of the SNA clients target one, and 50 percent target the other, contributing to large resource overhead. At a time when many customers are considering migrating away from traditional SNA technology to IP-based applications, the topology may be seen as an interim methodology with huge overhead associated. Alternatively, or additionally, it is likely that customers would have to change the MAC address on their FEP or CIP connections, which again could be considered as administrative overhead.
Adding Hub Technology
As can be seen from the DLSw+ Ethernet Redundancy configurations that do not use the mappings (that is, the full or partial mesh designs), the major restriction is the fact that the CAM table allows only one entry for the destination MAC address, which causes problems with load balancing. However, by placing a passive hub between the Catalyst 6500 Series switches and DLSw+ routers, the CAM table will contain one entry for the destination MAC address, and this resolves the issue of wanting to learn duplicate MAC addresses on the same switch.
Figure 5 shows the topology for DLSw+ Ethernet Redundancy configured with a passive hub.
Figure 5 DLSw+ Ethernet Redundancy with Hub Design
All caches were cleared and the DLSw+ circuits reset. The following output could be seen on the DLSw+ master router when all the normal reachability information had been populated:Jul 23 15:07:12.926: DLSW-ER:Et2/0:dm_action_j: Rcvd IW <- 0000.6c06.0080 0000.6c04.0000:4 0000.6c04.0200:18
In the previous output, two IWANTIT messages were received by the DLSw+ master router, the first one (IW Pending) was generated by itself, and the second was coming from the slave router (Rcvd IW). The master router made a decision and sent a CT (Circuit Taken) message back to the slave as follows:
This process occurred for all the circuits, where the master made a decision on each based on current number of circuits and reachability. A final check of the contents of the circuit cache on the master was detailed as follows:
The CAM table entry always pointed to the port where the hub resided (3/7), so there was no issue with any aging timers and so on:
There are many advantages to this implementation. For example, configuration is greatly simplified, which is a key consideration, and the design provides the ability to perform perfect load balancing. However, it does mean that passive hubs need to be placed between the Catalyst 6500 Series switches and DLSw+ routers, which results in additional cost. Also, the hub creates a single point of failure. To address this issue, hubs could be daisy-chained, so that the destination MAC address would be available via more than one port again, subsequently bringing spanning tree into operation. In this case, some delay may occur while spanning tree converges.
Cisco would prefer that customers move to a configuration with mappings, because this design provides effective (controlled and accurate) load balancing and fault tolerance. However, this choice might entail a large project, particularly if a customer is already on the verge or going through the process of migrating to IP-based SNA applications (via TN3270, HPR/IP, and so on). Changing many MAC address configurations on all SNA clients is a large task, particularly if it is an interim, short-term requirement.
The alternatives described in this document all have caveats. The full and partial mesh designs have issues associated with the CAM table, although the full mesh solution provides load balancing across the remote peers. The hub solution could be considered a step backward, but it resolves the load balancing issue and simplifies the configuration. It also adds a single point of failure into the equation.
Cisco Advanced Services would lean toward either the full-mesh solution or the hub solution, if the mapping of resources design is considered inappropriate. Either technique would be acceptable to most customers' end goal, which is to perform reasonable load balancing across the DLSw+ topology.
Cisco IOS Release 12.2 mainline is the software version associated with DLSw+ Ethernet Redundancy. Although the train is reaching a reasonable level of maturity for these technologies, it should be regularly monitored for new defects.
Finally, all the tests were performed in a controlled laboratory environment with basic routing configurations and Cisco IOS Release 12.2(10a). The SNA traffic generation was performed by a utility in the Cisco IOS Software called Downstream Physical Unit (DSPU) and emulates a PU 2 to host LLC2 session, providing the opportunity to observe circuits across DLSw+.
Appendix A: Configurations-Session Generators
For the standard tests, the initiator configuration was consistent. For the test with the mappings, the MAC addresses were changed to target the bit-swapped logical MAC address on each of the DLSw+ Ethernet Redundancy routers:
The host configuration was as follows: