Cisco BPX/IGX/IPX WAN Software

Introduction

The software architecture in use on the Cisco IGX 8400 series, BPX 8600 series, and IPX WAN switches restricts certain network modifications when there are one or more unreachable nodes in the network. This document explains why these restrictions are necessary.

Prerequisites

Requirements

Readers of this document should be knowledgeable of the following:

• Cisco WAN Switching Software for the Cisco IGX 8400 series, BPX 8600 series, and IPX WAN switches

Components Used

This document is not restricted to specific software and hardware versions.

Conventions

For more information on document conventions, see the Cisco Technical Tips Conventions.

Restricted Changes

The following changes are restricted any time there is an unreachable node in the network:

• Adding a new node

• Renumbering a node

• Adding a new trunk

• Changing the transmit or receive rate of an existing trunk

• Changing the CC Restrict parameter on any existing trunk

• Changing the Terrestrial/Satellite parameter on any existing trunk

Distributed Network Databases

The network software architecture rests on a distributed network database. No single, centralized component in the network (such as a node or a network management workstation) contains or uses the entire network configuration database. Fundamentally, no component, if damaged or eliminated, can make the entire network stop functioning or be unmanageable. This architecture eliminates the dangers associated with a single point of failure.

Instead, each node in the network maintains an up-to-date database that includes information about the following:

• Every other node in the network (including the node name, number, and type)

• All trunks in the network (including type, transmit rate, receive rate, processor traffic restriction, satellite vs. terrestrial, configured load summary, worst case queuing delays, and alarm status)

• All local modules, lines, and ports

• All Permanent Virtual Circuits (PVCs) that terminate on it

• All PVCs that traverse it

Any change in network topology characteristics is immediately broadcast to all other nodes in the network. This immediacy is required because each node in the network uses the information to determine the following:

• New routes for PVCs through the network

• The communication paths between the node's processors

• The layout of the network synchronization plan

Dangers of Nonsynchronized Databases

If network topology characteristics were to change while a node is unreachable, that node would not receive the database update. Different nodes in a network could have different versions of the same databases.

Network nodes have the ability to exchange databases with each other and they use such exchanges to update themselves and reconcile any differences. The reconciliation protocol is simple and consistent. Any database differences between nodes are resolved by deleting any database entries that do not agree. This is why a trunk can be deleted from a network with an unreachable node but a trunk cannot be added to a network with an unreachable node. When nodes reestablish communication, the databases reconcile conflicting entries, resulting in the deletion of the trunk from the node that was unreachable.

The greatest danger of nonsynchronized databases, specifically the topology database, is the possibility that a node might be unable to reestablish communication with its peers if the network topology has changed while it was unreachable. Each node uses Dijkstra's algorithm to determine which trunk to send messages to peer nodes. The key is that each node selects only the first hop of the best path to each remote node, relying on the downstream node to propagate the message packets to the next hop of the best path, and so on. This works because each node uses the same algorithm to analyze the same topology database. If one node had an incorrect database, then that node might be unable to establish communication with the other nodes.

For example, assume the following network:

Normally, node A communicates with node C over the path A-B-C. Similarly, node D communicates with node C over the path D-A-B-C.

Assume that node D becomes isolated (for example, its power is turned off or both of its trunks fail). This results in a communication failure condition (and perhaps other alarm conditions such as loss of signal) being detected on both trunks. Nodes A and E broadcast this topology change to all other nodes resulting in node D being declared unreachable by every other node in the network:

Assume that while D is unreachable, a new trunk is added between nodes C and E. Nodes A, B, C, E, and F are aware of the new trunk but node D is not:

Consider what happens when node D is restored:

As soon as trunks DA and DE clear their communication failure condition, node A determines that the best path for communicating with node C is A-D-E-C, thereby avoiding the lower-speed trunk BC.

Node D is unaware of the existence of trunk EC and still thinks that any messages for node C should be sent to node A. As a result, nodes C and D can never clear the unreachable state between them.

Furthermore, nodes A and C are now mutually unreachable, even though they could communicate before and during the isolation of node D.

Nodes A and D each think that the other is the correct path to node C, with the result that neither of them can communicate with node C.

Conclusion

Given the fundamental architecture of the distributed topology database as implemented in the Cisco IGX 8400 series, BPX 8600 series, and IPX WAN switches, network topology changes cannot be allowed to the network while any node in the network is unreachable.

Related Information

• Cisco WAN Switching Solutions - Cisco Documentation
• Guide to New Names and Colors for WAN Switching Products
• Downloads - WAN Switching Software
• Technical Support - Cisco Systems