The software architecture in use on the Cisco IGX 8400 series, BPX 8600
series, and IPX WAN switches restricts certain network modifications when there
are one or more unreachable nodes in the network. This document explains why
these restrictions are necessary.
Readers of this document should be knowledgeable of the
This document is not restricted to specific software and hardware
For more information on document conventions, see the
Cisco Technical Tips
The following changes are restricted any time there is an unreachable
node in the network:
Adding a new node
Renumbering a node
Adding a new trunk
Changing the transmit or receive rate of an existing trunk
Changing the CC Restrict parameter on any existing trunk
Changing the Terrestrial/Satellite parameter on any existing trunk
The network software architecture rests on a distributed network
database. No single, centralized component in the network (such as a node or a
network management workstation) contains or uses the entire network
configuration database. Fundamentally, no component, if damaged or eliminated,
can make the entire network stop functioning or be unmanageable. This
architecture eliminates the dangers associated with a single point of
Instead, each node in the network maintains an up-to-date database that
includes information about the following:
Every other node in the network (including the node name, number, and
All trunks in the network (including type, transmit rate, receive
rate, processor traffic restriction, satellite vs. terrestrial, configured load
summary, worst case queuing delays, and alarm status)
All local modules, lines, and ports
All Permanent Virtual Circuits (PVCs) that terminate on
All PVCs that traverse it
Any change in network topology characteristics is
immediately broadcast to all other nodes in the network. This
immediacy is required because each node in the network uses the information to
determine the following:
New routes for PVCs through the network
The communication paths between the node's processors
The layout of the network synchronization plan
If network topology characteristics were to change while a node is
unreachable, that node would not receive the database update. Different nodes
in a network could have different versions of the same databases.
Network nodes have the ability to exchange databases with each other
and they use such exchanges to update themselves and reconcile any differences.
The reconciliation protocol is simple and consistent. Any database differences
between nodes are resolved by deleting any database entries that do not agree.
This is why a trunk can be deleted from a network with an unreachable node but
a trunk cannot be added to a network with an unreachable node. When nodes
reestablish communication, the databases reconcile conflicting entries,
resulting in the deletion of the trunk from the node that was
The greatest danger of nonsynchronized databases, specifically the
topology database, is the possibility that a node might be unable to
reestablish communication with its peers if the network topology has changed
while it was unreachable. Each node uses
to determine which trunk to send
messages to peer nodes. The key is that each node selects only the first hop of
the best path to each remote node, relying on the downstream node to propagate
the message packets to the next hop of the best path, and so on. This works
because each node uses the same algorithm to analyze the same topology
database. If one node had an incorrect database, then that node might be unable
to establish communication with the other nodes.
For example, assume the following network:
Normally, node A communicates with node C over the path A-B-C.
Similarly, node D communicates with node C over the path D-A-B-C.
Assume that node D becomes isolated (for example, its power is turned
off or both of its trunks fail). This results in a communication failure
condition (and perhaps other alarm conditions such as loss of signal) being
detected on both trunks. Nodes A and E broadcast this topology change to all
other nodes resulting in node D being declared unreachable by every other node
in the network:
Assume that while D is unreachable, a new trunk is added between nodes
C and E. Nodes A, B, C, E, and F are aware of the new trunk but node D is
Consider what happens when node D is restored:
As soon as trunks DA and DE clear their communication failure
condition, node A determines that the best path for communicating with node C
is A-D-E-C, thereby avoiding the lower-speed trunk BC.
Node D is unaware of the existence of trunk EC and still thinks that
any messages for node C should be sent to node A. As a result, nodes C and D
can never clear the unreachable state between them.
Furthermore, nodes A and C are now mutually unreachable, even though
they could communicate before and during the isolation of node D.
Nodes A and D each think that the other is the correct path to node C,
with the result that neither of them can communicate with node C.
Given the fundamental architecture of the distributed topology database
as implemented in the Cisco IGX 8400 series, BPX 8600 series, and IPX WAN
switches, network topology changes cannot be allowed to the network while any
node in the network is unreachable.