Multishelf Configuration Example and Common issues with MS configuration and resolution

MS Configuration Procedure and Common issues with MS configuration and resolution

Introduction:
This Document describes the basic Multishelf Configuration example and common issues for troubleshooting.
Feature Identifier: Multishelf Configuration Example and common issues troubleshooting and resolution.

Prerequisites:

1) Basic knowlegde of multishelf concept.

2) Able to perform CTC and LCD panel provisioning.

3) Basic knowledge of Cisco 15454 M6 and M12 chassis

Document Overview:

This document presents an overview of the Cisco multishelf configuration for M12 and M6 shelf.
ONS 15454 multiservice transport platform (MSTP) multishelf configuration and common issues during new turn up and operational  Nodes.All the issues and workaround/solution are updated based on the various field issues reported.

What are shelf Types in multishelf Configuration:

•    Controller shelf
•    Subtended shelf

Hardware Requirement:

Controller shelf
One to twenty nine subtending shelves if used M6 shelf as node controller.
Two Catalyst switches or two MS-ISC cards (MS-ISC card will only be used when M12 shelf is used as node controller)

 Multishelf Connectivity Diagram:

 


M12 as Node Controller and connected to switch :

Switch to ONS connectivity details:
ONS 15454 Node controller shelf
•    TCC 7 to Catalyst 1 port 1
•    TCC 11 to Catalyst 2 port 1
ONS 15454 subtending shelves 1 to 7
•    N shelf TCC 7 to Catalyst 1 port n
•    N shelf TCC 11 to Catalyst 2 port n
Catalyst connections
•    Each Catalyst port 23 to network
•    Catalyst 1 port 22 to Catalyst 2 port 22
 Multishelf uses internal IP addresses
•    192.168.190.16x where x=shelf number: 2,3,4,5,6,7, or 8
Telnet to controller shelf
Login
Ping "192.168.190.162" and get a reply if shelf 2 is provisioned correctly

Provisioning the Controller Shelf:

• Navigate to Shelf> Provisioning> Multishelf
• Click Enable as Node Controller
• Choose Stand-alone
• Click Apply

Once it is selected and hit Apply then shelf will go for restart and will appear as Node controller Shelf.

Basic Catalyst Switch Configuration

Trunk ports
•    Ports 1 and 22 are trunk ports
Access ports
•    Ports 2 through 8 are access ports on VLAN 2
•    Ports 23 and 24 are access ports on VLAN 1
Remaining ports are disabled
Provisioning Port 1 and Port 22 as trunk ports:
Switch(config)#int fa0/1
Switch(config-if)#switchport mode trunk
Switch(config-if)#switchport trunk encap dot1Q
Switch(config-if)#switchport trunk allowed vlan 1,2
Switch(config-if)#switchport nonegotiate
Switch(config-if)#switchport trunk pruning vlan none

Provisioning Port 2 and Port 21  access  ports:
Switch(config)#int fa0/2
Switch(config-if)#switchport mode access
Switch(config-if)#switchport access vlan 2

Provisioning Port 23 in VLAN 1 as it will connect from switch to external network
Switch(config)#int fa0/23
Switch(config-if)#switchport mode access
Switch(config-if)#switchport access vlan 1

In Case MS-ISC is used MS connectivity then MS-ISC is having the preconfigured ML card and need to connect as mentioned below:
ONS 15454 controller shelf
•    TCC 7 to MS-ISC left port 9
•    TCC 11 to MS-ISC right port 9
ONS 15454 subtended shelves 1 to 7
•    N shelf TCC 7 to MS-ISC left port n
•    N shelf TCC 11 to MS-ISC right port n
MS-ISC connections
•    Each MS-ISC port 0 to network
•    MS-ISC left port 10 to MS-ISC right port 10

Subtending Shelf Provisioning

Connect to subtending shelf with CTC
•    Choose Shelf > Provisioning > Multishelf Config.
•    Choose Enable as Subtending Shelf.
•    Choose Shelf ID.
Or config from front panel
•    Scroll to MS menu.
•    Choose MS=Y.
•    Choose Shelf number = n.
•    Choose Ethernet = Y.





Results
•    Shelf reboots with neither TCC active
•    LCD window reports “Waiting for CT”
Go to Controller Shelf
•    Navigate to Shelf > Provisioning > Multishelf.
•    Right-click on first multishelf.
•    Choose Add Shelf with number = n.
•    Change Stand-alone to Ethernet.
•    Click Apply.
Results
•    Multishelf reboots and downloads from controller shelf
•    CTC window changes to multishelf configuration
Multishelf view from CTC  


Reference Links for Multishelf Provisioning Documents:
http://www.cisco.com/c/en/us/td/docs/optical/hardware/15454install/guide/hig15454/hig_15454.html#wp546337

http://www.cisco.com/en/US/partner/docs/optical/hardware/15454install/guide/hig_15454.html#wp547312

Common issues with Multishelf Provisioning and Resolution Details:

Problem 1:
Inserting New Shelf in existing Multishelf Configuration.

Resolution:

• Planned to insert the new multi-shelf i.e. shelf-4 in the existing 3 shelves multishelf configuration Setup but when tried to insert the new shelf,  were not receiving any options on the LCD panel to change the shelf ID. Removed both the controller cards from the shelf-4 and inserted the standby TNC card of shelf-3 slot-3 in slot-8 of shelf-4.After reconnecting the LAN cables were able to add the shelf-4 in CTC.
• But when tried to insert the spare TNC-E cards which was from shelf-4 in shelf-3 slot-8, it was not coming up.
• Even when tried to insert the TNCE card in shelf-4, they were not booting, so  thought the new TNCE cards are bad.
• Checked the Software release of the node which was 9.21 and  the TNCE card are not compatible with the TNC card in this release and they are compatible after release 9.30.
• We removed the TNC card we inserted in shelf-3 slot-8 and it came up.

Problem-2
Shelf-comm-fail on shelf 2

Resolution:

• Field Tech on site found both LAN ports in blocking/disabled state.
• Have unplugged and re-plugged back in which resolved the issue.

Problem-3

Adding M6 as subtended shelf to M12 NC with TCC3 Card and subtended card s/w download failing on subtended shelf TNC card.

Resolution:

• It was analyzed that the Node Controller TCC3 had lite software version as protect partition. Since TNC/TSC doesn't support lite version, getting this issue. Need to load full version as protect in node controller shelf.
• After putting the full version on protect partion(download the full version again) the subtended shelf was able to get the S/W from NC and multishelf configuration was successful.

Problem-4

Its Completely New Node*** Multi-shelf and VLAN provisioning button was grayed out in.

Resolution:
Not able to change the provisioning using the LCD buttons.Deleted CTC cache and no change.Deleted the db and after that were able change it to multishelf.

Problem-5
MW was scheduled to subtend  M12 shelf to M6 node controller.

Resolution:

• M12 shelf was prepared with TCC3 cards and subtended to NC but it never came up. Tried reset/reseat to all controllers of the shelf but shelf 2 never came up.
•  Arranged two TCC2P cards to site
• Added one of them to M12 and configured it to shelf 2 to subtend with NC but it failed too.
• Then used other spare TCC2P card. Downloaded correct MSTP software to it, as that of Node Controller and subtended it. This worked and shelf 2 got added. Then added TCC2P card  (which did not worked before) to stand by and it was
• Observed that it was having MSPP SW release and hence it failed. Same problem observed with other two TCC3 cards.
• When TCC3 card came up as stand by it copied correct software from active controller.
• Then side switch over was performed and added other TCC3 as well. Subtended shelf came up good with TCC3 cards.
• TCC3 cards were not having correct software copy.

Problem-6
9.21 to 9.605 2d shelf in multishelf upgrade failed.

After upgrading node from 9.221 to 9.605 shelf 2 did not upgrade.
In debug shelf 2 TNC slot 1 is not showing software 9.605 other TNC in slot 8 is showing stand by and has both softwares. Shelf Communication failure alarm was observed on shelf-2

Resolution:

• Remove TNC card from slot 1.
• Wait for 10 minutes if slot 8 does not take over then reseat the TNC card in slot 8.
• After shelf 2 comes up then insert slot 1 in shelf.

Issue resolved after executing the recommended action Plan

Problem-7
Adding new subtended shelf & TNC-E in new shelf remain in loading state.

Resolution:

•  The node controller having sw ver 9.203
• Subtended shelf having TNC-E controller card
• TNC-E card do not support software earlier than 9.3 which is causing issue and it was continuously showing into loading state.Node controller software upgraded to 9.605 and after that subtended shelf got added to main shelf successfully.

Problem-8

M6 subtended shelf-4 was not getting added in Multi-shelf with M12 node controller.

Resolution:

•  Successfully changed the multi-shelf config in shelf -4 like MS=Y, ID=4 and VLAN=Y via LCD.
• When connected shelf-4 with switch and it was not getting added as multishelf.Both TSC cards were not showing LED status.
• Card did not boot in shelf-4 after connecting to switch.
• Then removed LCD, remove sl-1 and kept only sl-8 TSC card in the shelf and connect the LAN cable in MSM ports in ECU and it worked and sl-8 card booted and became active.
• Then sl-1 and LCD inserted and normally shelf came in CTC shlef-4.

Problem-9
Unable to connect to M6 node from local login.

Resolution:

•  Were unable to ping the node.LCD panel LED show SC waiting for provisioning and multi-shelf.  This is
  Supposed to be a standalone node.Use the LCD buttons to disable to the multishelf config.  After the TNC reset were able to local login to Node.

Problem-10

All cards in shelf are constantly rebooting

• Node Software versin:9.211
  In Shelf 3,Slot 7 is active and slot 11 is standby.
   Slot 7 card going into loading state periodically after 1-2 minutes and all the transponder cards in slot 1,3,12,13,14,17 shows into loading state.
  When slot7 is in loading state then slot 11 is also going into loading state and not coming up as active.
   Tried to perform reset to this card (slot7)from telnet session but it is not allowing slot 11 to become as active.
   Once the slot 11 became active but after some time it again went into loading state and all the transponder card reporting equipment failure.

Resolution:
Action Performed when field tech was available on site:

• Slot7 removed and with slot 11 as active checking the other card LED indications. All other cards are showing in loading state.
• Removed the slot 11 card as well and then tried to insert the new spare card in slot7.
• Card came up properly but it was showing as node controller card and not able to communicate with the main controller card.
• Tried to local login to change it to subtended shelf but due to java incompatibility we were not able to login locally.
• Tried to make it multishelf from LCD panel but field tech was not able to get the option from LCD panel to MS configuration. Strange.
• Tried reseat of LCD but still situation was same.
•  Removed both the shelf controller cards and now removed the controller card from shelf2 and inserted in shelf 3 slot7 and card booted up properly and was getting the option to  change the MS configuration.(At this time we have removed all the LAN connections from shelf-3)
• Changed it to shelf 3 and connected as it was connected earlier in multishelf configuration and now we were able to get the shelf 3 communication back.
•  inserted the new TCC2P card in slot 11 and it came up as standby properly.
• Inserted all the cards one by one and all the cards booted properly and came back up.
•  After this activity not observed any further reboot to shelf and all the traffic card stop auto reboot.
• Arranged the new TCC card and placed in shlef2 slot11 from where we have took the card out for shelf3 restoration and it booted up and came as standby properly.
• Traffic confirmed stayed up.

Problem-11

M12 shelf running 9.6.05 on TCC3 to which  trying to add an M6 shelf running same version on TSC-E however the software download process never stopped for 18 hours.

Resolution:

• checked the switches configuration and it is fine.
• tried to remove the multishelf config.
• deleted 3rd shelf from node controller.
• removed it from Lan connection.
• booted it up separately.
• tried changing active/standby on node controller.
• added the new node again to the node controller.
• plugged in the LAN.
• slot 8 on new shelf3 goes through the software download process and keeps a loop on there.
• not coming up ever.
• removed slot 8 TSCE/
• it comes out of the software download process, but never comes up from Loading state.
• Resolution Summary:
• Removed the shelf 3 from MS configuration and performed flmdelete db,usb on
• shelf3 in standalone mode and directly login to shelf and made it subtended shelf-3 from CTC.
• observed the main Node controller was having the working S/W load as full version and protect s/w load as light version.
• Tried to download the full version on protect flash partition and connected the shelf-3 to MS and shelf 3 came up properly.

Problem-12

Shelf-4 was having shelf-comm failure alarm.
shelf-4 the TCC2P card in slot-7 was in cyclic reboot and slot-11 was only
showing PWR-A and PWR-B as green, else no lights on the card.

Resolution:

•  Checked via VxWorks (shelfConns) and found that shelf-4 was missing from the list of suspended shelves list.
• Slot-7 was in cyclic reboot so seems that the TCC2P card was not communicating to the node controller and declared shelf-comm alarm.
• Slot-11 was not showing any status neither active nor standby.
• Suspecting that the issue was with the TCC2P card in shelf-4 which was not communicating with the NC.
• Suggested to change the switch port connected to the shelf-4.
• By changing the switch port, issue resolved and  able to see the shelf-4.
• Reseat the slot-7 shelf-4 TCC card, as it was stuck in loading state and was
• Reporting EQPT failure and not came up.
• Tried to insert the spare TCC in slot-7 and it took around 20mins to boot completely.
• Slot-7 comes up as standby and slot-11 was active in shelf-4.

Problem-13

Cannot add M6 chassi with TSC's to existing M12 multi-shelf.

Resolution:

• As per problem statement, have TCC3 cards and full version of R9.603 in Node controller shelf (Node COntroller was M12 with TCC3 Cards.
• Webex'ed  and telnet'ed to node and dumped flmStat, this showed protect version had lite r9.603 loaded.
• Downloaded full versionR9.603 in Node controller and readded m6 chassis.TSC's came up fine after this step.

Problem-14

Attempting to add a M12 and a M6 node to an existing Multishelf node.

Resolution:

• The M12 shelf came up ok as shelf 3; however, the M6 shelf will not come up. The LEDs areoff  on the TNC cards, and only have Link/Act. The display shows "SC waiting Prov". The M6 reboot about every 10-15minutes.
•  The issue was identified the wrong patching of M6 port to switch. plugging the MS swt into the M6 TNC
• LAN port.  moved cable to MSM p1.  Node came up normal.

Problem-15

Shelf communication failure on shelf-2

Resolution:

• Both the shelf controller cards were not showing either Active or Standby.
• Removed both the controller cards and inserted the TCC from Node controller,but it was not coming up and could not able to change the shelf-ID through CTC as well as LCD.Inserted back the TCC card to node controller and let it boot up completely and when it was showing up properly as standby with DB sync then switched the active from slot-11 to slot-7 in Node controller.Observed the shelf communication restored back.

Problem-16
New Shelf addition is not taking place.

Resolution:

• New M6 shelf was with TSC cards.
• Software version on Node controoler shelf was 9.604 and 9.40 (lite version) in protect flash.M6 TSC card doesn't support 9.40 lite version that was the problem preventing SS M6 to come up.
• Downloaded full software version 9.604 on controller as well on M6 chassis, after making connection all came up fine.

Problem-17

During Software Upgrade lost subtended shelf, observing shelf Communication Failure.
An external switch was in use for multi-shelf configuration.

Resolution:

• During software activation TCC card takes reboot and because of that connectivity between TCC Ethernet port and switch Ethernet port drops.
•  After TCC boots got completed, communication between subtended shelf and main shelf didn't restored.
•  This communication between two shelf's should happen via switch.
•  Please note that Ethernet port on TCC card is of 10Mbps and half duplex.
  Whereas in this case, switch interface was 100Mbps and full duplex auto neg.
•  So it means after TCC reboot got completed switch interface failed to negotiate speed and duplex settings. So changed switch interface to half duplex and 10Mbps.

Problem-18

All cards in shelf5 of Node going for loading state.

Resolution:

• Observed the shelf 5 all the cards were in continuous loading state with slot8 as active
• When slot8 is in loading state then it was not allowing slot1 to be active.
• Tried to telent to shelf 5 but it was not allowing to go inside.
• Removed the card from slot8 and shelf 5 observed working stable.
• Telnet inside shelf 5 was possible and showing the correct status.
• Inserted new card in slot 8 and slot 8 copied all the provisioning information from slot8.
• Shelf 5 found working good.

Key Points before provisioning MS:

• Type of Node controller Cards in use (If it is M12 shelf with TCC3) then Make sure the TCC3 cards are loaded with full software version and should not have the light version if we are trying to    subtend the M6 shelf below M12 controller.
• When using the M6 shelf as Node controller then EMS port on ECU will be connected to external switch and MSM port will be used for multishelf node cascading.
• Verify the switch to controller card/subtended shelf card connectivity first before proceeding for any further equipment level workaround.
• Verify the LCD panel provisioning status which will indicate type of provisioning performed on the node.
• Verify the LED Indication on node controller and shelf controller.Do not reseat the card if other mate controller card is in loading state or having any critical alarm and contact TAC for further troubleshooting.