Contents

Introduction

Logs To be Collected

Basic Logs to be Collected

HeartBeat Miss Logs

EOBC/EPC Related logs

Fabric Related

Slice Manager

MPA Manager

LED

PD Install

Details

Analysis using card-mgr logs

Use show tech ctrace to do Offline Analysis

Decode ctrace offline on Ads Server

Identify the Primary SC and Primary SUP

Retrieve Post-Codes from Primary SC

Event-history from Primary RP point of view:

Triage Scenarios Example

How to Pull, Edit and Build Satori Lineup?

Command to Build the Kernel

How to Make the Changes for Different klms and get it Compiled?

Fretta Fixed Cards HW and FPGA Specs

Introduction

This document describes the guidelines on how to analyze and find the root causes of platform-dependent issues on the Fretta (NCS5500) family of routers.

Logs To be Collected

For any platform-dependent triages, collect the basic logs listed below. Depending on the suspected component or functionality area, collect other logs in addition to basic logs, as indicated in the subsections below.

Basic Logs to be Collected

- show logging

- show tech ctrace

- show tech card-mgr

- show tech shelf-mgr

- show tech os

- show tech obfl

HeartBeat Miss Logs

- show tech heart-beat miss

EOBC/EPC Related logs

Please refer to to traige EOBC/EPC specific issues.

- show tech control-ethernet

Fabric Related

- show tech fabric from xrvm

Slice Manager

- show tech-support sdr_mgr

- show tech-support install

- show tech-support fabric

- show tech-support fpd

- show tech-support cm

- show controller fabric health

- show platform slices

MPA Manager

- show tech-support mpa-mgr

LED

- show tech-support envmon

- show tech-support alarm_mgr

- show alarms

- show led

- show environment

Please refer these links for more detail on LED and status.

PD Install

Please refer to to traige PD Install issues

Details

Analysis using card-mgr logs

- Take some time to go through the Card Manager Detailed wiki at

- In order to debug any platform issue, you need to use show tech ctrace, show tech card-mgr, and show tech shelf_mgr logs to understand what is going on.

- show reboot-history card location <> from shelf_mgr logs provide the reboot history of the card.

- show controllers card-mgr event-history brief location <> and show controllers card-mgr event-history detail location <> from show tech card-mgr logs provide details on the card-mgr fsm state machine details.

- When a card fails to boot, you need to look at event history for that card, and based on which state/event the card is stuck/failed, you need to check with either BIOS, PD install, or card-mgr point of view. Each event has an associated postcode, which provides clues to the FSM movement.

Note: If a line card fails to boot and goes to FAILED/FAILED state, then you need to connect to the line card console after force reloading the card. This helps you to understand why the line card is failing to boot. show tech is unable to collect logs from a failed line card.

Please refer to card-mgr detailed wiki mentioned above to understand the meaning of specific state, event, and postcode details. Also refer to card manger smil file located at calvados/dc_common_pkg/drivers/card_mgr/card_mgr_fsm.smil. This file has a good description of the FSM states, events, and state transition.

Here is an example of working case event-history brief output when an LC is cold-reset:

sysadmin-vm:0_RP0# show controller card-mgr event-history brief location 0/1

Mon Dec 16 14:47:58.974 UTC+00:00

Card Event History for: 0/1

Card Event History as seen by Master (0/RP0)

Current State: CARD_READY

DATE  TIME (UTC)   STATE EVENT

----- ------------ ---------------------- ------------------------------

12/16 14:46:51.116 WAIT_CARD_INFO ev_card_info_synced

12/16 14:46:06.990 WAIT_SYSADMIN_VM_READY ev_sysadmin_vm_booted

12/16 14:45:57.375 HOST_OS_RUNNING ev_sysadmin_vm_started

12/16 14:45:39.554 BOOTLDR_STARTED ev_host_os_started

12/16 14:44:22.746 CARD_POWERED_ON ev_bootldr_started

12/16 14:44:19.142 IOFPGA_BOOTED ev_dm1_power_up_ok

12/16 14:44:12.825 IOFPGA_RESET_CHECK ev_inserted

12/16 14:44:12.325 CARD_IN_RESET ev_removed

12/16 14:44:10.224 PROCESS_PENDING_RESET if_pending_cold_reset_req

12/16 14:44:10.224 SYSADMIN_VM_GOING_DOWN ev_host_halting_os

12/16 14:43:50.258 SYSADMIN_VM_GOING_DOWN ev_cold_reset_req

12/16 14:43:34.275 CARD_READY ev_sysadmin_vm_shutdown

12/16 11:11:55.291 OIR_INSERT_NOTIF if_card_local_init_done

12/16 11:11:55.290 IDLE ev_card_info_synced

Example of event-history detailed output:

sysadmin-vm:0_RP0# show controller card-mgr event-history detail location 0/1

Mon Dec 16 14:49:20.850 UTC+00:00

Card Event History for: 0/1

Card Event History as seen by Master (0/RP0)

Event buffer info:

Total number of events recorded: 14

Number of events available for display: 14

Current State: CARD_READY

EVENT #: 13 (record index = 13)

TIMESTAMP: 2019/12/16 14:46:51.116090 UTC

STATE: WAIT_CARD_INFO

EVENT: ev_card_info_synced

EVENT DESC: Card info of the remote node has been received

EVENT #: 12 (record index = 12)

TIMESTAMP: 2019/12/16 14:46:06.990465 UTC

STATE: WAIT_SYSADMIN_VM_READY

EVENT: ev_sysadmin_vm_booted

EVENT DESC: SysAdmin VM has booted

EVENT #: 11 (record index = 11)

TIMESTAMP: 2019/12/16 14:45:57.375813 UTC

STATE: HOST_OS_RUNNING

EVENT: ev_sysadmin_vm_started

EVENT DESC: SysAdmin VM has been started from host

EVENT #: 10 (record index = 10)

TIMESTAMP: 2019/12/16 14:45:39.554589 UTC

STATE: BOOTLDR_STARTED

EVENT: ev_host_os_started

EVENT DESC: Host OS has started booting

EVENT #: 9 (record index = 9)

TIMESTAMP: 2019/12/16 14:44:22.746147 UTC

STATE: CARD_POWERED_ON

EVENT: ev_bootldr_started

EVENT DESC: Bootloader on the card has started booting

EVENT #: 8 (record index = 8)

TIMESTAMP: 2019/12/16 14:44:19.142021 UTC

STATE: IOFPGA_BOOTED

EVENT: ev_dm1_power_up_ok

EVENT DESC: I/O FPGA indicating power domain 1 was successfully powered up

EVENT #: 7 (record index = 7)

TIMESTAMP: 2019/12/16 14:44:12.825682 UTC

STATE: IOFPGA_RESET_CHECK

EVENT: ev_inserted

EVENT DESC: Card inserted into the chassis or I/O FPGA booted

EVENT #: 6 (record index = 6)

TIMESTAMP: 2019/12/16 14:44:12.325703 UTC

STATE: CARD_IN_RESET

EVENT: ev_removed

EVENT DESC: Card removed from chassis or I/O FPGA was power cycled

EVENT #: 5 (record index = 5)

TIMESTAMP: 2019/12/16 14:44:10.224354 UTC

STATE: PROCESS_PENDING_RESET

EVENT: if_pending_cold_reset_req

EVENT #: 4 (record index = 4)

TIMESTAMP: 2019/12/16 14:44:10.224343 UTC

STATE: SYSADMIN_VM_GOING_DOWN

EVENT: ev_host_halting_os

EVENT DESC: Host is performing halting of OS

EVENT #: 3 (record index = 3)

TIMESTAMP: 2019/12/16 14:43:50.258016 UTC

STATE: SYSADMIN_VM_GOING_DOWN

EVENT: ev_cold_reset_req

EVENT DESC: Client request to cold reset the card (I/O FPGA is also power-cycled)

EVENT #: 2 (record index = 2)

TIMESTAMP: 2019/12/16 14:43:34.275167 UTC

STATE: CARD_READY

EVENT: ev_sysadmin_vm_shutdown

EVENT DESC: SysAdmin VM shutdown operation has started

EVENT #: 1 (record index = 1)

TIMESTAMP: 2019/12/16 11:11:55.291184 UTC

STATE: OIR_INSERT_NOTIF

EVENT: if_card_local_init_done

EVENT #: 0 (record index = 0)

TIMESTAMP: 2019/12/16 11:11:55.290959 UTC

STATE: IDLE

EVENT: ev_card_info_synced

EVENT DESC: Card info of the remote node has been received

Example of post-codes observed on Primary SC and Primary SUP:

Identify the physical slot number of the card in question and the primary SC and SUP from this output:

location 0/1 physical slot number is 2 (ID column), Primary SC is 0/SC0, and Primary SUP is 0/RP0.

sysadmin-vm:0_RP0# show controller card-mgr inventory summary

Mon Dec 16 14:50:50.810 UTC+00:00

Card Manager Inventory Summary :

BP HW

Location Card Type      ID Serial Number Ver Card State

------------------------------------------------------------------------------

0/1   NC55-32T16Q4H-AT   2 JAE233813G2   0.302 CARD_READY

0/2   NC55-12X100GE-PROT 3 SAL1918EF3S   0.203 CARD_READY

0/FC1 NC55-5504-FC      22 JAE210600VD   0.3   CARD_READY

0/FC3 NC55-5504-FC      24 JAE210600XV   0.3   CARD_READY

0/FC5 NC55-5504-FC      26 JAE210600VX   0.3   CARD_READY

0/RP0 NC55-RP (Master)  27 SAL2044VUZT   1.0   CARD_READY

0/RP1 NC55-RP (Slave)   28 SAL1916DT8B   0.2040 CARD_READY

0/SC0 NC55-SC (Master)  29 SAL2046W07E   1.6   CARD_READY

Use this command to get the post code list for line card 0/1:

sysadmin-vm:0_RP0# show controller card-mgr trace cmgr_isr location 0/SC0 | inc "slot 2" | inc changed

Mon Dec 16 14:56:27.355 UTC+00:00

2019-12-16:11.14.44.916211712:[ISR]: POST Code for slot 2 changed to 0xa0

2019-12-16:11.14.44.916268544:[ISR]: POST Code for slot 22 changed to 0x54

2019-12-16:11.14.44.916295168:[ISR]: POST Code for slot 24 changed to 0x54

2019-12-16:11.14.44.916321280:[ISR]: POST Code for slot 26 changed to 0x54

2019-12-16:11.14.44.916347392:[ISR]: POST Code for slot 27 changed to 0xa0

2019-12-16:11.14.44.916373504:[ISR]: POST Code for slot 28 changed to 0xa0

2019-12-16:11.15.03.646569472:[ISR]: POST Code for slot 26 changed to 0xa0

2019-12-16:11.15.04.748022272:[ISR]: POST Code for slot 22 changed to 0xa0

2019-12-16:11.15.14.266484736:[ISR]: POST Code for slot 24 changed to 0xa0

2019-12-16:11.18.11.489846272:[ISR]: POST Code for slot 2 changed to 0x1e

2019-12-16:11.18.12.491101184:[ISR]: POST Code for slot 2 changed to 0xa0

2019-12-16:11.22.30.391535104:[ISR]: POST Code for slot 2 changed to 0x1e

2019-12-16:11.22.31.492875776:[ISR]: POST Code for slot 2 changed to 0xa0

2019-12-16:11.26.49.407702016:[ISR]: POST Code for slot 2 changed to 0x1e

2019-12-16:11.26.50.509097472:[ISR]: POST Code for slot 2 changed to 0xa0

2019-12-16:11.31.08.408430592:[ISR]: POST Code for slot 2 changed to 0x1e

2019-12-16:11.31.09.409682432:[ISR]: POST Code for slot 2 changed to 0xa0

2019-12-16:11.35.26.315185152:[ISR]: POST Code for slot 2 changed to 0x1e

2019-12-16:11.35.27.416556032:[ISR]: POST Code for slot 2 changed to 0xa0

2019-12-16:11.39.45.310315520:[ISR]: POST Code for slot 2 changed to 0x1e

2019-12-16:11.39.46.311528448:[ISR]: POST Code for slot 2 changed to 0xa0

2019-12-16:11.44.04.337517056:[ISR]: POST Code for slot 2 changed to 0x1e

2019-12-16:11.44.05.338741248:[ISR]: POST Code for slot 2 changed to 0xa0

2019-12-16:11.48.23.232193024:[ISR]: POST Code for slot 2 changed to 0x1e

2019-12-16:11.48.24.333538304:[ISR]: POST Code for slot 2 changed to 0xa0

2019-12-16:11.52.41.234022400:[ISR]: POST Code for slot 2 changed to 0x1e

2019-12-16:11.52.43.336457728:[ISR]: POST Code for slot 2 changed to 0xa0

2019-12-16:11.57.00.153080320:[ISR]: POST Code for slot 2 changed to 0x1e

2019-12-16:11.57.01.254410752:[ISR]: POST Code for slot 2 changed to 0xa0

2019-12-16:12.01.19.178457600:[ISR]: POST Code for slot 2 changed to 0x1e

2019-12-16:12.01.20.179703296:[ISR]: POST Code for slot 2 changed to 0xa0

2019-12-16:12.05.38.203790336:[ISR]: POST Code for slot 2 changed to 0x1e

2019-12-16:12.05.39.205028864:[ISR]: POST Code for slot 2 changed to 0xa0

2019-12-16:12.09.57.103055360:[ISR]: POST Code for slot 2 changed to 0x1e

2019-12-16:12.09.58.204383232:[ISR]: POST Code for slot 2 changed to 0xa0

2019-12-16:12.14.15.027237888:[ISR]: POST Code for slot 2 changed to 0x1e

2019-12-16:12.14.16.128579072:[ISR]: POST Code for slot 2 changed to 0xa0

2019-12-16:12.18.34.047417856:[ISR]: POST Code for slot 2 changed to 0x1e

2019-12-16:12.18.35.148794880:[ISR]: POST Code for slot 2 changed to 0xa0

2019-12-16:12.22.53.047706624:[ISR]: POST Code for slot 2 changed to 0x1e

2019-12-16:12.22.54.048883200:[ISR]: POST Code for slot 2 changed to 0xa0

2019-12-16:12.27.12.054199808:[ISR]: POST Code for slot 2 changed to 0x1e

2019-12-16:12.27.13.055494656:[ISR]: POST Code for slot 2 changed to 0xa0

2019-12-16:12.31.30.979380224:[ISR]: POST Code for slot 2 changed to 0x1e

2019-12-16:12.31.32.080705024:[ISR]: POST Code for slot 2 changed to 0xa0

2019-12-16:12.35.48.888316416:[ISR]: POST Code for slot 2 changed to 0x1e

2019-12-16:12.35.49.989663744:[ISR]: POST Code for slot 2 changed to 0xa0

2019-12-16:12.40.07.891782144:[ISR]: POST Code for slot 2 changed to 0x1e

2019-12-16:12.40.08.993085440:[ISR]: POST Code for slot 2 changed to 0xa0

2019-12-16:12.44.26.908366848:[ISR]: POST Code for slot 2 changed to 0x1e

2019-12-16:12.44.27.909621760:[ISR]: POST Code for slot 2 changed to 0xa0

2019-12-16:12.48.45.918578176:[ISR]: POST Code for slot 2 changed to 0x1e

2019-12-16:12.48.46.919841792:[ISR]: POST Code for slot 2 changed to 0xa0

2019-12-16:12.53.03.837281280:[ISR]: POST Code for slot 2 changed to 0x1e

2019-12-16:12.53.04.838517248:[ISR]: POST Code for slot 2 changed to 0xa0

2019-12-16:12.57.22.831639552:[ISR]: POST Code for slot 2 changed to 0x1e

2019-12-16:12.57.23.832911360:[ISR]: POST Code for slot 2 changed to 0xa0

2019-12-16:13.01.41.833031680:[ISR]: POST Code for slot 2 changed to 0x1e

2019-12-16:13.01.42.834268672:[ISR]: POST Code for slot 2 changed to 0xa0

2019-12-16:13.06.00.740024320:[ISR]: POST Code for slot 2 changed to 0x1e

2019-12-16:13.06.01.841394688:[ISR]: POST Code for slot 2 changed to 0xa0

2019-12-16:13.10.19.768019968:[ISR]: POST Code for slot 2 changed to 0x1e

2019-12-16:13.10.20.769302528:[ISR]: POST Code for slot 2 changed to 0xa0

2019-12-16:13.14.37.655355392:[ISR]: POST Code for slot 2 changed to 0x1e

2019-12-16:13.14.38.756755456:[ISR]: POST Code for slot 2 changed to 0xa0

2019-12-16:13.18.56.655229952:[ISR]: POST Code for slot 2 changed to 0x1e

2019-12-16:13.18.57.756587520:[ISR]: POST Code for slot 2 changed to 0xa0

2019-12-16:13.23.15.658801664:[ISR]: POST Code for slot 2 changed to 0x1e

2019-12-16:13.23.16.660048384:[ISR]: POST Code for slot 2 changed to 0xa0

2019-12-16:13.27.34.655034880:[ISR]: POST Code for slot 2 changed to 0x1e

2019-12-16:13.27.35.656287232:[ISR]: POST Code for slot 2 changed to 0xa0

2019-12-16:13.31.53.652897792:[ISR]: POST Code for slot 2 changed to 0x1e

2019-12-16:13.31.54.654104576:[ISR]: POST Code for slot 2 changed to 0xa0

2019-12-16:13.36.11.558914560:[ISR]: POST Code for slot 2 changed to 0x1e

2019-12-16:13.36.12.560167424:[ISR]: POST Code for slot 2 changed to 0xa0

2019-12-16:13.40.30.568370688:[ISR]: POST Code for slot 2 changed to 0x1e

2019-12-16:13.40.31.569627136:[ISR]: POST Code for slot 2 changed to 0xa0

2019-12-16:13.44.49.468186112:[ISR]: POST Code for slot 2 changed to 0x1e

2019-12-16:13.44.50.571635712:[ISR]: POST Code for slot 2 changed to 0xa0

2019-12-16:13.49.08.482063360:[ISR]: POST Code for slot 2 changed to 0x1e

2019-12-16:13.49.09.583393280:[ISR]: POST Code for slot 2 changed to 0xa0

2019-12-16:13.53.26.395422208:[ISR]: POST Code for slot 2 changed to 0x1e

2019-12-16:13.53.27.496771584:[ISR]: POST Code for slot 2 changed to 0xa0

2019-12-16:13.57.45.399475712:[ISR]: POST Code for slot 2 changed to 0x1e

2019-12-16:13.57.46.500909568:[ISR]: POST Code for slot 2 changed to 0xa0

2019-12-16:14.02.04.405213184:[ISR]: POST Code for slot 2 changed to 0x1e

2019-12-16:14.02.05.406433280:[ISR]: POST Code for slot 2 changed to 0xa0

2019-12-16:14.06.23.417884672:[ISR]: POST Code for slot 2 changed to 0x1e

2019-12-16:14.06.24.419138048:[ISR]: POST Code for slot 2 changed to 0xa0

2019-12-16:14.10.42.329566720:[ISR]: POST Code for slot 2 changed to 0x1e

2019-12-16:14.10.43.430938112:[ISR]: POST Code for slot 2 changed to 0xa0

2019-12-16:14.15.00.253901824:[ISR]: POST Code for slot 2 changed to 0x1e

2019-12-16:14.15.01.355243520:[ISR]: POST Code for slot 2 changed to 0xa0

2019-12-16:14.19.19.247721472:[ISR]: POST Code for slot 2 changed to 0x1e

2019-12-16:14.19.20.349063680:[ISR]: POST Code for slot 2 changed to 0xa0

2019-12-16:14.23.38.254869504:[ISR]: POST Code for slot 2 changed to 0x1e

2019-12-16:14.23.39.256110592:[ISR]: POST Code for slot 2 changed to 0xa0

2019-12-16:14.27.57.261724160:[ISR]: POST Code for slot 2 changed to 0x1e

2019-12-16:14.27.58.262965760:[ISR]: POST Code for slot 2 changed to 0xa0

2019-12-16:14.32.15.158858240:[ISR]: POST Code for slot 2 changed to 0x1e

2019-12-16:14.32.17.261378560:[ISR]: POST Code for slot 2 changed to 0xa0

2019-12-16:14.36.34.186439168:[ISR]: POST Code for slot 2 changed to 0x1e

2019-12-16:14.36.35.187675648:[ISR]: POST Code for slot 2 changed to 0xa0

2019-12-16:14.40.53.126042624:[ISR]: POST Code for slot 2 changed to 0x1e

2019-12-16:14.40.54.227419648:[ISR]: POST Code for slot 2 changed to 0xa0

2019-12-16:14.43.33.504493568:[ISR]: POST Code for slot 2 changed to 0xa1

2019-12-16:14.44.09.450505728:[ISR]: POST Code for slot 2 changed to 0x73

2019-12-16:14.44.18.369435136:[ISR]: POST Code for slot 2 changed to 0x1b

2019-12-16:14.44.21.973499392:[ISR]: POST Code for slot 2 changed to 0xe0

2019-12-16:14.44.45.599875072:[ISR]: POST Code for slot 2 changed to 0xe1

2019-12-16:14.45.26.660646400:[ISR]: POST Code for slot 2 changed to 0xe3

2019-12-16:14.45.28.064965632:[ISR]: POST Code for slot 2 changed to 0xe2

2019-12-16:14.45.30.167515648:[ISR]: POST Code for slot 2 changed to 0xe4

2019-12-16:14.45.33.070848000:[ISR]: POST Code for slot 2 changed to 0xe6

2019-12-16:14.45.38.777229312:[ISR]: POST Code for slot 2 changed to 0x50

2019-12-16:14.45.56.597211648:[ISR]: POST Code for slot 2 changed to 0x54

2019-12-16:14.46.06.211475968:[ISR]: POST Code for slot 2 changed to 0xa0

2019-12-16:14.46.14.720887296:[ISR]: POST Code for slot 2 changed to 0x17

2019-12-16:14.46.15.822237696:[ISR]: POST Code for slot 2 changed to 0xa0

2019-12-16:14.48.29.977753088:[ISR]: POST Code for slot 2 changed to 0x1e

2019-12-16:14.48.31.079104512:[ISR]: POST Code for slot 2 changed to 0xa0

2019-12-16:14.52.48.986328576:[ISR]: POST Code for slot 2 changed to 0x1e

2019-12-16:14.52.49.987563520:[ISR]: POST Code for slot 2 changed to 0xa0

Use show tech ctrace to do Offline Analysis

Decode ctrace offline on Ads Server

1. Copy the show tech ctrace archive to a directory on your ads server.

2. Untar the contents of the archive using - tar zxvf showtech-ctrace-admin-2019-Nov-06.174210.UTC.tgz

3. cd showtech-ctrace-admin-2019-Nov-06.174210.UTC

4. Use this command to decode the traces for card_mgr process under directory cmgr: /users/gonaidu/bin/showtech_ct_dec -d cmgr -p card_mgr

5. cd cmgr

Note: ctrace decoded logs are in the local time zone - If decode was done on Bangalore ADS server, ctraces are based on IST timezone. Please keep this in mind while comparing the router logs and offline decoded ctrace logs.

Identify the Primary SC and Primary SUP

Use "grep "Master Role" *" under cmgr directory:

card_mgr.0_RP0:299:2019-02-22:07.35.38.709224844:2580:calvados/dc_common_pkg/drivers/card_mgr/src/card_mgr_main.c:1539:main:cmgr_main:MAIN_HW_ARB_RESULT:[MAIN]: HW Arbitration Result = Master Role

card_mgr.0_SC0:96:2019-04-06:19.04.34.500975616:1976:calvados/dc_common_pkg/drivers/card_mgr/src/card_mgr_main.c:1539:main:cmgr_main:MAIN_HW_ARB_RESULT:[MAIN]: HW Arbitration Result = Master Role

Retrieve Post-Codes from Primary SC

Use this command to get post codes for LC 0/1: grep "slot 2" card_mgr.0_SC0 | grep changed

2019-04-06:19.04.34.759844864:1976:calvados/dc_common_pkg/drivers/card_mgr/src/card_mgr_interrupt.c:256:iofpga_check_card_post_code_change:cmgr_isr:ISR_CARD_POST_CODE_CHANGED:[ISR]: POST Code for slot 2 changed to 0xa0

Event-history from Primary RP point of view:

grep RCAPI_NOTIFY_CARD_POST_CODE_CB card_mgr.0_RP0 | grep "0\/1"

2019-04-06:19.04.40.665774834:2589:calvados/dc_common_pkg/drivers/card_mgr/src/card_mgr_rack_service.c:1919:cmgr_rack_notify_card_post_code_change_cb:cmgr_rack_capi:RCAPI_NOTIFY_CARD_POST_CODE_CB:[RACK CAPI]: 0/1 - CAPI cmgr_rack_notify_card_post_code_change_cb, client card_mgr (PID=1976,hdl=0x7f7c880ff728,slot=29)

Triage Scenarios Example

- card manager process CPU hog:

- Slice not coming up upon LC reload-Potenza 10C

How to Pull, Edit and Build Satori Lineup?

There can be multiple lineups co-existing. Engineer needs to know from where he/she should pull the view.  To know the appropriate satori lineup/devline, from sysadmin linux prompt run this command

cat /etc/build-info.txt

You get the ouput like this

### Thirdparty Information

SDK arm /auto/exr-yocto/SDK/WRL7/Fretta/REL0109/arm/kvm-host-arm-sdk.tgz

SDK x86_64 /auto/exr-yocto/SDK/WRL7/Fretta/REL0109/x86_64/kvm-host-x86_64-sdk.tgz

Refpoint = thirdparty/opensource/release@tp-main/289

Hostname     : calcium-99.cisco.com

Workspace    : /nobackup/hetsoi/satori-wrl7.release.20191209/target-n9000-gdb

Source Base  : ssh://wwwin-git-sjc-2/git/thinstack/satori.git

Devline      : cisco-xr-wr7

Devline Ver  : f53915539d9ca49d3dedec0882ee4eb12a408956

Devline Type : GIT Repository

Here Devline ‘cisco-xr-wr7’ should be used.

Before pulling the view, setup your environment

Step 2. Ensure this entry is present in your git config file ie; ~/.gitconfig

[apply]

whitespace = nowarn

ignorewhitespace = change

Step 3. Use this command to setup your environment before you do any activity

source /auto/exr-yocto/tools/scripts/set_yocto_env

Step 4. Use this command to pull the view.

acme pull -sb ios_ena -dev cisco-xr-wr7 -plat none

Step 5. Run the below command in your shell

unset CDPATH

Command to Build the Kernel

Step 1. Navigate to satori directory and run this command.

scripts/xr/build-release.sh  -f

Step 2. Once kernel builds successfully, it applies the cisco specific patches to various .c and .h files and code become available at this path for code walk.

satori/target-n9000-gdb/bitbake_build/tmp/work/n9000_gdb-wrs-linux/

Note: you are not supposed to make any permanent change here to get reflected in your binary because this is tmp location and shall get overwritten in the next build. Code at this location should be used for code walk and .patch file generation. The .patch file is discussed here

From a code location point of view, code for klm is located at two-path

- For Code walk and patch generation:

satori/target-n9000-gdb/bitbake_build/tmp/work/n9000_gdb-wrs-linux/

- Actual .c and .h files which are used in binary/sdk builds

satori/meta-cisco-nxos/recipes-kernel/

On the above path, you get two directories

A. cisco-klm —> This covers all the klm which are used in modular and fixed fretta systems.

B. cisco-klm-zermatt —> This covers all the klm which are only used in fixed fretta systems like klm_iofpga.

How to Make the Changes for Different klms and get it Compiled?

To make changes for klm_iofpga is very simple. Just go to this path and start making changes in .c or .h file you are interested in.

satori/target-n9000-gdb/bitbake_build/tmp/work/n9000_gdb-wrs-linux/cisco-klm-zermatt/0.1-r0/klm_iofpga

To make changes for all other klm(s) is a little tricky. As explained above, you need to go to tmp location, make the changes, generate the .patch file (how to generate a .patch file is explained here). Copy the .patch file to a specific location, make an entry for this new .patch file inthe .bb file and start the build.

Please find the steps to make the changes.

Step 1. Go to the klm specific directory where you want to make the changes. You find all the klms at this location.

satori/target-n9000-gdb/bitbake_build/tmp/work/n9000_gdb-wrs-linux/cisco-klm/0.1-r0

Step 2. Use the quilt tool to make the changes, so that the .patch file could be generated. Note, here acme diff does not work, so you need to use the quilt tool to generate the patch file.

Step 3. Setup the quilt tool alias

cd satori/target-n9000-gdb/bitbake_build/tmp/work/n9000_gdb-wrs-linux/cisco-klm/0.1-r0

alias quilt=/nobackup/rpanday/kernel-wr7/satori/target-n9000-gdb/bitbake_build/tmp/sysroots/x86_64-linux/usr/bin/quilt

quilt new patch_file.patch --> Instruct quilt to allocate a new .patch file with name patch_file.patch.

quilt top —> This command tells us that patch_file.patch is at top of the quilt stack and ready to be edited.

quilt edit klm_obfl/obfl_dc3.c —-> Instruct the quilt to capture the changes done in klm_obfl/obfl_dc3.c. Once this command is run, obfl_dc3.c is open and ready for edit. After the edits are done, write and exit.

quilt refresh —-> This command takes the diff from the edited file klm_obfl/obfl_dc3.c and puts it in patch_file.patch. In order to verify the same, open the file in vi.

Step 4. Once the patch file patch_file.patch is generated copy it to this path.

cp patches/patch_file.patch /nobackup/rpanday/kernel-wr7/satori/meta-cisco-nxos/recipes-kernel/cisco-klm/files

Step 5. Make an entry in the .bb file for this new .patch file. The .bb file is located at this location.

/nobackup/rpanday/kernel-wr7/satori/meta-cisco-nxos/recipes-kernel/cisco-klm/cisco-klm_0.1.bb

Fretta Fixed Cards HW and FPGA Specs

PID

Engineering Name

Card Type

HW  Spec

IOFPGA Spec

NCS-5502-SE

ZERMATT

2RU fixed

•       EDCS- 1515475

•       EDCS- 1026647

•       EDCS- 1516467

•       EDCS- 1193041

NCS-5501-SE

TURIN-MX

1RU fixed

•       EDCS- 1497433

•       EDCS- 1527505

•       EDCS- 1527506

NCS-5502-SE-PROTO

ZERMATT-PROTO

NCS-5502

ZERMATT-CR

2RU fixed

•       EDCS- 1515475

EDCS- 1516467

CPU IOFPGA FS: EDCS- 1026647

MIFPGA - EDCS -  1193041

NCS-5501

TAIHU

1RU fixed

EDCS- 1530044

IOFPGA - EDCS- 1541805

MIFPGA - EDCS- 1541804

NCS-5501-A2-SE

WINTERFELL

NCS-5501-A1-SE

OLDCASTLE

NCS-5501-A1

NCS-55A1-36H-S

OLDCASTLE

NCS-55A1-36H-SE-S

OLDCASTLE-SE

1RU fixed

•       EDCS-1563746

•       EDCS-1568105

•       EDCS-11402862

•       EDCS-11556985

NCS-55A1-36H-S

OLDCASTLE-CR

1RU fixed

•       EDCS-1563746

•       EDCS-1568105

•       EDCS-11402862

N540-X-24Z8Q2C-M

TORTIN-CONFORMAL-BOX

N540-24Z8Q2C-M

TORTIN-CR

N540X-ACC-SYS

TORTIN-16G-CR

N540-ACC-SYS

TORTIN-16G

NCS-5501-HD

N540-24Z8Q2C-M

TORTIN-CR

NCS-5501-A3

NCS-55A1-24H

PYKE

NCS-55A1-24H

PYKE

1RU fixed

•       EDCS-  11415948

•       EDCS- 1568105 EDCS- 1026647

•       EDCS- 1568940

NCS-55A2-MOD-SE-S

PEYTO WITH TCAM

2RU fixed

•       EDCS- 11601538

•       EDCS- 11632621

•       EDCS- 11632622

NC55A2-MOD-SE-H-S

PEYTO WITH TCAM & CC

2RU fixed

•       EDCS- 11601538

•       EDCS- 11632621

•       EDCS- 11632622

NCS-55A2-MOD-S

Peyto NonSE C-temp (TCAM)

2RU fixed

•       EDCS- 11601538

•       EDCS- 11632621

•       EDCS- 11632622

NCS-55A2-MOD-HD-S

PEYTO WITHOUT TCAM

2RU fixed

•       EDCS- 11601538

•       EDCS- 11632621

•       EDCS- 11632622

NCS-55A2-MOD-HX-S

PEYTO WITHOUT TCAM WITH Itemp CC

2RU fixed

•       EDCS- 11601538

•       EDCS- 11632621

•       EDCS- 11632622

NCS-55A1-48Q6H

BIFROST-T

1RU fixed

•       EDCS- 12914104

•       EDCS- 13259042

•       EDCS- 15599029

•       EDCS- 15676955

NCS-55A1-24Q6H-S

TURIN-CR

1RU fixed

•       EDCS- 12909672

•       EDCS- 13259042

•       EDCS- 15599029

•       EDCS- 15676955

NCS-55A1-24Q6H-SS

TURIN-CR