Introduction
This document describes the DDF reload event when the Cyclic Redundancy Check (CRC) Error occurred. The event is reported with Simple Network Management Protocol (SNMP) trap DDFreload. The trap is introduced with the CRC Error Restart Notification for Operation and Maintenance feature.
Problem
DDF Field-Programmable Gate Array (FPGA) is a DMA engine on the DPC and DPC2. DDF FPGAs are susceptible to receive a CRC_ERROR. The DDF FPGA driver decides whether or not the error can be recovered. This decision is based on how many times and the rate at which these errors occur. When the driver decides that these errors can be recovered, it indicates to the application program that such an error has occurred.
Thu Apr 01 02:54:09 2021 Internal trap notification 1332 (DDFreload) card 3 ddf-dev DDF1
2021-Apr-01+02:54:09.277 card 3-cpu1: Bad dheader magic number. previous=0xf1234567 (p[12345678.123456] mcdma: MDF/DDF FPGA 3 ch6 acket addr: 0xf2
2021-Apr-01+02:54:09.327 card 3-cpu0: [12345678.123789] DF2 Complex-0 Program DDF2 CAF_DF1_PROG_ERR error detected on SAD1234567
The sessmgr crashes with dhdr.bdh_magic can be observed here:
Assertion failure at midplane/libsn_midplane.c:1845
Function: sn_midplane_dma_buffer_packet_get()
Expression: packet->dhdr.bdh_magic == 0x1974
Proclet: sessmgr (f=87000,i=40)
Process: card=3 cpu=1 arch=X pid=40961 cpu=~18% argv0=sessmgr
Crash time: 2021-Apr-28+14:54:10 UTC
Recent errno: 11 Resource temporarily unavailable
Build_number: 76955
Stack (2680@0x0xffd28000):
[ffffe430/X] __kernel_vsyscall() sp=0xffd28378
[0d0d4c67/X] sn_assert() sp=0xffd283d8
[0d1cef88/X] sn_midplane_dma_buffer_packet_get() sp=0xffd28478
[06b85352/X] sessmgr_med_data_receive() sp=0xffd284f8
[0d15cca4/X] sn_epoll_run_events() sp=0xffd28548
[0d16979a/X] sn_loop_run() sp=0xffd289f8
[0ce5bc25/X] main() sp=0xffd28a68
Solution
The DDF reload fixes the CRC_ERROR issue and no further actions are required usually. Rarely the subscriber impact is reported after the DDF reload in such a case the manual card migration fixes the traffic issue.
# card migrate from <affected card> to <standby card>
In release 21.19 (or later) additional feature is introduced, the feature monitors the internal pipeline of the FPGA and triggers recovery if any issues after the DDF reload are detected.
Feedback