DM814x DSP firmware_loader stop/start hang

Dan,

I can successfully reproduce this DSP firmware load issue on my DM8148 EVM with the default EZSDK 5.05.02.00 / PSP 04.04.00.01.

Let us start from the error reported:

#firmware_loader 0 /usr/share/ti/ti-media-controller-utils/dm814x_c6xdsp.xe674 start -i2c 0
FIRMWARE: Memory map bin file not passed
FIRMWARE: isI2cInitRequiredOnM3: 0
FIRMWARE: Default memory configuration is used
Firmware Loader debugging not configured
Default FL_DEBUG: warning
Allowed FL_DEBUG levels: error, warning, info, debug, log
MemCfg: DCMM (Dynamically Configurable Memory Map) Version :  2.1.2.1
FIRMWARE: Ipc_CONTROLCMD_STARTCALLBACK Error: ProcMgr status 0xffffffff
FIRMWARE: Could not start: -1

The Ipc_CONTROLCMD_STARTCALLBACK call performs a handshake with the slave processor (DSP) typically enabling the slave-side calls to Ipc_attach() to succeed. This call is performed in the firmware_loader.c file:

/* Load the firmware and start the processor */
Int start_firmware (UInt16 procId, uint8_t* pMemCfgSpace, uint8_t ipcInit_memCfg)
{
.............
if (TRUE == ipcInit_memCfg) {
    status = Ipc_control (procId, Ipc_CONTROLCMD_STARTCALLBACK, NULL);

    if (status < 0)
    {
      UTL_TRACE(TRACE_ERROR, "FIRMWARE: Ipc_CONTROLCMD_STARTCALLBACK Error: " \
             "ProcMgr status 0x%x\n", status);
      return -5;
    }
  }
...................
return 0;
}

This Ipc_control() function is defined in:
ti-ezsdk_dm814x-evm_5_05_02_00/component-sources/syslink_2_20_02_20/packages/ti/syslink/ipc/hlos/usr/Ipc.c file:

Int Ipc_control(UInt16 procId, Int cmdId, Ptr arg)
{
...........
switch (cmdId) {
case Ipc_CONTROLCMD_STARTCALLBACK:
                status = _SharedRegion_setRegions();

                if (status < 0) {
                    status = Ipc_E_FAIL;
                    GT_setFailureReason(curTrace, GT_4CLASS, "Ipc_control",
                            status, "_SharedRegion_setRegions API Failed!");
                }
      
                /* register the terminate event with the driver */
                status = Ipc_addTerminateEvent(procId);

                if (status < 0) {
            
                    GT_setFailureReason(curTrace, GT_4CLASS, "Ipc_control",
                            status, "Ipc_addTerminateEvent failed");
                }
                break;
..................................
}

As you can see, this Ipc_control() function that actually failed is part of the syslink module (syslink_2_20_02_20), which makes me think the error should be related to the syslink module.

In your usecase, when you load/start the DSP firmware, the syslink module is already inserted from the load-hd-firmware.sh script. But when you unload/stop the DSP firmware, you have actually do not remove the syslink module. And I suspect this is the actual reason that some resource (memory or other) is used up at the 211 attempt to load/start the DSP firmware.

Taking as example the loading/unloading of the HDVICP2/HDVPSS firmware from the load-hd-firmware.sh script, we can see that before every load/start of the firmwares, we insert the syslink before that (modprobe syslink). And on every unload/stop of the firmwares, the syslink is removed after that (rmmod syslink). If I try the same approach with the DSP firmware load/unload, I can successfully start the DSP firmware for more than 211 times (I stopped the flow at 338). Below are the modifications:

root@dm814x-evm:~# /etc/init.d/pvr-init stop
root@dm814x-evm:~# /etc/init.d/matrix-gui-e stop
root@dm814x-evm:~# /etc/init.d/load-hd-firmware.sh stop
root@dm814x-evm:~# II=1
root@dm814x-evm:~# while true; do
>     echo; echo "========== $II: $(date) =========="
>     echo; echo "=== Starting DSP firmware =="; echo
>     modprobe syslink
>     sleep 1
>     firmware_loader 0 /usr/share/ti/ti-media-controller-utils/dm814x_c6xdsp.xe674 start -i2c 0    
>     sleep 1
>     echo; echo "=== Stoping DSP firmware =="; echo
>     firmware_loader 0 /usr/share/ti/ti-media-controller-utils/dm814x_c6xdsp.xe674 stop -i2c 0
>     rmmod syslink      
>     sleep 1
>     II=$(( II + 1 ))
> done

Please let me know what do you think about this approach (removing/inserting syslink along with the start/stop of the DSP firmware).

Best regards,
Pavel

Pavel,

First, thank you for responding

Pavel Botev said:

Please let me know what do you think about this approach (removing/inserting syslink along with the start/stop of the DSP firmware).

The main problem I have with this suggestion is that when I make a similar while loop to stop/start the load-hd-firmware.sh, I find that the result in much worse.

• On the DM814x-EVM, I got an extremely long (or maybe multiple) kernel panic dump, starting from a NULL dereference usually.  This usually occurs sometime between the 10th and 40th iteration.

• On our custom hardware, it seems more stable.  I do not get the kernel panic, but eventually the firmware_loader hangs in much the same way as the DSP load.  The number of successful cycles seems variable.  It's failed as quickly as 10 iterations, and as long as 1900+ iterations.  (Maybe it can go even higher - the hang is not as predictable as the DSP hang.)

Can you try this on your setup?

/etc/init.d/matrix-gui-e stop
/etc/init.d/pvr-init stop
II=1
while true; do
    echo; echo "========== $II: $(date) =========="
    echo; echo "=== Stopping M3 f/w =="; echo
    /etc/init.d/load-hd-firmware.sh stop
    sleep 1
    echo; echo "=== Starting M3 f/w =="; echo
    /etc/init.d/load-hd-firmware.sh start
    sleep 1
    II=$(( II + 1 ))
done

For your reference, I have attached the following logs in a zip file to this post:

• 2_m3_crash_with_latest_ti_code.txt - shows crash when running on DM814x-EVM, using the latest Arago ti81xx-master tags of linux-omap3 and u-boot-omap3, and the OMX/HDVPSS overlay.

• 3_m3_crash_with_original_EZDSK.txt - shows crash using the unaltered targetfs from the 5.05.02.00 EZSDK (and of course the unaltered uImage and u-boot.bin).

Dan -

Dan,

Daniel70334 said:

On the DM814x-EVM, I got an extremely long (or maybe multiple) kernel panic dump, starting from a NULL dereference usually.  This usually occurs sometime between the 10th and 40th iteration.

I run 3 tests on the DM814x-EVM, and I can reproduce this error (Unable to handle kernel NULL pointer dereference at virtual address 00000010) on 96th iteration, 288th iteration and 151th iteration.

=== Stopping M3 f/w ==

Unloading HDVICP2 Firmware
FIRMWARE: I2cInit will be done by M3
FIRMWARE: Memory map bin file not passed
Usage : firmware_loader <Processor Id> <Location of Firmware> <start|stop> [-mmap <memory_map_file>] [-i2c <0|1>]
===Mandatory arguments===
<Processor Id>         0: DSP, 1: Video-M3, 2: Vpss-M3
<Location of Firmware> firmware binary fUnable to handle kernel NULL pointer dereference at virtual address 00000010
ile
<start|stopgd = c0004000
p>           to [00000010] *pgd=00000000start/stop the f
irmware
===OptInternal error: Oops: 17 [#1]
last sysfs file: /sys/module/vpss/initstate
Modules linked in: tlc59108 ti81xxhdmi ti81xxfb vpss syslink ipv6 [last unloaded: syslink]
................

I will try to find the root cause of this kernel oops.

Meanwhile can you run tests with no sleep, sleep 3 and sleep 5, does it make any difference?

Best regards,
Pavel

I suspect this kernel oops is caused by the SysLink module. I found the below E2E thread which looks very related:

http://e2e.ti.com/support/embedded/linux/f/354/t/138501.aspx

On other (internal) E2E thread I found that "kernel oops: 17" occur when the kernel is under heavy loading. This is high utilization error of the kernel scheduler.

You can add more debug information output from the syslink module, when inserting it like:

load-hd-firmware.sh

modprobe syslink TRACE=1 TRACEENTER=1 TRACECLASS=3 TRACEFAILURE=1

See http://processors.wiki.ti.com/index.php/SysLink_UserGuide#Trace.2C_debug_and_build_configuration

BR
Pavel

Pavel,

Pavel Botev said:

I will try to find the root cause of this kernel oops.

Great, thank you!

Pavel Botev said:

Meanwhile can you run tests with no sleep, sleep 3 and sleep 5, does it make any difference?

I tested using the as-delivered EZSDK, without modification.  I got the crash with no sleep, sleep 1 (original report), sleep 3, and sleep 5.  All had the hang.

Pavel Botev said:

I suspect this kernel oops is caused by the SysLink module. I found the below E2E thread which looks very related:

http://e2e.ti.com/support/embedded/linux/f/354/t/138501.aspx

This post seems not applicable.  Robert Tivy said that Markus must "build syslink.ko with USE_SYSLINK_NOTIFY=0" - the EZSDK's Rules.mk already includes this:

# Syslink build variables
SYSLINK_BUILD_VARS = DEVICE=$(SYSLINK_PLATFORM) \
GPPOS=Linux \
LOADER=ELF \
SDK=EZSDK \
IPC_INSTALL_DIR="${IPC_INSTALL_DIR}" \
BIOS_INSTALL_DIR="${SYSBIOS_INSTALL_DIR}" \
XDC_INSTALL_DIR="${XDC_INSTALL_DIR}" \
LINUXKERNEL="${LINUXKERNEL_INSTALL_DIR}" \
CGT_ARM_INSTALL_DIR="${CSTOOL_DIR}" \
CGT_ARM_PREFIX="${CSTOOL_PREFIX}" \
CGT_C674_ELF_INSTALL_DIR="${CODEGEN_INSTALL_DIR}" \
USE_SYSLINK_NOTIFY=0

Also, Robert Tivy said "Or, you could upgrade to SysLink 2.03.00.82 which contains automatic detection of the Linux 'notify' module and configures SysLink accordingly."  The EZSDK 5.05.02.00 already has syslink 2.20.02.20.

So by both solutions, it should not be an issue.  No?

Pavel Botev said:

On other (internal) E2E thread I found that "kernel oops: 17" occur when the kernel is under heavy loading. This is high utilization error of the kernel scheduler.

How can the kernel be busy?  The TI matrix demo was already stopped, and the demo doesn't do anything by itself anyway.

Pavel Botev said:

You can add more debug information output from the syslink module, when inserting it like:

load-hd-firmware.sh

modprobe syslink TRACE=1 TRACEENTER=1 TRACECLASS=3 TRACEFAILURE=1

See http://processors.wiki.ti.com/index.php/SysLink_UserGuide#Trace.2C_debug_and_build_configuration

I tried adding this, but the logs are excessive - I cannot get past a single loop.  It seems like the loop execution just stops while the log output is dumping to the serial port.

 Dan -

Pavel,

Daniel70334 said:

You can add more debug information output from the syslink module, when inserting it like:

load-hd-firmware.sh

modprobe syslink TRACE=1 TRACEENTER=1 TRACECLASS=3 TRACEFAILURE=1

See http://processors.wiki.ti.com/index.php/SysLink_UserGuide#Trace.2C_debug_and_build_configuration