firmware_loader fails on first boot

Hi Markus,

How you are initiating the DSP link (ie through any scripts or manually) ?

FIRMWARE: Memory map bin file not passed

You have not passed the memory args for DSP link so It is taking default args and it is causing the error.

Refer the following TI wikis for DSP FAQs and troubleshooting.

http://processors.wiki.ti.com/index.php/DSPLink_FAQs#Generic_questions

http://processors.wiki.ti.com/index.php/Troubleshooting_DSPLink_configuration_issues
http://processors.wiki.ti.com/index.php/Changing_DSPLink_Memory_Map
http://processors.wiki.ti.com/index.php/Category:DSPLink
http://processors.wiki.ti.com/index.php/DSPLink_POOL_Module_Overview

Markus,

I'm wondering if its a timing issue with other parts of the kernel or file system. Try adding a delay in your script and see if it helps. Start with something large like 10 seconds and then reduce it until it fails.

~Ramsey

Hello Titus,

loading the dsp firmware is done using rc.d init-scripts

You have not passed the memory args for DSP link so It is taking default args…

That is correct…

and it is causing the error.

…this isn't, sorry. We have never passed any memory map bin files to the firmware_loader. It always uses the default args and it works fine.

The links you've posted all have "DSPLink" in their URLs. AFAIK DSPLink is deprecated and indeed the pages all show a bright red "deprecated" header. We're using syslink 2.20.02.20 from ezsdk 5.05.02.00 (I think I mentioned the ezsdk in the initial post…)

On of the reasons that we use no memory map bin file is that we've found no documentation how to create/use it. Do you have any recent documentation on that topic?

Thanks,
Markus

Hello Ramsey,

we'll try a delay and have a look if it works. It'll take a while because the bug can be reproduced only after waiting so long and the device is in use.

Nevertheless, I can't imagine it is a timing issue. Two invocations of the firmware_loader are working without errors, so all that's needed for syslink seems to work. On the other hand, I really have no better explanation. We'll see :)

Thanks for your help,
Markus

We tried the delay, but it didn't help :(

Markus,

I'm trying to understand the timing. Are you saying the board has to be up and running for a long period of time (> 1hr) and then when you power cycle it the DSP load fails? Or is it that the board has to be off for a long period of time and then when you power on the board the DSP load fails?

In either case, I understand that the same firmware script is used to load the DSP with the same executable during the Linux boot sequence. Is this correct?

Are there trace logs you can look at? I'm not familiar with the SDK but on some systems you can mount a debug file system and look at DSP trace logs.

Another idea is to check the log events on the DSP. After the failure occurs, use CCS to attach the DSP and look at its log buffer. Maybe you will find some clues.

I'm hoping someone on the SDK team will help out.

~Ramsey

Hello Ramsey,

That's the case:

is it that the board has to be off for a long period of time and then when you power on the board the DSP load fails?

In either case, I understand that the same firmware script is used to load the DSP with the same executable during the Linux boot sequence. Is this correct?

I'm not aware of any trace logs, but as you say maybe someone from ezsdk can help? In the meantime I'll do as you suggested and attach CCS after the failure occurs. I'll report back with any news.

Thanks,

Markus

If I attach my CCS after the error occurred, I get no meaningful messages from the log buffer. So I tried this:

Yesterday, I installed a firmware-version that blocks right after reaching main, so that I can attach my CCS before anything happens. This morning, I started the board, attached my CCS, released the DSP from the blocking loop and started stepping…

Int main(Int argc, Char* argv[])
{
    Int status = Ipc_S_SUCCESS;

#ifdef _BLOCK_AT_MAIN_
    #warning DSP will block on load and has to be released by a debugger
    volatile Bool bBlock=TRUE;
    while( bBlock  )
    {
    }
#endif

	initialiseCalculation();
	initialiseEDMAForFPGA();

    do {
        /* Call Ipc_start() */
        status = Ipc_start();
    } while (status != Ipc_S_SUCCESS);

    BIOS_start();

    return (0);
}

I also set a breakpoint in my MessageQ initialisation:

    /* Connect to remote processor. firmware_loader blocks until this step is passed. */
    while (Ipc_attach(rProcId) < 0)
    {
        Task_sleep(1);
    }

The target stoppt at Task_sleep(1). Unfortunately, I hit "continue" and didn't step on, so I don't know exactly what caused the following message that appeared:

Trouble Reading Memory Block at 0x9a000030 on Page 0 of Length 0x800:
Error 0x00000002/-1202
Error during: Memory,

CPU pipeline is stalled and the CPU is 'not ready'. This means
that the CPU has performed an access which has not
completed, and the CPU is waiting. The target may need to be
reset. The user can choose 'Yes' to force the CPU to be 'ready'.
When this is done, the user will have the ability to examine
the target memory and registers to determine the cause of the
CPU stall. If CPU hang is caused by application and it has been
forced to be 'ready', the CPU should not be run without a reset.

Yes - force CPU ready (might corrupt the code)
Disconnect - disconnect CCS so that it can be reset
Retry - attempt the command again

0x9a000030 is somewhere in the DSP_DATA section. Googling the error code shows two results, but I don't think they apply.

Since the error will appear again only tomorrow morning (sigh), I'll try then to step on. Has anyone an Idea what could cause that error?

cu
Markus

Just had a look at the .map file:

9a000030 xdc_runtime_LoggerBuf_Instance_State_0_entryArr__A

Hmm, what has the LoggerBuf to do with the problem? Or is it just coincidence?

Markus,

How is the SysLink driver module loaded? Sometimes, it takes a while after loading the module before you can use SysLink. If the module has not had enough time to fully initialize before the firmware loader starts, you will run into issues. When you added the delay to your startup script, was it between loading SysLink and running the firmware loader? If not, try adding a delay of 5 seconds.

As for debugging the startup sequence with CCS. This will probably not work. When SysLink loads and starts the DSP, if it does not handshake soon enough, SysLink assumes something went wrong on the DSP and puts it back into reset. It might be that when you attach CCS to the DSP, the handshake is delayed and SysLink will reset the DSP. The next time to attempt to single step in CCS, you might get an emulation error.

The best way to debug startup issues is to add Log events and then read the log buffer after the failure to see how far you get. However, LoggerBuf encodes the log statements, so this is difficult. I would try using LoggerSys just for now. This logger write formatted strings to memory, which you can then read from the debugger. In CCS, attach to the DebugSS (not the DSP) and then open a memory window to the LoggerSys buffer. You should be able to read the strings.

~Ramsey

Hello Ramsey,

We load syslink like this:

SYSLINK_PATH=`find /lib/modules/$(uname -r) -name 'syslink.ko'`
if [ $? -ne 0 ] ; then
	echo "####################################################################";
	echo "############## Syslink Module not found !!! ########################";
	echo "####################################################################";
	exit
fi
insmod $SYSLINK_PATH
until [[ -e /dev/syslinkipc_ProcMgr && -e /dev/syslinkipc_ClientNotifyMgr ]]
do                                                
	echo "Probing syslink devices..."
	sleep 1
done

And I think it is working because loading HDVICP2 and HDVPSS firmware succeeds. The DSP firmware is loaded a while after those two (see my first post in this thread)

The delay was added right before the call to firmware_loader, after insmod syslink and after the other two firmwares were loaded.

I'll follow your suggestion to have a look at the logger buffer. I've found DebugSS and it is completely new for me. What is it and where can I read more about it?

My only problem with this approach is: how can I find the address of the LoggerSys buffer? I had a look at the map file but found nothing that looked like it.

Markus,

Looks like the timing is not the issue. You added the delay in the correct place.

The DebugSS is simply a debug port your can attach to with CCS. I use it to execute gel functions and to look at memory using the system view (physical addresses). I don't use it for anything else. I don't know where to get more information on it. You could try the CCS forum.

I left out some details regarding LoggerSys, sorry. LoggerSys does not have a log buffer, it sends its output by calling System_printf() which uses the system support proxy. The default proxy is typcialy xdc.runtime.SysStd which sends the output of System_printf() to the CIO buffer. You could look for __CIOBUF_ in the map file, but this buffer is usually quite small. It happens to be the one that CCS reads to send data to the CCS Console window. But I recommend you use the xdc.runtime.SysMin proxy. This proxy has a buffer, called xdc_runtime_SysMin_Module_State_0_outbuf__A, which is used to store the output of System_printf(). You can configure the size of this buffer. This is where I would look using CCS/DebugSS.

Here is the config code you would need:

/* configure System module */
var SysMin = xdc.useModule('xdc.runtime.SysMin');
SysMin.bufSize = 0x1000;
SysMin.flushAtExit = false;

/* configure the system support proxy */
var System = xdc.useModule('xdc.runtime.System');
System.SupportProxy = SysMin;

/* create the logger instance */
var LoggerSys = xdc.useModule('xdc.runtime.LoggerSys');
var loggerSysP = new LoggerSys.Params();
Defaults.common$.logger = LoggerSys.create(loggerSysP);

When you look in the map file, use the list of symbols sorted alphabetically to find all objects related to a module. For example, search for xdc_runtime_SysMin to see all public symbols created by that module. Use the list of symbols sorted by address to see what is placed nearby and to compute the size of an object.

Note: when you use SysMin instead of SysStd, you will not get any output in the CCS Console window. But you should be able to see the same output in ROV. Look for the SysMin module and then the OutputBuffer tab. You will need to halt CCS.

~Ramsey

Hello Ramsey,

thank you for the clarification. I'll be quite busy for the next few days and won't be able to try it. I'll report back when I've made some progress.

cu

Markus