Endless spinning in EDMA3_RM_create when initializing EDMA3LLD with OpenMP enabled

Clemens Eisserer

Hello,

When compiling our application with OpenMP enabled, the EDMA3-LLD fails to initialize - even when we are not using any parallel sections / code.
The initialization is done in strictly serial code, I am quite puzzeld why it fails - however it works perfectly fine when compiling with openmp disabled.

The hang occurs in edma3init():

EDMA3_MAX_RM_INSTANCES is some bogus uninitialized value.

As soon as uncommenting the openmp-specific parts of the cfg-file and linking against rts6600_elf_mt.lib instead of rts6600_elf.lib, the following code prints instead of 8 some undefined negative value:

extern const uint32_t EDMA3_MAX_RM_INSTANCES;
void initDMA() {
printf("RM Instances %d \n", EDMA3_MAX_RM_INSTANCES);
}

I would be really grateful for ideas about what could be the issue here.
We are using cgt 7.4b2 with openmp sys/bios runtime 1.1.2.03_beta, and EDMA-LLD 2.11.5.

Thank you in advance, Clemens

over 13 years ago

0 Clemens Eisserer over 13 years ago

Expert 2430 points

I think this might be some kind of linker bug (or a really dubious problem).

I was able to reduce the issue to a small, self-contained testcase - I have uploaded the project: 2465.OmpLinkTest.zip

It would be really great if someone from TI could give it a try, to verify wether this is a real bug or rather a stupidity on my side.

To compile without OpenMP, please do the following steps:
- Uncheck the "Enable support for OpenMP 3.0" in Properties->Compiler->Advanced Options -> Advanced Optimizations
- In Properties->Linker->"File Search Path" replace rts6600_elf_mt.lib with rts6600_elf.lib
- Comment out the OpenMP related options in EV3D.cfg

When the program is executed without OpenMP, it reports correctly:

RM Instances 8

With OpenMP support turned on however, I get

RM Instances 0

when running with the Simulator, or some undefined value on the EMV6678 - even although it doesn't use any parallel constructs.

Thanks, Clemens

0 Yogesh Siraswar over 13 years ago in reply to Clemens Eisserer

TI__Mastermind 20805 points

Clemens,

I am unable to reproduce the problem with the attached OmpLinkTest.zip.

I always see the following output with OMP enabled:

[C66xx_0] RM Instances 8

These are the version of the component used:

- PDK Version 1.1.0.2

- IPC Version 1.24.2.2

- EDMA3 2.11.

- The platform file used: ti.sdo.ipc.examples.platforms.evm6678

Let me know if I am missing anything.

Thanks

Yogesh

0 Clemens Eisserer over 13 years ago in reply to Yogesh Siraswar

Expert 2430 points

Hi Yogesh,

Thanks a lot for taking a look at this issue.

Its really unfourtunate the problem is hard to repoduce :/
Did you use the cfg-file supplied with the project?

I have the following versions installed:
CGT 7.4b2
OpenMP: 1.1.2.03_beta
PDK: 1.1.0.2
IPC: 1.24.3.32
EDMA3: 2.11.5
Rtsc profile: debug (but happens with release as well)
Platform file used: ti.omp.examples.platforms.evm6678

I have attached the pre-compiled executable: 4212.EV3D.zip
When running that executeable in the functional simulator, by 1. starting, 2. loading on all cores, 3. waiting for the openmp-master-thread to reach the main-label and clicking run, I get: "[c66xx_0] RM Instances 0"

I've uploaded the output of a full project rebuild, hopefully it sheds some light what is going on:

Fullscreen 0636.build_log.txt Download

**** Build of configuration Debug for project EV3D ****

C:\ti\ccsv5\utils\bin\gmake -k all 
'Building file: ../EV3D.cfg'
'Invoking: XDCtools'
"C:/ti/xdctools_3_23_03_53/xs" --xdcpath="C:/ti/edma3_lld_02_11_05_02/packages;C:/ti/bios_6_33_05_46/packages;C:/ti/pdk_C6678_1_1_0_2/packages;C:/ti/ccsv5/ccs_base;C:/ti/ipc_1_24_03_32/packages;C:/ti/omp_1_01_02_03_beta/packages;" xdc.tools.configuro -o configPkg -t ti.targets.elf.C66 -p ti.omp.examples.platforms.evm6678 -r debug -c "C:/ti/cgt-7.4b2" "../EV3D.cfg"
making package.mak (because of package.bld) ...
generating interfaces for package configPkg (because package/package.xdc.inc is older than package.xdc) ...
configuring EV3D.xe66 from package/cfg/EV3D_pe66.cfg ...
Inside EDMA3 Drv Sample getLibs
    will link with ti.sdo.edma3.drv.sample:lib/c6678-evm/66/debug/edma3_lld_drv_sample.ae66
Inside EDMA3 Drv getLibs
    will link with ti.sdo.edma3.drv:lib/66/debug/edma3_lld_drv.ae66
Inside EDMA3 RM getLibs
    will link with ti.sdo.edma3.rm:lib/c6678-evm/66/debug/edma3_lld_rm.ae66
cle66 package/cfg/EV3D_pe66.c ...
'Finished building: ../EV3D.cfg'
' '
'Building file: ../newmain.c'
'Invoking: C6000 Compiler'
"C:/ti/cgt-7.4b2/bin/cl6x" -mv6600 --abi=eabi -g --display_error_number --diag_warning=225 --openmp --preproc_with_compile --preproc_dependency="newmain.pp" --cmd_file="./configPkg/compiler.opt"  "../newmain.c"
'Finished building: ../newmain.c'
' '
'Building target: EV3D.out'
'Invoking: C6000 Linker'
"C:/ti/cgt-7.4b2/bin/cl6x" -mv6600 --abi=eabi -g --display_error_number --diag_warning=225 --openmp -z -m"EV3D.map" --reread_libs --warn_sections -o "EV3D.out" -l"./configPkg/linker.cmd"  "./newmain.obj" -l"C:\ti\cgt-7.4b2\lib\rts6600_elf_mt.lib" 
<Linking>
warning #10247-D: creating output section ".TI.tls" without a SECTIONS
   specification
warning #10247-D: creating output section ".TI.tls_init" without a SECTIONS
   specification
'Finished building target: EV3D.out'
' '

**** Build Finished ****

Please also find the generated linker.cmd here: 8255.linker.zip

Please don't hestitate to ask me if further information would be useful to debug this problem.

Thanks again, Clemens

EDIT: Just imported the project on my production system (different CCS 5.2 installation on a different machine), and was able to reproduce the issue on my EVM6678:

[C66xx_0] RM Instances 1796653345

0 Yogesh Siraswar over 13 years ago in reply to Clemens Eisserer

TI__Mastermind 20805 points

Clemens,

I was able to reproduce the problem on simulator. I do see the value at address of EDMA3_MAX_RM_INSTANCES in MSMC_NOCACHE pointing to 0. Since OpenMP runtime maps DDR address to MSMC using MPAX register and makes it non-cacheable, I am suspecting issue with MPAX functionality in simulator. I will get back to you on this after confirming from the simulator team.

However on my EVM I do not see this issue even with the executable you attached. The same executable correctly over-writes the memory at EDMA3_MAX_RM_INSTANCES (0xA01949A4) with 8 after each load. Could you please verify the value at the register address. Also could you please provide the following information:

The bootmode for the EVM.
If you are using any gel file to configure the DDR.

Thanks,

Yogesh

0 Clemens Eisserer over 13 years ago in reply to Yogesh Siraswar

Expert 2430 points

Hi Yogesh,

Glad to hear it did show up at least in the simulator.

I will upload the gel-file tomorrow, however I am working with a LAN-capable debugger and the board is located 20 miles away from my office, so it will take till next monday until I can read the jumper settings. Is there any other (remote) way to determine the boot mode?

Thanks again, Clemens

0 Clemens Eisserer over 13 years ago in reply to Clemens Eisserer

Expert 2430 points

Hi Yogesh,

Just checked - I am using the evmc6678l.gel file located in omp_1_01_02_03_beta\packages\runscript\platforms\ on all 8 cores.
Bootmode will have to wait until Monday, except for the case it can be determined in a remote (trough the debugger or by executing code) way.

Update: The issue seems to occur only at the first run, right after the evm has been cold rebooted:

[C66xx_0] RM Instances -350306271
[C66xx_0] RM Instances 8
[C66xx_0] RM Instances 8

Thanks a lot, Clemens

PS: I made some screenshots of the debugger to show you how the memory is initialitzed:

0 Clemens Eisserer over 13 years ago in reply to Clemens Eisserer

Expert 2430 points

Hello again,

Any update on this problem?
I am running my EVM in "no boot" boot mode (all DIP-switches set to "on" expect the endian one is off).

Thanks you in advance, Clemens

0 Yogesh Siraswar over 13 years ago in reply to Clemens Eisserer

TI__Mastermind 20805 points

Clemens,

I am unable to reproduce this issue on EVM. I have a PG 1.0 EVM. Could you please let me know the version you are using?

Thanks

Yogesh

0 Clemens Eisserer over 13 years ago in reply to Yogesh Siraswar

Expert 2430 points

Hi Yogesh,

I was able to find the problem - it was project related. The linker settings had initialization model = none set, which didn't seem to cause any problems with single-threaded code, but caused unreliable behaviour with OpenMP enabled. With initialization model = ROM, everything works as expected on my EVM.

Glad that at least a Simulator-Bug was found through this mistake.

Thanks, Clemens

Processors

Processors forum

Endless spinning in EDMA3_RM_create when initializing EDMA3LLD with OpenMP enabled