RTOS/TMS320C6678: MessageQ_get and cache coherence

Hello

For my project I used as starting point the example in [INSTALL_DIR]\ipc_3_42_00_02\packages\ti\ipc\tests\messageq_multicore.c

I have 8 cores running simultanously. Cores from 1 to 6 send messages to CORE7. The sequence of events is the following
(NOTE: I remove code from my project to debug more easily)

FPGA sends IRQ to CORE7 (every 2.7 ms), CORE7 wakes up CORE1
CORE1 sends 2 messages to CORE7 and wakes up CORE2
CORE2 sends 2 messages to CORE7 and wakes up CORE3
CORE3 sends 2 messages to CORE7 and wakes up CORE4
CORE4 sends 2 messages to CORE7 and wakes up CORE5
CORE5 sends 2 messages to CORE7 and wakes up CORE6
CORE6 sends 2 messages to CORE7

So CORE7 receives 2*6 = 12 messages at each interrupt

In my .cfg file I have

var SHAREDMEM = 0x0c100000;
var SHAREDMEMSIZE = 0x00080000;

var SharedRegion = xdc.useModule('ti.sdo.ipc.SharedRegion');
SharedRegion.setEntryMeta(0,
{ base: SHAREDMEM,
len: SHAREDMEMSIZE,
ownerProcId: 0,
isValid: true,
cacheEnable: true,
cacheLineSize: 128,
createHeap: true,
name: "sharedMem",

});

At startup each core creates its own local list of messages by calling

MessageQ_Msg coreMsgList[5];
initFunc()
{
  for(int i=0; i< 5; i++)
  {
    coreMsgList[i] = MessageQ_alloc(HEAPID, size);
  }
}

size is 1244 bytes (MessageQ_MsgHeader + myBodySize). My message is:

typedef struct
{
MessageQ_MsgHeader msgHeader; /**< Required first field in every message. */
MyBody body; /**< Body of the message. */
} TMessageComm, *MessageComm_handle ;

MessageQ_free IS NEVER CALLED, each sender uses cyclically messages created at startup.

coreMsgList contains 5 messages, and 5 is big enough to ensure that the same message is not used twice at the same time
Please note that after the IRQ from FPGA, CORE7 takes 150 microsecond to receive all 12 messages (Oscilloscope + test point analisys)
and the FPGA sends an IRQ every 2.7 millisecond

Each sender does the following (pseudo code)

sendFunction()
{
  while((CSL_semAcquireDirect (semNum)) == 0)


  MessageQ_Msg msg = coreMsgList[fMsgIndex];

  //...read msgId from DDR3

  MessageQ_setMsgId((MessageQ_Msg)msg, msgId);

  //...write stuff in the body

  MessageQ_put(queueId, msg);

  fMsgIndex = (fMsgIndex+1)%5;

  printf("0x%08X [%d] %d %d\n",(int)msg, MultiProc_self(), MessageQ_getMsgId((MessageQ_Msg)msg), fMsgIndex);

  msgId++;

  //...write msgId to DDR3

  CSL_semReleaseSemaphore (semNum)
}

msgId is a global counter shared among cores (shared in DDR3, cache disabled) and incremented every time a message is sent

The receiver CORE7 does the following (pseudo code)

while(1)
{
  MessageComm_handle msg; //my Message (MessageQ_MsgHeader + myBody)

  MessageQ_get(messageQ, (MessageQ_Msg *)&msg, MessageQ_FOREVER);

  UInt16 msgId = MessageQ_getMsgId((MessageQ_Msg)msg);

  //Check for msgId correctness

  if(check fail)
  {
    printf("Receive Err get:%d expected:%d \n", msgId, expected);
    assert(1==0);

  }
}

*** TEST NUMBER 1 ***

For debug purpose, I send IRQ 'manually' from FPGA to CORE7, to activate the process. This is the output

NOTE: first column shows the address of message used, returned by MessageQ_alloc
second column is the sender coreID
third column is the global counter used to identify messages
fourth column is the index of coreMsgList

I send first interrupt, output:

0x0C10F900 [1] 1 0
0x0C111200 [1] 2 1
0x0C110300 [2] 3 0
0x0C110D00 [2] 4 1
0x0C10D100 [3] 5 0
0x0C10D600 [3] 6 1
0x0C113500 [4] 7 0
0x0C114E00 [4] 8 1
0x0C10DB00 [5] 9 0
0x0C10E500 [5] 10 1
0x0C112600 [6] 11 0
0x0C113000 [6] 12 1

As you can see this output is OK, no printf from CORE7, meneaning that the check is OK
Global counter is correctly incremented
Each core sends 2 messages as expected
the index of coreMsgList is correctly incremented

Second interrupt, output

0x0C111C00 [1] 13 2
0x0C112100 [1] 14 3
0x0C111700 [2] 15 2
0x0C114400 [2] 16 3
0x0C10E000 [3] 17 2
0x0C10EA00 [3] 18 3
0x0C116200 [4] 19 2
0x0C116C00 [4] 20 3
0x0C10EF00 [5] 21 2
0x0C10FE00 [5] 22 3
0x0C113A00 [6] 23 2
0x0C114900 [6] 24 3

Also this output is OK

third IRQ, output

0x0C112B00 [1] 25 4
0x0C10F900 [1] 26 0
0x0C115800 [2] 27 4
0x0C110300 [2] 28 0
0x0C10F400 [3] 29 4
0x0C10D100 [3] 30 0
Receive Err get:5 expected:30

We have the error, CORE3 sends message with ID 30, CORE7 read msgID 5

You can see that message with ID 30 uses address 0x0C10D100, already used with msgID 5, so I think CORE7 is reading the old message

0x0C10F900 [1] 1 0
0x0C111200 [1] 2 1
0x0C110300 [2] 3 0
0x0C110D00 [2] 4 1
0x0C10D100 [3] 5 0 ------> SAME ADDRESS
0x0C10D600 [3] 6 1
0x0C113500 [4] 7 0
0x0C114E00 [4] 8 1
0x0C10DB00 [5] 9 0
0x0C10E500 [5] 10 1
0x0C112600 [6] 11 0
0x0C113000 [6] 12 1
0x0C111C00 [1] 13 2
0x0C112100 [1] 14 3
0x0C111700 [2] 15 2
0x0C114400 [2] 16 3
0x0C10E000 [3] 17 2
0x0C10EA00 [3] 18 3
0x0C116200 [4] 19 2
0x0C116C00 [4] 20 3
0x0C10EF00 [5] 21 2
0x0C10FE00 [5] 22 3
0x0C113A00 [6] 23 2
0x0C114900 [6] 24 3
0x0C112B00 [1] 25 4
0x0C10F900 [1] 26 0
0x0C115800 [2] 27 4
0x0C110300 [2] 28 0
0x0C10F400 [3] 29 4
0x0C10D100 [3] 30 0 ------> SAME ADDRESS

--------------------------------------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------------------------------------

*** TEST NUMBER 2 ***

I modified the receiver code adding cache invalidate AFTER MessageQ_get

while(1)
{
  MessageComm_handle msg; //my Message (MessageQ_MsgHeader + myBody)

  MessageQ_get(messageQ, (MessageQ_Msg *)&msg, MessageQ_FOREVER);

  INVALIDATE_CACHE; <----------- ************************

  UInt16 msgId = MessageQ_getMsgId((MessageQ_Msg)msg);

  //Check for msgId correctness

  if(check fail)
  {
   printf("Receive Err get:%d expected:%d \n", msgId, expected);
   assert(1==0);

 }
}

My INVALIDATE_CACHE function does the following:

1) CACHE_invL1d of 1280 bytes (1280 > 1244 and multiple of 128)
2) prefetch buffer invalidation
3) mfence

The situation is better, I get the same error but after a big number of IRQ , usually > 10.000 IRQ from FPGA
Obviously I did it in free running, with the FPGA sending an IRQ every 2.7 millisecond, I remove all printf except the error print from CORE7

I remove sender printf but I create a shared buffer (4096 location, cyclic used) to store and verify ('post mortem') sender activities , senders are OK!

--------------------------------------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------------------------------------

*** TEST NUMBER 3 ***

I modified the receiver code adding cache invalidate BEFORE MessageQ_get

while(1)
{
  MessageComm_handle msg; //my Message (MessageQ_MsgHeader + myBody)

  INVALIDATE_CACHE; <----------- ************************

  MessageQ_get(messageQ, (MessageQ_Msg *)&msg, MessageQ_FOREVER);

  UInt16 msgId = MessageQ_getMsgId((MessageQ_Msg)msg);

  //Check for msgId correctness

  if(check fail)
  {
  printf("Receive Err get:%d expected:%d \n", msgId, expected);
  assert(1==0);

  }
}

SAME error occurs

--------------------------------------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------------------------------------

*** TEST NUMBER 4 ***

I modified the way I allocate Messages, each sender calls MessageQ_alloc just befor MessageQ_put
CORE7 calls MessageQ_free after MessageQ_get

No cache ivalidation

SAME error occurs

--------------------------------------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------------------------------------

*** TEST NUMBER 5 ***

Same as before with INVALIDATE_CACHE AFTER MessageQ_get

SAME error occurs
--------------------------------------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------------------------------------

*** TEST NUMBER 6 ***

Same as before with INVALIDATE_CACHE BEFORE MessageQ_get

SAME error occurs

What is wrong with my setup?

Thank you

Hello Fabio,

Thank you for the detailed explanation. I will need some time to look into this and get back to you.

In the meantime, a couple of things to try:

- Increase the heap size in your .cfg file.

- Alternatively, use MessageQ_staticMsgInit instead of messageq_alloc to allocate the messageQ statically:

Also, can you clarify whether your messageQ struct contains pointers to data or the actual message data?

Hello

I will take a look at the ping example.

For your information, I'm not using the last sdk available, Im using the following products

SysBIOS 6.45.1.29
IPC 3.42.0.02
MCSDK 2.1.2.6
MCSDK PDK 1.1.2.6
compiler 8.1.0
CodeComposer 6.1.3Is this a problem?

Below you can find
- map file
- linker.cmd
- cfg file

Thank you

SECTIONS
{
	.DDRSimMem:       > DDR_SIM


	.MSharedSram:  >  MSMCSRAM_DATA

	platform_lib:  >  MSMCSRAM_CODE
	.csl_vect:     >  MSMCSRAM_CODE

	.cinit:	       >  MSMCSRAM_CODE
	.const:        >  MSMCSRAM_CODE
	.switch:	   >  MSMCSRAM_CODE
}

SPSW_3D_map.txt

SpSwBios.cfg

Hello 

Ok I'm going to upgrade to the latest version, in the meantime can you take a look to my cfg and map files to see if they are ok?

thank you