TDA4VMXEVM: IPC LLD Sequence

Hi Thomas,

Thank you for your patience - it looks like your thread maybe got cut off or locked - can you provide context for the above?

Regards,
Mike

Hi Mike,

sure. I copied my original post below:

Hi,

I get in touch with IPC on J721e SoC. As I undestand, I can only use IPC Low Level Driver (PSDK RTOS Automotive), not higher level ones (MessageQ) which are available in PSDK RTOS.

I work through the documentation of pdk/packages/ti/drv/ipc/ipc.h. I assume, to establish a IPC connection between two cores it is necessary to "announce" and endpoint on Core_0 and "wait" for that endpoint on Core_1.

My question is: Is the order of "announce" (RPMessage_announce) and "wait (RPMessage_getRemoteEndPt) important ?

My observation in my example is like following:
Scenario 1:
Core_0: Announce endpoint.
Core_1: Wait for endpoint.
Result: Core_1 waits infinite long for endpoint. It seems to me, that it not sees the previous announced endpoint from Core_0.

Scenario 2:
Core_1: Wait for endpoint.
Core_0: Announce endpoint
Core_1: Receives endpoint and goes on.

Is this the defined behaviour of IPC LLD, or do I have a bug somehwere in my code? If I need behaviour of cores like scenario 2, I am not sure how implement this for two independed cores.

Best regards,
Thomas

PS: Yes I looked at the example code, but its not clear for me there if particular order is required.

Hi Thomas,

I am looking at this and will reply again this week.

Regards,
Mike

Thomas,

> My question is: Is the order of "announce" (RPMessage_announce) and "wait (RPMessage_getRemoteEndPt) important ?

Yes, announce must occur first. "Announce" is called by the server, and then "wait" is called on the client.

This page may help: https://software-dl.ti.com/jacinto7/esd/processor-sdk-rtos-jacinto7/latest/exports/docs/psdk_rtos_auto/docs/user_guide/developer_notes_ipc.html

Specifically take a look at these sections:

8.5.10.1.2. IPC server (listening to service requests)

8.5.10.1.3. IPC client (sending sevice requests)

Regards,
Mike

Thank you Michael,

I'll checkout the updated documentation of the new PSDK (v7). I'll come back if I have still problems.

Best regards,
Thomas

Michael,

I cannot confirm that announce must occur first. It seems to me, that when "wait" on the client is called first, and "Announce" by the server second, the IPC communication works "more often" than the other way around.

I think my missing piece of the puzzle may be the "pairing of virtio cores". But I don't understand exactly where it happens.

This is my code, which I think "pairing of virtio cores" is done under the hood.

    /* Step1 : Initialize the multiproc */
    retVal = Ipc_init(NULL);
    if(retVal != IPC_SOK)
        System_printf("Error during Ipc_init\n");

    Ipc_mpSetConfig(selfProcId, numProc, pRemoteProcArray);

    System_printf("IPC_echo_test (core : %s) .....\r\n%s\r\n",
                Ipc_mpGetSelfName(), IPC_DRV_VERSION_STR);

    //System_printf("Required Local memory for Virtio_Object = %d\r\n",
    //   numProc * Ipc_getVqObjMemoryRequiredPerCore());


    /* Step2 : Initialize Virtio */
    vqParam.vqObjBaseAddr = (void*)pSysVqBuf;
    vqParam.vqBufSize     = numProc * Ipc_getVqObjMemoryRequiredPerCore();
    vqParam.vringBaseAddr = (void*)VRING_BASE_ADDRESS;

    //Previously, it was defined for all core combinations (~ 15² = 255)
    //0x1C0 0000 / (256 x 512 + 256) = 255
    //Now we need only one core combination [MCU1_0 <-> MCU2_0], which requires 0x40200 size
    vqParam.vringBufSize  = IPC_VRING_BUFFER_SIZE;
    //vqParam.vringBufSize  = 0x40200;
    vqParam.timeoutCnt    = 100;  /* Wait for counts */
    Ipc_initVirtIO(&vqParam);

    /* Step 2.5 : Wait for the other cores*/
    for(int i = 0; i < gNumRemoteProc; i++)
    {
        while(!Ipc_isRemoteReady(remoteProc[i]))
            Task_sleep(1);
    }

    System_printf("Remote is ready!\n");
    System_flush();

    /* Step 3: Initialize RPMessage */
    RPMessage_Params cntrlParam;

    //System_printf("Required Local memory for RPMessage Object = %d\n",
    //   RPMessage_getObjMemRequired());

    /* Initialize the param */
    RPMessageParams_init(&cntrlParam);

    /* Set memory for HeapMemory for control task */
    cntrlParam.buf         = pCntrlBuf;
    cntrlParam.bufSize     = rpmsgDataSize;
    cntrlParam.stackBuffer = &pTaskBuf[index++ * IPC_TASK_STACKSIZE];
    cntrlParam.stackSize   = IPC_TASK_STACKSIZE;
    RPMessage_init(&cntrlParam);

Could you explain to me, where the pairing happens and my synchronization is wrong?

Best regards,
Thomas

Hi Thomas,

Thank you for your patience.  Pairing is handled by calling the Ipc_initVirtIO() function.

Just noticed a subtle change you need to make in your code above.  We need to wait for the cores to become ready before calling Ipc_initVirtIO (i.e., your code marked as step 2.5 should be moved before step 2).

Check out the note regarding Ipc_initVirtIO() here:

https://software-dl.ti.com/jacinto7/esd/processor-sdk-rtos-jacinto7/latest/exports/docs/psdk_rtos_auto/docs/user_guide/developer_notes_ipc.html#writing-ipc-applications-on-ti-rtos-pdk-ipc-driver

Regards,
Mike

Hello Mike,

thank you for your support.

I changed the code to your suggestion and wait for the other core before calling Ipc_initVirtIO. It did not help.
I debugged into Ipc_isRemoteReady() and noticed that it always returns true, when the given core is NOT running linux.

I use MCU1_0 with SysBios whereas my "clientapp" runs, and MCU2_0 with SysBios where my "serverapp" runs. So I dont have/use any Linux here.
I'm sorry if I forgot to mention that in the beginning.

I use CCS to initialize the cores, loading the apps and start MCU1_0 and MCU2_0.

This is the program flow in a condensed form (I removed Ipc_isRemoteReady since I do not use linux):

Clientapp:
c1. Ipc_initSciClient()
c2. Board_init()
c3. Ipc_init()
c4. Ipc_initVirtIO()
c5. RPMessage_init()
c6. RPMessage_create()
c7. RPMessage_getRemoteEndPt()
c8. RPMessage_send()

Serverapp:

s1. Ipc_initSciClient()
s2. Board_init()
s3. Ipc_init()
s4. Ipc_initVirtIO()
s5. RPMessage_init()
s6. RPMessage_create()
s7. RPMessage_announce()

When step c7 executes before s7, RPMessage_getRemoteEndPt() returns successfully when Serverapp executes s7.
When step s7 executes before c7, RPMessage_getRemoteEndPt() does *not* return.

According to your documentation, s7 should happen before c7. But my experience shows that this order does not work (for me). And there are other users which reported the same issue:

Can you help me finding the issue? If i need to synchronize MCU1_0 and MCU2_0 for my ipc application, how should I do that?

Best regards,
Thomas

Update:
I think I get now the point of how to use the IPC LLD. Mike, you said "We need to wait for the cores to become ready before calling Ipc_initVirtIO".
Could you please describe what you mean exactly? When is a core "ready"? I think I have to synchronize the cores earlier in my program flow.

Best Regards,

Hi Thomas,

The vring setup is not symmetric between  a Linux-to-RTOS vs a RTOS-to-RTOS case. Linux is responsible for allocating the buffers and initializing the vrings, so an  IPC_isRemoteReady() is needed before IPC_initVirtIO() on a remote processor when the other processor is running Linux. The IPC_isRemoteReady() is a no-op if the other processor is running RTOS, and each RTOS core sets up the vrings using a global static address (VRING_BASE_ADDRESS).

Ideally, the order should not matter. The endpoint announcement and lookup on different processors can occur in any order - we cannot control it. I suspect it is a bug, and somehow the task is not waking up once the announcement message has come in. 

regards

Suman

Hi Suman,

thank you very much for your explanation. I understand that a kind of synchronisation is required when Linux is involved as participant, and that IPC_isRemoteReady() is no-op if the other processor is NOT running linux.

However, for the RTOS-to-RTOS case:

Can you confirm/reproduce the potential bug on your side?
How can we progress further from here?
Is there any workaround?

Regards,
Thomas

Hi,

I narrowed down the circumstances when RPMessage_getRemoteEndPt does not return:

Program flow still the same as I previously described:

Clientapp:
c1. Ipc_initSciClient()
c2. Board_init()
c3. Ipc_init()
c4. Ipc_initVirtIO()
c5. RPMessage_init()
c6. RPMessage_create()
c7. RPMessage_getRemoteEndPt()
c8. RPMessage_send()

Serverapp:

s1. Ipc_initSciClient()
s2. Board_init()
s3. Ipc_init()
s4. Ipc_initVirtIO()
s5. RPMessage_init()
s6. RPMessage_create()
s7. RPMessage_announce()

When step s5 is not executed before c7, RPMessage_getRemoteEndPt() does *not* return.

So it seems to me that RPMessage_init() must be called from that core, which gets pointed by RPMessage_getRemoteEndPt().

@Suman, do you think this is still a bug?

Regards,
Thomas

Hi Thomas,

Both cores must call RPMessage_init(), this does the initialization of the RPMessage module.

In the IPC examples such as ipc_echo_test, an endpoint is created on both cores, announced from both cores, and a message is sent from both cores. This provides some synchronization indirectly. Can you try doing the same, and see if it improves the behavior? The sequence would be like:

Clientapp:
c1. Ipc_initSciClient()
c2. Board_init()
c3. Ipc_init()
c4. Ipc_initVirtIO()
c5. RPMessage_init()
c6. Ept1 = RPMessage_create()
c7. RPMessage_announce(Ept1)

c8. RPMessage_getRemoteEndPt(Ept2)
c9. RPMessage_send(Ept2)

Serverapp:

s1. Ipc_initSciClient()
s2. Board_init()
s3. Ipc_init()
s4. Ipc_initVirtIO()
s5. RPMessage_init()
s6. Ept2 = RPMessage_create()
s7. RPMessage_announce(Ept2)

s8. RPMessage_getRemoteEndPt(Ept1)

s9. RPMessage_send(Ept1)

Basically even the clientapp is doing an announce, and the serverapp is waiting for it and then sending a "hello"-type response message. This ensures that both cores are up and initialized before starting the regular client/server communication.

Can you try with this kind of change and let us know the behavior?

Thanks,

Angela

Hello Angela,

sorry for the delayed reply. I tried your suggested change, but it did not help in my case.

Derived from my observations, I would say that the key is following: The remote (server) core, must have already called RPMessage_init(), before the (client) core executes RPMessage_getRemoteEndPt(remote). So your suggested indirectly synchronization is to late at this point.

I think, the TI examples work "better" because there is some kind of increased delay, between RPMessage_init() and the correnspond RPMessage_getRemoteEndPt()  core.

As I skimmed through the ti examples, I've noticed that RPMessage_init() is called from the main function, whereas RPMessage_getRemoteEndPt() is called from Task-functions, which are created after RPMessage_init(). Maybe that kind of delay is already enough time.

How to progress further?

Best regards,

Thomas

Hi Thomas,

With your original client and server apps (without my suggested changes), if you load and run the client app first, and let it run until RPMessage_getRemoteEndPt(), then load and run the server app, it doesn't work for you? In that case, it should work, because the client would be completely ready to receive the announce message from the server, and the announce would cause RPMessage_getRemoteEndPt() to return. Can you confirm the behavior you are observing in this case?

Thanks,

Angela

Hi Angela,

to your question: I refer following to the original client server apps (program flow): (I understand "until" as non inclusive)

Clientapp:
c1. Ipc_initSciClient()
c2. Board_init()
c3. Ipc_init()
c4. Ipc_initVirtIO()
c5. RPMessage_init()
c6. RPMessage_create()
c7. RPMessage_getRemoteEndPt()
c8. RPMessage_send()

Serverapp:
s1. Ipc_initSciClient()
s2. Board_init()
s3. Ipc_init()
s4. Ipc_initVirtIO()
s5. RPMessage_init()
s6. RPMessage_create()
s7. RPMessage_announce()

1. I start and run Clientapp (executing c1 -  c6). And stop there.
2. I start and run Serverapp (executing s1 - s7)
3. I execute Clientapp c7. The call RPMessage_getRemoteEndPt() returns, it works!

Okay, now same example, but with "until" as inclusive:

1. I start and run Clientapp (executing c1 -  c7).
2. I start and run Serverapp (executing s1 - s7)
3. Clientapp will hang endless at c7. The call will never return. It does NOT work!

Does this answer your question?
Can you confirm this behaviour on your side?
Is this a bug in the IPC LLD?
If no, how can i synchronize all MCUs? I don't like to introduce some random wait time between the server announce and the client getRemoteEndPt, and hoping that it will work for all cases, as workaround.

Best regards,
Thomas

Hi Thomas,

I am not able to reproduce the behavior you are observing with the test in your previous post. In the test you did, both scenarios should have worked. I tried to reproduce it by using the ex01_bios_2core_echo_test included with IPC LLD and doing some small modification to be like your scenario. With this, if I run the client app completely first (until it is waiting in RPMessage_getRemoteEndPt), and then run the server app completely, it works.

The only difference in behavior that I could see between the 2 scenarios you mentioned in your previous reply, is that in the first one, the client app task that calls RPMessage_getRemoteEndPt is not being made to wait for the announcement because the announcement has already happened. I am suspecting for some reason in your case, the task is not waking up.

Also, you mentioned earlier:

Thomas Dorsch said:

As I skimmed through the ti examples, I've noticed that RPMessage_init() is called from the main function, whereas RPMessage_getRemoteEndPt() is called from Task-functions, which are created after RPMessage_init(). Maybe that kind of delay is already enough time.

To note, RPMessage_init() in the IPC examples is also called from a task and not in main. In your case are you also creating a task for your application?

Are you able to try the IPC LLD examples and confirm that they work for you?

Thanks,

Angela

Hello Angela,

thank you very much for trying to reproduce on your side. As you cannot reproduce it, I suspect a problem with my CCS projects / code.

After I figured out, that ex01_bios_2core_echo_test only works on MCU2_0 and C66xx_0, I tested both scenarios:
1. First started MCU2_0 and then C6xx_0 => worked!
2. First started C6xx_0 and then MCU2_0 => worked!

That's good! Seems no issues with IPC LLD, "only" something wrong with my code. Currently I build debug version of ex01_bios2core_echo and try to find the difference between this and my example.

>> To note, RPMessage_init() in the IPC examples is also called from a task and not in main. In your case are you also creating a task for your application?
Yes, you are right. RPMessage_init() is also called from a task in TI examples.
And yes, I also create a task for my application.

c1. Ipc_initSciClient() [Main Function]
c2. Board_init() [Main Function]
c3. Ipc_init() [Task1]
c4. Ipc_initVirtIO() [Task1]
c5. RPMessage_init() [Task1]
c6. RPMessage_create() [Task1]
c7. RPMessage_getRemoteEndPt() [Task1]
c8. RPMessage_send() [Task1]

Basically the same for server application.

Angela, you think that the task which waits for the RPMessage_getRemoteEndPt does not wake up when the announcement comes in. Do you have some ideas why this can happen? Maybe some directions where I can go to find the root issue?

Thanks and best regards,
Thomas

Hi Angela,

I think I figured it out now how to use the IPC LLD software. I document my outcome here, it may help others.

First of all, I'll have to correct my post from Aug 20, 2020 5:36 AM :
Both example do work! If the client runs until RPMessage_getRemoteEndPt(), then server start and run, the client function RPMessage_getRemoteEndPt() returns succesfully.

Back to my original question: Correct order of "RPMessage_Announce" and "RPMessage_getRemoteEndPt", and how to synchronize two RTOS cores.

According to TI documentation, the correct order should be first to "Announce" from server, and then "getRemoteEndPt" by the client.
From my observation: This is only correct, if the client has already called RPMessage_init() before the server calls "Announce".

If you do it the other way around (first "getRemoteEndPt" from client, and after that "Announce" from server), it works every time. Even if the server app does not run yet, and client app already called "getRemoteEndPt". So I am not sure, if TI documentation at this is maybe wrong.

Anyway, what to do to prevent any hangs when client app executes much later RPMessage_getRemoteEndPt?

Using following pattern on both (server and client) synchronizes and prevent a stucking client (regardless of timing between server and client):

1. RPMessage_create
2. RPMessage_announce
3. RPMessage_getRemoteEndPt
4. RPMessage_send( someDummyData)
5. RPMessage_receive( someDummyData)

After that, both cores "know" eath other and can send/recieve data across.

I suspect following issue: Initializing the IPC LLD software, a "RPMessage_CntrlTask" is created. This task proceeds (among other things) "Announce"-RPMessages (CNTRLMSG_ANNOUNCE) from remote cores. If such message has been received, RPMessage_CntrlTask puts that remote core and it's announced service in a list, which will be searched by function "RPMessage_getRemoteEndPt".

So when core1 sends a "Announce"-RPMessage to core2, and core2 starts RPMessage_CntrlTask later, this "Announce"-RPMessage will not get proceed!
A workaround is, that core1 sends after that a "Dummy"-RPMessage to core2, when RPMessage_CntrlTask on core2 has been started. This triggers a SWI on core2, which will proceed "Announce"-Message and "Dummy"-Message.

I think there may be something missing when startup RPMessage_CntrlTask. Maybe it should check if any "Announce"-RPMessages has been already received. But maybe thats not possible, because it only works with an interrupt from mailbox module.

I just wanted to share my thoughts.

Thank you very much and best regards,
Thomas

Hi Thomas,

I am glad to hear that you have made progress. Also, good to know that the issue we were discussing in Aug 20, 2020 5:36 AM is no longer present.

For your original observation about announce order, that is the situation that my sync sequence from https://e2e.ti.com/support/processors/f/791/p/917749/3440000#3440000 was meant to address. It looks like the new sequence you mentioned is doing something similar.

Currently, this kind of sync mechanism is needed in your case, because as you observed, it is possible for the announce message from the sending core to come before the receiving core has completely initialized. And cores may come up in any order. In that case, the message will indeed wait in the IPC shared memory until the next mailbox interrupt comes in after it has initialized, and at that time, it will process all the pending messages in the shared memory (including the announce message). We are planning to address this in our next release.

Thanks,

Angela