This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

[K2G ProcSDK Linux] Limited MessageQ size ?

Hi,

I'm using ti-processor-sdk-linux-k2g-evm-03.01.00.06 and now I'm trying to evaluate ipc_3_43_01_03/examples/66AK2G_linux_elf/ex02_messageq. Basically, it is running correctly. Here is the log on linux console:

root@k2g-evm:~# mpmcl reset dsp0
reset succeeded
root@k2g-evm:~# mpmcl load dsp0 core0.out
load succeeded
root@k2g-evm:~# run dsp0
-sh: run: command not found
root@k2g-evm:~# mpmcl run dsp0
[  155.684470]  remoteproc0: powering up 10800000.dsp0
[  155.689653]  remoteproc0: Booting unspecified pre-loaded fw image
[  155.696101]  remoteproc0: remote processor 10800000.dsp0 is now up
[  155.709164] virtio_rpmsg_bus virtio4: rpmsg host is online
[  155.714694] virtio_rpmsg_bus virtio4: creating channel rpmsg-proto addr 0              x3d
[  155.732898]  remoteproc0: registered virtio4 (type 7)
run succeeded
root@k2g-evm:~# ./app_host CORE0
--> main:
--> Main_main:
--> App_create:
App_create: Host is ready
<-- App_create:
--> App_exec:
App_exec: sending message 1
App_exec: sending message 2
App_exec: sending message 3
App_exec: message received, sending message 4
App_exec: message received, sending message 5
App_exec: message received, sending message 6
App_exec: message received, sending message 7
App_exec: message received, sending message 8
App_exec: message received, sending message 9
App_exec: message received, sending message 10
App_exec: message received, sending message 11
App_exec: message received, sending message 12
App_exec: message received, sending message 13
App_exec: message received, sending message 14
App_exec: message received, sending message 15
App_exec: message received
App_exec: message received
App_exec: message received
<-- App_exec: 0
--> App_delete:
<-- App_delete:
<-- Main_main:
<-- main:

And still, it works correctly if I increase the MessageQ buffer size like this :

==== ipc_3_43_01_03\examples\66AK2G_linux_elf\ex02_messageq\shared\AppCommon.h ====

typedef struct {`
     MessageQ_MsgHeader  reserved;`
     UInt32              cmd;`
 ^   UInt32^ ^   ^   ^   data[100];`//Newly added 
} App_Msg;`

but if I increase the messageQ buffer size, say data[200], it fails as below:

root@k2g-evm:~# mpmcl reset dsp0
[ 1128.021295]  remoteproc0: stopped remote processor 10800000.dsp0
reset succeeded
root@k2g-evm:~# mpmcl load dsp0 core0.out
load succeeded
root@k2g-evm:~# mpmcl run dsp0
[ 1137.042910]  remoteproc0: powering up 10800000.dsp0
[ 1137.047789]  remoteproc0: Booting unspecified pre-loaded fw image
[ 1137.061430]  remoteproc0: remote processor 10800000.dsp0 is now up
[ 1137.071946] virtio_rpmsg_bus virtio4: rpmsg host is online
[ 1137.077527] virtio_rpmsg_bus virtio4: creating channel rpmsg-proto addr 0x3d
[ 1137.086955]  remoteproc0: registered virtio4 (type 7)
run succeeded
root@k2g-evm:~# ./app_host CORE0
--> main:
--> Main_main:
--> App_create:
App_create: Host is ready
<-- App_create:
--> App_exec:
TransportRpmsg_put: send failed: 90 (Message too long)

It seems the messageQ buffer size is limited by IPC implementation.  How can I change the size limit for MessageQ ?

My goal is to know upper limit for MessageQ size and check the latency for larger messageQ packets from A15 to C66 and vice versa. But for me, it takes more time than I expected without no data[] in the messageQ buffer (default operation). From my evaluation, it takes1288709 nsec in App_exec() function. App_exec() sends totally 15 messageQs and gets their responses from C66. So it might be able to conclude that it takes 43 usec per way (A15 to C66, or, C66 to A15). If you have any suggestions to achieve smaller latency for messaging between A15 and C66, please let me know.

Best Regards,
Naoki

  • Hi Naoki,

    I've forwarded this to the TI processor SDK experts. Their feedback should be posted here.

    BR
    Tsvetolin Shulev
  • Thanks for following up this thread, so do you have any update on this ?

    Best Regards,
    Naoki
  • And I tried the same experimental test with Linux RT (ti-processor-sdk-linux-rt-k2g-evm-03.01.00.06).
    Now the messageQ container looks like this. That is the default state of the example code (ex02_example in IPC package):

    typedef struct {
        MessageQ_MsgHeader  reserved;
        UInt32              cmd;
    } App_Msg;

    Also, the host side code looks like this. The only difference from the original code can be referred as BENCH definition. 

    ====== \ipc_3_43_01_03\examples\66AK2G_linux_elf\ex02_messageq\host\App.c ====== 

    /*
     * Copyright (c) 2013-2014, Texas Instruments Incorporated
     * All rights reserved.
     *
     * Redistribution and use in source and binary forms, with or without
     * modification, are permitted provided that the following conditions
     * are met:
     *
     * *  Redistributions of source code must retain the above copyright
     *    notice, this list of conditions and the following disclaimer.
     *
     * *  Redistributions in binary form must reproduce the above copyright
     *    notice, this list of conditions and the following disclaimer in the
     *    documentation and/or other materials provided with the distribution.
     *
     * *  Neither the name of Texas Instruments Incorporated nor the names of
     *    its contributors may be used to endorse or promote products derived
     *    from this software without specific prior written permission.
     *
     * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
     * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
     * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
     * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
     * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
     * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
     * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS;
     * OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
     * WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR
     * OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE,
     * EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
     */
    
    /*
     *  ======== App.c ========
     *
     */
    
    /* host header files */
    #include <stdio.h>
    #include <unistd.h>
    
    /* package header files */
    #include <ti/ipc/Std.h>
    #include <ti/ipc/MessageQ.h>
    
    /* local header files */
    #include "../shared/AppCommon.h"
    #include "App.h"
    
    #define BENCH
    
    
    #ifdef BENCH
    #include <time.h>
    #endif
    
    /* module structure */
    typedef struct {
        MessageQ_Handle         hostQue;    // created locally
        MessageQ_QueueId        slaveQue;   // opened remotely
        UInt16                  heapId;     // MessageQ heapId
        UInt32                  msgSize;
    } App_Module;
    
    /* private data */
    static App_Module Module;
    
    
    /*
     *  ======== App_create ========
     */
    
    Int App_create(UInt16 remoteProcId)
    {
        Int                 status = 0;
        MessageQ_Params     msgqParams;
        char                msgqName[32];
    
        printf("--> App_create:\n");
    
        /* setting default values */
        Module.hostQue = NULL;
        Module.slaveQue = MessageQ_INVALIDMESSAGEQ;
        Module.heapId = App_MsgHeapId;
        Module.msgSize = sizeof(App_Msg);
    
        /* create local message queue (inbound messages) */
        MessageQ_Params_init(&msgqParams);
    
        Module.hostQue = MessageQ_create(App_HostMsgQueName, &msgqParams);
    
        if (Module.hostQue == NULL) {
            printf("App_create: Failed creating MessageQ\n");
            status = -1;
            goto leave;
        }
    
        /* open the remote message queue */
        sprintf(msgqName, App_SlaveMsgQueName, MultiProc_getName(remoteProcId));
    
        do {
            status = MessageQ_open(msgqName, &Module.slaveQue);
            sleep(1);
        } while (status == MessageQ_E_NOTFOUND);
    
        if (status < 0) {
            printf("App_create: Failed opening MessageQ\n");
            goto leave;
        }
    
        printf("App_create: Host is ready\n");
    
    leave:
        printf("<-- App_create:\n");
        return(status);
    }
    
    
    /*
     *  ======== App_delete ========
     */
    Int App_delete(Void)
    {
        Int         status;
    
        printf("--> App_delete:\n");
    
        /* close remote resources */
        status = MessageQ_close(&Module.slaveQue);
    
        if (status < 0) {
            goto leave;
        }
    
        /* delete the host message queue */
        status = MessageQ_delete(&Module.hostQue);
    
        if (status < 0) {
            goto leave;
        }
    
    leave:
        printf("<-- App_delete:\n");
        return(status);
    }
    
    
    /*
     *  ======== App_exec ========
     */
    Int App_exec(Void)
    {
        Int         status;
        Int         i;
        App_Msg *   msg;
    #ifdef BENCH
    	struct timespec time0, time1;
    	long overhead_nsec, diff_nsec;
    #endif
        printf("--> App_exec:\n");
    
    
    #ifdef BENCH
    
    	clock_gettime(CLOCK_MONOTONIC, &time0);
    	clock_gettime(CLOCK_MONOTONIC, &time1);
    	overhead_nsec = time1.tv_nsec - time0.tv_nsec;
    
    	clock_gettime(CLOCK_MONOTONIC, &time0);
    #endif
        /* fill process pipeline */
        for (i = 1; i <= 3; i++) {
    #ifndef BENCH
            printf("App_exec: sending message %d\n", i);
    #endif
            /* allocate message */
            msg = (App_Msg *)MessageQ_alloc(Module.heapId, Module.msgSize);
    
            if (msg == NULL) {
                status = -1;
                goto leave;
            }
    
            /* set the return address in the message header */
            MessageQ_setReplyQueue(Module.hostQue, (MessageQ_Msg)msg);
    
            /* fill in message payload */
            msg->cmd = App_CMD_NOP;
    
            /* send message */
            MessageQ_put(Module.slaveQue, (MessageQ_Msg)msg);
        }
    
        /* process steady state (keep pipeline full) */
        for (i = 4; i <= 15; i++) {
    
            /* wait for return message */
            status = MessageQ_get(Module.hostQue, (MessageQ_Msg *)&msg,
                MessageQ_FOREVER);
    
            if (status < 0) {
                goto leave;
            }
    
            /* extract message payload */
    
            /* free the message */
            MessageQ_free((MessageQ_Msg)msg);
    #ifndef BENCH
            printf("App_exec: message received, sending message %d\n", i);
    #endif
            /* allocate message */
            msg = (App_Msg *)MessageQ_alloc(Module.heapId, Module.msgSize);
    
            if (msg == NULL) {
                status = -1;
                goto leave;
            }
    
            /* set the return address in the message header */
            MessageQ_setReplyQueue(Module.hostQue, (MessageQ_Msg)msg);
    
            /* fill in message payload */
            if (i == 15) {
                /* Last message will tell the slave to shutdown */
                msg->cmd = App_CMD_SHUTDOWN;
            }
            else {
                msg->cmd = App_CMD_NOP;
            }
    
            /* send message */
            MessageQ_put(Module.slaveQue, (MessageQ_Msg)msg);
        }
    
        /* drain process pipeline */
        for (i = 1; i <= 3; i++) {
    #ifndef BENCH
    		printf("App_exec: message received\n");
    #endif
            /* wait for return message */
            status = MessageQ_get(Module.hostQue, (MessageQ_Msg *)&msg,
                MessageQ_FOREVER);
    
            if (status < 0) {
                goto leave;
            }
    
            /* extract message payload */
    
            /* free the message */
            MessageQ_free((MessageQ_Msg)msg);
        }
    
    #ifdef BENCH
    	clock_gettime(CLOCK_MONOTONIC, &time1);
    
    	diff_nsec = time1.tv_nsec - time0.tv_nsec - overhead_nsec;
    
        printf("nsec: %d\n", (int)diff_nsec);
    #endif
    
    leave:
        printf("<-- App_exec: %d\n", status);
        return(status);
    }

    The code itself is running correctly with LinuxRT, but the problem here is the latency for messageQ.
    As you see in the above code, I'm evaluating the time taken by message life cycle.
    When Linux is running, the result is 1288709 as I mentioned before, but when LinuxRT is running, the result is 1488251. It was worse score against the Linux case.
    I was expecting I would get the better score with LinuxRT... What do you think of this ? Do you think this is expected behavior ?
    I'm also waiting for your answer for the limitation of messageQ container size.

    Best Regards,
    Naoki





  • Hello again. I'm waiting for your reply here.

    Thank you in advance.
    Naoki

  • Any response ? Do you have any progress on this ? 

  • Hi, Naoki,

    The size should be bound by rpmsg and it is defined as 512 bytes. I am not sure why you have problem running at 200 bytes of payload. We are able to run it at 456 bytes on K2H.

    root@k2hk-evm:~# MessageQBench 100 456 1

    Using numLoops: 100; payloadSize: 456, procId : 1

    Entered MessageQApp_execute

    Local MessageQId: 0x80

    Remote queueId  [0x10080]

    Exchanging 100 messages with remote processor CORE0...

    CORE0: Avg round trip time: 86 usecs

    Leaving MessageQApp_execute

     

    The IPC isn't designed to send large package, and because kernel is involved in transmission between ARM and DSP with multiple copies during the process, your measurement is expected and matches our observation.

     

    What is your goal of performance improvement, from 43us to 30us or from 43us to 5us? Both are improvements but mean differently. What is the application for the desired performance?

     

    Rex

     

     

  • the payload is int32 so that's 800 bytes

  • Hello Rex,

    Thanks for your confirmation. I understand 512bytes requirement in rpmsg. Well, It is Audio application. I went to the customer to have technical meeting today. I have some feedback from them. it seems 40~50usec latency would be big for them. I suggested them to use CMEM for further improvement for small latency. They understand it, but as you know CMEM is just designed to allocate shared buffer between CA15 and C66, so some handshake protocol and low-level implementations would be required by themselves. Maybe that was negative point for them. Anyway, they will keep to work on TI device, but if you have any solution for latency with Linux, it is good to know.
    Well it is just aside, I tried same IPC test between CA15(SYS/BIOS) and C66(SYS/BIOS), and it was around 6usec. Much faster than Linux case, but they stick to Linux.

    Best Regards,
    Naoki
  • Dear Naoki

    I just wonder. You said that this is audio application. Does the ARM need access to the input at all? Can the input data gets into the DSP memory directly from the peripheral?

    Just wonder

    Best Regards

    Ran
  • Hello Ran,

    I cant't state in detail here, but they are evaluating K2G/AM57x whether these devices meet their requirements or not. Because they are considering to use them as an audio platform, there must be various use cases. As you know the dead line for real-time is very short in audio app, so I think audio input/output is basically handled by DSP, but some audio data may be passed to ARM(Linux), for example, to do some postprocessing (These data might go to ethernet finally). Anyway, shorter latancy for communicating DSP and ARM would be better for them.

    Best Regards,
    Naoki
  • The only suggesting that I have is the following - if the data buffer that moves from DSP to ARM is always same size (if not ignore what I say), and if it is always read in the same order that it is written ( if not ignore what I say), Then I would suggest to develop special communication code based on CMEM on the ARM side, fixed buffer (and buffer sizes) and use interrupt to tell the other side that a new message is ready, and interrupt to acknowledge.

    Ran

  • Hi Ran,

    Thank you for your suggestion. So my last question : In ARM context (Linux), do you have any light-weight interrupt notification mechanism between ARM and DSP ? (Just like IPC/Notify API in BIOS(CA15)-to-BIOS(DSP) application) . I wonder IPC inter-processor interrupt could be used (IPC_GR0 for DSP/IPC_GR8 for CA15) but these interrupt lines might be used/reserved by some Linux modules (IPC/MessageQ, mpmcl , and etc...). What do you think ?

    Best Regards,
    Naoki
  • Naoki,

    You are correct. I think what Ran meant is to develop a customized IPC.

    Rex