This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

SK-AM62A-LP: Sometimes the rpmsg driver on Linux side can't get rpmsg_driver->callback function be called after Linux rpmsg_send() a msg to the internal MCU.

Part Number: SK-AM62A-LP

Sometimes the rpmsg driver on Linux side can't get rpmsg_driver->callback function be called after Linux rpmsg_send() a msg to the internal MCU. 

Specially when Linux side tried to dump 256 values from the internal MCU.

The error log as below, would you please help to find the possible reason, and how to debug it?

---------------------------------------------------------------

[ 4687.661828] <DRV_ADS6311> [2024-05-18 00:10:17.988.122.430] <N> mcuctrl_dump_reg_segment()-8291 ====================MCU Register Dump from <__sensor_start_stream> Line: 3889=====================

[ 4687.679115] <DRV_ADS6311> [2024-05-18 00:10:18.005.409.120] <N> mcuctrl_dump_reg_segment()-8300 ---------0----1----2----3----4----5----6----7----8----9----a----b----c----d----e----f

[ 4687.797943] <DRV_ADS6311> [2024-05-18 00:10:18.124.235.710] <E> mcuctrl_read_reg_raw()-7736 rpmsg wait for completion time-out

[ 4687.809333] <DRV_ADS6311> [2024-05-18 00:10:18.135.627.315] <E> mcuctrl_read_reg_raw()-7744 rpmsg read reg:0x0000 failed, ret:-110, rpmsg_from_user: 0

[ 4687.822798] <DRV_ADS6311> [2024-05-18 00:10:18.149.091.650] <E> mcuctrl_dump_reg_segment()-8312 read register 0x0 failed ret:-110

[ 4687.822798]

[ 4688.354285] <DRV_ADS6311> [2024-05-18 00:10:18.680.574.820] <N> sensor_s_stream()-3937 --on:0, console_loglevel:7, dbg_ctrl:0x2, wkmode:6(PCM), linkFreq:500000000, exposureTime:1000 us, power_on:1, mcu_chipid:a62a, mcu_fw_type: 0xa, drv_version: 2.0.0_LM20240521a---

[ 4688.477944] <DRV_ADS6311> [2024-05-18 00:10:18.804.236.680] <E> mcuctrl_write_reg_raw()-7957 rpmsg wait for completion time-out for 100 ms.

[ 4688.490475] rpmsg_tx: 00000000: 03 11 00 88 00 00 95                             .......

[ 4688.498562] <DRV_ADS6311> [2024-05-18 00:10:18.824.855.850] <E> mcuctrl_write_reg_raw()-7970 rpmsg Write mcu reg:0x0088 failed, ret:-110, rpmsg_from_user: 0

[ 4688.512547] <DRV_ADS6311> [2024-05-18 00:10:18.838.841.005] <E> set_vcsel_state()-3445 set_vcsel_state(0) error, call from Line:3901.

[ 4688.512547]

[ 4704.421881] <DRV_ADS6311> [2024-05-18 00:10:34.748.171.675] <N> sensor_s_stream()-3937 --on:1, console_loglevel:7, dbg_ctrl:0x2, wkmode:1(PHR_BM), linkFreq:500000000, exposureTime:1000 us, power_on:0, mcu_chipid:a62a, mcu_fw_type: 0xa, drv_version: 2.0.0_LM20240521a---

[ 4704.549942] <DRV_ADS6311> [2024-05-18 00:10:34.876.233.415] <E> mcuctrl_write_reg_raw()-7957 rpmsg wait for completion time-out for 100 ms.

[ 4704.562471] rpmsg_tx: 00000000: 03 12 00 8a 00 01 95                             .......

[ 4704.570558] <DRV_ADS6311> [2024-05-18 00:10:34.896.852.670] <E> mcuctrl_write_reg_raw()-7970 rpmsg Write mcu reg:0x008a failed, ret:-110, rpmsg_from_user: 0

[ 4704.584544] <DRV_ADS6311> [2024-05-18 00:10:34.910.837.710] <E> mcuctrl_set_rx_workmode()-8212 Fail to set rx work mode: 1.

[ 4704.584544]

[ 4704.597140] <DRV_ADS6311> [2024-05-18 00:10:34.923.433.945] <E> __sensor_power_on()-4167 Failed to set rx work mode.

[ 4704.597140]

[ 4704.609127] <DRV_ADS6311> [2024-05-18 00:10:34.935.421.160] <E> sensor_runtime_resume()-4565 Fail to __sensor_power_on(), ret: -110

[ 4704.609127]

[ 4704.622418] <DRV_ADS6311> [2024-05-18 00:10:34.948.712.005] <E> sensor_s_stream()-3977 start stream failed ret:-110

[ 4704.622418]

[ 4704.634324] cdns-csi2rx 30101000.csi-bridge: Failed to start streams 0xf on subdev

[ 4704.651949] cdns-csi2rx 30101000.csi-bridge: Failed to stop stream0

[ 4704.668229] cdns-csi2rx 30101000.csi-bridge: Failed to stop stream1

[ 4704.684555] cdns-csi2rx 30101000.csi-bridge: Failed to stop stream2

[ 4704.700845] cdns-csi2rx 30101000.csi-bridge: Failed to stop stream3

  • Hello David,

    I support generic RPMsg, so I'm going to pass your question on to people who support the actual higher-level code that you are using.

    For the rest of yall, usually when I see RPMsg driver returning error 110, connection timed out, it means that Linux is flooding the remote core with more RPMsg message packets than the remote core can handle. Eventually all the RPMsg VIRTIO buffers fill up, and Linux is no longer able to write to the VIRTIO buffers.

    That issue usually pops up if people try to use the RPMsg protocol as the "data passing" mechanism for large amounts of data. In those usecases, I typically suggest that the user set aside a shared memory region that can hold much more than 496 bytes of information, and then only use RPMsg as the notification mechanism for when the shared memory region is ready to be consumed. That reduces the interrupts getting sent to the remote core, and allows for more efficient code.

    Regards,

    Nick

  • Hi Nick,

        Thanks for your fast reply. I will try your suggestion a little later. But I have 2 questions more:

    1. Sometimes, I also met rpmsg timeout when Linux side tried to write a MCU register by rpmsg. (Not many data need be transfered), for this case, what is possible reason to rpmsg timeout?

    2. Would you please help to provide the document or sample code about how to create and use a shared memory between Linux main CPU and R5 MCU?

    Thanks and Regards,

    David

  • Hello David,

    Just to verify: Are you writing your own IPC here, or are you using driver provided by TI that is handling the IPC? e.g., are you using a TI CSI driver? (I don't know anything about the CSI usecase, so please forgive my ignorance about the usecase and which company is writing which code)

    1.

    Linux does not use RPMsg to write to the MCU registers. Linux RPMsg writes data to a VIRTIO buffer that is in a shared memory region in DDR, and then the remote core is notified to go read that RPMsg buffer.

    So to double-check, are you writing directly to the MCU registers? If so, how are you doing that write, and what terminal output do you see?

    Or are you just seeing timeouts when doing RPMsg at a "slower" pace? If so, what is the rate that RPMsg messages are sent in this "slower" pace?

    2.

    For an example of a shared memory region, please refer to the zerocopy example:
    https://git.ti.com/cgit/rpmsg/rpmsg_char_zerocopy/ 

    Note that there are many possible ways to implement shared memory - this is just one method. Please also note that dma-buf-phys is the Linux driver we use for the example on kernel 6.1, but we will migrate the example to use remoteproc cdev on kernel 6.6.

    Regards,

    Nick

  • Hi Nick,

       I'm sorry I forgot to introduce some background to you in advance. We are porting our dToF sensor (as a v4l2 camera sensor sub device) to TI platform, the internal R5 MCU need to control some hardware component, while my Linux side driver will control other components. My Linux side driver also need communicate with the R5 MCU to get its status or send some commands to it, we defined some business layer registers for this communications, so here "MCU registers" only means our custom registers, not real MCU registers, so here the register actually means a MCU side variable which can be read or writen by Linux side driver by RpMsg driver. I hope you can understand what I'm saying. 

        Yes, sometimes, not so often, I can see rpmsg timeout even if Linux is reading/write one register.  I will add some mutex/lock for Rpmsg in my Linux side driver to see whether it will be improved. It sounds a little related with the  concurrency.

    Thanks and Best Regards

    David

  • For an example of a shared memory region, please refer to the zerocopy example:
    https://git.ti.com/cgit/rpmsg/rpmsg_char_zerocopy/ 

    Hi Nick,

       I noticed in this example, the Linux side code is based on Linux user space, since my Rpmsg communication is based on kernel space, would you please help to provide a example on kernel space?

    Thanks and Best regards,

    David

  • Hello David,

    Your thread is assigned to someone else for now (the electricity and internet is out for my part of the city and probably will be for the next couple days - I had to drive to another town to type this response).

    We don't currently provide an example of shared memory with the Linux kernel driver, though there should be other examples online (e.g., I've seen character drivers used to expose memory up to userspace - no reason they couldn't just use the memory directly from kernel space as well). The generic kernel RPMsg driver example is in the kernel under samples/rpmsg.

    Regards,

    Nick

  • Hi Nick,

        Thanks for your support and update.

    Thanks and Best regards,

    David

  • Hi David,

    Is there anything pending, or can we close this thread?

    Thanks

    Yogesh

  • Ok, I close it now, I will open a new case if I have new question about rpmsg.