AM6442: Creating IPC between Linux kernel driver and R5F

Hello Mack,

There is definitely an interface at the kernel level to send and receive data between Linux A53 and R5F. In fact, you can use the low level mailbox driver to do this kernel driver communication instead of RPMsg. Mailbox should be much lower latency than RPMsg.

I did a deep dive into how the mailbox is used to send a "graceful shutdown" message from Linux to the R5F in this thread. Hopefully this can give you an idea of how to get started with mailbox communication in kernel drivers: https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1420796/sk-am62b-p1-how-to-enable-graceful-shutdown-of-remote-cores-with-multithreading-in-freertos-threads

Please let me know if you need more guidance here. If you would like a code review, feel free to pass your code on to Kevin so that he can share it with me.

Regards,

Nick

Hi Mack,

Do you have any further support needs after following Nick's suggestions?

Thanks,

Kevin

Hi Kevin

According to Nick's suggestion, we are currently preparing for testing.

Thanks，

Mack

Hi Nick，

1. Can the mailbox handle sending and receiving data volumes of 8K?
2. In which file is the mailbox driver located in the Linux kernel? Because we have another driver that needs to directly access this mailbox,we want to modify this mailbox driver, Is this feasible? If so, could you provide a general idea of how to approach this?

Regards,

Mack

Hello Mack,

The mailbox itself is just an interrupt paired with a 32 bit register. So you would use a shared memory region to actually pass the data, and then use the mailbox to quickly send a notification when the memory region is ready to be read.

To make sure I understand: you want multiple different Linux drivers accessing the exact same mailbox queue? Why not just have a dedicated mailbox queue for each Linux driver?

Regards,

Nick

Hi Nick,

It might be my misunderstanding, as I thought the mailbox was a standalone driver. Based on what you said, if I want to set up a dedicated mailbox queue in my own Linux driver to communicate with R5F, how should I do it? Are there any examples?

Regards，

Mack

Hello Mack,

FYI: discussion is split into 2 threads

I have split the discussion into two threads:
The previous thread is about benchmarking IPC and the zerocopy example
This new thread is about creating IPC between a Linux driver and a remote core

More details about the mailbox peripheral

For more information, refer to the AM64x Technical Reference Manual (TRM), section Interprocessor Communication (IPC) > Mailbox.

The mailbox peripheral has 8 clusters in it. Each cluster has 16 queues. Each queue can hold up to 4 messages. Each message is 32 bits.

So when we say that we are "allocating a mailbox" for IPC between two cores, we are actually allocating a specific mailbox QUEUE.

That also has a couple important details for your design:

1) You need to allocate a different mailbox queue for each direction.

So let's say you want to communicate between 1 Linux driver, and 4 R5F cores. You would typically set up 8 mailbox queues for communication:
- 4 mailbox queues for sending messages from the Linux driver, to each of the R5F cores
- 4 mailbox queues for each R5F core to send a message to the Linux driver

2) You can only have up to 4 unread mailbox messages at a time. So if Linux tries to send a 5th mailbox message to one of the R5F cores, but the R5F core has not read any of the previous 4 mailbox messages, then Linux will wait to send the 5th mailbox message. We do NOT support overwriting messages that are already in the mailbox queue.

More details about the mailbox driver 

The mailbox driver is a low level kernel driver. That means that this driver is designed to be used by other Linux drivers, instead of being used by userspace.

The Linux remoteproc and RPMsg drivers are 2 drivers that use the mailbox driver for low level communication, so we can use them as an example for how you would want to write your custom kernel driver.

Let's look at AM64x SDK 10.0, arch/arm64/boot/dts/ti/k3-am642-evm.dts, for an example of how to configure the mailbox settings, and pass the mailbox information to the remoteproc driver:

// the mailboxes are configured here

&mailbox0_cluster2 {
    status = "okay";

    mbox_main_r5fss0_core0: mbox-main-r5fss0-core0 {
        ti,mbox-rx = <0 0 2>;
        ti,mbox-tx = <1 0 2>;
    };

    mbox_main_r5fss0_core1: mbox-main-r5fss0-core1 {
        ti,mbox-rx = <2 0 2>;
        ti,mbox-tx = <3 0 2>;
    };
};

&mailbox0_cluster4 {
    status = "okay";

    mbox_main_r5fss1_core0: mbox-main-r5fss1-core0 {
        ti,mbox-rx = <0 0 2>;
        ti,mbox-tx = <1 0 2>;
    };

    mbox_main_r5fss1_core1: mbox-main-r5fss1-core1 {
        ti,mbox-rx = <2 0 2>;
        ti,mbox-tx = <3 0 2>;
    };
};

&mailbox0_cluster6 {
    status = "okay";

    mbox_m4_0: mbox-m4-0 {
        ti,mbox-rx = <0 0 2>;
        ti,mbox-tx = <1 0 2>;
    };
};

// we pass the associated mailbox to the Linux remoteproc driver here

&main_r5fss0_core0 {
    mboxes = <&mailbox0_cluster2 &mbox_main_r5fss0_core0>;
    memory-region = <&main_r5fss0_core0_dma_memory_region>,
            <&main_r5fss0_core0_memory_region>;
};

&main_r5fss0_core1 {
    mboxes = <&mailbox0_cluster2 &mbox_main_r5fss0_core1>;
    memory-region = <&main_r5fss0_core1_dma_memory_region>,
            <&main_r5fss0_core1_memory_region>;
};

&main_r5fss1_core0 {
    mboxes = <&mailbox0_cluster4 &mbox_main_r5fss1_core0>;
    memory-region = <&main_r5fss1_core0_dma_memory_region>,
            <&main_r5fss1_core0_memory_region>;
};

&main_r5fss1_core1 {
    mboxes = <&mailbox0_cluster4 &mbox_main_r5fss1_core1>;
    memory-region = <&main_r5fss1_core1_dma_memory_region>,
            <&main_r5fss1_core1_memory_region>;
};

&mcu_m4fss {
    mboxes = <&mailbox0_cluster6 &mbox_m4_0>;
    memory-region = <&mcu_m4fss_dma_memory_region>,
            <&mcu_m4fss_memory_region>;
    status = "okay";
};

Regards,

Nick

Hi Nick

1、In the ti-linux-kernel-6.6.32 kernel, I found the mailbox driver located at drivers/mailbox/omap-mailbox.c. However, I only see the sending of mailbox data in it. Why is there no receiving of mailbox data?

2、Another question: Can a mailbox cluster send 2K of data at once, or does it need to be sent in batches?

Regards，

Mack

Hello Mack,

You can see that the devicetree node for the mailbox defines both a "tx" mailbox, and an 'rx' mailbox. So I would expect that the Linux mailbox driver is able to both "transmit", and "receive".

And if we check in the bindings documentation, we can see that there is RX and TX functionality:

ti-linux-kernel-6.6.32+git-ti/arch/arm64/boot/dts/ti$ grep -r --include=k3-am64* mailbox0_cluster4
k3-am64-main.dtsi:      mailbox0_cluster4: mailbox@29040000 {
k3-am642-evm.dts:&mailbox0_cluster4 {
k3-am642-evm.dts:       mboxes = <&mailbox0_cluster4 &mbox_main_r5fss1_core0>;
k3-am642-evm.dts:       mboxes = <&mailbox0_cluster4 &mbox_main_r5fss1_core1>;
grep: k3-am642-evm.dtb: binary file matches
k3-am64-phycore-som.dtsi:&mailbox0_cluster4 {
k3-am64-phycore-som.dtsi:       mboxes = <&mailbox0_cluster4 &mbox_main_r5fss1_core0>;
k3-am64-phycore-som.dtsi:       mboxes = <&mailbox0_cluster4 &mbox_main_r5fss1_core1>;
k3-am642-sk.dts:&mailbox0_cluster4 {
k3-am642-sk.dts:        mboxes = <&mailbox0_cluster4 &mbox_main_r5fss1_core0>;
k3-am642-sk.dts:        mboxes = <&mailbox0_cluster4 &mbox_main_r5fss1_core1>;
k3-am642-tqma64xxl.dtsi:&mailbox0_cluster4 {
k3-am642-tqma64xxl.dtsi:        mboxes = <&mailbox0_cluster4 &mbox_main_r5fss1_core0>;
k3-am642-tqma64xxl.dtsi:        mboxes = <&mailbox0_cluster4 &mbox_main_r5fss1_core1>;

// let's get the compatibility string so we can find the driver
ti-linux-kernel-6.6.32+git-ti/arch/arm64/boot/dts/ti$ vi k3-am64-main.dtsi

// now look at the bindings documentation
// this talks about devicetree usage
ti-linux-kernel-6.6.32+git-ti/arch/arm64/boot/dts/ti$ cd ../../../../../Documentation/devicetree/bindings/
ti-linux-kernel-6.6.32+git-ti/Documentation/devicetree/bindings$ grep -r 'ti,am64-mailbox'
mailbox/ti,omap-mailbox.yaml:      - ti,am64-mailbox  # for K3 AM64x SoCs
mailbox/ti,omap-mailbox.yaml:            - ti,am64-mailbox
ti-linux-kernel-6.6.32+git-ti/Documentation/devicetree/bindings$ vi mailbox/ti,omap-mailbox.yaml

And if you look in the driver, you will see ISRs that are set up for receiving data:

ti-linux-kernel-6.6.32+git-ti/Documentation/devicetree/bindings$ cd ../../../drivers/
ti-linux-kernel-6.6.32+git-ti/drivers$ grep -r 'ti,am64-mailbox'
grep: mailbox/omap-mailbox.o: binary file matches
mailbox/omap-mailbox.c:         .compatible     = "ti,am64-mailbox",
ti-linux-kernel-6.6.32+git-ti/drivers$ vi mailbox/omap-mailbox.c

I am not sure exactly how the driver is used, but I suspect that you do not call a specific function to .receive_data. Instead, you get an interrupt when a mailbox is received, and that ISR code runs.

The Remoteproc driver should provide more details on sending and receiving mailboxes.

Regards,

Nick