AM5718: MessageQ latency

Hello Sergey,

I am experiencing some network /hardware issues. It will probably be a few days before I can check on

* the feature request Rex submitted for low latency high throughput IPC

* if any request got submitted to update the MessageQ example to show real notification turnaround time (not average)

Alternatives to the IPC framework 

I suspect this does not apply to you, but I'll mention it just in case: We do have a PRU <-> RT Linux ARM userspace example that uses polling instead of interrupts to ensure timing requirements are met. In the example, the userspace thread is given elevated permissions, and it polls the first register of a shared memory buffer every 250 us. After the PRU core finishes loading data into that buffer, it writes a value to the first register in the buffer so the ARM knows to start processing the data. Some customers may be able to use that example as inspiration to implement their own custom interprocessor communication. As of SDK 6.0 that example can be found in the RT Linux AM335x SDK (not AM57xx SDK) under example-applications/pru-adc-x.x

Regards,

Nick

Thanks!

The polling is currently the only really working way to communicate fast enough for realtime audio/video streaming between ARM/DSP, but it is very limited way of doing things on such a powerful hardware.

Hello Sergey,

Update: I still do not have access to my hardware. I hope to get access later this week, but it could take a while. I am sorry for the delay.

* Feature request for high throughput with low latency: This is not implemented as of SDK 6.1. I am poking around some more. As always, feedback is welcome.

* The request to update the MessageQ example to show real notification turnaround time (not average) was not submitted. I just submitted it.

* I do not know if any effort has gone into putting an upper limit on the round trip latency you observed in the MessageQ example. I am still looking into this.

From your side, were you ever able to isolate where in the MessageQ process the increased latency was coming from?

Regards,

Nick

Hello,

Thanks for update.

Unfortunately I don't have enough expertise and resources to debug the IPC framework.

I tried to dive into the mailbox infrastructure as all I really need is just a fast notification between ARM and DSP cores, but unfortunately I'm not familiar with Linux/TI RTOS internals enough.

Hello Sergey,

Got it. Are you trying to get the interrupts and data to a Linux kernel driver, or to Linux userspace?

So far I have found a bare metal mailbox example at pdk_am57xx_1_0_15\packages\ti\csl\example\mailbox\mailbox_sender_receiver_app , but I have not had time to see if there are any Linux side resources for mailboxes.

Regards,

Nick

Hello Nick,

As far as I understand it does not differ much. I mean our application is working in userspace but it's not hard to make a simple kernel driver that would wake up the userspace app and pass the notification.

Our main goal is to offload some math to the DSP part as currently we only use Linux on ARM and DSP is just sleeping.

The problem is that it's a realtime stream of data, and we process it periodically, roughly - every 1.5ms. So if we are going to send some data to DSP (through shared ram) and notify it to start processing, and then wait for a completion notification, the notification turnaround time becomes very important. 

Our tests show that sending two notifications (ARM->DSP->ARM)  takes up to 1.2ms , and that is on an idle CPUs.

Thus DSP is virtually unusable for us.

Hello Sergey,

I apologize for the delay. I will take a look at mailboxing options soon. We do have a mailbox driver (drivers/mailbox/omap-mailbox.c, Documentation/devicetree/bindings/mailbox/omap-mailbox.txt) , and there should be a way for userspace to interact with the mailbox. I have not had time to look at it though, and I am not an RT Linux expert so I'm not sure if we could place a hard upper bound on the latency for the Linux driver or not.

Regards,

Nick

Hello Nick,

Thank you very much for your efforts.

I believe as soon as there would be a way to use a kinda "stripped-down IPC" solution that would just initiate an IRQ on a destination core without any  abstraction software layers (schedulers\queues etc) it would work just great.

On top of that - the IPC framework is kinda monstrous by itself, so it would be really great to have just an example of the mailbox messaging between DSP with RTOS/Cortex a15 with Linux(and ideally cortex M too), probably in a form of a benchmark.

As far as I understand - everything else is a matter of IRQ priority balancing (if the messaging code is fully decoupled from any kinds of high level software abstraction layers)

Hello Sergey,

I apologize for the delay. I started looking at mailbox communication for Linux ARM <-> DSP on AM57xx, but got pulled into a project to get mailboxing working on AM65xx between PRU and R5F. The AM65x example is working now (tell your friends!), so I'm shifting back to AM57xx Linux mailboxing.

At a broader level, I wanted to let you know that your requests for lower latency IPC examples, more transparency on IPC benchmarking, etc is being heard. It is taking a bit of time, but I am pushing to improve documentation, provide examples, etc. I am the point of contact for the IPC overhaul effort, so if you have any feedback that you want heard (or you want to make sure we didn't miss in this thread or any of your previous threads), feel free to let me know. Thanks for your patience over the past year!

Regards,

Nick

Hello Nick,

Thank you so much for an update!

In my personal opinion, some simple mailbox messaging (just a simple "kick" from one core to another) plus some shared memory management (and we already has TI CMEM for this as far as I understand ) would be a great and simple combo for any IPC duties.

Hello Sergey,

Update: I have written a kernel driver, device tree changes, and DSP sample code. I need to focus on some other end-of-year activities for a bit, so I am going to post what I have currently written. Feel free to play around with it and see if you can get it working.

This is NOT a finished product - I have not even tried compiling anything yet. So there may be compile errors, or runtime errors, or it might not even do what I think it should. You can find notes for how I expect the code to work in README_mailbox_example.

This example is written on SDK 6.1. I'm providing the patch to apply changes to the Linux SDK. RTOS SDK doesn't have git built in, so I'm providing the entire folder of ipc_3_50_04_07/examples/DRA7XX_linux_elf/ex02_messageq/dsp1

Regards,

Nick

mailbox-example-initial-commit.zip

Hello Nick,

thank you!

We are currently on SDK5 so we'll start porting our codebase to the 6.1 and then we will try your code. Thank you!

Hello Sergey,

I will start looking at the mailbox driver example again sometime over the next couple of weeks. If you have any feedback, let me know and I will try to incorporate it into the next draft of the example.

Regards,

Nick

Hello Nick, thanks for update!

Sorry I have no any new info yet

Hello Sergey,

Quick update: Have not had time to make much progress on this yet. A bare-bones mailbox example between Linux and DSP is still on my near-term TODO list though.

Regards,

Nick

Hello Nick,

thank you very much for update!