AM625: SIGBUS when memset() shared memory through /dev/mem

Hi Antonio,

I cannot explain the SIGBUS error because I never directly operate DDR from Linux user space, which is not a normal way to do in Linux.

My first question is that since you directly mmap 64KB at 0x91000000, did you reserve this DDR region in kernel devicetree?

Hi Bin Liu, 

Thanks for jumping in so quickly.

In the snippets above the address is something I came up with just now. In the real application it is indeed reserved through the devicetree.

Our application uses two mapped memory regions: PRU0's Data RAM and a 1MiB block from the DDR. 

As far as I could gather, this SIGBUS while trying to fill those regions with memset happens regardless of this being DDR or peripheral memory.

Also,  just to be clear, filling those memory regions using pointer accesses does work, regardless of the data type (1,2,4 or 8 bytes wide) as long as the access is aligned.

The same code built to run on the beaglebone black (with adjustment to the addresses) works just fine.

Best,

António

Hi,

I guess you're getting a strongly ordered (nGnRnE on arm64) mapping for that memory, so it's not "normal" memory, but something like a device register (from the processor's pov). Depending on the instructions used by memset (which could also depend on the size and the alignment) the accesses generated aren't suitable for that kind of mapping.

I guess it's simply wrong to use something like memset or memcpy with that kind of "special" memory.

The question is what kind of memory mapping you want for this region, and how to achieve that.

Regards,

Dominic

Thanks, for pointing out the kind of memory mapping that we may he dealing with, Dominic.

From your words and a very quick bit of reading on the interwebs it seems plausible to have a scenario where this kind of memory mapping, combined with the wrong kind of instructions, by memset, could run into that SIGBUS.

I am no expert on this topic, though, and would like to understand if it'd be possible, at all, to request a memory mapping which wouldn't have this kind of restriction.

Our application essentially treats these memory regions as buffers to communicate with a PRU. The accesses are tipically done member-by-member, with members being tipically integers up to 64 bits wide. Would be possible to pass appropriate flags to mmap()  - or the open("/dev/mem") - so that a more relaxed kind of mapping would be established?

Regards,

António

Hi Antonio,

I am not an expert on this topic either, but I am wondering if you can access this reserved memory in kernel space, or if this bus error still happens. I think directly accessing memory in user space via /dev/mem is somehow limited.

When communicating with the PRU you probably need uncached, or else you would need manual cache maintenance. 

I guess this is where the mapping is decided:

https://elixir.bootlin.com/linux/v6.11.6/source/arch/arm64/mm/mmu.c#L98

If the memory is not mapped (not sure what exactly that entails) you get nGnRnE. If it is, but you pass in O_SYNC you get uncached, else you get normal memory. At least for PRU data memory you won't get the necessary mapping. For the DDR buffer I suppose the problem is that it's reserved and hence unmapped.

You could look into DMA-Buf for a cleaner approach. That should give you a cached mapping and manual coherency. Or you write your own kernel driver, similar to char/mem.c, that maps with the desired attributes (normal uncached should work).

All of this is more effort, but you are "mis-using" interfaces for things they're not meant for. Unfortunately I often find that there's stuff missing for the "proper' solution. 

Maybe Andrew Davis or Nick Saulnier have better suggestions. I've seen both work on related stuff. 

Regards, Dominic

Hey Dominic, thanks for looping me in.

Hello António,

Questions

1) What version of Linux kernel are you using?

2) Will you grant userspace root level access in your final product? My assumption is that most applications do NOT want to give userspace that much control.

Some background 

We do provide a hacky example on AM335x of using mmap to access a DDR shared memory region and pass data back and forth between Linux and PRU:
https://git.ti.com/cgit/apps/tida01555/tree/ARM_User_Space_App/arm_user_space_app.c
https://www.ti.com/tool/TIDA-01555

With that said, I would only expect mmap to work in contexts where userspace has elevated permissions. Kernel developers will tell you that they usually do NOT want to give userspace access to directly expose memory.

There are several different ways to expose memory up to userspace, but as far as I can tell, the broader Linux community has not really aligned on a "correct" way to do it yet.

As far as TI examples, we do allocate a shared memory region in the zerocopy example (currently not written for PRU, but the same concepts apply).

For Linux kernel 5.10 & 6.1, we patched the dma-buf-phys driver to enable shared memory (i.e., this driver code worked on those kernel versions, but it was NOT upstreamed). Refer to branch ti-linux-6.1:
https://git.ti.com/cgit/rpmsg/rpmsg_char_zerocopy/?h=ti-linux-6.1

For Linux kernel 6.6, we moved to using the remoteproc CDev driver to enable shared memory. Note that this requires patches to the Remoteproc CDev driver that were also not upstreamed the last time I checked. For more information, refer to the Linux README.md on the main branch:
https://git.ti.com/cgit/rpmsg/rpmsg_char_zerocopy/tree/linux/README.md

Regards,

Nick

Hi Nick,

1. We are running ti-linux v6.1.105 (from the latest meta-ti-bsp kirkstone). We use the same kernel for both AM62 and AM33.

2. Yes, the application runs with privileges. This code goes some way back and we are avoiding to rework it to rely on something like rpms or similar.

Some additional info:

* If we mmap() with MAP_ANON instead of through "/dev/mem" all memory accesses work fine including memset(). Sadly this kind of mapping clears the memory when the mapping is established, which gets in the way of the operation of our application.

* would there be a reason for the mapping to work well with MAP_ANON but SIGBUS the process when going through /dev/mem? Please note that is happens for both DDR and peripheral regions

* I am aware of the security implications of this kind of setup, by exposing uninitialized memory to applications. Our system runs only our internal software so the risks are under control, even though I agree a solution that didn't rely on running with privileges would be better.

It's a bit late now. I'll have a look at the links tomorrow. Thanks for all the assistance. In addition to making this memory mapping work we are also looking for potential workarounds that work with a simple anonymous mapping.

Best regards,

António

Hi all, thanks for your inputs.

As an update, for what it's worth, I made some progress on this topic (other stuff has been getting in the way and I didn't have a lot of time to look further into this).

At this point, I got rid of the SIGBUS by modifying the devicetree reserved memory as follows:

	reserved-memory {
		#address-cells = <2>;
		#size-cells = <2>;
		ranges;

		input-pru@0x9c600000 {
			/* no-map; <---- commented this property!! */
			reg = <0x00000000 0x9c600000 0x000000 0x00100000>;
		};
    };

By commenting the `no-map` property it appears that the crash is gone. Memory access seems to work well from some tests that I did between the host CPU and the PRU running some bespoke firmware.

Why this devicetree change made any impact is something that eludes me, but I'll keep bringing up the PRU+Host setup and will keep you posted on how it goes.

In the meantime, if there is any kind of feedback or explanation as to why that change makes some difference, I'll be happy to learn about it.

Best regards,