AM5726: Prevent DSP CMA pools from being used by other linux processes

Hello,

Our expert handling this topic is out of the office currently. Hence, expect a delay in response. Thanks.

just wondering if there's any updates on the case.

thanks

Hello Daksh,

Due to bandwidth, I have not had a chance to revisit this.

Thanks for the patience.

-Josue

Hello Daksh,

The only way to prevent any other Linux processes from accessing the DSP memory allocations is to use the no-map property as you have tried.
When you did this, was this used in both the u-boot DTS and kernel DTS? 
Was the memory designated as a CARVEOUT on the resource table?

-Josue

I had another chance to look at the DTS that was shared with us:

If this is the way you are setting up the no-map property, then it is being used on the wrong node.

This property should be put in place of the reusable property on the node above.

Best,

Josue

we  have tried the no-map option in the node dsp1_cma_pool and dsp2_cma_pool

the DSPs fail to bind, and the following error is thrown:

[    3.514120] omap-rproc 40800000.dsp: device does not have specific CMA pool
[    3.552827] omap-rproc 41000000.dsp: device does not have specific CMA pool

the no-map in the CMEM pools is working as expected. we have defined  4x 2k buffers in the OCMC RAM 3 as shared buffers between c66x and A15.

Resource table remains the same correct?

Yes. the resource table is the same.

Daksh,

I will have to try this this week hopefully. It should not come up as a CMA pool, it should actually come up as a DMA pool once no-map is enabled.

Does the kernel log report a DMA pool being created?
example 

[    0.000000] Reserved memory: created DMA memory pool at 0x0000000095800000, size 56 MiB
[    0.000000] OF: reserved mem: initialized node ipu1-memory@95800000, compatible id shared-dma-pool

-Josue

here's the startup kernel messages i see related to those pools:

[    0.000000] Reserved memory: created DMA memory pool at 0x0000000040500000, size 0 MiB
[    0.000000] OF: reserved mem: initialized node cmemDSP1config@40500000, compatible id shared-dma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x0000000040502000, size 0 MiB
[    0.000000] OF: reserved mem: initialized node cmemDSP1status@40502000, compatible id shared-dma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x0000000040510000, size 0 MiB
[    0.000000] OF: reserved mem: initialized node cmemDSP2config@40510000, compatible id shared-dma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x0000000040512000, size 0 MiB
[    0.000000] OF: reserved mem: initialized node cmemDSP2status@40512000, compatible id shared-dma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x0000000095800000, size 80 MiB
[    0.000000] OF: reserved mem: initialized node dsp1_cma@95800000, compatible id shared-dma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x000000009a800000, size 80 MiB
[    0.000000] OF: reserved mem: initialized node dsp2_cma@9a800000, compatible id shared-dma-pool
[    0.000000] cma: dma_contiguous_reserve(limit ffffffff)
[    0.000000] cma: dma_contiguous_reserve: reserving 24 MiB for global area
[    0.000000] cma: cma_declare_contiguous(size 0x0000000001800000, base 0x0000000000000000, limit 0x00000000ffffffff alignment 0x0000000000000000)
[    0.000000] cma: Reserved 24 MiB at 0x00000000fd400000

after these i get the no specific CMA pool error.

thanks,

Daksh

Thank you for sharing Daksh.

Hi Josue, pinging this thread to see if you have further updates here? Thank you

Hello Abi, Daksh,

I have been trying to build the basic IPC examples for 6.03 but ran into some environment issues. Will continue to try to do this and will report back once I succeed

-Josue

Hi Josue, would you be able to let us know when you expect to have an update? Appreciate the help, thank you

Hello Abi, Daksh,

I am currently low on bandwidth. Will attempt to finish this this week.

Thanks for pinging!

-Josue

Hi Josue, 

Just checking in again here to see if you have an update? Thank you

Hello Daksh,

Other than appending the "no-map", can you confirm that you made changes to your resource table accordingly?

-Josue

Hi Josue,

please let us know what we need to update there.

we were under the impression that the resource table stays the same

Josue Zamitiz-Ayala said:

Resource table remains the same correct?

thanks,

Daksh

Daksh,

There are a couple of things that I believe need to be addressed:

1. From the last time had a look at your resource table, another change that could be needed is to make sure your make sure your memory allocation is compliant with DMA pools and the size is in powers of 2.See this note. For example, the DMA pool would allocate 64 MB instead of 63 in the case of 
   
   #define DSP_MEM_HEAP0_SIZE      (SZ_1M * 63)
   You could also try to split this up; seems like 64 MB would be easier
2. TYPE_CARVEOUTt is correct RSC
   
   
3. .From 

   Daksh Jaitly said:

   the DSPs fail to bind, and the following error is thrown:

   Fullscreen

   I think this error is from the memory-region  property where the DSP memory is being linked to the DSP node.

   As an experiment, can you remove this property from the DSP node and check to see if you are able to bind?

   -Josue

Josue,

the DSPs are not able to bind when i remove memory-region property from dsp nodes. (no specified CMA pool)

making it 64MB doesnt make it any different. if ethernet traffic is high, CMA pools deplete and the DSPs dont bind.

once again i would like to point out the original problem - other device/drivers are grabbing CMA pools reserved for the DSP.

when the CMA is available, the remoteproc is able to allocate the CMA pools with ease.

Daksh,

For this thread, we are only interested in enabling your DSP regions as "no-map" regions and therefore make them unavailable from Linux. Why other drivers are taking up the memory that has been reserved, that we cannot make any comments since we were unable to replicate the issue on our own devices as stated in a previous thread.

This sounds like the exact issue that you had here. Is there a chance that you did not have this set up correctly the first time or you see the same print in both cases?

Daksh Jaitly said:

making it 64MB doesnt make it any different.

This is a requirement of making it into a DMA pool , which is what you are doing effectively when using the "no-map" property.

Linked in previous response.

Daksh Jaitly said:

when the CMA is available, the remoteproc is able to allocate the CMA pools with ease.

Is this including the no-map property?

-Josue

with the no-map turned on, the DSPs are not able to bind.

no matter what we specify the HEAP0 size (63MB or 64MB)

Hello Daksh,

I will take a look tomorrow again.

-Josue

Hello Daksh, 

There is an example of the no-map property being used in our 6.03 SDK:

See the device tree for late attach: 

• https://git.ti.com/cgit/ti-linux-kernel/ti-linux-kernel/tree/arch/arm/boot/dts/dra7-ipu-common-early-boot.dtsi?h=ti-linux-4.19.y
  • the reasoning for the the other properties are not related to no-map property and are explained here

I do not see any reason why your device is not binding as needed. I will have to do some more investigation on my device at the office. 

I did not get a chance to get to this today but from a high level it seems like all the right nobs are aligned. Will continue next week.

Best 

Josue

Hello Daksh,

I ran a test today and I made this change to our default am572x-idk.dtb:

diff --git a/arch/arm/boot/dts/am572x-idk.dts b/arch/arm/boot/dts/am572x-idk.dts
index 31ed8070a6..3cf6a2686f 100644
--- a/arch/arm/boot/dts/am572x-idk.dts
+++ b/arch/arm/boot/dts/am572x-idk.dts
@@ -38,4 +38,9 @@
        reset-delay-us = <2>;   /* PHY datasheet states 1uS min */
 };
 
+&dsp2_memory_region {
+       /delete-property/ reusable;
+       no-map;
+};
+

And this is what I see in the boot:

[    0.000000] Reserved memory: created CMA memory pool at 0x0000000095800000, size 56 MiB
[    0.000000] OF: reserved mem: initialized node ipu2-memory@95800000, compatible id shared-dma-pool
[    0.000000] Reserved memory: created CMA memory pool at 0x0000000099000000, size 64 MiB
[    0.000000] OF: reserved mem: initialized node dsp1-memory@99000000, compatible id shared-dma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x000000009d000000, size 32 MiB
[    0.000000] OF: reserved mem: initialized node ipu1-memory@9d000000, compatible id shared-dma-pool
[    0.000000] Reserved memory: created DMA memory pool at 0x000000009f000000, size 8 MiB
[    0.000000] OF: reserved mem: initialized node dsp2-memory@9f000000, compatible id shared-dma-pool
[    0.000000] cma: Reserved 24 MiB at 0x00000000fe400000

root@am57xx-evm:~# dmesg | grep "remoteproc"
[    6.618037] remoteproc remoteproc0: 58820000.ipu is available
[    6.744791] remoteproc remoteproc1: 55020000.ipu is available
[    6.914822] remoteproc remoteproc2: 40800000.dsp is available
[    6.997098] remoteproc remoteproc3: 41000000.dsp is available
[    8.118993] remoteproc remoteproc1: powering up 55020000.ipu
[    8.124768] remoteproc remoteproc1: Booting fw image dra7-ipu2-fw.xem4, size 3747220
[    8.194454] remoteproc remoteproc1: registered virtio0 (type 7)
[    8.200402] remoteproc remoteproc1: remote processor 55020000.ipu is now up
[    8.214503] remoteproc remoteproc0: powering up 58820000.ipu
[    8.220194] remoteproc remoteproc0: Booting fw image dra7-ipu1-fw.xem4, size 6887428
[    8.459811] remoteproc remoteproc0: registered virtio1 (type 7)
[    8.473880] remoteproc remoteproc0: remote processor 58820000.ipu is now up
[    9.396076] remoteproc remoteproc3: powering up 41000000.dsp
[    9.401914] remoteproc remoteproc3: Booting fw image dra7-dsp2-fw.xe66, size 4744488
[    9.441030] remoteproc remoteproc3: registered virtio2 (type 7)
[    9.456518] remoteproc remoteproc3: remote processor 41000000.dsp is now up
[   10.935110] remoteproc remoteproc2: powering up 40800000.dsp
[   10.940968] remoteproc remoteproc2: Booting fw image dra7-dsp1-fw.xe66, size 20482288
[   11.131537] remoteproc remoteproc2: registered virtio3 (type 7)
[   11.151808] remoteproc remoteproc2: remote processor 40800000.dsp is now up
[   12.867870] remoteproc remoteproc4: 4b234000.pru is available
[   12.911946] remoteproc remoteproc5: 4b238000.pru is available
[   12.950053] remoteproc remoteproc6: 4b2b4000.pru is available
[   12.982487] remoteproc remoteproc7: 4b2b8000.pru is available
[   14.875257] remoteproc remoteproc7: powering up 4b2b8000.pru
[   14.903611] remoteproc remoteproc7: Booting fw image ti-pruss/am57xx-pru1-prueth-fw.elf, size 7256
[   14.948660] remoteproc remoteproc7: remote processor 4b2b8000.pru is now up
[   15.014614] remoteproc remoteproc6: powering up 4b2b4000.pru
[   15.028583] remoteproc remoteproc6: Booting fw image ti-pruss/am57xx-pru0-prueth-fw.elf, size 7228
[   15.086864] remoteproc remoteproc6: remote processor 4b2b4000.pru is now up

When unbinding and binding:

root@am57xx-evm:~# cd /sys/bus/platform/drivers/omap-rproc/
root@am57xx-evm:/sys/bus/platform/drivers/omap-rproc# echo 41000000.dsp > unbind
[  524.011998] omap-iommu 41501000.mmu: 41501000.mmu: version 3.0
[  524.017902] omap-iommu 41502000.mmu: 41502000.mmu: version 3.0
[  524.029526] remoteproc remoteproc3: failed to unmap 1048576/0
[  524.037464] remoteproc remoteproc3: stopped remote processor 41000000.dsp
[  524.045430] remoteproc remoteproc3: releasing 41000000.dsp
root@am57xx-evm:/sys/bus/platform/drivers/omap-rproc# echo 41000000.dsp > bind  
[  539.781911] omap-rproc 41000000.dsp: ignoring dependency for device, assuming no driver
[  539.790160] omap-rproc 41000000.dsp: ignoring dependency for device, assuming no driver
[  539.799932] omap-rproc 41000000.dsp: assigned reserved memory node dsp2-memory@9f000000
[  539.809117] remoteproc remoteproc3: 41000000.dsp is available
[  539.821878] remoteproc remoteproc3: powering up 41000000.dsp
[  539.827648] remoteproc remoteproc3: Booting fw image dra7-dsp2-fw.xe66, size 4744488
root@am57xx-evm:/sys/bus/platform/drivers/omap-rproc# [  539.837308] omap-iommu 41501000.mmu: 41501000.mmu: version 3.0
[  539.846114] omap-iommu 41502000.mmu: 41502000.mmu: version 3.0
[  539.860186] virtio_rpmsg_bus virtio2: rpmsg host is online
[  539.865823] remoteproc remoteproc3: registered virtio2 (type 7)
[  539.872923] remoteproc remoteproc3: remote processor 41000000.dsp is now up
[  539.880452] virtio_rpmsg_bus virtio2: creating channel rpmsg-proto addr 0x3d

Resource table:

root@am57xx-evm:/sys/kernel/debug/remoteproc/remoteproc3# cat resource_table
Entry 0 is of type vdev
  ID 7
  Notify ID 0
  Device features 0x1
  Guest features 0x1
  Config length 0x0
  Status 0x7
  Number of vrings 2
  Reserved (should be zero) [0][0]

  Vring 0
    Device Address 0x9f000000
    Alignment 4096
    Number of buffers 256
    Notify ID 0
    Physical Address 0x0

  Vring 1
    Device Address 0x9f004000
    Alignment 4096
    Number of buffers 256
    Notify ID 1
    Physical Address 0x0

Entry 1 is of type carveout
  Device Address 0x9f000000
  Physical Address 0x9f100000
  Length 0x100000 Bytes
  Flags 0x0
  Reserved (should be zero) [0]
  Name DSP_MEM_IPC_DATA

Entry 2 is of type carveout
  Device Address 0x95000000
  Physical Address 0x9f200000
  Length 0x100000 Bytes
  Flags 0x0
  Reserved (should be zero) [0]
  Name DSP_MEM_TEXT

Entry 3 is of type carveout
  Device Address 0x95100000
  Physical Address 0x9f300000
  Length 0x100000 Bytes
  Flags 0x0
  Reserved (should be zero) [0]
  Name DSP_MEM_DATA

Entry 4 is of type carveout
  Device Address 0x95200000
  Physical Address 0x9f400000
  Length 0x200000 Bytes
  Flags 0x0
  Reserved (should be zero) [0]
  Name DSP_MEM_HEAP0

Entry 5 is of type carveout
  Device Address 0x95300000
  Physical Address 0x9f600000
  Length 0x100000 Bytes
  Flags 0x0
  Reserved (should be zero) [0]
  Name DSP_MEM_HEAP1

Entry 6 is version 0 of type trace
  Device Address 0x9f000000
  Length 0x8000 Bytes
  Reserved (should be zero) [0]
  Name trace:dsp

Entry 7 is of type devmem
  Device Address 0xa0000000
  Physical Address 0x9f000000
  Length 0x100000 Bytes
  Flags 0x0
  Reserved (should be zero) [0]
  Name DSP_MEM_IPC_VRING

Entry 8 is of type devmem
  Device Address 0x60000000
  Physical Address 0x60000000
  Length 0x10000000 Bytes
  Flags 0x0
  Reserved (should be zero) [0]
  Name DSP_TILER_MODE_0_1

Entry 9 is of type devmem
  Device Address 0x70000000
  Physical Address 0x70000000
  Length 0x8000000 Bytes
  Flags 0x0
  Reserved (should be zero) [0]
  Name DSP_TILER_MODE_2

Entry 10 is of type devmem
  Device Address 0x78000000
  Physical Address 0x78000000
  Length 0x8000000 Bytes
  Flags 0x0
  Reserved (should be zero) [0]
  Name DSP_TILER_MODE_3

Entry 11 is of type devmem
  Device Address 0x4a000000
  Physical Address 0x4a000000
  Length 0x1000000 Bytes
  Flags 0x0
  Reserved (should be zero) [0]
  Name DSP_PERIPHERAL_L4CFG

Entry 12 is of type devmem
  Device Address 0x48000000
  Physical Address 0x48000000
  Length 0x200000 Bytes
  Flags 0x0
  Reserved (should be zero) [0]
  Name DSP_PERIPHERAL_L4PER1

Entry 13 is of type devmem
  Device Address 0x48400000
  Physical Address 0x48400000
  Length 0x400000 Bytes
  Flags 0x0
  Reserved (should be zero) [0]
  Name DSP_PERIPHERAL_L4PER2

Entry 14 is of type devmem
  Device Address 0x48800000
  Physical Address 0x48800000
  Length 0x800000 Bytes
  Flags 0x0
  Reserved (should be zero) [0]
  Name DSP_PERIPHERAL_L4PER3

Entry 15 is of type devmem
  Device Address 0x54000000
  Physical Address 0x54000000
  Length 0x1000000 Bytes
  Flags 0x0
  Reserved (should be zero) [0]
  Name DSP_PERIPHERAL_L4EMU

Entry 16 is of type devmem
  Device Address 0x4e000000
  Physical Address 0x4e000000
  Length 0x100000 Bytes
  Flags 0x0
  Reserved (should be zero) [0]
  Name DSP_PERIPHERAL_DMM

This is working on my end with the IPC examples.

Hopefully this helps!

-Josue