This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

am5728: CMEM issues

Part Number: AM5728

I use ti-processor-sdk-linux-am57xx-evm-03.03.00.04 on am5728

And want to allocate buffers for capture video. My dts:

reserved-memory {
		#address-cells = <0x2>;
		#size-cells = <0x2>;
		ranges;

		ipu1_cma@9d000000 {
			compatible = "shared-dma-pool";
			reg = <0x0 0x9d000000 0x0 0x2000000>;
			reusable;
			status = "okay";
			linux,phandle = <0xb5>;
			phandle = <0xb5>;
		};

		ipu2_cma@95800000 {
			compatible = "shared-dma-pool";
			reg = <0x0 0x95800000 0x0 0x3800000>;
			reusable;
			status = "okay";
			linux,phandle = <0xba>;
			phandle = <0xba>;
		};

		dsp1_cma@99000000 {
			compatible = "shared-dma-pool";
			reg = <0x0 0x99000000 0x0 0x4000000>;
			reusable;
			status = "okay";
			linux,phandle = <0xc2>;
			phandle = <0xc2>;
		};

		dsp2_cma@9f000000 {
			compatible = "shared-dma-pool";
			reg = <0x0 0x9f000000 0x0 0x800000>;
			reusable;
			status = "okay";
			linux,phandle = <0x127>;
			phandle = <0x127>;
		};

		cmem_block_mem@a0000000 {
			reg = <0x0 0xa0000000 0x0 0xA000000>;
			no-map;
			status = "okay";
			linux,phandle = <0x131>;
			phandle = <0x131>;
		};
		                
		myblock: cmem_block_mem@aa000000 {
			reg = <0x0 0xaa000000 0x0 0x2000000>;
			no-map;
			status = "okay";
		};

		cmem_block_mem@40500000 {
			reg = <0x0 0x40500000 0x0 0x100000>;
			no-map;
			status = "okay";
			linux,phandle = <0x132>;
			phandle = <0x132>;
		};
	};


	cmem {
		compatible = "ti,cmem";
		#address-cells = <0x1>;
		#size-cells = <0x0>;
		#pool-size-cells = <0x2>;
		status = "okay";

		cmem_block@0 {
			reg = <0x0>;
			memory-region = <0x131>;
			cmem-buf-pools = <0x1 0x0 0xA000000>;
		};

		cmem_block@1 {
			reg = <0x1>;
			memory-region = <&myblock>;
			cmem-buf-pools =<0x6 0x0 0x96000>,<0x6 0x0 0x3f4800>;
		};

		cmem_block@2 {
			reg = <0x2>;
			memory-region = <0x132>;
		};
	};

CMEM loads fine

root@am57xx-evm:~# cat /proc/cmem 

Block 0: Pool 0: 1 bufs size 0xa000000 (0xa000000 requested)

Pool 0 busy bufs:

Pool 0 free bufs:
id 0: phys addr 0xa0000000

Block 1: Pool 0: 6 bufs size 0x96000 (0x96000 requested)

Pool 0 busy bufs:

Pool 0 free bufs:
id 0: phys addr 0xabf6a000
id 1: phys addr 0xabed4000
id 2: phys addr 0xabe3e000
id 3: phys addr 0xabda8000
id 4: phys addr 0xabd12000
id 5: phys addr 0xabc7c000

Block 1: Pool 1: 6 bufs size 0x3f5000 (0x3f4800 requested)

Pool 1 busy bufs:

Pool 1 free bufs:
id 0: phys addr 0xab887000
id 1: phys addr 0xab492000
id 2: phys addr 0xab09d000
id 3: phys addr 0xaaca8000
id 4: phys addr 0xaa8b3000
id 5: phys addr 0xaa4be000

But sample program not works:

    CMEM_AllocParams prms;
    prms.alignment = 0;
    prms.flags = CMEM_NONCACHED;
    prms.type = CMEM_POOL;
    CMEM_init();
    void* ptr = CMEM_alloc(640*480*2,&prms);

    printf("Got buffer %08X\n",ptr);
    CMEM_free(ptr,&prms);
    CMEM_exit();

CMEM Error: registerAlloc: ioctl CMEM_IOCREGUSER failed for phys addr 0xa0000000: -1
TIOCL FATAL: The TI Multicore Tools daemon (/usr/bin/ti-mctd) is not running. Re-run application after starting the daemon. Refer User Guide for details.

When try to run ti-mctd:

CMEM Error: allocHeap: ioctl CMEM_IOCALLOCHEAPCACHED failed: -1
ti-mctd: /home/gtbldadm/processor-sdk-linux-krogoth-build/build-CORTEX_1/arago-tmp-external-linaro-toolchain/work/am57xx_evm-linux-gnueabi/opencl/1.1.12.0-r0.0/git/host/mct-daemon/cmem_allocator.h:94: CmemAllocator::CmemAllocator(): Assertion `msmc_alloc_dsp_addr_ == msmc_addr' failed.

  • The software team have been notified. They will respond here.
  • After some experiments it looks like ti-mctd breaks down after any change of cmem configuration(change partitioning of block0 or add new blocks with pool)
  • Hi, Konstantin,

    In your cmem_alloc() call, you didn't specify wihich block to allocate, it will allocate from first available. That is from cmem_block_mem@a0000000. It is why you see the error showing failed for phys addr 0xa0000000. If you need to allocate from a specific pool, you need to use cmem_alloc2().

    ti-mctd is only used by OpenCL. If your test program only does cmem_alloc(), the ti-mctd is not needed. However, you got a complaint of it not being run. That gives me a hint that opencl is running somewhere. If OpenCL is running, be sure there isn't any conflict in resource at 0xa0000000. That is possibly why the cmem_alloc() failed at the first place.

    Rex

  • In my application I want use OpenCL too and OpenCL initialization are later in sample code.

    Wnen device-tree

    	reserved-memory {
    		#address-cells = <0x2>;
    		#size-cells = <0x2>;
    		ranges;
    
    		ipu1_cma@9d000000 {
    			compatible = "shared-dma-pool";
    			reg = <0x0 0x9d000000 0x0 0x2000000>;
    			reusable;
    			status = "okay";
    			linux,phandle = <0xb8>;
    			phandle = <0xb8>;
    		};
    
    		ipu2_cma@95800000 {
    			compatible = "shared-dma-pool";
    			reg = <0x0 0x95800000 0x0 0x3800000>;
    			reusable;
    			status = "okay";
    			linux,phandle = <0xbd>;
    			phandle = <0xbd>;
    		};
    
    		dsp1_cma@99000000 {
    			compatible = "shared-dma-pool";
    			reg = <0x0 0x99000000 0x0 0x4000000>;
    			reusable;
    			status = "okay";
    			linux,phandle = <0xc5>;
    			phandle = <0xc5>;
    		};
    
    		dsp2_cma@9f000000 {
    			compatible = "shared-dma-pool";
    			reg = <0x0 0x9f000000 0x0 0x800000>;
    			reusable;
    			status = "okay";
    			linux,phandle = <0x12a>;
    			phandle = <0x12a>;
    		};
    
    		cmem_block_mem@a0000000 {
    			reg = <0x0 0xa0000000 0x0 0xc000000>;
    			no-map;
    			status = "okay";
    			linux,phandle = <0x134>;
    			phandle = <0x134>;
    		};
    
    		cmem_block_mem@b0000000 {
    			reg = <0x0 0xb0000000 0x0 0xc000000>;
    			no-map;
    			status = "okay";
    			linux,phandle = <0x135>;
    			phandle = <0x135>;
    		};
    
    		cmem_block_mem2@40500000 {
    			reg = <0x0 0x40500000 0x0 0x100000>;
    			no-map;
    			status = "okay";
    		};
    
    	};
    
    	cmem {
    		compatible = "ti,cmem";
    		#address-cells = <0x1>;
    		#size-cells = <0x0>;
    		#pool-size-cells = <0x2>;
    		status = "okay";
    
    		cmem_block@0 {
    			reg = <0x0>;
    			memory-region = <0x134>;
    			cmem-buf-pools = <0x1 0x0 0xc000000>;
    		};
    		cmem_block@1 {
    			reg = <0x1>;
    			memory-region = <0x135>;
    			cmem-buf-pools = <0x6 0x0 0x96000>,<0x6 0x0 0x3f5000>;
    		};
    	};

    Cmem loads fine, all buffers are free after boot

    root@am57xx-evm:~# cat /proc/cmem 
    
    Block 0: Pool 0: 1 bufs size 0xc000000 (0xc000000 requested)
    
    Pool 0 busy bufs:
    
    Pool 0 free bufs:
    id 0: phys addr 0xa0000000
    
    Block 1: Pool 0: 6 bufs size 0x96000 (0x96000 requested)
    
    Pool 0 busy bufs:
    
    Pool 0 free bufs:
    id 0: phys addr 0xbbf6a000
    id 1: phys addr 0xbbed4000
    id 2: phys addr 0xbbe3e000
    id 3: phys addr 0xbbda8000
    id 4: phys addr 0xbbd12000
    id 5: phys addr 0xbbc7c000
    
    Block 1: Pool 1: 6 bufs size 0x3f5000 (0x3f5000 requested)
    
    Pool 1 busy bufs:
    
    Pool 1 free bufs:
    id 0: phys addr 0xbb887000
    id 1: phys addr 0xbb492000
    id 2: phys addr 0xbb09d000
    id 3: phys addr 0xbaca8000
    id 4: phys addr 0xba8b3000
    id 5: phys addr 0xba4be000

    But ti-mctd cannot start at startup and OpenCL code not works:

    Mar 30 00:57:16 am57xx-evm systemd[1]: Starting TI MultiCore Tools Daemon...
    Mar 30 00:57:17 am57xx-evm ti-mctd[693]: CMEM Error: allocHeap: ioctl CMEM_IOCALLOCHEAPCACHED failed: -1
    Mar 30 00:57:17 am57xx-evm ti-mctd[693]: ti-mctd: /home/gtbldadm/processor-sdk-linux-krogoth-build/build-CORTEX_1/arago-tmp-external-linaro-toolchain/work/am57xx_evm-linux-gnueabi/opencl/1.1.12.0-r0.0/git/host/mct-daemon/cmem_allocat
    or.h:94: CmemAllocator::CmemAllocator(): Assertion `msmc_alloc_dsp_addr_ == msmc_addr' failed.
    

    When try run it manual:

    root@am57xx-evm:~# ti-mctd
    [  341.836543] CMEMK Error: ioctl: failed to allocate heap buffer of size 0xc000000
    CMEM Error: allocHeap: ioctl CMEM_IOCALLOCHEAPCACHED failed: -1
    ti-mctd: /home/gtbldadm/processor-sdk-linux-krogoth-build/build-CORTEX_1/arago-tmp-external-linaro-toolchain/work/am57xx_evm-linux-gnueabi/opencl/1.1.12.0-r0.0/git/host/mct-daemon/cmem_allocator.h:94: CmemAllocator::CmemAllocator(): Assertion `msmc_alloc_dsp_addr_ == msmc_addr' failed.
    Aborted (core dumped)

    What I want to do for allocate buffers with cmem(and get dmabuf for it) and use it for OpenCL?

  • Hi, Konstantn,

    Are you able to run ti opencl example? ti-mctd should have been started when the Kernel boots up. I suspect your issue may have something to do with ti-mctd not running. Ignore those omap_nwmod or omap_mmuoutputs in my opencl run.


    Arago Project http://arago-project.org am57xx-evm ttyS2

    Arago 2016.12 am57xx-evm ttyS2

    am57xx-evm login: root
    root@am57xx-evm:~# ps -ef | grep ti-mctd
    root       608     1  0 00:56 ?        00:00:00 /usr/bin/ti-mctd
    root      1164  1159  0 01:04 ttyS2    00:00:00 grep ti-mctd
    root@am57xx-evm:~# cd /usr/share/ti/examples/opencl/vecadd
    root@am57xx-evm:/usr/share/ti/examples/opencl/vecadd# ./vecadd
    [  479.784259] omap_hwmod: mmu0_dsp2: _wait_target_disable failed
    [  479.790170] omap-iommu 41501000.mmu: 41501000.mmu: version 3.0
    [  479.797004] omap-iommu 41502000.mmu: 41502000.mmu: version 3.0
    [  479.812336] omap_hwmod: mmu0_dsp1: _wait_target_disable failed
    [  479.818243] omap-iommu 40d01000.mmu: 40d01000.mmu: version 3.0
    [  479.824307] omap-iommu 40d02000.mmu: 40d02000.mmu: version 3.0
    DEVICE: TI Multicore C66 DSP

    Offloading vector addition of 8192K elements...

    Kernel Exec : Queue  to Submit: 14 us
    Kernel Exec : Submit to Start : 89 us
    Kernel Exec : Start  to End   : 29124 us

    Success!
    root@am57xx-evm:/usr/share/ti/examples/opencl/vecadd#

  • Try change your CMEM configuration (add new memory region and pool with blocks). OpenCL will broken.

    When CMEM configuration is

        cmem {
            compatible = "ti,cmem";
            #address-cells = <0x1>;
            #size-cells = <0x0>;
            #pool-size-cells = <0x2>;
            status = "okay";

            cmem_block@0 {
                reg = <0x0>;
                memory-region = <0x131>;
                cmem-buf-pools = <0x1 0x0 0xc000000>;
            };

            cmem_block@1 {
                reg = <0x1>;
                memory-region = <0x132>;
            };
        };

    Output:

    [   35.868787] usb 3-1.1.1: new high-speed USB device number 4 using xhci-hcd
    [  OK  ] Started TI MultiCore Tools Daemon.
    [  OK  ] Started Telephony service.
    .......

    root@am57xx-evm:/usr/share/ti/examples/opencl/vecadd# cat /proc/cmem

    Block 0: Pool 0: 1 bufs size 0xc000000 (0xc000000 requested)

    Pool 0 busy bufs:
    id 0: phys addr 0xa0000000 (cached)

    Pool 0 free bufs:
    root@am57xx-evm:/usr/share/ti/examples/opencl/vecadd# ./vecadd
    [ 3852.511708] omap_hwmod: mmu0_dsp2: _wait_target_disable failed
    [ 3852.517626] omap-iommu 41501000.mmu: 41501000.mmu: version 3.0
    [ 3852.525792] omap-iommu 41502000.mmu: 41502000.mmu: version 3.0
    [ 3852.541179] omap_hwmod: mmu0_dsp1: _wait_target_disable failed
    [ 3852.547083] omap-iommu 40d01000.mmu: 40d01000.mmu: version 3.0
    [ 3852.554601] omap-iommu 40d02000.mmu: 40d02000.mmu: version 3.0
    DEVICE: TI Multicore C66 DSP

    Offloading vector addition of 8192K elements...

    Kernel Exec : Queue  to Submit: 15 us
    Kernel Exec : Submit to Start : 49 us
    Kernel Exec : Start  to End   : 29110 us

    Success!
    root@am57xx-evm:/usr/share/ti/examples/opencl/vecadd#

    BUT: when I change CMEM config in device tree to:

    reserved-memory {
    ...

            cmem_block_mem@a0000000 {
                reg = <0x0 0xa0000000 0x0 0xc000000>;
                no-map;
                status = "okay";
                linux,phandle = <0x131>;
                phandle = <0x131>;
            };

            my_cmem: cmem_block_mem@b0000000 {
                reg = <0x0 0xb0000000 0x0 0xc000000>;
                no-map;
                status = "okay";
            };

            cmem_block_mem@40500000 {
                reg = <0x0 0x40500000 0x0 0x100000>;
                no-map;
                status = "okay";
                linux,phandle = <0x132>;
                phandle = <0x132>;
            };
        };


        cmem {
            compatible = "ti,cmem";
            #address-cells = <0x1>;
            #size-cells = <0x0>;
            #pool-size-cells = <0x2>;
            status = "okay";

            cmem_block@0 {
                reg = <0x0>;
                memory-region = <0x131>;
                cmem-buf-pools = <0x1 0x0 0xc000000>;
            };

            cmem_block@1 {
                reg = <0x1>;
                memory-region = <&my_cmem>;
                cmem-buf-pools = <0x1 0x0 0xc000000>;
            };

            cmem_block@2 {
                reg = <0x2>;
                memory-region = <0x132>;
            };

        };

    recompile dtb and reboot, I see error in console:

    ....

    [  OK  ] Started Permit User Sessions.
    [FAILED] Failed to start TI MultiCore Tools Daemon.
    See 'systemctl status ti-mct-daemon.service' for details.
    [  OK  ] Started Save/Restore Sound Card State.
    ....

    root@am57xx-evm:~# systemctl status ti-mct-daemon.service

    [[0;1;31mb[[0m ti-mct-daemon.service - TI MultiCore Tools Daemon
       Loaded: loaded (/lib/systemd/system/ti-mct-daemon.service; enabled; vendor preset: enabled)
       Active: [[0;1;31mfailed[[0m (Result: core-dump) since Thu 2017-03-30 00:57:17 UTC; 1min 57s ago
      Process: 661 ExecStart=/usr/bin/ti-mctd [[0;1;31m(code=dumped, signal=ABRT)[[0m

    Mar 30 00:57:15 am57xx-evm systemd[1]: Starting TI MultiCore Tools Daemon...
    Mar 30 00:57:16 am57xx-evm ti-mctd[661]: CMEM Error: allocHeap: ioctl CMEM_IOCALLOCHEAPCACHED failed: -1
    Mar 30 00:57:16 am57xx-evm ti-mctd[661]: ti-mctd: /home/gtbldadm/processor-sdk-linux-krogoth-build/build-CORTEX_1/arago-tmp-external-linaro-toolchain/work/am57xx_evm-linux-gnueabi/opencl/1.1.12.0-r0.0/git/host/mct-daemon/cmem_allocat
    or.h:94: CmemAllocator::CmemAllocator(): Assertion `msmc_alloc_dsp_addr_ == msmc_addr' failed.
    Mar 30 00:57:17 am57xx-evm systemd[1]: [[0;1;39mti-mct-daemon.service: Control process exited, code=dumped status=6[[0m
    Mar 30 00:57:17 am57xx-evm systemd[1]: [[0;1;31mFailed to start TI MultiCore Tools Daemon.[[0m
    Mar 30 00:57:17 am57xx-evm systemd[1]: [[0;1;39mti-mct-daemon.service: Unit entered failed state.[[0m
    Mar 30 00:57:17 am57xx-evm systemd[1]: [[0;1;39mti-mct-daemon.service: Failed with result 'core-dump'.[[0m

    root@am57xx-evm:~# cat /proc/cmem

    Block 0: Pool 0: 1 bufs size 0xc000000 (0xc000000 requested)

    Pool 0 busy bufs:

    Pool 0 free bufs:
    id 0: phys addr 0xa0000000

    Block 1: Pool 0: 1 bufs size 0xc000000 (0xc000000 requested)

    Pool 0 busy bufs:

    Pool 0 free bufs:
    id 0: phys addr 0xb0000000

    root@am57xx-evm:~# cd /usr/share/ti/examples/opencl/vecadd
    root@am57xx-evm:/usr/share/ti/examples/opencl/vecadd# ./vecadd
    CMEM Error: registerAlloc: ioctl CMEM_IOCREGUSER failed for phys addr 0xa0000000: -1
    TIOCL FATAL: The TI Multicore Tools daemon (/usr/bin/ti-mctd) is not running. Re-run application after starting the daemon. Refer User Guide for details.
    Aborted (core dumped)

  • Konstantin,

    Let me play around with the CMEM blocks and get back to you.

    Rex
  • Hi Konstantin,

       If you add an extra CMEM block, please add it after the first 2 CMEM blocks.  OpenCL will use the first 2 CMEM blocks: one DDR, one OCMC, and leave the rest alone.  In next PSDK release, OpenCL CMEM block usage will be addressed in a JSON config file which you can change.  For the time being, you can do,

    cmem {
      ...
      cmem_block@0 {
    ... // ddr block
    }
    cmem_block@1 {
    ... // OCMC block
    }
    cmem_block@2 {
    ... // your block
    }
    }

        Alternatively, since you are using OpenCL already.  If all you want is to get some memory in CMEM, you can use the OpenCL memory allocation (they are all allocated in CMEM blocks):

    1) Create an OpenCL buffer, read/write/map/unmap for host access, or

    2) use TI-extension __malloc_ddr()/__free_ddr(), then create a buffer with USE_HOST_PTR property for DSP access.

    Details can be found in our OpenCL user guide:

    If you want more memory for OpenCL use, you can simply increase the CMEM block sizes in DTS file.

    - Yuan

  • Thank you for your time.
    How I found solution after rebuild cmemk in debug mode and compare logs.

    Cmemk have some bug: when I create 3 pools like:

    cmem {
    ...
    cmem_block@0 {
    reg = <0x0>;
    memory-region = <0x131>;
    cmem-buf-pools = <0x1 0x0 0xc000000>;
    };
    cmem_block@1 {
    reg = <0x1>;
    memory-region = <0x132>;
    };
    cmem_block@2 {
    reg = <0x2>;
    memory-region = <&my_cmem>;
    cmem-buf-pools = <0x6 0x0 0x96000>,<0x6 0x0 0x3f5000>;
    };
    };


    I cannot see cmem_block@2 in /proc/cmem. This deceived me. Today I tryed allocate in block 2 with CMEM_alloc2 and it works.

    I want cmem buffers for get dmabuf and able use OpenCL without copy. I try export dmabuf of pointer from __malloc_ddr(), but it not works.

    www.khronos.org/.../cl_arm_import_memory.txt
    This extension will be good feature for OpenCL implementation.
  • Glad you got the CMEM part working. I looked at the ARM extension, I think we essentially have the same idea of importing pre-allocated device memory into OpenCL application so that there will be zero copy. TI extension is achieved by __malloc_ddr() plus buffer created with USE_HOST_PTR property.

    I don't know much about dma_buf. Does it require some initialization in kernel space? ARM extension seems to differentiate between dma_buf memory and host memory. I don't know the significance of the difference since I don't know much about dma_buf. But if you can achieve the same effect by having your own CMEM block, I don't see why you cannot do it with the CMEM memory allocated by OpenCL runtime. They are all CMEM memory, after all. Is it possible that dma_buf interface require physical address of memory instead of virtual address? __malloc_ddr() does the CMEM memory allocation, but returns the mapped virtual address to ARM. If dma_buf requires physical address, you can call certain CMEM APIs to get the physical address back.

    - Yuan