Remoteproc: getting valid rproc handle

Why do you want to get the rproc handle?

rproc objects are allocated by specific architecture drivers, and the handle to the object is stored for later use by the driver.

Jouko Nikula said:

My aim is to use rproc to start the DSP software.

rproc loads and starts a DSP firmware file.  Do you want to both load and start, or just start the DSP?

Jouko Nikula said:

Could someone please point me to the right direction? Or if someone has an example on booting a DSP using remoteproc it would be greatly appreciated

Depending on the architecture, you would name your DSP firmware file according to the architecture's specification and insmod/modprobe the remoteproc modules, at which point your DSP firmware gets loaded and started.

Regards,

- Rob

Robert Tivy said:

rproc loads and starts a DSP firmware file.  Do you want to both load and start, or just start the DSP?

I want to load and start the DSP.

Robert Tivy said:

Depending on the architecture, you would name your DSP firmware file according to the architecture's specification and insmod/modprobe the remoteproc modules, at which point your DSP firmware gets loaded and started.

Thanks, I was on the wrong track here. I'm not obviously interested in the rproc handle anymore. My problem rather is that when I load the remoteproc modules:

modprobe remoteproc
modprobe da8xx_remoteproc da8xx_fw_name=rproc-dsp-fw

I can see that firmware file gets loaded and the resource table is parsed, however rproc_boot() is never called. If I understand correctly rproc_virtio_find_vqs() should be calling rproc_boot(), but who should call rproc_virtio_find_vqs()? Should I be loading some other module as well?

Additionally, I noticed that if I try to load the dsp to DSP L1 memories (device is OMAP-L138), my kernel will die in the very beginning. It seems that it dies in the da8xx_rproc_reserve_cma() at the device initialization. Should it be possible to use DSP memories with remoteproc or am I trying to do something that is not supported? In  other words, if I put on kernel commandline rproc_mem=32K@0x11e00000 my kernel dies silently without anything written onto kernel console.

regards,

     - Jouko

Jouko Nikula said:

modprobe remoteproc
modprobe da8xx_remoteproc da8xx_fw_name=rproc-dsp-fw

I can see that firmware file gets loaded and the resource table is parsed, however rproc_boot() is never called. If I understand correctly rproc_virtio_find_vqs() should be calling rproc_boot(), but who should call rproc_virtio_find_vqs()? Should I be loading some other module as well?

Yes, you also need to load virtio_rpmsg_bus.ko, with:
    % modprobe virtio_rpmsg_bus

Jouko Nikula said:

In  other words, if I put on kernel commandline rproc_mem=32K@0x11e00000 my kernel dies silently without anything written onto kernel console

Hmmm, I'm not sure what's going on with that, I've never tried direct loading into DSP L1/L2 RAM.  I will look into the issue, but I sense that this is a general kernel issue that can't be overcome without changing the generic kernel or the dma mapping subsystem.  The function da8xx_rproc_reserve_cma() is called very early in the boot process, when lots of elements of the system are not yet active/initialized, so we may be limited in what we can do here.

Regards,

- Rob

Robert Tivy said:

Yes, you also need to load virtio_rpmsg_bus.ko, with:
    % modprobe virtio_rpmsg_bus

This does not seem to change anything. The virtio_rpmsg_busmodule loads without errors, but dsp is still not loaded. rproc_boot() does not get called. It seems that I'm still missing something? If I use a self made test module to call rproc_boot() (using the handle from platform driver) dsp is loaded and started.

Robert Tivy said:

Hmmm, I'm not sure what's going on with that, I've never tried direct loading into DSP L1/L2 RAM.  I will look into the issue, but I sense that this is a general kernel issue that can't be overcome without changing the generic kernel or the dma mapping subsystem.  The function da8xx_rproc_reserve_cma() is called very early in the boot process, when lots of elements of the system are not yet active/initialized, so we may be limited in what we can do here.

Ok I understand. I made an ugly hack to replace the carveout dma allocations with iomemory allocations and was able to load the dsp directly to L1 memory.  But this is not very clean solution.

regards,

    - Jouko

Jouko Nikula said:

This does not seem to change anything. The virtio_rpmsg_busmodule loads without errors, but dsp is still not loaded. rproc_boot() does not get called. It seems that I'm still missing something?

So this brings us back to your original question of "who calls rproc_boot()?".  As you stated, rproc_boot() is called by rproc_virtio_find_vqs(), which is assigned to the ".find_vqs" method of the "struct virtio_config_ops rproc_virtio_config_ops" in drivers/rpmsg/virtio_rpmsg_bus.c.  This method can be invoked by many entities, but in this case it is called by rpmsg_probe() in virtio_rpmsg_bus.c (and rpmsg_probe() gets called automatically when the virtio_ipc_driver is registered, which happens in rpmsg_init() in virtio_rpmsg_bus.c). 

FYI, as I just noticed, virtio_rpmsg_bus.ko is automatically installed when you install da8xx_remoteproc.ko (an "lsmod" command will show this).  Regardless, you're saying that rproc_boot() is not getting called, even though your virtio_rpmsg_bus.ko is getting installed.  I don't know what's going on there, it should get called when virtio_rpmsg_bus.ko is installed.  Can you show your console output when attempting this?

Jouko Nikula said:

Ok I understand. I made an ugly hack to replace the carveout dma allocations with iomemory allocations and was able to load the dsp directly to L1 memory.  But this is not very clean solution.

In the remoteproc world, "carveout" means an area that is allocated with dma_alloc_coherent().  Remoteproc uses a CMA region assigned to the device driver to service the dma_alloc_coherent() call, and then maps it into the DSP's MMU (if present) with iommu_map().  A remoteproc "devmem" entry does just the iommu_map().

Since the OMAP-L138 doesn't have an iommu, this handling of a devmem entry seems to be mostly a NOP (aside from a translation table entry).  It doesn't even create a kernel mapping for it (as is done with dma_alloc_coherent()), so I'm surprised that the remoteproc ELF loader was able to write sections to DSP L1P memory at all.  Are you sure that the DSP actually got loaded (when you changed to devmem entry), or did you only see Linux not complain?

FYI, I did try telling remoteproc to assign its CMA region to DSP L1P (by setting rproc_mem=16K@0x11e00000 on the kernel command line), but this hung the kernel right away.

Regards,

- Rob

 

Robert Tivy said:

FYI, as I just noticed, virtio_rpmsg_bus.ko is automatically installed when you install da8xx_remoteproc.ko (an "lsmod" command will show this).  Regardless, you're saying that rproc_boot() is not getting called, even though your virtio_rpmsg_bus.ko is getting installed.  I don't know what's going on there, it should get called when virtio_rpmsg_bus.ko is installed.  Can you show your console output when attempting this?

Sure, but I've just started my holidays so I'm not able to do this now. I will get back with this in August when I'm back at the office.

Robert Tivy said:

In the remoteproc world, "carveout" means an area that is allocated with dma_alloc_coherent().  Remoteproc uses a CMA region assigned to the device driver to service the dma_alloc_coherent() call, and then maps it into the DSP's MMU (if present) with iommu_map().  A remoteproc "devmem" entry does just the iommu_map().

Since the OMAP-L138 doesn't have an iommu, this handling of a devmem entry seems to be mostly a NOP (aside from a translation table entry).  It doesn't even create a kernel mapping for it (as is done with dma_alloc_coherent()), so I'm surprised that the remoteproc ELF loader was able to write sections to DSP L1P memory at all.  Are you sure that the DSP actually got loaded (when you changed to devmem entry), or did you only see Linux not complain?

No I did not use the "devmem" entry. I still use "carveout" in the resource table, but in rproc_handle_carveout() I added calls to request_mem_region() and ioremap_nocache() . I store the pointer given by ioremap() and use this pointer in rproc_da_to_va() and then in rproc_elf_load_segments() I replaced the memcopy() with  memcopy_toio().  This seems to work for me. I can see our DSP blinking me a led and I can do IPC with it. So I'm sure it works.  I'm loading to both DSP L1 memories and to shared memory. I guess my options are to use this kind of hack or to load to SDRAM?

Robert Tivy said:

FYI, I did try telling remoteproc to assign its CMA region to DSP L1P (by setting rproc_mem=16K@0x11e00000 on the kernel command line), but this hung the kernel right away.

Yes, same here. Now I just used a dummy value pointing to SDRAM. I'm not using this memory area in the resource table.

regards,

     - Jouko

Robert Tivy said:

FYI, as I just noticed, virtio_rpmsg_bus.ko is automatically installed when you install da8xx_remoteproc.ko (an "lsmod" command will show this).  Regardless, you're saying that rproc_boot() is not getting called, even though your virtio_rpmsg_bus.ko is getting installed.  I don't know what's going on there, it should get called when virtio_rpmsg_bus.ko is installed.  Can you show your console output when attempting this?

My holidays are over and I tried this now. For me  virtio_rpmsg_bus.ko is not automatically loaded when I modprobe da8xx_remoteproc.ko. Here's what I get:

# uname -a
Linux XPU 3.10.0-rc4-13792-gc89afd8 #3 PREEMPT Mon Aug 5 12:47:48 EEST 2013 armv5tejl GNU/Linux
# lsmod
Module Size Used by Not tainted
# modprobe da8xx_remoteproc da8xx_fw_name=courier_dsp.out
remoteproc0: dsp is available
remoteproc0: Note: remoteproc is still under development and considered experimental.
remoteproc0: THE BINARY FORMAT IS NOT YET FINALIZED, and backward compatibility isn't yet guaranteed.
# lsmod
Module Size Used by Not tainted
da8xx_remoteproc 2431 0
remoteproc 17321 1 da8xx_remoteproc
firmware_class 9901 1 remoteproc
virtio_ring 8901 1 remoteproc
virtio 3622 1 remoteproc

I also manually installed the  virtio_rpmsg_bus.ko and added trace to see what happens when it is loaded. I can see that rpmsg_init() gets called but rpmsg_probe() is never called. Any ideas what is missing?

Robert Tivy said:

I'm not very familiar with the mechanisms involved in automatically loading dependent modules so I probably can't help here, but one question - have you done a fresh "% depmod -a" before doing the modprobes?

I'm doing this with a full fresh build which should generate the module dependencies for me.

Robert Tivy said:

No, I don't know why rpmsg_probe() is not getting called automatically.  I would expect that the kernel would print something indicating what's wrong.  Have you checked for kernel error messages?

No I don't get any kernel error messages.

I think I'm gonna give up with this, because this seems to take too much effort and I already have simple workaround for this. The workaround is that I created a very simple kernel module, which in init calls platform_get_drvdata(&da8xx_dsp) and uses the platform driver's rproc handle to call rproc_boot(). I've defined da8xx_dsp as extern and exported it. In exit it calls rproc_shutdown() with same handle. I think this is essentially the same that rpmsg_probe()  would do for me.

Thanks a lot for your help and effort Rob! Although we didn't find the correct solution you guided me to the right direction to get this working.

There aren't too many examples in the Internet for this and I found this quite difficult. So I'll try to summarize here what I did in order to get DSP load and start to work with remoteproc. I hope this may help others:

1. Compile remoteproc as kernel module.
2. Reserve memory areas for rproc in the kernel command line e.g. rproc_mem=256K@0xc3000000. At least for me this needed to be in the main (DDR) memory. This area or areas should match with the DSP linker file. So that you reserve the areas where DSP wants to be loaded.
3. Define resource table in the DSP code and linker file. For us the resource table needed to be defined as const in order to get an elf section that remoteproc recognizes.
4. Read remoteproc.txt in your kernel documentation. Also found this useful when defining the resource table: http://processors.wiki.ti.com/index.php/IPC_Resource_customTable
5. If you get similar problems as me that remoteproc does not start automatically make simple kernel module to start it as I did above.

regards,

    - Jouko

Thank you for the thorough investigation, and this writeup Jouko. Can you please share how you got a handle to your remoteproc module, to boot/stop that other module? I think I have to do the same thing, because I used workqueue to boot my module, but then I cannot rmmod my module after that. I think another module that boots my module after it has been probed (and loaded into remote CPU) is safer.

Of course, if TI can tell us how the virtio_rpmsg_bus module is supposed to be probed, that would be even better!

Henry Choi said:

Thank you for the thorough investigation, and this writeup Jouko. Can you please share how you got a handle to your remoteproc module, to boot/stop that other module?

This is what I used in a kernel module's init:

    /* Get rproc handle */
    handle = platform_get_drvdata(&da8xx_dsp);
    if (!handle) {
        printk(KERN_ERR "dspload: Failed to get rproc handle.\n");
        return -EFAULT;
    }
    if ((ret = rproc_boot(handle)) != 0) {
        printk(KERN_ERR "dspload: rproc_boot failed (%d).\n", ret);
    }
    return ret;

Additionally I had 

EXPORT_SYMBOL(da8xx_dsp);
in /linux/arch/arm/mach-davinci/devices-da8xx.c
and
extern struct platform_device da8xx_dsp;
in /linux/arch/arm/mach-davinci/include/mach/da8xx.h

As you know this is not the proper way to do this, but just a workaround...

Thank you Jouko, exporting the singleton instance is a nice workaround. I am on Xilinx Zynq platform (the module is zynq_remoteproc), but the technique I came up with last night (adding a RW attribute; described here: henryomd.blogspot.com/.../zynq-amp-linux-on-cpu0-and-bare-metal.html) should a valid for any other platform. I had to modify the device driver, so your workaround is good to learn as well!