DRA718: MessageQ can't be opened between M4 in early start and A15

Hi Gabriel,

Can you clarify which MessageQ_open is failing? Is it a MessageQ_open from the A15 or a MessageQ_open from the M4?

You said you are following processors.wiki.ti.com/.../Early_Boot_and_Late_Attach_in_Linux, but that you are using Android.
For 6AO.1.3 release, please follow the Android-specific Early boot and Late attach page:
processors.wiki.ti.com/.../Early_Boot_and_Late_Attach

Some things to check:
1. After early loading and late attach, check the status and traces of the remote core to make sure it is running. You can find instructions of checking remoteproc info here: www.ti.com/.../sprac12.pdf in Section 4.
2. Check that the MessageQ that you are trying to open has been successfully created on the core that is creating the MessageQ.

Thanks,
Angela

thanks

Hello Angela, thanks for your help, we tried many things but we are still struggling with this problem.

A brief summary of what we tried:

Firstly we changed the dtb in accordance to this specification

processors.wiki.ti.com/.../Early_Boot_and_Late_Attach

since we had in the u-boot the option "Enable using a DMA pool instead of a CMA pool" enabled, but without any noticeable difference.

Then we decided to disable the option "Enable using a DMA pool instead of a CMA pool" and we have been working like that in the rest of our tests.

(I am copying the related parts of our dtsi at the end of the email)


The results of our tests:

I ran a program to only turn on a light and turn it off after 2 seconds, there is nothing else in it (no other code) and everything runs without a problem. The light is turned on right after u-boot and the program never crash, so we think early start and late attach is working.

When I add this code inside a while(1) in a thread

if (hostStatus == 0) {

hostStatus = Resource_getVdevStatus(VIRTIO_ID_RPMSG);

FAMP_PM_SVER("Waiting start of RPMSG: %d\n", hostStatus);

if (hostStatus != 0) {

IpcMgr_ipcStartup();

FAMP_PM_SVER("IpcMgr_ipcStartup\n");

}

}


As soon as the IpcMgr_ipcStartup() is executed the program crashes.

This is the log from the terminal

[ 10.545040] remoteproc0: crash detected in 58820000.ipu: type mmufault

[ 10.561888] remoteproc0: handling crash #1 in 58820000.ipu

[ 10.597489] omap-rproc 58820000.ipu: omap rproc 58820000.ipu crashed

[ 10.616278] remoteproc0: crash detected in 58820000.ipu: type device exception

This is the log from cat /sys/kernel/debug/remoteproc/remoteproc0/trace0_last (there is a sleep of 5 seconds in the thread)

[0][ 0.004] Wake up condition 2

[0][ 0.004] power on by powerButton is 0

[0][ 0.004] Entering pm_getSwitchOnValue

[0][ 5.004] Waiting start of RPMSG: 0

[0][ 10.004] Waiting start of RPMSG: 0

[0][ 15.004] Waiting start of RPMSG: 7

[0][ 15.004] Exception occurred at (PC) = 0000b902

[0][ 15.004] CPU context: thread

[0][ 15.004] BIOS Task name: famp_pm handle: 0x80049380.

[0][ 15.004] BIOS Task stack base: 0x800493d0.

[0][ 15.004] BIOS Task stack size: 0x1000.

[0][ 15.004] [t=0x00000001:7c9163b5] ti.sysbios.family.arm.m3.Hwi: ERROR: line 1104: E_hardFault: FORCED

[0][ 15.004] ti.sysbios.family.arm.m3.Hwi: line 1104: E_hardFault: FORCED

[0][ 15.004] [t=0x00000001:7c93ac8f] ti.sysbios.family.arm.m3.Hwi: ERROR: line 1181: E_busFault: PRECISERR: Immediate Bus Fault, exact addr known, address: 60002000

[0][ 15.004] ti.sysbios.family.arm.m3.Hwi: line 1181: E_busFault: PRECISERR: Immediate Bus Fault, exact addr known, address: 60002000

[0][ 15.004] R0 = 0x60002000 R8 = 0x8004a550

[0][ 15.004] R1 = 0x60002000 R9 = 0x00000012

[0][ 15.004] R2 = 0x9d000000 R10 = 0x00000010

[0][ 15.004] R3 = 0x00000000 R11 = 0x00000000

[0][ 15.004] R4 = 0x80053c0c R12 = 0x00003000

[0][ 15.004] R5 = 0x60001000 SP(R13) = 0x8004a200

[0][ 15.004] R6 = 0x60000000 LR(R14) = 0x00000000

[0][ 15.004] R7 = 0x8004a528 PC(R15) = 0x0000b902

[0][ 15.004] PSR = 0x61000000

[0][ 15.004] ICSR = 0x00438803

[0][ 15.004] MMFSR = 0x00

[0][ 15.004] BFSR = 0x82

[0][ 15.004] UFSR = 0x0000

[0][ 15.004] HFSR = 0x40000000

[0][ 15.004] DFSR = 0x00000000

[0][ 15.004] MMAR = 0x60002000

[0][ 15.004] BFAR = 0x60002000

[0][ 15.004] AFSR = 0x00000000

[0][ 15.004] Stack trace

[0][ 15.004] -- [op 01000000] 00058005

[0][ 15.004] -- [op 2001f8bf] 00004197

[0][ 15.004] 00 [op fe4bf7fe] 0000ca8b (ret from call to 0000b720)

[0][ 15.004] -- [op 00000000] 00014839

[0][ 15.004] 01 [op f9dcf7fe] 0000e569 (ret from call to 0000c920)

[0][ 15.004] 02 [op ff54f7f4] 000195f9 (ret from call to 0000e4a0)

[0][ 15.004] -- [op f0064630] 0000ffff

[0][ 15.004] -- [op f0064630] 0000ffff

[0][ 15.004] 03 [op faa6f003] 0001607d (ret from call to 000195c8)

[0][ 15.004] -- [op 1f04f854] 00008005

[0][ 15.004] 04 [op ff45f005] 00010187 (ret from call to 00016010)

[0][ 15.004] 05 [op faeaf005] 0000ab6d (ret from call to 00010140)

[0][ 15.004] -- [op 42a88800] 00010701

[0][ 15.004] -- [op 0001c3b9] 000163f9

[0][ 15.004] -- [op bd08fefd] 0000aae3

[0][ 15.004] -- [op 0001c3b9] 000163f9

[0][ 15.004] -- [op 80055b54] 0001ca31

[0][ 15.004] Stack dump base 800493d0 size 4096 sp 8004a200:

[0][ 15.004] 8004a200: 8004a238 00000000 0000cb8c 80053c0c 00000005 000220c4 00000000 00004197

[0][ 15.004] 8004a220: 00000000 00000000 00000000 0000ca8b 00000014 80055888 00014839 00000000

[0][ 15.004] 8004a240: 00000024 8004a240 00000000 8004a25c 00000000 00000001 00000001 00000008

[0][ 15.004] 8004a260: 00000000 00000018 8004a264 00000000 8004a274 00000008 00000000 00000001

[0][ 15.004] 8004a280: 00000000 8004a4e8 00000000 00000090 00000090 00000000 000169f4 ffffffff

[0][ 15.004] 8004a2a0: ffffffff 0000e569 00000000 00000000 8004a2e0 00021a7c 8004a31c 00000000

[0][ 15.004] 8004a2c0: 8004a4e8 00000000 ffffffff ffffffff ffffffff 000195f9 00000020 8004a31c

[0][ 15.004] 8004a2e0: 00000020 8004a2e0 00000000 8004a2f8 00000000 00000000 00000008 00000000

[0][ 15.004] 8004a300: ffffffff 00000000 ffffffff 00000000 00000000 0001607d 8004a384 00000000

[0][ 15.004] 8004a320: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

[0][ 15.004] 8004a340: 00000000 00000000 00000000 00000020 8004a34c 00000000 8004a364 00000000

[0][ 15.004] 8004a360: 00000000 00000008 00000000 000220c4 00000005 80055a44 00000000 00010187

[0][ 15.004] 8004a380: 8004a3a0 ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff

[0][ 15.004] 8004a3a0: ffffffff 0000ab6d 00000000 00000000 00010701 000163f9 00000000 00000000

[0][ 15.004] 8004a3c0: 0000aae3 000163f9 0001ca31 bebebebe 00000000 00005c30 00000000 8004a3f0

[0][ 15.004] Terminating execution...

The "weird" thing is that the program in the ipu1 is restarted after it crashs a couple of times and then it all just starts ok, this is the extended log:

[ 9.281936] ueventd: firmware: loading 'dra7-ipu1-fw.xem4' for '/devices/platform/44000000.ocp/58820000.ipu/remoteproc0/firmware/dra7-ipu1-fw.xem4'

[ 9.315670] ueventd: loading /devices/platform/44000000.ocp/58820000.ipu/remoteproc0/firmware/dra7-ipu1-fw.xem4 took 0.0339983 seconds

[ 9.329237] remoteproc0: powering up 58820000.ipu

[ 9.338417] remoteproc0: Booting fw image dra7-ipu1-fw.xem4, size 149968

[ 9.352755] remoteproc0: remote processor 58820000.ipu is now up

[ 9.386097] remoteproc0: registered virtio0 (type 7)

[ 11.202577] remoteproc0: crash detected in 58820000.ipu: type mmufault

[ 11.226272] remoteproc0: handling crash #1 in 58820000.ipu

[ 11.232062] remoteproc0: recovering 58820000.ipu

[ 11.258035] remoteproc0: crash detected in 58820000.ipu: type device exception

[ 11.280763] remoteproc0: stopped remote processor 58820000.ipu

[ 11.287076] remoteproc0: Direct firmware load for dra7-ipu1-fw.xem4 failed with error -2

[ 11.302772] remoteproc0: Falling back to user helper

[ 11.322333] ueventd: firmware: loading 'dra7-ipu1-fw.xem4' for '/devices/platform/44000000.ocp/58820000.ipu/remoteproc0/firmware/dra7-ipu1-fw.xem4'

[ 11.339071] remoteproc0: powering up 58820000.ipu

[ 11.347782] ueventd: loading /devices/platform/44000000.ocp/58820000.ipu/remoteproc0/firmware/dra7-ipu1-fw.xem4 took 0.0257506 seconds

[ 11.360111] remoteproc0: Booting fw image dra7-ipu1-fw.xem4, size 149968

[ 11.388709] remoteproc0: remote processor 58820000.ipu is now up

[ 11.427711] remoteproc0: registered virtio0 (type 7)

I can start lad_dra7xx and my own process in the Android user space and I am able to connect and communicate with a queue created in the ipu1.

We are not sure what is happening and why it is crashing, do you have any idea or something we can try to solve this issue?

As a side note, we never changed anything in the resource table, in fact we are not sure which resource table is being compiled at the time, it is "packages/ti/ipc/remoteproc/rsc_table_vayu_ipu.h"? Could this problem be derived from a bad definition there?

Thanks in advance for your help.

Gabriel

PS: I forgot to mention that we tried the example messageQ_single with the same results

DTSI:

#include "dra71-famp-common.dtsi"
#include <dt-bindings/net/ti-dp83867.h>

/ {

memory {
device_type = "memory";
reg = <0x0 0x80000000 0x0 0x80000000>; /* 2GB */
};

reserved_mem: reserved-memory {
#address-cells = <2>;
#size-cells = <2>;
ranges;

ipu2_cma_pool: ipu2_cma@95800000 {
compatible = "shared-dma-pool";
reg = <0x0 0x95800000 0x0 0x3800000>;
reusable;
status = "okay";
};

ipu1_cma_pool: ipu1_cma@9d000000 {
compatible = "shared-dma-pool";
reg = <0x0 0x9d000000 0x0 0x2000000>;
reusable;
status = "okay";
};
};
};

#define LATE_ATTACH(label) &label { \
ti,late-attach;\
ti,no-idle-on-init;\
ti,no-reset-on-init; }

#define NO_MAP(label) &label { \
/delete-property/ reusable; \
no-map; }

LATE_ATTACH(ipu1);
LATE_ATTACH(timer11);
LATE_ATTACH(timer7);
LATE_ATTACH(timer8);
//LATE_ATTACH(mmu_ipu1);
LATE_ATTACH(gpio4);

&mbox_ipu1_ipc3x {
ti,no-idle-on-init;
ti,no-reset-on-init;
};

//#define USE_DMA

#ifdef USE_DMA
&mmu_ipu1 {
ti,late-attach;
ti,late-attach-dma-pool;
ti,no-idle-on-init;
ti,no-reset-on-init;
};

&ipu1_cma_pool {
/delete-property/ reusable;
no-map;
ti,no-idle-on-init;
ti,no-reset-on-init;
};

#else
LATE_ATTACH(mmu_ipu1);
#endif


&reserved_mem {
latea_pagetbl: late_pgtbl@bfc00000 {
reg = <0x0 0xbfc00000 0x0 0x100000>;
no-map;
status = "okay";
};
};

&dra7_pmx_core {
pinctrl_fsync_pwm: fsync_pwm_grp {
pinctrl-single,pins = <
DRA7XX_CORE_IOPAD(0x3774, (PIN_OUTPUT | MUX_MODE10)) /* gpio6_10.ehrpwm2A */
>;
};
};

&ipu2 {
status = "disabled";
memory-region = <&ipu2_cma_pool>;
};

&ipu1 {
status = "okay";
memory-region = <&ipu1_cma_pool>;
};

Hi Garbriel,

>>[0][ 15.004] [t=0x00000001:7c93ac8f] ti.sysbios.family.arm.m3.Hwi: ERROR: line 1181: E_busFault: PRECISERR: Immediate Bus Fault, exact addr known, address: 60002000

This is showing IPU virtual address 0x60002000 as the address that caused the issue. This is IPC vring memory, looks like it is unable to access it. In early loading case, the physical memory used to program the MMU for this address is allocated from the CMA region. The CMA region address needs to match between u-boot early loading code and kernel dts config.

Did you make any changes to the default CMA address in bootloader (default CMA address for IPU1 is 0x9d000000)? If so, then it needs to be updated in the kernel dts ipu1_cma_pool node. Also, the resource table address for the VRING needs to be updated in the IPU1 firmware. Can you check that all of these are the same?


>> The "weird" thing is that the program in the ipu1 is restarted after it crashs a couple of times and then it all just starts ok, this is the extended log:

This could be due to CMA address mis-match between bootloader and kernel dts. When the remote core is reloaded after the crash, then it is doing normal (non-early) remoteproc loading and will only use the kernel CMA address from dts file for loading the remote core, and for IPC communication.


>> We are not sure what is happening and why it is crashing, do you have any idea or something we can try to solve this issue?

>>As a side note, we never changed anything in the resource table, in fact we are not sure which resource table is being compiled at the
>>time, it is "packages/ti/ipc/remoteproc/rsc_table_vayu_ipu.h"? Could this problem be derived from a bad definition there?

Can you please check the CMA address configuration across these 3 locations?

- bootloader (lateattach.c)
- kernel dts
- resource table vring physical address (PHYS_MEM_IPC_VRING).

The default resource table used is packages/ti/ipc/remoteproc/rsc_table_vayu_ipu.h. To use a custom table, you can follow the instructions here processors.wiki.ti.com/.../IPC_Resource_customTable. If you need to update the vring physical address entries, you'll need to use a custom resource table.


Thanks,
Angela

Hi Angela

As far we know the following files matches CMA address

lateattach.c

dts's

packages/ti/ipc/remoteproc/rsc_table_vayu_ipu.h

We are a bit confused by the address of the following files

$CUSTOM_PROJ_DIR\ipu_ammu.cfg

/* Overwrite smallPage[1] so that 16K is covered. H/w reset value configures
 * only 4K */
AMMU.smallPages[1].pageEnabled = AMMU.Enable_YES;
AMMU.smallPages[1].logicalAddress = 0x40000000;
AMMU.smallPages[1].translatedAddress = 0x55080000;
AMMU.smallPages[1].translationEnabled = AMMU.Enable_YES;
AMMU.smallPages[1].size = AMMU.Small_16K;

/* L2RAM: 64K mapped using 4 smallPages(16K); cacheable; translated */
/* config small page[2] to map 16K VA 0x20000000 to PA 0x55020000  */
AMMU.smallPages[2].pageEnabled = AMMU.Enable_YES;
AMMU.smallPages[2].logicalAddress = 0x20000000;
AMMU.smallPages[2].translatedAddress = 0x55020000;
AMMU.smallPages[2].translationEnabled = AMMU.Enable_YES;
AMMU.smallPages[2].L1_writePolicy = AMMU.WritePolicy_WRITE_BACK;
AMMU.smallPages[2].L1_allocate = AMMU.AllocatePolicy_ALLOCATE;
AMMU.smallPages[2].L1_posted = AMMU.PostedPolicy_POSTED;
AMMU.smallPages[2].L1_cacheable = AMMU.CachePolicy_CACHEABLE;
AMMU.smallPages[2].size = AMMU.Small_16K;

/* config small page[3] to map 16K VA 0x20004000 to PA 0x55024000 */
AMMU.smallPages[3].pageEnabled = AMMU.Enable_YES;
AMMU.smallPages[3].logicalAddress = 0x20004000;
AMMU.smallPages[3].translatedAddress = 0x55024000;
AMMU.smallPages[3].translationEnabled = AMMU.Enable_YES;
AMMU.smallPages[3].L1_writePolicy = AMMU.WritePolicy_WRITE_BACK;
AMMU.smallPages[3].L1_allocate = AMMU.AllocatePolicy_ALLOCATE;
AMMU.smallPages[3].L1_posted = AMMU.PostedPolicy_POSTED;
AMMU.smallPages[3].L1_cacheable = AMMU.CachePolicy_CACHEABLE;
AMMU.smallPages[3].size = AMMU.Small_16K;

/* config small page[4] to map 16K VA 0x20008000 to PA 0x55028000 */
AMMU.smallPages[4].pageEnabled = AMMU.Enable_YES;
AMMU.smallPages[4].logicalAddress = 0x20008000;
AMMU.smallPages[4].translatedAddress = 0x55028000;
AMMU.smallPages[4].translationEnabled = AMMU.Enable_YES;
AMMU.smallPages[4].L1_writePolicy = AMMU.WritePolicy_WRITE_BACK;
AMMU.smallPages[4].L1_allocate = AMMU.AllocatePolicy_ALLOCATE;
AMMU.smallPages[4].L1_posted = AMMU.PostedPolicy_POSTED;
AMMU.smallPages[4].L1_cacheable = AMMU.CachePolicy_CACHEABLE;
AMMU.smallPages[4].size = AMMU.Small_16K;

/* config small page[5] to map 16K VA 0x2000C000 to PA 0x5502C000 */
AMMU.smallPages[5].pageEnabled = AMMU.Enable_YES;
AMMU.smallPages[5].logicalAddress = 0x2000C000;
AMMU.smallPages[5].translatedAddress = 0x5502C000;
AMMU.smallPages[5].translationEnabled = AMMU.Enable_YES;
AMMU.smallPages[5].L1_writePolicy = AMMU.WritePolicy_WRITE_BACK;
AMMU.smallPages[5].L1_allocate = AMMU.AllocatePolicy_ALLOCATE;
AMMU.smallPages[5].L1_posted = AMMU.PostedPolicy_POSTED;
AMMU.smallPages[5].L1_cacheable = AMMU.CachePolicy_CACHEABLE;
AMMU.smallPages[5].size = AMMU.Small_16K;

/* ISS: Use 3 small pages(1 4K and 2 16K) for various ISP registers; non-cacheable; translated */
/* config small page[6] to map 16K VA 0x50000000 to PA 0x55040000 */
/* non cacheable by default */
AMMU.smallPages[6].pageEnabled = AMMU.Enable_YES;
AMMU.smallPages[6].logicalAddress = 0x50000000;
AMMU.smallPages[6].translatedAddress = 0x55040000;
AMMU.smallPages[6].translationEnabled = AMMU.Enable_YES;
AMMU.smallPages[6].size = AMMU.Small_16K;

/* config small page[7] to map 16K VA 0x50010000 to PA 0x55050000 */
/* non cacheable by default */
AMMU.smallPages[7].pageEnabled = AMMU.Enable_YES;
AMMU.smallPages[7].logicalAddress = 0x50010000;
AMMU.smallPages[7].translatedAddress = 0x55050000;
AMMU.smallPages[7].translationEnabled = AMMU.Enable_YES;
AMMU.smallPages[7].size = AMMU.Small_16K;

/* config small page[8] to map 4K VA 0x50020000 to PA 0x55060000 */
/* non cacheable by default */
AMMU.smallPages[8].pageEnabled = AMMU.Enable_YES;
AMMU.smallPages[8].logicalAddress = 0x50020000;
AMMU.smallPages[8].translatedAddress = 0x55060000;
AMMU.smallPages[8].translationEnabled = AMMU.Enable_YES;
AMMU.smallPages[8].size = AMMU.Small_4K;

/*********************** Medium Pages *************************/
/* ISS: The entire ISS register space using a medium page (256K); cacheable; translated */
/* config medium page[0] to map 256K VA 0x50000000 to PA 0x55040000 */
/* Make it L1 cacheable */
AMMU.mediumPages[0].pageEnabled = AMMU.Enable_YES;
AMMU.mediumPages[0].logicalAddress = 0x50000000;
AMMU.mediumPages[0].translatedAddress = 0x55040000;
AMMU.mediumPages[0].translationEnabled = AMMU.Enable_YES;
AMMU.mediumPages[0].size = AMMU.Medium_256K;
AMMU.mediumPages[0].L1_cacheable = AMMU.CachePolicy_CACHEABLE;
AMMU.mediumPages[0].L1_posted = AMMU.PostedPolicy_POSTED;


/*********************** Large Pages *************************/
/* Instruction Code: Large page  (512M); cacheable */
/* config large page[0] to map 512MB VA 0x0 to L3 0x0 */
AMMU.largePages[0].pageEnabled = AMMU.Enable_YES;
AMMU.largePages[0].logicalAddress = 0x0;
AMMU.largePages[0].translationEnabled = AMMU.Enable_NO;
AMMU.largePages[0].size = AMMU.Large_512M;
AMMU.largePages[0].L1_cacheable = AMMU.CachePolicy_CACHEABLE;
AMMU.largePages[0].L1_posted = AMMU.PostedPolicy_POSTED;

/* Peripheral regions: Large Page (512M); non-cacheable */
/* config large page[1] to map 512MB VA 0x60000000 to L3 0x60000000 */

AMMU.largePages[1].pageEnabled = AMMU.Enable_YES;
AMMU.largePages[1].logicalAddress = 0x60000000;
AMMU.largePages[1].translationEnabled = AMMU.Enable_NO;
AMMU.largePages[1].size = AMMU.Large_512M;
AMMU.largePages[1].L1_cacheable = AMMU.CachePolicy_NON_CACHEABLE;
AMMU.largePages[1].L1_posted = AMMU.PostedPolicy_POSTED;

/*
AMMU.largePages[1].pageEnabled = AMMU.Enable_YES;
AMMU.largePages[1].logicalAddress = 0x40000000;
AMMU.largePages[1].translatedAddress = 0x60000000;
AMMU.largePages[1].translationEnabled = AMMU.Enable_YES;
AMMU.largePages[1].size = AMMU.Large_512M;
AMMU.largePages[1].L1_cacheable = AMMU.CachePolicy_NON_CACHEABLE;
AMMU.largePages[1].L1_posted = AMMU.PostedPolicy_NON_POSTED;
AMMU.largePages[1].L2_cacheable = AMMU.CachePolicy_NON_CACHEABLE;
AMMU.largePages[1].L2_posted = AMMU.PostedPolicy_NON_POSTED;
*/

/* Private, Shared and IPC Data regions: Large page (512M); cacheable */
/* config large page[2] to map 512MB VA 0x80000000 to L3 0x80000000 */
AMMU.largePages[2].pageEnabled = AMMU.Enable_YES;
AMMU.largePages[2].logicalAddress = 0x80000000;
AMMU.largePages[2].translationEnabled = AMMU.Enable_NO;
AMMU.largePages[2].size = AMMU.Large_512M;
AMMU.largePages[2].L1_cacheable = AMMU.CachePolicy_CACHEABLE;
AMMU.largePages[2].L1_posted = AMMU.PostedPolicy_POSTED;

/* TILER & DMM regions: Large page (512M); cacheable */
/* config large page[3] to map 512MB VA 0xA0000000 to L3 0xA0000000 */
AMMU.largePages[3].pageEnabled = AMMU.Enable_YES;
AMMU.largePages[3].logicalAddress = 0xA0000000;
AMMU.largePages[3].translationEnabled = AMMU.Enable_NO;
AMMU.largePages[3].size = AMMU.Large_512M;
AMMU.largePages[3].L1_cacheable = AMMU.CachePolicy_CACHEABLE;
AMMU.largePages[3].L1_posted = AMMU.PostedPolicy_POSTED;

$CUSTOM_PROJ_DIRR\Debug\j6_m4_demo.map

MEMORY CONFIGURATION

         name            origin    length      used     unused   attr    fill
----------------------  --------  ---------  --------  --------  ----  --------
  L2_ROM                00000000   00004000  000005d0  00003a30  RWIX
  EXT_CODE              00004000   000fc000  0001f90a  000dc6f6  RW X
  L2_RAM                20000000   00010000  00000000  00010000  RWIX
  OCMC_RAM1             40300000   00080000  00000000  00080000  RWIX
  OCMC_RAM2             40400000   00100000  00000000  00100000  RWIX
  OCMC_RAM3             40500000   00100000  00000000  00100000  RWIX
  EXT_DATA              80000000   00200000  00056caa  001a9356  RW  
  EXT_HEAP              80200000   00300000  00000000  00300000  RW  
  TRACE_BUF             9f000000   00060000  00008000  00058000  RW  
  EXC_DATA              9f060000   00010000  00000200  0000fe00  RW  
  PM_DATA               9f070000   00020000  00000000  00020000  RW X

/pdk_dra7xx_1_0_7/packages/ti/build/tda2ex/mem_segment_definition_1024mb_bios.xs

1- We are trying to relate address configured at mem_segment_definition_1024mb_bios.xs, with generated .map file, and rsc_table_vayu_ipu.h.

Could you help with that?

2-SR1_FRAME_BUFFER_ADDR & SR1_FRAME_BUFFER_SIZE, should match with IPU_MEM_IPC_VRING  & PHYS_MEM_IPC_VRING, translated to PHYS_MEM_IPC_VRING ? It should be showed at .map, isn't it?

Regards

Hi Christian,

What is the address for DRA7_RPROC_CMA_BASE_IPU1 and DRA7_RPROC_CMA_SIZE_IPU1 in lateattach.c in your bootloader code?
What is the value for "reg" for ipu1_cma_pool in your dts file?

Regarding your other questions:

>> 1- We are trying to relate address configured at mem_segment_definition_1024mb_bios.xs, with generated .map file, and rsc_table_vayu_ipu.h.

Your map file is generated based on your defined memory segments. The resource table will list all addresses which need programming in the MMU. All memory accessed by the IPU1 needs to be included in the resource table.

More information on the resource table can be found here processors.wiki.ti.com/.../IPC_Resource_customTable and here www.kernel.org/.../remoteproc.txt

>> 2-SR1_FRAME_BUFFER_ADDR & SR1_FRAME_BUFFER_SIZE, should match with IPU_MEM_IPC_VRING & PHYS_MEM_IPC_VRING, translated to PHYS_MEM_IPC_VRING ? It should be showed at .map, isn't it?

I'm not familiar with SR1_FRAME_BUFFER_ADDR & SR1_FRAME_BUFFER_SIZE. It is not related to IPC Vring.
The Vring memory does not need to show in the .map file, but it does need to have an entry in the resource table.

Thanks,
Angela

Hi Angela

dts
ipu1_cma_pool: ipu1_cma@9d000000
...
reg = <0x0 0x9d000000 0x0 0x2000000>
...

lateattach.c
#define DRA7_RPROC_CMA_BASE_IPU1 0x9d000000
#define DRA7_RPROC_CMA_SIZE_IPU1 0x02000000

The files are the right ones?
/pdk_dra7xx_1_0_7/packages/ti/build/tda2ex/mem_segment_definition_1024mb_bios.xs
$CUSTOM_PROJ_DIRR\Debug\j6_m4_demo.map
Should the following iInformation should be in .map?
SR0
IPU1_CODE_MEM
IPU1_DATA_MEM
REMOTE_LOG_MEM
SYSTEM_IPC_SHM_MEM

Regards

Hi Christian,

Thanks for providing the values. They match the defaults.

>> The files are the right ones?
>> /pdk_dra7xx_1_0_7/packages/ti/build/tda2ex/mem_segment_definition_1024mb_bios.xs
>> $CUSTOM_PROJ_DIRR\Debug\j6_m4_demo.map
>> Should the following iInformation should be in .map?
>> SR0
>> IPU1_CODE_MEM
>> IPU1_DATA_MEM
>> REMOTE_LOG_MEM
>> SYSTEM_IPC_SHM_MEM
I am not following what these are. Are these memory segment names that you are expecting to show up in the map file, but are not there?


Can we try with the messageq_single.xem4 example only for now? You see the same behavior with early loading messageq_single, right?

Thanks,
Angela

Hello Angela, thanks for your continues help,

Angela Stegmaier said:

>> SR0
>> IPU1_CODE_MEM
>> IPU1_DATA_MEM
>> REMOTE_LOG_MEM
>> SYSTEM_IPC_SHM_MEM

Yes, we are expecting to see these values somewhere and we thought they would appear in the map file, but they are not there.

So we are unsure where this values are taken from, we think they may be obtained from here

/pdk_dra7xx_1_0_7/packages/ti/build/tda2ex/mem_segment_definition_1024mb_bios.xs

Could you please confirm this information?

Regarding the messageQ_single we tried with that and a similar result, but with a different sequence:

On booting everything seems to be going ok:

[    9.276351] ueventd: firmware: loading 'dra7-ipu1-fw.xem4' for '/devices/platform/44000000.ocp/58820000.ipu/remoteproc0/firmware/dra7-ipu1-fw.xem4'
[    9.303756] ueventd: loading /devices/platform/44000000.ocp/58820000.ipu/remoteproc0/firmware/dra7-ipu1-fw.xem4 took 0.0276581 seconds
[    9.317312]  remoteproc0: powering up 58820000.ipu
[    9.326131]  remoteproc0: Booting fw image dra7-ipu1-fw.xem4, size 116096
[    9.342510]  remoteproc0: remote processor 58820000.ipu is now up
[    9.375815]  remoteproc0: registered virtio0 (type 7)

But as soon as we call messageQApp the ipu1 crashes:

famp:/ # messageQApp 10 2                                                      
Using numLoops: 10; procId : 2
Entered MessageQApp_execute
Local MessageQId: 0x80[   80.454615] omap-iommu 58882000.mmu: 58882000.mmu: version 2.1

[   80.462621] omap-iommu 58882000.mmu: iommu fault: da 0x0 flags 0x0
[   80.468831]  remoteproc0: crash detected in 58820000.ipu: type mmufault
[   80.475488] omap-iommu 58882000.mmu: 58882000.mmu: errs:0x00000002 da:0x00000000 pgd:0xf1ee0000 *pgd:px00000000
[   80.485693]  remoteproc0: handling crash #1 in 58820000.ipu
[   80.491291]  remoteproc0: recovering 58820000.ipu
recvfrom failed: Link has been severed (67)
rpmsgThreadFxn: transportGet failed on fd 8, returned -20
[   80.533272]  remoteproc0: stopped remote processor 58820000.ipu
[   80.539319] omap-rproc 58820000.ipu: failed to send mailbox message, status = -22

After the crash the IPU1 goes offline and there is no trace or logs in

/sys/kernel/debug/remoteproc/remoteproc0/trace0

If I restart the IPU1 firmware with an "unbind" - "bind", then everything works and the messageQApp can successfully communicate with the queue in the IPU1.

Regards,

Gabriel

Hi Gabriel,

Gabriel G said:

Yes, we are expecting to see these values somewhere and we thought they would appear in the map file, but they are not there.

So we are unsure where this values are taken from, we think they may be obtained from here

/pdk_dra7xx_1_0_7/packages/ti/build/tda2ex/mem_segment_definition_1024mb_bios.xs

Could you please confirm this information?

Can I ask why you are expecting to see these specific values? Have you seen them somewhere else? Your bld file for your firmware configures the externalMemoryMap. This is where your external memory map is configured.

Gabriel G said:

Regarding the messageQ_single we tried with that and a similar result, but with a different sequence:

On booting everything seems to be going ok:

[    9.276351] ueventd: firmware: loading 'dra7-ipu1-fw.xem4' for '/devices/platform/44000000.ocp/58820000.ipu/remoteproc0/firmware/dra7-ipu1-fw.xem4'
[    9.303756] ueventd: loading /devices/platform/44000000.ocp/58820000.ipu/remoteproc0/firmware/dra7-ipu1-fw.xem4 took 0.0276581 seconds
[    9.317312]  remoteproc0: powering up 58820000.ipu
[    9.326131]  remoteproc0: Booting fw image dra7-ipu1-fw.xem4, size 116096
[    9.342510]  remoteproc0: remote processor 58820000.ipu is now up
[    9.375815]  remoteproc0: registered virtio0 (type 7)

At this point, after booting, but before running messageQApp, can you capture the following debug information?

cat /d/remoteproc/remoteproc0/state

cat /d/remoteproc/remoteproc0/name

cat /d/remoteproc/remoteproc0/trace0

cat /d/omap_iommu/58882000.mmu/regs

cat /d/omap_iommu/58882000.mmu/pagetable

Thanks,

Angela

Angela Stegmaier said:

Can I ask why you are expecting to see these specific values? Have you seen them somewhere else? Your bld file for your firmware configures the externalMemoryMap. This is where your external memory map is configured.

We are working at the same time with the sdk vision for the IPU2, and the guide we are using to set the memory map is this:

processors.wiki.ti.com/.../VisionSDK_UserGuide_MemoryMap

Here is an example of the resulting map, as you can see it contains the location of the segments we were asking you about before, we were expecting to find something similar when working with the IPU1 (even though they are different sdks they are based in the same build system)

IPU2 memory map (SDK vision, please ignore the values, they are from a different project)

MEMORY CONFIGURATION

         name            origin    length      used     unused   attr    fill
----------------------  --------  ---------  --------  --------  ----  --------
  L2_ROM                00000000   00004000  00000184  00003e7c  RWIX
  L2_RAM                20000000   00010000  00000000  00010000  RWIX
  OCMC_RAM1             40300000   00080000  00000000  00080000  RWIX
  OCMC_RAM2             40400000   00100000  00000000  00100000  RWIX
  OCMC_RAM3             40500000   00100000  00000000  00100000  RWIX
  DSP1_L2_SRAM          40800000   00048000  00000000  00048000  RWIX
  NDK_MEM               84000000   00200000  00000000  00200000  RWIX
  SR1_BUFF_ECC_ASIL_MEM 84200000   00001000  00000000  00001000  RWIX
  SR1_BUFF_ECC_QM_MEM   84201000   00001000  00000000  00001000  RWIX
  SR1_BUFF_NON_ECC_ASIL 84202000   00001000  00000000  00001000  RWIX
  SR1_FRAME_BUFFER_MEM  84203000   0fa00000  00000000  0fa00000  RWIX
  IPU2_CODE_MEM         99000000   00800000  00610be8  001ef418  RWIX
  IPU2_DATA_MEM         99800000   03c00000  02f2d735  00cd28cb  RWIX
  IPU1_0_CODE_MEM       9e000000   00800000  00000000  00800000  RWIX
  IPU1_0_DATA_MEM       9e800000   01400000  00000000  01400000  RWIX
  SR0                   a0100000   00100000  00100000  00000000  RWIX
  REMOTE_LOG_MEM        a0200000   00040000  000278a0  00018760  RWIX
  LINK_STATS_MEM        a0240000   00080000  00024da4  0005b25c  RWIX
  SYSTEM_IPC_SHM_MEM    a02c0000   00080000  00054640  0002b9c0  RWIX
  HDVPSS_DESC_MEM       a0340000   00100000  0001b340  000e4cc0  RWIX
  TRACE_BUF             a0440000   00060000  00008000  00058000  RWIX
  EXC_DATA              a04a0000   00010000  00000000  00010000  RWIX
  PM_DATA               a04b0000   00080000  00000000  00080000  RWIX
  DSP1_CODE_MEM         a1000000   00200000  00000000  00200000  RWIX
  DSP1_DATA_MEM         a1200000   01800000  00000000  01800000  RWIX
  SR2_MEM               a9000000   04000000  00000000  04000000  RWIX

But the memory map obtained in the IPU1 is this (which is based in our config.bld)

         name            origin    length      used     unused   attr    fill
----------------------  --------  ---------  --------  --------  ----  --------
  L2_ROM                00000000   00004000  000005d0  00003a30  RWIX
  EXT_CODE              00004000   000fc000  0001f7a6  000dc85a  RW X
  L2_RAM                20000000   00010000  00000000  00010000  RWIX
  OCMC_RAM1             40300000   00080000  00000000  00080000  RWIX
  OCMC_RAM2             40400000   00100000  00000000  00100000  RWIX
  OCMC_RAM3             40500000   00100000  00000000  00100000  RWIX
  EXT_DATA              80000000   00200000  00056caa  001a9356  RW  
  EXT_HEAP              80200000   00300000  00000000  00300000  RW  
  TRACE_BUF             9f000000   00060000  00008000  00058000  RW  
  EXC_DATA              9f060000   00010000  00000200  0000fe00  RW  
  PM_DATA               9f070000   00020000  00000000  00020000  RW X

It seems that the changes we make in the resource table are affecting the positioning of the segments but they are not reflected in the map file, right?

The bottom line of this, and the main reason we are asking all this questions, is that at some point we will have to harmonize all cores (A15, IPU1, IPU2) to use the same memory map, so we need to understand and be sure where all the values are set. In the meantime, understanding the memory map can also help us identify if there is something that can cause our current problem with the message queues.

Angela Stegmaier said:

cat /d/remoteproc/remoteproc0/state

suspended (1)

Angela Stegmaier said:

cat /d/remoteproc/remoteproc0/name

58820000.ipu

Angela Stegmaier said:

cat /d/remoteproc/remoteproc0/trace0

[0][      0.000] Watchdog enabled: TimerBase = 0x68824000 SMP-Core = 0 Freq = 19200000
[0][      0.000] Watchdog enabled: TimerBase = 0x68826000 SMP-Core = 1 Freq = 19200000
[0][      0.000] Watchdog_restore registered as a resume callback
[0][      0.000] 18 Resource entries at 0x3000
[0][      0.000] messageq_single.c:main: MultiProc id = 2
[0][      0.000] registering rpmsg-proto:rpmsg-proto service on 61 with HOST
[0][      0.000] tsk1Fxn: created MessageQ: SLAVE_IPU1; QueueID: 0x20080
[0][      0.000] Awaiting sync message from host...

Angela Stegmaier said:

cat /d/omap_iommu/58882000.mmu/regs

[ 3677.560335] omap-iommu 58882000.mmu: 58882000.mmu: version 2.1
            REVISION: 00000021
           IRQSTATUS: 00000000
           IRQENABLE: 0000001e
          WALKING_ST: 00000000
                CNTL: 00000006[ 3677.566720] omap-iommu 58882000.mmu: 58882000.mmu: version 2.1

            FAULT_AD: 00000000
                LOCK: 00000000
              LD_TLB: 00000000
                 CAM: 00000000
                 RAM: 00000000
              GFLUSH: 00000000
         FLUSH_ENTRY: 00000000
            READ_CAM: 00000000
            READ_RAM: 00000000
        EMU_FAULT_AD: 00000000

Angela Stegmaier said:

cat /d/omap_iommu/58882000.mmu/pagetable

L:      da:     pte:
--------------------------

Regards,

Gabriel

Hi Gabriel,

Are you also loading/running an IPU2 image during this test?

Your output for "/d/omap_iommu/58882000.mmu/regs" is missing the trace that shows the TTB address, and also the pagetable looks to be empty. Can you get at bootloader prompt the value from the following register?

IPU1_MMU_TTB: 0x5888 204C

What address are you using for the pagetable in the bootloader? Is it the default one of 0xbfc00000? Or has it been modified? Are you using a 1GB board? If so, you may need to follow the suggestion found here: e2e.ti.com/.../2568213

Thanks,
Angela

Hi Angela

Original values

U-Boot

md.l 0x5888204C 1

0xbfc00000

Linux

memdev2 0x5888204C

0xbfc00000

Changes applied

Linux dts

&reserved_mem {
   latea_pagetbl: late_pgtbl@9eff8000 {
      reg = ⟨0x0 0x9eff8000 0x0 0x4400⟩;
      no-map;
      status = "okay";
   };
};

U-Boot

Patch e2e.ti.com/.../2568213

md.l 0x5888204C 1

0x9eff8000

WIth this value, system halt at "Starting kernel"

Regards

Hi Christian,

So, you are using a 1GB board?

Is the behavior the same if you don't add the new reserved_mem kernel entry?

Can you enable earlyprintk so we can see what is happening in the kernel?

Thanks,
Angela

Angela

• We are using 1GB board
• With this implementation

&reserved_mem {
   latea_pagetbl: late_pgtbl@bfc00000 {
      reg = ⟨0x0 0xbfc00000 0x0 0x4400⟩;
      no-map;
      status = "okay";
   };
};

Kernel boot, but obviously IPU crash

• Are you sure about the late change request?

With this one IPU1 page table address is inside IPU CMA range, so in DTS

ipu1_cma_pool:ipu1_cma@9d000000{
    compatible = "shared-dma-pool";
    reg = <0x0 0x9d000000 0x0 0x2000000>  /*=9f000000*/
    reusable;
    status = "okay";
};

&ipu1{
   memory-region = <&ipu1_cma_pool>;
   ti, late-attach;
   ti, no-reset-on-init;
   ti, no-idle-on-init;
   status = "okay";
 }

&reserved_mem {   
     latea_pagetbl: late_pgtbl@9eff8000 {      
     reg = <0x0 0x9eff8000 0x0 0x4400>;       
     no-map;      
     status = "okay";    
     };
 };

Note that the proposed patch was intended for the VISION SDK, which has a quite different memory map.

• We ll try earlyprintk and update you
  
• There is a pending question that becomes more and more important, about how to harmonize the memory map.
  
  • Is mem_segment_definitionxxxxx.xs used by TI-RTOS?
    
  • Where is IPC shared data structure (named SR0 by VISION)?
  • Where is IPU text and data?

Regards

Hi Christian,

It looks like this entry might be the issue preventing kernel boot:

&reserved_mem {
latea_pagetbl: late_pgtbl@9eff8000 {
reg = <0x0 0x9eff8000 0x0 0x4400>;
no-map;
status = "okay";
};
};

Instead, can you please try reserving 2MB at 0xafc00000 as a test? I have tried this on my setup and it is working with messageq_single early loading:

&reserved_mem {
latea_pagetbl: late_pgtbl@afc00000 {
reg = <0x0 0xafc00000 0x0 0x200000>;
no-map;
status = "okay";
};
};

And update the bootloader with the same change to start address of the IPU1 pagetable memory (0xafc00000).

These changes would be in place of the previous patch. Can you try with these changes with early loading and see the behavior?

Regarding the memory map questions:

- When you say "TI-RTOS" are you referring to the SDK or to the OS?
- SR0 ("SharedRegion 0") is not used in the IPC messageq_single example. SharedRegion 0 is not needed for MessageQ communication between HLOS and remotecore running SYS/BIOS because it is based on rpmsg. SharedRegion 0 is used for IPC between remote cores running SYS/BIOS, and also if using GateMP between HOST and remotecore.
- IPU text and data will be in the sections specified in the memory map that you provided in your bld file.

Thanks,
Angela

Hi Angela

• Page table address change works!. Thanks for your help.
• For a proper implementationm the changes should be done in the following files (Please confirm for double checking)
  
  • Linux kernel dts (Kernel)
  • U-Boot lateattach (U-Boot SPL)
  • "packages/ti/ipc/remoteproc/rsc_table_vayu_ipu.h" (TI-RTOS SDK)
  • custom bld file (TI-RTOS SDK)
• The following files should not be modified in TI-RTOS SDK (Please confirm for double checking)
  
  • mem_segment_definitionxxxxx.xs
  • ipu_ammu.cfg
• The changes enable A15-IPU1 IPC (MessageQ), but for IPU1-IPU2 IPC other regions (SRx, IPU2 CMA) should be enabled. We ll come back with this later. Right?

Thanks a lot