Linux/DRA75: How to Validate Late Attach Feature in Linux Kernel

https://e2e.ti.com/cfs-file/__key/communityserver-discussions-components-files/791/dtb.7zHi Shravan,

I am attaching dtb file we are using.

Regards,

Sreeju

Hi Sreeju,

Are you building the IPU firmware using Vision-SDK? If yes, you need to late-attach 9(which I notice you're doing), but also set the timer attribute in ipu1 to timer9 and timer11 in the DTB.

Can you also provide the dmesg logs?

Regards
Shravan

dmesg_lateattach_logs.txtHi Shravan,

Yes, building IPU1 firmware using Vision-SDK.

We will set timer attribute of IPU1 and check

dmesg log attached

Regards,

Sreeju

Hi Shravan,

We have tried with the timer9 and timer11 in the dts for ipu1, still the state indicates offline.

Regards,
Sreeju

Hi Sreeju,

Can you confirm your /lib/firmware folder has the dra7-ipu1-fw.xem4 file? This file should match the same firmware being loaded from bootloader (but shouldn't be compressed). Also are the states of all all remoteprocs offline (i.e cat /sys/kernel/debug/remoteproc/*/state is offline for all?). Can you please output the contents of the /lib/firmware contents of your SD card.

Regards
Shravan

Hi Shravan,

Please find the inline reply below:

1. Can you confirm your /lib/firmware folder has the dra7-ipu1-fw.xem4 file? 

Yes, we have the dra7-ipu1-fw.xem4 binary present in /lib/firmware.

root@mmt2020-a880:~# ls /lib/firmware/

bcm89335          dra7-ipu2-fw.xem4.map
dra7-ipu1-fw.xem4         dra7w.bgz
dra7-ipu1-fw_old.xem4      mv8887
dra7-ipu2-fw.xem4      mxt540_DM8.cfg
dra7-ipu2-fw.xem4.ipumm-fw      mxt641T_SDM8.cfg

2. Also are the states of all all remoteprocs offline (i.e cat /sys/kernel/debug/remoteproc/*/state is offline for all?

Yes,all the remoteproc states are shown offline

root@mmt2020-a880:~# cat /sys/kernel/debug/remoteproc/*/state

offline (0)

offline (0)

 

 Regards,

Sneha

Hi Sneha,

Couple of observations / comments:

1. In your /lib/firmware folder I don't see DSP firmware, however, from the logs Sreeju sent, I notice that there's an DSP MDMA fault. Are the logs provided by Sreeju (earlier in this thread) consistent with your current observations?

2. From Sreeju's logs it looks to me like the user-space firmware load fails. This is why you're seeing both remote-cores to be offline. Can you attach the output of the below command run on your target (attach config.txt file):

zcat /proc/config.gz  > config.txt

3. Can you revert this patch '0004-config_fragments-Make-remoteproc-built-in-enable-use.patch' and try again?

Regards

Shravan

config_remotecore_module.txtconfig_remotecore_inbuilt.txt

Hi Shravan,

Please find my observations below:

1. In your /lib/firmware folder I don't see DSP firmware, however, from the logs Sreeju sent, I notice that there's an DSP MDMA fault. Are the logs provided by Sreeju (earlier in this thread) consistent with your current observations?

The logs shared by Sreeju is consistent with my observations,and we aren't loading the DSP firmware  from /lib/firmware.

2. From Sreeju's logs it looks to me like the user-space firmware load fails. This is why you're seeing both remote-cores to be offline. Can you attach the output of the below command run on your target (attach config.txt file):

Actually according to this patch "0004-config_fragments-Make-remoteproc-built-in-enable-use.patch", we had made the remotecore built-in.

But as suggested by you,I  have reverted the patch.Attaching the previous(config_remotecore_inbuilt) and the reverted(config_remotecore_module) kernel configs.

As the Remotecore drivers are now loaded as modules,I observed that while booting its loading the firmware again.Is this behavior expected in case of Early boot - Late attach, since it was already loaded by bootloader? And why do we need to load it again?

[    1.392830]  remoteproc0: powering up 58820000.ipu
[    1.392845]  remoteproc0: Booting fw image dra7-ipu1-fw.xem4, size 21731188

Regards,

Sneha

Hello Shravan,


Below the terminal output of /sbin/vsdkapps

[HOST] OSA: MEM: 0: Mapped 0xa0100000 to 0xb68d7000 of size 0x00100000
[HOST] OSA: MEM: 1: Mapped 0x84203000 to 0xb28d7000 of size 0x04000000
[HOST] OSA: MEM: 2: Mapped 0xa0200000 to 0xb2817000 of size 0x00040000
[HOST] OSA: MEM: 3: Mapped 0xa02c0000 to 0xb2857000 of size 0x00080000
[HOST] OSA: MEM: 4: Mapped 0x00000000 to 0x00000000 of size 0x00000000
[HOST] OSA: IPU1-0 Remote Log Shared Memory @ 0xa0200000
[HOST] OSA: HOST Remote Log Shared Memory @ 0xa024f140
ASSERT (system_ipc.c|System_ipcInit|119)
[HOST] [HOST ] 22.606905 s: SYSTEM: System A15 Init in progress !!!
[HOST] [HOST ] 22.607027 s: SYSTEM: IPC: Init in progress !!!
[HOST] [HOST ] 22.607057 s: SYSTEM: IPC: Notify init in progress !!!
[HOST] [HOST ] 22.607118 s: SYSTEM: IPC: [IPU1-0] socket bind failed (Invalid argument, 22) !!!
[HOST] [HOST ] 22.607149 s: SYSTEM: IPC: [IPU1-0] Notify RX channel create failed (endpoint = 81) !!!
[HOST] [HOST ] 22.602848 s: Enter 'e' to exit


Regards,
Sreeju

Hi Sreeju,

The error (ASSERT (system_ipc.c|System_ipcInit|119)), could mean the memcache module isn't inserted. Can you run lsmod on your target after running vision_sdk_load.sh. Can you also provide the dmesg logs after reverting the patch mentioned.

Regards
Shravan

dmesg_lateattch_modules.txt

Hi  Shravan,

memcache module is inserted.

below the lsmod output.

root@mmt2020-a880:~# lsmod

Module                  Size  Used by

vfat                    9929  1

fat                    56346  1 vfat

sd_mod                 29404  2

sd8xxx                460729  0

cfg80211              475989  1 sd8xxx

mlan                  400565  1 sd8xxx

memcache                3386  2

pvrsrvkm              420510  2

cryptoloop              2331  0

loop                   17903  1 cryptoloop

ti_fpd3_serdes         12665  0

smsc95xx               24465  0

smsc75xx               30894  0

asix                   29524  0

usbnet                 32885  3 asix,smsc75xx,smsc95xx

mii                     4959  4 asix,usbnet,smsc75xx,smsc95xx

usb_storage            48552  1

scsi_mod              154529  2 usb_storage,sd_mod

rpmsg_proto             8283  4

omap_remoteproc        14935  2

rpmsg_rpc              29517  0

virtio_rpmsg_bus       14717  2 rpmsg_rpc,rpmsg_proto

remoteproc             32601  4 omap_remoteproc,virtio_rpmsg_bus,rpmsg_rpc,rpmsg_proto

virtio                  6417  2 remoteproc,virtio_rpmsg_bus

virtio_ring            13423  2 remoteproc,virtio_rpmsg_bus

dwc3                   86589  0

udc_core               13976  1 dwc3

dwc3_omap               4636  0

xhci_plat_hcd           5102  0

xhci_hcd              175106  1 xhci_plat_hcd

usbcore               214433  7 asix,usb_storage,xhci_plat_hcd,usbnet,smsc75xx,smsc95xx,xhci_hcd

usb_common              4358  3 dwc3,udc_core,usbcore

Attaching dmesg log

Regards,

Sreeju

Hello Shravan,

We also tried again with remoteproc built in, but loaded firmware via sysfs

echo 1 > /sys/class/firmware/dra7-ipu1-fw.xem4/loading
cat /lib/firmware/dra7-ipu1-fw.xem4 > /sys/class/firmware/dra7-ipu1-fw.xem4/data
echo 0 > /sys/class/firmware/dra7-ipu1-fw.xem4/loading

and couldn't see any ipc error with vsdkapps, below the log:

root@mmt2020-a880:~# /sbin/vsdkapps
[HOST] OSA: MEM: 0: Mapped 0xa0100000 to 0xb68b7000 of size 0x00100000
[HOST] OSA: MEM: 1: Mapped 0x84203000 to 0xb28b7000 of size 0x04000000
[HOST] OSA: MEM: 2: Mapped 0xa0200000 to 0xb27f7000 of size 0x00040000
[HOST] OSA: MEM: 3: Mapped 0xa02c0000 to 0xb2837000 of size 0x00080000
[HOST] OSA: MEM: 4: Mapped 0x00000000 to 0x00000000 of size 0x00000000
[HOST] OSA: IPU1-0 Remote Log Shared Memory @ 0xa0200000
[HOST] OSA: HOST Remote Log Shared Memory @ 0xa024f140
[HOST] [HOST ] 110.173122 s: DISP_DIST_SRC_6 : Init done - linkid-0x2000062
[HOST] [HOST ] 110.173214 s: DISP_DIST_SRC_7 : Init done - linkid-0x2000063
[HOST] [HOST ] 110.173366 s: ALGORITHM_0 : Init done - linkid-0x2000031
[HOST] [HOST ] 110.173458 s: ALGORITHM_1 : Init done - linkid-0x2000032
[HOST] [HOST ] 110.173580 s: ALGORITHM_2 : Init done - linkid-0x2000033
[HOST] [HOST ] 110.173702 s: ALGORITHM_3 : Init done - linkid-0x2000034
[HOST] [HOST ] 110.173793 s: ALGORITHM_4 : Init done - linkid-0x2000035
[HOST] [HOST ] 110.173915 s: ALGORITHM_5 : Init done - linkid-0x2000036
[HOST] [HOST ] 110.174037 s: ALGORITHM_6 : Init done - linkid-0x2000037
[HOST] [HOST ] 110.174159 s: ALGORITHM_7 : Init done - linkid-0x2000038
[HOST] [HOST ] 110.174312 s: DUP0 : Init done - linkid-0x0
[HOST] [HOST ] 110.174495 s: DUP1 : Init done - linkid-0x0
[HOST] [HOST ] 110.182608 s: NETWORK_CTRL: Starting Server (port=5000) !!!
[HOST] [HOST ] 110.182699 s: NETWORK_CTRL: Starting Server ... DONE (port=5000) !!!
[HOST] [HOST ] 110.183004 s: Enter 'e' to exit
[HOST] [HOST ] 110.213383 s: IPC_OUT_0 : Create in progress !!!

Regards,
Sreeju

Hello Shravan,

We have used the debugger and step through the code as you suggested and found the _enable () for "dss_core" is causing the display off. trace is below.

Display is going off at "_disable_optional_clocks()"

Below the call traces.

_disable_optional_clocks
_enable_sysc
_enable
_setup_reset
_setup
_omap_hwmod_for_each
_omap_hwmod_setup_all

We also compared the file : omap_hwmod_7xx_data.c, found the flag HWMOD_CONTROL_OPT_CLKS_IN_RESET setting for dss in our case. Removed this flag according to the vision sdk change. Now display is not going off.

Still we are unable to communicate with IPU1 (vsdkapps log output is same as before.)
We also tried with modetest, there is no error reported, but no pattern got displayed on the screen.

Can we look in to IPU1, DSS, VPE registers, or configuration specifically that enables A15 to IPU1 communication. ?


Thanks,
Sreeju

Hello Shravan,

Late attach is working with timer3 for IPU1, We have issue, when we are changing timer attribute to timer9 for IPU1, It is not working when we load both IPU1 and IPU2, we are unable to display in this case(it is working when we load IPU1 alone with timer9). Below the currently assigned timer attributes in dts for IPUs

&ipu2 {
status = "okay";
memory-region = <&ipu2_cma_pool>;
mboxes = <&mailbox6 &mbox_ipu2_ipc3x>;
timers = <&timer3>;
/* watchdog-timers = <&timer4>, <&timer9>; */
};

&ipu1 {
status = "okay";
memory-region = <&ipu1_cma_pool>;
mboxes = <&mailbox5 &mbox_ipu1_ipc3x>;
timers = <&timer9>, <&timer11>;
/* watchdog-timers = <&timer7>, <&timer8>; */e2e.ti.com/.../777231
};

Suspect, there could be some conflict between IPU1 and IPU2, when timer9 is used for IPU1. even though timer 9 is not mentioned for IPU2 in dts.
How we can confirm that IPU2 will not be using timer9 ?, How we can change timer attribute and rebuild the IPU2 firmware ?

Regards,
Sreeju

Hello Shravan,

Yes IPU1 firmware is loading from the Bootloader. If I load IPU1 alone, there is no issue, it is working fine.
Issue is only with below case:
IPU1 with timer9 load from bootloader, and IPU2 from /lib/firmware (no late attach for IPU2 and its attributes)

Yes I will create a new thread for this.

Regards,
Sreeju

Created new thread: e2e.ti.com/.../786499