How to activate IPC late attach feature

Rabindra Paikaray said:

<6> remoteproc0: ipu1 is available
<3> remoteproc0: failed to load dra7-ipu1-fw.xem4

The most likely cause of this is that the above file is not present in /lib/firmware.  The same file that is loaded by u-boot also needs to be present in /lib/firmware.  Please check that you don't have dra7-ipu1-fw.xem4 there, and if not then create a copy of the file that is loaded by u-boot (should be the same name but in a different place).

Rabindra Paikaray said:

Want to know where it has failed exactly. Is there any way to know from loaded xem4 is running or not? How to debug from here to find what goes wrong here?

You could always try connecting to the IPU1 core through CCS to see if it's running or not.

Other than CCS, there is debugfs content from remoteproc:
    % mount -t debugfs none /debug
    % cd /debug/remoteproc/remoteproc0
and you can check the files 'trace0' (Logger output from SYS/BIOS on IPU1) and 'state' (running, stopped, etc.), although the debugfs framework isn't created until a successful connection is made.

Regards,

- Rob

Hi Rob,

Thanks for your answer. We are using an Android environment . Yes it is true dra7-ipu1-fw.xem4 not available on the Android location /vendor/firmware/.  My question here is if the xem4 is loaded from u-boot , why do we need it again to place the same dra7-ipu1-fw.xem4 file in /vendor/firmware/ ? Is the image started running after getting loaded from u-boot or it is still waiting for Android to come to run. 

After putting dra7-ipu1-fw.xem4 under /vendor/firmware/ directory I can see below trace logs.

[0][ 0.000] 18 Resource entries at 0x3000
[0][ 0.000] [t=0x007a19cd] xdc.runtime.Main: --> main:
[0][ 0.000] registering rpmsg-proto:rpmsg-proto service on 61 with HOST
[0][ 0.000] [t=0x1de56c5f] xdc.runtime.Main: --> smain:
[0][ 0.000] [t=0x1de7098b] Server: Server_create: Slave is ready
[0][ 0.000] [t=0x1de79f8b] Server: <-- Server_create: 0
[0][ 0.000] [t=0x1de81645] Server: --> Server_exec:

But when I am trying to run the messageQApp built earlier in ipc_3_20_00_06. Slave is not responding at all.

LAD server logs are attached .

5556.LAD_Logs.txt

Few more updates regarding the issue.

Built the xem4 and Android messageQApp from the latest Android release 6AJ.1.3 .

Put messageq_single.xem4 as dra7-ipu1-fw.xem4 in for u-boot and /vendor/firmware.

I can see IPU1 slave is coming up with  below logs attached:

1524.IPU1_logs.txt

Started running lad_dra7xx and then messageQApp. It fails with below logs.

shell@gminfo3:/ # messageQApp.latest 2 2
Ipc_start: NameServer_setup() failed: -1
Using numLoops: 2; procId : 2
Ipc_start failed: status = -1

Any idea what could be the reason.

Thanks,

Rabi

Rabindra Paikaray said:

Yes it is true dra7-ipu1-fw.xem4 not available on the Android location /vendor/firmware/.  My question here is if the xem4 is loaded from u-boot , why do we need it again to place the same dra7-ipu1-fw.xem4 file in /vendor/firmware/ ? Is the image started running after getting loaded from u-boot or it is still waiting for Android to come to run.

Thanks for finding the actual location of /vendor/firmware (not /lib/firmware as I suggested).  I'm still learning the Android packaging.

For late-attach mode, the Linux remoteproc kernel module needs to examine the firmware binary in order to know where its sections are loaded/located, so it can know the resource table and the location of the trace buffer, among other things.

The firmware image is certainly started running by u-boot.  There is actually one component of the firmware that doesn't activate until the remoteproc runs during Linux boot - the VirtQueue component.  This component contains a "sync" point that isn't completed until remoteproc runs.  The Ipc tests & examples configure a SYS/BIOS "startup" function that contains this sync point, and being a "startup" function it is run before Tasks/Swis/Hwis are enabled, effectively holding off all those thread levels.  In the "early-boot/late-attach" model, it is advised to place this "sync" call at the beginning of a low priority Task, so that other higher priority Tasks (and Swis and Hwis) can run before remoteproc comes up during Linux boot.  Let me know if you want some direction there.

Rabindra Paikaray said:

[0][ 0.000] [t=0x1de7098b] Server: Server_create: Slave is ready
[0][ 0.000] [t=0x1de79f8b] Server: <-- Server_create: 0
[0][ 0.000] [t=0x1de81645] Server: --> Server_exec:

But when I am trying to run the messageQApp built earlier in ipc_3_20_00_06. Slave is not responding at all

The trace from the firmware indicates you are not running messageq_single.xem4, which is required for MessageQApp.  I assume you've already figured this out, based on the content in your next post (to which I will respond shortly, but I wanted the Forum database to contain this info).

Regards,

- Rob

Rabindra Paikaray said:

I can see IPU1 slave is coming up with  below logs attached:

1524.IPU1_logs.txt

Yep, that output looks normal for the messageq_single.xem4 running on IPU1.

Rabindra Paikaray said:

Started running lad_dra7xx and then messageQApp. It fails with below logs.

shell@gminfo3:/ # messageQApp.latest 2 2
Ipc_start: NameServer_setup() failed: -1
Using numLoops: 2; procId : 2
Ipc_start failed: status = -1

Would you be able to show the LAD logging?  There are many reasons for NameServer_setup() returning -1, so I'd like to see further output from LAD.

Regards,

- Rob

shell@gminfo3:/ # messageQApp.latest 2 2

Ipc_start: NameServer_setup() failed: -1
Using numLoops: 2; procId : 2
Ipc_start failed: status = -1

Would you be able to show the LAD logging?  There are many reasons for NameServer_setup() returning -1, so I'd like to see further output from LAD.

Please check the logs from LAD.

4276.LAD_Logs.txt

Thanks,

Rabi

Want to update one more thing here. When I am switching back to old Kernel with remoteproc loading option. messageQ application able to connect and communicating with SLAVE IPU1 successfully. Where as same set of .xem4 messageQApp and all required libraries through this error with Late Attach feature Linux Kernel.

Any idea what could be the reason here?

Thanks,

Rabi

Nothing jumps to mind here.

Would you be able to attach your messageq_single.xem4 and Linux kernel boot log statements?  (maybe requires doing so in a zip file).

Where do you get your kernel and u-boot?

So, to summarize what I understand ... you're able to reconfigure the kernel to *not* do late attach on IPU1 and you rebuilt u-boot to *not* early boot IPU1, and MessageQApp/messageq_single.xem4 test works well.

Regards,

- Rob

Yes I am attaching here below list of things in a tar file:

1. Kernel boot logs.

2. Tracebuf logs for IPU1 Slave 

3. messageq_single.xem4 striped and unstripped one.

4. Kernel configuration.

5. Logs file also described steps followed to get Android application and u-boot branches merged for Late Attach feature.

3630.TI_E2E_Shared_Logs.tar.gz

We got the 6AJ.1.3 release from TI applied the required patches from TI commits to our customized u-boot and kernel. 

One more observation I am getting recently is :

[t=0x19a2e02b] ti.ipc.family.vayu.VirtQueue: ERROR: "VirtQueue.c", line 296: 

This above error coming consistently after first /second boot of the board. 

I have never seen anykind of connectivity between A15 and IPU1. Earlier while debugging connection fail we found out NameServer timeout was a problem. Increased the same but still not even a single communication happens between HLOS and SLAVE IPU1.

No I was trying with 6AJ.1.2 Kernel and u-boot without having any Late Attach feature enabled. It has the remoteproc and rpmesg functionality enabled with it. Keeping same Android application which is Built under 6AJ.1.3 works fine and messageq_single able to communicate with each other.

Hope I will get some pointer at least to move forward.

Thanks,

Rabi

Rabi,

Could you try an experimental change to u-boot?

In the file <u-boot>/common/cmd_elf.c:
    - in function handle_vdev, add memset after alloc_mem
        pa = alloc_mem(RPMSG_TOTAL_BUF_SPACE, 6);
        memset(pa, 0, RPMSG_TOTAL_BUF_SPACE);
    -  in function alloc_vring, add memset after alloc_mem:
        pa = alloc_mem(size, order);
        memset(pa, 0, size);

Let me know how that goes.

Regards,

- Rob