Linux/AM5716: IPU timer access issues when Linux is running

Oleg Semyonov said:

Timers are configured in Linux device tree as no idle (otherwise they can't be initialized from M4 core being in suspended state):

&timer4 {
        ti,no-idle;
        ti,timer-alwon;
};

Please configure as follows:

&timer4 {
    status = "disabled";
    ti,no-idle;
};

If the issue still persists, please use devmem2 to read the value of the CM_L4PER3_CLKSTCTRL register (address 0x4A00 9910):

devmem2 0x4A009910

Best regards,
Brad

I tested this solution on both latest kernel and filesystem from PDK Linux RT 06.00.00.07 with am572x-idk.dtb, and also with my own kernel/dtb/filesystem build.

If I modify timer4 as you suggested, it works, devmem 0x4A009910 returns 0x00000403.

If I disable IPU1 (set status = "disabled" in device tree) or remove the demo firmware dra7-ipu1-fw.xem4.opencl-monitor from /lib/firmware, the IPU1 remote processor does not start, and timer4 does not work. Its IDLEST=2, and devmem 0x4A009910 reads 0x00000003.

I can't guess why IPU1 since this timer is reserved by IPU2. I can disable IPU2 without issues, but IPU1 is the key. In the end I need to boot IPU before Linux starts, and do not use remoteproc API for it. So Linux must never touch the IPU and timer4 at all. But for now timer4 doesn't work without IPU1 remote processor running.

Can you try writing CM_L4PER3_CLKSTCTRL = 2 at run-time like this:

devmem2 0x4A009910 w 2

That will force CM_L4PER3 to SW_WKUP state which I suspect will fix your issue.  (Though watch out for Linux changing it back!  We would ultimately need a kernel change to do this sort of thing.)

We may need to repeat your experiment one more time looking at one other register.  There are two clocks related to Timer4, and 2 registers associated with those clocks.

• Functional clock : TIMER4_GFCLK : CM_L4PER_TIMER4_CLKCTRL (0x4A00 9748)
• Interface clock: L4PER_L3_GICLK : CM_L4PER3_CLKSTCTRL (0x4A00 9910)

In particular, I'm interested in knowing the value of both of these registers with respect to any experiments that pass or fail.

Oleg Semyonov said:

static void TimerPRCMConfigure(void)

{

    HW_WR_REG32(SOC_L4PER_CM_CORE_BASE + CM_L4PER_TIMER4_CLKCTRL, 0x2);

    while ((HW_RD_REG32(SOC_L4PER_CM_CORE_BASE + CM_L4PER_TIMER4_CLKCTRL) & (0x00030000)) != 0x0) ;

}

I'm also wondering whether this code is correct.  In particular, it is waiting for the module to become "fully functional".  I'm not aware of any specific issues related to allowing that interface clock to idle.  In other words, it may be sufficient to simply change this while loop above.  For example:

    while ((HW_RD_REG32(SOC_L4PER_CM_CORE_BASE + CM_L4PER_TIMER4_CLKCTRL) & (0x00010000)) != 0x0) ;

With this small change an IDLEST value of 0 or 2 would be sufficient for the timer. 

Oleg Semyonov said:

If I disable IPU1 (set status = "disabled" in device tree) or remove the demo firmware dra7-ipu1-fw.xem4.opencl-monitor from /lib/firmware, the IPU1 remote processor does not start, and timer4 does not work.

IPU1 reserves Timer11 and Timer14 in the kernel.  Timer14 has the same interface clock as Timer4 (L4PER_L3_GICLK).  This whole issue seems to revolve around the interface clock so I believe that's the connection.

By the way, this is a lot of info I just threw at you.  I recommend trying the code change first.  We don't want to make a bunch of kernel changes if in fact everything is working fine with the interface clock idled.

Thanks for all the info. I will try everything and report results ASAP.

Here are some results. Unfortunately, with IPU1 disabled we were not able to make timer4 interrupts usable in IPU with Linux running.

TEST 1: IPU1 enabled (demo firmware started by remoteproc API on Linux boot), Timer4 disabled (status=disabled and ti,no-idle)

CM_L4PER3_CLKSTCTRL (before): 0x0403
CM_L4PER3_CLKSTCTRL. Write 0x2
CM_L4PER3_CLKSTCTRL (after): 0x0502

CM_L4PER_TIMER4_CLKCTRL (before): 0x0002
CM_L4PER_TIMER4_CLKCTRL. Write 0x2
CM_L4PER_TIMER4_CLKCTRL (after): 0x0002

IPU1 restarted using own firmware. It works at first glance, but hangs. Looks like there are some conflicts with Linux IPU subsystem and own code. Actually, the same as with default device tree with both IPU and Timer4 enabled.

TEST 2: IPU1 disabled (status = disabled), Timer4 default to enable (no status)

CM_L4PER3_CLKSTCTRL (before): 0x0003
CM_L4PER3_CLKSTCTRL. Write 0x2
CM_L4PER3_CLKSTCTRL (after): 0x0102

CM_L4PER_TIMER4_CLKCTRL (before): 0x30000
CM_L4PER_TIMER4_CLKCTRL. Write 0x2
CM_L4PER_TIMER4_CLKCTRL (after): 0x30002

Timer4 can't be started.

TEST 3: IPU1 disabled (status = disabled), Timer4 disabled (status = disabled and ti,no-idle)

CM_L4PER3_CLKSTCTRL (before): 0x0003
CM_L4PER3_CLKSTCTRL. Write 0x2
CM_L4PER3_CLKSTCTRL (after): 0x0102

CM_L4PER_TIMER4_CLKCTRL (before): 0x20002
CM_L4PER_TIMER4_CLKCTRL. Write 0x2
CM_L4PER_TIMER4_CLKCTRL (after): 0x20002

Firmware running on IPU1 core has one-time timer interrupt (overflow for this test), but no interrupt source flags bit is set (all zeroes). After exiting ISR no more timer interrupts happen. But they start happen when the main loop on M4 core reads any timer register (does not matter which one - either status register or value). When any register is read, interrupt happens, but approximately every 2nd time it has interrupt source bits set to zero (and each other time they are correct according to interrupt reason). Looks like something steals interrupts and process them before M4 core.

ISR code reads timer registers and clears flags, so it is not the explanation why some interrupts have zero flags, and why interrupts happen only when main code reads any timer register.

The same M4 code works well when U-Boot is running (no Linux). The same M4 code almost work (at least, regarding interrupts) when using default DTB with IPU1 and Timer enabled. But only if Linux starts IPU1 firmware (default opencl-monitor in this case), and this conflicts with reloaded firmware for some reason (even when M4 is reset, and interrupt vector table is replaced).

So it looks like ignoring interface clock in init makes some progress, but timer still is not fully operational when Linux is running (interrupt issues). We have no idea what else should we try. Starting a dummy IPU firmware is not an option in any case. We need to boot that core from SPL and Linux must not touch it later (including clocks, power domains, etc). We are ok to patch the Linux kernel but need to know what happens first.

Oleg Semyonov said:

TEST 1: IPU1 enabled (demo firmware started by remoteproc API on Linux boot), Timer4 disabled (status=disabled and ti,no-idle)

CM_L4PER3_CLKSTCTRL (before): 0x0403
CM_L4PER3_CLKSTCTRL. Write 0x2
CM_L4PER3_CLKSTCTRL (after): 0x0502

CM_L4PER_TIMER4_CLKCTRL (before): 0x0002
CM_L4PER_TIMER4_CLKCTRL. Write 0x2
CM_L4PER_TIMER4_CLKCTRL (after): 0x0002

IPU1 restarted using own firmware. It works at first glance, but hangs. Looks like there are some conflicts with Linux IPU subsystem and own code. Actually, the same as with default device tree with both IPU and Timer4 enabled.

Did you check CM_L4PER3_CLKSTCTRL after the hang?  I wonder if perhaps Linux changed back the value and that caused the hang.

Oleg Semyonov said:

When any register is read, interrupt happens, but approximately every 2nd time it has interrupt source bits set to zero (and each other time they are correct according to interrupt reason). Looks like something steals interrupts and process them before M4 core.

This sounds very much like the auto-idling is causing major issues for this subsystem.  The test above where you write CM_L4PER3_CLKSTCTRL = 2 does exactly that, but I suspect that Linux may have changed this value and reintroduced the issue.  We may need a small change to the kernel to fix it.  Let me get back to you on that.

Here's a similar thread and associated solution:

https://e2e.ti.com/support/processors/f/791/p/746597/2779684#2779684

More specifically, I think you need to delete the following highlighted line from arch/arm/mach-omap2/clockdomains7xx_data.c:

static struct clockdomain l4per3_7xx_clkdm = {
.name = "l4per3_clkdm",
.pwrdm = { .name = "l4per_pwrdm" },
.prcm_partition = DRA7XX_CM_CORE_PARTITION,
.cm_inst = DRA7XX_CM_CORE_L4PER_INST,
.clkdm_offs = DRA7XX_CM_CORE_L4PER_L4PER3_CDOFFS,
.dep_bit = DRA7XX_L4PER3_STATDEP_SHIFT,
.flags = CLKDM_CAN_HWSUP_SWSUP,
};

You may still need to perform a write to CM_L4PER3_CLKSTCTRL = 2, but with the above line deleted you should not have Linux try to change it (assuming that's your underlying issue).

Thanks for a lot of info. Unfortunately, the suggested solution hasn't helped.

With IPU1 disabled, it has exactly same behaviour with CLKDM_CAN_HWSUP_SWSUP set or missing. All registers have the same value as set, and Linux did not try to change them. Still no timer working or unreliable interrupts.

When switched to the latest SDK 06.00.00.07, the behavior changed a bit: with IPU1 enabled (default, running opencl-monitor firmware by remoteproc API) and timer4 disabled, it now does not hang and seems to work. It is better than nothing. But in the end we need to run M4 core from SPL, so can't allow Linux to restart it as a remote processor.

Not same, but related question (maybe I should post a new thread for it): as said, we need to run IPU and DSP cores from SPL. From what I see, it is supported for K3 family of SoC, but not supported for am57xx in latest PDK. I saw that the so called feature "Early Boot - Late Attach" was supported for DRA7 family in automotive SDK branch, but wasn't merged into the latest PDK. Can anybody point me how to do the same with latest PDK U-Boot and kernel? I guess I could be able to init cores in SPL and completely ignore them in Linux. May be this will also resolve the issue above. Didn't you see a discussion thread about such feature supporting AM57xx family?

Oleg Semyonov said:

With IPU1 disabled, it has exactly same behaviour with CLKDM_CAN_HWSUP_SWSUP set or missing. All registers have the same value as set, and Linux did not try to change them. Still no timer working or unreliable interrupts.

Did you try manually writing CM_L4PER3_CLKSTCTRL = 2 in this case?  Before you mentioned that it worked for a bit and then the IPU hung.  My suspicion was that the hang occurred due to Linux changing this value back to 3.  With the change in Linux I mentioned, Linux should no longer touch this register.

Oleg Semyonov said:

Not same, but related question (maybe I should post a new thread for it): as said, we need to run IPU and DSP cores from SPL

This is being done right now specifically for the IPU.  I don't have a firm timeline, but I would say roughly by the end of this year (e.g. SDK 6.02).  DSP isn't yet committed. I expect we will need a few months before we know more.

Brad Griffis said:

Did you try manually writing CM_L4PER3_CLKSTCTRL = 2 in this case?  Before you mentioned that it worked for a bit and then the IPU hung.  My suspicion was that the hang occurred due to Linux changing this value back to 3.  With the change in Linux I mentioned, Linux should no longer touch this register.

No, it worked with IPU1 enabled after starting it as remote processor by Linux with some demo code, and then being reloaded with own firmware via CCS. Then Linux went mad dumping in the console something about data access when functional access active or like that. And M4 core's PC in CCS went out of code range. So it was not a clock stop. In the same case all is fine with the latest  kernel: M4 reloaded works, gets interrupts and Linux feels good with that. But, again, it is with IPU1 enabled and started by Linux first.

But it never worked when IPU1 was disabled in device tree and started from CCS. Either timer could not be inited (30002), or it was partially enabled (20002) but never generated interrupts on M4 core regardless of status of those registers and .flags value in kernel. UNLESS we read from any timer register in the main loop, and then interrupt happened. Writing 0x2 did not help since all registers were already correct.

So for now we can debug using the first approach (reloading IPU core after Linux start), but we still can't have IPU started without Linux remoteproc first and have it completely isolated (removed or disabled in the device tree).

Sorry, it seems like we are so close to having this figured out, but I'm still not seeing it...  Can you please try this:

1. While IPU2 is "happy" please execute this command from the Linux command line "omapconf export ctt good.rd1".
2. In the case when IPU1 is disabled and IPU2 won't work, please run "omapconf export ctt bad.rd1".

Please zip the files and attach.  

By the way, I was reviewing this thread some more and wanted to understand why you're using Timer4 specifically.  IPU2 is expected to use Timer3 for its main timer (i.e. for the RTOS scheduler).  Timer4 is configured as a watchdog timer in the dts by default.

Regarding timers used: it was an arbitrary choice. We can use any timers because we do not use IPU remoteproc feature or RTOS on IPU and wish to run them as early as possible without Linux involved. As far as I understand, early start for IPU is planned for coming releases, hope to see the same for DSP as well. Maybe then other issues will also be resolved.

Sorry, missed the request with omapconf. We used own boards and own filesystem builds. So I have to restore original SDK to IDK and provide the info requested after I reproduce the issue there.

I recommend switching to Timer3 for use with IPU2.  There is a lot of coordination to have everything work properly, and Linux is expecting Timer3 to be used for IPU2 scheduler.

I suggested omapconf because I thought you would have it handy.  Alternatively there's a JTAG script that is OS-independent:

http://git.ti.com/sitara-dss-files/am57xx-dss-files/blobs/raw/master/am57xx-ctt.dss

Directions on how to use the script are available here:

http://git.ti.com/sitara-dss-files/am57xx-dss-files/blobs/master/README

Please switch over to Timer3 (Clock.timerId=2) first though.  

Did I miss the point?

The goal is to use both IPU1, IPU2 and some timers interrupts routed to IPU cores, booted as early as possible (from SPL). Linux must not touch anything of them (no clock changes, no power domain switches, no IPU schedulers, etc).

We can configure everything from SPL, but Linux tries to control them and conflicts with them. We are looking for a method to disable any impact of Linux on both IPUs and timer operations. Preferably, without patching the driver code if possible.

Currently (with Linux running) timer4 interrupts routed to IPU1 core only happen if IPU2 core is started by Linux (because timer 4 uses same clocks as timer used by IPU2 with default device tree). Of course, another timer can be used, but again - this may work since Linux is expecting something about IPU and timers. But I want to make Linux know nothing about both IPU1, IPU2 and few timers, do not disable timer clocks, and do not suspend IPUs and timers.

Oleg Semyonov said:

We can configure everything from SPL, but Linux tries to control them and conflicts with them. We are looking for a method to disable any impact of Linux on both IPUs and timer operations. Preferably, without patching the driver code if possible.

What you're describing is called "late attach".  That's a feature that's under development right now.  It's not as simple as just cutting these resources out of Linux.  There are shared clock domains where Linux may inadvertently interfere with the IPU cores while doing other stuff.  The more complete solution is that Linux needs to be aware that the IPU core is there and that it needs to be left alone.  I'm expecting this feature to be available for IPU at the end of the year.