Part Number: AM3359
Hi All
We have a custom board utilizing an AM3359 SoC and Linux running on that hardware.
With that system we have an issue with DMTimer 2. The problem does not occur often, sometimes it takes days, weeks or even months to reproduce it, but it happens periodically. We have systems that are supposed to run 24/7 and we are losing customers because of this problem.
After a long time of investigations and tests with different patches, we think that we now understand in detail what happens and it looks like an chip internal synchronization issue inside the AM3359 SoC.
The timer operates in one-shot mode. To start a one-shot cycle the software first writes a variable value to the TCRR register and then sets the ST bit in the TCLR register to start the timer. When the timer overflows, it generates an interrupt and stops itself. After that the one-shot cycle starts all over again.
We have introduces a patch (details see below) that immediately reads back the TCLR register after the software has written the ST Bit and when our problem occurs, this bit is not set!! Because the timer is not started there will be no interrupt and the timer stalls completely. We can recover from this error situation by manually setting the ST bit in the TCLR register.
We were able to catch the register values in this error situation:
0x0 TIDR Identification Register : 1342116609
0x10 TIOCP_CFG Timer OCP Configuration : 12
0x20 IRQ_EOI Timer IRQ End-of-Interrupt : 0
0x24 IRQSTATUS_RAW Timer Status Raw : 0
0x28 IRQSTATUS Timer Status : 0
0x2C IRQENABLE_SET Timer Interrupt Enable Set : 2
0x30 IRQENABLE_CLR Timer Interrupt Enable Clear : 2
0x34 IRQWAKEEN Timer IRQ Wakeup Enable : 2
0x38 TCLR Timer Control : 0
0x3C TCRR Timer Counter : 4294852622
0x40 TLDR Timer Load : 4294727296
0x44 TTGR Timer Trigger : 4294967295
0x48 TWPS Timer Write Posting Bits : 0
0x4C TMAR Timer Match : 0
0x50 TCAR1 Timer Capture : 0
0x54 TSICR Timer Synchronous Interface Control : 4
0x58 TCAR2 Timer Capture : 0
There are several comments in the TRM and the Errata regarding synchronization issues between the DMTimer and the OCP interface but none of them explained what we are observing.
Since we are using the posted synchronization mode the software does check the TWPS register bits before a write or read access to the timer registers.
This is the patch that we were using for debugging:
Subject: [PATCH] read back timer start flag after write
---
arch/arm/plat-omap/include/plat/dmtimer.h | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/arch/arm/plat-omap/include/plat/dmtimer.h b/arch/arm/plat-omap/include/plat/dmtimer.h
index dd79f30..b021558 100644
--- a/arch/arm/plat-omap/include/plat/dmtimer.h
+++ b/arch/arm/plat-omap/include/plat/dmtimer.h
@@ -392,8 +392,13 @@ static inline void __omap_dm_timer_load_start(struct omap_dm_timer *timer,
u32 ctrl, unsigned int load,
int posted)
{
+ u32 ctrl_reg = 0;
__omap_dm_timer_write(timer, OMAP_TIMER_COUNTER_REG, load, posted);
__omap_dm_timer_write(timer, OMAP_TIMER_CTRL_REG, ctrl, posted);
+
+ ctrl_reg = __omap_dm_timer_read(timer, OMAP_TIMER_CTRL_REG, posted);
+ WARN( (!(ctrl_reg & OMAP_TIMER_CTRL_ST)) && (ctrl & OMAP_TIMER_CTRL_ST) && (timer->id == 2),
+ "__omap_dm_timer_load_start(): Timer 2 not started!!\nload=0x%x ; ctrl=0x%x ; ctrl_reg=0x%x, posted=0x%x\n", load, ctrl, ctrl_reg, posted);
}
static inline void __omap_dm_timer_int_enable(struct omap_dm_timer *timer,
--
This patch generates the following output, when the error occurs:
[361249.217197] WARNING: CPU: 0 PID: 4411 at /jenkins_data/wim-build-release/skybase/yocto_build/tmp/work-shared/skyboard-evb/kernel-source/arch/arm/plat-omap/include/plat/dmtimer.h:403 omap2_gp_timer_set_next_event+0xb4/0xcc
[361249.217205] __omap_dm_timer_load_start(): Timer 2 not started!!
[361249.217205] load=0xfffe400e ; ctrl=0x1 ; ctrl_reg=0x0, posted=0x1
[361249.217209] Modules linked in: skyboard_asyncevent(O) skyboard_fpga(O) uio jiffies(O) ti_am335x_adc kfifo_buf industrialio spidev rtc_ds1307 ti_am335x_tscadc
[361249.217253] CPU: 0 PID: 4411 Comm: pidof Tainted: G W O 4.10.17-yocto-standard #1
[361249.217257] Hardware name: Generic AM33XX (Flattened Device Tree)
[361249.217289] [<c010fb64>] (unwind_backtrace) from [<c010c848>] (show_stack+0x20/0x24)
[361249.217305] [<c010c848>] (show_stack) from [<c039cdd8>] (dump_stack+0x20/0x28)
[361249.217323] [<c039cdd8>] (dump_stack) from [<c012e52c>] (__warn+0xe0/0x10c)
[361249.217337] [<c012e52c>] (__warn) from [<c012e5a0>] (warn_slowpath_fmt+0x48/0x50)
[361249.217349] [<c012e5a0>] (warn_slowpath_fmt) from [<c011aba8>] (omap2_gp_timer_set_next_event+0xb4/0xcc)
[361249.217371] [<c011aba8>] (omap2_gp_timer_set_next_event) from [<c017ef14>] (clockevents_program_event+0x12c/0x160)
[361249.217387] [<c017ef14>] (clockevents_program_event) from [<c017fcd0>] (tick_program_event+0x88/0x94)
[361249.217406] [<c017fcd0>] (tick_program_event) from [<c0172e40>] (hrtimer_interrupt+0xa4/0x188)
[361249.217420] [<c0172e40>] (hrtimer_interrupt) from [<c011aab8>] (omap2_gp_timer_interrupt+0x30/0x40)
[361249.217434] [<c011aab8>] (omap2_gp_timer_interrupt) from [<c016651c>] (__handle_irq_event_percpu+0x88/0x1b4)
[361249.217446] [<c016651c>] (__handle_irq_event_percpu) from [<c0166674>] (handle_irq_event_percpu+0x2c/0x68)
[361249.217457] [<c0166674>] (handle_irq_event_percpu) from [<c01666e8>] (handle_irq_event+0x38/0x4c)
[361249.217472] [<c01666e8>] (handle_irq_event) from [<c0169880>] (handle_level_irq+0xd0/0x104)
[361249.217496] [<c0169880>] (handle_level_irq) from [<c0165960>] (generic_handle_irq+0x28/0x38)
[361249.217512] [<c0165960>] (generic_handle_irq) from [<c0165e6c>] (__handle_domain_irq+0x90/0xb0)
[361249.217527] [<c0165e6c>] (__handle_domain_irq) from [<c010143c>] (omap_intc_handle_irq+0x88/0xa0)
[361249.217545] [<c010143c>] (omap_intc_handle_irq) from [<c068830c>] (__irq_svc+0x6c/0x90)
[361249.217552] Exception stack(0xcdf29ed0 to 0xcdf29f18)
[361249.217561] 9ec0: ccaaf840 00000000 ccaaf840 c023f3a0
[361249.217573] 9ee0: ccaaf840 cd50e890 cdd91190 cd501b28 c01080e8 cdf28000 00000000 cdf29f5c
[361249.217583] 9f00: cdf29f60 cdf29f20 c023f194 c0684554 a0030013 ffffffff
[361249.217608] [<c068830c>] (__irq_svc) from [<c0684554>] (_cond_resched+0x0/0x50)
[361249.217626] [<c0684554>] (_cond_resched) from [<c023f3b8>] (____fput+0x18/0x1c)
[361249.217646] [<c023f3b8>] (____fput) from [<c01477a0>] (task_work_run+0x88/0x9c)
[361249.217671] [<c01477a0>] (task_work_run) from [<c010c0ac>] (do_work_pending+0xb4/0xd0)
[361249.217684] [<c010c0ac>] (do_work_pending) from [<c0107f34>] (slow_work_pending+0xc/0x20)
[361249.217692] ---[ end trace 9a96982c011d5553 ]
