This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

Linux/AM3354: Linux/AM335x: freeze/lockup

Part Number: AM3354
Other Parts Discussed in Thread: TPS65910

Tool/software: Linux

Hi, we have designed a custom board with the AM3354. Sometimes we have a freeze of 180 seconds or less in user space. It occurs after hours or days. On the AM3354 runs a linux with kernel 4.14. Everything works well after the freeze of 180 seconds. During the freeze are hardware as DCLK and Backlight PWM still working well.There are no interrupts of timer 2 within the 180 seconds (read before and after freeze from /proc/interrupts: INTC 68 Level gp_timer).

We enabled the CONFIG_DEBUG_TIMEKEEPING and CONFIG_DEBUG_PREEMPT in the kernel defconfig. After enabling these configs, we can observe the following kernel message before the freeze occurs:

[372307.567958] INFO: timekeeping: Cycle offset (4291823686) is larger than the 'timer1' clock's 50% safety margin (2147483647)
[372307.567968] timekeeping: Your kernel is still fine, but is feeling a bit nervous

Our clock source is based on an external 24MHz crystal. It seems that timer 2 generates no interrupt and the system hangs for these 3 minutes until next overflow occurs (2^32 / 24MHz = 179s). 

Is there any advise what we can do? All the rules of the design guideline should have been realized in our pcb design.

Many thanks and kind regards

  • Thanks for your post.

    Could you please tell us more about where you get and how you build your kernel and filesystem? If you are not using the ones provided with the SDK, could you try those on your board to see if you can reproduce the freeze? Or, have you tried your kernel and filesystem on a beaglebone black?

    Thanks.
  • Thank you for your prompt reply to my post.

    We build our kernel and fs with yocto (release 2.5). The built kernel is 4.14, but we have also tested kernel version 4.9 without any improvement. We tried our kernel and fs on another vendors AM3354 board but never on a beaglebone black. We couldn't reproduce this freeze on the other board.

    Thanks

  • Have you considered Advisory 1.0.30 in the AM335x Errata: http://www.ti.com/lit/er/sprz360i/sprz360i.pdf

    Can you try one of the fs from the SDK? You might also want to consider a very minimal fs to see if this is a kernel issue or a fs issue.

    I hope this is helpful to you.

  • I've tried our kernel and RootFS also with the TI am335x-evmsk and it works normally without any freeze. I never tried a built kernel from the TI SDK but I will do that in the next days.

    We have considered Advisory 1.0.30, so this should not be the problem. We have also made a redesign of the pcb layout but the error occurs in both designs.

  • tortuguita said:
    I've tried our kernel and RootFS also with the TI am335x-evmsk and it works normally without any freeze.

    '

    This points to a hardware issue with your board.  And given that the error is timer-related, the clocking is the immediate suspect.

    A good test would be to remove your crystal and see if you can provide an externally generated 24 MHz 1.8V square wave clock source into the XTALIN pin.  You need to be careful however not to be inputting this 1.8V signal into the AM335x while the AM335x is powered down.  There are a couple ways you could do this:

    1. Power up the AM335x while manually holding down the reset (if you have a button).  After power is stable, turn on your your clock source, then release the AM335x from reset.

    2. Use a buffer (e.g. on bread board, etc.) between the 1.8V square wave and the AM335x.  Power the buffer using the VDDS_OSC power rail.  That way as VDDS_OSC ramps up, so will the associated clock.

  • We made the test with an external 24MHz clock today. It still occurs...

  • Ok, well that seems to exonerate the clock. Are you using the ZCZ package? What's the voltage on VDD_CORE? What PMIC are you using?
  • We use the AM3354BZCZ80 and a PMIC TPS65910A3A1. Our VDD_Core Voltage is measured 1,1V (standard setting).
  • First, what I'm about to ask you to do is for diagnostic purposes, not as a solution... Please make this change in u-boot:

    diff --git a/board/ti/am335x/board.c b/board/ti/am335x/board.c
    index 8015012..0d4e05c 100644
    --- a/board/ti/am335x/board.c
    +++ b/board/ti/am335x/board.c
    @@ -438,7 +438,7 @@ void scale_vcores_generic(int freq)
    return;

    /* Second, update the CORE voltage. */
    - if (tps65910_voltage_update(CORE, TPS65910_OP_REG_SEL_1_1_0))
    + if (tps65910_voltage_update(CORE, TPS65910_OP_REG_SEL_1_2_0))
    return;

    }

    After making the change please verify that VDD_CORE now equals 1.2V. Then let me know whether that impacts your issue.

    Thanks,
    Brad
  • tortuguita,

    Since we haven't heard back from you in a while, we're assuming you've moved past this issue. If not, please reply back with more information or start a new post if you have a new issue. Thanks.
  • Sorry for the late answer. Now, we have solved the problem. I've seen that the DPLL's for MPU and CORE are not correct set in the first stage loader . If I set the DPLL registers, then the error does not occur anymore.

    One additional observation: If we configure a 26MHz clock source (SPL, SYSBOOT resistors and crystal), then the error does not occur, also if wen don't set the DPLL's in the first stage loader. Can you explain me this effect?

    Thanks

  • Sorry for the late answer. Now, we have solved the problem. I've seen that the DPLL's for MPU and CORE are not correct set in the first stage loader . If I set the DPLL registers, then the error does not occur anymore.

    One additional observation: If we configure a 26MHz clock source (SPL, SYSBOOT resistors and crystal), then the error does not occur, also if we don't set the DPLL's in the first stage loader. Can you explain me this effect? How does it work exactly with the CORE and MPU DPLL. If I read the CM_CLKSEL_DPLL_MPU register, then I can calculate and verify the correct MPU frequency (for example 800MHz), also if I don't set the DPLL in the first stage loader. But in the case of CORE I get a different frequency. What's the max CORE frequency? Is it the max value of the ADPLL (2GHz)?

    Thanks
  • tortuguita said:
    I've seen that the DPLL's for MPU and CORE are not correct set in the first stage loader

    What are you referring to as "first stage loader"?  Are you referring to the ROM code or u-boot-SPL?  How did you diagnose that the frequencies are incorrect?  This would surprise me greatly...

    tortuguita said:
    If I set the DPLL registers, then the error does not occur anymore.

    Which registers?  What do you set them to?  Your original description stated that this issue often occurred minutes or even hours after boot.  Was that because the clocks were wrong the entire time? 

    I have some scripts for Code Composer Studio that can help us. The general process for running these scripts is described here:

    http://git.ti.com/sitara-dss-files/am335x-dss-files/blobs/master/README

    In your particular instance, you should use this script which scrapes all clock-related registers:

    http://git.ti.com/sitara-dss-files/am335x-dss-files/blobs/raw/master/am335x-ctt.dss

    I suggest potentially running the script for several cases:

    1. At run-time when Linux is up and running in your original configuration.
    2. At run-time when Linux is up and running in your newly working configuration.
    3. After the first stage boot loader, i.e. the place where you think there's an issue.
    4. At run-time when Linux is up and running using the 26 MHz clock.

    I suggest renaming the generated files after each run to something more meaningful, e.g. am335x-linux-bad.rd1, am335x-linux-good.rd1, am335x-spl.rd1, am335x-linux-26mhz.rd1.  Please zip the files and attach them.

    tortuguita said:
    What's the max CORE frequency? Is it the max value of the ADPLL (2GHz)?

    We don't specify the max, only what we recommend.  This is given in the TRM in Table 8-22. Core PLL Typical Frequencies (MHz).

  • Hi Brad

    Brad Griffis said:
    What are you referring to as "first stage loader"?  Are you referring to the ROM code or u-boot-SPL?  How did you diagnose that the frequencies are incorrect? 

    It's the u-boot SPL or MLO. I cannot say that the frequencies are incorrect but the DPLL register CM_CLKSEL_DPLL_MPU and CM_CLKSEL_DPLL_CORE have never been set in the u-boot SPL. 

    Brad Griffis said:
    Which registers?  What do you set them to?  Your original description stated that this issue often occurred minutes or even hours after boot.  Was that because the clocks were wrong the entire time? 

    The CM_CLKSEL_DPLL_MPU and CM_CLKSEL_DPLL_CORE. Now I do the same as in the am335x-evmsk board.c file. Yes, the error occurred days, hours or minutes after boot (randomly).

    Brad Griffis said:
    I have some scripts for Code Composer Studio that can help us

    Okay, thanks. I will do this.

    Thanks & BR

  • Hi Brad

    Here the first logs. Unfortunately did not work the JTAG debugger configuration. I read back the registers with devmem2. I will try it once again tomorrow so that I can read it back in u-boot (or with the md command).

    Brad Griffis said:
    • At run-time when Linux is up and running in your original configuration.
    • At run-time when Linux is up and running in your newly working configuration.
    • After the first stage boot loader, i.e. the place where you think there's an issue.
    • At run-time when Linux is up and running using the 26 MHz clock.

    It looks that the dpll of core is set to 50MHz (OPP50). In the good configuration I've set to 1GHz.

    Thanks

    0x44e00000	 0x0022C502
    0x44e00004	 0x0000000A
    0x44e00008	 0x00000102
    0x44e0000c	 0x00000056
    0x44e00010	 0x00070000
    0x44e00014	 0x00000002
    0x44e00018	 0x00000002
    0x44e0001c	 0x00070000
    0x44e00020	 0x00070000
    0x44e00024	 0x00000002
    0x44e00028	 0x00000002
    0x44e0002c	 0x00000002
    0x44e00030	 0x00000002
    0x44e00034	 0x00030000
    0x44e00038	 0x00030000
    0x44e0003c	 0x00030000
    0x44e00040	 0x00030000
    0x44e00044	 0x00030000
    0x44e00048	 0x00030000
    0x44e0004c	 0x00030000
    0x44e00050	 0x00030000
    0x44e00054	 0x00030000
    0x44e00058	 0x00030000
    0x44e00060	 0x00000002
    0x44e00064	 0x00000002
    0x44e00068	 0x00030000
    0x44e0006c	 0x00000002
    0x44e00070	 0x00030000
    0x44e00074	 0x00000002
    0x44e00078	 0x00000002
    0x44e0007c	 0x00030000
    0x44e00080	 0x00000002
    0x44e00084	 0x00000002
    0x44e00088	 0x00030000
    0x44e0008c	 0x00030000
    0x44e00090	 0x00000002
    0x44e00094	 0x00030000
    0x44e00098	 0x00030000
    0x44e0009c	 0x00030000
    0x44e000a0	 0x00030000
    0x44e000a4	 0x00030000
    0x44e000a8	 0x00030000
    0x44e000ac	 0x00000002
    0x44e000b0	 0x00000002
    0x44e000b4	 0x00000002
    0x44e000b8	 0x00030000
    0x44e000bc	 0x00000002
    0x44e000c0	 0x00030000
    0x44e000c4	 0x00030000
    0x44e000c8	 0x00030000
    0x44e000cc	 0x00030000
    0x44e000d0	 0x00000002
    0x44e000d4	 0x00030000
    0x44e000d8	 0x00000002
    0x44e000dc	 0x00000002
    0x44e000e0	 0x00000002
    0x44e000e4	 0x00070000
    0x44e000e8	 0x00070000
    0x44e000ec	 0x00030000
    0x44e000f0	 0x00030000
    0x44e000f4	 0x00030000
    0x44e000f8	 0x00030000
    0x44e000fc	 0x00000002
    0x44e00100	 0x00000002
    0x44e00104	 0x00030000
    0x44e0010c	 0x00030000
    0x44e00110	 0x00000002
    0x44e0011c	 0x0000007A
    0x44e00120	 0x00000002
    0x44e00124	 0x00070000
    0x44e00128	 0x00030000
    0x44e0012c	 0x00000012
    0x44e00130	 0x00040002
    0x44e00134	 0x00030000
    0x44e00138	 0x00030000
    0x44e0013c	 0x00030000
    0x44e00140	 0x00000002
    0x44e00144	 0x00000012
    0x44e00148	 0x00000012
    0x44e0014c	 0x00030000
    0x44e00150	 0x00000001
    0x44e00400	 0x00007716
    0x44e00404	 0x00000002
    0x44e00408	 0x00040002
    0x44e0040c	 0x00000002
    0x44e00410	 0x00000002
    0x44e00414	 0x12510000
    0x44e00418	 0x00000009
    0x44e0041c	 0x00000000
    0x44e00420	 0x00000001
    0x44e00424	 0x00000000
    0x44e00428	 0x00000000
    0x44e0042c	 0x00006402
    0x44e00430	 0x00000000
    0x44e00434	 0x00000001
    0x44e00438	 0x00000000
    0x44e0043c	 0x00000000
    0x44e00440	 0x00019017
    0x44e00444	 0x00000000
    0x44e00448	 0x00000001
    0x44e0044c	 0x00000000
    0x44e00450	 0x00000000
    0x44e00454	 0x0001873B
    0x44e00458	 0x00000000
    0x44e0045c	 0x00000001
    0x44e00460	 0x00000000
    0x44e00464	 0x00000000
    0x44e00468	 0x00003217
    0x44e0046c	 0x00000000
    0x44e00470	 0x00000001
    0x44e00474	 0x00000000
    0x44e00478	 0x00000000
    0x44e0047c	 0x00000000
    0x44e00480	 0x00000221
    0x44e00484	 0x00000221
    0x44e00488	 0x00000007
    0x44e0048c	 0x00000007
    0x44e00490	 0x00000007
    0x44e00494	 0x00000007
    0x44e00498	 0x00000007
    0x44e0049c	 0x0403C017
    0x44e004a0	 0x00000201
    0x44e004a4	 0x00000222
    0x44e004a8	 0x00000201
    0x44e004ac	 0x00000285
    0x44e004b0	 0x00040002
    0x44e004b4	 0x00000002
    0x44e004b8	 0x00030000
    0x44e004bc	 0x00000002
    0x44e004c0	 0x00030000
    0x44e004c4	 0x00000002
    0x44e004c8	 0x00030000
    0x44e004cc	 0x00000006
    0x44e004d0	 0x00000002
    0x44e004d4	 0x00030000
    0x44e004d8	 0x00000001
    0x44e00504	 0x00000001
    0x44e00508	 0x00000001
    0x44e0050c	 0x00000001
    0x44e00510	 0x00000001
    0x44e00514	 0x00000004
    0x44e00518	 0x00000001
    0x44e0051c	 0x00000001
    0x44e00520	 0x00000000
    0x44e00528	 0x00000000
    0x44e0052c	 0x00000000
    0x44e00530	 0x00000000
    0x44e00534	 0x00000000
    0x44e00538	 0x00000001
    0x44e0053c	 0x00000000
    0x44e00600	 0x00000006
    0x44e00604	 0x00000002
    0x44e00700	 0x00000080
    0x44e00800	 0x00000002
    0x44e00804	 0x00000302
    0x44e00900	 0x00000302
    0x44e00904	 0x00000002
    0x44e00908	 0x00070000
    0x44e0090c	 0x00000002
    0x44e00910	 0x00030000
    0x44e00914	 0x00030000
    0x44e00a00	 0x00000002
    0x44e00a20	 0x00030000
    0x44e00b00	 0x00000000
    0x44e00b04	 0x00000500
    0x44e00b08	 0x00000000
    0x44e00b0c	 0x00000100
    0x44e00b10	 0x00000000
    0x44e00c00	 0x00000003
    0x44e00c04	 0x00000000
    0x44e00c08	 0x01E60007
    0x44e00c0c	 0xEE0000EB
    0x44e00d00	 0x00000000
    0x44e00d04	 0x00000008
    0x44e00d08	 0x00000000
    0x44e00d0c	 0x00000020
    0x44e00e00	 0x01FF0007
    0x44e00e04	 0x000003F7
    0x44e00e08	 0x00000000
    0x44e00f00	 0x00000000
    0x44e00f04	 0x00001006
    0x44e00f08	 0x00000000
    0x44e00f0c	 0x78000017
    0x44e00f10	 0x00000003
    0x44e00f14	 0x00000000
    0x44e00f18	 0x00000003
    0x44e00f1c	 0x00000000
    0x44e01000	 0x00000004
    0x44e01004	 0x00000000
    0x44e01100	 0x00060047
    0x44e01104	 0x00000000
    0x44e01110	 0x00000037
    0x44e01114	 0x00000001
    0x44e01200	 0x00000003
    0x44e01204	 0x00000007
    0x44e10040	 0x00400337
    
    0x44e00000	 0x0022C502
    0x44e00004	 0x0000000A
    0x44e00008	 0x00000102
    0x44e0000c	 0x00000056
    0x44e00010	 0x00070000
    0x44e00014	 0x00000002
    0x44e00018	 0x00000002
    0x44e0001c	 0x00070000
    0x44e00020	 0x00070000
    0x44e00024	 0x00000002
    0x44e00028	 0x00000002
    0x44e0002c	 0x00000002
    0x44e00030	 0x00000002
    0x44e00034	 0x00030000
    0x44e00038	 0x00030000
    0x44e0003c	 0x00030000
    0x44e00040	 0x00030000
    0x44e00044	 0x00030000
    0x44e00048	 0x00030000
    0x44e0004c	 0x00030000
    0x44e00050	 0x00030000
    0x44e00054	 0x00030000
    0x44e00058	 0x00030000
    0x44e00060	 0x00000002
    0x44e00064	 0x00000002
    0x44e00068	 0x00030000
    0x44e0006c	 0x00000002
    0x44e00070	 0x00030000
    0x44e00074	 0x00000002
    0x44e00078	 0x00000002
    0x44e0007c	 0x00030000
    0x44e00080	 0x00000002
    0x44e00084	 0x00000002
    0x44e00088	 0x00030000
    0x44e0008c	 0x00030000
    0x44e00090	 0x00000002
    0x44e00094	 0x00030000
    0x44e00098	 0x00030000
    0x44e0009c	 0x00030000
    0x44e000a0	 0x00030000
    0x44e000a4	 0x00030000
    0x44e000a8	 0x00030000
    0x44e000ac	 0x00000002
    0x44e000b0	 0x00000002
    0x44e000b4	 0x00000002
    0x44e000b8	 0x00030000
    0x44e000bc	 0x00000002
    0x44e000c0	 0x00030000
    0x44e000c4	 0x00030000
    0x44e000c8	 0x00030000
    0x44e000cc	 0x00030000
    0x44e000d0	 0x00000002
    0x44e000d4	 0x00030000
    0x44e000d8	 0x00000002
    0x44e000dc	 0x00000002
    0x44e000e0	 0x00000002
    0x44e000e4	 0x00070000
    0x44e000e8	 0x00070000
    0x44e000ec	 0x00030000
    0x44e000f0	 0x00030000
    0x44e000f4	 0x00030000
    0x44e000f8	 0x00030000
    0x44e000fc	 0x00000002
    0x44e00100	 0x00000002
    0x44e00104	 0x00030000
    0x44e0010c	 0x00030000
    0x44e00110	 0x00000002
    0x44e0011c	 0x0000007A
    0x44e00120	 0x00000002
    0x44e00124	 0x00070000
    0x44e00128	 0x00030000
    0x44e0012c	 0x00000012
    0x44e00130	 0x00040002
    0x44e00134	 0x00030000
    0x44e00138	 0x00030000
    0x44e0013c	 0x00030000
    0x44e00140	 0x00000002
    0x44e00144	 0x00000012
    0x44e00148	 0x00000012
    0x44e0014c	 0x00030000
    0x44e00150	 0x00000001
    0x44e00400	 0x00007716
    0x44e00404	 0x00000002
    0x44e00408	 0x00040002
    0x44e0040c	 0x00000002
    0x44e00410	 0x00000002
    0x44e00414	 0x12510000
    0x44e00418	 0x00000009
    0x44e0041c	 0x00000000
    0x44e00420	 0x00000001
    0x44e00424	 0x00000000
    0x44e00428	 0x00000000
    0x44e0042c	 0x00032017
    0x44e00430	 0x00000000
    0x44e00434	 0x00000001
    0x44e00438	 0x00000000
    0x44e0043c	 0x00000000
    0x44e00440	 0x00019017
    0x44e00444	 0x00000000
    0x44e00448	 0x00000001
    0x44e0044c	 0x00000000
    0x44e00450	 0x00000000
    0x44e00454	 0x0001873B
    0x44e00458	 0x00000000
    0x44e0045c	 0x00000001
    0x44e00460	 0x00000000
    0x44e00464	 0x00000000
    0x44e00468	 0x0003E817
    0x44e0046c	 0x00000000
    0x44e00470	 0x00000001
    0x44e00474	 0x00000000
    0x44e00478	 0x00000000
    0x44e0047c	 0x00000000
    0x44e00480	 0x0000022A
    0x44e00484	 0x00000228
    0x44e00488	 0x00000007
    0x44e0048c	 0x00000007
    0x44e00490	 0x00000007
    0x44e00494	 0x00000007
    0x44e00498	 0x00000007
    0x44e0049c	 0x0403C017
    0x44e004a0	 0x00000201
    0x44e004a4	 0x00000222
    0x44e004a8	 0x00000201
    0x44e004ac	 0x00000285
    0x44e004b0	 0x00040002
    0x44e004b4	 0x00000002
    0x44e004b8	 0x00030000
    0x44e004bc	 0x00000002
    0x44e004c0	 0x00030000
    0x44e004c4	 0x00000002
    0x44e004c8	 0x00030000
    0x44e004cc	 0x00000006
    0x44e004d0	 0x00000002
    0x44e004d4	 0x00030000
    0x44e004d8	 0x00000004
    0x44e00504	 0x00000001
    0x44e00508	 0x00000001
    0x44e0050c	 0x00000001
    0x44e00510	 0x00000001
    0x44e00514	 0x00000004
    0x44e00518	 0x00000001
    0x44e0051c	 0x00000001
    0x44e00520	 0x00000000
    0x44e00528	 0x00000000
    0x44e0052c	 0x00000000
    0x44e00530	 0x00000000
    0x44e00534	 0x00000000
    0x44e00538	 0x00000001
    0x44e0053c	 0x00000000
    0x44e00600	 0x00000006
    0x44e00604	 0x00000002
    0x44e00700	 0x00000080
    0x44e00800	 0x00000002
    0x44e00804	 0x00000302
    0x44e00900	 0x00000302
    0x44e00904	 0x00000002
    0x44e00908	 0x00070000
    0x44e0090c	 0x00000002
    0x44e00910	 0x00030000
    0x44e00914	 0x00030000
    0x44e00a00	 0x00000002
    0x44e00a20	 0x00030000
    0x44e00b00	 0x00000000
    0x44e00b04	 0x00000500
    0x44e00b08	 0x00000000
    0x44e00b0c	 0x00000100
    0x44e00b10	 0x00000000
    0x44e00c00	 0x00000003
    0x44e00c04	 0x00000000
    0x44e00c08	 0x01E60007
    0x44e00c0c	 0xEE0000EB
    0x44e00d00	 0x00000000
    0x44e00d04	 0x00000008
    0x44e00d08	 0x00000000
    0x44e00d0c	 0x00000020
    0x44e00e00	 0x01FF0007
    0x44e00e04	 0x000003F7
    0x44e00e08	 0x00000000
    0x44e00f00	 0x00000000
    0x44e00f04	 0x00001006
    0x44e00f08	 0x00000000
    0x44e00f0c	 0x78000017
    0x44e00f10	 0x00000003
    0x44e00f14	 0x00000000
    0x44e00f18	 0x00000003
    0x44e00f1c	 0x00000000
    0x44e01000	 0x00000004
    0x44e01004	 0x00000000
    0x44e01100	 0x00060047
    0x44e01104	 0x00000000
    0x44e01110	 0x00000037
    0x44e01114	 0x00000001
    0x44e01200	 0x00000003
    0x44e01204	 0x00000007
    0x44e10040	 0x00400337
    
    0x44e00000	 0x00024502
    0x44e00004	 0x0000000A
    0x44e00008	 0x00000102
    0x44e0000c	 0x00000056
    0x44e00010	 0x00070000
    0x44e00014	 0x00000002
    0x44e00018	 0x00000002
    0x44e0001c	 0x00070000
    0x44e00020	 0x00070000
    0x44e00024	 0x00000002
    0x44e00028	 0x00000002
    0x44e0002c	 0x00000002
    0x44e00030	 0x00000002
    0x44e00034	 0x00030000
    0x44e00038	 0x00030000
    0x44e0003c	 0x00030000
    0x44e00040	 0x00030000
    0x44e00044	 0x00030000
    0x44e00048	 0x00030000
    0x44e0004c	 0x00030000
    0x44e00050	 0x00030000
    0x44e00054	 0x00030000
    0x44e00058	 0x00030000
    0x44e00060	 0x00000002
    0x44e00064	 0x00000002
    0x44e00068	 0x00030000
    0x44e0006c	 0x00000002
    0x44e00070	 0x00030000
    0x44e00074	 0x00000002
    0x44e00078	 0x00000002
    0x44e0007c	 0x00030000
    0x44e00080	 0x00000002
    0x44e00084	 0x00030000
    0x44e00088	 0x00030000
    0x44e0008c	 0x00030000
    0x44e00090	 0x00000002
    0x44e00094	 0x00030000
    0x44e00098	 0x00030000
    0x44e0009c	 0x00030000
    0x44e000a0	 0x00030000
    0x44e000a4	 0x00030000
    0x44e000a8	 0x00030000
    0x44e000ac	 0x00000002
    0x44e000b0	 0x00000002
    0x44e000b4	 0x00000002
    0x44e000b8	 0x00030000
    0x44e000bc	 0x00000002
    0x44e000c0	 0x00030000
    0x44e000c4	 0x00030000
    0x44e000c8	 0x00030000
    0x44e000cc	 0x00030000
    0x44e000d0	 0x00000002
    0x44e000d4	 0x00030000
    0x44e000d8	 0x00000002
    0x44e000dc	 0x00000002
    0x44e000e0	 0x00000002
    0x44e000e4	 0x00070000
    0x44e000e8	 0x00070000
    0x44e000ec	 0x00030000
    0x44e000f0	 0x00030000
    0x44e000f4	 0x00030000
    0x44e000f8	 0x00030000
    0x44e000fc	 0x00000002
    0x44e00100	 0x00000002
    0x44e00104	 0x00030000
    0x44e0010c	 0x00030000
    0x44e00110	 0x00000002
    0x44e0011c	 0x0000007A
    0x44e00120	 0x00000002
    0x44e00124	 0x00070000
    0x44e00128	 0x00030000
    0x44e0012c	 0x00000012
    0x44e00130	 0x00040002
    0x44e00134	 0x00030000
    0x44e00138	 0x00030000
    0x44e0013c	 0x00030000
    0x44e00140	 0x00000002
    0x44e00144	 0x00000012
    0x44e00148	 0x00000012
    0x44e0014c	 0x00000002
    0x44e00150	 0x00000012
    0x44e00400	 0x00007716
    0x44e00404	 0x00000002
    0x44e00408	 0x00040002
    0x44e0040c	 0x00000002
    0x44e00410	 0x00000002
    0x44e00414	 0x12510000
    0x44e00418	 0x00000009
    0x44e0041c	 0x00000000
    0x44e00420	 0x00000001
    0x44e00424	 0x00000000
    0x44e00428	 0x00000000
    0x44e0042c	 0x0001900C
    0x44e00430	 0x00000000
    0x44e00434	 0x00000001
    0x44e00438	 0x00000000
    0x44e0043c	 0x00000000
    0x44e00440	 0x00019019
    0x44e00444	 0x00000000
    0x44e00448	 0x00000001
    0x44e0044c	 0x00000000
    0x44e00450	 0x00000000
    0x44e00454	 0x00018719
    0x44e00458	 0x00000000
    0x44e0045c	 0x00000001
    0x44e00460	 0x00000000
    0x44e00464	 0x00000000
    0x44e00468	 0x00003219
    0x44e0046c	 0x00000000
    0x44e00470	 0x00000001
    0x44e00474	 0x00000000
    0x44e00478	 0x00000000
    0x44e0047c	 0x00000000
    0x44e00480	 0x00000221
    0x44e00484	 0x00000221
    0x44e00488	 0x00000007
    0x44e0048c	 0x00000007
    0x44e00490	 0x00000007
    0x44e00494	 0x00000007
    0x44e00498	 0x00000007
    0x44e0049c	 0x0403C019
    0x44e004a0	 0x00000201
    0x44e004a4	 0x00000225
    0x44e004a8	 0x00000201
    0x44e004ac	 0x00000285
    0x44e004b0	 0x00040002
    0x44e004b4	 0x00000002
    0x44e004b8	 0x00030000
    0x44e004bc	 0x00000002
    0x44e004c0	 0x00030000
    0x44e004c4	 0x00000002
    0x44e004c8	 0x00030000
    0x44e004cc	 0x00000006
    0x44e004d0	 0x00000002
    0x44e004d4	 0x00000002
    0x44e004d8	 0x00000001
    0x44e00504	 0x00000001
    0x44e00508	 0x00000001
    0x44e0050c	 0x00000001
    0x44e00510	 0x00000001
    0x44e00514	 0x00000004
    0x44e00518	 0x00000001
    0x44e0051c	 0x00000001
    0x44e00520	 0x00000000
    0x44e00528	 0x00000000
    0x44e0052c	 0x00000000
    0x44e00530	 0x00000000
    0x44e00534	 0x00000000
    0x44e00538	 0x00000001
    0x44e0053c	 0x00000000
    0x44e00600	 0x00000006
    0x44e00604	 0x00000002
    0x44e00700	 0x00000080
    0x44e00800	 0x00000002
    0x44e00804	 0x00000302
    0x44e00900	 0x00000002
    0x44e00904	 0x00070000
    0x44e00908	 0x00070000
    0x44e0090c	 0x00000002
    0x44e00910	 0x00030000
    0x44e00914	 0x00030000
    0x44e00a00	 0x00000002
    0x44e00a20	 0x00030000
    0x44e00b00	 0x00000000
    0x44e00b04	 0x00000500
    0x44e00b08	 0x00000000
    0x44e00b0c	 0x00000100
    0x44e00b10	 0x00000000
    0x44e00c00	 0x00000003
    0x44e00c04	 0x00000000
    0x44e00c08	 0x01E60007
    0x44e00c0c	 0xEE0000EB
    0x44e00d00	 0x00000000
    0x44e00d04	 0x00000008
    0x44e00d08	 0x00000000
    0x44e00d0c	 0x00000020
    0x44e00e00	 0x01FF0007
    0x44e00e04	 0x000003F7
    0x44e00e08	 0x00000000
    0x44e00f00	 0x00000000
    0x44e00f04	 0x00001006
    0x44e00f08	 0x00000000
    0x44e00f0c	 0x78000017
    0x44e00f10	 0x00000003
    0x44e00f14	 0x00000000
    0x44e00f18	 0x00000003
    0x44e00f1c	 0x00000000
    0x44e01000	 0x00000004
    0x44e01004	 0x00000000
    0x44e01100	 0x00060047
    0x44e01104	 0x00000001
    0x44e01110	 0x00000037
    0x44e01114	 0x00000000
    0x44e01200	 0x00000003
    0x44e01204	 0x00000007
    0x44e10040	 0x00C00337
    

  • I did a little massaging of the formatting and imported your 24 MHz "good" and "bad" into Clock Tree Tool.  The differences are purely with respect to the DPLL_MPU and DPLL_CORE configuration. Here are a few screenshots:

  • So my questions/comments based on these register dumps are:

    1. Where/how are you setting these different PLL configurations? 
    2. Neither of those dumps showed MPU @ 1 GHz, so I think your PLL parameters are incorrect.
    3. What type of DDR memory are you using, e.g. DDR2, DDR3?
    4. Why are you using OPP50 for DPLL_CORE?
    5. Are you lowering VDD_CORE too?  It's not required, but I want to understand the full scenario.
    6. Are you able to do some further experimentation where you change only ONE of these PLL's at run-time?  If you could only change one PLL for a given test you can hopefully narrow down which PLL relates to the issues you're having.

    Best regards,
    Brad

  • Here are some screenshots from the 26 MHz registers:

    You said this works fine, right?  I'm trying to understand if I can draw any conclusions from this as compared to the other data. 

  • Hi Brad

    Brad Griffis said:
    Where/how are you setting these different PLL configurations? 

    I set the configuration in the u-boot MLO (within the board specific am33xx_spl_board_init). 

    do_setup_dpll(&dpll_core_regs, &dpll_core_opp100);
    do_setup_dpll(&dpll_mpu_regs, &dpll_mpu_opp100);

    The values are unchanged from the arch/arm/cpu/armv7/am33xx/clock_am33xx.c. The max MPU frequency is read by the AM3354 efuse_sma register:

    struct dpll_params dpll_mpu_opp100 = {
                    CONFIG_SYS_MPUCLK, OSC-1, 1, -1, -1, -1, -1};
    const struct dpll_params dpll_core_opp100 = {
                    1000, OSC-1, -1, -1, 10, 8, 4};

    The 800MHz MPU frequency is set by the linux kernel. We fix the governor of our system to performance (in our case 800MHz). So, I haven't got changed the u-boot MPU frequency from 550MHz (default) to 1000MHz because we change it by the linux kernel.

    Brad Griffis said:
    Neither of those dumps showed MPU @ 1 GHz, so I think your PLL parameters are incorrect.

    We set MPU frequency to 800MHz not 1000MHz. Only the core dpll value is set to 1000MHz.

    Brad Griffis said:
    What type of DDR memory are you using, e.g. DDR2, DDR3?

    We use DDR3 memory

    Brad Griffis said:
    Why are you using OPP50 for DPLL_CORE?

    We do not want to use OPP50 for DPLL_CORE. We didn't set it in u-boot. That was our bug. So the value in the DPLL_CORE register was 50 MHz (As defined in arch/arm/cpu/armv7/am33xx/clock_am33xx.c):

    const struct dpll_params dpll_core = {
                    50, OSC-1, -1, -1, 1, 1, 1};

    Brad Griffis said:
    Are you lowering VDD_CORE too?  It's not required, but I want to understand the full scenario

    No, we don't set it lower. We measured the CORE voltage and it is/was in the specified range.

    Brad Griffis said:
    Are you able to do some further experimentation where you change only ONE of these PLL's at run-time?  If you could only change one PLL for a given test you can hopefully narrow down which PLL relates to the issues you're having.

    Okay, I will do that.

    Thanks

  • Ok, given that you didn't want to use OPP50 for the CORE, I think you should fix that and then see if the issue is still present.

    Have you tried disabling the governor altogether and setting the CPU frequency in u-boot? The performance governor simply sets the CPU frequency to the max allowed speed. It would be better to do that in u-boot because then you're booting at the fastest speed. Right now if you're booting at 600 MHz then your boot speed will be slower.

    I'm interested to know if setting the CPU speed in u-boot fixes the issue altogether. If so, that's a good sign that we've narrowed it down. If not, I think my conclusion would be that these DPLL differences are irrelevant and the true issue is some kind of clocking/noise issue. That said, yet another test would be to remove your crystal and input a 24 MHz square clock externally. (Though be sure the clock is not applied while the device is unpowered.)
  • Brad Griffis said:
    Ok, given that you didn't want to use OPP50 for the CORE, I think you should fix that and then see if the issue is still present.

    We fixed that and the problem does not occur now.

    Brad Griffis said:
    Have you tried disabling the governor altogether and setting the CPU frequency in u-boot? The performance governor simply sets the CPU frequency to the max allowed speed. It would be better to do that in u-boot because then you're booting at the fastest speed. Right now if you're booting at 600 MHz then your boot speed will be slower.

    No, I never tried it. But it will be a good idea to set the 800MHz in the u-boot's MLO, so it will be speed up boot time. Thanks

    Brad Griffis said:
    I'm interested to know if setting the CPU speed in u-boot fixes the issue altogether. If so, that's a good sign that we've narrowed it down. If not, I think my conclusion would be that these DPLL differences are irrelevant and the true issue is some kind of clocking/noise issue. That said, yet another test would be to remove your crystal and input a 24 MHz square clock externally. (Though be sure the clock is not applied while the device is unpowered.)

    We dismounted the crystal and clocked the AM3354 with an external clock generator. The error occured also with the external clock generator. A change to another clock frequency such as 19,2MHz, 25MHz didn't change anything (freeze still present). Only a configuration with 26MHz crystal works or the early boot pll settings in our u-boot's MLO. 

    Another interesting thing is the following:

    We also configured our board for 25MHz (SYSBOOT and u-boot) and clocked it with 26MHz. This configuration worked well as the fully correct 26MHz configuration (no freeze). A 25MHz configuration (SYSBOOT & u-boot) with 26MHz input clock frequency does not work well (freeze occurs). This was only a test...

  • If I make for the am335x-evmsk the same configuration (0x00003217) in the CM_CLKSEL_DPLL_CORE register, then the freeze occurs also on the ti evalboard. In general we solved the problem but I would like to understand the problem. 

  • Hello,

    I've continued to try to look into this and our feeling is that this is not a valid configuration and it is leading to underclocking the timer. The guidelines found in Table 8-22 Core PLL Typical Frequencies need to be followed and only the documented use cases have been verified. Using something different could certainly lead to unexpected behavior. I hope this helps close this issue, especially for any that might find it later via search.

    Thank you.

  • Thank you everyone for this posting.

    Our Java application would randomly lock up threads for around two minutes. This would only happen every couple of hours. 

    After much head scratching and taking everything apart we arrived at the same place at the OP and found this page. Sure enough our u-boot MLO code was not doing the initialization correctly either. After correcting this, we have so far not seen the issue.

    Before we could consistently duplicate the issue by running the following java code:

    public class FreezeTest implements Runnable	{
    	
    	public static void main(String[] args) {
    
    		FreezeTest supervisor = new FreezeTest();
            	Thread th = new Thread(supervisor);
    		th.start();
    		int i;
    		for (;;) {
    			++i;
    		}
    	}
    	
    	@Override public void run() {
    		for (;;) {
    			long beforeSleep = System.currentTimeMillis();
            		try {	Thread.sleep(2000);} catch (Exception ex) {}   
    		        long elapsedS = (System.currentTimeMillis() - beforeSleep) / 1000;
            		if (elapsedS > 5)
    				System.out.println("Elapsed: " + elapsedS);
    		}
    	}	
    }

    Basically just running an endless loop that maxes out the CPU and monitoring the time in a different thread to see if it freezes. Running this on 10 boards would cause the issue at least a few times per hour. The time lapse is always around 95 seconds when it happens, and during that time the gp_timer interrupts don't increment. Running an equivalent program in C shows the same symptoms, but it does not freeze as frequently.

    Thanks again for this post - it might have saved our project !!