Linux/AM3352: Slow boot continues through WDT

James Miller39 said:

We've had a design in the field for 3 years that exhibits a 'slow' boot condition during which normal execution occurs at maybe 1/20 of the expected speed

The first thought that comes to mind based on your description is that perhaps the PLL has not properly locked.  If you were in bypass mode then you might experience this sort of issue.  Something like that might happen if there's excessive noise on the VDDS_PLL_* rails.

James Miller39 said:

Our work around has been to check the reset reason in u-boot and issue a reset from the PMIC if the reset reason is WDT.  This resolves the issue and the device is able to boot normally after initially failing.  However,  the over all boot time is creating problems

How much time is this workaround adding?  I would have expected it to be very small, e.g. <1sec.  What is your boot time requirement?

Yes, it is DDR_CKE Ball G3 on the ZCZ package that is toggling.  The only indication that we know we're running in 'slow mode' is the rate at which messages are produced on the console and the activation of the WDT reset, which, oddly fails to restore normal operation.  We typically use reset reason == WDT to indicate that we should issue a PMIC reset to recover,  This is known be be effective.  We're currently using u-boot as our boot loader which checks to ensure the PLL is locked prior to continuing the boot process and this appears to be working.  If the PLL fails to lock uboot will hang until WDT resets, but, what we see is messages continue to be output on the serial port.  Our JTAG port is available, what scripts can you suggest we try to verify configuration?

Thanks,

Jim

I have some scripts for Code Composer Studio that can help us. The general process for running these scripts is described here:

http://git.ti.com/sitara-dss-files/am335x-dss-files/blobs/master/README

In your particular instance, there are several scripts which you should run:

http://git.ti.com/sitara-dss-files/am335x-dss-files/blobs/raw/master/am335x-boot.dss

http://git.ti.com/sitara-dss-files/am335x-dss-files/blobs/raw/master/am335x-ctt.dss

http://git.ti.com/sitara-dss-files/am335x-dss-files/blobs/raw/master/am335x-ddr-analysis.dss

Please zip up and attach the output files.

Thank you for the scripts.  We'll get this up and running with the output back to you as soon as possible.

Another interesting thing we've found is that a watchdog timer reset on a normally running device causes this slow running condition approximately 30% of the time after the reset.  We see this by running our application and turning the pet of the watchdog off.  Is there any particular reason why a WDT reset wouldn't return the CPU back to a stable state like POR?

Also, we're  running an older version of the TI kernel 3.12.10.  Are you aware of any issues with locking of the PLL by the kernel?  We've ruled out failure of u-boot to lock the PLL as u-boot waits specifically for the PLL to lock and halts if the PLL doesn't lock.  We don't see a halt, the kernel loads as normal but everything runs s l o w l y.

Thanks.

I like the idea of updating to resolve these sorts of issues, but sometimes you trade old issues for new, unknown issues.  But, moving from 3,12, what kernel might you recommend?

James Miller39 said:

I like the idea of updating to resolve these sorts of issues, but sometimes you trade old issues for new, unknown issues.

I've not had a single instance where a customer regretted moving forward.  Sure, a few issues will pop up during the migration.  That's why it takes a month to migrate.  Usually the issues that pop up are things that other people are also encountering where solutions are readily available.

If you're close to production, you might use SDK 5.03 which is our final release on the 4.14 kernel.  However, if you still have a year or more of development work, you might consider moving all the way to SDK 6.00 when it is released in July.  It will be on the 4.19 kernel.  If you move to SDK 6.00, you should budget for one more small migration to a later release such as 6.03 which will be the final 4.19 release and out by this time a year from now.  It will be a rather small effort migrating from SDK 6.00 to 6.0x, likely just a few days of effort as it will mostly contain bug fixes and not a lot of huge feature additions.

Also, keep in mind that the kernels being chosen for the TI SDK releases are all LTS kernels.  So there will be much better community support on those kernels if you run into an issue a couple years from now.

Hi Brad, I work with James and am trying to make this work. I have a Segger JLink connected to our device. It can connect just fine on its own using JLink Commander:

SEGGER J-Link Commander V6.34f (Compiled Sep  5 2018 13:25:56)
DLL version V6.34f, compiled Sep  5 2018 13:25:28

Connecting to J-Link via USB...O.K.
Firmware: J-Link V9 compiled Oct 25 2018 11:46:07
Hardware version: V9.20
S/N: 59200642
License(s): GDB
VTref=3.316V


Type "connect" to establish a target connection, '?' for help
J-Link>connect
Please specify device / core. <Default>: AM3352
Type '?' for selection dialog
Device>?
Please specify target interface:
  J) JTAG (Default)
  S) SWD
  T) cJTAG
TIF>J
Device position in JTAG chain (IRPre,DRPre) <Default>: -1,-1 => Auto-detect
JTAGConf>-1
ERROR while parsing value for DRPre. Using default: -1.
Specify target interface speed [kHz]. <Default>: 4000 kHz
Speed>4000
Device "AM3352" selected.


Connecting to target via JTAG
TotalIRLen = 6, IRPrint = 0x01
TotalIRLen = 10, IRPrint = 0x0011
JTAG chain detection found 2 devices:
 #0 Id: 0x3BA00477, IRLen: 04, CoreSight JTAG-DP
 #1 Id: 0x2B94402F, IRLen: 06, TI ICEPick
AP map detection skipped. Manually configured AP map found.
AP[0]: AHB-AP (IDR: Not set)
AP[1]: APB-AP (IDR: Not set)
AP[2]: JTAG-AP (IDR: Not set)
Iterating through AP map to find AHB-AP to use
AP[0]: Skipped. Not an APB-AP
AP[1]: APB-AP found
Found Cortex-A8 r3p2
6 code breakpoints, 2 data breakpoints
Debug architecture ARMv7.0
Data endian: little
Main ID register: 0x413FC082
I-Cache L1: 32 KB, 128 Sets, 64 Bytes/Line, 4-Way
D-Cache L1: 32 KB, 128 Sets, 64 Bytes/Line, 4-Way
Unified-Cache L2: 256 KB, 512 Sets, 64 Bytes/Line, 8-Way
System control register:
  Instruction endian: little
  Level-1 instruction cache enabled
  Level-1 data cache enabled
  MMU enabled
  Branch prediction enabled
Memory zones:
  [0]: Default (Default access mode)
  [1]: AHB-AP (AP0) (DMA like acc. in AP0 addr. space)
  [2]: APB-AP (AP1) (DMA like acc. in AP1 addr. space)
Cortex-A8 identified.
J-Link>

This took a couple of tries (I deleted the first connect failure lines as irrelevant).

Then, following the instructions in the README, CCS says this in its Debug tab:


As per the instructions, I did NOT try to connect to the unit in any way (after I saw that the JTAG was working all the way through to the target, I disconnected and re-connected it physically and then re-invoked CCS to get the above screen shot.) But, when trying to execute the script as per the README instructions, I get this:
js:> loadJSFile c:\workspace\am335x-dss-files\am335x-boot.dss
Could not open session. No matching devices found. (c:\workspace\am335x-dss-files\am335x-boot.dss#60)
js:>

Does #60 mean "line 60" in that file?
Line 60 is this:

debugSessionDAP = ds.openSession("*","CS_DAP_DebugSS");

Thanks in advance.

Looks like your Segger can connect to the Cortex A8 but not the DAP. The scripts all use the DAP since it avoids the MMU. Do you have any TI XDS debuggers? If not you might consider getting an XDS110 for this purpose. It is only $99.

You might be able to make the script work by changing CS_DAP_DebugSS to CortxA8.  If the MMU is not yet enabled that will work ok. Once Linux enables that won't work.

am335x-script-results.zipPlease see attached zip file.

No, this was a properly running card. I just induced "slow" operation and ran the scripts again (multiple resets but still in slow mode!). Please see attached zip: am335x-scripts-results-slow.zip

Stephen Biggs said:

Also, there is this (in the normal running):
EMIF: PWR_MGMT_CTRL = 0x00000000
* ERROR: Bits 7:4 (reg_sr_tim) are in violation of Maximum Self-Refresh Command Limit
* Please see the silicon errata for more details.

It relates to Section 3.1.2 DDR3: JEDEC Compliance for Maximum Self-Refresh Command Limit.  Having the lower byte of the register set as 0xA0 is appropriate.  That's why there's no warning in the case of your slow boot, i.e. because the register is configured as 0x2A0.  That's actually a properly configured value for configuring self-refresh.  The only issue is you shouldn't be doing self-refresh!!!

Stephen Biggs said:

Yes, I saw this too. As far as I know, this is the same boot code being executed after a watchdog reset.

I wonder if you explicitly zero out that register in u-boot if that would resolve your issue.

Stephen Biggs said:

The question is: why is this being put into self-refresh in the first place?

I think for starters you should check to see if adding in code to u-boot that forces that to a defined value (either 0 or 0xA0) fixes the issue.  In other words, before we get overly obsessed with this register, it would be good to have some further confirmation that this truly has a big impact on the issue.  I think it will, but let's verify.

The next question in my mind would be WHEN is this register taking on this value of 0x2A0.  You might need to add some additional prints into u-boot and/or Linux to understand that one.  Given that this issue happens "after some time", I expect it's happening in Linux, and then never getting cleared out by u-boot.  So I think having u-boot initialize this register is good.

Your script flags an error:

EMIF: PWR_MGMT_CTRL = 0x00000000
 * ERROR: Bits 7:4 (reg_sr_tim) are in violation of Maximum Self-Refresh Command Limit
 * Please see the silicon errata for more details.

The errata says:

When using DDR3 EMIF Self-Refresh, it is possible to violate the maximum refresh command requirement specified in the JEDEC standard DDR3 SDRAM Specification (JESD79-3E, July 2010). This requirement states that the DDR3 EMIF controller should issue no more than 16 refresh commands within any 15.6-μs interval.

To avoid this requirement violation, when using the DDR3 EMIF and Self-Refresh (setting LP_MODE = 0x2 field in the PMCR), the SR_TIM value in the PMCR must to be programmed to a value greater than or equal to 0x9.

But with a 0 in reg_lp_mode, self refresh is not enabled, so your script is wrong to flag this, IMO.

Stephen Biggs said:

Missing a post that got sent to moderation where I questioned the validity of your script flagging an errata issue with the EMIF register when we weren't doing self refresh.

Stephen Biggs said:

But since LP_MODE is 0, we are not doing self-refresh so this error is not correct and should not be flagged unless we are doing self-refresh (LP_MODE == 2) and the reg_sr_tim is actually out of range.

I've made it more intelligent now.  It only flags an error if you're using a combination of reg_lp_mode=2 with an invalid reg_sr_tim.  It gives a warning otherwise saying that the value being used could be a problem if it was ever used with reg_lp_mode=2.

Thank you for your help on this.  We can detect the issue in u-boot and clear the self refresh register and it prevents the case of continuously slow operation through a software watchdog reset event.  My question now is why this is occurring.  I would think this register would be set to a known value during reset, but perhaps this isn't happening because it's persistent through software watchdog reset.  Is this a possibility?  You also mentioned read-modify-write.  Would this refresh setting persist through a software reset and r-m-w?  We know POR typically always resolves the issue.  

James Miller39 said:

I would think this register would be set to a known value during reset, but perhaps this isn't happening because it's persistent through software watchdog reset.  Is this a possibility?

It certainly seems that this EMIF_PWR_MGMT register is persistent through a watchdog reset.  Take a look in the Technical Reference Manual at Table 8-26. Reset Sources.  The watchdog reset does not reset the PLL's and also puts the DDR into self-refresh.  A cold reset on the other hand will reset everything.

James Miller39 said:

You also mentioned read-modify-write.

That was specific to Linux suspend operation.

James Miller39 said:

We know POR typically always resolves the issue. 

Yes, a POR will clear this register.  However, ensuring the DDR initialization code writes to this register (which is what is done in the current u-boot) is something you should be doing.

I'm looking in the Technical Reference manual rev P table 8-26 and I see where the PLLs are reset differently but you mention that watchdog reset puts the DDR into self refresh.  I don't see that in the table, where is this indicated?  I'm also looking where EMIF_PWR_MGMT is persistent though watchdog reset and I'm not seeing that, can you help oint that register out for me?  Thanks

Thank you for the explanation and pointing me to the right information.  One final question, are there any warm reset conditions under which self-refresh mode would not be entered, i.e. PWR_MGMT = 0x00 on reset?  I'm not sure that we're seeing evidence of self-refresh after every warm reset.  Certainly, we do see lp_mode set on most warm resets but not on every warm reset.

Jim