Linux/AM4379: QSPI Boot Problems

We might be able to find what's wrong by dumping some info via JTAG.  First, please download these scripts which are used on conjunction with CCS and a XDS-class JTAG probe:

http://git.ti.com/sitara-dss-files/am43xx-dss-files/blobs/raw/master/am43xx-ctt.dss

http://git.ti.com/sitara-dss-files/am43xx-dss-files/blobs/raw/master/am43xx-ddr-analysis.dss

http://git.ti.com/sitara-dss-files/am43xx-dss-files/blobs/raw/master/padconf/am43xx-padconf.dss

What I'd like you to do is:

1. Boot one of your "good" boards from QSPI and hit a key to pause at u-boot command line.

2. Follow the instructions from the README for each of the above 3 scripts.  The output from the scripts will land on your desktop.

3. Please zip up all these files as good.zip to attach.

4. Repeat for the "bad" board and create similar bad.zip file to attach.

You should be able to diff some of these files and perhaps draw some conclusions of your own based on what's different.

Best regards,
Brad

3858.not_working.zip8640.working.zipHi Brand,

I did a little bit more research on the boot problems and figured out, that one of my new boards boot up, one boots up but stops in kernel with faults and one does not boot. spl comes up every time, u-boot stops sometimes during initializing and sometimes comes up ok, but this specific boards stops always at "Starting kernel ...".

Because I thought, it has to do something with the qspi, I changed the frequency of qspi in kernel and u-boot from 48Mhz to 300kHz but nothing changed. Exact same behavior. So I guess it has to be something different.

I did as you told me and run the debug scripts. "Not_working.zip" is the board, which stops at "starting kernel..." and "working.zip" is the board, which boots up normally.

I already compared the output. And the only differences are in am43xx-ddr-config.csv. And the differences are at 0x4c000080 (EMIF4D_PERFORMANCE_CTR_1), 0x4c000084 (EMIF4D_PERFORMANCE_CTR_2), 0x4c000090 (EMIF4D_PERFORMANCE_CTR_TIME) and many EMIF4D_PHY_STS_xx registers.

The board has two ddr3 ram chips, designed in fly-by architecture. Because there is a small mistake in the board, currently the reference voltage for the ram is supplied by a resistive divider, which will be replaced in the next revision, by the default power supply.

If the ram is the problem, why it boots up with mmc card and with qspi it does not? Are there any differences in power consumption during boot up?

Stefan Haensel said:

Because there is a small mistake in the board, currently the reference voltage for the ram is supplied by a resistive divider, which will be replaced in the next revision, by the default power supply.

When you say "reference voltage" are you talking about VREF?  On a related note, do you intend to support any type of suspend/resume capability in your hardware?

I want to make sure that VTT is correctly implemented with a regulator that can both sink and source current such as the TPS51200.  If you used a resistor divider for VTT that would be a major issue.

Stefan Haensel said:

one of my new boards boot up, one boots up but stops in kernel with faults and one does not boot. spl comes up every time, u-boot stops sometimes during initializing and sometimes comes up ok, but this specific boards stops always at "Starting kernel ...".

I suspect all of these relate to DDR issues.  There were some major differences in the leveling output between good and bad boards.  I expect this can be remedied by using the EMIF Tools app note and accompanying spreadsheet to determine the proper configuration:

http://www.ti.com/sprac70

FYI, we are in the process of upstreaming some patches to u-boot related to hardware leveling and invert_clkout=1.  Most of the patches relate to properly handling hardware leveling in the case of suspend/resume.  If you're not using suspend/resume then the one critical patch you'll need is this one:

https://git.ti.com/~bradgriffis/ti-u-boot/bradgriffis-ti-u-boot/commit/92538daea23db37377841ae9176a59c4f9177098

And while it's not critical, if you want to avoid seeing an L3 timeout error you should also implement this simple fix:

https://git.ti.com/~bradgriffis/ti-u-boot/bradgriffis-ti-u-boot/commit/2c5c712ee240d1947979d2c9612aa52a87fe8cff

If you intend to have suspend/resume capability, I have a series of 6 patches that are needed:

https://git.ti.com/~bradgriffis/ti-u-boot/bradgriffis-ti-u-boot/commits/am437x_hw_leveling

FYI, we are in the process of upstreaming these to u-boot (and there's a kernel patch needed too if you need suspend/resume).  We are working actively to make sure the code in u-boot and Linux is the most robust possible, so I apologize that in the near term you will need to apply some of these patches manually.

So next steps should be:

1. Download the EMIF Tools app note and spreadsheet and fill in all the details.
2. Update your u-boot configuration based on the spreadsheet.
3. Make sure the necessary patches from above are integrated, e.g. that first one should be sufficient just to boot reliably.
4. Pause at u-boot and re-run the DSS script pertaining to DDR.  We want to compare the captured values against the DDR spreadsheet to make sure everything is configured correctly.

Hi Brad,

no we don’t want to have any suspend/resume capability.
Yes, with reference voltage I meant VREF of DDR. For the Supply of DDR we used TPS51200DRCT, but forgot to connect REFOUT with VREF. So currently we use a resistive divider together with some capacitors to generate VREF. But the termination voltage is generated by TPS51200. So this should be no problem.

The whole build environment including u-boot and the kernel I use from colleagues, which build up a similar system two years ago. DDR3 memory and other components are the same. So, for EMIF registers I even used their values. But today I had a look into the spreadsheet and the results were some other register values. I copied them to my u-boot configuration and the boards did not booted up. After applying the first patch (invert_clkout=1) the boards booted up.

After flashing qspi and spi flash again I still get kernel panic or with the “bad” board stopping at “starting kernel…”.

I even re-run the DSS script on two boards (“bad” board and board with kernel panic). Those files are attached.

Tomorrow I will have a closer look at the u-boot code. Because I using it from my colleagues, which distinguish at some points between spi boot and other boots, because spi boot is their production boot and everything else debug boot. Maybe they are activating some hardware in spi boot, what they don’t during debug boot.

Do you have any other idea, what I can check? I also tried to run mtest during uboot to have a confirmation that it is really about DDR. I tried “mtest 0x8000000 0xAFFFFFFF”, what should test al 1GB ram, which is installed. And everything worked. In MMC boot and qspi boot. I am little bit confused, because I assumed, that when I boot from qspi that uboot is stored in ram and mtest should overwrite this region and it should crash. But it didn’t. So can you confirm, that this is the right command?

I also attach the datasheet of our ram and my spreadsheet.
Our design was simulated in hyperlinx so there should not be any problem on our ram connection.

newSettingsNotWorking.zipnewSettingsAlmostWorking.zipKopie von SPRAC70_AM437x_EMIF_Configuration_Tool_V20.xlsx4Gb_1_35V_DDR3L.pdf

I haven't been able to review everything yet, but can you please try updating the DDR_PHY_CTRL_1 value from 0x00048008 to 0x00048009?

I generally do my memory testing in Linux using memtester, so I'm not sure on your question about u-boot mtest. One possible explanation is that some of the code is already cached and so it continues to work ok. In general though that doesn't seem like the right thing to do, i.e. you shouldn't blast the entire DDR. Checking a MB or 2 is probably sufficient just to see if it functions.