TDA4VM sdhc controler driver error---PHY IO calibration failed

Cunlei Sun said:

disable DLL_CALB,the board works fine

Hi,

Thanks for the details. So We do have a working combination but i believe the question is why the disable is needed? Also once the config is disabled then no failure seen on all 6 boards? Can you confirm that?

Best Regards,

Keerthy

Dear Keerthy,

     Here are some more information to update:

     1. This EMMC error only happens in -40℃， when they test in -30℃ it will not happen.

     2. As customer described, they located the error at IO calibration in kernel.

     3. Also, as customer described, they currently use u-boot boot. As a result, they need to do IO calibration in u-boot and kernel. They did twice calibration.

However, they also have other projects, in that project they use SBL boot and in that case they only need to do calibration once.

     4. Currently we find that in -40℃, we can pass calibration in U-boot while halted in kernel. So current workaround is disable the calibration in kernel.

     Also we have some suspects need to be verified:

     1. We are worried that this may be caused by incomplete release of resouces in -40℃。

     2. Test1: do calibration twice in u-boot by reset emmc controller.

     3. Test2: Disable then enable clock module right after entering kernel so that we do calibration twice in kernel.

     So here are customer's questions:

     1. Why in current case, they can pass the calibration in -30℃ but failed in -40℃？

     2. If in SBL boot mode we can only do calibration once, do they need to calibration twice with u-boot boot mode? (u-boot once and kernel once)

     3. Is it normal that we will face such error when we do calibration twice?

     4. Do we have any potential risk that we disable calibration in kernel?

     Hope you can help check the questions, thanks.

BR

Sikai

Hi,

Unlocking this thread for further communication.

- Keerthy

Dear Experts,

     From Marco:

     I found a very similar issue from earlier this year where a workaround was found by increasing the timeout period from 20 to 200 (ms?) for low temp rather than disabling IO calibration in kernel: https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1094942/tda4vm-sdhci-am654-probe-of-4f80000-sdhci-failed-with-error--110?tisearch=e2e-sitesearch&keymatch=dll_calib# 

     I am not yet sure if IO calibration in both kernel and u-boot is necessary, but it would seem safer not to recommend disabling either.

     From Marco:

     Adding onto this, actually have another thread just opened today with the same premise, and the same calibration timeout increase workaround applied: https://e2e.ti.com/support/processors-group/processors---internal/f/processors---internal-forum/1181780/tda4al-q1-emmc-io-calibration-timeout 

     Would any of you have any background on this:

     1. What are the steps in kernel IO calibration?

     2. What is causing IO calibration timeout at default 20us?

     3. Why is the timeout value set at 20us? Is it calculated by hardware or just an emulation value?

     From Joe:

     We tried to increase timeout value from 20us to 2000us, this issue is still there.

     Please refer to the below steps in kernel IO calibration, it is same with u-boot IO calibration.

      /* Reset OTAP to default value */

     mask = OTAPDLYENA_MASK | OTAPDLYSEL_MASK;

     regmap_update_bits(sdhci_am654->base, PHY_CTRL4, mask, 0x0);

     if (sdhci_am654->flags & DLL_CALIB) {

         regmap_read(sdhci_am654->base, PHY_STAT1, &val);

         if (~val & CALDONE_MASK) {

             /* Calibrate IO lines */

             regmap_update_bits(sdhci_am654->base, PHY_CTRL1,

                  PDB_MASK, PDB_MASK);

             ret = regmap_read_poll_timeout(sdhci_am654->base,

                 PHY_STAT1, val,

                 val & CALDONE_MASK,

                 1, 20);

BR

Sikai

hI Sikai,

Can you guys dump the MMCSD register in uboot and in kernel to see if there is any difference in that.

You can dump the 1kb of memory from MMCSD0_SS_CFG base address and see if there is any difference 

Regards
Diwakar

Dear Diwakar,

     Thanks for your advice. We are going to have a Chinese vocation for the next 7-10 days, so forgive for our late reply.

     By the way, do you know why we desgin such twice calibration at the very beginning?

BR

Sikai

Hi Sikai 

From where you are loading the Environment variable  if it is from FAT i don't think you calibration is getting done in uboot 

like ->Loading Environment from FAT

can you stop at the uboot promt and check the value of the MMCSD0_SS_CFG_MMCSD0_SS_PHY_STAT_1_REG register.

if you are getting 000000E0  then calibration is not happening in uboot .

and later kernel do that.

Regards
Diwakar

Dear Diwakar,

    Sorry to reply this after such a long time.

    1. Customer does not only boot from emmc, they boot u-boot from ospi flash and boot kernel from emmc.  So is their condition the same as you mentioned?

Diwakar Dhyani said:

From where you are loading the Environment variable  if it is from FAT i don't think you calibration is getting done in uboot 

like ->Loading Environment from FAT

    2. Here are the register dump in u-boot stage and kernel stage. Since they have some difficulty in reproducing the case so we only have dump in good condition. We will update the bad condition's once we reproduce it.

68414a00 00000000 00000000 00000000 201030c8 24ecc801 98000407 00000000  00
00000000 00000100 00000004 00000002 00000004 00000002 00000001 00000000  20
00000002 00000001 00000000 00000000 00000000 00000000 00000000 00000000  40
80000000 00000010 00000000 00000000 00000008 00000000 00000000 00000000  60
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000  80
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000  a0
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000  c0
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000  e0
00010001 00000000 10ff10ff 00000000 00000000 00000000 00000000 00000000 100
00000000 00000000 00000000 00000000 00000092 00000000 00000000 00000000 120
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 140
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 

68414a00 00000000 00000000 00000000 201030c8 64ecc801 98000407 00000000  00
00000000 00000100 00000004 00000002 00000004 00000002 00000001 00000000  20
00000002 00000001 00000000 00000000 00000000 00000000 00000000 00000000  40
80000000 00000010 00000000 00000000 00000008 00000000 00000000 00000000  60
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000  80
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000  a0
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000  c0
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000  e0
00010083 00000000 10ff10ff 00102000 00000000 00000000 00000000 00000000 100
00000000 00000000 00000000 00000000 00000093 00000000 00000000 00000000 120
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 140
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

    3. As you can see from the comparison below, even in good condition they have different dump in u-boot and kernel stage. Will this be a cause?

BR

Sikai

Hi Sikai

Can you check in the boot log are you seeing this line ?

Loading Environment from FAT

Regards
Diwakar

Dear Diwakar,

     1. Here is their whole boot log and I cannot find anything with key work "environment" or "fat".

         3386.log_all.txt

         After confirming with customer, they use default environment variables in u-boot instead of from File system or OSPI. They disable loading env in the configuraton file:

    2.  They add log in the process of calibration and we can see from the log that we have entered the process and do the calibration successfully in u-boot.

         加了打印，证明uboot进入了calibration流程的日志.txt

    3. Is it normal to have such differences?

Sikai Lu said:

    3. As you can see from the comparison below, even in good condition they have different dump in u-boot and kernel stage. Will this be a cause?

       

BR

Sikai

Hi Sikai 

Is it possible to dump the register just before the calibration in uboot and same in kernel?

What we can do is if there is a timeout in the driver we will do while 1 , can you add a logic like this and then dump the register 

Regards
Diwakar

Hi Sikai 

You can dump the register like this so that it is easy to read 

Regards
Diwakar