• State TI Thinks Resolved
  • Locked Locked
  • Replies 17 replies
  • Answers 1 answer
  • Subscribers 99 subscribers
  • Views 1918 views
  • Users 0 members are here
Support feedback
Options
Options
Related

This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

DRA744: Ti OMAP_HSMMC host controller issue with software time out in IRQ context.

Vivek Pernamitta
Prodigy 170 points

Part Number: DRA744

HI TI Team,

Platform: TOI DRA74x SOC

eMMC    : samsung

We have one problem when we are issuing  "sanitize" operation from host to eMMc device. Once we issue Sanitize operation the response can take up to  4min(240 sec) according to eMMC standard . But Ti has one errata (snippet is attached below) where maximum time out is always 700 ms when host is running at 192MHZ. For to over come this Ti has already added software timer(which is coming  approx 3 sec in our hardware)  for the response. This time out is applicable  for all the responses, if the response comes in less than 4sec timer is deleted,  but sanitize requires more time (240 sec)  according to specs.  We have observed that once we issue sanitize operation CMD6 is send  from   host will wait for 4 sec( 700ms from H/W time out + 3 sec from S/W timer) to get respnse, If response is not with in 4 secs driver will HPI interrupt to interrupt  sanitize operation using CMD13 untill it is success. We have observed that system will go for hang after certain time due to RCU lock. To over come this sanitize time out  We have tried some experiments by  increased  time out to 240 sec in "omap_hsmmc_do_irq()" in drivers/mmc/host/omap_hsmmc.c. After applying the below change We observed that response(for CMD6 sanitize) is coming after 12 seconds . As we cannot pre-determine for response  we have to wait for  4mins max accoding to standard.

 omap_hsmmc_do_irq(){

.....

        OMAP_HSMMC_WRITE(host->base, STAT, status);
        if (end_cmd || ((status & CC_EN) && host->cmd)) {
                omap_hsmmc_cmd_done(host, host->cmd);
                if (host->data_timeout) {
                        unsigned long timeout;
 
                        if ((host->mrq->cmd->arg >>16 & 0xFF) == EXT_CSD_SANITIZE_START) {
                                host->data_timeout=240000000000;

                        }

                        timeout = jiffies +
                                  nsecs_to_jiffies(host->data_timeout);
                        mod_timer(&host->timer, timeout);
                }
        }
        if ((end_trans || (status & TC_EN)) && host->mrq)
                omap_hsmmc_xfer_done(host, data);
}
Here there is a new problem, We connot wait for 240 seconds in ISR context, which will impact system performace and lead to other errors. We would like to know is there any reasons for running  software timer  in ISR context. Can you suugest any approaches  for this.

Above is the highlighted we have added for experimental purpose.

Errata:

                  * DRA7 Errata No i834: When using high speed HS200 and
                 * SDR104 cards, the functional clock for MMC module
                 * will be 192MHz. At this frequency, the maximum
                 * obtainable timeout (DTO =0xE) in hardware is
                 * (1/192MHz)*2^27 = 700ms. Commands taking longer than
                 * 700ms will be affected by this small window frame
                 * and will be timing out frequently even without a
                 * genuine timeout from the card. Workaround for
                 * this errata is use a software timer instead of
                 * hardware timer to provide the timeout requested
                 * by the upper layer.

Thanks and Regards,

Vivek

 
static void omap_hsmmc_do_irq(struct omap_hsmmc_host *host, int status)
  • 0
    TI__Mastermind 34130 points

    HI Vivek,

    Could you try the patch below
    http://review.omapzoom.org/#/c/38998/

    Regards,
    Vishal

  • 0 in reply to Vishal Mahaveer
    Prodigy 170 points

    Hi Vishal,

    looks all  the three patches are same. Did any thing is missed.Looks patch is for MMC_Erase operation, but sanitize is different.

    Please let me know if i missed anything.

    In the mean time i will try the patch.

    Thanks & regards,

    Vivek

  • 0 in reply to Vivek Pernamitta
    Prodigy 170 points

    HI Vishal,

    We have tried above patch We are seeing kernel  is hanged (that is the actual issue).  Looks above patch is meant for mmc_erase operation, Please correct me if i am wrong.

    Thanks and Regards,

             Vivek

  • 0 in reply to Vivek Pernamitta
    TI__Mastermind 34130 points

    Hi Vivek,

    Could you try reverting this patch from Kernel?

    mmc: host: omap_hsmmc: use sw timeout for busy timeout as well

    Regards,
    Vishal

  • 0 in reply to Vishal Mahaveer
    Prodigy 170 points

    HI Vishal,

    We have tried reverting the above patch, But we are still facing the same issue.

    Thanks and Regards,

    Vivek

  • 0 in reply to Vivek Pernamitta
    TI__Mastermind 34130 points

    Could you share the full kernel log when the issue happens?

    Could you also share the steps to reproduce this, I want to try this on TI EVM.

  • 0 in reply to Vishal Mahaveer
    Prodigy 170 points

    HI Vishal,

    Please find the dmesg logs attached at the time of sanitize operation being run, For sanitize operation max timeout should be 240 sec as per JEDEC standard, But host controller is waiting only for 4sec(700ms for hardware  timeout interrupt [192Mhz clock] + 3 sec using Software timer) and then sending CMD13 (status) and then cmd13(HPI  interrupt) and failed to receive response. The reproduction step is we are  calling  sanitize ioctl from user space.

    HaS_hpi_HT_RCU_error.txtvdev_mmc_debug_ioctl-dump_intp.txt

    Thanks and Regards,

               Vivek

  • 0 in reply to Vivek Pernamitta
    TI__Mastermind 34130 points

    Hi Vivek,

    I was trying the sanitize command on my 4.4 setup with TI EVM and seeing a different behavior. 
    There is no failed response, but there is a CMD13 after 4 seconds.

    On the driver query we are still looking in to it. Will get back to you.

    jacinto6evm:/ # mmc_utils sanitize /dev/block/mmcblk0 <
    [ 268.647297] omap_hsmmc 480b4000.mmc: enabled
    [ 268.651600] mmc0: starting CMD6 arg 03b30001 flags 0000049d
    [ 268.658274] mmc0: req done (CMD6): 0: 00000800 00000000 00000000 00000000
    [ 268.665109] mmc0: starting CMD13 arg 00010000 flags 00000195
    [ 268.670797] mmc0: req done (CMD13): 0: 00000900 00000000 00000000 00000000
    [ 268.677711] mmc0: ioctl_do_sanitize - SANITIZE IN PROGRESS...
    [ 268.683536] mmc0: starting CMD6 arg 03a50101 flags 0000049d
    [ 272.463455] mmc0: starting CMD13 arg 00010000 flags 00000195
    [ 272.469206] mmc0: req done (CMD13): 0: 00000e00 00000000 00000000 00000000
    [ 272.476131] mmc0: starting CMD13 arg 00010001 flags 00000015
    [ 272.481874] mmc0: req done (CMD13): 0: 00000e00 00000000 00000000 00000000
    [ 272.488798] mmc0: starting CMD13 arg 00010000 flags 00000195
    [ 272.494576] mmc0: req done (CMD13): 0: 00000900 00000000 00000000 00000000
    [ 272.501500] mmc0: mmc_wait_for_req_done: Interrupted sanitize
    [ 272.507347] mmc0: starting CMD13 arg 00010000 flags 00000195
    [ 272.513042] mmc0: req done (CMD13): 0: 00000900 00000000 00000000 00000000
    [ 272.519967] mmc0: ioctl_do_sanitize - SANITIZE COMPLETED
    [ 272.623448] omap_hsmmc 480b4000.mmc: disabled


    Regards,
    Vishal

  • 0 in reply to Vishal Mahaveer
    Prodigy 170 points

    Hi Vishal.

    Thanks for analysis

    As from above logs maximum time is waiting for four seconds, and then cmd 13 (status. starting CMD13 arg 00010000 flags 00000195) is being sent and cmd13 is responded from device . Then CMD13 with HPI interrupt (starting CMD13 arg 00010001 flags 00000015) is being send and sanitize operation is being  interrupted and  device interrupted sanitize  successfully and responded for CMD13 HPI.

    But some times device may not respond for HPI interrupt/status and system will go for hang. So  to over come that  mmc core driver is  setting maximum time out is 240 seconds. Attached snippet below.

    Code:

    ...

    #define MMC_SANITIZE_REQ_TIMEOUT 240000

    file: drivers/mmc/card/block.c

    ..

    static int ioctl_do_sanitize(struct mmc_card *card)
    {
            int err;

            if (!mmc_can_sanitize(card)) {
                            pr_warn("%s: %s - SANITIZE is not supported\n",
                                    mmc_hostname(card->host), __func__);
                            err = -EOPNOTSUPP;
                            goto out;
            }

            pr_debug("%s: %s - SANITIZE IN PROGRESS...\n",
                    mmc_hostname(card->host), __func__);

            err = mmc_switch(card, EXT_CSD_CMD_SET_NORMAL,
                                            EXT_CSD_SANITIZE_START, 1,
                                            MMC_SANITIZE_REQ_TIMEOUT);

            if (err)
                    pr_err("%s: %s - EXT_CSD_SANITIZE_START failed. err=%d\n",
                           mmc_hostname(card->host), __func__, err);

            pr_debug("%s: %s - SANITIZE COMPLETED\n", mmc_hostname(card->host),
                                                 __func__);
    out:
            return err;
    }

    This issue will not be observed always  only few times. So to over come this  we have added more delay(240)  through software timer only for sanitize operation. So please suggest us it OK to go or any recommendations from your side.

    Thanks and Regards,

    Vivek

  • 0 in reply to Vivek Pernamitta
    TI__Mastermind 34130 points

    Hi Vivek,

    Could you share the changes you did?

    Regards,
    Vishal

  • 0 in reply to Vishal Mahaveer
    Prodigy 170 points

    Hi Vishal,

    Please find the changes below

    We have  increased  time out value to 240 sec in "omap_hsmmc_do_irq()" in drivers/mmc/host/omap_hsmmc.c. After applying the below change We observed that once response(for CMD6 sanitize) was coming after 12 seconds .

    file:  drivers/mmc/host/omap_hsmmc.c

     omap_hsmmc_do_irq(){

    .....
            OMAP_HSMMC_WRITE(host->base, STAT, status);
            if (end_cmd || ((status & CC_EN) && host->cmd)) {
                    omap_hsmmc_cmd_done(host, host->cmd);
                    if (host->data_timeout) {
                            unsigned long timeout;
     
                            if ((host->mrq->cmd->arg >>16 & 0xFF) == EXT_CSD_SANITIZE_START) {
                                    host->data_timeout=240000000000;

                            }

                            timeout = jiffies +
                                      nsecs_to_jiffies(host->data_timeout);
                            mod_timer(&host->timer, timeout);
                    }
            }
            if ((end_trans || (status & TC_EN)) && host->mrq)
                    omap_hsmmc_xfer_done(host, data);
    }


    Thanks and Regards,

    Vivek

  • 0 in reply to Vivek Pernamitta
    TI__Mastermind 34130 points

    Hi Vivek,

    You can also use the sanitize_busy member of struct mmc_command to detect sanitize command. But the main issue is why the upper layer timeout is not propagating to host->data_timeout.

    Can you check,
    - What is the value of data_timeout just before the command is sent?
    - Looking at set_data_timeout(), it only sets host->data_timeout if it is a data command or if RSP_BUSY flag is set. Can you check whether this is happening?

  • 0 in reply to Vishal Mahaveer
    Prodigy 170 points

    Hi Vishal,

    We have checked host->data_timeout is set to 3776798720 through RSP_BUSY flag set in set_data_timeout(). and value remains same until  command is sent out "omap_hsmmc_start_command()".

    So for the issue you want me use i sanitize_busy  (as a flag) member of struct mmc_command to change timeout value for software timer in irq handler

    Attached log

    Sanitize_timeout_raw_value_24_9_2019.txt

    Thanks and Regards,

    Vivek

  • 0 in reply to Vivek Pernamitta
    TI__Mastermind 34130 points

    Hi Vivek,

    Thank you for the logs.
    We are still working on finding the reason why upper layer timeout is not propagating to host->data_timeout.

    Not everyone in our team have a board with eMMC part that supports SANITIZE.
    Could you please share below information as well,
    1. In __mmc_switch() which is called from ioctal_do_sanitize(), what is the value of host->max_busy_timeout and timeout_ms?
    2. In set_data_timeout() (omap_hsmmc.c), what dto value gets written to SYSCTL and what value does host->data_timeout have?

    Regards,
    Vishal

  • 0 in reply to Vishal Mahaveer
    Prodigy 170 points

    HI Vishal,

    Please find the values you have requested for sanitize operation

    1.In __mmc_switch() which is called from ioctal_do_sanitize(), what is the value of host->max_busy_timeout and timeout_ms?

             timeout_ms = [240000]    

    host->max_busy_timeout = [0]

    2. In set_data_timeout() (omap_hsmmc.c), what dto value gets written to SYSCTL and what value does host->data_timeout have?

    host->data_timeout = [3776798720] 

    dto=[14]

    please find the full log attached.ti_raw_timeout_value_24_09_19.txt

    Thanks and Regards,

    Vivek

  • 0 in reply to Vivek Pernamitta
    TI__Mastermind 34130 points

    Hi Vivek,

    We are still working on this. Will get back to you as soon as we have an update. 

    Regards,
    Vishal

  • 0 in reply to Vishal Mahaveer
    TI__Mastermind 34130 points

    Hi Vivek,

    For now you can use your change as a solution for SANITIZE command. It's safe to use.

    Regards,
    Vishal