Linux/AM3357: MMC Driver in linux kernel 4.1- Interrupt

Hi Shankar,

shankar sm said:

We are using beagle bone based custom board running linux 4.1.18

Do you use AM335x TI PSDK 2.00.02.11?

shankar sm said:

and we have a Wifi chip connected to the MMC0 port.

WL12xx? Wl18xx?

shankar sm said:

We initially had kernel 3.12 and we are trying to migrate to 4.1.18. One of the main reasons being, in kernel 3.12 the mmc driver did not have interrupt support and it was only polling which was causing us to get a very low throughput on Wi-Fi

I see we have DMA support there. Have you try it?

http://processors.wiki.ti.com/index.php/AM335x_MMC/SD_Driver's_Guide

http://processors.wiki.ti.com/index.php/AM335x-PSP_04.06.00.11_Features_and_Performance_Guide#MMC.2FSD_Driver

The below wiki pages are valid for PSDK 2.0.02.11 which comes with kernel 4.1.18:

processors.wiki.ti.com/index.php/Linux_Core_MMC/SD_User's_Guide

http://processors.wiki.ti.com/index.php?title=Processor_SDK_Linux_Kernel_Performance_Guide&oldid=216407#MMC.2FSD_Driver

Regards,
Pavel

Hi Pavel,

Thanks for the response. My answers are as below.

Pavel Botev said:

shankar sm

We are using beagle bone based custom board running linux 4.1.18

Do you use AM335x TI PSDK 2.00.02.11?

We are using 2.00.02 TI SDK through Timesys. Im not sure on the last version. But i dont think there has been any changes on the mmc driver. The changes seem to be same in kernel 4.4 as well.

Pavel Botev said:

and we have a Wifi chip connected to the MMC0 port.

WL12xx? Wl18xx?

Marvell 88w8801 WiFi Chip

Pavel Botev said:

shankar sm

We initially had kernel 3.12 and we are trying to migrate to 4.1.18. One of the main reasons being, in kernel 3.12 the mmc driver did not have interrupt support and it was only polling which was causing us to get a very low throughput on Wi-Fi

I see we have DMA support there. Have you try it?

I haven't tried the DMA support yet. I will give that a shot. However I want to understand what is wrong. I dont want to end in the same spot with DMA if it is some fundamental problem like pin configuration or something else.

I believe understanding where in the driver is the interrupt flags (like IRQF_TRIGGER_LOW etc) set will throw some light on it. Could you please point me to the driver code where this is done?
Some background on above point is I understand in the omap_hsmmc_probe() there is code as below. The 4th parameter is 0 which corresponds to IRQF_TRIGGER_NONE. Is this correct? If this is true what i dont understand is where in the code is it changed to something like IRQF_TRIGGER_LOW etc.

ret = devm_request_irq(&pdev->dev, host->irq, omap_hsmmc_irq, 0,
               mmc_hostname(mmc), host);

shankar sm said:

We have set up the mmc to run in interrupt mode and the configuration in device tree is same/very similar to what is there in the kernel Documentation in ti-omap-hsmmc.txt

I would suggest to try also with the configuration of the am33xx.dtsi file

linux-kernel/arch/arm/boot/dts/am33xx.dtsi

mmc1: mmc@48060000 {
            compatible = "ti,omap4-hsmmc";
            ti,hwmods = "mmc1";
            ti,dual-volt;
            ti,needs-special-reset;
            ti,needs-special-hs-handling;
            dmas = <&edma_xbar 24 0 0
                &edma_xbar 25 0 0>;
            dma-names = "tx", "rx";
            interrupts = <64>;
            interrupt-parent = <&intc>;
            reg = <0x48060000 0x1000>;
            status = "disabled";
        };

shankar sm said:

        interrupts-extended = <&intc 64 &gpio2 28 0>;

On which AM335x pin exactly you receive the interrupt request? Is it some of the MMC pins or GPIO2_28 pin?

Pavel,

Thanks for the response.

I figured out that the IRQF_TRIGGER_LOW can probably be given through the device tree as below, However that did not solve the problem I'm facing.
interrupts-extended = <&intc 64 &gpio2 28 IRQ_TYPE_LEVEL_LOW>;

Initially i thought the problem was in the suspend mode sue to incorrect gpio configuration but does not seem so. What i see is there is a CIRQ interrupt event through the line does not go low.
As per my understanding the CIRQ interrupt is generated if the mmc_cmd line goes low after enabling the interrupt. I checked the waveform using a logic analyser and it seemed like command lines are not pulled low. Not very sure how the interrupt is getting triggered.

Pavel Botev said:

I mean do you have any of the bits in SD_STAT (OMAP_HSMMC_STAT) register auto set to 1?

No, the register values are as below for reference.
inside omap_hsmmc_enable_sdio_irq()
STAT=00
ISE=0x100
IE=0x100
CON=0x10E00

In the interrupt handler inside omap_hsmmc_irq
STAT=0x100
ISE=0x100
IE=0x100
CON=0x10E00

Pavel,

Pavel Botev said:

Can you check if polling mode is working fine for you in kernel 4.1.8?

I have verified that it works in polling mode without any issues.

The device tree configuration for polling mode is as below.

&mmc1 {
        compatible = "ti,am33xx-hsmmc";
        vmmc-supply = <&vmmcsd_fixed>;
        bus-width = <4>;
        pinctrl-names = "default";
        pinctrl-0 = <&mmc1_pins_default>;
        status = "okay";
};

I will go through 18.2.4 section one more time to see if there anything that I have missed.

Thanks
Shankar

shankar sm said:

I have verified that it works in polling mode without any issues.

SD_STAT register should be used for both polling and interrupt mode. Can you share what value you have in SD_STAT register in polling mode and in interrupt mode?

Can you also explain what type of interrupt you expect to be logged in SD_STAT? Is it CIRQ or BWR or TC/CC or else?

I see also you mention GPIO2_28 pin. How is this pin involved in your setup?

Regards,
Pavel

Hi Pavel,

Let me tell you the whole story.

We are using the AM335x custom board with Marvell 88W8801 chip connected to mmc0 pins. The MMC0 is configured to operate in 4 bit mode.

Polling Mode:
When we configured the device tree as mentioned in above posts, polling mode was enabled and the Wi-Fi worked. However the throughput was very low (~1-4 MBps). So we started considering the interrupt mode to get more throughput.

Interrupt Mode:
In the case of interrupt mode with the device tree configuration as mentioned in above post, it did not seem not work. The Marvell driver was constantly sending CMD53. Further debugging on the Marvell driver we saw that the specific CMD53 is sent only when the SDIO interrupt occurs.

We checked with Marvell to see if their driver works with 4.1.18 kernel and they seem to claim that they have tested this on other hardwares and they dont have any issues.
Excerpt from Marvell is as below

I enabled full debug on my system running kernel 4.1 to highlight steps of FW download
and FW ready (see attached log), driver uses polling for FW download and interrupt when FW is active.

1. Detect SDIO card
2. Disable SDIO host interrupt
3. Start SDIO probe
4. Start FW download
5. Poll card status every 1 msec
6. FW downloaded , ~ 300 msec
7. Enable host interrupt
8. FW is active, ~ 200 msec after downloaded

When I tried debugging to see what is different from interrupt mode and polling, itseems like the interrupt mode is only enabled when GPIO configuration is enabled for DAT1 line (this is to avoid being in infinite wait in suspend mode). I see that when this is enabled there is a constant IRQ immediately after the interrupt is enabled. The stat and other registers(like CON, ISE etc) seemed to be fine as I have mentioned in the above posts.

In interrupt mode, there was a dmesg error log saying it failed with error code -84 while sending CMD53, arg 0x10000040. This happens immediately after a CMD52, arg 0x0a00 every time, which happens to be an I/O abort command. The CMD52 and CMD53 repeats infinitely. I have attached the dmesg logs for the same.

I see also you mention GPIO2_28 pin. How is this pin involved in your setup?

For GPIO_28 pin, is the multiplexed DAT1 line of mmc. In the case of suspend mode this line will be pulled low to wake AM335x. This is same as the device tree configuration that is there under linux/Documentation/devicetree/bindings/mmc/ti-omap-hsmmc.txt

Thanks
Shankar

5811.dmesgLogs_with_extra_logging.txt

shankar sm said:

Polling Mode:
When we configured the device tree as mentioned in above posts, polling mode was enabled and the Wi-Fi worked. However the throughput was very low (~1-4 MBps). So we started considering the interrupt mode to get more throughput.

With TI SDK7 (linux 3.12), we have better throughput (ARM is running at 1000MHz):

http://processors.wiki.ti.com/index.php?title=Sitara_Linux_SDK_Kernel_Performance_Guide&amp;oldid=172232#MMC.2FSD_Driver

I think these numbers (in example 17.19 MBps in ext2 read) are achieved in EDMA mode.

The MMC/SD driver performance in PSDK 2.00.02 (linux 4.1.18) is little better (in example 21.21 MBps in ext2 read):

http://processors.wiki.ti.com/index.php?title=Processor_SDK_Linux_Kernel_Performance_Guide&amp;oldid=216407#MMC.2FSD_Driver

I will check the other points and come back to you when I have something on these.

Regards,
Pavel

shankar sm said:

We are using the AM335x custom board with Marvell 88W8801 chip connected to mmc0 pins. The MMC0 is configured to operate in 4 bit mode.

shankar sm said:

Interrupt Mode:
In the case of interrupt mode with the device tree configuration as mentioned in above post, it did not seem not work. The Marvell driver was constantly sending CMD53. Further debugging on the Marvell driver we saw that the specific CMD53 is sent only when the SDIO interrupt occurs.

Let me confirm what we have so far. You are using MMC0 controller in SDIO 4-bit mode, and you receive interrupt through the mmc0_dat1 pin (muxmode gpio2_28), and this interrupt is successfully logged in the SD_STAT[8] CIRQ bit. In polling mode you are checking this [8] CIRQ bit by software, thus you have slow performance. You want to enable interrupt mode, thus when [8] CIRQ bit is set, to generate MMCSD0INT (SINTERRUPTN) (#63) signal to the Cortex-A8 MPU, thus to invoke interrupt handler routine. Could you please confirm my understanding is correct?

And the problem is that you have many unexpected CIRQ events?

The controller interrupt must be clear by setting SD_IE[8] CIRQ_ENABLE to 0, then the host driver must start the interrupt service with card (clearing card interrupt status) to remove card interrupt source. Otherwise the Controller interrupt will be reasserted as soon as SD_IE[8] CIRQ_ENABLE is set to 1.


Also we have some info regarding OMAP35x CIRQ, which may also apply here:

Marvell 8686 needs the SDIO Clock to be active (driven by OMAP3530 SDIO Controller) in order for the WLAN to be able to generate an interrupt in 4 bit mode whereas OMAP3530 was designed for optimal low power and thus shuts off the SDIO clock when the bus is not active (i.e. while waiting for SDIO interrupts).  Thus we cannot support 4-bit mode with IRQs when using the Marvell 8686.  With the Marvell 8686, the SDIO interrupts work great in 1-bit mode since the clock is not required to be active on the Marvell side.

See also the below e2e thread:

Why is SDIO driver not interrupt driven? - Processors forum - Processors - TI E2E support forums

e2e.ti.com

Other Parts Discussed in Thread: AM3359 It appears to me, that the SDIO driver is not interrupt driven. This gives poor pertormance. Is there a reason for implementing

Regards,
Pavel

Hi Pavel,
    Sorry for the late reply I somehow missed your reply on the second tab and assumed you were still looking into it.

Let me confirm what we have so far. You are using MMC0 controller in SDIO 4-bit mode, and you receive interrupt through the mmc0_dat1 pin (muxmode gpio2_28), and this interrupt is successfully logged in the SD_STAT[8] CIRQ bit. In polling mode you are checking this [8] CIRQ bit by software, thus you have slow performance. You want to enable interrupt mode, thus when [8] CIRQ bit is set, to generate MMCSD0INT (SINTERRUPTN) (#63) signal to the Cortex-A8 MPU, thus to invoke interrupt handler routine. Could you please confirm my understanding is correct?

One small change, In polling mode, the CIRQ line is not used. The mmc driver sleeps for 10ms in sdio_irq_thread (see code below from file "drivers/mmc/core/sdio_irq.c") and after this timeout it queries the chip to see if there is new data. So this 10ms sleep is what is causing a slow throughput. Reducing this timeout will increase the throughput. Other parts of the understanding are correct.

static int sdio_irq_thread(void *_host)
{
        struct mmc_host *host = _host;
        struct sched_param param = { .sched_priority = 1 };
        unsigned long period, idle_period;
        int ret;

        sched_setscheduler(current, SCHED_FIFO, &param);

        /*
         * We want to allow for SDIO cards to work even on non SDIO
         * aware hosts.  One thing that non SDIO host cannot do is
         * asynchronous notification of pending SDIO card interrupts
         * hence we poll for them in that case.
         */
        idle_period = msecs_to_jiffies(10);
        ....
}

And the problem is that you have many unexpected CIRQ events?

    We have few updates, regarding this. We had fair amount of help from TI FAE (Khayami, Mohsen) who had come over this week to our office and
    we debugged together for a couple of days and made some progress. We continued our investigation later and the details that we have so far are as below.

    It seems there are 2 issues that we are currently seeing

1. 
   
   1. We had initially thought the problem for Wifi Chip to not work was mainly due to CIRQ interrupt.
   2. However when we debugged using lauterbach debugger (that Mohsen brought with him) and looked at the registers we found that the Pin Mux of the Dat1 line was not getting changed from GPIO (IDLE mode) to MMC line (Normal Mode) i.e the register 0x44E108F8 was 0x3F which corresponds to GPIO instead of 0x30 which corresponds to MMC.
   3. Using lauterbach debugger when we changed the register to 0x30, when it was stuck, it immediately started to work.
   4. We found that the pin remux happens in omap mmc driver code in functions omap_hsmmc_runtime_suspend() and omap_hsmmc_runtime_resume().
   5. After few more printk's we found that every time the marvel driver is calling request_firmware(), the issue seems to happen.
   6. Also we see that in kernel log that omap_hsmmc_runtime_suspend was called few miniseconds before the call to request_firmware was made, but we dont see a omap_hsmmc_runtime_resume call.
   7. We have tried the same marvell software on kernel 3.12 with a patch to support interrupts and it seems to work. We do see a omap_hsmmc_runtime_resume before the request_firmware call.
   8. Currently the issue always seems to happen when request_firmware kernel api call is made. However we are not sure who the culprit is. Meaning we dont know if  there is any other kernel thread or marvell thread that is supposed to be doing something that it is not.
2. The CIRQ interrupt is continuously getting triggered even though the line does not go low during the idle, which we have talked about the previously in above posts. With this the wifi will work however the interrupt causes the Driver to query the chip for data even though the chip has nothing to send.

        We are trying to concentrate on issue 1 for now. Couple of questions regarding the same
        1. From what I read from datasheet, the PM module will kick the MMC module out of idle mode to normal mode if there is any mmc transaction, Is that correct?
        2. If point 1 is true does sending a mmc msg such as cmd52 or cmd53 is considered to be a transaction?
        3. Before marvell driver makes a call to request_firmware kernel API does it have to do anything to bring the mmc back to normal mode or does request_firmware handle it?
        4. Do you know if there are any patches to the PM module that we should be using?
       

Let me know if there are any debugging suggestions that you might have or if you need any more details.
Below is the attachment for dmesg logs that has some extra printk msg.

6505.Wifi_Issue_4.1.18.txt

Thanks
Shankar

Pavel,

Yes it seems like the problem is when it is switching between idle mode to normal mode. We are not sending any commands manually to switch between the modes. From what I understand this is done by the PM module (Referring to AM335x TRM under MMCHS Pin List(18.2.3) there is a section for MMC power management that talks about Transition from Normal mode to smart-idle Mode). It looks like if there is no transaction on the MMC module then the PM module tells the MMC module on the next clock cycle to enter idle mode. When there is a new command to be sent then the MMC module wakes up. Thats my understanding.

Not very sure how to debug this.

Thanks,
Shankar

Finally we found the issue. It seems to be in sdk code which does not seem to be pushed to mainline yet. The problem is in else case of omap_hsmmc_runtime_resume function attached below. If you look at else of omap_hsmmc_runtime_suspend, pinctrl_pm_select_idle_state(dev) is called which is changing the mmc state to idle where as in runtime resume the pins are not changed to normal state, if host->pinctrl is not set. Seems like host->pinctrl is changed for specific boards such as dra7 and not for am335x. This  has been acknowledged by our TI FAE as well. Hopefully this will be fixed in the upcoming sdk release and mainlined.

Thanks for all the suggestions and help to isolate this issue.

static int omap_hsmmc_runtime_suspend(struct device *dev)
{
        struct omap_hsmmc_host *host;
        unsigned long flags;
        int ret = 0;

        host = platform_get_drvdata(to_platform_device(dev));
        omap_hsmmc_context_save(host);
        dev_dbg(dev, "disabled\n");

        spin_lock_irqsave(&host->irq_lock, flags);
        if ((host->mmc->caps & MMC_CAP_SDIO_IRQ) &&
            (host->flags & HSMMC_SDIO_IRQ_ENABLED)) {
                /* disable sdio irq handling to prevent race */
                OMAP_HSMMC_WRITE(host->base, ISE, 0);
                OMAP_HSMMC_WRITE(host->base, IE, 0);

                if (!(OMAP_HSMMC_READ(host->base, PSTATE) & DLEV_DAT(1))) {
                        /*
                         * dat1 line low, pending sdio irq
                         * race condition: possible irq handler running on
                         * multi-core, abort
                         */
                        dev_dbg(dev, "pending sdio irq, abort suspend\n");
                        OMAP_HSMMC_WRITE(host->base, STAT, STAT_CLEAR);
                        OMAP_HSMMC_WRITE(host->base, ISE, CIRQ_EN);
                        OMAP_HSMMC_WRITE(host->base, IE, CIRQ_EN);
                        pm_runtime_mark_last_busy(dev);
                        ret = -EBUSY;
                        goto abort;
                }

                pinctrl_pm_select_idle_state(dev);
        } else {
                pinctrl_pm_select_idle_state(dev);
        }

abort:
        spin_unlock_irqrestore(&host->irq_lock, flags);
        return ret;
}

static int omap_hsmmc_runtime_resume(struct device *dev)
{
        struct omap_hsmmc_host *host;
        unsigned long flags;
        int ret;

        host = platform_get_drvdata(to_platform_device(dev));
        omap_hsmmc_context_restore(host);
        dev_dbg(dev, "enabled\n");

        spin_lock_irqsave(&host->irq_lock, flags);
        if ((host->mmc->caps & MMC_CAP_SDIO_IRQ) &&
            (host->flags & HSMMC_SDIO_IRQ_ENABLED)) {

                pinctrl_pm_select_default_state(host->dev);

                /* irq lost, if pinmux incorrect */
                OMAP_HSMMC_WRITE(host->base, STAT, STAT_CLEAR);
                OMAP_HSMMC_WRITE(host->base, ISE, CIRQ_EN);
                OMAP_HSMMC_WRITE(host->base, IE, CIRQ_EN);
        } else {
                if (host->pinctrl) {
                        ret = pinctrl_select_state(host->pinctrl,
                                                   host->pinctrl_state);
                        if (ret)
                                dev_err(mmc_dev(host->mmc),
                                        "failed to activate pinctrl state\n");
                }
        }
        spin_unlock_irqrestore(&host->irq_lock, flags);
        return 0;
}