This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

TMS320C5515: CSL v3.04 MMC_write() Returns Spurious CSL_EMMCSD_WRITE_ERROR

Part Number: TMS320C5515

Hi,

We have a pcb which uses the C5515 and an SD card in conjunction with CSL v3.04.

We've seen spurious errors with the file system, and have debugged it back to the method MMC_write() in CSL v3.04.

Occasionally this method returns the value CSL_EMMCSD_WRITE_ERROR.

The biggest problem is getting it to fail predictively, which we have not been able to do.

The error can occur on an SD card which was just reformatted too (the reformat is being done in Windows 7), so no user data is on the card.

The code fails in the below section:

/* Added Pedro's response  KR032010 */
response = (hMmcsd->mmcRegs->MMCRSP7 << 8);
response = response << 8;
response |= hMmcsd->mmcRegs->MMCRSP6;
if( (response & 0xFFFF00FF) != 0)
{
    return (CSL_EMMCSD_WRITE_ERROR);
}

The value of response has been 0x00400900 when the error occurs, I will try to find out if this value is consistent with the failure.

Does anyone know what could lead to this type of error?

The next action item would be to find out what the value 0x00400900 means as a response which I will post here if I can figure it out.

Thanks!

Ben

  • I was able to get the documentation for the SD Card specification to see what the response means.

    With a response value of 0x00400900 it seems that bits 22, 11, and 8 are set in the card status bits area of a response of type R1.

    Bit 22 = ILLEGAL_COMMAND
    Bit 11 is part of the CURRENT_STATE (bits 12:9) and when is the only bit set in the CURRENT_STATE, CURRENT_STATE = 4 or tran (I will have to find out more about this state).
    Bit 8 = READY_FOR_DATA

    It seems as though that an illegal command is somehow being sent to the SD card.

    Ben
  • Hi,

    I've notified the sw team. They will post their feedback directly here.

    Best Regards,
    Yordan
  • Ben,

    We are looking into this and shall get back to you.

    Lali
  • Thanks Lali.

    If I find anything else out I will post here.

    Ben
  • Ben,

    To get a better understanding, a few questions:

    - Was this issue seen with a new batch of boards where this problem was not noticed on an older rev?
    - Has anything changed in your HW?
    - Are you able to put a logic analyzer on the MMC bus and see the transactions when this error occurs? This would perhaps provide some clues on the problem?

    Lali
  • Hi Lali,

    Answers to questions below in red:

    - Was this issue seen with a new batch of boards where this problem was not noticed on an older rev?  The problem was reported on a fairly recent batch of PCB's, but I have been able to reproduce the error on my development PCB which is much older than the boards we originally saw the problem on.

    - Has anything changed in your HW? No

    - Are you able to put a logic analyzer on the MMC bus and see the transactions when this error occurs? This would perhaps provide some clues on the problem? I can use our oscilloscope, but I'm not sure if the signals are readily available on our PCB, but I will check.

     


    Thanks,


    Ben

  • Hi Lali,

    We do have access to the SD card signals, but only have a 4 channel scope, and no logic analyzer.

    Ben
  • Ben,

    I'm looking into this and will post back more feedback.

    Following up on your previous reply. How did you reproduce it on your development PCB?
    Trying to understand why this was not observed before. Was there some SW "knob" you had to tweak on your development PCB to start seeing this error?

    Lali

  • Hi Lali,

    In regards to why we didn't see the issue in the past, unfortunately the issue is not 100% reproducible and is spurious at best.

    Below is a more comprehensive timeline of events.

    Recently a customer reported issues with the file system in the unit when they tried to save data.

    We have 2 file medias in the unit, on board flash and the micro SD card.

    We told them to use the on board flash on our unit instead of the SD card file system, and the file issue went away.

    This caused us to evaluate the SD cards closer.

    We used a Windows utility which checked the integrity of the micro SD card by writing to the whole card.

    This utility also posted the write and read speed that Windows saw when interacting with the micro SD card, and we found that the micro SD cards that gave us trouble had a write speed of roughly 4-5MB/s or below.

    We went with that criteria for the time being (write speed) to reject micro SD cards, but then saw an issue with the file system on a 16GB micro SD card from PNY that had a reported write speed of 9MB/s.

    This caused alarm since it threw our write speed criteria right out the window, if this micro SD card failed then many more may be subject to the failure.

    I then started taking micro SD cards that failed our write speed test to see if I could debug the problem down to its source.

    I found that the unit would potentially fail when interacting with the SD card in any of the 3 scenarios:

    1.  At startup our unit checks to see if a micro SD card is present.  If a micro SD is present it attempts to initialize it with the default folders.

    2.  When the user creates a file.

    3.  When the user saves data to a file.

    These steps are in order of time, so some SD cards could pass step 1, and then fail on step 2 or 3.

    Since catching the failure at step 1 was the soonest in time that is where I have spent my debugging time with the code.

    It is during this area that I was able to catch the SD card driver in CLS v3.04 returning the error code.

    Thanks,

    Ben

  • One other item to mention.
    The default media the unit will use is the on board flash.
    The user has to explicitly switch the media choice if they wish to use the micro SD card.
    That being said, there may be units out in the field that have this issue but it is unknown to the user since they are using the on board flash and not the micro SD card.

    Ben

  • Hi Ben,

    Just following up on some of the questions:

    - Please try with a 1 GB SD card to see if there is any change in behavior.
    - Confirmed that pull-ups are on DATA and CMD lines
    - Operation mode for MMC_open(): CSL_MMCSD_OPMODE_POLLED.
    - Try reducing block size to <512 for MMC_write() or MMC_read() (Using something greater than 512 bytes triggers multiple block R/W commands which needs a stop command to stop the R/W transaction).
    - How can we reproduce the issue on a TI C5515 EVM?

    Lali
  • Hi Lali,
    We have the TI eZdsp v2 USB Stick Rev D.
    If I can get the issue to occur on this PCB will that work on your end?
    I will also post my findings for the different transfer sizes (below 512), and 1GB SD card once tested.
    Ben

    Edit:

    (Added Rev D to the eZdsp model)

  • Hi Ben,

    Yes, it helps to be able to reproduce on this EVM.

    Lali
  • Hi Lali,
    Below are my findings so far:
    - ILLEGAL_COMMAND failures occur on multiple SanDisk 2GB micro SD cards, and a PNY 16GB micro SD card.
    - Same test on a 4GB SanDisk micro SD card had no issues as of yet.
    - We do not have a 1GB micro SD card on hand, but will look for one to test.
    - I tested writing 256 bytes (below 512 byte test), and this failed on the first three SanDisk 2GB micro SD cards I tested, so I went no further. The failure was different in this test. The TI driver got stuck in the do while loop on lines 4614 to 4659 and never returned. I did not perform this test on the 16GB micro SD card since it has to address it by sector, and not individual byte addresses.
    - The tests I've now been performing are just a continuous for loop that writes/reads data to/from the SD card (with the modified mmctod and mmctor timeouts). I can port this off of our hardware to another development platform, but all that we have right now is the TI eZDSP v2 USB Stick Rev D. We do not have the TI C5515 EVM, should we purchase the TI C5515 EVM to perform the micro SD card test on?
    - Also, below is a link which has the post back in April 2015 outlining why we changed the mmctod and mmctor register values.
    e2e.ti.com/.../418216
    Thanks,
    Ben
  • Ben,

    Thanks for the information.
    Is the eZdsp you have this one www.ti.com/.../TMDX5515EZDSP ?

    If so, then having the problem reproducible on this board would suffice.

    Lali
  • Hi Lali,

    Yes, that is the board we have.

    I will proceed with porting the code over to this board and let you know what I find.

    Ben
  • Hi Lali,
    I ported the minimum amount of code to reproduce the error on the eZdsp board.
    The project is standalone and does not require any external references.
    How best to send the code to you?
    Thanks,
    Ben
  • Ben,
    Is this something you can attach here, or would need another mechanism?

    Lali
  • Hi Lali,

    Another mechanism would be preferable.

    Also, would exporting the project (File-->Export-->General-->Archive File) be the best method?

    Or just zip up the entire project directory in the code composer working directory folder?

    Thanks,

    Ben
  • Hi Ben
    Archive file would be best.
    I will ask someone from out team to reach out to you, and you can pass the file on to them to relay to us.

    Lali
  • Hi Lali,
    That sounds good.
    I will prep the archive file and await contact email.
    Thanks,
    Ben
  • Ben,
    One other thing to test. Can you please try the MMC_read/write only (without file system)?
    We want to rule out the FS as a cause.

    Lali
  • Hi Lali,

    I have done testing without the file system and the error still happens.

    The example code I will provide is pretty short and to the point in regards to getting the bug to show.

    It is just a for loop writing to and reading from the micro SD card.

    Ben

  • Hi Lali,

    We have yet to receive contact email for sending the project.

    The project is ready to go once the contact email is sent to us.

    Thanks,

    Ben
  • Ben,
    We got your project. Please give us some time to dig into it.

    Lali
  • Hi Lali,

    Thanks for the update.

    If you have any questions about the project please let me know.

    Thanks,

    Ben

  • Hi Ben,

    We are able to reproduce the error with a certain brand of SD cards, in this case it was a Transcend 8GB HC SD card.
    The error DID NOT happen with Sandisk 8GB, Kingston 2GB, Samsung 32GB.

    The problem seems to be caused by a Write command being sent before a Read completes (while the SD card itself tries to complete the Read transaction). There isn't a way to find out the status of the transaction of the SD card.
    So, to get around this problem, please try the code snippet sent via email (from the sales folks. Wasn't sure if you were ok posting the code here).

    It essentially does 5 tries of the MMC_WRITE with a 1ms delay in case there is a write failure. This worked the few times that it was tested on the same card that was failing (fix is in: do_write_test_method()).

    Please give this a try and let us know if it fixed the failures.
    What brand SD card are you using?

    Lali
  • Hi Lali,

    We have tested on the following micro SD cards:

    - 2GB SanDisk

    - 4GB SanDisk

    - 16GB PNY

    The supplied code in do_write_test_method() seems to have worked around the issue with the ILLEGAL_COMMAND.

    We have yet to see retryCnt exceed 2.  It seems that after a single delay and 1 retry, the micro SD card write always succeeds.

    The code path that tries to write less than 512 bytes still fails in the same spot, but we are always writing 512 bytes so this doesn't concern us.

    We will continue testing and let you know what we find.

    Thank you for the quick response.

    Ben

  • Ben,
    Thanks for the update. Keep us posted.

    Lali
  • Hi Lali,

    Have you tried the below parameters for the test method on the micro SD card that failed to see if it still failed?

    do_write_test_method(false, 512);

    If the bug was something to do with the micro SD card not being ready for a write right after a read, this code path should hopefully reveal that, but we have not seen this.

    Prior to your changes in the test code, a micro SD card that failed with the below parameters:

    do_write_test_method(true, 512);

    Would pass with the below parameters:

    do_write_test_method(false, 512);

    Ben

  • Hi Ben,

    Please try the below code snippet for the FALSE section. I think what you had previously was incorrect.

    /*
    * This code path to date has had no issue with the reads or writes when number_of_bytes_to_transfer
    *  is equal to 512.  If number_of_bytes_to_transfer is 256, the write method will get stuck.  No other values other
    *  than 512 and 265 have been tested yet.  This code path does not change the timeout registers of the mmc peripheral.
    */
    
    if(MMC_write(mmcsdHandle, (number_of_writes), number_of_bytes_to_transfer, write_buffer_test) != CSL_SOK)
    {
    if(MMC_write(mmcsdHandle, (number_of_writes), number_of_bytes_to_transfer, write_buffer_test) != CSL_SOK)
                  while(1);
    }
    
    if(MMC_read(mmcsdHandle, (number_of_writes), number_of_bytes_to_transfer, write_buffer_test) != CSL_SOK)
    {
    if(MMC_read(mmcsdHandle, (number_of_writes), number_of_bytes_to_transfer, write_buffer_test) != CSL_SOK)
                  while(1);
    }

    The above code passed for FALSE as well for a card that was previously failing.

    Lali

  • Hi Lali,

    So far we have not had a micro SD card fail with the below parameters being passed to the test function:

    do_write_test_method(false, 512);

    If our initial issue was somehow something to do with successive reads/writes with the micro SD card, I think this code path should have exposed the issue, but to date has not.

    I don't think the successive reads/writes with the micro SD card are the source of the issue, but perhaps the constant changing of the mmctod or mmctor timeout registers?

    Thanks,

    Ben

  • Ben,

    We think that the problem is with doing a MMC_config for every disk_read and disk_write call.

    status = MMC_config(mmcsdHandle, &mmcConfig);
    
       if(status != CSL_SOK)
        {
           return RES_ERROR;
        }

    If you take a look at the CSL example located at C:\ti\c55_lp\c55_csl_3.08\ccs_v6.x_examples\mmc_sd\CSL_MMCSD_MmcCardExample\csl_mmcsd_MmcCardExample.c

    The MMC_config is done one time. There may be other initializations in your read write functions that would also need to be reviewd. Please use the example I point to as reference. There are other examples in the C:\ti\c55_lp\c55_csl_3.08\ccs_v6.x_examples\mmc_sd directory as well.

    We tested by omitting this routine and it seemed to have solved the problem with your original code.

    Could you try it out without the RETRY fixed we suggested and report back? Thanks.

    Lali

  • Hi Lali,

    We will test out and provide feedback.

    Also, should MMC_config() only be called once?

    I wasn't able to find anything in the documentation which provided any restrictions on this method other than MMC_open() should be called prior.

    Thanks!

    Ben

  • Hi Lali,

    The code was retested and the following was found.
    Scenario 1: The RETRY code you introduced was removed, and the micro SD card went back to failing when calling do_write_test_method(true, 512);
    Scenario 2: The RETRY code you introduced was removed, and the modifications of the mmctod and mmctor were removed. The calls to MMC_getConfig() and MMC_config() remained. The method do_write_test_method(true, 512); was then called and the same micro SD card which failed in above Scenario 1, did not fail in Scenario 2.

    It would seem as though that modifying the mmctod and mmctor registers is what is causing the issue.
    Should this not be done, if so why?

    Thanks,

    Ben
  • Ben,

    Thanks for confirming.
    What's the initial reason behind re-doing the TOD and TOR registers for every transaction?

    Lali
  • Hi Lali,
    Originally we found that the uC was seemingly getting stuck in the read/write methods of CSL for micro SD card.
    The user would perceive this as the unit freezing.
    We lowered the timeout registers so at least control would go back to the user if there was some kind of error with the micro SD card.
    We established different timeout values for a read and write of the micro SD card (write seemingly needed more because otherwise it would timeout prematurely even though everything would have been ok had it gotten more time).
    This didn't solve the problem but made it recoverable until we could find a fix.
    One of the other things we changed in addition to the timeout registers was disable interrupts before the read/write of the micro SD card thinking this may have had something to do with causing the CSL micro SD card driver to get stuck.
    We are running a small RTOS which theoretically could have preempted the task reading/writing to the micro SD card in the CSL read/write methods, and it would not have had control returned to it for maybe 1ms - 25ms potentially.
    We were never able to catch it in the act of preempting the task while in the CSL micro SD read/write methods though.
    The original timeout issue seemed to have been fixed, however, this latest issue is what seems to have popped up as a result of the fix for the timeout issue.
    Ben
  • Ben,

    I checked with some colleagues and they mentioned that you can have a smaller value for TOR and TOD. Also, you don't need to set these values repeatedly.

    Lali
  • Hi Lali,

    We brought up the idea of setting the timeout to the larger of the two (this way constantly setting them would not have to occur).

    Is the constant setting of these registers somehow interfering with proper operation of the MMC peripheral?

    Thanks,

    Ben

  • Hi Ben,
    I'm looking into your question internally and shall get back to you.

    Lali
  • Thanks Lali.

    We've tested with only modifying the timeout registers at initialization and so far everything seems good.

    We just want to make sure there is nothing else going on.

    Ben

  • Hi Ben,

    We think that the constant setting of the TOD/TOR is causing the problem. This is based on the testing we have done and the behavior that you have confirmed from your testing as well.
    At this point, I don't have any other feedback.

    Lali
  • Hi Lali,

    Thank you for all the support.

    This current solution seems to successfully work around the issue so far.

    We would like to be able to get to the source of the issue to make sure it does not appear again later down the road.

    It does seem to have something to do with the constant setting of the TOD/TOR registers, but what could it be at the core?

    Thanks,

    Ben

  • Hi Ben

    Based on our review of the user guide and internal spec, the MMC SD controller seems to be expect that these registers are configured during MMC_config phase and one time. The MMCSD CSL example illustrates this.

    The way you had originally implemented the routine may not necessarily be the way it is expected to be configured, and could have led to unexplained behavior of the peripheral. Unfortunately we would not be able to root cause this further.

    Let me know if I am missing something in order to alleviate your concerns to close this out.

    Lali