• State
  • Locked Locked
  • Replies 12 replies
  • Subscribers 31 subscribers
  • Views 4029 views
  • Users 0 members are here
Support feedback
Options
Options
Related

This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

TM4C1294NCPDT: FATFS with SDCard write/read optimization

Luis Afonso
Guru 20670 points
Part Number: TM4C1294NCPDT

Hi everyone!

Long time! Finally some time for projects now that I am in vacation from University! Let's see how my coding is (and also how my english is too).
So I am now trying to finally use SD cards (I am probably saying for the past 2 years that I should do that).
Been successful using the FATFS with the port that comes with tivaware.But I am not liking the write times so I want to optimize that for data logging. This might be a bit more related to the third party software so it's probably more directed to the forum users and their experience rather than TI employees. 



Been reading up in the best ways to log data into the sd card, but there's still an issue. A write will always take a couple milliseconds (sometimes a bunch of milliseconds). Basically f_write will block the code while that happens. Could implement DMA on the SPI or something similar but it seems f_write tries to read/write more stuff from/to the sd card. This means that f_write will still block the code for some milliseconds. Am I correct? This seems a bit weird because it seems the FATFS functions shouldn't be touched, only the port functions - so even when using an OS this seems unsolvable! (if I don't edit the FATFS functions and actually use them).

Before going and implementing DMA on the SPI I of course want to know if this will actually help.

I've seen suggestions to when actually logging data to avoid FATFS and write directly into the sectors (and previously having a file with pre-allocated size). Here DMA on the SPI would help avoid blocking the code for a long time.



I hope some of you can give me a boost and/or point if any of my line of thought is wrong.
I hope you have a continuation of a good day!
Luís

  • 0
    Guru 117855 points

    *** Noted *** After TWO YEARS - The much hoped/desired, "Return of Luis!"       (might it be that poster Bruno left the dungeon (pardon) "work-place" door (bit) awry - enabling your escape?)     Your English is fine - but for the "Brazilian accent, cadence & phraseology."

    Welcome Luis - beyond "help" from those here - might  your investigation into the "means/methods of (top flite) commercial products" using such "SD card storage media" - describe/define "What is "reasonably" achievable/possible?"     Such should provide the "top end" - should be well marked w/in their Tech Lit. - and should save you substantial time/effort.

    In general - might a boost (or point) result from your "writing such data into SRAM (first on chip) and later (off chip) as/if/where required?"     Once the essential data has been captured - it may (then) be logged to the "slow" SD card (without great speed concerns) - for later analysis and/or processing...

    Should (other) answers arrive - yet not to your liking - I'll contact via our (standard) means.      (never mentioned - the "effectiveness of our hack-saw blades" - which embedded among tech shipment - strived to  free you from your servitude...and B's claims that you "enjoyed" bread & water.)

  • 0 in reply to cb1_mobile
    Guru 20670 points
    Hi cb1,

    I am very tired right now (insomnia is a pain...) so just a short answer for now. The only dungeon (if any) keepers are the teachers from automation masters...
    If anything goes over my head of what you're writing hopefully tomorrow/later I will read it better.

    OS would probably solve a lot of the problems derived from the SD card slowness unlike I questioned in the first post.

    Your solution is around what I am thinking of using - but it will still mean that the MCU will need to dispense (probably wrong translation) a long, continuous time to just dump data into the SD card (at least a few milliseconds every 512 bytes). Still pondering if this is acceptable.

    Actually a way english teachers tells us to write in english is to from the phrases/word in brazilian. The grammar (sintax?) is more similar to english. In Portuguese from Portugal, the phrases almost always need to be completely rearranged and many words that seem similar are not correct (like the "dispense time", the meaning might be kinda similar but not appropriate in english).

    I believe there is a somewhat elegant solution to my problem from what I've read that can avoid an OS. Hopefully tomorrow with better rest I will be able to decide a solution.
  • 0 in reply to Luis Afonso
    Guru 13040 points
    Great to see Luis around!

    So that's the chain and whip noises I hear when driving nearby Coimbra! I assumed they were related to the ghosts of the Roman Empire, not to Tech Uni masters!!!

    I wouldn't see a problem that the writing takes a few miliseconds - IF that's what the actual physical transfer takes, then it is what it is. BUT it is important to understand what is the "system available" feedback provided by the SDCard. Is there some sort of "I'm ready" message which is part of FAT protocol? I know for a fact that it ain't done by additional GPIO logical signals.

    Using OS or not would not affect help at all the transfer part of it. It would be useful only if different "sub-sections" of the FAT transfer can respond to generated flags/semaphores. And even then, I'd expect these sub-functions to be conditionally called by the application, without OS as well - does anyone know if this is the case?

    Something that is certainly not useful is a "freezing" loop inside the SDCard library, some sort of "while(CARD_IS_STILL_BUSY){}", that throws your MCU to the trash bin not allowing it do anything else during the wait period. Again, does anyone here know if such happens?

    As for the physical transfer itself, indeed that is a phase where DMA can be useful. A properly sized buffer and a flip-flop-ping-pong-zig-zag smoothly implemented RAM to SPI engine would certainly alleviate the CPU burden for those byte hauling actions.

    Cheers

    Bruno
  • 0 in reply to Bruno Saraiva
    Guru 20670 points

    Still sleepy zzz. I will answer here for future people that actually search forum.

    I've seen it take 4ms and sometimes 100ms btw.

    "Is there some sort of "I'm ready" message which is part of FAT protocol? I know for a fact that it ain't done by additional GPIO logical signals."

    It does as far as I've seen. There is "wait_ready" in the port.

    "Something that is certainly not useful is a "freezing" loop inside the SDCard library, some sort of "while(CARD_IS_STILL_BUSY){}", that throws your MCU to the trash bin not allowing it do anything else during the wait period. Again, does anyone here know if such happens?"

    This does happen. This would easily solved with OS actually. Dunno how to solve without editing fatfs instead of just the port otherwise (besides what I am trying right now that is so far working).

    "As for the physical transfer itself, indeed that is a phase where DMA can be useful. A properly sized buffer and a flip-flop-ping-pong-zig-zag smoothly implemented RAM to SPI engine would certainly alleviate the CPU burden for those byte hauling actions."

    It might not help because of how fatfs works. Not sure if it's in certain conditions (cluster miss, cluster update and stuff like that) or always.

  • 0 in reply to Bruno Saraiva
    Guru 20670 points
    Just 1 note,

    Tried doing something I've seen advised. Pre alocated a file in contiguous sectors (had to check fatfs sample code because I am using FATFS prior to f_expand being added, c'mon Tivaware update your package, it's at least 3 versions behind, I know it's third party but still). This all using FATFS.
    I get the sector where the file begging, then I directly write into the SD (fatfs free) and keep track of the useful data of the file . I get constant times so far - just bellow 2ms. This way the DMA will help much, much more which is a plus (and of course predictable times is much better than having couple of fast times and an ocasional 200ms delay...)

    More test is needed. I wrote 10 sectors in this test. Need to see if it's possible to neatly cut the file size if after logging there are empty sectors and much more. But this seems to work better than just using FATFS.
  • 0 in reply to Luis Afonso
    Guru 13040 points

    Luis Afonso said:
    This does happen. This would easily solved with OS actually.

    I'm not sure this would be "easily solved with OS" - except if the OS actually slices time, automatically calls the taks for a period, and after a while regains control of the main system despite of what that transfer task was doing. Still, whatever period that is, the cpu will be doing nothing, stuck in that loop.

    I assume it would be more efficient to have a brief call to something like

    BackGroundLoop()
    {
    if (!CARD_IS_STILL_BUSY)
    {
      EnableDMATransferOfSomeMoreDataToCard();
    }
    ContinueDoingOtherThingsNotRelatedToSDCard();
    }

    Not too creative this morning to explain the idea, but I hope it makes sense.

    Bruno

  • 0 in reply to Bruno Saraiva
    Guru 117855 points

    Mes Amis,

    Is it possible that TWO Central Objectives - at minimum - (should) be "in play" here?

    • Capture acquired data into memory - as fast as possible (so that critical, incoming data is "not" (forever) lost!)
    • Later - under far less "execution time" pressure - copy data from (assumed) SRAM into a non-volatile storage medium.   (such as SD card)

    The posting (seems) to "skimp over" the requirement to,  "Capture LIVE Data" at the highest possible rate!     MCU and Analog vendors make every effort to "maximize such ADC rates" - any data logging system should NOT (hold hostage) data conversion rates to a flawed storage medium!      Again - this critical point appears to have escaped notice (but for here) w/in this thread - it has (some) importance - does it not?

    The suggestion to review the efforts & achievements of "top cabin" commercial vendors (still) appears a valuable source of insight (i.e. provides a storage time "benchmark") into what rates may be achievable via "SD card."     Yet - most always - this is (very) much secondary to, "Capture of acquired data at the highest possible rate!"

    Note too - certain "pro data loggers" write captured data "directly into RAM" - and then transfer that RAM data to a separate, attached device (PC usually) via Ethernet or USB - completely avoiding the "slow-down" and extra time/effort enforced by the SD card!      (perhaps they "know" something...)

  • 0 in reply to cb1_mobile
    Guru 20670 points
    Hi cb1,

    Do not worry your suggestion was not ignored! I was just fishing for more.
    Your suggestion is very good, especially for how I've seen (and you probably know) the SD card working. At first "okay but what if it's ALOT of data? RAM ends and is needed for something else. Maybe it's enough ram". But the SD card seems to work really well by sending a ton of data in a row. The write performance increases dramatically when writing the same amount of data in even bigger multiples of 512 bytes. Seems to be due to SD card housekeeping.

    Already have a very good performance (more than enough) with a 4096byte buffer. Still needs more tests of course, this is a very friendly test that I made.
    The CPU still gets blocked by about the same time - simply in that time more data is transferred. But how I am doing it right now, if it keeps working well, that can be solved if it becomes a issue.

    "Note too - certain "pro data loggers" write captured data "directly into RAM" - and then transfer that RAM data to a separate, attached device (PC usually) via Ethernet or USB - completely avoiding the "slow-down" and extra time/effort enforced by the SD card! (perhaps they "know" something...)"
    Yes you are correct. It's very common the data bank to be remote in some way. Like a server. Wifi, ethernet (maybe GSM if you really need it and want to pay the tax). Seen it done many times in the craze of IoT that has been going on.
  • 0 in reply to Luis Afonso
    Guru 117855 points

    Hi again, Luis,

    So good to have you back. As to your "use of SD card" - it's fine for non-volatile storage - AFTER the "Live Data" has been captured into the fastest memory available. (which - as you well know - is "never" an SD card!)

    Firm/my belief (garnered during years "in the trenches") is that, "Capturing Live Data at the Highest possible data rate" is (always) JOB #1. If data capture is voluminous - external RAM can be deployed - the alternative (capturing into ... S L O W ... SD card is (rarely) seen on any "pro/serious" such Data-Logging System.     "Blocking the MCU" as you note - should be "avoided at all costs" - which disqualifies the SD card as the (sole) data storage medium w/in such a system...

  • 0 in reply to cb1_mobile
    Guru 20670 points
    Agreed.

    I guess it's more of me always trying to push the hardware and software to see where it breaks and why.
  • 0 in reply to Luis Afonso
    Guru 117855 points

    Luis Afonso said:
    I guess it's more of me always trying to push the hardware and software to see where it breaks and why.

    That's "well & good" - but should "never" compromise the mission.     Your subject: "Write/Read Optimization" flies in the face of "High Speed Data Capture."     (i.e. Applying even the latest/greatest "lip gloss" to a "pig" - cannot insure "high finish" in the pending, "Ms. America" contest - even when the lips are (nicely) colored & glossed...)     Indeed you may be able to "improve" SD card transfer rates - but to "What purpose?"

    Your, "Pushing HW/SW to "tease out" learning is one means - yet further developing of your "investigative skills" - may prove faster, even better!      This is so as there is "much to be gleaned" from reading the specs & product reviews provided by "market leaders."    Such review may "save you" from techniques and/or implementations which have been properly found to be, "ineffective" - and which may have "eaten" substantial amounts of  (your)  time/effort.

    Luis - sometimes "Choosing what to do" may prove more advantageous than, "Charging forward - minus a proper investigation..."      (i.e. perhaps your "pushing the investigation" - before HW & SW - warrants your consideration...)

  • 0 in reply to cb1_mobile
    Guru 20670 points
    I guess the purpose of me doing that is more of a challenge like some do crosswords or sudoku