• State Resolved
  • Locked Locked
  • Replies 8 replies
  • Subscribers 31 subscribers
  • Views 2165 views
  • Users 0 members are here
Support feedback
Options
Options
Related

This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

Implementing a database on the SD Card

ROM_Jamie
Intellectual 810 points


I've been trying to decide how I will log and query data on the SD card. So far I've just been creating text log files and haven't fully decided on how I will be querying them. It seems somewhat non-trivial to get data from the last X loggings or from a range of loggings. A lot of seek, read, reverse seek, with the possibility of having to traverse multiple log files to get a range (I archive log files after a certain size so the last 10 loggings will probably be in log.log, but may be split over log.log and log.log20140819)

I was wondering if anyone has had luck implementing an existing DBMS? Porting SQLite or NewLib (as suggested here, http://stackoverflow.com/questions/11453404/database-without-operating-system-required-for-embedded-system)

  • 0
    Guru 27665 points

    SQLite might be using a sledgehammer to swat a fly in this case.

    If you are just looking at doing searches like the last 10 records or records 40 to 100 then simply changing to fixed length records solves the problem.  Reading any record or set of records if fairly trivial with such a scheme.

    For slightly more complex work you can add an index scheme.  There's probably some publically available DBase clones that you can borrow from to provide fixed length records with indexing if you really need that level of complexity for log files.

     

    Robert

  • 0 in reply to Robert Adsett
    Intellectual 810 points

    That's more or less what I was thinking but I thought I may be in trouble if I wanted to start querying based on something like a time stamp, which is looking fairly likely.

    I may have data coming in on an irregular interval with a time stamp, I don't know how I could go about extracting data from a range of time without reading each line in reverse order until I hit the time range.

  • 0 in reply to ROM_Jamie
    Guru 27665 points

    If it's time stamp you are needing to search on, then since the data is ordered, a simple binary search will work quite effectively. If your files are timestamped you can further reduce the amount of data you have to search to find the range.  And finally you could maintain an index or even a partial index in RAM (say keep 1000 evenly spaced time stamps with log file and offset) and reduce the search time further (to RAM lookup for a full index).

    There's always the option of copying the log file to a PC with a 'real' database engine to get full indexing and search capabilities.  If you are searching by timestamp you probably don't need real-time access at the same time.

    Robert

  • 0 in reply to Robert Adsett
    Guru 27665 points

    I suspect the binary search would be sufficiently fast.  Before trying something complex measure the performance of a simple solution.  You might find that even a linear search is more than fast enough if there are other performance constraints.

    If 100mS is fast enough to satisfy the user's needs don't spend a lot of time trying to make it work in 1mS.  Simple code is usually more robust.

    Robert

  • 0 in reply to Robert Adsett
    Guru 47900 points

    Robert Adsett said:
    Before trying something complex measure the performance of a simple solution.

    Gold Star #1

    Robert Adsett said:
    If 100mS is fast enough...don't spend a lot of time trying to make it work in 1mS.

    Gold Star #2

    Robert Adsett said:
    Simple code is usually more robust.

    Gold Star #3   (although this reporter cannot resist: Simple code is (usually) Robust Code...)

    Some here have been suffering, "Robert withdrawal" ...welcome your return...

  • 0 in reply to Robert Adsett
    Intellectual 810 points

    Thanks for the pointers Robert.  I'll see if I can figure out the binary search method.

    The efficiency requirement boils down to increasing battery life for my app. It's something that will be running for extended periods of time and communicating the results of these queries wirelessly. The tighter I can make it the longer it lasts in the field. When the battery dies it could be a while before it's charged again.

  • 0 in reply to ROM_Jamie
    Guru 47900 points

    ROM_Jamie said:
    requirement boils down to increasing battery life for my app.

    You may consider vendor's MSP family - not an ARM but long famed for very low power...  Historically - this vendor's ARM MCUs - not so much...  (new family has improved - yet a critical comparison/review of MCU power specs seems (now) very much indicated...)

  • 0 in reply to ROM_Jamie
    Guru 27665 points

    ROM_Jamie said:
    The efficiency requirement boils down to increasing battery life for my app

    Then you'll definitely want to measure the energy usage (as opposed to time).

    And thanks for the welcome back cb1

     

    Robert