• State Resolved
  • Locked Locked
  • Replies 11 replies
  • Subscribers 33 subscribers
  • Views 2458 views
  • Users 0 members are here
Support feedback
Options
Options
Related

This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

Best practices for timers, sensor data storage, and GPRS transmission?

Lee Shaver
Intellectual 390 points
Other Parts Discussed in Thread: EK-TM4C123GXL

I am using custom hardware driven by a TIVA TM4C (specifically, EK-TM4C123GXL) to control a microgrid with battery energy storage, solar panels, four loads controlled by relay, and a SIM900 GSM GPRS module. Users can push buttons to toggle relays, or send an SMS to toggle relays.

Three ADCs are recording battery voltage, total load current, and total load power.

I want to periodically log the relay states and ADC voltages - say once per minute. Each time the user toggles a relay (either by switch or SMS) I want to log this as well.

Then, every hour or so, I will send this data as a packet of data points via GPRS to my server.

With all of this data I can compare to solar radiation databases to see if my microgrid management algorithm is performing optimally (allowing full use of power during the day, preserving enough power through the night, keeping the batteries healthy, acting on SMS input promptly, etc).

I am writing C code using Keil u4.

Here are the questions/concerns I have:

  1. Do these time intervals make sense? Should I log more or less often? Transmit more or less often? What questions should I ask to figure this out?
  2. Should I use EEPROM to do this, or just hold the values in memory for a while before transmitting? The system is in a lab environment so I'm not too concerned about external factors upsetting hardware.
  3. What's the best way to implement timers? I have several 32-bit timers with 16-bit subtimers, and several 64-bit timers with 32-bit subtimers. I'm unclear on how/why to use different timer arrangements for different tasks.

I've worked with each of these pieces individually, but wondering what the best means to implement them all together is.

  • 0
    Guru 47900 points

    May I note your post as very well conceived - drawing in multiple inputs most always proves worthwhile - and often detects issues/methods which would not have "bubbled up" - had you not broadcast here.

    I've "instant" ideas but better to reduce these to key questions - enable my further probe for items of substance - which may answer your concerns while (perhaps) expanding (or tightening) your design, goals, use-case & implementation...   (like "Arnold"...I'll be back...)

  • 0
    Guru 27665 points

    A few comments, maybe more later.

    Lee Shaver said:
    Should I use EEPROM to do this, or just hold the values in memory for a while before transmitting?

    Read the EE errata, consider external storage. It's fairly easy to estimate the amount of storage you need you need if you have a known sample rate and size. The good news is if the micro doesn't trip on a watchdog or something similar you should have a fairly hefty backup. An SD card lets you gather data by hand even if it fails to make its way across the network.

    Lee Shaver said:
    Do these time intervals make sense? Should I log more or less often?

    The issue here is what level of detail you want to capture. Insolation can change very rapidly (cloud cover etc..), the battery hopefully not so rapid. For average you are probably fine but for quick changes you need to evaluate (maybe with an actual sensor). Solar radiation databases are unlikely to be faster than 1 minute and probably not strongly correlated to your position even if they are, even at 1km separation correlation is quite small on a short term basis.

    Lee Shaver said:
    What's the best way to implement timers?

    Determine the needed

    • accuracy
    • precision
    • range

    Then you can determine what you need to do with the timers.

    Robert

  • 0 in reply to Robert Adsett
    Intellectual 390 points
    Thanks for the feedback, Robert. cb1, I'm looking forward to your input.

    DATA STORAGE:
    I would prefer to use external storage but at this point it's not an option - the hardware is deployed and I'm not able to make any changes for a while. I am leaning towards EEPROM since there will be some level of backup, but I'm concerned about how writing this much data - four 32-bit integers stored each minute for two-three weeks of the study period. Even if I'm cycling through the whole EEPROM block so that I don't re-write one interval too much, I'd need to use one to hold a starting address of some sort - though I suppose I could send this info to my server each time and then just get the last EEPROM address back from the server in case of power failure.

    INTERVALS:
    Your question lead me to check into solar insolation sources for the study site - Mysore, India - and I found there is nothing better than daily averages. So, recording once per minute will be more than sufficient. I'll just be comparing to sunrise/sunset times and historically averaged hourly capacity factors.

    But what about my transmit intervals? how do I figure out if once per hour is sufficient, or too much?

    TIMERS:
    Accuracy/precision are not my chief concerns - anything within a few seconds will be fine. I'm more interested in analyzing trends.

    The range, of course, depends on how I end up selecting intervals. At this point, I'd need one timer cycling once per minute, and another once per hour.
  • 0 in reply to Lee Shaver
    Guru 10570 points
    Ok, for that kind of comparison once every 5minutes is probably more than sufficient.

    EE, really check the errata for EE and Flash. They are not minor. As for marker, you don't need a fixed index, a simple rolling index in front of each entry works, you just look for the end of the stored records on start. Consider if you need an integrity check on the record.
    A single timer should be sufficient, my concern would be drift especially considering the temperature variations you are likely to see.

    Transmission is over an inherently unreliable medium. If your transport protocol doesn't have a reliability layer built in you will need to add it and account for retries. I don't know about acceptable transmission sizes but I'd be tempted to err on the small side to start so if reliable transmission turns out to be an issue you'll have a better chance of getting each packet through and less overhead on a retry. Cb1probably has more experience here than I do.

    Have you considered adding an error log?

    You haven't burdened yourself with the overhead of updates via the phone network I hope? That would probably at least double the complexity.

    Robert
  • 0 in reply to Robert Adsett72
    Guru 117855 points

    Friend Robert has, "Stolen most of my thunder" yet I'll soldier on.

    I'm in agreement w/most of Robert's points - but my frame of reference (and interest) tends more to the, "Why, How & Justification" for the project.   Now - it can be said that such is "none of our business" - yet minus that input - our contributions can stray wildly off-mark.   That wastes time/effort - does it not?   So - it's great that you sought help here - not so great that key details were not included! 

    Lee's quote, "...prefer to use external storage but at this point it's not an option - the hardware is deployed and I'm not able to make any changes..." drives a stake into each helper's heart - does it not?   We've a conflicted blend here of: "Already Done" vs. "How best do I do it?"   Pardon - but that's "Krazy Making" - is it not?

    Now - my turn to make you crazy!   Shouldn't this data be (typically known) if not @ your exact location - at least via some reasonably similar locations?   It is always (very) poor form to start w/out ANY baseline!   (even a crude one)   And - would not many of your questions yield to the read/review of data-sheets from similar (or better) equipment - already released?   Further - I guarantee that near identical questions are faced by other - "non-related" industries or applications - and a scan of their literature would prove revealing.   (inexact yes - but you'll learn the basis for their decisions - and that's (exactly) what you're asking for - here.)

    Silent is your list of sensors.   Our firm's preference is for serial-based ones - so that a single cable - or single radio/commo channel - can pass the data.   Just as Robert notes - small packets trump large ones - and multiple transmissions (even when not needed) vastly increase data reception robustness.

    I tend toward design "Over-Kill" it's the rate client who cries, "You did this too well!"   Again Robert's phrase, "Beware Premature Optimization" rings loudly - second only to our phrase, "Use KISS!"   Repeatedly we are hired to rescue projects which were designed "too close to their limits" - and almost never can the (failed) designers justify their (too limited/restricted) approach.   It is always vastly easier to, "Remove components from a WORKING system - than add them to a FAILED SYSTEM!"   

    Again - my friend identified, "Error Log."   I'd go even further - you should set your operational limits fairly tightly - and log, "everything under the (solar) sun!"   At this early stage - How can you know what's important?   Is it degree of, or frequency of - fault(s)  which most concerns?   (or should concern)   And - I'd strive to catch faults, "As SOON AS POSSIBLE!"   This (usually) means monitor FAR BEYOND that which you've identified.   Your ability to predict all of the fault sources cannot exceed ours - and we stink!   (and the MCU is simply great at digesting vast data - over multiple channels - and catching things we mortals miss...)

    As to Timers - the wide timers make sense when the event under scrutiny requires time in excess of the lesser timers' capability.   For critical systems we often run TWO Timers in parallel (commonly clocked) and then compare them.  (proves more difficult to assault two devices!)

    Promise fulfilled - surely Robert (possibly moi) have earned (slight) splash of green...

  • 0 in reply to cb1_mobile
    Intellectual 390 points

    Robert -

    Thanks for the pointers on EEPROM. I will look into the errata and check back with any questions.

    Your point on the timer – you’re saying I could just use a single timer for each interval (five minutes) where I want to store data, and transmit at the same time? Rather than logging data for a bit and then transmitting?

    This does sound like a simpler approach – and the point on reducing overhead when re-trying missed packets is well taken.

    An error log is also a good idea – fortunately due to time constraints it is something I’ll have to push back a bit.

    And no, I’m not taking any updates over the phone network. Right now it’s just handling periodic SMS messages, and the GRPS transmission that I’ll be initiating.

    cb1 –

    I’m happy to add more detail on application. Typically, in forums such as these I find it’s best for the initial post to be to the point – so that readers don’t fall asleep before they get to the end!

    The main purpose of the project is to gather data on how battery voltage of the microgrid system fluctuates over the course of the day, for a period of some two to three weeks. This will accomplish two goals – to determine if the sizing of the batteries and solar panels is sufficient for the loads we are serving, and secondly, whether the voltage setpoints we’ve programmed for disabling loads are appropriate to ensure sufficient battery over-night, without overly limiting use of power during the day. We can also demonstrate to our sponsors, in the form of a chart, how the system responds to user input, both via manual manipulation and the basic SMS commands we have set up for turning relays on and off.

    As further background, I am a graduate student in electrical engineering at the University of Wisconsin-Madison, and this project is part of research we are conducting on microgrid control hardware and algorithms for rural and off-grid environments, particularly in developing countries. The hardware I’m working on was developed a few semesters ago and wasn’t intended to be used in this application originally, but has the requisite components – MCU, relays, GSM module – to be serviceable for research purposes, and inform our design choices as we’re getting a new version of the hardware ready.

    To your point, yes – much of this data is available. But, in December we’ll be taking the next hardware version to a remote school – and we need a proof of concept. As you well know, datasheets can be unreliable, and skim over crucial details.

    Our sensors are rather crudely implemented at this point – all three come into the TM4C via on-board ADCs. Battery voltage uses a simple resistor network voltage divider. For current and power we are using an LT2940 monitor with current mode power and current outputs. Accuracy is not a chief concern here – we’re more interested in trends.

    I’m happy to give green, but like to be further along lest I find myself abandoned!

  • 0 in reply to Lee Shaver
    Guru 117855 points

    Early bird notes "small world" - U of WI @ Madison has long been a client of ours - and in fact was involved w/our "Command/Control Display Module" which (twice) flew w/in the Shuttle. I'm no fan of that town's politics - yet the school is top-notch. And much new VC activity has spread southward - to the S.E. corner of the state. (cannot equal our Chicago activity - which pales when contrasted w/ "SJ.")

    I'd think that the voltage setpoints should be dynamic (perhaps they are) and shift based upon: sun's intensity, load, time of day, etc.

    Five or so years past we participated in a similar project - this one on the US east-coast (CT) and we found that different portions of the solar array were critically impacted by the simple arrival of clouds (which only "partially" blocked the solar array) and the sun's movement. As a defense - much like the monitoring of a series string of Li-Ion batteries - we had to monitor the solar panels (sectionally).   This "partial solar blocking" drove our (and others) electronics Nutz!!!

    Sometimes these panels are mounted such that they may be - w/in limits, "re-positioned" and best directed to harvest sunlight optimally.

    I noted your earlier mention of "4 relays" - unless environmentally sound - we find these - in time - to prove suspect. Would not a 2016 version employ sub-1mΩ Power FETs (as on our BLDC Controllers) - which eliminate the power-hungry relay coil - and relay contact arcing, bouncing, sticking & corrosion?  Those antiquated, power wasting, failure prone devices "fly in the face" of your "Green Initiative" - do they not?


    I continue in the belief that (near similar) data (as you seek) does exist - and that enhanced investigation could make your "proof of concept" efforts even more compelling. (VCs always ask - "Who else does similarly - how is yours better/different - what IS your central Value Proposition?"   Thorough (investigation) most always proves (most) impressive (as it is SO rare - and so valued!)    (never/ever claim "research" - that's the playing field of Abbott, Baxter, Boeing - not you/I or (other) garage...IRS has very strict R&D laws - you/I/99.9% of those here perform D&D - R&D - NO way!) 

    Wish you well - go Badgers (but not so fast/far as UCLA Bruins...)

  • 0 in reply to Lee Shaver
    Guru 47900 points

    Lee Shaver said:
    Typically, in forums such as these I find it’s best for the initial post to be to the point – so that readers don’t fall asleep before they get to the end!

    Is it not desirable that those, "Sleep induced posters" escape to dream-land - and thus cannot respond?

    There's a mid-ground (surely you know) and that's similar to, "Executive Summary" which lists your primary goals, constraints, budgets & timeline.   This is placed "up top" thus sure to be read/reviewed.  

    Yet separate from - and beneath this summary (happily co-existing) - may appear "acres of fine detail" available for those who are really interested - and may be (both) sufficiently motivated & skilled to guide/comment/assist.

    Seizing the moment makes great sense - sometimes you'll have, "Just one chance."   And we've observed that the (enforced) discipline of such an approach yields a vastly improved report (or) forum request...

  • 0 in reply to Lee Shaver
    Guru 27665 points

    Lee Shaver said:
    Your point on the timer – you’re saying I could just use a single timer for each interval (five minutes) where I want to store data, and transmit at the same time? Rather than logging data for a bit and then transmitting?

    I was thinking either

    a- free running timer and you pick of the intervals you need for sampling and transmittal or

    b - transmitting every n sample rather than every x minutes. Where n could be as small as 1.

    Side question: Is this going to have published/publically available results?

    Robert

  • 0 in reply to Robert Adsett
    Intellectual 390 points
    cb1 -

    These are all good question and pointers. The main difference between this system and most others is the cost and scale. We are working with 100W solar panels, 10A relays, 40Ah batteries. All of this is based on a 12V DC back-bone. Eventually we'd likely scale up to perhaps 400W or so, but for the customer base we're looking at, 24VDC is likely the highest we'd go - using anything above 24VDC increases the cost and (more importantly) reduces access. In most parts of the world 24VDC batteries are fairly easy to come by.

    The relays we're using are latching, so power draw is not an issue. And while they do have their issues, they are again cheap, plentiful, and easy to replace.

    Literature review efforts are on-going in parallel with what I'm working on - but alas, time is of the essence! Sponsors will be on campus, physically, in about a month - and seeing the physical results of our physical system will go a long way towards heartening them in their continued sponsorship efforts.

    Robert -

    Thanks for clarifying. I'll likely go with option b. There is an RTC on the GSM module I'm using, so I plan to set a timer to grab a sample every five minutes, grab the clock time, and then transmit the two pieces of data together.

    Still need to figure out how to handle user interactions - it's probably simpler just to transmit this data when it occurs, but I could store it and just wait until the next transmit interval to push it to my server. I'll try the former and switch to the latter if I have any issues.

    At some point and in some form, this data will likely be published - at the very least in a graphical form for the poster session I'll be presenting in May. Why do you ask?
  • 0 in reply to Lee Shaver
    Guru 10570 points
    A few reasons

    I'm naturally curious ;)
    I've been interested in appropriate technologies and alternate energy for something like 40years
    And it's somewhat related to what I do professionally.

    So if it's publicly available I'd like to take a look at it when you're finished.

    Robert