• State Resolved
  • Locked Locked
  • Replies 6 replies
  • Subscribers 83 subscribers
  • Views 496 views
  • Users 0 members are here
Support feedback
Options
Options
Related

This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

Concerto Communication Optimization

Ronaldo Pace
Intellectual 875 points


Hi guys,

I'm developing a huge project on the Concerto controller and it's mostly working fine, I2C, Ethernet, serial, EEPROM, analogue inputs, ePWMs, etc.

But I'm starting to reach some funnel on so many communications operating in parallel and not sure if there are tricks I could do optimize it, so I came to ask the experts.

and please guys, I know it's a big topic, but I know there are a lot of experts and smart TI employees around the forum, so come on, I need this help!

What is happening is that with the latest addition of some constant pooling on the serial, the other serial and the ethernet became considerably less stable and constantly locking up one comm channel or other.

Let me explaining how is setup:
I upped both cores stack_size, M3 have 8K available and  C28 have 1K. As you'll see some specially large buffers are pre-allocated on S-RAM. Both applications are being compiled with Full Symbolic Debug (because it's still being debuged) and no optimization as I was planning to put those to the maximum closer to the end of development. 

There is a separate buffer for in/out uart0/uart1 (4 in total) and all of them with 256bytes each located at S7 using PRAGMA
There is the whole S8 (using pragma) area for the Ethernet response buffer which can reply 2 to 3 ethernet packs.

All comms parsing is done by the same code, no matter from which com it arrived. The comm parsing code is built upon cmdline.c that TI provides.
For sure WRITE_EEPROM and READ_EEPROM commands are effectively the slowest ones because the comms parser have to communicate with the EEPROM over the I2C before replying to it. But what mostly happen is one READ_EEPROM when an external device connects to the Concerto and it might or not send further READ/WRITE but with several minutes of space between them. 

UART0 and UART1 input interruption handlers buffers the chars while (UARTCharsAvail(UARTx_BASE)) and once the message termination char is received it adds a char 0 at the end of the buffer and pass as pointers its input and output buffer to the comm parsing routines. If the parser reply is > 0 then it sends back the message stored on the output buffer, or a fixed "BAD COMMAND" if =< 0

UART0 also instantly echoes back each char received, so it can be seen on the terminal. 

UARTs reads and echoes uses UARTCharGetNonBlocking and UARTCharPutNonBlocking and UARTs comm parsed reply uses UARTCharPut as on the beginning of the developing I saw some issues on it, maybe I should revert it back? Could I increase UART internal buffer to be able to use NonBlocking? I'm not sure on this.

Ethernet handling is copied from the uip_eth example and honest to God I tried but I can't figure how to move it to a timed interrupt or comm interrupt without breaking it, so it's running inside my main loop, and it's the only thing on the main loop.
The ethernet input comes from the same ucUIPBuffer as in the example uip_eth and the cmdline places answers on the S8 memory and a small routine decides if the whole buffer can go in one package or divide it into 1400bytes long packages, uip_send() them and updates the pointer to the next start-of-package to be processed on uip_acked(). At the end of a reply it sends a uip_close(). For the next poll whatever decide is communicating with it have to open the connection again.

So on this mess of comms all was working fine:
UART0 was debug only and barely used at this advanced stage of development;
UART1 was sending a message every 500ms to a connected LCD to update it's values and only receiving anything if the user touched the touch-screen, which was just Start, Stop and Reset commands, almost never)
Ethernet was the one working harder as it was connecting to the engineering station that was pooling values 3 times a second and the user (me) sending commands every once in a while.

The problem started when (as a purely development and testing tool before the final electronics boards be ready for testing) I started sending every 310ms some extra info (32bytes) over UART0.
Now I have Ethernet randomly stopping,  the screen times-out every once in a while, even the UART0 stops receiving the values.
I can't see as it's being some memory corruption thing cause every communication channel have it's own input and output buffers that is passed as parameters.

I'm about to test with compiler optimization, remove the echo back from UART0 and change the UART send to NonBlocking.
But I'm not very confident that any of those will have significant impact on the those comms randomly failing, and I reckon that removing the NonBlocking will give me headache again on the uart.

So, do the experts have some good comments, suggestions, tips-n-tricks, or anything to help me on this one?

I still have in a near future make the board be able to handle communication with more than one device constantly pooling the ethernet and with current problems I don't see it happening.

thanks!!!

  • 0
    TI__Mastermind 19810 points

    Hi Ronaldo,

    nice applciation. So if you stop sending the extra 32 bytes the problem goes away?

    Ronaldo Pace said:

    The problem started when (as a purely development and testing tool before the final electronics boards be ready for testing) I started sending every 310ms some extra info (32bytes) over UART0.
    Now I have Ethernet randomly stopping,  the screen times-out every once in a while, even the UART0 stops receiving the values.
    I can't see as it's being some memory corruption thing cause every communication channel have it's own input and output buffers that is passed as parameters.

    I would start by looking at below;

    1.> where is the code getting this extra 32 bytes from, which buffer? or is the buffer being overflown. If you are using same buffer across multiple tasks, I would suggest adding a semaphore control for mutex operation.

    2.>  Timing - try plaing with the priority of tasks and see how things change, it could give you a hint.

    3.> Interrupt priorities.

    4.> is the code running out of stack?

     

    Best Regards

    Santosh

  • 0 in reply to Santosh Athuru
    TI__Guru* 92780 points

    I'm wondering if a scheduler would help in this case - then you could have both hardware and software tasks.  There are also some nice analysis tools in sys/bios that will help you visualize where the bottlenecks are.   

  • 0 in reply to Lori Heustess
    Intellectual 875 points

    Hi guys,

    I feel a bit embarrassed now, how come I miss that .... sorry guys, honestly!

    Santosh,

    1. Yes without the extra data arriving the problem goes away.
    2. & 3. Even thou it's fixed now, I would like to know more about it. How do one prioritize timer and events interruptions?
    4. yes. Stack. The M3 stack I had to increase a few times during the project development due to all the stuff it's doing.

    Lori, 

    unfortunately, everything is running on stack, I didn't use sys/bios. And I guess scheduler and the analysis tools are just for sys/bios, right?
    Just for my future reference, a project that size would be better to be done using it? 

    Answer to the data corruption mystery:

    This extra data is simulating the high precision analog inputs that the final electronics will have via the I2C and is being sent from the laptop via the debug USB/serial connection and it ignores all the echo and responses that the MCU is sending back. I'm using this simulated input so we can carry on testing the hardware until we have the proper boards with all the electronics.

    At the end, the whole problem was not the Concerto not handling the load, but.... wait for it...... wait for it .....  the laptop was not handling all the load.
    This data is coming from a high precision data logger that uses some very old VB6 style .dll for the communication. And as it happens the run(), isReady() and getValues() routines from the .dll completely suck all the CPU from whatever computer you're running it in. So to solve the problem I put the thread where those run in the lowest Windows priority as possible and increase the sleep() that I had on the "While Not isRead()".

    What was happening is that the dll methods sucked all the CPU time from the laptop, the background thread sending the data to the MCU was already getting corrupted and sending garbage to the MCU, the MCU was not properly parsing the values and the LCD updates were freezing, and back to the laptop, even thou the MCU reply to all Ethernet messages form the Engineering GUI, the GUI itself was missing codes because it is running on the same computer.

    Thank you guys for the replies, I'm really sorry for the misleading thread.

  • 0 in reply to Ronaldo Pace
    TI__Mastermind 19810 points

    Ronaldo,

    interesting debug, I'm glad you thought out of box and looked for the problem at the host, otherwise these problems could eat up days together because we hardly suspect hosts to fail.

    Regarding interrupt priorities it is explained in the device TRM, section 27.5.16. As Lori suggested adding a RTOS support to your project would modularize and improve the performance a lot.

    Best Regards

    Santosh

  • 0 in reply to Santosh Athuru
    Intellectual 875 points

    Santosh,

    yep, start from the beginning, check what is being sent, what is being received, etc, etc.

    I'll check the priorities. And would u be able to tell me how difficult would be to migrate an existing code to RTOS ?
    I'm asking because I guess I might have time to implement it after I finish the Android application that will be used to monitor the system.

    Thanks,

  • 0 in reply to Ronaldo Pace
    TI__Mastermind 19810 points

    Ronaldo,

    Migrating the existing non-RTOS code to RTOS could be a major work and depends on how your current SW is written and how much you can re use. I guess you will be starting with an RTOS which is already ported on to the device. After you pick the RTOS that meets the system needs, you will have to identify SW entities in you current application which can operate independently after the initial set up is done. For ex: after Ethernet is Initialized, you could have a receive task, a transmit task and a timer call back task each to handle received Ethernet packets (responsibility of this task begins when there is a packet in incoming queue, queued by the ETH RX INT handler and the responsibility is completed when the packet is handled and queued into the application RX queue), Ethernet packets to be transmitted (responsibility begins when application queues a packet in TX queue and ends when Ethernet TX Complete event is signaled by the ISR), time out envents handled by timer task respectively.

    Depending on your system resources you can combine two or three operations into a single task, and you will have to identify and code the tasks - here you an probably re use some of the SW you already wrote. Next comes the queues and semaphore and memory management. Depending on the RTOS you ahve chose you can use the built in memory management and task synchronization modules.

    Next thing will be writing your ISRs to work with the RTOS, you would normally write the ISRs to quickly finish the job and leave the major data copying or other time taking work to the RTOS application tasks, some RTOS provide infrastructure to register and handle the ISRs for some you can use your own.

    Thats a quick preview on the work and it helps because you can set priorities for each task and have mutex operations when operating on shared data. Having an RTOS scheduler helps because you can have a high priority taks pre-empt the lower priority task depending on the current system events/state.

    hope it helps.

    Best Regards

    Santosh