This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

Embedded HTTP Web Server (NDK) stops upon Task Block/Malfunction

Other Parts Discussed in Thread: SYSBIOS

MCU: TM4C1294NCPDT

TI-RTOS: v2.01.00.03

NDK: v2.23.01.01

CCS: v6.0.1.0040

-

Hi,

  I have a firmware running on Tiva C MCU on top of TI-RTOS. The firmware also makes use of Embedded HTTP Web Server based out of the example as illustrated here  .

  In the code, I have multiple tasks running simultaneously. The tasks have priorities of 3, 5, 7, 9, 10 & 12. What I mean is that, the task priorities are selected between these few numbers only. For example, there are 10 tasks which have priority as 3 & 2 tasks which have priority 7, and so on. Now, due to some coding error (or something else which I need to diagnose), one task having task priority as 3 is executing in an infinite loop without any OS function call like Task_sleep() or Semaphore_pend() or Mailbox_pend(), etc. This is not happening at the beginning of the firmware running. This occurs after about 2 hours of continuous firmware run. This will get corrected if the hardware is restarted.

  Now, in this situation, I find all higher priority tasks are running; whereas all tasks having priorities lower than or equal to 3 are getting blocked. Due to some urgency in delivering the code, I have reduced the priority of this erroneous task from 3 to 2. After doing this, I find that the task (whose priority I reduced from 3 to 2) has no effect on any other task as all other tasks are having priority greater than 2.

  But, surprisingly, I find that the Embedded Web Server stops working once this phenomena (as explained above) occurs. Can anyone suggest where to look for? I have the following settings in the .cfg file, is any change of parameters required in the below mentioned script of the .cfg file?

EMAC.libType = EMAC.LibType_NonInstrumented;
Global.IPv6 = false;
Ip.autoIp = true;
Ip.address = "";
Ip.mask = "255.255.255.0";
Ip.gatewayIpAddr = "192.168.1.2";
Ip.domainName = "domain.com";
var http0Params = new Http.Params();
var http0 = Http.create(http0Params);
Global.lowTaskPriLevel = 3;
Global.stackInitHook = "&AddWebFiles";
Global.stackDeleteHook = "&RemoveWebFiles";

Global.networkIPAddrHook = "&mynetworkIPAddrHook";
Global.networkOpenHook = "&functionNetworkOpenHook";

Global.netSchedulerPri = Global.NC_PRIORITY_LOW;
Global.ndkThreadStackSize = 2048;
Global.lowTaskStackSize = 2048;
Global.normTaskStackSize = 2048;
Global.highTaskStackSize = 2048;

  Also I find the following settings, is any change in this section help in resolving the issue, can anyone tell me why this even happens (the HTTP server non responsive)?

TI-RTOS -> Products -> NDK -> Networking - Scheduling Options -> 

Network Scheduler Task Priority: Low Priority

Priority Level for Low Priority NDK Tasks : 3

Priority Level for Normal Priority NDK Tasks : 5

Priority Level for High Priority NDK Tasks : 7

Priority Level for Kernel Priority NDK Tasks : 9

-

NOTE: I do not have JTAG access to the hardware. The bootloader is permanently loaded on the MCU & new firmware is loaded using USB Thumb Drive and/or remotely via Ethernet/LAN or GSM/GPRS modem interface.

-

Regards

Soumyajit

  • Soumyajit,

    Just to be clear… after you’ve bumped the priority of the task that gets stuck to 2, you still see the web server stop working, right?

    If yes, is it possible this stuck task at priority 2 is improperly interacting with NDK?  Or could it have masked or disabled interrupts?

    If the stuck task doesn’t yield then the Idle task won’t get to run.  Does the app have any Idle functions that need to be run, at least occasionally?

    Thanks,
    Scot

  • Hi Scott,

                    Thanks for your reply.

    1. Yes, reducing the malformed task’s priority from 3 to 2 still makes the NDK non responsive.
    2. This malformed task is not interacting with NDK & it doesn’t have control over interrupts. It basically collects data from an UART port, analyses the data & updates some global variables.
    3. I remember that during some driver implementation (may be the HTTP server of NDK), I had to enable the idle Task. May be it was required to be enabled. My codes do not require the idle task to run as all are put in Tasks with priority above the idle Task. But the particular driver required me to insert “Task.enableIdleTask = true;” in .cfg file.

    To investigate further, I opened TI-RTOS -> Products -> SysBIOS -> Scheduling -> Idle Basic Options -> All Idle Functions. Here I find the following functions added to the list.

    • ti.sysbios.knl.Task.deleteTerminatedTasksFunc
    • ti.sysbios.smp.Load,idleFxn
    • ti.uia.sysbios.LoggerIdle.idleWriteEvent

    But I don’t know how these functions (if at all) are related to NDK!!

    -

    Although, this may not be a proper solution, but for the time being, I have added "Task_sleep(1)" in the malformed task so that the infinite loop in the Task is forced to pass through the Task_sleep(1). This seems to somewhat resolve the issue but I suspect that NDK might be running slow which we may face/realise in future after thorough testing of the system. Also, I may sometimes miss characters/bytes coming in through the UART port.

    -

    Regards

    Soumyajit

  • Soumyajit,

    Thanks for the additional information.

    In your list of Idle functions you have: ti.sysbios.smp.Load,idleFxn

    The ti.sysbios.smp.Load module is for other, multicore ARM devices, and should not be used on a Tiva device.  I think you probably wanted the ti.sysbios.utils.Load module instead.  

    If you change this, or disable this, does the problem with the task still occur?

    Thanks,
    Scott

  • Hi Scott,

      I am unable to disable/change these functions, they are grey shaded in the .cfg GUI, (See attached images). Moreover, they don't exist if I am viewing the .cfg file in a text editor!! Can you help me in changing them? Where to go to enable editing them?

      Also, the checkbox "Add the idle function management module to my configuration" (as shown in the first image) was unchecked initially & all the User Idle Function fields were grey shaded too (cannot be edited), I ticked on this checkbox to enable editing them!

    -

    Regards

    Soumyajit

  • Hi Soumyajit,

    I was out of office, sorry for the delay.

    Thanks for the screenshots.  These idle functions are being pulled in by some other modules that are configured for the application.  Can you post the full text of your .cfg file?   If you can’t post it directly to the forum you can send it to me privately by first sending me a forum friend request.

    Also, over the weekend I was wondering if you’ve tried replacing the task that gets stuck with a simple infinite loop.  This will keep any same or lower priority tasks from running.  It seems from your description that the rest of the application should continue to function.  You will obviously be missing the functionality of the task that gets stuck.  But it will stop whatever this task is doing that results in it getting stuck.  That should be a good data point.

    Regards,
    Scott

  • Hi Scott,

     I have tried two things as described below:

    1. Replacing the malfunctioning task with "while(1){Task_sleep(1);}", this allowed all other Tasks to run properly including the NDK.

    2. Replacing the malfunctioning task with "while(1);", this blocked all Tasks having priority equal to or lower than this task. Even the NDK stopped functioning.

    -

      I have sent you a community friend request. Please accept the same, I will send you the .cfg file via private message.

    -

    Regards

    Soumyajit

  • Hi Soumyajit,

    Thanks for sending the .cfg file.  A few things:

    1) The smp.Load module is explicitly pulled in by line 57 in the .cfg file:

    var ti_sysbios_smp_Load = xdc.useModule('ti.sysbios.smp.Load');

    This line should be removed.

    2) There are two tasks (not one) set for priority of “2”.  Spinning in the first task will also block the second.

    3) Just to confirm, since their priority level is being set lower than Global.NC_PRIORITY_LOW, the two tasks at priority “2” are not making any NDK API calls, right?

    4) Global.lowTaskPriLevel is set to “3”.  I think this means that the NDK scheduler will be running at priority of one less, or “2”.  So if your priority “2” tasks are not yielding then the NDK scheduler task will not get to run.  If you are still seeing issue after removing the smp.Load module, in your experiment, can you bump the “stuck” task (or the while(1) substitute task) to a priority of “1” and see if NDK continues to function in that case?

    Thanks,
    Scott

  • Hi Scott,

    Thanks for your reply once again.

    1. Commenting out "var ti_sysbios_smp_Load = xdc.useModule('ti.sysbios.smp.Load');" throws the following error: "Undefined reference to 'ti_sysbios_smp_Load_getCPULoad__E' in file ./fatsdusbcopy.obj .xdchelp /DL_Tiva_v2 C/C++ Problem". Is this xdc Module inclusion has anything to do with the function by which I am getting the CPU usage in my code?

    2. Yes, you are right. Since both the task are inter related in my application, I just reduced both their priorities from 3 to 2, just in case the other one also causes any issues.

    3. Yes, those two tasks at priorities '2' are not making any NDK API call.

    4. Since, I am unable to remove smp.Load module, I did the following for experimental purposes.

      I put "while(1);" in the task with priority 2. As usual, the NDK got blocked.

      Now, I made the task with priority 2 to priority 1 while "while(1);" is still there in code. Now I find that the NDK is fully functioning.

    -

      Can you suggest me a way of figuring out whether NDK is blocked or not. I'll explain you what I meant. In my application, one high priority task is monitoring all low priority tasks. This is accomplished by making multiple global variables increment by one in all tasks in each loop re-entry. If the high priority task finds that a particular task's re-entry counter is not changing/incrementing for long (say 60 seconds), then the high priority task will understand that the particular task got blocked & depending on requirement, may issue a system restart.

      Can you suggest a way of implementing such a thing in NDK? In NDK, is their any task that is running continuously (may be at a low priority) where I can put something like "counter_NDK++;"?

    -

    Thanks

    -

    Regards

    Soumyajit

  • Hi Soumyajit,
    OK, thanks.
    Where is this fatsdusbcopy.obj coming from?  If it is referencing ti.sysbios.smp.Load it seems it was built for a different (multicore) ARM processor.  I think we need to focus on that first, and get the SMP load module out of your program. 
    Regarding detecting if NDK is blocked… Unless you modify the NDK sources, I don’t think there is an easy way to check this.  One thought is to add another task at the same priority as the NDK scheduler task, that simply increments the counter and periodically sleeps.  That way you’d at least be able to see if the NDK scheduler task is blocked from running.
    Regards,
    Scott
  • Hi Scott,

    1.  When I started developing this project, I imported the Tiva C TI-RTOS example code named FatSdUsbCopy from CCS Resource Explorer as a starting point. Then gradually I kept on adding my own code as well renaming the project to "DL_Tiva_v2". At that time, two object files were created in the "Debug" folder of the CCS project. The two files are "EK_TM4C1294XL.obj" & "fatsdusbcopy.obj"; the same exists today. The same applies for the .c files too (see attached image).

    2.  Ok, if I have to add a new task having priority same as the NDK task, then what should be the required task priority; I am asking this because the NDK itself runs on different task priorities like LowTaskPriLevel, NormTaskPriLevel, HighTaskPriLevel & KernelTaskPriLevel.

    -

    Thanks

    -

    Regards

    Soumyajit

  • Hi Soumyajit,

    1. After deleting the line in the .cfg file did you try cleaning and rebuilding the project?  It seems that fatusbobjcopy.obj still somehow has reference to ti.sysbios.smp.Load. (I only have newer versions of TI-RTOS for TivaC installed, and don’t see that FatSdUsbCopy example.) If clean and rebuild doesn't work, can you check your sources for references or include of a header for ti/sysbios/smp/Load.h?

    2. I think you want to set the priority to one less than what you are setting for Global.lowTaskPriLevel.

    Regards,
    Scott

  • Hi Scott,

      Thanks for the reply.

    1. Yes, "#include <ti/sysbios/smp/Load.h>" is there in the code. If I comment it out, then the "Load_getCPULoad()" function won't work. Can you suggest a workaround?

    2. Okay.

    -

    Thanks

    -

    Regards

    Soumyajit

  • Hi Soumyajit,

    Sorry for the long delay.  I’ve been working on a product release and am just catching up on email.

    Instead of the SMP Load module, I think you can use the non-SMP version.

    In your .cfg file replace the useModule for SMP Load with this:

        var Load = xdc.useModule('ti.sysbios.utils.Load');

    And in your .c files change the #include to:

        #include<ti/sysbios/utils/Load.h>.

    Does that work?

    Thanks,
    Scott

  • Hi Scott,
    Thanks for the suggestion. Yes, it did work fine.
    -
    Thanks
    -
    Regards
    Soumyajit