This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

LAUNCHXL-CC1352P: Zstackapi.c sendReqDefaultRsp() stuck in infinite loop

Part Number: LAUNCHXL-CC1352P
Other Parts Discussed in Thread: SYSBIOS, Z-STACK

We have implemented a work around in sendReqDefaultRsp() where it is getting stuck in an infinite loop.  See attached and search for "JCI Patch".

While this is working for us (for now), it does not explain what this rogue message is or how it is getting into the queue.

This lockup occurs when our application starts getting busy sending and receiving multiple proprietary messages over-the-air. 

We should work together to find for the root cause of this lockup just in case it is pointing to something more serious. 

- Billzstackapi.c

  • Hi Bill,

    Please provide more information on how to recreate the issue so that we can further evaluate on our side. Can you debug the project and place a breakpoint inside of sendReqDefaultRsp to capture the call stack and packet information of the failed message? What is the device type and is there a specific message which is causing the problem to exist or does this have more to do with network traffic?

    Regards,
    Ryan
  • Attached you will find more information. The device is a line powered router.  There is no specific message, they are all the same (proprietary) message.  This high traffic condition will work several times in a row, then it will get stuck in this loop when we do not have the patch implemented.  With the patch, it always works.

    Let me know when you have more questions.

    - Bill  

    ps:  define DEFAULT_KEY {0x01, 0x03, 0x05, 0x07, 0x09, 0x0B, 0x0D, 0x0F, 0x00, 0x02, 0x04, 0x06, 0x08, 0x0A, 0x0C, 0x0D}

    Lockup.zip

  • Can you share the changes that you made to the sample application in order to get this behavior and at least one of the messages that trigger this infinite loop. This is in order to reproduce the bug on my end.

  • We're not running a sample app. We're running our proprietary production app which is too involved to share.

    The lockup.c file that I provided in the zip is the code that builds the message that is sent to Zstackapi_AfDataReq(). Our code is locked up waiting for a return. This same snippet of code works successfully most of the time, which indicates some sort of timing issue. It may very well be something that we're doing, but we're looking for help.

    We made changes in the zstackapi.c which is attached above which fixes this lockup. One additional thing that we noticed was that...

    pCmdStatus = (zstackmsg_genericReq_t*)OsalPort_msgReceive(appServiceTaskId);

    was returning a NULL, which would get put on the queue. That might be the issue.

    - Bill
  • I'm not able to replicate this issue with the provided function. I'll need you to share the inputs fot BACnet_TransmitPkt() that trigger the loop.
  • We are working to provide you with a simple app that will recreate the issue.  We have a new work-around that appears to be even more robust that we will share once we can recreate the issue.

    Thanks

  • We are working to provide you with a simple app that will recreate the issue. We have a new work-around that appears to be even more robust that we will share once we can recreate the issue.

    We also just noticed that the generic app is setting its semaphore to ti_sysbios_knl_Semaphore_Mode_BINARY and we have set this to ti_sysbios_knl_Semaphore_Mode_COUNTING in our application.

    Is there an issue with using zstack and COUNTING semaphores?

    If we monitor appSemHandle->count when the app gets busy servicing zstackmsg_CmdIDs_AF_INCOMING_MSG_INDs , we will see it jumping up to 300-400 counts at times, then back to 0. This doesn't seem right. Is something in the stack doing semaphore posts in some sort of loop?
  • Hi Bill,

    Do you have any updates to provide? I am not aware of any issues using ti_sysbios_knl_Semaphore_Mode_COUNTING with Z-Stack, this is the mode used by default for messages/events in the SimpleLink CC13x2/CC26x2 v2.40 SDK (zstackstartup.c).

    Regards,
    Ryan

  • Still working on it. We have not been able to recreate the issue starting with a simple app. But we can see similar symptoms which leads to a crash in our case. I hope to have something for you this week.
  • It is unnecessary to be afraid of infinite loop. If your program has entered into infinite loop, there maybe some issue in your user task. You may misuse this OS.
  • Hi Ryan,

    We haven’t been able to recreate this same issue using a simplified generic application, but the attached project does show some of the same characteristics.

    /cfs-file/__key/communityserver-discussions-components-files/158/zstackapi_5F00_fix.docx

    /cfs-file/__key/communityserver-discussions-components-files/158/zr_5F00_genericapp_5F00_CC1352P_5F00_2_5F00_LAUNCHXL_5F00_tirtos_5F00_ccs.zip

    The fix that we implemented has been solid and we have not seen any adverse effects. We'd like to see this implemented so we don't have to maintain these files over the next release.

    Let me know if you have any questions.

    Thanks,

    - Bill

  • Can you try to send the frame that triggers the loop with a fresh new project? we need to find if the issue is present out-of-the-box or if something on your custom application is causing this loop to occur.

  • Jose,

    I'm not sure what you are looking for.  

    We're not able to tell what triggers the sendReqDefaultRsp to loop. All we can do is observe the app's semaphore count ramping up.  

    I've attached the out-of-the-box application.  If you compare this to the code that I previously sent, you can see the simple additions/changes that we made.  Basically all we did was add a very fast random timer that sends a message to the coordinator.  When received, the coordinator turns that message around and sends it back. 

    The second app/task (zcl_genericapp2.c) does the same thing, just at a slower rate.  I'm not sure if the second task is really required, we just wanted to add some more random events into the test.

    /cfs-file/__key/communityserver-discussions-components-files/158/7725.zr_5F00_genericapp_5F00_CC1352P_5F00_2_5F00_LAUNCHXL_5F00_tirtos_5F00_ccs.zip

  • I'm still not able to replicate the issue and there is no count ramping on my tests. On the project where the issue present, are you running two applications as well or this is just for the generic app test?

  • Jose,

    How are you running your test?  Are you at least seeing the rapid fire comms between the two radios?  Did you use the code exactly as it was delivered?  The code is for the Rev E chip, so you can't use the standard 1352P Launchpad (unless you have one that's not available to us).  You also won't see the semaphore ramp up unless it is configured as Counting.

    semParam.mode = ti_sysbios_knl_Semaphore_Mode_COUNTING;
    Semaphore_construct(&appSem, 0, &semParam);
    appSemHandle = Semaphore_handle(&appSem);

  • Hi Bill,

    I'm using the code that you provided with the changes for sample apps and main.c and the semaphore is configured for countingI just didn't include the fixes for the stack that you provided expecting the issue to pop-up put I'm not able to replicate. Please look at the attached OTA log.

    sendReqDefaultRsp-NO LOOP.zip

  • From the log, it appears that you are only running the application on the Router.  If you run the application built for the Coordinator configuration, the Coordinator will echo each received message back to the Router.  You can then monitor the Semaphore count on the Coordinator and see the problem.

    Here's an example of what the OTA should look like.

    /cfs-file/__key/communityserver-discussions-components-files/158/Lockup3.zip

    Thanks for sticking with this.

    - Bill

  • Hi Bill,

    As an update, Jose has been able to reproduce the issue and is currently investigating the MAC libraries to determine the root cause. We will provide an update as soon as we are able to.

    Regards,
    Ryan
  • Thanks for the update.

    I did just download the 3.10 SDK and noticed that there is a change to the osal_port relative to adding items to the queue [OsalPort_msgEnqueue()].  I haven't tried it, but this may be related and could possibly fix the ramping up of the semaphore count. 

    The other part of the problem in zstackapi is the way the sendReqDefaultRsp() [and others] waits for the response.  Pulling a message off the queue and putting it back on the bottom gets messages out of order which can potentially cause other problems.  Our proposed fix addresses that too.

    Anyway, keep me informed and let me know if there is anything you like us to try out.

    Thanks,

    - Bill

  • Hi Bill,

    I apologize for the delay, here is what has been discussed internally since the last update:

    - The behavior can be recreated using the provided projects, however removing CPUdelay(10000*16); from zcl_genericapp2.c appears to resolve the issue. Such a hard delay is not recommended as it can be easily seen as to how this would interfere with over-the-air communications.

    - Concerning semaphore counting modes and priorities, tasks of lower or same priority are starved during the CPUDelay, but higher tasks like the Swi and Hwi will still run as needed. The CPUdelay could cause the mismatch in pends and posts on the two tasks which drives up the count.

    - Given the above, it is believed that the issue is caused by the application but then masked by optimizing the OsalPort solution. R&D recognizes the improvements implemented and are further evaluating so that it can be implemented in a future SDK.

    Regards,
    Ryan
  • Hi Ryan,

    The CPUDelay was put there to simulate work that the task could be doing. How is this any different than just doing a bunch of processing in a loop?  Also, since both the genericapp and genericapp2 tasks are set to priority 1, I don't see how they could interfere with anything going on with the stack running at priority 3. 

    - Bill

  • Hi Bill,

    CPUDelay does not disable interrupts, so only tasks of lower (or the same) priority are starved. So if a priority 1 task is calling it, other priority 1 (and Idle) tasks will not run during the call. Higher tasks, Swi and Hwi will still run as needed.

    You are correct then that the Z-Stack task (priority 3) should not be affected. However the other application task is the same priority and thus affected. It is possible that this task may be blocked enough such that its semaphore count overflows and creates the issue observed. This is difficult to imagine with a semaphore count allocation of uint16 but the delay (10 ms) is quite large and the ZR does send multiple ZCL messages to the ZC at a rapid speed. You could further evaluate by increasing the first application task's priority to 2.

    Regards,
    Ryan