This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

CC3120: Device entering to infinite while loop in _SlDrvMsgReadCmdCtx function and not responding to Host MCU during the cyber security test.

Part Number: CC3120
Other Parts Discussed in Thread: CC3200, CC3100

Hello Ti Team,


I'm facing system hang issue from _SlDrvMsgReadCmdCtx function  and I've debugged inside the code, I observed that CC3120 is not giving respond and _SlNonOsSemGet function is entering to infinite loop. I've already done a good time of researches in this forum related same kind of topic but none of them given any  solution for my issue. 

About test case : we have an embedded device with CC3120 NWP, which is perfectly working for webHMI communication. Before the product SOP we have given our embedded device for Cyber security attack testing and they found 2 major issue. 

Test -1 

DoS attack :It was possible to crash the embedded device by sending a specially crafted IP packet with a null length for IP option#0xE4. When this plugin was rerun multiple times on embedded device, TCP 21,80 and UDP port 53 were down permanently during and after the test. After 30 minutes, it was observed that the client connected to the embedded device was kicked off and the re-connection to embedded device was possible only after the manual reboot of the device.

Test -2

Slow Http test : During Range attack, it was observed that tcp ports 21 and 80 was down permanently during and after the test. Ports were recovered into normal state only after the device was manually
rebooted. During slow read test, it was observed that tcp port 21 was down permanently during and after the test. Port 21 recovered into normal state only after the device was manually rebooted. Port 80
was also found to be down multiple times during the test duration. It is recommended the web server should be able to handle http requests which are send very slowly. As a security measure, set an absolute timeout. Define the minimum incoming data rate and drop connections slower than that rate. Also, some server specific measures like setting connection Timeout, will also minimize the impact of slow HTTP attacks.

When I debug the code :

For test - 1: embedded device is entering to the infinite loop and I have enable Timeout decrements code inside  nonos.c then I got [SIMPLELINK]  FATAL ERROR: No Cmd Ack detected [cmd opcode = 0x9c0b] . later it given  [SIMPLELINK] FATAL ERROR: Driver Abort detected.

For test -2 : embedded device throwing error is [SIMPLELINK] FATAL ERROR: Driver Abort detected..

Device Environment details:

MCU : K61 from Freescale

RTOS : MQX 4.2 

NWP : 3.10.0.5

MAC: 2.0.0.0

PHY : 2.2.0.6

WIFI Chip : CC3120

Embedded device wifi stack driver codes are up to date with NWP version.

Additional information : Code is hanging this way, _httpsrv_server_task --> MUTEX_SECTION(new_sock, sl_Accept((server->sock_v4), ( struct SlSockAddr_t *)&ClientAddres, (SlSocklen_t*)&AddrSize)); -->

sl_Accept -->  VERIFY_RET_OK(_SlDrvCmdOp((_SlCmdCtrl_t *)&_SlAcceptCmdCtrl, &Msg, NULL)); --> _SlDrvCmdOp -->  RetVal = _SlDrvMsgReadCmdCtx(pCmdCtrl->Opcode, IsLockRequired); /* will free global lock */ -->  RetVal = sl_SyncObjWait(&g_pCB->CmdSyncObj, CmdCmpltTimeout); --> _SlNonOsSemGet () -->  infinite Loop.

Could you please give me a solution for this two issues. Our device is using for highly critical monitoring application which won't allow to reset the device. Since our project in SOP stage, need to fix this issue immediately.

I'm looking forward your response.

Thanks in advance.

regards,

Paul

  • Hi Paul,

    It seems that the NWP crashes or at least responds very slowly in this DOS attack (what is your the timeout that you use for the command response? can you try to increment the value?).

    We will try to reproduce and debug this. 

    Can you provide the exact description of the test (what is the setup and what are the tool that you use to create the attack)?

    It seems that you are running and CC3200 (Gen1) driver that you've ported and not the formal Gen2 driver. It seems that is not related to the issue but i would suggest that you try to use our official driver (why did you take this approach to begin with?). We are updating the (gen2) driver periodically  with fixes that new capabilities.

    Have you tried using the SimpleLinkGeneralEventHandler() to reset the NWP or the MCU?

    In general, the Simplelink was designed with focus on protecting data on the device and over the network and not against DOS attacks (so the service might be influenced by such attack), however if the NWP indeed crashes - we will debug and fix this.

        

    Br,

    Kobi

     

     

  • Hi again,

    In addition, it will be helpful if you can provide NWP logs (see http://processors.wiki.ti.com/index.php/CC3120_%26_CC3220_Capture_NWP_Logs) for the failure.

    Thanks,

    Kobi 

  • Hello Kobi,

    Thank you for your kind reply.

    1. what is your the timeout that you use for the command response? can you try to increment the value?.
    Ans: It was 30000 i.e, SL_DRIVER_TIMEOUT_SHORT. Even I tried to use SL_DRIVER_TIMEOUT_LONG (65535) but did not solve the issue.

    2. Can you provide the exact description of the test (what is the setup and what are the tool that you use to create the attack)?
    Ans:

    PFA.

    Test -1

    3com RAS 1500 / Wyse Winterm Malformed Packet Remote DoS Plugin ID#11475.
    Plugin ID is #11475.

    you can refer this link for more information about test . Link : www.tenable.com/.../11475
    its a common open source nessus tool test. I'm attaching pcapng file which used by our cyber security test team. ( we are running this file in Linux system).

    Test -2

    You can refer in google about slow http attack test procedure. Our cyber security testing team told you can download any open source tool to recreate it.

    3. It seems that you are running and CC3200 (Gen1) driver that you've ported and not the formal Gen2 driver.
    It seems that is not related to the issue but i would suggest that you try to use our official driver (why did you take this approach to begin with?).
    We are updating the (gen2) driver periodically with fixes that new capabilities.

    Ans: I have not completely understood your suggestion about Gen1 and Gen2 driver. I've downloaded simplelink_sdk_wifi_plugin_2_40_00_22 and ported CC3120 driver code to host mcu code. Mainly I have taken ti\simplelink_sdk_wifi_plugin_2_40_00_22\source\ti\drivers\net\wifi folder files changes. Could you please guide about Gen1 and Gen2 drivers?  Did I take wrong driver code for cc3120 communication? if yes Please let me know how can I find the correct driver codes (Gen2).

    4. Have you tried using the SimpleLinkGeneralEventHandler() to reset the NWP or the MCU?

    Ans : yes, I have used. I try to reset NWP when Timeout happened and before time out also.
    when time expires I got [SIMPLELINK] FATAL ERROR: No Cmd Ack detected [cmd opcode = 0x9c0b] . later it given [SIMPLELINK] FATAL ERROR: Driver Abort detected.
    later I have tried to restart NWP (without resetting MCU ) I'm getting below error .

    Debug Log 

    Timeout =1
    Timeout =0
    restarting required
    KillWifiTask done

    [2000-2-23 18:19:37.250] [SIMPLELINK] FATAL ERROR: Driver Abort detected.
    [2000-2-23 18:19:37.256] [HTTP Service] [sock 80] Socket Accept -2005 ERROR!!!
    [2000-2-23 18:19:37.257] [HTTP Service] [sock -2005] ERROR!!! Can't close HTTP socket.
    [2000-2-23 18:19:37.275] [HTTP Service]
    HTTP Server exit!!
    [2000-2-23 18:19:37.277] [JSON]
    JSON task exited!!!!

    Wifi Disabled

    [2000-2-23 18:19:37.290] [HTTP Service] HTTP server released!!!
    [2000-2-23 18:19:37.375] [SIMPLELINK] ERROR!!! Failed to stop WiFi!!!!
    [2000-2-23 18:19:37.375] [SIMPLELINK] WiFi SSID: "WIFI_RER605_JP"
    [2000-2-23 18:19:37.376] [SIMPLELINK] WiFi Password: "qwerty123"
    [2000-2-23 18:19:37.376] [SIMPLELINK] WiFi mDNS: "rer605.abb.com"
    [2000-2-23 18:19:37.377] [SIMPLELINK] ERROR!!! Failed to configure WiFi in AP mode!!!!

    Wifi Enabled

    [2000-2-23 18:19:52.395] [MQX] WiFi Task Started
    [2000-2-23 18:19:52.420] [SIMPLELINK] WiFi SSID: "WIFI_RER605_JP"
    [2000-2-23 18:19:52.420] [SIMPLELINK] WiFi Password: "qwerty123"
    [2000-2-23 18:19:52.421] [SIMPLELINK] WiFi mDNS: "rer605.abb.com"
    [2000-2-23 18:19:52.421] [SIMPLELINK] ERROR!!! Failed to configure WiFi in AP mode!!!!

    Additional note:

    "An open point from our cyber security team is, NWP using WPA2 passwords.  As it is proven now’ it is quite vulnerable and could hacked with ease. Check with TI if they can supports WPA3 standard? "

    Could you please let me know if we can use WPA3 standards?

    CDEC Issues.zip

    I'm looking for your reply.

    Thanks in advance.

    Br,

    Paul

  • Hello Again,

    Please find the logs

    Timeout =4
    Timeout =3
    Timeout =2
    Timeout =1
    Timeout =0

    [2000-2-22 00:11:42.336] [SIMPLELINK] FATAL ERROR: No Cmd Ack detected [cmd opcode = 0x9403]

    [2000-2-22 00:11:42.346] [HTTP Service] [sock 80] Socket Accept -2005 ERROR!!!
    [2000-2-22 00:11:42.355] [HTTP Service] [sock -2005] ERROR!!! Can't close HTTP socket.

    if I tried to trying again to communicate..

    [SIMPLELINK] FATAL ERROR: Driver Abort detected. is coming from NWP.

    Br,

    Paul

  • Hi Paul,

    The _SlNonOsSemGet  was used by the CC3100 driver, and was replaced in simplelink_sdk_wifi_plugin_2_40_00_22. Thus was the source of the question.

    Are you sure you are porting the right driver?

    When trying to reset the device - are you calling sl_Stop(0), i.e. with 0 timeout? If yes, than this may be related to a known issue and you'll need to use full MCU reset currently (a fix to this NWP reset issue will be included in one of the coming releases).

    We will add the WPA3.0 support in one of the next releases.

    The logs above are terminal (host) logs and not the NWP (FW) ones. Please follow the instructions in the link to provide the NWP log ( http://processors.wiki.ti.com/index.php/CC3120_%26_CC3220_Capture_NWP_Logs).

    Br,

    Kobi

  • Also, I'm not sure how you can set a timeout to the sl_Accept. are you working with the blocking or non-blocking socket?

    The "sl_Accept" can wait for very long time until a peer device will connect to it.

    Br, 

    Kobi

  • Hello Kobi,

    I’m using non blocking socket. May be here it’s waiting for a long time. but attacker try to slowdown the server connection or stopping the server  by flooding with packets.

    Br,

    Paul Jins

  • Hello Kobi,

    Q : The logs above are terminal (host) logs and not the NWP (FW) ones. Please follow the instructions in the link to provide the NWP log ( http://processors.wiki.ti.com/index.php/CC3120_%26_CC3220_Capture_NWP_Logs).

    Ans: Currently I don't have any CC3120 development board and we are using customized hardware board along with our MCU. MCU to NWP communication through SPI. At this moment I'm little confused about that how I can get the logs from NWP . Could you please guide ? How can I get the NWP logs through SPI communication with out changing hardware setup.

    Please let me know.

    Br,

    Paul 

  • Hello Kobi,

    Q : The _SlNonOsSemGet  was used by the CC3100 driver, and was replaced in simplelink_sdk_wifi_plugin_2_40_00_22. Thus was the source of the question.

    Ans: 

    #ifdef DeviceFamily_CC26X2
    #define sl_SyncObjWait(pSyncObj,Timeout) SemaphoreP_pend( \
    (*(pSyncObj)),(Timeout * 100))
    #else
    #define sl_SyncObjWait(pSyncObj,Timeout) SemaphoreP_pend( \
    (*(pSyncObj)),Timeout)
    #endif

     

    Are you referring about this code? If yes  then,  instead using SemaphoreP_pend  I have used _SlNonOsSemGet  only ( Intention was don't want to do any major change in porting code since our project in SOP state. As per your point I have added MQX semaphore option but result seems same. semaphore is not getting released and also this change making some failure in my MCU host application  code.

    Q: Are you sure you are porting the right driver? 

    Ans: Yes. I've ported correct driver only.

     

    Q: When trying to reset the device - are you calling sl_Stop(0), i.e. with 0 timeout?

    Ans: Yes: but I'm getting negative response from NWP i.e, [SIMPLELINK] ERROR!!! Failed to stop WiFi!!!!

    I have one more open point about CC3120 certification. I knew you may have done WLAN certification testing for NWP but have you done any cyber security attack tests? if yes, can you guide me find the certification information.

    Our product release dead line is approaching, Since we have not expected these type of issue in NWP and we have failed cyber security test, our product release is highly critical now. Could you please provide me a work around soon.

    Do needful. Thanks in advance.

    Br,

    Paul

     

  • Hello Kobi,

    Do you have any updates on this issue? Please let me know.

    Regards,

    Paul

  • Hello Kobi,

    I'm awaiting for your reply.

    One more open point : Is there any possibility to recover from this attack without resetting the MCU ? like only resetting the NWP or re-initializing the NWP.

    I was not successful in resetting the NWP and re-initializing the NWP. Can you guide me whether this method would solve this issue?

    Looking forward your reply soon,

    Br,

    Paul

  • Hi Paul,

    The NWP logs are coming of GPIO_07 (which is mux-ed to PIN 62 on the launchpad, hopefully you can get this GPIO out on your board).

    The next SP (to be released by end of the month should include a fix that will allow to reset the NWP after getting the abort error.

    Br,

    Kobi

  • Hi Kobi,

    Thank you for the response.

    I have an open point!! why  can not we reinitialize/recover NWP successfully, after reset of NWP  (in current firmware ),giving some time(1sec to 30mins) delay and re-initializing same like normal  MCU power cycle start?

    I would like to know what is the exact dependency between NWP firmware and MCU firmware ? Why current NWP firmware demanding MCU reset?

    I'm awaiting your reply.

    Thanks in advance.

    Br,

    Paul

  • Hello Kobi,

    Thank you for the response.

    I had a discussion with my hardware team and they said they will able to generate NWP Logs.. I will try to share logs as early as possible.

    I hope the next SP will give me a workaround for my issue but if continuous cyber attack happened!! my communication would be permanently  down due  to the continuous restart of NWP. Can I expect some permanent solution fix for this cyber attack in coming month? 

    Or is there any way to block that client from my side ? like blocking that particular IP after getting the first attack?

    Please let me know.

    Br,

    Paul 

  • Hello Kobi,

    When can I expect next SP release with this fix?

    Please let me know.

    Br,

    Paul

  • Hi Paul,

    The new SDK (3.30) is out there.

     I believe the fix you are waiting for is in the SimpleLink Driver and not in the SP (but updating to the latest SP is always recommended).

    Br,

    Kobi