This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

CC2652R: OTBR-RCP: RadioSpinelNoResponse (radio tx timeout)

Part Number: CC2652R
Other Parts Discussed in Thread: , SYSCONFIG, UNIFLASH, LAUNCHXL-CC26X2R1, SYSBIOS,

Hi, 

We are using otbr-agent in our Linux (version  4.19.183) platform with CC2652R1 through UART(115200) communication. OTBR was synced in March, 2023 and latest rcp image from simplelink_cc13xx_cc26xx_sdk_7_10_00_98 are used.

version:
OPENTHREAD/; POSIX; May 3 2023 14:38:20

rcp version:(rcp_CC26X2R1_LAUNCHXL_tirtos7_ticlang_sdk_7_10_00_98.bin)
TI-OPENTHREAD/1.2.4.0; CC1352; May 12 2023 13:12:48

When we add more thread sensors, we get this error message and otbr-agent crashes. Looks like, TI-RCP is not responding after some time.

crit otbr-agent[3315]: 00:31:09.962 [C] Platform------: HandleRcpTimeout() at radio_spinel_impl.hpp:2275: RadioSpinelNoResponse
warn otbr-agent[3315]: 00:31:09.962 [W] Platform------: radio tx timeout

If we have 1-3 thread sensors, then issue is less frequent. When we add more sensors around 10, then we see this issue more frequently. 

We use the default setting everywhere (UART baud rate 115200)

otbr-agent -I wpan0 -B eth0 spinel+hdlc+uart:///dev/ttyH0 trel://eth0

Any suggestion? 

Is there any need to increase the baud rate?  if so, what is teh recommended value and where we should change? (rcp_CC26X2R1_LAUNCHXL_tirtos7_ticlang\platform\uart.C ?)

Thanks

Sen

  • Hi Sen,

    Is the LED still blinking on RCP?

    Is there any need to increase the baud rate?

    Yes, increasing baud rate could be worth trying.

    In otbr-posix, search for "OPENTHREAD_SIMULATION_UART_BAUDRATE".

    In RCP, you should be able to change the baudrate in the UART section.

    Thanks,
    Toby

  • Hi Toby,

    We don't use the eval board. We have CC2652R in our PCB connected through UART (AP).

    So should I change this following file for rcp baud rate?

    rcp_CC26X2R1_LAUNCHXL_tirtos7_ticlang\platform\uart.C 

    Our host processor support B460800, so i ll try that after you confirm the rcp file.

    Thanks

    Sen

  • rcp_CC26X2R1_LAUNCHXL_tirtos7_ticlang\platform\uart.C 

    Actually yes, that is the right file to change, since it is hard-coding the baudrate (i.e. "params.baudRate         = 115200;").

  • Hi ,

    I could reproduce the same issue with Raspberry PI (OTBR) with LAUNCHXL-CC26X2R1 board (simplelink_cc13xx_cc26xx_sdk_7_10_00_98). I connected 6 sensors and started pinging. After few minutes, otbr-agent exits with this message. I see only one led is lights up and it is a green LED close to XDS110 Debugger. 


    May 15 16:23:17 raspberrypi otbr-agent[1977]: 00:23:15.571 [W] Platform------: radio tx timeout
    May 15 16:23:17 raspberrypi otbr-agent[1977]: 00:23:15.571 [C] Platform------: HandleRcpTimeout() at radio_spinel_impl.hpp:2248: RadioSpinelNoResponse

    No improvement with baud rate. Any advise  ?

    NOTE:

    For the same setup Raspberry (OTBR ) and 6 sensors;  If i replace CC2652R1 eval board with and Nordic nrf8240 usb dongle, that works fine.

  • I see only one led is lights up and it is a green LED close to XDS110 Debugger.

    You should see the red LED blink every 2 seconds (red LED is DIO6, near the antenna on the LP_CC2652R7). If that is not blinking, then the RCP is probably not running.

    Can you further describe your system? How often is the RCP TX'ing? How often are the sensors TX'ing? How large are the packets? Is it a start network or mesh network?

  • I dont see the led blinking but rcp running without any issues for 5-20mins randomly.

    I can easily reproduce with star topology with 6 sensors connected. 

    I tested with 300bytes ping message every second. This way, i can reproduce it much quicker. Even if dont ping and add 20 sensors, i can see the issue. 

  • What devices are you using for the sensors? Are the sensors sleep devices? If sleepy, how often do they wake up to poll?

  • Hi ,

    We are pinging every second and sleepy end-device will not go to sleep. All the sensors USB powered dongles. Why do you think, sensor causes the rcp timeout with otbr?  If you cant reproduce the issue, do you want us to test anything else?

    1) TI CC2652R (Leader and border router)

    childsupervision interval

    129

    2) Sensor: Nordic NRF52840 (ot-cli-mtd.hex)

    > childtimeout
    240

    > pollperiod (milliseconds)
    236000

  • I think it could be related to too much traffic. This could potentially be resolved by increasing some "sizes" on RCP side.

    Can you try increasing this value for the RCP?
    OTSTACK_PROC_QUEUE_MAX_MSG

    Can you share sniffer logs?

    Can you share fuller serial logs (otbr-posix)?

  • Thanks

    could you reproduce the issue or you don't have a setup?

    1) #define OTSTACK_PROC_QUEUE_MAX_MSG      (16). ; do you recommend any value to test?  Shall i keep 2048?

    2) I don't have a sniffer, i ll try to convert a sensor as sniffer.

    3) Do you want me to enable with -d 7 in otbr-agent during the start up or need to compile the otbr-agent with any specific changes?

  • otbr_log.txt
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:12.395 [I] Platform------: | 00 00 40 1E 00 00 00 00 | 00 00 00 00 60 00 00 00 | ..@.........`...
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:12.395 [I] Platform------: | 00 43 11 FF FE 80 00 00 | 00 00 00 00 A4 7F 5B F5 | .C..~.......$.[u
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:12.396 [I] Platform------: | 53 A0 81 4E FE 80 00 00 | 00 00 00 00 3C 3D 18 48 | S .N~.......<=.H
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:12.396 [I] Platform------: | 84 39 B1 7A 4D 4C 4D 4C | 00 43 5F 70 00 15 12 00 | .91zMLML.C_p....
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:12.396 [I] Platform------: | 00 00 00 00 00 00 01 43 | D9 FE 7D A6 64 13 62 27 | .......CY~}&d.b'
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:12.396 [I] Platform------: | F2 89 21 7F 00 00 00 00 | 00 00 00 00 B8 30 00 20 | r.!.........80.
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:12.396 [I] Platform------: | 00 00 00 00 00 00 00 00 | 0E 2C 0A 00 08 00 6B 00 | .........,....k.
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:12.397 [I] Platform------: | 00 00 00 00 00 00 00 00 | 00 00 00 00 00 00 40 1E | ..............@.
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:12.397 [I] Platform------: | 00 00 00 00 00 00 00 00 | 60 00 00 00 00 43 11 FF | ........`....C..
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:12.397 [I] Platform------: | FE 80 00 00 00 00 00 00 | A4 7F 5B F5 53 A0 81 4E | ~.......$.[uS .N
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:12.397 [I] Platform------: | FE 80 00 00 00 00 00 00 | 3C 3D 18 48 84 39 B1 7A | ~.......<=.H.91z
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:12.397 [I] Platform------: | 4D 4C 4D 4C 00 43 1E D0 | 00 15 11 00 00 00 00 00 | MLML.C.P........
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:12.397 [I] Platform------: | 00 00 01 7B 4C B9 D4 4D | 47 9F EA DB 47 3D 04 E6 | ...{L9TMG.j[G=.f
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:12.397 [I] Platform------: | 00 00 00 00 00 00 00 00 | B8 30 00 20 00 00 00 00 | ........80. ....
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:12.397 [I] Platform------: | 69 C2 00 00 1D 2F 0A 00 | 08 00 5C 01 .. .. .. .. | iB.../....\.....
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:12.398 [I] Platform------: ------------------------------------------------------------------------
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:12.615 [I] MeshForwarder-: Received IPv6 ICMP6 msg, len:348, chksum:f83c, ecn:no, from:0xd806, sec:yes, prio:normal, rss:-38.125
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:12.615 [I] MeshForwarder-:     src:[fd69:fb99:a55a:1:36e0:d2cb:d296:5183]
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:12.615 [I] MeshForwarder-:     dst:[fd00:b2ff:fe12:34:dea6:32ff:fe12:2cd6]
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:12.615 [I] Platform------: [netif] Packet from NCP (348 bytes)
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:12.615 [I] Platform------: ===============================[ len=348]===============================
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:12.615 [I] Platform------: | 60 00 00 00 01 34 3A 05 | FD 69 FB 99 A5 5A 00 01 | `....4:.}i{.%Z..
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:12.615 [I] Platform------: | 36 E0 D2 CB D2 96 51 83 | FD 00 B2 FF FE 12 00 34 | 6`RKR.Q.}.2.~..4
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:12.615 [I] Platform------: | DE A6 32 FF FE 12 2C D6 | 80 00 F8 3C 00 01 0A 2F | ^&2.~.,V..x<.../
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:12.615 [I] Platform------: | 00 2D 7E 91 00 00 00 00 | 00 00 00 00 00 00 00 00 | .-~.............
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:12.615 [I] Platform------: | 00 00 00 00 00 00 00 00 | 00 32 AD 79 00 00 00 00 | .........2-y....
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:12.616 [I] Platform------: | 00 00 00 00 00 00 00 00 | 00 02 D8 00 .. .. .. .. | ..........X.....
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:12.616 [I] Platform------: | 00 00 00 00 00 00 00 00 | 81 00 00 00 00 00 00 00 | ................
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:12.616 [I] Platform------: | 00 00 00 00 A0 30 00 20 | 00 00 00 00 00 00 00 00 | .... 0. ........
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:12.616 [I] Platform------: | 00 00 00 00 30 00 26 00 | 0C 00 00 00 00 00 00 00 | ....0.&.........
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:12.616 [I] Platform------: | 00 00 00 00 00 00 00 06 | 60 00 00 00 00 00 00 00 | ........`.......
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:12.616 [I] Platform------: | 00 00 00 00 00 00 00 00 | 3C 3D 18 48 00 00 00 00 | ........<=.H....
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:12.616 [I] Platform------: | 00 00 00 00 00 00 00 00 | 00 00 00 00 00 00 00 00 | ................
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:12.616 [I] Platform------: | 00 00 00 00 00 32 0E CE | 00 15 0C 26 00 00 00 00 | .....2.N...&....
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:12.617 [I] Platform------: | 00 00 01 04 00 02 D8 00 | 0B 08 21 1F CA B8 40 0B | ......X...!.J8@.
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:12.617 [I] Platform------: | FB 36 09 0A 84 00 00 00 | 00 00 00 00 00 00 00 00 | {6..............
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:12.617 [I] Platform------: | 00 00 00 00 00 00 00 00 | 00 00 00 00 00 00 00 00 | ................
    May 19 13:26:40 raspberrypi otbr-agent[78586]: message repeated 4 times: [ 00:09:12.617 [I] Platform------: | 00 00 00 00 00 00 00 00 | 00 00 00 00 00 00 00 00 | ................]
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:12.617 [I] Platform------: | 00 00 00 00 00 00 00 00 | A0 30 00 20 00 00 00 00 | ........ 0. ....
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:12.618 [I] Platform------: | FA A1 00 00 A9 7A 2D 00 | 08 00 5C 01 .. .. .. .. | z!..)z-...\.....
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:12.618 [I] Platform------: ------------------------------------------------------------------------
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:12.619 [I] Platform------: [netif] Packet to NCP (348 bytes)
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:12.619 [I] Platform------: ===============================[ len=348]===============================
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:12.619 [I] Platform------: | 60 0B 93 2A 01 34 3A 3F | FD 00 B2 FF FE 12 00 34 | `..*.4:?}.2.~..4
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:12.619 [I] Platform------: | DE A6 32 FF FE 12 2C D6 | FD 69 FB 99 A5 5A 00 01 | ^&2.~.,V}i{.%Z..
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:12.619 [I] Platform------: | 36 E0 D2 CB D2 96 51 83 | 81 00 F7 3C 00 01 0A 2F | 6`RKR.Q...w<.../
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:12.619 [I] Platform------: | 00 2D 7E 91 00 00 00 00 | 00 00 00 00 00 00 00 00 | .-~.............
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:12.619 [I] Platform------: | 00 00 00 00 00 00 00 00 | 00 32 AD 79 00 00 00 00 | .........2-y....
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:12.619 [I] Platform------: | 00 00 00 00 00 00 00 00 | 00 02 D8 00 .. .. .. .. | ..........X.....
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:12.619 [I] Platform------: | 00 00 00 00 00 00 00 00 | 81 00 00 00 00 00 00 00 | ................
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:12.619 [I] Platform------: | 00 00 00 00 A0 30 00 20 | 00 00 00 00 00 00 00 00 | .... 0. ........
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:12.620 [I] Platform------: | 00 00 00 00 30 00 26 00 | 0C 00 00 00 00 00 00 00 | ....0.&.........
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:12.620 [I] Platform------: | 00 00 00 00 00 00 00 06 | 60 00 00 00 00 00 00 00 | ........`.......
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:12.620 [I] Platform------: | 00 00 00 00 00 00 00 00 | 3C 3D 18 48 00 00 00 00 | ........<=.H....
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:12.620 [I] Platform------: | 00 00 00 00 00 00 00 00 | 00 00 00 00 00 00 00 00 | ................
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:12.620 [I] Platform------: | 00 00 00 00 00 32 0E CE | 00 15 0C 26 00 00 00 00 | .....2.N...&....
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:12.620 [I] Platform------: | 00 00 01 04 00 02 D8 00 | 0B 08 21 1F CA B8 40 0B | ......X...!.J8@.
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:12.621 [I] Platform------: | FB 36 09 0A 84 00 00 00 | 00 00 00 00 00 00 00 00 | {6..............
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:12.621 [I] Platform------: | 00 00 00 00 00 00 00 00 | 00 00 00 00 00 00 00 00 | ................
    May 19 13:26:40 raspberrypi otbr-agent[78586]: message repeated 3 times: [ 00:09:12.621 [I] Platform------: | 00 00 00 00 00 00 00 00 | 00 00 00 00 00 00 00 00 | ................]
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:12.622 [I] Platform------: | 00 00 00 00 00 00 00 00 | 00 00 00 00 00 00 00 00 | ................
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:12.622 [I] Platform------: | 00 00 00 00 00 00 00 00 | A0 30 00 20 00 00 00 00 | ........ 0. ....
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:12.622 [I] Platform------: | FA A1 00 00 A9 7A 2D 00 | 08 00 5C 01 .. .. .. .. | z!..)z-...\.....
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:12.622 [I] Platform------: ------------------------------------------------------------------------
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:12.739 [I] MeshForwarder-: Received IPv6 UDP msg, len:84, chksum:1553, ecn:no, from:a67f5bf553a0814e, sec:no, prio:net, rss:-34.0
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:12.739 [I] MeshForwarder-:     src:[fe80:0:0:0:a47f:5bf5:53a0:814e]:19788
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:12.739 [I] MeshForwarder-:     dst:[ff02:0:0:0:0:0:0:2]:19788
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:12.740 [I] Mle-----------: Receive Parent Request (fe80:0:0:0:a47f:5bf5:53a0:814e)
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:12.740 [I] Mle-----------: Delay Parent Response (fe80:0:0:0:a47f:5bf5:53a0:814e)
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:12.968 [I] Mle-----------: Send delayed message (fe80:0:0:0:a47f:5bf5:53a0:814e)
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:13.049 [I] MeshForwarder-: Received IPv6 ICMP6 msg, len:348, chksum:355b, ecn:no, from:0xd805, sec:yes, prio:normal, rss:-39.25
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:13.049 [I] MeshForwarder-:     src:[fd69:fb99:a55a:1:d4a1:e4d1:4747:725c]
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:13.049 [I] MeshForwarder-:     dst:[fd00:b2ff:fe12:34:dea6:32ff:fe12:2cd6]
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:13.049 [I] Platform------: [netif] Packet from NCP (348 bytes)
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:13.049 [I] Platform------: ===============================[ len=348]===============================
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:13.049 [I] Platform------: | 60 00 00 00 01 34 3A 05 | FD 69 FB 99 A5 5A 00 01 | `....4:.}i{.%Z..
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:13.049 [I] Platform------: | D4 A1 E4 D1 47 47 72 5C | FD 00 B2 FF FE 12 00 34 | T!dQGGr\}.2.~..4
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:13.049 [I] Platform------: | DE A6 32 FF FE 12 2C D6 | 80 00 35 5B 00 02 0A 35 | ^&2.~.,V..5[...5
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:13.049 [I] Platform------: | 00 2A 22 15 00 00 00 00 | 00 00 00 00 00 00 00 00 | .*".............
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:13.049 [I] Platform------: | 00 00 00 00 00 00 00 00 | 00 00 00 00 00 00 00 00 | ................
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:13.049 [I] Platform------: | B8 30 00 20 00 00 00 00 | 00 00 00 00 .. .. .. .. | 80. ............
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:13.049 [I] Platform------: | 00 00 54 00 28 00 00 00 | 5A E3 00 00 30 09 00 00 | ..T.(...Zc..0...
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:13.050 [I] Platform------: | 00 00 00 06 60 00 00 00 | 00 2C 11 FF FE 80 00 00 | ....`....,..~...
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:13.050 [I] Platform------: | 00 00 00 00 A4 7F 5B F5 | 53 A0 81 4E FF 02 00 00 | ....$.[uS .N....
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:13.050 [I] Platform------: | 00 00 00 00 00 00 00 00 | 00 00 00 02 4D 4C 4D 4C | ............MLML
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:13.050 [I] Platform------: | 00 2C 15 53 00 15 13 00 | 00 00 00 00 00 00 01 35 | .,.S...........5
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:13.050 [I] Platform------: | D8 43 BB 3F CE CE 39 9F | 93 85 9B 63 B5 23 34 CE | XC;?NN9....c5#4N
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:13.050 [I] Platform------: | 49 86 4D 3D 00 00 00 00 | 00 00 00 00 00 00 00 00 | I.M=............
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:13.050 [I] Platform------: | 00 00 00 00 00 00 00 00 | 00 00 00 00 00 00 00 00 | ................
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:13.050 [I] Platform------: | 00 00 00 00 00 00 00 00 | B8 30 00 20 00 00 00 00 | ........80. ....
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:13.050 [I] Platform------: | 00 00 00 00 00 00 00 00 | 30 00 26 00 0C 00 00 00 | ........0.&.....
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:13.050 [I] Platform------: | 00 00 00 00 00 00 00 00 | 00 00 00 06 60 00 00 00 | ............`...
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:13.051 [I] Platform------: | 00 00 00 00 00 00 00 00 | 00 00 00 00 3C 3D 18 48 | ............<=.H
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:13.051 [I] Platform------: | 00 00 00 00 00 00 00 00 | 00 00 00 00 00 00 00 00 | ................
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:13.052 [I] Platform------: | 00 00 00 00 00 00 00 00 | 00 32 0E CE 00 15 0C 26 | .........2.N...&
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:13.052 [I] Platform------: | 00 00 00 00 00 00 00 00 | B8 30 00 20 00 00 00 00 | ........80. ....
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:13.052 [I] Platform------: | AE 9E 00 00 2D 1E 2A 00 | 08 00 5C 01 .. .. .. .. | ....-.*...\.....
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:13.052 [I] Platform------: ------------------------------------------------------------------------
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:13.052 [I] Platform------: [netif] Packet to NCP (348 bytes)
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:13.053 [I] Platform------: ===============================[ len=348]===============================
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:13.053 [I] Platform------: | 60 05 2B 70 01 34 3A 3F | FD 00 B2 FF FE 12 00 34 | `.+p.4:?}.2.~..4
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:13.053 [I] Platform------: | DE A6 32 FF FE 12 2C D6 | FD 69 FB 99 A5 5A 00 01 | ^&2.~.,V}i{.%Z..
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:13.053 [I] Platform------: | D4 A1 E4 D1 47 47 72 5C | 81 00 34 5B 00 02 0A 35 | T!dQGGr\..4[...5
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:13.053 [I] Platform------: | 00 2A 22 15 00 00 00 00 | 00 00 00 00 00 00 00 00 | .*".............
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:13.053 [I] Platform------: | 00 00 00 00 00 00 00 00 | 00 00 00 00 00 00 00 00 | ................
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:13.054 [I] Platform------: | B8 30 00 20 00 00 00 00 | 00 00 00 00 .. .. .. .. | 80. ............
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:13.054 [I] Platform------: | 00 00 54 00 28 00 00 00 | 5A E3 00 00 30 09 00 00 | ..T.(...Zc..0...
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:13.054 [I] Platform------: | 00 00 00 06 60 00 00 00 | 00 2C 11 FF FE 80 00 00 | ....`....,..~...
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:13.054 [I] Platform------: | 00 00 00 00 A4 7F 5B F5 | 53 A0 81 4E FF 02 00 00 | ....$.[uS .N....
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:13.054 [I] Platform------: | 00 00 00 00 00 00 00 00 | 00 00 00 02 4D 4C 4D 4C | ............MLML
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:13.054 [I] Platform------: | 00 2C 15 53 00 15 13 00 | 00 00 00 00 00 00 01 35 | .,.S...........5
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:13.054 [I] Platform------: | D8 43 BB 3F CE CE 39 9F | 93 85 9B 63 B5 23 34 CE | XC;?NN9....c5#4N
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:13.054 [I] Platform------: | 49 86 4D 3D 00 00 00 00 | 00 00 00 00 00 00 00 00 | I.M=............
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:13.054 [I] Platform------: | 00 00 00 00 00 00 00 00 | 00 00 00 00 00 00 00 00 | ................
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:13.054 [I] Platform------: | 00 00 00 00 00 00 00 00 | B8 30 00 20 00 00 00 00 | ........80. ....
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:13.054 [I] Platform------: | 00 00 00 00 00 00 00 00 | 30 00 26 00 0C 00 00 00 | ........0.&.....
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:13.054 [I] Platform------: | 00 00 00 00 00 00 00 00 | 00 00 00 06 60 00 00 00 | ............`...
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:13.054 [I] Platform------: | 00 00 00 00 00 00 00 00 | 00 00 00 00 3C 3D 18 48 | ............<=.H
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:13.054 [I] Platform------: | 00 00 00 00 00 00 00 00 | 00 00 00 00 00 00 00 00 | ................
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:13.055 [I] Platform------: | 00 00 00 00 00 00 00 00 | 00 32 0E CE 00 15 0C 26 | .........2.N...&
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:13.055 [I] Platform------: | 00 00 00 00 00 00 00 00 | B8 30 00 20 00 00 00 00 | ........80. ....
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:13.055 [I] Platform------: | AE 9E 00 00 2D 1E 2A 00 | 08 00 5C 01 .. .. .. .. | ....-.*...\.....
    May 19 13:26:40 raspberrypi otbr-agent[78586]: 00:09:13.055 [I] Platform------: ------------------------------------------------------------------------
    May 19 13:26:41 raspberrypi otbr-agent[78586]: 00:09:13.489 [I] MeshForwarder-: Received IPv6 UDP msg, len:84, chksum:12da, ecn:no, from:a67f5bf553a0814e, sec:no, prio:net, rss:-34.0
    May 19 13:26:41 raspberrypi otbr-agent[78586]: 00:09:13.490 [I] MeshForwarder-:     src:[fe80:0:0:0:a47f:5bf5:53a0:814e]:19788
    May 19 13:26:41 raspberrypi otbr-agent[78586]: 00:09:13.490 [I] MeshForwarder-:     dst:[ff02:0:0:0:0:0:0:2]:19788
    May 19 13:26:41 raspberrypi otbr-agent[78586]: 00:09:13.490 [I] Mle-----------: Receive Parent Request (fe80:0:0:0:a47f:5bf5:53a0:814e)
    May 19 13:26:41 raspberrypi otbr-agent[78586]: 00:09:13.490 [I] Mle-----------: Delay Parent Response (fe80:0:0:0:a47f:5bf5:53a0:814e)
    May 19 13:26:41 raspberrypi otbr-agent[78586]: 00:09:13.567 [W] Platform------: radio tx timeout
    May 19 13:26:41 raspberrypi otbr-agent[78586]: 00:09:13.567 [C] Platform------: HandleRcpTimeout() at radio_spinel_impl.hpp:2248: RadioSpinelNoResponse
    

    i could not upload sniffer log here but it does not give any new input.  Attached the otbr logs. 

    i changed the value from 16 to 64 but no help. still same rcp timeout. 

    #define OTSTACK_PROC_QUEUE_MAX_MSG      (64)

    Can you help me in debugging the rcp firmware or can you try to  reproduce it?  it should be straight forward to reproduce with Raspberry PI and few sensors. 

  • No, I have not setup and reproduced the issue. I will try to get setup and reproduce in next week or so.

  • As this issue is critical for us, Could you please point us any documentation on how we can debug the Thread "RCP" firmware with Raspberry PI - TI CC2652R eval board? 

  • Hi Sen,

    Are you aware of the Debugging section of the TI OpenThread User's Guide?  It is possible to have the RCP UART communication ongoing with the Raspberry PI while debugging the CC2652R device through JTAG. 

    Regards,
    Ryan

  • Thanks    I ll have a look and see, if i can debug. 

  • Hi Ryan / Toby

    Have a good day.

    Thanks for your information update.

    I'm SW Danny from WNC, we've real projects built with CC2562R1/R7,

    currently, our customer (Extreme / Sen) met this Thread issue and the failure symptoms can replicated on Raspberry PI (OTBR) with LAUNCHXL-CC26X2R1 board (simplelink_cc13xx_cc26xx_sdk_7_10_00_98) as he mentioned.

    so, could you please escalate the issue priority to get more efficient support for this critical issue fix? 

    Thanks

    Danny

  • Hello Sen, Danny,

    I apologize for the delay, please understand that we have limited resources with which to address this issue at the moment.  It would help greatly if you are able to connect the CC2652RX to JTAG and further debug to determine where the firmware is getting stuck.  

    I also recommend evaluating the RCP project from the ot-ti github repo.  This is the branch that will be supported in the near future in favor of the SDK, and there is a concern that a change in API from the older SDK resource could be causing a communication mismatch with the host. It should be sufficient to run the build with the LaunchPad name being evaluated.

    Also, please send a more complete log of the openthread daemon. Including the packets before will allow us to see if something was wrong with the packet structure.

    Regards,
    Ryan

  • Hi Ryan,

    Sen said this issue happen more easily when the connected sensors become more (3 --> 10 sensors). Does this mean it relates to memory ? Is it helpful to suggest to use CC2652R7 (with bigger memory) ?

  • Hi Ryan, Jerry

    Thanks for update quickly, our customer (EXTR) Sen will follow suggestion to try it.

    Thanks again.

    Danny

  • Hi Jerry,

    SRAM memory overflow could be an issue, but if this is the case then it would be better to test increasing HEAPSIZE from the *.cmd file since it is statically allocated.  I do not expect the CC2652R7 would improve performance, although it could be attempted.  Increasing the PLATFORM_UART_RECV_BUF_LEN and SysConfig UART2 Ring Buffer Sizes can also be evaluated.

    Regards,
    Ryan

  • HEAPSIZE

      Thanks Ryan.

    We located the files, could you please suggest the safe values to start with to experiment the issue?

    1) cc13x2_cc26x2_tirtos7.cmd
    HEAPSIZE = 0x4000; /* Size of heap buffer used by HeapMem */

    2) uart.c
    /**
    * Size of the statically allocated buffer to pass data from the callback to
    * the processing loop.
    */
    #define PLATFORM_UART_RECV_BUF_LEN 32

    3)
    RX Ring Buffer Size 32
    TX Ring Buffer Size 32

  • 1) Try increasing to 0x6000

    2) and 3) Double the values to 64

    Please comment whether any change delays the RCP failure from occurring.

    Regards,
    Ryan

  • Thanks  

    I followed the recommendations and it definitely improving the performance. when i add more and more sensors, i see the issue. Could you please recommend the upper limit for the above Heap and buffer size that we can change to. 

  • I'm glad to hear that there is improvement.  First, identify which value actually improves behavior (I assume the HEAPSIZE).  I don't know whether an upper limit has ever been evaluated by the RCP, you could try doubling or tripling the value but I recommend rigorously testing your system while continuing to monitor the stack and heap memory.

    Regards,
    Ryan

  • Thanks Ryan, Doubling the UART helped then tripling did not help. Then I kept the UART doubled, then doubled the HEAP, then it improved a bit. I may try tripling the Heap. Just worried, if  there is any safe guidelines. Even though, you did not experiment, any theoretical guidelines would be beneficial. 

  •   I have this setup. 

    https://software-dl.ti.com/simplelink/esd/simplelink_cc26x2_sdk/2.10.00.44/exports/docs/thread/html/cc26x2/hardware-thread.html#thread-debug-uart-connections

    Can i use the "secondary serial debug connection" to debug the "rcp timeout". CC2652R eval board is connected to Raspberry  through UART. I have this secondary serial debug connection as in the link above. Com port is  showing  up in my PC but i could not connect and debug through CCS. Any suggestion or documentation?  

  • Hi Sen,

    I have not used this interface before, however I did align with a co-worker who informed me that the secondary serial debug connection corresponds to the debug_uart.c file but has been deprecated since the transition to UART2 functionality several quarters ago (the UART TI Driver was replaced around v6.20 according to the Release Notes).  Thus this documentation is no longer valid and should be removed from the User's Guide.  A JTAG debug connection remains as the recommended approach.

    Regards,
    Ryan

  • Thanks  for letting us know.

    We don't have a XDS110 Debug Probe but I could use one of the TI eval board as XDS110 usb debug probe. I could download the RCP image using UniFlash though the JTAG debugger (on TI CC2652 Eval board). I could not connect the CCS to debug. I checked your debug link but was not successful.

    Could you help on this?

    Thanks

  • Please share your connection setup and what specific errors/issues you observe from CCS. Pictures and screenshots would be preferred.  You can also review the "Advanced use of the LaunchPad hardware" section of the LAUNCHXL-CC26X2R1 User Guide as well as the Debugging JTAG page.

    Regards,
    Ryan

  •   Now i could connect the CCS to the target. How to find the cause of the reset? Looks like target reset causes the rcp timeout. When the issue happens, the CCS losses the connection too (due to target reset). So i need to monitor the heap runtime(not sure, how to do right now) and should have a breakpoint before the reset so that i can get more details. Any suggestion? meantime, i am going through the document you shared earlier. 

    You suggestion may speed up the troubleshooting process. Thanks again. 

  • otPlatReset is the software reset function and otPlatGetResetReason can determine the reason for the last reset during startup.  A hardware reset is unexpected as the RCP is designed to get trapped in a fault or while(1).  You can use the ROV (TI-RTOS7 project) to track the heap.

    Regards,
    Ryan

  • Hi Ryan,

    I don't see the heap size changes while i was pinging a sensor. Am i looking at the wrong place?

    [
    {
    "module": "ti.sysbios.heaps.HeapMem",
    "view": "Detailed"
    },
    {
    "elements": [
    {
    "address": "0x2000628c",
    "symbol": "BIOS_heapMemObject",
    "buf": "0x200064c8",
    "minBlockAlign": "8",
    "totalSize": "16384",
    "totalFreeSize": "15456",
    "largestFreeSize": "15456"
    }
    ]
    }
    ]

    Also, don't see any error code.

  • You are observing the correct heap location for HEAPSIZE.  Does the code not reach otPlatReset -> SysCtrlSystemReset when the issue occurs?  What are you able to observe from the ROV after replicating the problem?

    Regards,
    Ryan

  • I did have a break point but never reached there or not working properly during the reset. Where can i find the Runtime Object View (rov.json) file?  I guess, i need that to observe. 

  • Here is the Runtime Object View User Guide which includes information pertaining to JSON files.

    Regards,
    Ryan

  • Hi Ryan,

    I see overview.rov.json for different projects but not for Thread in simplelink_cc13xx_cc26xx_sdk_7_10_00_98.

    I could not get any meaningful details. Heap memory map is not changing. 

    CPU has error; Error: Module ti.sysbios.utils.Load does not exist in the application's configuration

    Stack space: not showring anything while ruining ping or during the reset

  • There are TI-RTOS features which can be added to the RCP project through SyscConfig.  Where is the device inside the call stack if you are able to pause the debugger after the crash occurs?  Can you provide an updated otbr log?

    Regards,
    Ryan

  • I enabled the otbr-agent with -d 7 flag and this is the only logs i get.

    May 19 13:26:41 raspberrypi otbr-agent[78586]: 00:09:13.567 [W] Platform------: radio tx timeout
    May 19 13:26:41 raspberrypi otbr-agent[78586]: 00:09:13.567 [C] Platform------: HandleRcpTimeout() at radio_spinel_impl.hpp:2248: RadioSpinelNoResponse

    Why my stack or cpu usage is not showing up?

    What should i change here?

      

  • My apologies for the late response.  You are asking for CPU Load settings which would be made available from the TI RTOS -> UTILS -> Load module. There are several TI-RTOS 7 Configurations which are not made available inside of the SysConfig Editor, you can add/modify these by accessing the SysConfig file from a Text Editor.  That being said, I'm not certain whether/how these graphs can be enabled for SimpleLink devices.

    Focusing again on the crashing device state, where is the code located in the call stack when it is unresponsive?  Is it stuck in a loop or FaultISR?  Please try to load the symbols in CCS and determine where the PC is located.  

    Regards,
    Ryan

  • Ryan,

    I compiled from https://github.com/TexasInstruments/ot-ti and loaded the ot-rcp.out for the R7. It is forming as a Leader. Then i added one TI-CC2652R1 as MTD sensor, that one also compiled form git (ot-cli-mtd.out). Now I can see the sensor in the child table of the Leader but child is always in detached state. When i compile RCP image from SDK-CCS, it works fine. 

    factoryreset

    networkkey 13eb99025e216a54d06c647ceda46700

    dataset active commit

    ifconfig up

    thread start

    > ifconfig
    up
    Done
    > ipaddr
    fdde:ad00:beef:0:955a:1725:a45f:7e85
    fe80:0:0:0:3c3a:12a7:154c:ec7e
    Done
    > state
    detached

  • Are the channels and pan IDs aligned for these devices, and does the CLI MTD use the "thread start" command after "ifconfig up"?  You can reference either the ot-ti README or SDK README for command sets.

    Regards,
    Ryan

  • Yes, all matched. 

    joiner start J01NME command is not available in default ot-cli-mtd.out. So added panid, channel and network key manually at the sensor 
    before start the thread.

    > child table
    | ID | RLOC16 | Timeout | Age | LQ In | C_VN |R|D|N|Ver|CSL|QMsgCnt|Suprvsn| Extended MAC |
    +-----+--------+------------+------------+-------+------+-+-+-+---+---+-------+-------+------------------+
    | 1 | 0x9801 | 240 | 0 | 3 | 70 |1|0|0| 4| 0 | 0 | 129 | f696fe6b47adf213 |



    > ifconfig up
    Done
    > thread start
    Done
    > extaddr
    f696fe6b47adf213 <--- you can see this extaddr in child table but child is detached.
    Done
    > state
    detached
    Done



  • I will try to find the time to recreate this setup later in the week.

    Regards,
    Ryan

  • Hi Sen,

    I encountered the same issue as yourself, thank you for reporting this behavior.  I have asked the Software Development Team for further guidance.

    Regards,
    Ryan

  • Thanks Ryan for confirming the issue with GitHub build. Could you please guide me on original rcp timeout issue with SDK build? That build works fine. If you can reproduce that issue with SDK rcp image, that would be great. 

  • Please provide steps to reproduce the RCP issue using Thread end devices. 

    Regards,
    Ryan

  • Hi Ryan,

    Just ping from the sensor to an outside machine. If you have more sensors, you will see the rcp_timeout in otbr-agent sooner.

    i can reproduce the issue, with 3 sensors pinging outside Thread network: I kept 500 Bytes payload. You can increase, if you want to see it faster with 1 or 2 sensors. Even without any pinging, if i have 10 sensors, then i see that issue. 

    ping fdbe:ef11:1122:beef:dea6:32ff:fe12:2cd6 500 600 1 5 1 

  • Hi Sen,

    For the github build, did you set the networkkey on the MTD to match the RCP network?

    I believe I have recreated the behavior you've described with the v7.10 SDK with two child devices pinging every second.  However I notice that although the RPi host times out the RCP does not reset and remains active, in so much as that the RPi otbr-agent is able to re-establish the connection shortly after with no change to the RCP state.  Can you try a similar test and are you confident that the RCP is resetting as stated earlier?  Can you try increasing the timeout from the OTBR agent host?

    Regards,
    Ryan

  • Hi Ryan,

    For the GitHub build, I added the network key to MTD and had that child state = detached, even though leader has it in the child table. Then i tried with commissioning with PSKd and found the same issue. Only for GitHub, not for SDK.

    Yes, RCP is not resetting when "rcp timeout" occurs. When i set the OT_RCP_RESTORATION_MAX_COUNT=5, it works but "rcp timeout" still occurs at certain frequency. Sometimes I am loosing 40% packets when i have more sensors dur to multiple otbr restoration.  

    When I increase the timeout from OTBR, still it gets "rcp_timeout". never get back the reply from rcp.

    #define TX_WAIT_US (10 * US_PER_S)

    I would like have your help in finding out the root cause of the "rcp timeout".

    Thanks

    Sen

  • Have you set up a Thread sniffer to log the over-the-air activity of 10 or more devices which are not pinging?

    Regards,
    Ryan