This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

LP-CC1352P7: Openthread RCP at 460800 baudrate failing with RadioSpinelNoResponse

Part Number: LP-CC1352P7
Other Parts Discussed in Thread: CC2652P7

I am building Openthread RCP example from Simplelink 7.10.02.23 for LP_CC1352P7-4 board, however having trouble when I set the baudrate to 460800.

On starting otbr-agent it sends a bunch of spinel frames successfully but then consistently hangs when the call to RcpSetMac() occurs. It seems somewhat racy though as maybe 20% of the time, it does succeed without the timeout error.

Mar 23 23:57:50 6b091069403f otbr-agent[195]: 00:00:00.004 [D] P-RadioSpinel-: Sent spinel frame, flg:0x2, iid:0, tid:8, cmd:PROP_VALUE_SET, key:PHY_ENABLED, enabled:1
Mar 23 23:57:50 6b091069403f otbr-agent[195]: 00:00:00.004 [D] P-RadioSpinel-: Wait response: tid=8 key=32
Mar 23 23:57:50 6b091069403f otbr-agent[195]: 00:00:00.007 [D] P-RadioSpinel-: Received spinel frame, flg:0x2, iid:0, tid:8, cmd:PROP_VALUE_IS, key:PHY_ENABLED, enabled:1
Mar 23 23:57:50 6b091069403f otbr-agent[195]: 00:00:00.007 [D] P-RadioSpinel-: Sent spinel frame, flg:0x2, iid:0, tid:9, cmd:PROP_VALUE_SET, key:MAC_15_4_PANID, panid:0xffff
Mar 23 23:57:50 6b091069403f otbr-agent[195]: 00:00:00.007 [D] P-RadioSpinel-: Wait response: tid=9 key=54
Mar 23 23:57:50 6b091069403f otbr-agent[195]: 00:00:00.011 [D] P-RadioSpinel-: Received spinel frame, flg:0x2, iid:0, tid:9, cmd:PROP_VALUE_IS, key:MAC_15_4_PANID, panid:0xffff
Mar 23 23:57:50 6b091069403f otbr-agent[195]: 00:00:00.011 [D] P-RadioSpinel-: Sent spinel frame, flg:0x2, iid:0, tid:10, cmd:PROP_VALUE_SET, key:MAC_15_4_SADDR, saddr:0x0000
Mar 23 23:57:50 6b091069403f otbr-agent[195]: 00:00:00.011 [D] P-RadioSpinel-: Wait response: tid=10 key=53
Mar 23 23:57:50 6b091069403f otbr-agent[195]: 00:00:00.014 [D] P-RadioSpinel-: Received spinel frame, flg:0x2, iid:0, tid:10, cmd:PROP_VALUE_IS, key:MAC_15_4_SADDR, saddr:0x0000
Mar 23 23:57:50 6b091069403f otbr-agent[195]: 00:00:00.014 [D] P-RadioSpinel-: Sent spinel frame, flg:0x2, iid:0, tid:11, cmd:PROP_VALUE_GET, key:PHY_RX_SENSITIVITY
Mar 23 23:57:50 6b091069403f otbr-agent[195]: 00:00:00.014 [D] P-RadioSpinel-: Wait response: tid=11 key=39
Mar 23 23:57:50 6b091069403f otbr-agent[195]: 00:00:00.017 [D] P-RadioSpinel-: Received spinel frame, flg:0x2, iid:0, tid:11, cmd:PROP_VALUE_IS, key:PHY_RX_SENSITIVITY, sensitivity:-90
Mar 23 23:57:50 6b091069403f otbr-agent[195]: 00:00:00.018 [D] P-RadioSpinel-: Sent spinel frame, flg:0x2, iid:0, tid:12, cmd:PROP_VALUE_SET, key:RCP_MAC_KEY, keyIdMode:8, keyId:1, prevKey:***, currKey:***, nextKey:***
Mar 23 23:57:50 6b091069403f otbr-agent[195]: 00:00:00.018 [D] P-RadioSpinel-: Wait response: tid=12 key=2048
Mar 23 23:57:52 6b091069403f otbr-agent[195]: 00:00:02.020 [W] P-RadioSpinel-: Wait for response timeout
Mar 23 23:57:52 6b091069403f otbr-agent[195]: 00:00:02.020 [C] Platform------: HandleRcpTimeout() at radio_spinel.cpp:2092: RadioSpinelNoResponse



Tested using latest OTBR Docker image, but also same results in the Home Asssitant OTBR Addon.

Are there any other settings I need to tweak to get this working at 460800 baudrate?

  • Also when it does startup successfully it pretty quickly falls over with this error:

    otbr-agent[167]: 00:00:46.618 [I] MeshForwarder-: Sent IPv6 UDP msg, len:84, chksum:8fdd, ecn:no, to:0xffff, sec:no, prio:net, radio:all
    otbr-agent[167]: 00:00:46.618 [I] MeshForwarder-:     src:[fe80:0:0:0:f086:1ecd:e5bc:c1a5]:19788
    otbr-agent[167]: 00:00:46.618 [I] MeshForwarder-:     dst:[ff02:0:0:0:0:0:0:2]:19788
    otbr-agent[167]: 00:00:47.083 [I] MeshForwarder-: Received IPv6 UDP msg, len:129, chksum:015e, ecn:no, from:7ed41eb1cab938dd, sec:no, prio:net, rss:-64.0, radio:15.4
    otbr-agent[167]: 00:00:47.083 [I] MeshForwarder-:     src:[fe80:0:0:0:7cd4:1eb1:cab9:38dd]:19788
    otbr-agent[167]: 00:00:47.083 [I] MeshForwarder-:     dst:[fe80:0:0:0:f086:1ecd:e5bc:c1a5]:19788
    otbr-agent[167]: 00:00:47.084 [I] Mle-----------: Receive Parent Response (fe80:0:0:0:7cd4:1eb1:cab9:38dd,0xdc00)
    otbr-agent[167]: 00:00:47.084 [I] RadioSelector-: RadioSelector: NewRadio(OnRx) 15.4 - neighbor:[7ed41eb1cab938dd rloc16:0xdc00 radio-pref:{15.4:200} state:ParentRes]
    otbr-agent[167]: 00:00:47.363 [I] Mle-----------: Send Child ID Request (fe80:0:0:0:7cd4:1eb1:cab9:38dd)
    otbr-agent[167]: 00:00:47.363 [I] Mle-----------: AttachState ParentReq -> ChildIdReq
    otbr-agent[167]: 00:00:48.613 [I] Mle-----------: AttachState ChildIdReq -> Idle
    otbr-agent[167]: 00:00:51.557 [I] Mle-----------: Send Advertisement (ff02:0:0:0:0:0:0:1)
    otbr-agent[167]: 00:00:52.364 [W] P-RadioSpinel-: radio tx timeout
    otbr-agent[167]: 00:00:52.364 [C] Platform------: HandleRcpTimeout() at radio_spinel.cpp:2092: RadioSpinelNoResponse
    [14:59:56] WARNING: otbr-agent exited with code 6 (by signal 0).

  • What is the correct location to be building openthread RCP from? I notice openthread is removed in the 7.40 SDK?

    I tried building from this branch, and seems happier at 460800 so far
    github.com/.../

  • Even the above build is still not very stable and crashes out in a few hours, with just 2 thread devices connected

    otbr-agent[167]: 00:04:07.792 [I] MeshForwarder-:     src:[fdae:4848:e485:1:73b5:1463:d663:4a65]:34301
    otbr-agent[167]: 00:04:07.793 [I] MeshForwarder-:     dst:[fdae:4848:e485:1:725c:5a4f:d67a:a3cc]:5540
    otbr-agent[167]: 00:04:07.822 [I] MeshForwarder-: Sent IPv6 UDP msg, len:105, chksum:0683, ecn:no, to:0xdc00, sec:yes, prio:low, radio:15.4
    otbr-agent[167]: 00:04:07.822 [I] MeshForwarder-:     src:[fdae:4848:e485:1:73b5:1463:d663:4a65]:34301
    otbr-agent[167]: 00:04:07.822 [I] MeshForwarder-:     dst:[fdae:4848:e485:1:725c:5a4f:d67a:a3cc]:5540
    otbr-agent[167]: 00:04:07.932 [I] MeshForwarder-: Received IPv6 UDP msg, len:131, chksum:f603, ecn:no, from:0xdc00, sec:yes, prio:normal, rss:-63.5, radio:15.4
    otbr-agent[167]: 00:04:07.932 [I] MeshForwarder-:     src:[fdae:4848:e485:1:725c:5a4f:d67a:a3cc]:5540
    otbr-agent[167]: 00:04:07.932 [I] MeshForwarder-:     dst:[fdae:4848:e485:1:73b5:1463:d663:4a65]:34301
    otbr-agent[167]: 00:04:07.967 [I] MeshForwarder-: Sent IPv6 UDP msg, len:82, chksum:9449, ecn:no, to:0xdc00, sec:yes, prio:low, radio:15.4
    otbr-agent[167]: 00:04:07.967 [I] MeshForwarder-:     src:[fdae:4848:e485:1:73b5:1463:d663:4a65]:34301
    otbr-agent[167]: 00:04:07.968 [I] MeshForwarder-:     dst:[fdae:4848:e485:1:725c:5a4f:d67a:a3cc]:5540
    otbr-agent[167]: 00:04:17.710 [I] Mle-----------: Send Advertisement (ff02:0:0:0:0:0:0:1)
    otbr-agent[167]: 00:04:17.710 [C] Platform------: Write() at hdlc_interface.cpp:253: Input/output error

  • Hi,

    Correct, it's recommended to build from the github.

    Can you try with this branch?
    https://github.com/TexasInstruments/ot-ti/tree/release/thread-1.3-certification-support

    Thanks,
    Toby

  • Hi Toby,
      That is the branch that I built from on Github. It starts up fine and runs for a few hours but crashes out after some time with either RX timeout or IO errors.

    Also for reference running this with upstream otbr agent from commit 2279c02f3c3373f074899fc8d993b8ddb72910a2


    - Tim

  • Have you tested with other baudrates?

    The default one TI tests is 115200.

  • No I havent tested 115200 with the git branch, I will try it and see if the issues persist. However ultimately 115200 is too slow for a Openthread RCP, especially with no flow control.

  • I tested with 115200 baudrate and seem to face the same issues:

    otbr-agent[175]: 01:20:21.553 [I] MeshForwarder-:     src:[fd6b:ad5a:8eb0:1:9e2c:4bc6:3dd0:66b2]:5540
    otbr-agent[175]: 01:20:21.553 [I] MeshForwarder-:     dst:[fd6b:ad5a:8eb0:1:b388:2732:4775:bede]:45095
    otbr-agent[175]: 01:20:22.219 [I] MeshForwarder-: Sent IPv6 UDP msg, len:105, chksum:f050, ecn:no, to:0xdc00, sec:yes, prio:low, radio:15.4
    otbr-agent[175]: 01:20:22.220 [I] MeshForwarder-:     src:[fd6b:ad5a:8eb0:1:b388:2732:4775:bede]:45095
    otbr-agent[175]: 01:20:22.220 [I] MeshForwarder-:     dst:[fd6b:ad5a:8eb0:1:9e2c:4bc6:3dd0:66b2]:5540
    otbr-agent[175]: 01:20:22.326 [I] MeshForwarder-: Received IPv6 UDP msg, len:131, chksum:2571, ecn:no, from:0xdc00, sec:yes, prio:normal, rss:-48.0, radio:15.4
    otbr-agent[175]: 01:20:22.326 [I] MeshForwarder-:     src:[fd6b:ad5a:8eb0:1:9e2c:4bc6:3dd0:66b2]:5540
    otbr-agent[175]: 01:20:22.327 [I] MeshForwarder-:     dst:[fd6b:ad5a:8eb0:1:b388:2732:4775:bede]:45095
    otbr-agent[175]: 01:20:24.639 [I] Mle-----------: Send Advertisement (ff02:0:0:0:0:0:0:1)
    otbr-agent[175]: 01:20:27.330 [W] P-RadioSpinel-: radio tx timeout
    otbr-agent[175]: 01:20:27.330 [C] Platform------: HandleRcpTimeout() at radio_spinel.cpp:2092: RadioSpinelNoResponse
    [21:58:32] WARNING: otbr-agent exited with code 6 (by signal 0).

  • Thanks for the update.

    I'll look into this, expect an update in ~3 business days.

  • Thanks Toby

  • Hi Tim,

    Reading through this post again, I remembered a previous post which detailed a similar issue: https://e2e.ti.com/support/wireless-connectivity/zigbee-thread-group/zigbee-and-thread/f/zigbee-thread-forum/1226166/cc2652r-otbr-rcp-radiospinelnoresponse-radio-tx-timeout#pifragment-323417=1

    The fix that was introduced there was (Nov 20, 2023): https://github.com/TexasInstruments/ot-ti/commit/cc4e6d111ca18828164a7abc190bbea3e3fead8e

    Since then, there have been other changes introduced, and I wonder if somehow those are affecting the RCP performance, with regards to the stability of the serial (UART) communication.

    Can you checkout the commit above (Nov 20, 2023) and re-build the RCP, to see if the issue is resolved?
    If it is, then I will need to work with my R&D team to see 

  • Sure I will test with that commit and report back how it goes.

  • Building from commit cc4e6d111ca18828164a7abc190bbea3e3fead8e , otbr starts ok, but fails to bring up the Thread network properly.

    I had to cherry-pick PR #9 (74813db170c24d7c54525608821915a75b5e2e13) as well for things to work. I will let this run for a while and see if there are any changes in stability.

  • I will keep monitoring this, but so far has been stable with no RCP dropouts over about 11hrs.

  • Thanks for the confirming.

    I'll reach out to our R&D team to see if they might have some ideas.

  • 24hrs now without a dropout. I will try the same build at 460800 baud and see how it goes.

  • Thanks Tim!

  • 460800 dropped out after 18 hrs with the " Input/output error" shown previously. This is an improvement on my initial testing (with latest commits) though

  • Thanks for the update.

    We are working to setup a test to catch this regression.

    Can you share other parameters in your test? So far I have the following, and also some requested info:

    1. 2 Thread devices connected -- are these both directly connected to the RCP (like Thread_0 <--> RCP <--> Thread_1)? Or is it a mesh (like RCP <--> Thread_0 <--> Thread_1)?
    2. What does the traffic profile look like from RCP perspective (how often packets are received/transmitted, how often the other Thread devices are transmitting, etc)
    3. Size of Thread payload per packet
    4. Environment: radio chamber (isolated) or typical (e.g. office space)

    I'll keep you posted on updates from my end.

    Some other suggestions to potentially try in the meantime:

    1. Use UART in blocking mode: https://github.com/TexasInstruments/ot-ti/blob/cc4e6d111ca18828164a7abc190bbea3e3fead8e/src/uart.c#L50 
    2. Increase size of UART queue (perhaps to 10): https://github.com/TexasInstruments/ot-ti/blob/cc4e6d111ca18828164a7abc190bbea3e3fead8e/src/uart.c#L56 

    Additionally, here is a FW image of RCP for CC1352P7-4 with latest changes (not yet merged into github):
    ot-rcp-LP_CC1354P7-4.out

  • Hi Toby,

    1. Currently the two devices I am testing with are Eve Motion + Eve Plug AU, both are connected directly to RCP
    2. Frames seem to come in roughly every 3secs but with bursts a few per second.
    3. sizes range for 30bytes ~ 120bytes
    4. Typical office space with wifi access point and a couple of zigbee networks, however channels are selected to avoid overlap. Roughly 2-3m line of sight between RCP and devices.

    I will add that I have not observed the rx timeout errors with the current test build, only the input/output errors.

    I will try your suggestions and pre built FW image over the next few days.

  • Hi Tim,

    Thanks for the update, these will be useful parameters for our internal testing.

    Will let you know when I hear back about the progress of that testing.

    Regards,
    Toby

  • Additionally, here is a FW image of RCP for CC1352P7-4 with latest changes (not yet merged into github):
    ot-rcp-LP_CC1354P7-4.out

    Additionally, here is a FW image of RCP for CC1352P7-4 with latest changes (not yet merged into github):
    ot-rcp-LP_CC1354P7-4.out

    Hi Toby,
      I ran this build all weekend, without any RCP Errors occurring in that time. Which is a definite improvement on any other builds i have tried! When will these changes be released to github?

  • Hi Tim,

    I'm glad to hear!

    I expect this to be released by the end of April in a github branch.

  • Still seeing occasional RCP errors with this build, but timeframe of dropouts occuring now is probably 2-3 days.

    otbr-agent[167]: 22:12:26.896 [I] MeshForwarder-:     src:[fe80:0:0:0:a0d9:2ab8:2b4:3d49]:19788
    otbr-agent[167]: 22:12:26.896 [I] MeshForwarder-:     dst:[ff02:0:0:0:0:0:0:1]:19788
    otbr-agent[167]: 22:12:31.440 [N] MeshForwarder-: Dropping (reassembly queue) IPv6 UDP msg, len:154, chksum:0be0, ecn:no, sec:yes, error:ReassemblyTimeout, prio:normal, rss:-39.0, radio:15.4
    otbr-agent[167]: 22:12:31.440 [N] MeshForwarder-:     src:[fdae:4848:e485:1:2a98:131c:ea85:ca42]:5540
    otbr-agent[167]: 22:12:31.441 [N] MeshForwarder-:     dst:[fdae:4848:e485:1:73b5:1463:d663:4a65]:44446
    otbr-agent[167]: 22:12:32.352 [I] Mle-----------: Send Advertisement (ff02:0:0:0:0:0:0:1)
    otbr-agent[167]: 22:12:33.418 [I] MeshForwarder-: Received IPv6 UDP msg, len:92, chksum:4565, ecn:no, from:a2d92ab802b43d49, sec:no, prio:net, rss:-20.0, radio:trel
    otbr-agent[167]: 22:12:33.418 [I] MeshForwarder-:     src:[fe80:0:0:0:a0d9:2ab8:2b4:3d49]:19788
    otbr-agent[167]: 22:12:33.419 [I] MeshForwarder-:     dst:[ff02:0:0:0:0:0:0:1]:19788
    otbr-agent[167]: 22:12:33.419 [I] Mle-----------: Receive Advertisement (fe80:0:0:0:a0d9:2ab8:2b4:3d49,0x8800)
    otbr-agent[167]: 22:12:37.354 [W] P-RadioSpinel-: radio tx timeout
    otbr-agent[167]: 22:12:37.354 [C] Platform------: HandleRcpTimeout() at radio_spinel.cpp:2092: RadioSpinelNoResponse

  • it actually appears that the device completely hangs up when these radio timeouts occur and simply restarting otbr is not sufficient to get it running again.

    otbr-agent[167]: [NOTE]-AGENT---: Running 0.3.0-2279c02-dirty
    otbr-agent[167]: [NOTE]-AGENT---: Thread version: 1.3.0
    otbr-agent[167]: [NOTE]-AGENT---: Thread interface: wpan0
    otbr-agent[167]: [NOTE]-AGENT---: Radio URL: spinel+hdlc+uart:///dev/ttyACM0?uart-baudrate=460800
    otbr-agent[167]: [NOTE]-AGENT---: Radio URL: trel://end1
    otbr-agent[167]: [NOTE]-ILS-----: Infra link selected: end1
    otbr-agent[167]: [INFO]-NCP-----: OpenThread log level changed to 4
    otbr-agent[167]: 50d.02:27:29.023 [W] P-RadioSpinel-: Wait for response timeout
    otbr-agent[167]: 50d.02:27:29.023 [I] P-RadioSpinel-: RCP self reset successfully
    otbr-agent[167]: 50d.02:27:31.025 [W] P-RadioSpinel-: Wait for response timeout
    otbr-agent[167]: 50d.02:27:31.025 [C] P-RadioSpinel-: Failed to communicate with RCP - no response from RCP during initialization
    otbr-agent[167]: 50d.02:27:31.025 [C] P-RadioSpinel-: This is not a bug and typically due a config error (wrong URL parameters) or bad RCP image:
    otbr-agent[167]: 50d.02:27:31.025 [C] P-RadioSpinel-: - Make sure RCP is running the correct firmware
    otbr-agent[167]: 50d.02:27:31.025 [C] P-RadioSpinel-: - Double check the config parameters passed as `RadioURL` input
    otbr-agent[167]: 50d.02:27:31.025 [C] Platform------: HandleRcpTimeout() at radio_spinel.cpp:2092: RadioSpinelNoResponse

  • So after a week of testing, I can say that both errors are still occurring with the latest build just on a much less frequent basis.

    - rx/tx timeout errors seem to occur along with a hard lockup of cc2652P7 (not sure if this is the cause of timeout or a symptom)
    - I/O errors dont lockup, so a restart of otbr does bring back up RCP.

  • Hi Tim,

    Thanks for the updates.

    Our RnD team is currently investigating RCP stability, current plan is to resolve this before the next release (scheduled for end of April).

    I'll provide an update here as I receive them.

  • Hi Tim,

    I have received an update from RnD.

    After they changed UART to blocking mode, it seems to be more stable (passes 24hr test) -- https://github.com/TexasInstruments/ot-ti/blob/6f30243676724ef1472d20098e00eb20b3f20679/src/uart.c#L50 
    This change (among others) should be included in the release scheduled for end of April.

    We believe the root cause may have some relation between the XDS/UART (since UART is routed through the XDS), and this is planned to be further investigated after the April release.

    Thanks,
    Toby

  • We believe the root cause may have some relation between the XDS/UART

    While I do test on the LP_CC1352 board, our actual devices use CP2102N UART and we do generally see the same issues on these (although I unable to test that last build you sent as these have different Xosc Tuning values)

    After they changed UART to blocking mode, it seems to be more stable (passes 24hr test) -- https://github.com/TexasInstruments/ot-ti/blob/6f30243676724ef1472d20098e00eb20b3f20679/src/uart.c#L50 

    I will try this change against existing tree and see if makes any difference. btw 24hrs test may not be long enough though to ensure stability.

  • btw 24hrs test may not be long enough though to ensure stabili

    I'll pass this feedback to the team.

  • Hi Toby,
      Any updates on the new release?

  • Hi Tim,

    Toby is out of office this week, please allow him some time to get situated next Monday and I am sure he will be able to get back in touch with you by Tuesday or Wednesday with an update.

    Regards,
    Ryan

  • Hi Tim,

    Certification process is in progress, mainly logistical (e.g. getting cert id(s)).

    My estimate would be end of week, subject to delays associated with external certification process(es).

    Thanks,
    Toby

  • Thanks for the update Toby

  • Hi Toby,
      Any updates on when this will be released?

  • Hi Tim,

    I've checked and seen that it has not been released yet...

    Our team is currently is actively handling it.

    In the meantime, I can potentially provide a RCP FW to you to test from our cert branch. (I can share as early as tomorrow).
    Just to confirm what you'd need:

    1. LP_CC1352P7-4
    2. baudrate of 460800

    Thanks,
    Toby

    1. LP_CC1352P7-4
    2. baudrate of 460800

    Yes this is what I would need for testing, thanks

  • Please find attached, let me know if it is working for you.

    ot-rcp-460800baud-LP_CC1352P7-4_ce396456.out

  • Thanks Toby, I will set this up and monitor stability over the coming days 

  • Still seeing the otbr agent crash out with IO Errors

    otbr-agent[172]: 00:04:47.632 [I] Mle-----------: Send Advertisement (ff02:0:0:0:0:0:0:1)
    otbr-agent[172]: 00:04:47.632 [C] Platform------: Write() at hdlc_interface.cpp:253: Input/output error
    [18:59:45] WARNING: otbr-agent exited with code 5 (by signal 0).

  • Sure, will test it.