CC2652P7: When the otbr-agent exits during traffic generation, how to troubleshoot the issue?

Hi zhimin,

Can you please provide the log of the otbr-agent, like in these examples

https://groups.google.com/g/openthread-users/c/wqR1URcAUxM?pli=1 

tail -f /var/log/syslog | grep otbr > otbr.txt

https://github.com/openthread/ot-br-posix/issues/742 

cat /var/log/syslog | grep otbr-agent

It would also help if you supplied the ot-br-posix and ot-ti repository branch IDs.  The spaces I hyperlinked would also be good locations for you to post your issue in case this is a ot-br-posix issue and not related to the CC2652P7 RCP.  Otherwise you can run the RCP image in the CCS debugger to determine whether the device is in an undefined state.

Regards,
Ryan

otbr-agent-log.txthello, there is no /var/log/syslog file in my device. This is the print log of otbr-agent before it exited. Can you tell what information can be gleaned from it?

This is enough to tell that a "radio tx timeout" occurs followed by "RCP failure detected" from which "Software reset RCP" does not recover.  So this appears to be an issue with the RCP firmware crashing.

Ryan Brown1 said:

It would also help if you supplied the ot-br-posix and ot-ti repository branch IDs.

Ryan Brown1 said:

you can run the RCP image in the CCS debugger to determine whether the device is in an undefined state.

Also, how does one replicate the behavior quickly and reliably?

Regards,
Ryan

I use CCS (Code Composer Studio) with an XDS110 debug probe to debug the Thread chip. After an issue occurred during Thread traffic testing, I used the pause function in CCS, and the Thread program halted at the following call stack position. It seems the Thread chip got stuck while executing a reset command. Could you please explain why a reset command is issued during Thread traffic testing? And which part of the otbr-agent does this reset command correspond to?

CC1354P10.ccxml [Code Composer Studio - Device Debugging]
Texas Instruments XDS110 USB Debug Probe_0/Cortex_M4_0 (Suspended)
NOROM_RFCDoorbellSendTo() at rfc.c:104 0x0000D30C
RF_executeDirectImmediateCmd() at RFCC26X2_multiMode.c:4,041 0x0000B746
RF_executeDirectImmediateCmd() at RFCC26X2_multiMode.c:4,032 0x0000B740
RF_runDirectImmediateCmd() at RFCC26X2_multiMode.c:4,094 0x0000B736
rfCoreModifyRxAutoPend() [Y:\0_workspace\6_CC2652\1_git_1204\TI-OT-CC13XX_CC26XX-RCP\build\bin\ot-rcp.out] at 0x00003E3E
otPlatRadioEnableSrcMatch() at radio.c:1,760 0x00003E18
ot::Radio::EnableSrcMatch(bool) [Y:\0_workspace\6_CC2652\1_git_1204\TI-OT-CC13XX_CC26XX-RCP\build\bin\ot-rcp.out] at 0x000072F6
ot::Radio::Init() at radio.cpp:47 0x000072F0
ot::Instance::ResetRadioStack() at instance.cpp:319 0x0000524E
ot::Ncp::NcpBase::CommandHandler_RESET(unsigned char)() at ncp_base.cpp:1,204 0x000012BC
ot::Ncp::NcpBase::CommandHandler_RESET(unsigned char)() at ncp_base.cpp:1,187 0x000012B6
ot::Ncp::NcpBase::HandleCommand(unsigned char)() at ncp_base.cpp:894 0x0000196C
ot::Ncp::NcpBase::HandleReceive(unsigned char const*, unsigned short)() at ncp_base.cpp:360 0x000019DE
ot::Ncp::NcpHdlc::HandleFrame(otError)() at multi_frame_buffer.hpp:146 0x00000C32
ot::Hdlc::Decoder::Decode(unsigned char const*, unsigned short)() at hdlc.cpp:261 0x00011CF8
platformUartProcess() at uart.c:189 0x00004AC0
otSysProcessDrivers() at system.c:256 0x0000498C
app_main() at main.c:102 0x00000220

The ot-br-posix Linux solution is calling for a RCP reset after the device becomes unresponsive (RCP timeout), however this is ineffective given that the MCU seems to be stuck inside of NOROM_RFCDoorbellSendTo.

Similar behavior has been observed by TI in the past and thus I would ask for you to try the following change to the ti_radio_config.c pOverrides.  This file is generated by SysConfig automatically, so additional steps are necessary to use a customized ti_radio_config.c file before rebuilding:

1. Build the target project via ./script/build <BOARD>
2. Open ot-ti/src/CMakeLists.txt
   1. Comment out the ti_devices_config.h file listed in the SYSCONFIG_OUTPUT_C structure

      1. if(TI_SIMPLELINK_KERNEL STREQUAL "freertos")
             set(SYSCONFIG_OUTPUT_C
                 ${CMAKE_CURRENT_BINARY_DIR}/syscfg/ti_devices_config.c
         #        ${CMAKE_CURRENT_BINARY_DIR}/syscfg/ti_devices_config.h
                 ${CMAKE_CURRENT_BINARY_DIR}/syscfg/ti_drivers_config.c
                 ${CMAKE_CURRENT_BINARY_DIR}/syscfg/ti_drivers_config.h

         
         
   2. Comment out the sysconfig custom command as follows:

      1. #add_custom_command(
         #    OUTPUT
         #        ${SYSCONFIG_OUTPUT_C}
         #        ${SYSCONFIG_OUTPUT_OTHER}
         #    COMMAND
         #        ${syscfg_cmd}
         #    DEPENDS
         #        ${sysconfig_file} 
         #    VERBATIM
         #)

         
         
3. Open ti_radio_config.c in the project's build directory: ot-ti/build/src/syscfg/ti_radio_config.c
   1. Append the following overrides to the pOverrides[] array before the last element of (uint32_t)0xFFFFFFFF

      1. // Overrides for CMD_RADIO_SETUP_PA
         uint32_t pOverrides[] =
         {
             // override_ieee_802_15_4_10_dbm.json
             // Rx: Set LNA bias current offset to +15 to saturate trim to max (default: 0)
             (uint32_t)0x000F8883,
             // Set VCTRIM to 0 for 20 dBm PA
             ADI_REG_OVERRIDE(1,26,0x00),
             // override_ieee_coex.json
             // CoEx: set the radio IOMUX to enable coex signals
             HW_REG_OVERRIDE(0x1110,0xCE30),
             // CoEx: Enable CoEx
             (uint32_t)0x000188F3,
             // CoEx: Enable 32 bit write for the pointer
             (uint32_t)0xC0040371,
             // CoEx: Place the pointer to the configuration struct at byte index 220
             (uint32_t)&coexConfig,
             // START OF ADDED CONTENT
             // IEEE 802.15.4: Set pilot tone length to 35 us
             HW_REG_OVERRIDE(0x5320,0x0690),
             // IEEE 802.15.4: Set pilot tone length to 35 us
             HW_REG_OVERRIDE(0x6024,0x5B20),
             // IEEE 802.15.4: Enable fast settling during pilot tone
             (uint32_t)0x10108413,
             // IEEE 802.15.4: Compensate for modified pilot tone length
             (uint32_t)0x034902A3,
             // END OF ADDED CONTENT
             (uint32_t)0xFFFFFFFF
         };

         
         
4. Build the target project via ./script/build <BOARD>

Regards,
Ryan

After modifying the pOverrides, the frequency of unresponsiveness during Thread traffic testing has indeed decreased. Before the modification, the device would become unresponsive after 3 to 5 minutes of traffic testing, while after the modification, it can now run for more than 20 minutes before unresponsiveness occurs. Could we further modify the pilot tone length parameter? If so, for example, if we set the pilot tone length parameter to 50 us, what would be the corresponding register value?

I'm glad to hear that the situation has improved, unfortunately I do not have further expertise to advise on additional changes.  How much time does the Thread traffic testing need, and is this a requirement for any protocol certifications?  Would duty cycling, or decreasing packet intervals, be a valid workaround to this issue?  Are you enabling WiFi coexistence in your design?  And are you able to replicate this behavior with LAUNCHXL-CC2652P7 EVMs?

I realized my previous response does not have "(uint32_t)0x00000263," added to the end of the pOverrides, which you will want to try.

// Overrides for CMD_RADIO_SETUP_PA
uint32_t pOverrides[] =
{
    // override_ieee_802_15_4_10_dbm.json
    // Rx: Set LNA bias current offset to +15 to saturate trim to max (default: 0)
    (uint32_t)0x000F8883,
    // Set VCTRIM to 0 for 20 dBm PA
    ADI_REG_OVERRIDE(1,26,0x00),
    // override_ieee_coex.json
    // CoEx: set the radio IOMUX to enable coex signals
    HW_REG_OVERRIDE(0x1110,0xCE30),
    // CoEx: Enable CoEx
    (uint32_t)0x000188F3,
    // CoEx: Enable 32 bit write for the pointer
    (uint32_t)0xC0040371,
    // CoEx: Place the pointer to the configuration struct at byte index 220
    (uint32_t)&coexConfig,
    // START OF ADDED CONTENT
    // IEEE 802.15.4: Set pilot tone length to 35 us
    HW_REG_OVERRIDE(0x5320,0x0690),
    // IEEE 802.15.4: Set pilot tone length to 35 us
    HW_REG_OVERRIDE(0x6024,0x5B20),
    // IEEE 802.15.4: Enable fast settling during pilot tone
    (uint32_t)0x10108413,
    // IEEE 802.15.4: Compensate for modified pilot tone length
    (uint32_t)0x034902A3,
    (uint32_t)0x00000263,
    // END OF ADDED CONTENT
    (uint32_t)0xFFFFFFFF
};

You may also consider trying this updated library folder https://e2e.ti.com/cfs-file/__key/communityserver-discussions-components-files/158/conan_5F00_package-_2800_1_2900_.tgz

Please download the .tgz file, extract it, then merge only the source folder (i.e. not the .metadata or docs folders) with the existing ot-ti/third_party/ti_simplelink_sdk/repo/source directory (i.e. only replace duplicate files, do not delete original files which are not updated).  You will need to delete the build folders from both ot-ti/third_party/ti_simplelink_sdk/repo and ot-ti directories, then rebuild the RCP image before loading the new image onto your device.

Let me know if any combination of these suggestions helps improve performance.

Regards,
Ryan