This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

PROCESSOR-SDK-AM437X: Getting timeout message when we enable CAN interface

Part Number: PROCESSOR-SDK-AM437X

Hi,

For the below-attached post, Sorry, I was unable to reply the my post, so I have created a new thread to reply 

PROCESSOR-SDK-AM437X: Getting timeout message when we enable CAN interface. - Processors forum - Processors - TI E2E support forums

[Monish]: 

We are doing a continuous environmental test for all the interfaces available in our custom board. Each interface tests are sequential and each iteration will take around 1.5 mins to complete testing all the interfaces in the custom board. And we are using two CAN interfaces in our custom board and we do loopback both the ports for environmental testing. And we can transmit and receive messages on both ports. We set the bitrate and enable the CAN ports by using these commands.

#ip link set can0 type can bitrate 1000000 # set bit rate
#ip link set can0 up # set can0 up

#ip link set can1 type can bitrate 1000000 # set bit rate
#ip link set can1 up # set can1 up

When we use the above command, we are getting the timeout message like below for each iteration (Log from Syslog). Why we are getting a timeout message? Whether the frequent timeout error will cause any problem if the system runs for more than 24 hours.

2021-06-04T14:07:42.535150+00:00 am437x-custom kernel: [10615.455475] net can0: c_can_hw_raminit_wait_syscon: time out
2021-06-04T14:07:42.535237+00:00 am437x-custom kernel: [10615.461843] c_can_platform 481cc000.can can0: setting BTR=2701 BRPE=0000
2021-06-04T14:07:42.584839+00:00 am437x-custom kernel: [10615.512659] net can1: c_can_hw_raminit_wait_syscon: time out
2021-06-04T14:07:42.595194+00:00 am437x-custom kernel: [10615.519524] c_can_platform 481d0000.can can1: setting BTR=2701 BRPE=0000
2021-06-04T14:07:52.615153+00:00 am437x-custom kernel: [10625.533893] c_can_platform 481cc000.can can0: setting BTR=2701 BRPE=0000
2021-06-04T14:07:52.665631+00:00 am437x-custom kernel: [10625.584586] net can1: c_can_hw_raminit_wait_syscon: time out
2021-06-04T14:07:52.665726+00:00 am437x-custom kernel: [10625.590801] c_can_platform 481d0000.can can1: setting BTR=2701 BRPE=0000
2021-06-04T14:08:01.450176+00:00 am437x-custom crond[1000]: (root) WRONG FILE OWNER (/var/spool/cron/root)
2021-06-04T14:09:01.451354+00:00 am437x-custom crond[1000]: (root) WRONG FILE OWNER (/var/spool/cron/root)
2021-06-04T14:09:07.065079+00:00 am437x-custom kernel: [10699.987593] c_can_platform 481cc000.can can0: setting BTR=2701 BRPE=0000
2021-06-04T14:09:07.115002+00:00 am437x-custom kernel: [10700.038186] net can1: c_can_hw_raminit_wait_syscon: time out
2021-06-04T14:09:07.115089+00:00 am437x-custom kernel: [10700.044657] c_can_platform 481d0000.can can1: setting BTR=2701 BRPE=0000
2021-06-04T14:09:17.144960+00:00 am437x-custom kernel: [10710.067504] net can0: c_can_hw_raminit_wait_syscon: time out
2021-06-04T14:09:17.145153+00:00 am437x-custom kernel: [10710.073994] c_can_platform 481cc000.can can0: setting BTR=2701 BRPE=0000
2021-06-04T14:09:17.205233+00:00 am437x-custom kernel: [10710.122861] c_can_platform 481d0000.can can1: setting BTR=2701 BRPE=0000

[Schuyler Patton]

Which version of the kernel are you using?

Could you please post the output for each interface?

ip -d -s link show <can interface>

Regards, 

Monish P

  • Please find the below response,

    [TI] Which version of the kernel are you using?

    [Monish] Kernel version: 4.19

    [TICould you please post the output for each interface?

    [Monish] We are doing loopback between CAN0 and CAN1. First, we are transmitting messages from CAN0 and receiving them through CAN1 and then vice-versa. Please find the below output.

    CAN0 Transmit.....CAN1 Receive

    interface = can0, family = 29, type = 3, proto = 1
    interface = can0, family = 29, type = 3, proto = 1
    interface = can0, family = 29, type = 3, proto = 1
    interface = can0, family = 29, type = 3, proto = 1
    interface = can0, family = 29, type = 3, proto = 1
    interface = can0, family = 29, type = 3, proto = 1
    interface = can0, family = 29, type = 3, proto = 1
    interface = can0, family = 29, type = 3, proto = 1
    interface = can0, family = 29, type = 3, proto = 1
    interface = can0, family = 29, type = 3, proto = 1
    interface = can0, family = 29, type = 3, proto = 1

    Received Messages from CAN0

    read: Network is down
    interface = can1, family = 29, type = 3, proto = 1
    <0x00a> [8] 08 07 06 05 04 03 02 01
    <0x00a> [8] 08 07 06 05 04 03 02 01
    <0x00a> [8] 08 07 06 05 04 03 02 01
    <0x00a> [8] 08 07 06 05 04 03 02 01
    <0x00a> [8] 08 07 06 05 04 03 02 01
    <0x00a> [8] 08 07 06 05 04 03 02 01
    <0x00a> [8] 08 07 06 05 04 03 02 01
    <0x00a> [8] 08 07 06 05 04 03 02 01
    <0x00a> [8] 08 07 06 05 04 03 02 01
    <0x00a> [8] 08 07 06 05 04 03 02 01
    <0x00a> [8] 08 07 06 05 04 03 02 01


    ./pre_compliance_i2c_lcd_remove.sh: line 847: 23340 Killed ( while true; do
    msg_candump="$( candump can1 )"; if [[ ${#msg_candump} > 1 ]]; then
    echo "$msg_candump";
    fi; sleep 0.025;
    done )
    CAN1 Transmit.....CAN0 Receive

    interface = can1, family = 29, type = 3, proto = 1
    interface = can1, family = 29, type = 3, proto = 1
    interface = can1, family = 29, type = 3, proto = 1
    interface = can1, family = 29, type = 3, proto = 1
    interface = can1, family = 29, type = 3, proto = 1
    interface = can1, family = 29, type = 3, proto = 1
    interface = can1, family = 29, type = 3, proto = 1
    interface = can1, family = 29, type = 3, proto = 1
    interface = can1, family = 29, type = 3, proto = 1
    interface = can1, family = 29, type = 3, proto = 1
    interface = can1, family = 29, type = 3, proto = 1

    Received Messages from CAN1

    read: Network is down
    interface = can0, family = 29, type = 3, proto = 1
    <0x005> [8] 01 02 03 04 05 06 07 08
    <0x005> [8] 01 02 03 04 05 06 07 08
    <0x005> [8] 01 02 03 04 05 06 07 08
    <0x005> [8] 01 02 03 04 05 06 07 08
    <0x005> [8] 01 02 03 04 05 06 07 08
    <0x005> [8] 01 02 03 04 05 06 07 08
    <0x005> [8] 01 02 03 04 05 06 07 08
    <0x005> [8] 01 02 03 04 05 06 07 08
    <0x005> [8] 01 02 03 04 05 06 07 08
    <0x005> [8] 01 02 03 04 05 06 07 08
    <0x005> [8] 01 02 03 04 05 06 07 08

    read: Network is down
    ./pre_compliance_i2c_lcd_remove.sh: line 847: 23376 Killed ( while true; do
    msg_candump="$( candump can0 )"; if [[ ${#msg_candump} > 1 ]]; then
    echo "$msg_candump";
    fi; sleep 0.025;
    done )

    [TIip -d -s link show <can interface>

    [Monish] Please find the below output image.

    Regards,

    Monish P

  • Hi,

    Thank you for the detailed explanation.  Looking through the code of the driver the time out appears to be related to HW that might be busy. I will be not be able to debug your test script. But based on the timeout my initial thinking here is that there needs to be some flow control added to the test script. It looks like the script is sending messages faster than the network stack and the HW can handle them. My recommendation would be to modify your test to start slow such compare on a 1 to 1 basis an incrementing message stream in the 8 bytes from CAN0 to CAN1 and then perhaps build from there.

    Best Regards,

    Schuyler

  • Hi Schuyler,

    Thanks for the response. 

    While transmitting a message in the loop we are having a 0.5s delay between the messages. Whether the delay between messages need to be increased to 1 sec? 

    Also, find the below CAN test script.

    echo " "
    echo "CAN0 Transmit.....CAN1 Receive"
    echo " "
    
    ip link set can0 type can bitrate 1000000	# set bit rate
    ip link set can0 up				# set can0 up
    
    ip link set can1 type can bitrate 1000000	# set bit rate
    ip link set can1 up				# set can1 up
    
    sleep 1
    a=0;                                                          
    (       
      while true;                                                 
      do                                                          
        msg_candump="$( candump can1 )" 		# read CAN    
        if [[ ${#msg_candump} > 1 ]]; then 		#received msg          
          echo "$msg_candump"                                     
        fi                                                       
        sleep 0.025       
      done                                       
    ) &                                                     
    PID=$! 
    
    echo 0 > /sys/class/gpio/gpio160/value		# Enable CAN0 Transeiver sleep pin
    echo 0 > /sys/class/gpio/gpio161/value		# Enable CAN1 Transeiver sleep pin
    
    sleep 0.5
    
    while true;
    do
        cansend can0 --identifier=10 8 7 6 5 4 3 2 1 	#Transmit CAN message
        sleep 0.5
        if [ $a -eq 10 ]
        then
          break
        fi
        ((a=a+1))
    done
    echo " "
    echo "Received Messages from CAN0 "
    echo " "
    ip link set can0 down                                         
    ip link set can1 down
    
    kill -9 $PID
    
    echo " "
    echo " "
    sleep 2
    echo "CAN1 Transmit.....CAN0 Receive"
    
    echo " "
    
    ip link set can0 type can bitrate 1000000		# set bit rate
    ip link set can0 up					# set can0 up
    
    ip link set can1 type can bitrate 1000000		# set bit rate
    ip link set can1 up					# set can1 up
    
    sleep 1 
    
    a=0;                                                          
    (     
      while true;                                                 
      do                                                          
        msg_candump="$( candump can0 )" 			# read CAN   
        if [[ ${#msg_candump} > 1 ]]; then 			#received msg          
          echo "$msg_candump"                                     
        fi                                                       
        sleep 0.025       
      done                                       
    ) &                                                     
    PID=$!
    
    sleep 0.5
    
    while true;
    do
        cansend can1 --identifier=5 1 2 3 4 5 6 7 8		#Transmit CAN message
        sleep 0.5
        if [ $a -eq 10 ]
        then
          break
        fi
        ((a=a+1))
    done
    
    echo " "                              
    echo "Received Messages from CAN1 "
    echo " " 
    
    ip link set can0 down				#Down can 0
    ip link set can1 down				#Down can 1
    
    echo 1 > /sys/class/gpio/gpio160/value		# Disable CAN0 Transeiver sleep pin
    echo 1 > /sys/class/gpio/gpio161/value		# Disable CAN1 Transeiver sleep pin
    kill -9 $PID
    

  • Hi,

    Could you try reducing the message count in the loops to 1 as a means of identifying if the issue is an overrun condition. How long has the test been running before seeing the first timeout? Could describe how you are bringing up the interfaces and when the test starts? Is the test continuously running or is being stopped and restarted?

    Best Regards,

    Schuyler

  • Hi Schuyler,

    Could you try reducing the message count in the loops to 1 as a means of identifying if the issue is an overrun condition.

    • We can perform this debug action and update you by tomorrow

    How long has the test been running before seeing the first timeout?

    • Timeout is observed in all the test cases of CAN i.e., in the first testing case itself it is observed.
    • Also, this timeout is observed when we normally do disable and enable the CAN interface. Please find the below image.

    Could describe how you are bringing up the interfaces and when the test starts?

    #ip link set can0 type can bitrate 1000000          

    #ip link set can0 up           

    After transmitting all the messages, we disable the can interface.                   

    #ip link set can0 down                     

    Is the test continuously running or is being stopped and restarted?

    • Overall testing is done continuously with various interfaces in which CAN is one of the interfaces which will be disabled and restarted in each sequence.

    Regards,

    Monish P

  • Hi,

    Thank you for the update. The error occurring on disabling and enabling the interface I will look into with the development team. Also are you seeing any message loss?

    Best Regards,

    Schuyler

  • Hi,

    I need the results of uname -a too please. 

    Thanks and Best Regards,

    Schuyler

  • Hi Schuyler,

    We are not observing any message loss. Actually, we are transmitting 11 data from CAN0 to CAN1, and vice-versa, we observe the expected output on both cases.

    Regards,

    Monish P

  • Hi, 

    Please find the result of uname -a.

  • Hi,

    I apologize, I should have asked for the results of uname -a when I asked for kernel version. That kernel you are running is not a TI released kernel. Is there a reason this kernel version is being used? Where is the kernel source being cloned from?

    I would recommend switcing back to using the kernel that is in the TI SDK v6.03 which is kernel Version: 4.19.94-rt39. The reason I recommend this is because pulling from mainline and compiling for RT is likely to miss patches that TI applies and tests. After the SDK is released testing and auto-merge stop on RT kernels.

    Here is the link to ti kernel tree for the 6.03 RT Kernel. 

    Best Regards,

    Schuyler

  • Hi Schuyler,

    Our image is based on ti 4.19.94-rt39

    On this version, we have also kept ourselves updated on a few of the patches that have been applied on upstream 4.19-rt-kernel and also few patches for CVE vulnerabilities.

    Regards,

    Monish P

  • Hi Monish,

    If possible I would like to recommend as a means of problem isolation trying the pre-built TI kernel & drivers and see if you see the same issue.  

    Best Regards,

    Schuyler