AM5728: Unable to SSH into TI AM5728

Devashish Tiwari

Part Number: AM5728

Hi,

I have a custom board based on TI AM5728 EVM. I am using TI LINUX PSDK 6.03 on it.

I am facing the following issue on the board -

I have connected the board to ethernet (LAN) on eth0 port. Say, after the first reboot, I have provided it with the IP address, netmask, and gateway. I am able to PING the IP address of the board. I am also able to SSH (remote-login) into the board. Now suppose I rebooted the board, there is a 50-50% chance that SSH will stop working. That is, the board will have the same IP, netmask, and gateway and the PING will also work but SSH will not work.

This may happen just after the second reboot or after the 'N' number of reboots. It is not fixed and also not reproducible after a fixed interval. But after several reboots, there will be one time when SSH will stop working but PING will work until I reboot again. The only solution to this issue is to reboot again and again till SSH starts working.

So I decided to take some observations when I hit the problem (PING working but SSH not working).

1. ifconfig & route table snapshot - looks OK to me

Fullscreen

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
root@am57xx-evm:~# ifconfig 
eth0      Link encap:Ethernet  HWaddr D0:03:EB:6C:50:36  
          inet addr:192.168.8.97  Bcast:192.168.8.255  Mask:255.255.255.0
          inet6 addr: fe80::d203:ebff:fe6c:5036/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:201188 errors:0 dropped:3032 overruns:0 frame:0
          TX packets:816 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:18021841 (17.1 MiB)  TX bytes:249325 (243.4 KiB)
          Interrupt:91 
eth1      Link encap:Ethernet  HWaddr D0:03:EB:6C:50:37  
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

root@am57xx-evm:~# ifconfig 
eth0      Link encap:Ethernet  HWaddr D0:03:EB:6C:50:36  
          inet addr:192.168.8.97  Bcast:192.168.8.255  Mask:255.255.255.0
          inet6 addr: fe80::d203:ebff:fe6c:5036/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:201188 errors:0 dropped:3032 overruns:0 frame:0
          TX packets:816 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:18021841 (17.1 MiB)  TX bytes:249325 (243.4 KiB)
          Interrupt:91 

eth1      Link encap:Ethernet  HWaddr D0:03:EB:6C:50:37  
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:2 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:140 (140.0 B)  TX bytes:140 (140.0 B)

root@am57xx-evm:~# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.8.1     0.0.0.0         UG    0      0        0 eth0
192.168.8.0     0.0.0.0         255.255.255.0   U     0      0        0 eth0
root@am57xx-evm:~#

2. Dropbear service status -

Fullscreen

1
2
3
4
5
6
7
8
9
10
11
12
13
root@am57xx-evm:~# ps -ef | grep dropbear
root      1686  1088  0 01:36 ttyS2    00:00:00 grep dropbear
root@am57xx-evm:~# 
root@am57xx-evm:~# fuser 22/tcp                                                                                                                                                                      
root@am57xx-evm:~# 
root@am57xx-evm:~# systemctl is-active dropbear                                                                                                                                               
inactive
root@am57xx-evm:~# 
root@am57xx-evm:~# systemctl status dropbear.service                                                                                                                                               
�● dropbear.service - LSB: Dropbear Secure Shell server
   Loaded: loaded (/etc/init.d/dropbear; generated)
   Active: inactive (dead)
     Docs: man:systemd-sysv-generator(8)
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

root@am57xx-evm:~# ps -ef | grep dropbear
root      1686  1088  0 01:36 ttyS2    00:00:00 grep dropbear
root@am57xx-evm:~# 
root@am57xx-evm:~# fuser 22/tcp                                                                                                                                                                      
root@am57xx-evm:~# 
root@am57xx-evm:~# systemctl is-active dropbear                                                                                                                                               
inactive
root@am57xx-evm:~# 
root@am57xx-evm:~# systemctl status dropbear.service                                                                                                                                               
�● dropbear.service - LSB: Dropbear Secure Shell server
   Loaded: loaded (/etc/init.d/dropbear; generated)
   Active: inactive (dead)
     Docs: man:systemd-sysv-generator(8)

3. When I try SSH into the board from a client -

It gets stuck after "Local version string SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.4"

Fullscreen

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
root@darsh-pc:/home/darsh# ssh -vvvvvvv root@192.168.8.97
OpenSSH_8.2p1 Ubuntu-4ubuntu0.4, OpenSSL 1.1.1f  31 Mar 2020
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug2: resolve_canonicalize: hostname 192.168.8.97 is address
debug2: ssh_connect_direct
debug1: Connecting to 192.168.8.97 [192.168.8.97] port 22.
debug1: Connection established.
debug1: identity file /root/.ssh/id_rsa type -1
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: identity file /root/.ssh/id_dsa type -1
debug1: identity file /root/.ssh/id_dsa-cert type -1
debug1: identity file /root/.ssh/id_ecdsa type -1
debug1: identity file /root/.ssh/id_ecdsa-cert type -1
debug1: identity file /root/.ssh/id_ecdsa_sk type -1
debug1: identity file /root/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /root/.ssh/id_ed25519 type -1
debug1: identity file /root/.ssh/id_ed25519-cert type -1
debug1: identity file /root/.ssh/id_ed25519_sk type -1
debug1: identity file /root/.ssh/id_ed25519_sk-cert type -1
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

root@darsh-pc:/home/darsh# ssh -vvvvvvv root@192.168.8.97
OpenSSH_8.2p1 Ubuntu-4ubuntu0.4, OpenSSL 1.1.1f  31 Mar 2020
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug2: resolve_canonicalize: hostname 192.168.8.97 is address
debug2: ssh_connect_direct
debug1: Connecting to 192.168.8.97 [192.168.8.97] port 22.
debug1: Connection established.
debug1: identity file /root/.ssh/id_rsa type -1
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: identity file /root/.ssh/id_dsa type -1
debug1: identity file /root/.ssh/id_dsa-cert type -1
debug1: identity file /root/.ssh/id_ecdsa type -1
debug1: identity file /root/.ssh/id_ecdsa-cert type -1
debug1: identity file /root/.ssh/id_ecdsa_sk type -1
debug1: identity file /root/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /root/.ssh/id_ed25519 type -1
debug1: identity file /root/.ssh/id_ed25519-cert type -1
debug1: identity file /root/.ssh/id_ed25519_sk type -1
debug1: identity file /root/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /root/.ssh/id_xmss type -1
debug1: identity file /root/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.4

The client (the laptop) here is waiting for the server (the board) to send its version string to the client. The connection hangs here because the client hasn't received the version string from the server.

If at this time I check logs on the server (the board), I get the following -

Fullscreen

1
2
3
4
5
6
7
8
9
root@am57xx-evm:~# ps -ef | grep drop
root      1703     1  0 01:41 ?        00:00:00 /usr/sbin/dropbear -i -r /etc/dropbear/dropbear_rsa_host_key -B
root      1705  1088  0 01:42 ttyS2    00:00:00 grep drop
root@am57xx-evm:~# journalctl -f
Mar 17 01:42:12 am57xx-evm dropbear[1703]: Exit before auth: Exited normally
Mar 17 01:42:13 am57xx-evm systemd[1]: Started SSH Per-Connection Server (192.168.8.191:34614).
Mar 17 01:42:13 am57xx-evm dropbear[1708]: Child connection from ::ffff:192.168.8.191:34614
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

root@am57xx-evm:~# ps -ef | grep drop
root      1703     1  0 01:41 ?        00:00:00 /usr/sbin/dropbear -i -r /etc/dropbear/dropbear_rsa_host_key -B
root      1705  1088  0 01:42 ttyS2    00:00:00 grep drop

root@am57xx-evm:~# journalctl -f

Mar 17 01:42:12 am57xx-evm dropbear[1703]: Exit before auth: Exited normally
Mar 17 01:42:13 am57xx-evm systemd[1]: Started SSH Per-Connection Server (192.168.8.191:34614).
Mar 17 01:42:13 am57xx-evm dropbear[1708]: Child connection from ::ffff:192.168.8.191:34614

I don't understand what is causing this issue. Also why this issue is intermittent.

How can I resolve this?

Kindly acknowledge and revert.

Regards,

Devashish

over 3 years ago

0 Devashish Tiwari over 3 years ago

Expert 1435 points

Hi,

This is a gentle reminder that I am awaiting your reply to this post.

Regards,

Devashish

0 Devashish Tiwari over 3 years ago in reply to Devashish Tiwari

Expert 1435 points

Hello TI Experts,

The resolution of this issue is very critical for us.

This is a gentle reminder that I am awaiting your reply to this post.

Regards,

Devashish

0 jian35385 over 3 years ago in reply to Devashish Tiwari

TI__Mastermind 23125 points

Devashish,

Sorry for the delayed response. I was on travel then timeoff.

Can you confirm by "reboot" you mean by power cycle or warmreset?

If warmreset is the cause, we need to focus to see if you have anything that is reset isolated.

Jian

0 Devashish Tiwari over 3 years ago in reply to jian35385

Expert 1435 points

Hi,

Both power cycle or warmreset work.

0 jian35385 over 3 years ago in reply to Devashish Tiwari

TI__Mastermind 23125 points

Devashish,

I am not sure what you mean by "work". You mean both PORz and Warmreset will reproduce the 50/50 change issue?

Jian

0 Devashish Tiwari over 3 years ago in reply to jian35385

Expert 1435 points

That's exactly what I mean.

0 jian35385 over 3 years ago in reply to Devashish Tiwari

TI__Mastermind 23125 points

Devashish,

can you confirm if you have done any of the basic debugs, specifically:

1. when ssh fails for ping works, did you restart sshd on the board and retry?

2. have you tried to eliminate rest of network by directly connect the board with a pc via a cable, then set static IP and retry

3. can you ssh out of the board to a pc?

regards

JIan

0 Devashish Tiwari over 3 years ago in reply to jian35385

Expert 1435 points

Hi,

1. There is no sshd on the board. Remote functionality is provided using Dropbear. I have tried to restart /etc/init.d/dropbear service but no success

2. This I will have to try.

3. No. ssh from the board is also not possible. But ping works from the board.

Regards,

Devashish

0 Devashish Tiwari over 3 years ago in reply to Devashish Tiwari

Expert 1435 points

Hi JIan,

This is a reminder that I am awaiting your reply to this post.

Kindly acknowledge.

Regards,

Devashish

0 jian35385 over 3 years ago in reply to Devashish Tiwari

TI__Mastermind 23125 points

Devashish,

Please also update the outcome if you tried 2.

I am not familiar with Dropbear. But if it is possible to try sshd in your system that would give a reference against EVM.

Also a guess, as I am not sure how Dropbear is using the crypto driver, you may try to remove the crypto module driver in the filesystem, then reboot and see if the issue get eliminated. This is not likely, as otherwise, you will see 100% identical behavior. So the first two test should give us more information.

Jian

0 Devashish Tiwari over 3 years ago in reply to jian35385

Expert 1435 points

Hi Jian,

Which crypto driver are you talking about?

0 Devashish Tiwari over 3 years ago in reply to jian35385

Expert 1435 points

The latest Linux PSDK 6.03 for AM5728 also contains Dropbear. There is no sshd in the SDK.

I am using the same Linux PSDK 6.03 on my custom board.

Processors

Processors forum

AM5728: Unable to SSH into TI AM5728