AM625: tispl download u-boot

Hi,

Could you please attach the full console sequence? 

Is this a custom board or a TI EVM?

What was the default block size that tftp was using?

Best Regards,

Schuyler

Hello Schuyler,

Thanks for your response,

This is a custom board.

Below is our config related to Block size

CONFIG_IP_DEFRAG=y
CONFIG_NET_MAXDEFRAG=65535
CONFIG_TFTP_BLOCKSIZE=65535

and timeout is down from 10s to 1s

#define TIMEOUT 1000UL

We attach the full console: 

02000000011a0000616d3632780000000000000048535345010001000100010002a6000002000200d68ecb2c055dff11ade95bd927e837d2a53bc23b0a2800cebce4f106bcf309df2213912d77a157a8b7c2df40672a06a918034aa4c7d603e462481475225d49b839109fc8da497caaca402cdb8566ccee2e94a6514077f4253513d2c9eabf277294ba87b257ba58d8bff4fb1797d5e6b6d2c6df8c07a210045e4ad37505f5be34e7f9a0180a54b90a2841c73f268690ad53a4a3a5be07c9e2dcb2e2cb90896a42C
U-Boot R5 SPL 2021.01-00015-gaa48999c6a (Oct 14 2024 - 10:02:03 +0700)
SYSFW ABI: 3.1 (firmware rev 0x0009 '9.0.5--v09.00.05 (Kool Koala)')
Switching: UART -> ETH RMII boot...
SPL initial stack usage: 13424 bytes
Trying to boot from eth device
eth0: ethernet@8000000port@1
ethernet@8000000port@1 PHY link status . is up(done)
ethernet@8000000port@1:100BASE-T1 100 Mbit/s
link up on port 1, speed 100, full duplex
Using ethernet@8000000port@1 device
TFTP from server 192.168.1.20; our IP address is 192.168.1.2
Filename 'tispl.bin'.
Load address: 0x82000000
Loading: #################
         1.6 MiB/s
done
Bytes transferred = 869192 (d4348 hex)
Authenticating image at address 0x9e780000
Authenticating image of size 49897 bytes
Authentication passed
Authenticating image at address 0x9e800000
Authenticating image of size 442065 bytes
Authentication passed
Authenticating image at address 0x89000000
Authenticating image of size 151589 bytes
Authentication passed
Authenticating image at address 0x80080000
Authenticating image of size 203721 bytes
Authentication passed
Authenticating image at address 0x800b1580
Authenticating image of size 20555 bytes
Authentication passed
init_env from device 4 not supported!
Starting ATF on ARM64 core...

NOTICE:  BL31: v2.8(release):v2.8-226-g2fcd408bb3-dirty
NOTICE:  BL31: Built : 00:42:57, Jan 13 2023

U-Boot A53 SPL 2021.01-00015-gaa48999c6a (Oct 14 2024 - 10:01:54 +0700)
SYSFW ABI: 3.1 (firmware rev 0x0009 '9.0.5--v09.00.05 (Kool Koala)')
Trying to boot from eth device
eth0: ethernet@8000000port@1
ethernet@8000000port@1 PHY link status  is up(done)
ethernet@8000000port@1:100BASE-T1 100 Mbit/s
link up on port 1, speed 100, full duplex
Using ethernet@8000000port@1 device
TFTP from server 192.168.1.20; our IP address is 192.168.1.2
Filename 'u-boot.img'.
Load address: 0x82000000
Loading: T ##############
         93.8 KiB/s
done
Bytes transferred = 668712 (a3428 hex)
Authenticating image at address 0x80800000
Authenticating image of size 634353 bytes
Authentication passed
Authenticating image at address 0x8089a780
Authenticating image of size 33358 bytes
Authentication passed


U-Boot 2021.01-00015-gaa48999c6a (Oct 14 2024 - 10:01:54 +0700)

SoC:   AM62X SR1.0 HS-SE
Model: Texas Instruments AM625 SK
DRAM:  512 MiB
MMC:
Error, wrong i2c adapter 0 max 0 possible
In:    serial@2800000
Out:   serial@2800000
Err:   serial@2800000
Net:   eth0: ethernet@8000000port@1
Hit any key to stop autoboot:  0
jedec_spi_nor flash@0: Issue 0xB7, This changes CFR2V[7] to 1 (4-byte addr mode)
CFR5N: 40 40
CFR4N: a8 a8
CFR3N: 08 00
CFR2N: 88 88
CFR1N: 00 04
STR1N: 00 00
The hybrid sector is already enabled (CFR3N[3]=0) of die-2
Already setting top 4KB (CFR1N[2]=1) of die-2
params->page_size:512
CFR3V: 18 10
non_uniform---> done
CFR3V: 38 30---> done
The 1rst die is already in DDR mode!!!
The 2nd die is already in DDR mode!!!
SF: Detected s28hs02gt with page size 512 Bytes, erase size 256 KiB, total 256 MiB
ethernet@8000000port@1 PHY link status  is up(done)
ethernet@8000000port@1:100BASE-T1 100 Mbit/s
link up on port 1, speed 100, full duplex
Using ethernet@8000000port@1 device
TFTP from server 192.168.1.20; our IP address is 192.168.1.2
Filename 'debrick.scr'.
Load address: 0x80000000
Loading: #
         834 KiB/s
done
Bytes transferred = 3419 (d5b hex)
## Executing script at 80000000

Thanks.

Cong.

I'd like to add more info:

Package capture from WireShark show that there are 2 timeout with Read Request package.

the time between 2 data packet is not greater than downloading of tispl.

Why the average speed is much smaller than? Does it calculate all the time that include ARP packet?

Thanks,

Cong.

Hi,

Could you please attach individual wireshark captures of the downloads for tiboot3.bin, tispi.bin and u-boot.img. We do not see the timeouts that you are experiencing. This possibly could be a DHCP server issue.

Best Regards,

Schuyler

Hello Schuyler,

Thanks for your response,

I cannot upload file here, so please see the attached file on the email.

Thanks.

Cong.

Could you please provide your email or guide me how to upload file here. It is *.pcapng file.

Hello Cong,

Could you please zip/compress the *.pcapng file and try to re-upload it here?

Regards,
Krithika

Download_Uboot.zip

Hello Schuyler, Krithika,

Zip file is put here.

Please help us check.

Thanks,

Cong.

What is your network architecture or topology. Is there a DHCP server? 192.168.1.20 is the TFTP server, 192.168.1.2 is the AM625? A Linux server running TFTP server connected in a dedicated LAN to the AM625 device? Or is it connected using some enterprise network?

Looking at the packet capture in wireshark couple things jump out.

1. TFTP looks to be using 512 byte frames, could there be some setting in the TFTP server limiting to such small MTU? I would have expected ~1500 bytes. This is probably minor.

2. At the very beginning it looks like it takes 30s or so to get a ARP response. 192.168.1.20 is looking for 192.168.1.0 (default gateway) but does not get a response for a long time. Then loading tispl to 192.168.1.2 begins. This points to some issue with default gateway settings?

3. Just from the wireshark the bandwidth of loading tispl.bin looks to be (start at 66.872 seconds into capture, end at 67.403) 0.53 seconds. While the u-boot.img is (start 73.334, end at 76.231) is 2.9 seconds. File sizes are 869kB and 666kB so bandwidth should be around 1.6MB/s and 0.23MB/s. This does not match your screenshot printout of 2.2MB/s and 44kB/s? Are those bandwidth calculated by your SW? Wireshark shows bandwidth delta of 6-7x not 50x.

4. Based on the item 3 it looks like loading u-boot.img image acknowledging each TFTP frame takes 3-5x as long as each frame of tispl.bin. Is the TFTP client software, cache settings and memory map identical?

  Pekka

Pekka Varis said:

1. TFTP looks to be using 512 byte frames, could there be some setting in the TFTP server limiting to such small MTU? I would have expected ~1500 bytes. This is probably minor.

Ok, looks like 512 bytes is the TFTP default, so that explains it.

Hello Pekka,

Thanks for your analysis.
I am following this ticket, and I will be answer the question below.

Is there a DHCP server?
=> No
192.168.1.20 is the TFTP server, 192.168.1.2 is the AM625?
=> Yes
A Linux server running TFTP server connected in a dedicated LAN to the AM625 device? Or is it connected using some enterprise network?
=> No, Personal computer with OS Windows10 running TFTP server connected in a LAN to the AM625 device. below is my configure TFTP server.

2. I tried configure static IP without default IP, the speed at which it downloads data does not change, so i think it is not the cause of slowness.

3. Log from SW and Wireshark get at the same time.
Bandwidth is calculated in SW which is the default code in the kernel.

The way to calculate bandwidth in SW is when SW is ready to receive data, and the way to calculate in wireshark is when data between TFTP server has received a request to send data, so these two values ​​cannot be compared.

I got some new information when debugging inside the kernel and noticed that initially there was no data available.

this is where i put log in kernel

I observed the two logs simultaneously, and found that these two points were at the same time.

From here we can see that the cause lies in the initialization process and connection to the TFTP server.

I attach the log here 6428.Log_debug.zip

Regards,

Huy

huy khuat said:

From here we can see that the cause lies in the initialization process and connection to the TFTP server.

I'd tend to agree. It looks like there is 5 seconds of trying to figure out IP addresses when loading u-boot.img. Your screenshot has that 5s (starts at around 32s from the capture start), where nothing is transferred but the Windows machine and the board being loaded send ARP messages to figure out which MAC address has which IP (looks like Windows is sending at least three ARPs, messages 3408, 3409, 3410 one second apart). In the loading of tispl.bin at the very beginning there is no multiple ARP messages. So if you include this 5s of nothing going because the TFTP server and the board asking for the image have not completed ARP on in the bandwidth calculation that could explain the lower throughput.

Your debug print log seems to be in the Ethernet driver printing the Ethertype, 0x800 is IPv4, 0x806 is ARP. So it is probably message 3412, which Wireshark says is ARP from Windows to the AM62x.

What are you using for the packet capture / wireshark? Do you have a network tap like https://www.profitap.com/profishark-1g/ or are you using Wireshark in the Windows machine?

Are you able to run a Linux machine as the TFTP server, I'm not familiar with Windows networking stack and what could be causing multiple ARP messages. 

  Pekka

Hello Pekka,

We do not have a network tap, we are using Wireshark in the Windows machine

Additional information: We use this Media Converter to convert from standard ethernet to Automotive ethernet.

I attached the zip file that included log file and pcap file when we run a Linux machine as TFTP server.

Please help me check

Download_Uboot_Ubuntu_0412.zip

Thanks.

Cong.

From all three wireshark logs, one screenshot at the beginning and then the two from the zip files (one Windows, one Linux) there looks to be a repeating pattern. Loading of tispl.bin, which happens with the boot rom seems to be fine. Sequence is ARP request sent as broadcast from the TI device (beginning of the MAC address 28:B5:e8, OUI registered to TI), response from 192.168.1.20, then the download starts. tispl.bin is loaded quick. Then the second image download (u-boot.bin) using TFTP with probably SPL running has 4-10s of ARP messages before addressing is figured out and download starts. See messages arounf number 3406 in both logs. Once it starts the speed is fine (maybe 2s to download). This MAC and IP address sorting out is why the downloading of u-boot.img appears slow. The logs also have a third file being downloaded where there does not seem to be any addressing issues but the TFTP server says no such file.

So the issue is sorting out the interaction of the networking stack and ARP resolution/cache in SPL software, media converter, and the TFTP server. What could explain this is some hacks on MAC address, or some dropped Ethernet frames. In the Windows packet capture, the MAC address of the TFTP server seems to be the Gopel Electronic (OUI is 00:06:81), in the Ubuntu one it is an unregistered one (OUI is b6:c3:97). What might fit here is that the SPL misses the ARP response (no 3406 in Ubuntu log), something (the adapter? PHY configuration?) drops the frame. Note the wireshark you capture is from the TFTP server, so shows what leaves and arrives there, not what makes it to the TI device.

3405	44.230910243	28:b5:e8:cd:57:95	Broadcast	ARP	60	Who has 192.168.1.20? Tell 192.168.1.2
3406	44.230930572	b6:c3:97:fe:e3:f3	28:b5:e8:cd:57:95	ARP	42	192.168.1.20 is at b6:c3:97:fe:e3:f3
3407	48.091626906	b6:c3:97:fe:e3:f3	28:b5:e8:cd:57:95	ARP	42	Who has 192.168.1.2? Tell 192.168.1.20
3408	49.115611308	b6:c3:97:fe:e3:f3	28:b5:e8:cd:57:95	ARP	42	Who has 192.168.1.2? Tell 192.168.1.20
3409	49.231214758	28:b5:e8:cd:57:95	Broadcast	ARP	60	Who has 192.168.1.20? Tell 192.168.1.2
3410	49.231232546	b6:c3:97:fe:e3:f3	28:b5:e8:cd:57:95	ARP	42	192.168.1.20 is at b6:c3:97:fe:e3:f3
3411	50.139626816	b6:c3:97:fe:e3:f3	28:b5:e8:cd:57:95	ARP	42	Who has 192.168.1.2? Tell 192.168.1.20
3412	50.139890824	28:b5:e8:cd:57:95	b6:c3:97:fe:e3:f3	ARP	60	192.168.1.2 is at 28:b5:e8:cd:57:95
3413	54.232190870	28:b5:e8:cd:57:95	Broadcast	ARP	60	Who has 192.168.1.20? Tell 192.168.1.2
3414	54.232311369	b6:c3:97:fe:e3:f3	28:b5:e8:cd:57:95	ARP	42	192.168.1.20 is at b6:c3:97:fe:e3:f3
3415	54.232538042	192.168.1.2	192.168.1.20	TFTP	71	Read Request, File: u-boot.img, Transfer type: octet, timeout=1

So need is to sort out the ARP traffic or possible dropped frames after tispl.bin was loaded, when loading u-boot.img is requested.

  Pekka

Hello Pekka,

Many thanks for your feedback.

We keep the configuration of ethernet stack of SPL as the initial stack

You worried about the ETH converter hidding the real MAC, causing the need to do some address resolution.

what I don't get is - why does it happen only with the uboot download?

Cong.

I'm aligned with the suspicion. One possibility is on the PHY initialization, does the SPL do something? As a workaround if you can add the TFTP server MAC address as 192.168.1.20 to the SPL ARP cache, that would bypass it asking with a broadcast who has 192.168.1.20.

  Pekka

Hello Pekka,

Could you please guide me how to add the TFTO server MAC address as 192.168.1.20 to the SPL ARP cache?

Thanks.

Cong.

Hi,

The media converter and automotive ethernet raises some new questions. Does you board use a single pair ethernet PHY? If so which PHY is being used?

While the boot sequence is obviously happening the ROM is not capable of understanding link status on SPE PHYs. 

Has the u-boot code been modified to work with SPE PHYs?

We will need to discuss with the development team next week your question how to bypass the ARP cache. 

Is the ethernet on the PC connected to the media converter at 1G? And the SPE PHY is limited to 100Mbps?

Best Regards,

Schuyler

Hello Schuyler,

Thanks for your feedback. We are waiting the feedback from your development team.

Our board use TJA1101B as ethernet PHY

Has the u-boot code been modified to work with SPE PHYs? -> Yes

Is the ethernet on the PC connected to the media converter at 1G? And the SPE PHY is limited to 100Mbps?

-> We configured the media converter to 100Mbps.

Cong.

Update:

I tried bypass the ARP packets and speed is improved significant. 

What I did is simply comment this line of code:

But without ARP packets, it has side effect 

it take about 3 timeouts (1s for each timeout) in the beginning of sending packet.

This is just 1 timeout if ARP packet is not bypassed.

Below is the attach file.

.Download_Uboot0912.zip

Could you please help me analyze it.

Thanks.

Cong.

Cong Pham said:

I tried bypass the ARP packets and speed is improved significant. 

Great. This confirms your download speed degradation is caused by issues with IP network addressing, specifically ARP broadcast messages and responses.

Cong Pham said:

But without ARP packets, it has side effect 

it take about 3 timeouts (1s for each timeout) in the beginning of sending packet.

Looks like these 3 messages again stall loading of u-boot.img for 3 seconds, probably again the reason why loading is observed to be slow (~3s of nothing being loaded when u-boot.img is requested). The messages you highlight are the TFTP server asking with a broadcast message who has 192.168.1.0 IP address, which (address ending in 0) typically is the default gateway (the address you send to if the IP address is not local to the LAN/subnet) in the network. So this is some setting in your TFTP server networking stack. In your packet capture very beginning, it also looks like there is about 40 ARP requests at once per second by the TFTP server for the address 192.168.1.0 (typically default gateway) , and I don't see a response. My suspicion is if you get that default gateway ARP request answered or hard coded in the TFTP server those messages will go away.

Cong Pham said:

Our board use TJA1101B as ethernet PHY

That looks to me like 100BASE-T1 PHY. I thought our boot ROM does not support 100M operation with a "fixed link" or what SPE PHY's use. But as the tispl.bin looks to load just fine, I guess we do. I still think this PHY configuration and/or the adapter is causing some packets to be dropped causing the original issue that ARP cache hard coding bypasses. The TFTP server sending ARPs for 192.168.1.0 is a different issue, I suspect there is no one in the LAN to answer this. Usually a router as the one answering, so configuring the network stack of the TFTP not to ask is probably a good route. Or add someone to answer those queries.

  Pekka

Hello Pekka,

Even if I remove the default gateway 192.168.1.0 in the ethernet configuration of TFTP server, timeout still occur.

So I think the reason for timeout is not related to default gateway.

Below is the log and trace

Download_Uboot1112.zip

Thanks.

Cong.

Hi Pekka, 

I have a question, why this 4 package firstly with protocol = 0x0, I'm reviewing the code and see that when length is 0 (below)

that is there is data in HW received from tftp server, it jumps into the net_process_received_packet() function, but when it gets inside, the protocol is 0x0.

Thanks, 

Huy

huy khuat said:

that is there is data in HW received from tftp server, it jumps into the net_process_received_packet() function, but when it gets inside, the protocol is 0x0.

Lets make sure I understand what are you printing out here. Is it that you modified the Ethernet receive function in SPL, the hex numbes you printo out are the Ethertype of the received frame? Maybe a minor suggestion but I'd add some text to the debug printout to say what are you printing out? 0x0 in Ethertype is not good, that packet will probably get dropped. I'd focus on where these are coming from, do they match something the TFTP server was thinking they send out. 0x800 means it was in IPv4 packet. 0x806 is an ARP packet.

I'd also print out a little more of the frame, not just the Ethertype. For example which MAC address sent it.

  Pekka

yes, I modified the ethernet receive function in net.c file, what is printed on the screen is Ethertype, this is where i print it.

And my current question is why does it always lose the first 4 packets of tispl.bin image?, but tiboot3.bin and uboot.img image are not like that

Thanks. 

Huy

huy khuat said:

And my current question is why does it always lose the first 4 packets of tispl.bin image?, but tiboot3.bin and uboot.img image are not like that

Based on your log printing out the Ethertype, it looks like there is 4 frames with Ethertype as 0 received, which will be dropped. I think 0 is not allowed see https://www.iana.org/assignments/ieee-802-numbers/ieee-802-numbers.xhtml#ieee-802-numbers-1 . And based on the timestamps that consumes 5s of no other traffic.

  Pekka

Hello Pekka,

 it looks like there is 4 frames with Ethertype as 0 received, which will be dropped

-> It seems server has sent ACK successful but the board cannot receive completed data.

The length looks good but the data is not.

We do not know why those packets are dropped while tispl download u-boot but tiboot3 download tispl?

Do you have any idea or suggestion?

Thanks,

Cong.

The wireshark you have attached the screenshot is from the TFTP server capturing what it sends. It has the read request received from the spl. It is not sending a data packet until 3472.

You also had the modified spl networking stack, printing out EtherType as 0. Which is never ok. I would print out more of the frame the spl receives at the spl, are all fields 0 ? For example is the source MAC address correct.

  Pekka

Hello Pekka,

I have send you the attachment which include log uart and wireshark at one time, it was printed out EtherType, Len and Data.

with 4 EtherType as 0, the frame at the spl receives are all fields 0.

I modified spl networking stack as below.

12.17.24_2.zip

Thanks,

Huy

huy khuat said:

with 4 EtherType as 0, the frame at the spl receives are all fields 0.

Stating the obvious so there are 4 Ethernet frames with all zeroes coming to to the spl, each one second apart from the previous one. These seem to match your TFTP server sending something that left it not being all zeroes. So maybe going in circles here but pretty clear this is where 4s of zero bits per second of TFTP bandwidth happening.

What is converting the frames from the TFTP server to all zeroes for a few seconds? My guess is the media converter or maybe more likely the PHY setup in the spl. u-boot.img is the file being loaded so focus should be on the tispl.img . Sent frames seem to be fine (the TFTP request).

Seems like the boot ROM is fine with the PHY (a little unclear why) and loading tispl.bin. Is the spl modified to work with TJA1101B? I would try to remove all code related to the PHY from the spl, everything with the PHY was fine before spl started running so leave the PHy as is.

  Pekka

Hello Pekka,

First time, the boot ROM is not fine with the PHY, then we ask TI to support and TI reflash the ROM code that support the PHY and send back to us.

1. boot ROM load tiboot3 -> OK

2. tiboot3 load tispl.bin -> OK

3. tispl.bin load uboot -> NOT OK

#2 and #3 use the same source code of PHY. Why #3 is NOT OK?

the difference between #2 and #3 is the running core, tiboot3 run on R5 core and tispl run on A53 core.

Source code of TJA1101B is porting from drivers/net/phy/dp83867.c. Do you have any alternative driver for TJA1101B?

Thanks.

Cong.

Cong Pham said:

First time, the boot ROM is not fine with the PHY, then we ask TI to support and TI reflash the ROM code that support the PHY and send back to us.

The ROM is ROM code, there is no way to update it. So I'm not clear what does "TI reflash the ROM" mean?

Cong Pham said:

3. tispl.bin load uboot -> NOT OK

#2 and #3 use the same source code of PHY. Why #3 is NOT OK?

What we know based on your spl printout is 4 frames received, all fields zeroes. I guess transmit is fine(?). PHY config is one possibility, not necessarily the underlying reason.

Cong Pham said:

Source code of TJA1101B is porting from drivers/net/phy/dp83867.c. Do you have any alternative driver for TJA1101B?

TJA1101B is an NXP PHY chip, 100BASE-T1 SPE. DP83867 is TI, 1000BASE-TX, not SPE. I can see https://github.com/torvalds/linux/blob/master/drivers/net/phy/nxp-tja11xx.c ? I'm not familiar with NXP PHY's and how similar or not their drivers are to TI, but 1000BASE-TX and 100BASE-T1 are different, for example 100BASE-T1 there is no speed negoatiation, it is hard coded (fixed-link) to 100M.

100BASE-T1 is hard coded 100M mode. Could it be there is something like auto-negotiation going on? tispl.bin is it hard coded to 100M ?

  Pekka