Horrible download performance for TI Yocto components

Janne Paalijarvi

Hello,

I have been researching new TI Kirkland Yocto and I seem to be experiencing horrible download performance.

I had the build running overnight and the waiting time had climbed to 8 hours. I restarted the build. In the screenshot it shows about 26 minutes waiting time. But now that I have been writing this message, the time has gone up to 41 minutes. I have a feeling it will go up more and make this build very cumbersome, especially in this situation where I am integrating the new build.

There is nothing wrong with my internet connection I'd say:

I see that the downloads go up to 25% and then some kind of aggressive throttling kicks in and everything grinds to almost halt.

What might be going on?

over 2 years ago

0 Andreas Dannenberg over 2 years ago

TI__Guru 70392 points

Hi Janne,

download performance could be dependent on a number of factors including time of day and other items. Since ~95% of the components downloaded get pulled from open-source project repositories (rather than "TI repositories") there's not much we can do to directly improve those in a generic way, if what you see is really because of download performance.

Generally what helps is that the download only happens once per SW component+version, and download files get cached in a "downloads" folder. So successive builds, even full builds, should not really be impacted by this. Please do make sure you preserve the "downloads" folder across builds/platforms, and note that's also something you can setup/share between projects and even across developers/build machines internally. Please also see here https://docs.yoctoproject.org/ref-manual/variables.html?highlight=dl_dir#term-DL_DIR

Regards, Andreas

0 Janne Paalijarvi over 2 years ago in reply to Andreas Dannenberg

Prodigy 105 points

Hello and thank you for answering. Everything else comes via pipe blazingly fast. But there seems to be something problematic with TI servers.

See:

Also, trying to access their Web Git via Google top results yields problems:

They seem to be down completely now.

0 Olav RS over 2 years ago in reply to Janne Paalijarvi

Prodigy 41 points

Hi, I have the same issue. Related: e2e.ti.com/.../cloning-from-ti-linux-kernel-git-is-extraordinarily-slow---150k-s

0 Andreas Dannenberg over 2 years ago in reply to Olav RS

TI__Guru 70392 points

Janne, All,

after discussing some with the team it turns out there indeed have been performance degradations involving the TI servers due to much increased traffic which I wasn't aware of when I made my original reply. Apparently capacity was added but the steps taken don't seem to be enough and the team realizes that. Currently there are discussions ongoing on and there's definitely awareness that this needs further improvement so this is being tackled, and I also provided them with a reference to this support thread here to underscore the importance of this. I do not have an ETA for further changes/improvements at this moment but I wanted to let you know.

If you can't download a specific file/repository at all (even repeatedly), please do let me know, as this will have an even higher urgency than general performance degradation.

Regards, Andreas

0 Daniel Ammann over 2 years ago in reply to Andreas Dannenberg

Prodigy 190 points

Hi, I have the same issue. Specifically, a clone of git.ti.com/.../ti-linux-kernel.git fails repeatedly.

0 Olav RS over 2 years ago in reply to Andreas Dannenberg

Prodigy 41 points

For my part, I'm having issues with cloning the following repos:
https://git.ti.com/cgit/optee/ti-optee-os
https://git.ti.com/cgit/atf/arm-trusted-firmware

They are cloned as part of a setup script in the J721E SDK.

0 Andreas Dannenberg over 2 years ago in reply to Olav RS

TI__Guru 70392 points

Some additional improvements are being made right now but sounds like this is pretty severe. Some increase in bot activity was being reported, that's probably at least part of the issue. We'll see how we can make this more robust.

Regards, Andreas

0 Andreas Dannenberg over 2 years ago in reply to Andreas Dannenberg

TI__Guru 70392 points

I was just informed that we blocked a bot agent that was stressing the system around 10:45am CST today 30-Jan-24 and the load has since returned to normal. Please everybody let me know if there are still issues.

Regards, Andreas

0 Randy Kwong over 2 years ago

Prodigy 110 points

Thanks for the update, Andreas.

I was piggybacking off of another thread, and you had led me here. https://e2e.ti.com/support/site-support-group/site-support/f/site-support-forum/1318449/cloning-from-ti-linux-kernel-git-is-extraordinarily-slow---150k-s

We are also trying to get Linux files (seems like everybody is trying to get similar sets of files) -

yocto/meta-ti/recipes-kernel/linux/linux-ti-staging_5.10.bb:do_fetch

Unfortunately, we are not able to cache our downloaded libraries and files as we have to use a clean VM for every build run. I'll try the build again later today to see if things improve.

0 Daniel Ammann over 2 years ago in reply to Andreas Dannenberg

Prodigy 190 points

Unfortunately, there still are intermittent issues. Affected repositories include:
git.ti.com/.../ti-linux-kernel.git
git.ti.com/.../ti-linux-firmware.git

0 Olav RS over 2 years ago in reply to Andreas Dannenberg

Prodigy 41 points

I'm still having issues as well.

0 Janne Paalijarvi over 2 years ago in reply to Andreas Dannenberg

Prodigy 105 points

Hello,

Unfortunately the issue seems to be there again for me also.

0 Andreas Dannenberg over 2 years ago in reply to Janne Paalijarvi

TI__Guru 70392 points

The web team needs more specifics in order to be able to debug this further. What is everybody seeing (time) for cloning the full TI kernel repo directly from the command line? Below is what I got when I just tried it. I did it from within the TI firewall / being on the TI network, so this will not be representative of what you all will see outside "in the real world", but I wanted to post this as a working example/reference point anyways. Please post your complete log if you do run the experiment (don't just give the time).

a0797059@dasso:~/tmp
$ time git clone https://git.ti.com/git/ti-linux-kernel/ti-linux-kernel.git
Cloning into 'ti-linux-kernel'...
remote: Enumerating objects: 13310168, done.
remote: Counting objects: 100% (13310168/13310168), done.
remote: Compressing objects: 100% (1778085/1778085), done.
Receiving objects: 100% (13310168/13310168), 2.94 GiB | 66.09 MiB/s, done.
remote: Total 13310168 (delta 11477057), reused 13301513 (delta 11468668), pack-reused 0
Resolving deltas: 100% (11477057/11477057), done.
Updating files: 100% (52224/52224), done.

real    5m39.378s
user    8m45.337s
sys     0m30.461s

Thanks, Andreas

0 Michael Lynch over 2 years ago in reply to Andreas Dannenberg

Prodigy 130 points

This is what I see so far (at this rate it will take > 5 hours. I'll paste the actual results if/when they are available:

$ time git clone git.ti.com/.../ti-linux-kernel.git
Cloning into 'ti-linux-kernel'...
remote: Enumerating objects: 13310168, done.
remote: Counting objects: 100% (13310168/13310168), done.
remote: Compressing objects: 100% (1778085/1778085), done.
Receiving objects: 2% (387121/13310168), 223.15 MiB | 166.00 KiB/s

Mike

0 Randy Kwong over 2 years ago in reply to Andreas Dannenberg

Prodigy 110 points

Running the same command as you are, I'm getting around the same time as you are

sw-build@swbuild-pc:~/ti_test$ time git clone git.ti.com/.../ti-linux-ke
rnel/ti-linux-kernel.git -b linux-5.4.y
Cloning into 'ti-linux-kernel'...
remote: Enumerating objects: 13310168, done.
remote: Counting objects: 100% (13310168/13310168), done.
remote: Compressing objects: 100% (1778085/1778085), done.
Receiving objects: 100% (13310168/13310168), 2.94 GiB | 40.48 MiB/s, done.
remote: Total 13310168 (delta 11477056), reused 13301514 (delta 11468668), pack- reused 0
Resolving deltas: 100% (11477056/11477056), done.
Checking out files: 100% (65727/65727), done.

real 8m4.655s
user 9m52.502s
sys 2m28.661s

Our yocto recipe grabs the same URI, but at the 5.10 branch.

0 Randy Kwong over 2 years ago in reply to Randy Kwong

Prodigy 110 points

Update: the git clone from our build system is still timing out for only that particular package.

There were two other packages (prueth-fw-am65x-sr2_git, ti-sci-fw/ti-sci-fw_git) taking an exceptionally longer time to fetch, but they did fetch eventually within the timeout period

0 Andreas Dannenberg over 2 years ago in reply to Randy Kwong

TI__Guru 70392 points

Randy Kwong said:
real 8m4.655s

What you saw on the command line here for the Linux kernel looks completely normal to me. Are you saying only under Yocto it is much slower? If that's the case, one thing I'd like you to double-check. I've seen doing a Git clone using the 'git' protocol can be signifcantly slower than the 'https' protocol/transport in some cases. As far as I know our SDK v8.6-based Kernel 5.10 recipes (assuming this is what you are using?) already uses 'https' protocol. Here it is for reference, please see line 36 where it says KERNEL_GIT_PROTOCOL = "https". Please double check that this is what you also have.

https://git.ti.com/cgit/arago-project/meta-ti/tree/recipes-kernel/linux/linux-ti-staging_5.10.bb?h=dunfell#n36

(also see here, this came in with SDK v8.4 (meta-ti tag 08.04.01.003) it looks like https://git.ti.com/cgit/arago-project/meta-ti/commit/recipes-kernel/linux/linux-ti-staging_5.10.bb?h=dunfell&id=daccda80cd637e7afc61481e5d601dc66dbe8edd).

Randy Kwong said:
There were two other packages (prueth-fw-am65x-sr2_git, ti-sci-fw/ti-sci-fw_git) taking an exceptionally longer time to fetch, but they did fetch eventually within the timeout period

So ti-linux-kernel is still the main concern here, correct?

Regards, Andreas

0 Manuel Rota over 2 years ago

Prodigy 20 points

Same issue here with git.ti.com/.../ti-linux-firmware.git since yesterday, download speeds capped at 50kb/s

0 Michael Lynch over 2 years ago in reply to Andreas Dannenberg

Prodigy 130 points

Cloning is being done via use of 'git clone' on the command line. Not via any SDK or Yocto/Bitbake as can be seen in previous replies in this thread. Problem is still happening for me on different machines on different networks. Cloning from other git repository sources (git.kernel.org) works as expected.

0 Michael Lynch over 2 years ago in reply to Michael Lynch

Prodigy 130 points

Final results of clone attempt started yesterday late morning:

$ time git clone git.ti.com/.../ti-linux-kernel.git
Cloning into 'ti-linux-kernel'...
remote: Enumerating objects: 13310168, done.
remote: Counting objects: 100% (13310168/13310168), done.
remote: Compressing objects: 100% (1778085/1778085), done.
remote: Total 13310168 (delta 11477056), reused 13301514 (delta 11468668), pack-reused 0
Receiving objects: 100% (13310168/13310168), 2.94 GiB | 136.00 KiB/s, done.
Resolving deltas: 100% (11477056/11477056), done.
Updating files: 100% (52224/52224), done.

real 381m48.007s
user 10m14.564s
sys 1m27.803s

0 Andreas Dannenberg over 2 years ago in reply to Michael Lynch

TI__Guru 70392 points

Michael Lynch said:
real 381m48.007s

That's pretty awful.

I just tried it again on my systems.

Company-provided PC on the corporate network: About 5min30s, almost exactly the same as yesterday
My home PC, on the public internet (w/ 300Mbit/s service): I started it a while ago but it looks like it will take a REALLY long time. Coming in at about 500KiB/s.

I'm sure the IT folks monitoring things but let me touch base again internally.

Thanks everybody for the reports, and your patience as the team is trying to get this resolved.

Regards, Andreas

0 Randy Kwong over 2 years ago in reply to Andreas Dannenberg

Prodigy 110 points

Yeah the linux-kernel is the primary concern here.

I should mention that our script actually changes the protocol to https so it's not actually running on git:// but I can double check on that

0 Mukul Bhatnagar over 2 years ago in reply to Randy Kwong

TI__Guru* 85235 points

Randy

Sharing this link from Ryan , per our discussion if this helps with any stop gap measures while we work this out

https://www.thegoodpenguin.co.uk/blog/improving-yocto-build-time/

Regards

Mukul

0 Andreas Dannenberg over 2 years ago in reply to Mukul Bhatnagar

TI__Guru 70392 points

Everybody,

quick update here from a TI POV. It sounds like our internal IT team, after extensive troubleshooting, has identified an issue at one of our internet service providers that looks like could well be the cause of the recent slowdowns seen by some of you. Please do know this is being worked on with very high priority. We'll have another update probably tomorrow.

Regards, Andreas

0 Jonathan Cormier over 2 years ago in reply to Andreas Dannenberg

Genius 4041 points

Thanks Andreas for the updates

+1 Andreas Dannenberg over 2 years ago in reply to Jonathan Cormier

TI__Guru 70392 points

Everybody,

as it turns out our ISP for the company had a faulty circuit, and they fixed that last night. Speeds should be back up to where they were before this incident. Please let us know what you see on your end.

Thanks, Andreas

0 Jonathan Cormier over 2 years ago in reply to Andreas Dannenberg

Genius 4041 points

Downloading well right now

0 Michael Lynch over 2 years ago in reply to Andreas Dannenberg

Prodigy 130 points

Lastest attempt here. Looks like issue is resolved.

$ time git clone git.ti.com/.../ti-linux-kernel.git
Cloning into 'ti-linux-kernel'...
remote: Enumerating objects: 13312376, done.
remote: Counting objects: 100% (13312376/13312376), done.
remote: Compressing objects: 100% (1779232/1779232), done.
Receiving objects: 100% (13312376/13312376), 2.94 GiB | 41.22 MiB/s, done.
remote: Total 13312376 (delta 11478880), reused 13302932 (delta 11469721), pack-reused 0
Resolving deltas: 100% (11478880/11478880), done.
Updating files: 100% (52224/52224), done.

real 6m11.892s
user 6m45.379s
sys 0m12.807s

0 Janne Paalijarvi over 2 years ago in reply to Andreas Dannenberg

Prodigy 105 points

I get this good now via Wifi:

Receiving objects: 4% (607475/13313697), 339.86 MiB | 10.47 MiB/s

I consider this resolved. Thanks.

Processors

Processors forum

Horrible download performance for TI Yocto components