• State Resolved
  • Locked Locked
  • Replies 8 replies
  • Subscribers 99 subscribers
  • Views 2075 views
  • Users 0 members are here
Support feedback
Options
Options
Related

This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

Arago | Bitbake | ftp problem

Alyona
Expert 1295 points

Hello everybody,

I'm launching bitbake arago-base-image (for example, but it could be another configuration) to build a filesystem for Linux Embedded. As Arago is an overlay for OpenEmbedded, I read on the OpenEmbedded page (http://www.openembedded.org/wiki/Required_software) the list of necessary utilites and have installed all of them (in different directories BTW just because not exactly me was who did it, but our system administrator ).

The problem is that on 36, 42 and some other tasks (steps of bitbake operation) it persistently tries to download the same packages from ftp that I've already installed. And my host machine has not got the access to ftp. Moreover I don't see the sense of downloading and installing it again.

Please, help me somehow who know what to do! 

  • 0
    TI__Expert 5450 points

    Alyona,

    A log is more helpful than steps because depending on what machine you are building for, etc the task numbers can change.

    That being said, you should realize that while you install some helper programs on the host, the bitbake operation may also want to build some of those programs for the target.  This means that you have to download and fetch the sources from cross-compile.

    Without a log I can't tell if bitbake is trying to build the utilities for the host or the target (hopefully the log will make that clear) but not having ftp access is going to be an issue when using bitbake.  There are going to be a lot of sources that need to be downloaded either from the web with http or ftp as well as from source repositories with svn and git to name a few.

    Chase 

  • 0 in reply to Chase Maupin
    Expert 1295 points

    Hello, Chase,

    I chose a part of log concerning one download of autoconf: /unfortunately it is in Russian so I translated important parts/

    --------------------------------------------------------- 

    NOTE: Running task 42 of 3832 (ID: 312, virtual:native:/home/alyona.nikiforova/oe/arago-oe-dev/recipes/autoconf/autoconf_2.65.bb, do_fetch)
    --2012-02-16 17:38:01-- ftp://ftp.gnu.org/gnu/autoconf/autoconf-2.65.tar.bz2
    => «/home/alyona.nikiforova/oe/downloads/autoconf-2.65.tar.bz2»
    Address transforming ftp.gnu.org... NOTE: package autoconf-native-2.65-r14.1: task do_fetch: Started
    NOTE: fetch ftp://ftp.gnu.org/gnu/autoconf/autoconf-2.65.tar.bz2
    140.186.70.20
    Connecting to ftp.gnu.org|140.186.70.20|:21... error: connection timed out.
    Continuing with attempts.

    <4 more same attempts>

    --2012-02-16 17:39:57-- ftp://mirrors.kernel.org/gnu/autoconf/autoconf-2.65.tar.bz2
    => «/home/alyona.nikiforova/oe/downloads/autoconf-2.65.tar.bz2»
    Address transforming mirrors.kernel.org... 149.20.4.71
    Connecting to mirrors.kernel.org|149.20.4.71|:21... NOTE: fetch ftp://mirrors.kernel.org/gnu/autoconf/autoconf-2.65.tar.bz2
    error: connection timed out.
    Continuing with attempts.

    <4 more same attempts>

    --2012-02-16 17:41:52-- ftp://ftp.cs.ubc.ca/mirror2/gnu/autoconf/autoconf-2.65.tar.bz2
    => «/home/alyona.nikiforova/oe/downloads/autoconf-2.65.tar.bz2»
    Address transforming ftp.cs.ubc.ca... 142.103.6.49
    Connecting to ftp.cs.ubc.ca|142.103.6.49|:21... NOTE: fetch ftp://ftp.cs.ubc.ca/mirror2/gnu/autoconf/autoconf-2.65.tar.bz2
    error: connection timed out.
    Continuing with attempts.

    <4 more same attempts>

    --2012-02-16 17:43:47-- ftp://sunsite.ust.hk/pub/gnu/autoconf/autoconf-2.65.tar.bz2
    => «/home/alyona.nikiforova/oe/downloads/autoconf-2.65.tar.bz2»
    Address transforming sunsite.ust.hk... error: name or service is unknown.
    wget: can't get the host address «sunsite.ust.hk»
    --2012-02-16 17:43:47-- ftp://ftp.ayamura.org/pub/gnu/autoconf/autoconf-2.65.tar.bz2
    => «/home/alyona.nikiforova/oe/downloads/autoconf-2.65.tar.bz2»
    Address transforming ftp.ayamura.org... 205.178.189.131
    Connecting to ftp.ayamura.org|205.178.189.131|:21... error: connection timed out.
    Continuing with attempts.

    <4 more same attempts>

    --2012-02-16 17:45:42-- http://mirrors.openembedded.org//autoconf-2.65.tar.bz2
    Address transforming proxy.planar.chel.ru... 192.168.1.249
    Connecting to proxy.planar.chel.ru|192.168.1.249|:8080... connected.
    Request to Proxy was sent, waiting for answer... 503 Service Unavailable
    2012-02-16 17:45:52 ERROR 503: Service Unavailable.

    --2012-02-16 17:45:52-- http://sources.openembedded.org//autoconf-2.65.tar.bz2
    Address transforming proxy.planar.chel.ru... 192.168.1.249
    Connecting to proxy.planar.chel.ru|192.168.1.249|:8080... connected.
    Request to Proxy was sent, waiting for answer... 404 Not Found
    2012-02-16 17:45:52 ERROR 404: Not Found.

    --2012-02-16 17:45:52-- http://www.angstrom-distribution.org/unstable/sources/autoconf-2.65.tar.bz2
    Address transforming proxy.planar.chel.ru... 192.168.1.249
    Connecting to proxy.planar.chel.ru|192.168.1.249|:8080... connected.
    Request to Proxy was sent, waiting for answer... 200 OK
    Length: 1332522 (1,3M) [application/x-bzip2]
    Saving to: «/home/alyona.nikiforova/oe/downloads/autoconf-2.65.tar.bz2» 

    ------------------------------------------------

    I noticed that it was downloaded in the end via http, but all previous attempts (5 for every mirror)  consumes so much time. If you could just tell me whether it is possible to build bitbake settings somehow to start off with http I would be almost happy.

    And also another question arises, I achieved just only 50 task but there are 3900 of them and I already pinched with free space on hard disc. Do you have at least an idea how much space it could take?

  • 0 in reply to Alyona
    TI__Expert 5450 points

    Alyona,

    I believe you can unset the MIRROR values to disable the source mirrors.

    As for the amount of space this can be 30GB or more.  One way to save some space is to uncomment the INHERIT += rm_work line in your local.conf to allow you to cleanup the work directory for each package after it is built.

    Chase

  • 0 in reply to Alyona
    TI__Prodigy 355 points

    Well, Russian shouldn't be a problem for me... :)

    Anyway, as you can see from the logs, it first tries to download autoconf from it's original location, which is ftp://ftp.gnu.org, and when failed, it starts going through the list of mirrors, first few of them being ftp, and then some http, until it gets to a mirror that has the correct file.

    So, it appears you cannot disable MIRRORS, like Chase was suggesting...

    On the other hand, I see that you use a proxy for HTTP connections, I wonder if you can simply use it for FTP as well. You probably already have http_proxy variable defined and set to point to proxy.planar.chel.ru:8080 - try the same with ftp_proxy:

    $ export ftp_proxy=http://proxy.planar.chel.ru:8080

    Many Web proxies are capable of accessing FTP servers, while serving clients over HTTP.

    Another option would be to play with MIRRORS and PREMIRRORS settings, but that may not be trivial, as there's no complete mirror out there with all the sources, where you can point to and forget all the troubles...

    Denys

  • 0 in reply to Denys Dmytriyenko
    Expert 1295 points

    Hello, Chase and Denys!

    Chase,

    is this space 30 GB necessary only once or for every compilation? I mean will several similar compilations (for example base-images) and other compilations (for example for special board) use things that was downloaded before, won't it?

    Denys,

    Thanks for the idea about ftp_proxy, I will try it when have more space on Virtual Box. As for MIRRORS I haven't made a conclusive decision but if these downloads will appear only once (for the first time) maybe it is reasonable just to bear rather than killing time on digging in complicated settings.

  • 0 in reply to Alyona
    TI__Expert 5450 points

    Alyona,

    Much of the space will be shared.  i.e. building busybox for two devices with the same architecture will only be built once and shared for both.

    Chase

  • 0 in reply to Alyona
    TI__Prodigy 355 points

    Alyona,

    Majority of that space is being utilized by build directories, not by downloads.

    Downloads (i.e. source tarballs, or SCM checkouts) are shared between builds all the time and are downloaded once, they won't be re-downloaded, until you need a different version.

    Build directories is where all the sources are being built - compiled and linked. Most of them are shared, when building for the same target architecture (i.e. apps for ARMv7a are usually shared), but some are machine-specific, such as kernel, bootloader etc.

    BTW, my fresh clean build of current AM-SDK images preserving all the sources is around 60-70GB...

    Denys

  • 0 in reply to Denys Dmytriyenko
    Expert 1295 points

    Thank you very much.

    Yeah, 60-70 GB is quite conspicuous...