This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

[FAQ] AM6x: Using multilib to allow running 32-bit binaries on Yocto Dunfell-based v8.x SDKs for Sitara 64-bit MPUs

Part Number: AM62x, AM62Ax, AM64x, AM65x

Overview

Multilib is one of the solutions allowing users to run applications built for various application binary interfaces (ABIs) of the same architecture. One use of this is to run 32-bit applications built for older generation of ARMv7A-based Sitara SoCs such as AM335x and AM437x on modern-day 64-bit ARMv8A-based AM6x SoCs. This E2E FAQ outlines a way adding multilib support to Yocto Dunfell-based v8.x SDK for AM6x SoCs, and shows an example of a 32-bit AM335x application running on an 64-bit AM62x device. The steps were tested and developed using the Yocto Dunfell-based AM62x SDK v8.6, and should similarly apply to other Sitara MPUs (AM62A, AM64, AM65) as well as other Yocto Dunfell-based SDK v8.x variants.

ExclamationDisclaimer: The steps outlined here involve switching to using Yocto-internal toolchain for image builds, which in the context of Yocto Dunfell-based v8.x SDKs is not considered a production solution. If a multilib solution is required for production purposes it is recommended to wait for the release of the upcoming Yocto Kirkstone-based SDK v9.x series arriving sometime around late 2Q2023 which by default will use a Yocto-internal toolchain, making the addition of multilib support very trivial. The solution discussed here nevertheless can be helpful to experiment with multilib support in general, and prepare for the upcoming Yocto Kirkstone-based SDK v9.x series. If a production-ready solution for Yocto Dunfell-based v8.x SDKs is required at this time it is recommended to explore a Docker-based solution, which is considered easier to use and integrate as it greatly simplifies any runtime dependency management through containerization. Docker support is readily available through the tisdk-thinlinux-image and tisdk-default-image image build targets.

Steps to enabling multilib Build and Runtime Environment

The multilib solution discussed here is dependent on using a Yocto-internal toolchain for image build process to minimize breakages and updates that would be required to different packages and recipes to get this to work using the Yocto Dunfell-based v8.x SDKs. The needed toolchain related changes are included in the patch tarball that is provided as part of this E2E FAQ. More details on those changes can be found here: https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1202155/faq-am6x-using-a-yocto-internal-toolchain-for-target-image-builds-and-cross-development-sdk-installer-generation-with-yocto-dunfell-based-v8-x-sdks-for-sitara-mpus

Configure the TOOLCHAIN_TYPE variable to internal. This can either be done in conf/local.conf by simply appending this to the end of the file:

# Use internal toolchain, the common Yocto way
TOOLCHAIN_TYPE = "internal"

or by making an update in the Yocto distribution layer that is used (meta-arago/meta-arago-distro/conf/distro/arago.conf in case of the default TI SDK).

ExclamationNote: Upcoming Yocto Kirkstone-based Sitara SDKs v9.x will use a Yocto-internal toolchain by default, and the above step is not applicable/needed.

Then, configure the build and runtime environments for multilib support to allow running 32-bit ARMv7A applications for example built with the AM335x and AM437x SDKs by using the exact same architectural "tune" configuration. Note that in case of AM62x we disable GPU support for the time being, as the associated drivers are not compatible with the 32-bit multilib build, as some of them currently can only be build for 64-bit operation. However it should be possible to allow GPU support while limiting it to the 64-bit environment and this E2E post will get updated once a solution for this is available.

# Define multilib target
require conf/multilib.conf
MULTILIBS = "multilib:lib32"
DEFAULTTUNE:virtclass-multilib-lib32 = "armv7athf-neon"

# Specify 32-bit packages to be added
IMAGE_INSTALL:append = " lib32-glibc lib32-libgcc lib32-libstdc++"

# AM62x GPU drivers don't support 32-bit, remove for now
MACHINE_FEATURES:remove = "gpu"

# The kms++ packages are not compatible with multilib builds, remove for now
RDEPENDS:packagegroup-arago-base:remove = "kms++ kms++-python"

Lastly, several patches need to be applied to the meta-arago, meta-arm, and meta-ti layers to fix a couple of issues the multilib transition and toolchain change expose. The patches are listed below and are provided in a tarball. To apply the patches, extract the tarball into the Yocto sources/ folder after SDK setup, and apply the patches one by one using git am to the respective Yocto layers based on their file location as shown below.

~/tisdk/am62xx-evm/sources
$ tar -xvf tisdk-yocto-dunfell-multilib-3-mar-2023.tar.gz */*.patch
meta-arago/0001-meta-toolchain-arago-fix-internal-toolchain-installe.patch
meta-arago/0002-meta-arago-use-nonarch_base_libdir-for-firmware-inst.patch
meta-arm/0001-optee-os_git.bb-Update-LIBGCC_LOCATE_CFLAGS-sysroot-.patch
meta-arm/0002-optee-os-Add-missing-sysroot-compiler-flag.patch
meta-ti/0001-optee-os-Fix-SRC_URI-to-allow-inherited-patch-to-get.patch
meta-ti/0002-meta-ti-use-nonarch_base_libdir-for-firmware-install.patch

~/tisdk/am62xx-evm/sources
$ git -C meta-arago am 0001-meta-toolchain-arago-fix-internal-toolchain-installe.patch
$ git -C meta-arago am 0002-meta-arago-use-nonarch_base_libdir-for-firmware-inst.patch
$ git -C meta-arm am 0001-optee-os_git.bb-Update-LIBGCC_LOCATE_CFLAGS-sysroot-.patch
$ git -C meta-arm am 0002-optee-os-Add-missing-sysroot-compiler-flag.patch
$ git -C meta-ti am 0001-optee-os-Fix-SRC_URI-to-allow-inherited-patch-to-get.patch
$ git -C meta-ti am 0002-meta-ti-use-nonarch_base_libdir-for-firmware-install.patch

Building a multilib-enabled Target Image and Cross-Development SDK Installer

After applying the outlined changes to the Yocto configuration file conf/local.conf and applying the provided patches, a multilib image capable of also running 32-bit applications can be build just as a target image would normally get build using the steps outlined in the SDK documentation. This example here builds the "default" image for AM62x, which is a full-featured image variant containing a variety of features and demo applications. The resulting target images can be found in the arago-tmp-default-glibc/deploy/images/am62xx-evm/ folder after the build. You can then use the usual process of for example programming the tisdk-default-image-am62xx-evm.wic.xz image to an SD card to be able to run this on an AM62-SK board.

# Build the AM62x Default Target System Image (same as SDK instructions, except no
# TOOLCHAIN* environmental variable setup needed)
$ MACHINE=am62xx-evm bitbake -k tisdk-default-image

Also since we are using a Yocto-internal toolchain in this setup, in order to generate the cross-development SDK installer with the target-specific toolchain and sysroots containing libraries, headers, etc. according to the configured packages we use the standard Yocto way of using the '-c populate_sdk' bitbake argument like shown below:

# Build the cross-development SDK installer tailored for the given target image
# After the build, the installer can be found at arago-tmp-default-glibc/deploy/sdk/
$ MACHINE=am62xx-evm bitbake tisdk-default-image -c populate_sdk

Note that since our multilib-capable target image supports two different variants of the ARM architecture, the cross-development SDK installer also contains two different sysroots actually, together with two different toolchains, one for the ARMv8A architecture, and another one for the ARMv7A architecture. When doing cross-compilation make sure you use the the environment-setup-* script according to the architecture variant you want to build your application for. But using / building for either architecture variant will result in an application image that will be able to run on the associated target image (here: tisdk-default-image).

Testing the multilib-enabled Solution

This section shows how to use the AM335x Linux SDK to cross-compile a simple C application into a 32-bit binary, and then how this binary is then executed on an AM62-SK board running on the multilib-enabled 64-bit/32-bit combo image created earlier.

Create a simple 32-bit Test Application that writes some strings to /tmp/test.txt and copy that Application to the SD card

$ cd ~/ti/ti-processor-sdk-linux-am335x-evm-08.02.00.24/linux-devkit/
$ ll
total 56
drwxrwxr-x 3 a0797059 a0797059  4096 Feb 28 16:00 ./
drwxr-xr-x 8 a0797059 a0797059  4096 Feb 28 15:57 ../
-rw-rw-r-- 1 a0797059 a0797059  3769 Feb 28 15:59 environment-setup
-rw-r--r-- 1 a0797059 a0797059  4326 Feb 28 15:59 environment-setup-armv7at2hf-neon-linux-gnueabi
-rw-r--r-- 1 a0797059 a0797059  1492 May 14  2022 post-relocate-setup.sh
-rw-r--r-- 1 a0797059 a0797059 22672 May 14  2022 site-config-armv7at2hf-neon-linux-gnueabi
drwxrwxr-x 4 a0797059 a0797059  4096 May 14  2022 sysroots/
-rw-r--r-- 1 a0797059 a0797059   124 May 14  2022 version-armv7at2hf-neon-linux-gnueabi
a0797059@ula0797059:~/ti/ti-processor-sdk-linux-am335x-evm-08.02.00.24/linux-devkit
$ source environment-setup
[linux-devkit]:~/ti/ti-processor-sdk-linux-am335x-evm-08.02.00.24/linux-devkit> cd ~/tmp/am335x-hello-world/
[linux-devkit]:~/tmp/am335x-hello-world> ll
total 12
drwxrwxr-x 2 a0797059 a0797059 4096 Mar  3 20:30 ./
drwxrwxr-x 7 a0797059 a0797059 4096 Feb 28 16:02 ../
-rw-rw-r-- 1 a0797059 a0797059  364 Feb 28 16:07 linux-test.c
[linux-devkit]:~/tmp/am335x-hello-world> cat linux-test.c 
#include <stdio.h>

int main(int argc, char *argv[])
{
    FILE *fp;

    printf("Hello World!\n");
    
    printf("Doing basic file write test...\n");
    fp = fopen("/tmp/test.txt", "w+");
    fprintf(fp, "This is testing for fprintf...\n");
    fputs("This is testing for fputs...\n", fp);
    fclose(fp);
    
    printf("All done!\n");
    
    return 0;
}

[linux-devkit]:~/tmp/am335x-hello-world> $CC linux-test.c -o linux-test
[linux-devkit]:~/tmp/am335x-hello-world> ll
total 24
drwxrwxr-x 2 a0797059 a0797059  4096 Mar  3 20:31 ./
drwxrwxr-x 7 a0797059 a0797059  4096 Feb 28 16:02 ../
-rwxrwxr-x 1 a0797059 a0797059 12176 Mar  3 20:31 linux-test*
-rw-rw-r-- 1 a0797059 a0797059   364 Feb 28 16:07 linux-test.c
[linux-devkit]:~/tmp/am335x-hello-world> file linux-test
linux-test: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for GNU/Linux 3.2.0, with debug_info, not stripped
[linux-devkit]:~/tmp/am335x-hello-world> sudo cp -v linux-test /media/a0797059/root/home/root
[sudo] password for a0797059: 
'linux-test' -> '/media/a0797059/root/home/root/linux-test'

Run the simple 32-bit Test Application on an AM62-SK Board

 _____                    _____           _         _   
|  _  |___ ___ ___ ___   |  _  |___ ___  |_|___ ___| |_ 
|     |  _| .'| . | . |  |   __|  _| . | | | -_|  _|  _|
|__|__|_| |__,|_  |___|  |__|  |_| |___|_| |___|___|_|  
              |___|                    |___|            

Arago Project am62xx-evm ttyS2

Arago 2021.09 am62xx-evm ttyS2

am62xx-evm login: root
root@am62xx-evm:~# ls -al
total 28
drwx------ 3 root root  4096 Mar  3 18:37 .
drwxr-xr-x 3 root root  4096 Mar  3 18:37 ..
-rw------- 1 root root   244 Mar  3  2023 .bash_history
drwx------ 3 root root  4096 Mar  3  2023 .config
-rwxr-xr-x 1 root root 12176 Mar  4  2023 linux-test
root@am62xx-evm:~# file linux-test 
linux-test: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for GNU/Linux 3.2.0, with debug_info, not stripped
root@am62xx-evm:~# ./linux-test 
Hello World!
Doing basic file write test...
All done!
root@am62xx-evm:~# cat /tmp/test.txt 
This is testing for fprintf...
This is testing for fputs...
root@am62xx-evm:~# 

tisdk-yocto-dunfell-multilib-3-mar-2023.tar.gz