PROCESSOR-SDK-AM64X: How to build "Full SDK"

Making another pass at the question.  More specifically, my requirement is to have an SDK with the full kernel headers in it.  It appears that these are in the board-support folder of the processor-sdk as distributed.

I was able to build my own version of the processor SDK by including this in my custom SDK recipe:

require recipes-core/images/tisdk-core-bundle.bb
TOOLCHAIN_TARGET_TASK:append = " cdk-staticdev"

This is a much bigger build.  It gives me a tar.xz file that, when extracted, gives something that looks a lot like TI's own processor-sdk.  When I unpack it and run the "setup-sdk.sh" script therein, I get a "linux-devkit" folder, but my customizations are not there.  And in fact, I can't find the customization files anywhere in the unpacked tree.

In contrast, when I build using

require recipes-core/meta/meta-toolchain-arago-tisdk.bb
TOOLCHAIN_TARGET_TASK:append = " cdk-staticdev"

I do get my customizations.


So I am wondering if building the "Full SDK" is barking up the wrong tree.  Perhaps I should instead be using my second approach but baking in some sort of Board Support recipe?  I started trying to sleuth that out and there are no recipes called "board-support*".  I tried grep instead, but there are hundreds of references to "board-support" in the tree, way too many to easily make sense of.

So restating my question to more directly reflect what I am starting to think my requirements are, "What is the correct way to build a customized SDK that includes the board support stuff?"

Thanks,
Brad

Actually, now that I dig into the board-support tree, I find that it may not resolve my issue either.  One of the files that eventually gets included by clocksource.h is <asm/rwonce.h>.

But when I look at the board-support folder in the Full SDK that is generated, only the arch/alpha version of the file is present:

~/lseb-processor-sdk$ find . -name 'rwonce.h' -print
./board-support/linux-rt-5.10.65+gitAUTOINC+541ec9a699-g541ec9a699/arch/alpha/include/asm/rwonce.h
./board-support/linux-rt-5.10.65+gitAUTOINC+541ec9a699-g541ec9a699/include/asm-generic/rwonce.h
./linux-devkit/sysroots/aarch64-linux/usr/include/asm-generic/rwonce.h

I do find promising looking files in the kernel-source and many others, but they are not getting included in the built SDK (results edited):

~/tisdk/build/arago-tmp-external-arm-glibc$ find . -name cache.h | grep asm | grep arm
./work/x86_64-nativesdk-arago-linux/nativesdk-linux-libc-headers/5.10-r0.arago0/git/arch/arm64/include/asm/cache.h
./work/x86_64-nativesdk-arago-linux/nativesdk-linux-libc-headers/5.10-r0.arago0/git/arch/arm/include/asm/cache.h
./work/am64xx_evm-linux/kernel-devsrc/1.0-r0/image/lib/modules/5.10.65-rt53-g541ec9a699/build/arch/arm64/include/asm/cache.h
./work/am64xx_evm-linux/kernel-devsrc/1.0-r0/package/lib/modules/5.10.65-rt53-g541ec9a699/build/arch/arm64/include/asm/cache.h
./work/am64xx_evm-linux/kernel-devsrc/1.0-r0/packages-split/kernel-devsrc/lib/modules/5.10.65-rt53-g541ec9a699/build/arch/arm64/include/asm/cache.h
./work/am64xx_evm-linux/tisdk-core-bundle/1.0-r0.tisdk19/rootfs/board-support/linux-rt-5.10.65+gitAUTOINC+541ec9a699-g541ec9a699/arch/arm64/include/asm/cache.h
./work/am64xx_evm-linux/tisdk-core-bundle/1.0-r0.tisdk19/rootfs/board-support/linux-rt-5.10.65+gitAUTOINC+541ec9a699-g541ec9a699/arch/arm/include/asm/cache.h
./work/am64xx_evm-linux/tisdk-core-bundle/1.0-r0.tisdk19/rootfs/board-support/u-boot-2021.01+gitAUTOINC+15769936a5-g15769936a5/arch/arm/include/asm/cache.h
./work/am64xx_evm_k3r5-linux-gnueabi/u-boot-ti-staging/1_2021.01+gitAUTOINC+15769936a5-r16/git/arch/arm/include/asm/cache.h
./work/am64xx_evm_k3r5-linux-gnueabi/u-boot-ti-staging/1_2021.01+gitAUTOINC+15769936a5-r16/srcipk-staging/board-support/u-boot-2021.01+gitAUTOINC+15769936a5-g15769936a5/arch/arm/include/asm/cache.h
./work/am64xx_evm_k3r5-linux-gnueabi/u-boot-ti-staging/1_2021.01+gitAUTOINC+15769936a5-r16/packages-split/u-boot-ti-staging-src/board-support/u-boot-2021.01+gitAUTOINC+15769936a5-g15769936a5/arch/arm/include/asm/cache.h
./work/armv7at2hf-vfp-linux-gnueabi/external-arm-toolchain/2019.12-r0/git/arch/arm64/include/asm/cache.h
./work/armv7at2hf-vfp-linux-gnueabi/external-arm-toolchain/2019.12-r0/git/arch/arm/include/asm/cache.h
./work-shared/am64xx-evm/kernel-source/arch/arm64/include/asm/cache.h
./work-shared/am64xx-evm/kernel-source/arch/arm/include/asm/cache.h

It seems like maybe the Full SDK is *trying* to pull in the file, but is pulling in the wrong arch?

As an aside, several days ago I was trying to build linux-libc-headers into my image, which appears to have a lot of these headers.  But it was not working.  I finally discovered one of the Arago recipes that gets baked in to the SDK has

   PREFERRED_PROVIDER_linux-libc-headers = "external-arm-toolchain"  

that causes the standard linux-libc-headers recipe to not run, and many of these key headers to not be included in the built SDK.  Instead of the standard recipe, or even the one in 
sources/meta-arago/meta-arago-distro/recipes-kernel/linux-libc-headers/linux-libc-headers_5.10.bb, what you end up getting something from the ARM toolchain, and it does not appear to include the <asm/...> headers.

One more data point.  I do see arch/alpha/include/asm/rwonce.h at numerous places in the built tree after baking the Full SDK.

But to be explicit, I am bitbaking the Full SDK using "MACHINE=am64xx-evm":

  MACHINE=am64xx-evm ARAGO_RT_ENABLE=1 bitbake sdk-lseb

where sdk-lseb.bb is the recipe that I have quoted snippets of earlier.  There's nothing I am doing that should be selecting the alpha arch.  Here is the console output from bitbake:

Build Configuration:
BB_VERSION = "1.46.0"
BUILD_SYS = "x86_64-linux"
NATIVELSBSTRING = "ubuntu-20.04"
TARGET_SYS = "aarch64-linux"
MACHINE = "am64xx-evm"
DISTRO = "arago"
DISTRO_VERSION = "2021.09"
TUNE_FEATURES = "aarch64"
TARGET_FPU = ""

[...]

Build Configuration:
BB_VERSION = "1.46.0"
BUILD_SYS = "x86_64-linux"
NATIVELSBSTRING = "ubuntu-20.04"
TARGET_SYS = "arm-linux-gnueabi"
MACHINE = "am64xx-evm-k3r5"
DISTRO = "arago"
DISTRO_VERSION = "2021.09"
TUNE_FEATURES = "arm armv7a vfp thumb callconvention-hard"
TARGET_FPU = "hard"

One more datum.  I do see what looks like a desired rwonce.h file under

/build/arago-tmp-external-arm-glibc/work-shared/am64xx-evm/kernel-build-artifacts/arch/arm64/include/generated/asm/rwonce.h

Apologies in the last message above for jumping back and forth between searches for "cache.h" and "rwonce.h".  Both are examples of files I'm having trouble with.

Braden Hines said:

I have successfully used Yocto/Arago to build and deploy a customized SDK using the "meta-toolchain-arago-tisdk" target, per the instructions at https://software-dl.ti.com/processor-sdk-linux/esd/docs/08_00_00_21/linux/Overview_Building_the_SDK.html

Yes that is the official target to build the cross toolchain installer. And TOOLCHAIN_TARGET_TASK:append is the step needed to add custom packages to it, which (unfortunately) needs to be kept in sync manually with the packages you make part of your image.

Braden Hines said:

However, the instructions at the link above for building the Full SDK are clearly out of date - there is no arago-core-tisdk-bundle target anywhere in the TI source tree.

The arago-core-tisdk-bundle target was changed to/replaced with tisdk-core-bundle, which was recently updated in our documentation as well. If you look at the corresponding section from the current SDK you'll see it documented there: https://software-dl.ti.com/processor-sdk-linux/esd/AM64X/08_06_00_42/exports/docs/linux/Overview_Building_the_SDK.html#build-options

I have not tried to re-create your header-file related concern but if you can perhaps you can setup a test build using a Yocto-internal toolchain approach? IMHO this is the much easier/simpler Yocto way of creating those cross-toolchain installers, and should have been the way our SDKs v8.x were built in the first place (note, upcoming SDK v9.x series will use this standard approach however). Please check this E2E FAQ here: https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1202155/faq-am6x-am335x-am437x-using-a-yocto-internal-toolchain-for-target-image-builds-and-cross-development-sdk-installer-generation-with-yocto-dunfell-based-v8-x-sdks-for-sitara-mpus   Should this work, we can look for a way to fix the SDK standard build with using the external toolchain if still needed.

Regards, Andreas

Thanks, Andreas, this is good info.

I think I have uncovered a deeper root cause for the missing headers - the timer driver simply isn't being included in the built kernel or the SDK.

I have determined that I believe that I'd like to add CONFIG_OMAP_DM_TIMER to the kernel, to pull in drivers/clocksource/timer-ti-dm.c.  (To refresh, this is for AM6442.)

But for some reason I don't understand, the option is not showing up in my .config and is not available in "make menuconfig".  I can search for it in "make menuconfig" and it is disabled, and is not available as something I can set.  I've tried manually editing .config and rebuilding.   I've tried adding "CONFIG_OMAP_DM_TIMER=y" to connectivity.cfg and building a new defconfig.  I've tried "make oldconfig". 

When I edit connectivity.cfg, the option CONFIG_OMAP_DM_TIMER does make it into my "ti_sdk_arm64_rt_release_defconfig" file, but no matter what I do, when I do the "make ti_sdk_arm64_rt_release_defconfig" step (note that I am building the RT kernel), CONFIG_OMAP_DM_TIMER disappears.

I've also searched the Kconfig files throughout the entire kernel source tree and am not finding any dependencies that would preclude OMAP_DM_TIMER from being included.

I'm not an expert on rebuilding kernels.  Is there some trick to get my selection to "take" and get timer-ti-dm.ko and friends to get built?

Thanks,
Brad

Braden Hines said:

I have determined that I believe that I'd like to add CONFIG_OMAP_DM_TIMER to the kernel, to pull in drivers/clocksource/timer-ti-dm.c.  (To refresh, this is for AM6442.)

This config should be enabled by default when building for any TI K3 device which includes all AM6x devices.

a0797059@dasso:~/git/linux (HEAD detached at 08.06.00.006)
$ git grep -A 5 'config.*OMAP_DM_TIMER'
drivers/clocksource/Kconfig:config OMAP_DM_TIMER
drivers/clocksource/Kconfig-    bool "OMAP dual-mode timer driver" if ARCH_K3 || COMPILE_TEST
drivers/clocksource/Kconfig-    default y if ARCH_K3
drivers/clocksource/Kconfig-    select TIMER_OF
drivers/clocksource/Kconfig-    help
drivers/clocksource/Kconfig-      Enables the support for the TI dual-mode timer driver.

Sure enough when creating a new .config for AM6x it's included there:

a0797059@dasso:~/git/linux (HEAD detached at 08.06.00.006)                                                                                                            
$ ti_config_fragments/defconfig_builder.sh                                                                                                                            
         1. v7 ARM Architecture                                                                                                                                       
         2. v8 ARM Architecture                                                                                                                                       
Please choose an architecture to build for or 'q' to exit: 2                                                                                                          
         1. SDK_Debug_Defconfigs                                                                                                                                      
         2. SDK_Release_Defconfigs                                                                                                                                    
                                                                                                                                                                      
Please choose a defconfig type to build for or 'q' to exit: 2                                                                                                         
Available SDK_Release_Defconfigs defconfig build options:                                                                                                             
                                                                                                                                                                      
        1. ti_sdk_arm64_release                                                                                                                                       
        2. ti_sdk_arm64_rt_release                                                                                                                                    
                                                                                                                                                                      
Please enter the number of the defconfig to build or 'q' to exit: 1                                                                                                   
Creating defconfig file /home/a0797059/git/linux/arch/arm64/configs/ti_sdk_arm64_release_defconfig
a0797059@dasso:~/git/linux (HEAD detached at 08.06.00.006)                                                                                                            
$ make -j16 ARCH=arm64 CROSS_COMPILE="ccache $TOOLCHAIN_PATH_ARMV8/bin/aarch64-none-linux-gnu-" ti_sdk_arm64_release_defconfig                                                                                                                            
  HOSTCC  scripts/basic/fixdep                                                                                                                                        
  HOSTCC  scripts/kconfig/conf.o                                                                                                                                      
  HOSTCC  scripts/kconfig/confdata.o                                                                                                                                  
  HOSTCC  scripts/kconfig/expr.o                                                                                                                                      
  LEX     scripts/kconfig/lexer.lex.c                                                                                                                                 
  YACC    scripts/kconfig/parser.tab.[ch]                                                                                                                             
  HOSTCC  scripts/kconfig/preprocess.o                                                                                                                                
  HOSTCC  scripts/kconfig/symbol.o                                                                                                                                    
  HOSTCC  scripts/kconfig/util.o
  HOSTCC  scripts/kconfig/lexer.lex.o
  HOSTCC  scripts/kconfig/parser.tab.o
  HOSTLD  scripts/kconfig/conf
#
# configuration written to .config
#
a0797059@dasso:~/git/linux (HEAD detached at 08.06.00.006)
$ grep DM_TIMER .config                                                             
CONFIG_OMAP_DM_TIMER=y

Also here's a neat trick how you can check if a certain CONFIG options is enabled on a running system/kernel. You can use this to double-check on your end of the CONFIG option is really your issue here.

root@am62xx-evm:/# zcat /proc/config.gz | grep DM_TIMER
CONFIG_OMAP_DM_TIMER=y

Regards, Andreas

Braden Hines said:

After some sleuthing, I deduced that it appears that the asm headers are associated with kerneI, and that need the "Full SDK" in order to do "kernel development".  (I am not doing kernel development with this application, but it appears that I need the headers for my application.)

Also what Kernel headers specifically you want to make available in the sysroot? Usually the Kernel only exports the UAPI (Userspace API) headers from <kernel_dir>/include/uapi. But sounds like you want headers beyond that (which risks/will be tying your application to a specific Kernel version, usually something people try to avoid!).

If so your solution is probably along the lines of this here https://docs.windriver.com/bundle/Wind_River_Linux_Tutorial_Exporting_Custom_Kernel_Headers_to_the_Sysroot_LTS_18_1/page/blt1582642421401.html   I've not checked yet if/how this applies to generic Yocto (and with that to the TI SDK), but it at least points into the right direction.

Regards, Andreas

Hi Andreas,

Thanks for your detailed reply.  

On my system, I am not getting the same result.  First, this is what I have in drivers/clocksource/Kconfig.  I think it's fine though, just an earlier version:

config I8253_LOCK
bool

config OMAP_DM_TIMER
bool

config CLKBLD_I8253
def_bool y if CLKSRC_I8253 || CLKEVT_I8253 || I8253_LOCK

Now I build the RT release config:

brad@MSI-Brad:~/tisdk/build/arago-tmp-external-arm-glibc/work/am64xx_evm-linux/linux-ti-staging-rt/5.10.65+gitAUTOINC+541ec9a699-r0b.arago1.tisdk0/git$ ti_config_fragments/defconfig_builder.sh
1. v7 ARM Architecture
2. v8 ARM Architecture
Please choose an architecture to build for or 'q' to exit: 2
1. SDK_Debug_Defconfigs
2. SDK_Release_Defconfigs

Please choose a defconfig type to build for or 'q' to exit: 2
Available SDK_Release_Defconfigs defconfig build options:

1. ti_sdk_arm64_release
2. ti_sdk_arm64_rt_release

Please enter the number of the defconfig to build or 'q' to exit: 2
Creating defconfig file /home/brad/tisdk/build/arago-tmp-external-arm-glibc/work-shared/am64xx-evm/kernel-source/arch/arm64/configs/ti_sdk_arm64_rt_release_defconfig

Make:

brad@MSI-Brad:~/tisdk/build/arago-tmp-external-arm-glibc/work/am64xx_evm-linux/linux-ti-staging-rt/5.10.65+gitAUTOINC+541ec9a699-r0b.arago1.tisdk0/git$ make ARCH=arm64 ti_sdk_arm64_rt_release_defconfig
arch/arm64/configs/ti_sdk_arm64_rt_release_defconfig:1850:warning: override: PREEMPT_RT changes choice state
#
# configuration written to .config
#

Grep:

brad@MSI-Brad:~/tisdk/build/arago-tmp-external-arm-glibc/work/am64xx_evm-linux/linux-ti-staging-rt/5.10.65+gitAUTOINC+541ec9a699-r0b.arago1.tisdk0/git$ grep DM_TIMER .config

[returns nothing]

As far as I can tell, the only thing I'm doing differently is selecting the RT version of the configuration in the final step.

And I'm on a diffferent SDK version - 08.02.00.23.

Any ideas?

Thanks,

Brad

Hi Brad,

I noticed this case is still open, please let me know if this still needs attention.

Regards, Andreas

HI Andreas,

Thanks for checking back.  Yes, this issue still exists, but it's on hold, and it may go away.

We have decided to bite the bullet and move on to Kirkstone and Ubuntu 22.04 the latest Arago.  It's possible that the difference in results is due to us being out of date.

Some other stuff has moved to the front burner, though, so this may not get attention for a little while.  I don't know if an issue can be parked in a "Dormant" state.  If so, that would be an appropriate flag for this topic for now.

I can update you when we return to this in a couple of months.

Thanks,
Brad

Braden Hines said:

We have decided to bite the bullet and move on to Kirkstone and Ubuntu 22.04 the latest Arago.

Yes good call, using this new/current SDK will ensure continued support in case of any issues. Also note that our Kirkstone-based SDK 9.0 should support generating toolchain installers in the more Yocto standard way of doing for example MACHINE=am62xx-evm bitbake tisdk-default-image -c populate_sdk instead of using the TI-specific target of meta-toolchain-arago-tisdk which has been used until now. When I experimented and made the internal toolchains work with SDK v8.x I noticed the resulting cross-toolchain installers are much more comprehensive, and didn't need tinkering with TOOLCHAIN_TARGET_TASK:append to get all the libs/headers one wants. Note that I said "should" I have not tried this with the official 9.0 release yet, but will do so very soon. You can find more info on this topic in the E2E FAQ I created for 8.x (https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1202155/faq-am6x-am335x-am437x-using-a-yocto-internal-toolchain-for-target-image-builds-and-cross-development-sdk-installer-generation-with-yocto-dunfell-based-v8-x-sdks-for-sitara-mpus) which I plan on updating with some 9.0 content once I get a chance to try things out.

Braden Hines said:

I don't know if an issue can be parked in a "Dormant" state.  If so, that would be an appropriate flag for this topic for now.

That is not possible. We need to close the issue for the time being, and once you are ready to pick this back up again, and you run into similar issues even with 9.0, please file a new E2E ticket (it would be good to link back to this thread here too for completeness sake).

Thanks, Andreas