Original question:
Hello.
Recently I have moved to SDK_08_05. The SBL created with "make sbl_mmcsd_img_hlos" creates an SBL that fails to run mulltiimage with FreeRtos image in MC1_0 and Arago in A72.
The same app was launched with a solution done in the compilation in SDK_08_04 that I was using before.
I have detected changes in the TIFS binary for every SDK version so that I also try to change it, but the issue is still happening.
Could you clarify how to solve this in 08_05 version?
Thanks
Hello Parh.
App created with combined_img make option, including CAN_profile demo for MCU1_0 from MCUSW package (with minor changes for UART messages and loop iterations) and all needed to launch ARAGO in A72.
The situation is that the app is launched using SBL HLOS built from SDK 8.4 (UART_MCU and UART_MAIN0 show flow execution expected, but not for the one generated from 8.5 with same app. Content of SD card is always the same, only switching tiboot3.bin with SBL from SDK8.4 or SDK8.5
Here log for SBL generated from 8.5 (compiled with LOG_LEVEL 3);
Here is the one from 8.4 (SBL compiled with LOG_LEVEL 3):
App in MCU is running:
Thanks
Hello. For the TIFS, I have tested both. Currently, TIFS in SD Card is the one provided in 8.5, and also leads sbl from 8.4 to run correctly:
Thus, TIFS from 8.5 is ok for SBL for 8.4
App, as I explained, is unvaried, so is the one generated when I used SDK 8.4. Should I generate new app (also compiling MCUSW project from the new SDK) from the new SDK (8.5)?
BR
Hello Nikhil.
Monica says that CAN DEM is happening every time, but is not. Sometimes happens, but I don't have log for that. As soon as it happens I will attach to the post.
Besides that, Arago most of the tries gets stuck in remote proc (not clear if is a timing occurrence or is happing because of specific operations on kernel load). We have done tries in 3 different setups, creating in each setup all binaries needed (SBL, patching, CAN profile, DTB, multi-image, etc) and we observed the same behavior in all of them. We don't have a clear statistic, but we assume that in around 90% of PowerOff-PowerON tests, A72 is stuck in remoteproc operations (before the stopping point failure "-22" is observed instead of result of "-2" in remote proc operations when is not stuck)
(*) attached complete log for MCU and A72 in booting
7144.remote_proc_A72_stuck.zip
As it seemed that this failure could be caused by the absence of firmware for the rest of the cores, we modified conf.mk to generate an app with all of them integrated. Then, we assume that SBL (when HLOS_BOOT is optimized in multiimage creation) will be responsible for loading in proper allocation all binaries needed by kernel. We also assume that when SPL booting is used, loading binaries are done by Uboot (please, could you confirm if whole of this understanding is correct?)(**)
In this case, SBL gets stuck in loading some firmware at @0x400000.
(**) Attaching conf.mk with modifications
Hello Nikhil. Many thanks for your support.
About issue observed some days ago, and comented by me and Monica (https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1198329/processor-sdk-j721e-sbl-for-hlos-sd-card-fails-in-sci-client/4544700#4544700) now (once vision APPs are running using SBL and combined image in optimized mode) we observe on every run one DEM error that stops CAN transmission.
After some time running CAN_profile (that is patched with the pathc you provided), at some point of loading kernel in A72 CAN the transmission is stopped and DEM Event (internally coded as 7) is appearing. Busoff is deteted (Can_mcanCancelAllPendingMessages is called because of that in Can_mcanProcessISR)
(I modified code to identify which DemConf_DemEventParameter_CAN_E_HARDWARE_ERROR was causing the error, then coded to 7)
Debugging can profile it is observed MCU_MCAN0_CFG (@0x40528050) has value 0xB800000 when Can_mcanCancelAllPendingMessages is called, so PEA, BO, EW and EP are actived
I guess this is happenign because of DTB used by kernel (memories or clock path, etc). But I can not say at this exact A72 stage is this happening.
Could you help on this?
Thanks
Summary of the ticket
Goal: To be able to run a CAN application in MCU1_0 concurrently with one application from vision_apps in the Linux side.
Due to the need of having early CAN we wanted to use SBL HLOS in development mode
Current status / Steps to reproduce:
#define APP_NUM_ITERATION (300U)
The steps we follow from a clean SDK to build and run the applications are:
1. In the linux kernel source base path (board-support/linux-5.10.153+gitAUTOINC+90c3a58fd2-g90c3a58fd2) we apply the following patches:
- Linux_changes.patch
- 1616.0001_2D00_vision_5F00_apps_2D00_Integrated.patch
The only conflict with these two paths are the bootargs line in k3-j721e-common-proc-board.dts (line 17), which we resolve as:
`bootargs = "console=ttyS2,115200n8 earlycon=ns16550a,mmio32,0x02800000 root=/dev/mmcblk1p2 rw rootfstype=ext4 rootwait";`
- Apply vision_apps_integrated.patch
2. Apply mcusw_changes.patch in the mcusw folder of the sdk_rtos
3. Clean SD partitions setup.
- ./psdk_rtos/scripts/mk-linux-card.sh /dev/mmcblk0
- ./psdk_rtos/scripts/install_to_sd_card.sh
4. Build the Linux Kernel DTBS (from ${SDK_LINUX})
- make linux-dtbs
5. Build CAN profile application (from ${PSDK_RTOS}/mcusw/build)
- make -s can_profile_app BOARD=j721e_evm SOC=j721e BUILD_PROFILE=debug CORE=mcu1_0 BUILD_OS_TYPE=freertos
6. Prepare combined image by modifying ${PSDK_RTOS}/pdk_jacinto_08_05_00_36/packages/ti/boot/sbl/tools/combined_appimage/config.mk (by default optimized mode for SBL_HLOS is set)
- DTB_IMG to point to the previously compiled arch/arm64/boot/dts/ti/k3-j721e-common-proc-board.dtb
- IMG1 to point to the previously compiled mcusw/binary/can_profile_app_freertos/bin/j721e/can_profile_app_freertos_mcu1_0_debug.xer5f
- IMG2-IMG6 point to the vision apps binaries (MCU2_0, MCU2_1, C66_1, C66_2 and C7)
7. Generate combined image
- make
8. Compile sbl_mmcsd_hlos in debug mode. Since bulding SBL HLOS in debug mode (BUILD_PROFILE=debug option) gives an error, to bypass it we have commented these lines in ti-processor-sdk-rtos-j721e-evm-08_05_00_11/pdk_jacinto_08_05_00_36/packages/ti/boot/sbl/sbl_component.mk
# SBL not supported for any profile # other than release # >>>>>> Comment these lines to enable debug build #ifneq ($(BUILD_PROFILE), release) #sbl_LIB_LIST = #sbl_EXAMPLE_LIST = #SBL_CFLAGS = #endif # ifneq ($(BUILD_PROFILE), release)
also modify pdk_jacinto_08_05_00_36/packages/ti/boot/sbl/soc/k3/sbl_log.h to get all prints to UART
//#define SBL_log(dbg_level, ...) if ((int32_t)(dbg_level) <= SBL_LOG_LEVEL) { UART_printf(__VA_ARGS__); }
#define SBL_log(dbg_level, ...) UART_printf(__VA_ARGS__);
- make sbl_mmcsd_img_hlos BUILD_PROFILE=debug BOARD=j721e_evm SOC=j721e
9. Copy binaries to SD.
- cp ti-processor-sdk-rtos-j721e-evm-08_05_00_11/pdk_jacinto_08_05_00_36/packages/ti/boot/sbl/binary/j721e_evm/mmcsd /bin/sbl_mmcsd_img_hlos_mcu1_0_debug.tiimage /media/$USER/BOOT/tiboot3.bin
- cp ti-processor-sdk-rtos-j721e-evm-08_05_00_11/pdk_jacinto_08_05_00_36/packages/ti/drv/sciclient/soc/V1/tifs.bin /media/$USER/BOOT/
- cp ti-processor-sdk-rtos-j721e-evm-08_05_00_11/pdk_jacinto_08_05_00_36/packages/ti/boot/sbl/tools/combined_appimage/bin/j721e_evm/combined.appimage /media/$USER/BOOT/app
Problems we are facing:
1. DEM error reported in CAN Profile
Once vision APPs are running using SBL and combined image in optimized mode we observe on every run one DEM error that stops CAN transmission.
After some time running CAN_profile (that is patched with the patch provided), at some point of loading kernel in A72 CAN the transmission is stopped and DEM Event is appearing. To further debug, we instrument the code and see that the eventID of DEM is internally coded as 7. Busoff is deteted (Can_mcanCancelAllPendingMessages is called because of that in Can_mcanProcessISR)
Debugging Can profile it is observed MCU_MCAN0_CFG (@0x40528050) has value 0x0B80000 when Can_mcanCancelAllPendingMessages is called, so PEA, BO, EW and EP are actived
We think this is happening because of DTB used by kernel (memories or clock path, etc)
Other tests we’ve tried:
2. Coredumps in the Kernel (SERDES)
We are facing some problems with SERDES as it is causing core-dumps in the Kernel
[ 1.625229] wiz: probe of bus@100000:wiz@5000000 failed with error -12
[ 1.641722] ------------[ cut here ]------------
[ 1.646494] wiz bus@100000:wiz@5010000: Unable to create SERDES platform device
[ 1.653988] WARNING: CPU: 0 PID: 61 at drivers/phy/ti/phy-j721e-wiz.c:1551 wiz_probe+0xbe8/0x1118
[ 1.663053] Modules linked in:
[ 1.666169] CPU: 0 PID: 61 Comm: kworker/0:2 Tainted: G W 5.10.153-g90c3a58fd2 #1
[ 1.675144] Hardware name: Texas Instruments K3 J721E SoC (DT)
[ 1.681105] Workqueue: events deferred_probe_work_func
[ 1.686352] pstate: 60000005 (nZCv daif -PAN -UAO -TCO BTYPE=--)
[ 1.692486] pc : wiz_probe+0xbe8/0x1118
[ 1.696398] lr : wiz_probe+0xbe8/0x1118
[ 1.700310] sp : ffff800011713a80
[ 1.703688] x29: ffff800011713a80 x28: ffff00087fa1a880
[ 1.709112] x27: 0000000000000000 x26: 0000000000000002
[ 1.714536] x25: 0000000000000002 x24: ffff0008273e5c10
[ 1.719960] x23: ffff00082891d108 x22: ffff00082891d0e8
[ 1.725385] x21: 0000000000000002 x20: ffff0008273e5c10
[ 1.730808] x19: ffff00082891d080 x18: 0000000000000010
[ 1.736233] x17: 0000000000000000 x16: 0000000012117bb7
[ 1.741657] x15: ffff000827c9d930 x14: 0000000000000140
[ 1.747080] x13: ffff000827c9d930 x12: 00000000ffffffea
[ 1.752504] x11: ffff8000111b04b0 x10: ffff800011198470
[ 1.757928] x9 : ffff8000111984c8 x8 : 0000000000017fe8
[ 1.763352] x7 : c0000000ffffefff x6 : 0000000000000001
[ 1.768776] x5 : 0000000000000000 x4 : 0000000000000000
[ 1.774200] x3 : 00000000ffffffff x2 : ffff800011140440
[ 1.779623] x1 : d5be478556c6da00 x0 : 0000000000000000
[ 1.785048] Call trace:
[ 1.787540] wiz_probe+0xbe8/0x1118
[ 1.791097] platform_drv_probe+0x54/0xa8
[ 1.795187] really_probe+0xec/0x3e0
[ 1.798833] driver_probe_device+0x58/0xb8
[ 1.803012] __device_attach_driver+0xb8/0xe0
[ 1.807457] bus_for_each_drv+0x78/0xc8
[ 1.811370] __device_attach+0xf8/0x188
[ 1.815282] device_initial_probe+0x14/0x20
[ 1.819549] bus_probe_device+0x9c/0xa8
[ 1.823462] deferred_probe_work_func+0x88/0xc0
[ 1.828087] process_one_work+0x1a0/0x328
[ 1.832177] worker_thread+0x1f8/0x420
[ 1.836002] kthread+0x140/0x160
[ 1.839293] ret_from_fork+0x10/0x34
[ 1.842939] ---[ end trace f86c66cf3a0c6091 ]---
[ 1.847904] wiz: probe of bus@100000:wiz@5010000 failed with error -12
[ 1.864361] ------------[ cut here ]------------
Nikhil provided some improvements in FAQ
but we are still seeing the same problem, so we have not taken his change so far
Hi Pablo,
The CAN transceiver seems to be getting probed by Linux, which is interfering with the CAN on RTOS. Please disable the CAN transceiver node from dts and try.
Please see the changes required below:
With these changes I have been able to run the CAN profile app without any issues for 300 iterations.
Regards,
Parth
Hi Parth.
We applied your patch, and issue is solved finally. Thanks!
Rewinding to the things listed by us as "Summary of the ticket" , there still are two important things to resolve our booting for early Can with no issues in Linux
Summary of the ticket
1.- Combined image can only be created for Optimized, and we want Development mode, in order to have U-Boot run in A72 startup
2.- SERDES issue.
We would apreciate any suggestions/tips/resolutions on that.
Thanks