AM62A7: TIDSS Error

Jason Zeng

Part Number: AM62A7
Other Parts Discussed in Thread: TDA4VH

SDK ver: 10_00_00_08

My application was running normally on version 10.00.00.08. Since we needed to update the TIDL tool to version 10.1, I referred to the linked documentation and updated the TIDL tool on our custom board to version 10.1. As our custom board lacks a Python environment and other env, I manually replaced the following files: vx_app_rtos_linux_c7x_1.out.signed, vx_app_rtos_linux_c7x_1.out, tidl_lib, and created a symbolic link from libonnxruntime.so.1.15.0to libonnxruntime.so. Afterward, the 10.1 version of the model ran normally. However, a new issue has emerged: the image intermittently flickers, accompanied by the error message:

tidss 30200000.dss: CRTC1 SYNC LOST: (irq 800)

How can this be resolved?

Here is the download link for the c7x firmware and tidl lib.

https://software-dl.ti.com/jacinto7/esd/tidl-tools/10_01_03_00/FIRMWARES/am62a/edgeai/10_0/firmware.tar.gz

https://software-dl.ti.com/jacinto7/esd/tidl-tools/10_01_03_00/FIRMWARES/am62a/edgeai/10_0/tidl_lib.tar.gz

5 months ago

0 Jason Zeng 5 months ago

Intellectual 375 points

After I restored the firmware and the tild lib, this tidss error no longer appeared.

0 Reese Grimsley 5 months ago in reply to Jason Zeng

TI__Mastermind 18176 points

Hi Jason,

This is an interesting observation. TIDSS should not have a direct relationship to the TIDL components, but I have an idea why this is happening.

The TIDL 10.1 release made some changes to performance-sensitive components, and some models have different performance now. So why is this relevant?

The tidss message you are receiving often means that synchronization between the gstreamer pipeline and the monitor was lost, likely because some frames were dropped. This may happen if your model is running at a slower rate than your monitor's output.

I believe this should resolve if you set that 'sync' setting on the kmssink plugin to be false. The last plugin of most gstreamer pipelines I use is like the following:

kmssink driver-name=tidss force-modesetting=true sync=false

You didn't specifically mention that you use gstreamer but I have assumed so. Please correct if this is the assumption is wrong.

0 Jason Zeng 5 months ago in reply to Reese Grimsley

Intellectual 375 points

Hi Reese

Just as you mentioned, I'm using GStreamer for video output. My output frame rate is 60fps while the algorithm runs at 25-30fps. Although I've set sync=falsefor kmssink, I haven't enabled force-modesetting=true.

    kmssink->set_property("sync", FALSE);
    kmssink->set_property("can-scale", FALSE);
    kmssink->set_property("skip-vsync", TRUE); // 60 fps

Jason

0 Jason Zeng 5 months ago in reply to Reese Grimsley

Intellectual 375 points

Hi Reese

I forgot to mention that I also updated the libtivision_apps.solibrary.This is also included in the tidl_lib.tar.gzpackage.

Jason

0 Reese Grimsley 5 months ago in reply to Jason Zeng

TI__Mastermind 18176 points

Hi Jason,

The force-modesetting parameter has more impact on setting the monitor resolution to the same output resolution as the stream coming from gstreamer. If you were already setting sync=false, then perhaps it did not alter the behavior on your side. Is the display still flickering?

I seem to recall internal dialogue on this same topic -- let me ask a few development folks if there was a different solution (assuming the issue is present still).

Does inserting a queue before kmssink alter any behavior either?

Edit: I found a externally published JIRA that was relevant to this issue. For that demo-application that runs by default, we had swapped in a model with lower resolution/ latency so circumvent. I'll see if there are any other recommendations we can provide

BR,
Reese

0 Jason Zeng 5 months ago in reply to Reese Grimsley

Intellectual 375 points

Hi Reese

The issue I'm encountering occurs specifically when setting sync=false for kmssink, and there's already a queue present before kmssink. The complete pipeline is as follows:

cam -> msc -> caps -> q -> mosaic -> q -> kmssink
       |
       +----> caps -> q -> msc2 -> caps -> q -> appsink(algo)

Jason

0 Reese Grimsley 5 months ago in reply to Jason Zeng

TI__Mastermind 18176 points

Hi Jason,

I've been discussing this with our development team. This display-starvation issue does not have a clean fix in place, either for your SDK version or the newer 11.1. The main workaround is to avoid running display pipelines at <30fps

This flickering issue can be resolved by running a smaller / faster model such that there is not a bottleneck to limit performance < 30 fps going into the KMSSINK plugin. You may also try removing anything like tiperfoverlay plugin if there is one between the tiovxmosaic and kmssink plugins.

Is 30 fps display part of your key requirements? We include a display subsystem on this device, but since there is no GPU it has more limited utility. This DSS component is commonly used in debugging and development, but less often in the production system. A more common use-case is to use the video encoder to create output videos

Another alternative is to use application code such that it forces an output image to be available at 30 FPS rate, even if that means replaying an old frame (and recognizing when an inference should be skipped to maintain FPS).

BR,
Reese

0 Jason Zeng 5 months ago in reply to Reese Grimsley

Intellectual 375 points

Hi Reese

Our application is the Camera Mirror System, which requires developing a product compliant with R46 and ISO 26262(ASIL-B) standards. Therefore, a display component is essential. To achieve better visual performance and lower display latency, our current camera operates at 60fps, the tidss size is 1920x720, so the video stream also runs at 60fps.

Regarding the "smaller/faster model" you mentioned, from which dimensions should it be evaluated? Is it based on model input size? Network size?

BR,

Jason

0 Reese Grimsley 5 months ago in reply to Jason Zeng

TI__Mastermind 18176 points

Hi Jason,

Understood on the system requirement. Yes this would require full rate without flickering, then. You may also consider to run your AI models with TIDL asynchronously from the output video stream. With gstreamer, this would require appsrc / appsink and corresponding application code

Jason Zeng said:
Regarding the "smaller/faster model" you mentioned, from which dimensions should it be evaluated

Fundamentally, this is latency-driven. You want a faster runtime if this model is running synchronously with your imaging / video pipeline.

Reducing the input dimensions is one way to do this (and for many CNN's, doesn't require retraining). Otherwise, you can reduce the size of layers (e.g. number of channels for a convolution) and the number of them (i.e. the depth of the deep learning model).

Keep in mind that running smaller / faster models usually includes some accuracy loss.

In the external Jira ticket [1] I had mentioned before, we replaced the deeplabv3 model with a smaller segmentation model. This is both a simpler model and has smaller dimensions (512->384).

BR,
Reese

0 Jason Zeng 5 months ago in reply to Reese Grimsley

Intellectual 375 points

Hi Reese

Thank you. I will coordinate with our algorithm colleagues to try this approach in the coming days and will report back with the results.

BR,

Jason

0 Reese Grimsley 5 months ago in reply to Jason Zeng

TI__Mastermind 18176 points

Understood, thanks for the quick confirmation, Jason.

Let me know if you would like any references for appsrc / appsink usage as a starting point for asynchronous handling (if you choose to go in that direction).

0 Jason Zeng 5 months ago in reply to Reese Grimsley

Intellectual 375 points

Hi Reese

I've implemented asynchronous processing using appsink. Here's the connection diagram of my main media pipeline nodes: In MSC_1 I obtain a 608x352 image frame, which gets delivered to the appsink, then a dedicated thread retrieves this frame from the sink and feeds it into the algorithm for processing.

Jason

0 Reese Grimsley 5 months ago in reply to Jason Zeng

TI__Mastermind 18176 points

Oh, Forgive my misunderstanding. You are experiencing display starvation despite already running your AI algorithms in appsink / async thread.

For the multiscaler and mosaic, are you setting the target property for these? There are 2 physical instances of MSC (target 0 and 1). Both these mosaic and multiscaler plugins use these MSC's. Also, what are the two feeds going into the mosaic plugin? Is it just one feed but an overlay / background image is added?

I'm also curious if you've used any gstreamer functions (including our gst-tracers tool [1]) to determine where the frame drops are in this case. Appsink for algorithms will use DDR of course, but otherwise this pipeline should be able to keep up with 60FPS, even for larger input frames (like 5MP) from camera

And if you remove the appsink and corresponding processing, then I suppose you are seeing expected smooth 60 FPS? Please confirm.

[1] https://texasinstruments.github.io/processor-sdk-doc/processor-sdk-edgeai-AM62AX/esd/docs/11_01_07_05/edgeai/measure_perf.html#parse-gst-tracers

BR,
Reese

0 Jason Zeng 5 months ago in reply to Reese Grimsley

Intellectual 375 points

Hi Reese

I have set different target for 2 MSC(0 & 1).

In our application, the screen resolution is 1920x720, and the camera resolution is 1920x1536@60. For meet the field-of-vision for both R46 Class II and Class IV.We split the single camera frame into two by MSC: One compliant with Class II and the other with Class IV.

For chn0, the display area on the screen starts at coordinates (0,0) with a size of 1280x720 pixels, while chn1 begins at (1280,0) with a size of 640x720 pixels. Therefore, we need to use mosaic processing to merge these two channels into a single 1920x720 display output.

During previous testing, I distinctly observed that frame stuttering consistently occurred when the system output the log message "tidss 30200000.dss: CRTC1 SYNC LOST: (irq 800)". Therefore, I deliberately avoided using tracer tools for diagnosis.

I will verify whether removing appsink (i.e., disabling algorithm processing) can maintain 60 FPS performance, and will provide you with an update within this timeframe.

Jason

0 Jason Zeng 4 months ago in reply to Reese Grimsley

Intellectual 375 points

Hi Reese

After removing the appsink branch, there were no error logs from tidss.

Jason

0 Reese Grimsley 4 months ago in reply to Jason Zeng

TI__Mastermind 18176 points

Hi Jason,

That is an interesting finding. I am curious now if the issue is tied to C7xMMA running TIDL, or if it is more related to high load on DDR (A53 may be relevant as well).

You can simulate DDR loads using the stress-ng command, like

stress-ng --memrate 1 --memrate-wr 2000 --memrate-rd 2000 -v

If you run this in the background during your application, do you see similar sync-lost messages, even when appsrc is not present? I have not been able to replicate similar behavior on my side for this sync-loss issue

I have on other question -- is weston or wayland window management software running while you are doing this? You can check with 'systemctl' and see if that service is present in the interactive logging shown.

0 Jason Zeng 4 months ago in reply to Reese Grimsley

Intellectual 375 points

Hi Reese

Here is the complete pipeline of our application. There are two sources in total: one is the camera (cam), and the other is videotestsrc which outputs a black screen to display when no camera is connected. Additionally, there are four sinks: one is kmssink for real-time display, and the other three correspond to different functional modules. Among them, appsink_alg is used for algorithm processing, appsink_detect performs image quality detection using OpenCV, and appsink_venc is used for video recording. After several rounds of verification, I found that when all three functional modules are enabled simultaneously, DSS-related errors occur along with screen tearing. However, if only any two of the three modules (alg, detect, venc) are enabled, no screen tearing occurs. Before we upgraded the TIDL version, running all three functional modules simultaneously had always worked fine. Therefore, I proceeded to verify the DDR usage issue you mentioned. While running stress-ng --memrate 1 --memrate-wr 2000 --memrate-rd 2000 -vin the background and enabling only real-time display (i.e., all three functional modules disabled), no screen tearing occurred.

Additionally, since our imgis extended from a minimal filesystem, components like Weston and Wayland were not included in the filesystem, so these processes do not run in the background either.

Jason

0 Reese Grimsley 4 months ago in reply to Jason Zeng

TI__Mastermind 18176 points

Hi Jason,

The comprehensive view here is helpful and appreciated. Thank you as well for running these tests.

Jason Zeng said:
Therefore, I proceeded to verify the DDR usage issue you mentioned. While running stress-ng --memrate 1 --memrate-wr 2000 --memrate-rd 2000 -vin the background and enabling only real-time display (i.e., all three functional modules disabled), no screen tearing occurred.

So you say that the pipeline behaves well when only 2 of these appsinks are active, and fails when you add the third, regardless of which one that is. Can you run this stress-test alongside those 2/3 appsinks? You can also simulate CPU (A53) loads with the same tool like so:

stress-ng --cpus <ncpus> --cpu-load <percentage as int like 40> --timeout <duration_in_seconds>

I've noted from performance indicators (DDR, CPU load) that the behavior is bursty and may go above the % set for a short interval of time. This should make the error easier to reproduce, if it is resource contention.
If you are running TIDL at a different rate than the image pipeline, you will certainly have bursty periods of DDR utilization

What I'm trying to determine, is if this sync-lost effect is a result of system DDR load / CPU load. libtivision_apps will touch the underlying implementation of gstreamer plugins via TI OpenVX, which the TIDL libraries have dependency on. C7x firmware itself should only impact C7x behavior and system DDR load. I'm not sure how these libraries could cause screen tearing unless the issue is related to system-level performance

Does dmesg/kernel logs ever show an messages from VPU aka wave5 video encoder/decoder? If you could share the dmesg logs, that may be informative as well

I'm also curious how close your application is to maximum DDR allocation. If you are close to this allocation, then I could see a scenario where buffers from queues are trying to allocate, but cannot in time such that frames are dropped. Are you modifying any buffer-pool sizes for tiovx* plugins?

Also, I assume (but you may verify if willing) that gstreamer will recognize frames being dropped. Have you analyzed where these may be happening? KMSSink is the obvious one, but it may be happening elsewhere too. [1]

[1] https://software-dl.ti.com/processor-sdk-linux/esd/AM62AX/11_01_07_05/exports/docs/edgeai/measure_perf.html#parse-gst-tracers

BR and thank you for your patience -- this is a challenging one,
Reese

0 Jason Zeng 4 months ago in reply to Reese Grimsley

Intellectual 375 points

Hi Reese

Let me update my previous statement with new findings.

Jason Zeng said:
if only any two of the three modules (alg, detect, venc) are enabled, no screen tearing occurs

In reality, my testing revealed that screen tearing along with the "CRTC1 SYNC LOST" error occurs when appsink_algo is enabled alongside one or more additional functional modules. For example:

Enabling algo + detect
Enabling algo + venc
Enabling all three (algo + venc + detect)

However, if only detectand vencare enabled (without algo), the issue does not occur.

Additionally, I tested running stress-ngto simulate memory pressure while using a confirmed problematic pipeline configuration. The results showed that the frequency of screen tearing did not significantly increase under DDR stress.

root@am62axx-evm:~# stress-ng --memrate 1 --memrate-wr 2000 --memrate-rd 2000 -v
stress-ng: debug: [1220] invoked with 'stress-ng --memrate 1 --memrate-wr 2000 --memrate-rd 2000 -v' by user 0 'root'
stress-ng: debug: [1220] stress-ng 0.17.05 g4e68895f4fe6
stress-ng: debug: [1220] system: Linux am62axx-evm 6.6.32 #46 SMP PREEMPT Wed Oct 22 09:41:51 HKT 2025 aarch64, gcc 13.3.0, glibc 2.39
stress-ng: debug: [1220] RAM total: 3.2G, RAM free: 2.8G, swap free: 0.0
stress-ng: debug: [1220] temporary file path: '/root', filesystem type: ext2 (26415103 blocks available)
stress-ng: debug: [1220] 4 processors online, 4 processors configured
stress-ng: info:  [1220] defaulting to a 1 day, 0 secs run per stressor
stress-ng: debug: [1220] cache allocate: using cache maximum level L2
stress-ng: debug: [1220] CPU data cache: L1: 32K, L2: 512K
stress-ng: debug: [1220] cache allocate: shared cache buffer size: 512K
stress-ng: info:  [1220] dispatching hogs: 1 memrate
stress-ng: debug: [1220] starting stressors
stress-ng: debug: [1220] 1 stressor started
stress-ng: debug: [1221] memrate: [1221] started (instance 0 on CPU 1)
stress-ng: info:  [1221] memrate: using buffer size of 262144K, cache flushing disabled
stress-ng: info:  [1221] memrate: cache flushing can be enabled with --memrate-flush option

Here is the dmesg log after CRTS1 SYNC LOST error

Fullscreen error_dmesg.log Download

[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd034]
[    0.000000] Linux version 6.6.32 (flork@ti) (aarch64-oe-linux-gcc (GCC) 13.3.0, GNU ld (GNU Binutils) 2.42.0.20240620) #46 SMP PREEMPT Wed Oct 22 09:41:51 HKT 2025
[    0.000000] KASLR disabled due to lack of seed
[    0.000000] Machine model: Texas Instruments AM62A7 SK
[    0.000000] earlycon: ns16550a0 at MMIO32 0x0000000002800000 (options '')
[    0.000000] printk: bootconsole [ns16550a0] enabled
[    0.000000] efi: UEFI not found.
[    0.000000] Reserved memory: created CMA memory pool at 0x00000000c0000000, size 576 MiB
[    0.000000] OF: reserved mem: initialized node linux,cma, compatible id shared-dma-pool
[    0.000000] OF: reserved mem: 0x00000000c0000000..0x00000000e3ffffff (589824 KiB) map reusable linux,cma
[    0.000000] OF: reserved mem: 0x0000000080000000..0x000000008007ffff (512 KiB) nomap non-reusable tfa@80000000
[    0.000000] Reserved memory: created DMA memory pool at 0x0000000099800000, size 1 MiB
[    0.000000] OF: reserved mem: initialized node c7x-dma-memory@99800000, compatible id shared-dma-pool
[    0.000000] OF: reserved mem: 0x0000000099800000..0x00000000998fffff (1024 KiB) nomap non-reusable c7x-dma-memory@99800000
[    0.000000] Reserved memory: created DMA memory pool at 0x0000000099900000, size 31 MiB
[    0.000000] OF: reserved mem: initialized node c7x-memory@99900000, compatible id shared-dma-pool
[    0.000000] OF: reserved mem: 0x0000000099900000..0x000000009b7fffff (31744 KiB) nomap non-reusable c7x-memory@99900000
[    0.000000] Reserved memory: created DMA memory pool at 0x000000009b800000, size 1 MiB
[    0.000000] OF: reserved mem: initialized node r5f-dma-memory@9b800000, compatible id shared-dma-pool
[    0.000000] OF: reserved mem: 0x000000009b800000..0x000000009b8fffff (1024 KiB) nomap non-reusable r5f-dma-memory@9b800000
[    0.000000] Reserved memory: created DMA memory pool at 0x000000009b900000, size 15 MiB
[    0.000000] OF: reserved mem: initialized node r5f-dma-memory@9b900000, compatible id shared-dma-pool
[    0.000000] OF: reserved mem: 0x000000009b900000..0x000000009c7fffff (15360 KiB) nomap non-reusable r5f-dma-memory@9b900000
[    0.000000] Reserved memory: created DMA memory pool at 0x000000009c800000, size 1 MiB
[    0.000000] OF: reserved mem: initialized node r5f-dma-memory@9c800000, compatible id shared-dma-pool
[    0.000000] OF: reserved mem: 0x000000009c800000..0x000000009c8fffff (1024 KiB) nomap non-reusable r5f-dma-memory@9c800000
[    0.000000] Reserved memory: created DMA memory pool at 0x000000009c900000, size 30 MiB
[    0.000000] OF: reserved mem: initialized node r5f-dma-memory@9c900000, compatible id shared-dma-pool
[    0.000000] OF: reserved mem: 0x000000009c900000..0x000000009e6fffff (30720 KiB) nomap non-reusable r5f-dma-memory@9c900000
[    0.000000] OF: reserved mem: 0x000000009e800000..0x000000009fffffff (24576 KiB) nomap non-reusable optee@9e800000
[    0.000000] OF: reserved mem: 0x00000000a0000000..0x00000000a0ffffff (16384 KiB) nomap non-reusable edgeai-rtos-ipc-memory-region
[    0.000000] Reserved memory: created DMA memory pool at 0x00000000a1000000, size 32 MiB
[    0.000000] OF: reserved mem: initialized node edgeai-dma-memory@a1000000, compatible id shared-dma-pool
[    0.000000] OF: reserved mem: 0x00000000a1000000..0x00000000a2ffffff (32768 KiB) nomap non-reusable edgeai-dma-memory@a1000000
[    0.000000] OF: reserved mem: initialized node edgeai_shared-memories, compatible id dma-heap-carveout
[    0.000000] OF: reserved mem: 0x00000000a3000000..0x00000000adffffff (180224 KiB) map non-reusable edgeai_shared-memories
[    0.000000] Reserved memory: created DMA memory pool at 0x00000000ae000000, size 288 MiB
[    0.000000] OF: reserved mem: initialized node edgeai-core-heap-memory@ae000000, compatible id shared-dma-pool
[    0.000000] OF: reserved mem: 0x00000000ae000000..0x00000000bfffffff (294912 KiB) nomap non-reusable edgeai-core-heap-memory@ae000000
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000080000000-0x00000000ffffffff]
[    0.000000]   DMA32    empty
[    0.000000]   Normal   [mem 0x0000000100000000-0x00000008ffffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000080000000-0x000000008007ffff]
[    0.000000]   node   0: [mem 0x0000000080080000-0x00000000997fffff]
[    0.000000]   node   0: [mem 0x0000000099800000-0x000000009e6fffff]
[    0.000000]   node   0: [mem 0x000000009e700000-0x000000009e7fffff]
[    0.000000]   node   0: [mem 0x000000009e800000-0x00000000a2ffffff]
[    0.000000]   node   0: [mem 0x00000000a3000000-0x00000000adffffff]
[    0.000000]   node   0: [mem 0x00000000ae000000-0x00000000bfffffff]
[    0.000000]   node   0: [mem 0x00000000c0000000-0x00000000ffffffff]
[    0.000000]   node   0: [mem 0x0000000880000000-0x00000008ffffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000080000000-0x00000008ffffffff]
[    0.000000] psci: probing for conduit method from DT.
[    0.000000] psci: PSCIv1.1 detected in firmware.
[    0.000000] psci: Using standard PSCI v0.2 function IDs
[    0.000000] psci: Trusted OS migration not required
[    0.000000] psci: SMC Calling Convention v1.4
[    0.000000] percpu: Embedded 20 pages/cpu s43496 r8192 d30232 u81920
[    0.000000] pcpu-alloc: s43496 r8192 d30232 u81920 alloc=20*4096
[    0.000000] pcpu-alloc: [0] 0 [0] 1 [0] 2 [0] 3 
[    0.000000] Detected VIPT I-cache on CPU0
[    0.000000] CPU features: detected: GIC system register CPU interface
[    0.000000] CPU features: detected: ARM erratum 845719
[    0.000000] alternatives: applying boot alternatives
[    0.000000] Kernel command line: console=ttyS2,115200n8 earlycon=ns16550a,mmio32,0x02800000 mtdparts=spi-nand0:512k(ospi_nand.tiboot3),2m(ospi_nand.tispl),4m(ospi_nand.u-boot),256k(ospi_nand.env),256k(ospi_nand.env.backup),98048k@32m(ospi_nand.rootfs),256k@130816k(ospi_nand.phypattern) root=PARTUUID=076c4a2a-02 rw rootfstype=ext4 rootwait
[    0.000000] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes, linear)
[    0.000000] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 1032192
[    0.000000] mem auto-init: stack:all(zero), heap alloc:off, heap free:off
[    0.000000] software IO TLB: area num 4.
[    0.000000] software IO TLB: mapped [mem 0x00000000fbfff000-0x00000000fffff000] (64MB)
[    0.000000] Memory: 2806884K/4194304K available (12672K kernel code, 1234K rwdata, 4304K rodata, 2432K init, 503K bss, 797596K reserved, 589824K cma-reserved)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
[    0.000000] rcu: Preemptible hierarchical RCU implementation.
[    0.000000] rcu: 	RCU event tracing is enabled.
[    0.000000] rcu: 	RCU restricting CPUs from NR_CPUS=256 to nr_cpu_ids=4.
[    0.000000] 	Trampoline variant of Tasks RCU enabled.
[    0.000000] 	Tracing variant of Tasks RCU enabled.
[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
[    0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4
[    0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
[    0.000000] GICv3: GIC: Using split EOI/Deactivate mode
[    0.000000] GICv3: 256 SPIs implemented
[    0.000000] GICv3: 0 Extended SPIs implemented
[    0.000000] Root IRQ handler: gic_handle_irq
[    0.000000] GICv3: GICv3 features: 16 PPIs
[    0.000000] GICv3: CPU0: found redistributor 0 region 0:0x0000000001880000
[    0.000000] ITS [mem 0x01820000-0x0182ffff]
[    0.000000] GIC: enabling workaround for ITS: Socionext Synquacer pre-ITS
[    0.000000] ITS@0x0000000001820000: Devices Table too large, reduce ids 20->19
[    0.000000] ITS@0x0000000001820000: allocated 524288 Devices @880800000 (flat, esz 8, psz 64K, shr 0)
[    0.000000] ITS: using cache flushing for cmd queue
[    0.000000] GICv3: using LPI property table @0x0000000880040000
[    0.000000] GIC: using cache flushing for LPI property table
[    0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000880050000
[    0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.
[    0.000000] arch_timer: cp15 timer(s) running at 200.00MHz (phys).
[    0.000000] clocksource: arch_sys_counter: mask: 0x3ffffffffffffff max_cycles: 0x2e2049d3e8, max_idle_ns: 440795210634 ns
[    0.000000] sched_clock: 58 bits at 200MHz, resolution 5ns, wraps every 4398046511102ns
[    0.000336] Console: colour dummy device 80x25
[    0.000385] Calibrating delay loop (skipped), value calculated using timer frequency.. 400.00 BogoMIPS (lpj=800000)
[    0.000397] pid_max: default: 32768 minimum: 301
[    0.000474] LSM: initializing lsm=capability,integrity
[    0.000574] Mount-cache hash table entries: 8192 (order: 4, 65536 bytes, linear)
[    0.000593] Mountpoint-cache hash table entries: 8192 (order: 4, 65536 bytes, linear)
[    0.002219] RCU Tasks: Setting shift to 2 and lim to 1 rcu_task_cb_adjust=1.
[    0.002291] RCU Tasks Trace: Setting shift to 2 and lim to 1 rcu_task_cb_adjust=1.
[    0.002464] rcu: Hierarchical SRCU implementation.
[    0.002469] rcu: 	Max phase no-delay instances is 1000.
[    0.002713] Platform MSI: msi-controller@1820000 domain created
[    0.002916] PCI/MSI: /bus@f0000/interrupt-controller@1800000/msi-controller@1820000 domain created
[    0.003175] EFI services will not be available.
[    0.003399] smp: Bringing up secondary CPUs ...
[    0.003984] Detected VIPT I-cache on CPU1
[    0.004055] GICv3: CPU1: found redistributor 1 region 0:0x00000000018a0000
[    0.004070] GICv3: CPU1: using allocated LPI pending table @0x0000000880060000
[    0.004116] CPU1: Booted secondary processor 0x0000000001 [0x410fd034]
[    0.004787] Detected VIPT I-cache on CPU2
[    0.004838] GICv3: CPU2: found redistributor 2 region 0:0x00000000018c0000
[    0.004852] GICv3: CPU2: using allocated LPI pending table @0x0000000880070000
[    0.004883] CPU2: Booted secondary processor 0x0000000002 [0x410fd034]
[    0.005457] Detected VIPT I-cache on CPU3
[    0.005504] GICv3: CPU3: found redistributor 3 region 0:0x00000000018e0000
[    0.005516] GICv3: CPU3: using allocated LPI pending table @0x0000000880080000
[    0.005545] CPU3: Booted secondary processor 0x0000000003 [0x410fd034]
[    0.005613] smp: Brought up 1 node, 4 CPUs
[    0.005629] SMP: Total of 4 processors activated.
[    0.005634] CPU features: detected: 32-bit EL0 Support
[    0.005639] CPU features: detected: CRC32 instructions
[    0.005696] CPU: All CPU(s) started at EL2
[    0.005700] alternatives: applying system-wide alternatives
[    0.007222] devtmpfs: initialized
[    0.016214] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[    0.016240] futex hash table entries: 1024 (order: 4, 65536 bytes, linear)
[    0.029887] pinctrl core: initialized pinctrl subsystem
[    0.030333] DMI not present or invalid.
[    0.030927] NET: Registered PF_NETLINK/PF_ROUTE protocol family
[    0.031682] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations
[    0.031799] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations
[    0.031951] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations
[    0.032013] audit: initializing netlink subsys (disabled)
[    0.032187] audit: type=2000 audit(0.032:1): state=initialized audit_enabled=0 res=1
[    0.032615] thermal_sys: Registered thermal governor 'step_wise'
[    0.032622] thermal_sys: Registered thermal governor 'power_allocator'
[    0.032663] cpuidle: using governor menu
[    0.032837] hw-breakpoint: found 6 breakpoint and 4 watchpoint registers.
[    0.032918] ASID allocator initialised with 65536 entries
[    0.045143] platform a40000.pinctrl: Fixed dependency cycle(s) with /bus@f0000/pinctrl@a40000/cpsw-cpts
[    0.047060] platform 30200000.dss: Fixed dependency cycle(s) with /bus@f0000/i2c@20020000/bridge-lvds@29
[    0.048927] platform panel-lvds: Fixed dependency cycle(s) with /bus@f0000/i2c@20020000/bridge-lvds@29
[    0.049622] Modules: 27440 pages in range for non-PLT usage
[    0.049631] Modules: 518960 pages in range for PLT usage
[    0.050312] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages
[    0.050323] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page
[    0.050329] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages
[    0.050333] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page
[    0.050337] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages
[    0.050341] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page
[    0.050346] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages
[    0.050350] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page
[    0.051614] k3-chipinfo 43000014.chipid: Family:AM62AX rev:SR1.0 JTAGID[0x0bb8d02f] Detected
[    0.052277] iommu: Default domain type: Translated
[    0.052288] iommu: DMA domain TLB invalidation policy: strict mode
[    0.052552] SCSI subsystem initialized
[    0.052666] libata version 3.00 loaded.
[    0.052829] usbcore: registered new interface driver usbfs
[    0.052856] usbcore: registered new interface driver hub
[    0.052890] usbcore: registered new device driver usb
[    0.053327] mc: Linux media interface: v0.10
[    0.053369] videodev: Linux video capture interface: v2.00
[    0.053414] pps_core: LinuxPPS API ver. 1 registered
[    0.053419] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[    0.053435] PTP clock support registered
[    0.053599] EDAC MC: Ver: 3.0.0
[    0.054083] scmi_core: SCMI protocol bus registered
[    0.054329] FPGA manager framework
[    0.054434] Advanced Linux Sound Architecture Driver Initialized.
[    0.055288] vgaarb: loaded
[    0.055591] clocksource: Switched to clocksource arch_sys_counter
[    0.055800] VFS: Disk quotas dquot_6.6.0
[    0.055829] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    0.061912] Carveout Heap: Exported 176 MiB at 0x00000000a3000000
[    0.062017] NET: Registered PF_INET protocol family
[    0.062201] IP idents hash table entries: 65536 (order: 7, 524288 bytes, linear)
[    0.065097] tcp_listen_portaddr_hash hash table entries: 2048 (order: 3, 32768 bytes, linear)
[    0.065143] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)
[    0.065194] TCP established hash table entries: 32768 (order: 6, 262144 bytes, linear)
[    0.065410] TCP bind hash table entries: 32768 (order: 8, 1048576 bytes, linear)
[    0.066220] TCP: Hash tables configured (established 32768 bind 32768)
[    0.066352] UDP hash table entries: 2048 (order: 4, 65536 bytes, linear)
[    0.066440] UDP-Lite hash table entries: 2048 (order: 4, 65536 bytes, linear)
[    0.066669] NET: Registered PF_UNIX/PF_LOCAL protocol family
[    0.067086] RPC: Registered named UNIX socket transport module.
[    0.067097] RPC: Registered udp transport module.
[    0.067101] RPC: Registered tcp transport module.
[    0.067104] RPC: Registered tcp-with-tls transport module.
[    0.067108] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    0.067116] NET: Registered PF_XDP protocol family
[    0.067135] PCI: CLS 0 bytes, default 64
[    0.068466] Initialise system trusted keyrings
[    0.068664] workingset: timestamp_bits=46 max_order=20 bucket_order=0
[    0.068948] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    0.069178] NFS: Registering the id_resolver key type
[    0.069201] Key type id_resolver registered
[    0.069206] Key type id_legacy registered
[    0.069225] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
[    0.069231] nfs4flexfilelayout_init: NFSv4 Flexfile Layout Driver Registering...
[    0.102906] Key type asymmetric registered
[    0.102916] Asymmetric key parser 'x509' registered
[    0.102964] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 243)
[    0.103103] io scheduler mq-deadline registered
[    0.103110] io scheduler kyber registered
[    0.103137] io scheduler bfq registered
[    0.106133] pinctrl-single 4084000.pinctrl: 34 pins, size 136
[    0.106822] pinctrl-single f4000.pinctrl: 151 pins, size 604
[    0.108755] pinctrl-single a40000.pinctrl: 512 pins, size 2048
[    0.114480] Serial: 8250/16550 driver, 12 ports, IRQ sharing enabled
[    0.127892] loop: module loaded
[    0.128871] megasas: 07.725.01.00-rc1
[    0.131298] tun: Universal TUN/TAP device driver, 1.6
[    0.132316] VFIO - User Level meta-driver version: 0.3
[    0.133485] usbcore: registered new interface driver usb-storage
[    0.133870] i2c_dev: i2c /dev entries driver
[    0.134032] IR NEC protocol handler initialized
[    0.135529] sdhci: Secure Digital Host Controller Interface driver
[    0.135540] sdhci: Copyright(c) Pierre Ossman
[    0.135812] sdhci-pltfm: SDHCI platform and OF driver helper
[    0.136581] ledtrig-cpu: registered to indicate activity on CPUs
[    0.136878] SMCCC: SOC_ID: ARCH_SOC_ID not implemented, skipping ....
[    0.137551] usbcore: registered new interface driver usbhid
[    0.137557] usbhid: USB HID core driver
[    0.137843] omap-mailbox 29000000.mailbox: omap mailbox rev 0x66fca100
[    0.137984] omap-mailbox 29010000.mailbox: omap mailbox rev 0x66fca100
[    0.138093] omap-mailbox 29020000.mailbox: omap mailbox rev 0x66fca100
[    0.138151] omap-mailbox 29030000.mailbox: no available mbox devices found
[    0.146519] hw perfevents: enabled with armv8_cortex_a53 PMU driver, 7 counters available
[    0.147178] optee: probing for conduit method.
[    0.147206] optee: revision 4.2 (12d7c4ee)
[    0.147478] optee: dynamic shared memory is enabled
[    0.148177] random: crng init done
[    0.148261] optee: initialized driver
[    0.149827] Initializing XFRM netlink socket
[    0.149862] NET: Registered PF_PACKET protocol family
[    0.149924] Key type dns_resolver registered
[    0.157832] registered taskstats version 1
[    0.157986] Loading compiled-in X.509 certificates
[    0.171298] ti-sci 44043000.system-controller: ABI: 4.0 (firmware rev 0x000a '10.0.8--v10.00.08 (Fiery Fox)')
[    0.209097] i2c 0-0048: Fixed dependency cycle(s) with /bus@f0000/i2c@20000000/pmic@48/regulators/buck5
[    0.209288] omap_i2c 20000000.i2c: bus 0 rev0.12 at 400 kHz
[    0.210471] pca953x 1-0023: supply vcc not found, using dummy regulator
[    0.210599] pca953x 1-0023: using AI
[    0.231727] pca953x 1-0023: failed writing register
[    0.236840] pca953x: probe of 1-0023 failed with error -121
[    0.236932] omap_i2c 20010000.i2c: bus 1 rev0.12 at 100 kHz
[    0.237878] i2c 2-002a: Fixed dependency cycle(s) with /bus@f0000/ticsi2rx@30102000/csi-bridge@30101000
[    0.238102] platform panel-lvds: Fixed dependency cycle(s) with /bus@f0000/i2c@20020000/bridge-lvds@29
[    0.238168] platform 30200000.dss: Fixed dependency cycle(s) with /bus@f0000/i2c@20020000/bridge-lvds@29
[    0.238224] i2c 2-0029: Fixed dependency cycle(s) with /panel-lvds
[    0.238273] i2c 2-0029: Fixed dependency cycle(s) with /bus@f0000/dss@30200000
[    0.238526] omap_i2c 20020000.i2c: bus 2 rev0.12 at 400 kHz
[    0.238842] ti-sci-intr 4210000.interrupt-controller: Interrupt Router 5 domain created
[    0.238988] ti-sci-intr bus@f0000:interrupt-controller@a00000: Interrupt Router 3 domain created
[    0.239208] ti-sci-inta 48000000.interrupt-controller: Interrupt Aggregator domain 28 created
[    0.239461] ti-sci-inta 4e0a0000.interrupt-controller: Interrupt Aggregator domain 200 created
[    0.240215] ti-udma 485c0100.dma-controller: Number of rings: 82
[    0.242707] ti-udma 485c0100.dma-controller: Channels: 48 (bchan: 18, tchan: 12, rchan: 18)
[    0.245194] ti-udma 485c0000.dma-controller: Number of rings: 150
[    0.249846] ti-udma 485c0000.dma-controller: Channels: 35 (tchan: 20, rchan: 15)
[    0.251748] ti-udma 4e230000.dma-controller: Number of rings: 6
[    0.252290] ti-udma 4e230000.dma-controller: Channels: 6 (bchan: 0, tchan: 0, rchan: 6)
[    0.253407] printk: console [ttyS2] disabled
[    0.253700] 2800000.serial: ttyS2 at MMIO 0x2800000 (irq = 252, base_baud = 3000000) is a 8250
[    0.253751] printk: console [ttyS2] enabled
[    0.253757] printk: bootconsole [ns16550a0] disabled
[    0.254978] 2820000.serial: ttyS4 at MMIO 0x2820000 (irq = 254, base_baud = 3000000) is a 8250
[    0.658424] spi-nand: probe of spi0.0 failed with error -110
[    0.699606] davinci_mdio 8000f00.mdio: davinci mdio revision 9.7, bus freq 1000000
[    0.701000] am65-cpsw-nuss 8000000.ethernet: initializing am65 cpsw nuss version 0x6BA01103, cpsw version 0x6BA81103 Ports: 3 quirks:00000006
[    0.701133] am65-cpsw-nuss 8000000.ethernet: initialized cpsw ale version 1.5
[    0.701141] am65-cpsw-nuss 8000000.ethernet: ALE Table size 512
[    0.701584] pps pps0: new PPS source ptp0
[    0.701829] am65-cpsw-nuss 8000000.ethernet: CPTS ver 0x4e8a010c, freq:500000000, add_val:1 pps:1
[    0.706904] am65-cpsw-nuss 8000000.ethernet: set new flow-id-base 19
[    0.710800] xhci-hcd xhci-hcd.0.auto: xHCI Host Controller
[    0.710828] xhci-hcd xhci-hcd.0.auto: new USB bus registered, assigned bus number 1
[    0.710998] xhci-hcd xhci-hcd.0.auto: USB3 root hub has no ports
[    0.711006] xhci-hcd xhci-hcd.0.auto: hcc params 0x0258fe6d hci version 0x110 quirks 0x0000008020000010
[    0.711055] xhci-hcd xhci-hcd.0.auto: irq 267, io mem 0x31000000
[    0.711838] hub 1-0:1.0: USB hub found
[    0.711878] hub 1-0:1.0: 1 port detected
[    0.713978] xhci-hcd xhci-hcd.1.auto: xHCI Host Controller
[    0.714005] xhci-hcd xhci-hcd.1.auto: new USB bus registered, assigned bus number 2
[    0.714129] xhci-hcd xhci-hcd.1.auto: USB3 root hub has no ports
[    0.714138] xhci-hcd xhci-hcd.1.auto: hcc params 0x0258fe6d hci version 0x110 quirks 0x0000008020000010
[    0.714186] xhci-hcd xhci-hcd.1.auto: irq 268, io mem 0x31100000
[    0.714915] hub 2-0:1.0: USB hub found
[    0.714953] hub 2-0:1.0: 1 port detected
[    0.717010] rtc-ti-k3 2b1f0000.rtc: registered as rtc0
[    0.717043] rtc-ti-k3 2b1f0000.rtc: setting system clock to 1970-01-01T00:00:09 UTC (9)
[    0.718263] cpufreq: cpufreq_online: CPU0: Running at unlisted initial frequency: 1200000 KHz, changing to: 1250000 KHz
[    0.720008] mmc0: CQHCI version 5.10
[    0.720150] mmc1: CQHCI version 5.10
[    0.755612] mmc1: SDHCI controller on fa00000.mmc [fa00000.mmc] using ADMA 64-bit
[    0.756151] mmc0: SDHCI controller on fa10000.mmc [fa10000.mmc] using ADMA 64-bit
[    0.788778] mmc1: new high speed SDXC card at address 5048
[    0.789344] mmcblk1: mmc1:5048 METOR 117 GiB
[    0.791348]  mmcblk1: p1 p2
[    0.907662] mmc0: switch to bus width 8 failed
[    0.909029] mmc0: unexpected status 0x800900 after switch
[    0.919762] mmc0: mmc_select_hs200 failed, error -110
[    0.924825] mmc0: error -110 whilst initialising MMC card
[    0.999575] tps6594-rtc tps6594-rtc.7.auto: registered as rtc1
[    0.999739] pinctrl-single f4000.pinctrl: Invalid number of rows: 0
[    1.006008] pinctrl-single f4000.pinctrl: no pins entries for main-gpio1-ioexp-intr-default-pins
[    1.014908] max96714_probe start
[    1.014953] max96714 2-002a: max96714_get_hw_resources success[0]
[    1.044273] mmc0: Command Queue Engine enabled
[    1.044289] mmc0: new HS200 MMC card at address 0001
[    1.044779] mmcblk0: mmc0:0001 8GUF4R 7.28 GiB
[    1.045846]  mmcblk0: p1
[    1.046202] mmcblk0boot0: mmc0:0001 8GUF4R 31.9 MiB
[    1.046893] mmcblk0boot1: mmc0:0001 8GUF4R 31.9 MiB
[    1.047532] mmcblk0rpmb: mmc0:0001 8GUF4R 4.00 MiB, chardev (238:0)
[    1.059595]  ------------------------------------- 
                
[    1.083746] twl max96714 read device id [c9] 
[    1.084065] max96714 2-002a: Cur Link speed :2
[    1.084072] max96714 2-002a: Reset Link speed :1, reg[1]
[    1.084355] max96714 2-002a: Now Link speed :1
[    1.084389] link freq[500000000]
[    1.084393] twl max96714_parse_dt_txport num_data_lanes[4]
[    1.085621] max96714 2-002a: Deserializer: max96714
[    1.085625] twl max96714_log_status 
[    1.085784] max96714 2-002a: Link locked:0
[    1.085945] twl max96714_link_status lock: 0 
[    1.086104] max96714 2-002a: Pipe vidlock:0
[    1.086264] twl max96714_pipe_status Pipe vidlock: 0 
[    1.086578] max96714 2-002a: CSI controller DPLL freq:1000MHz CSIPHY enabled:0
[    1.086739] twl max96714_csi_status CSI controller DPLL freq:1000MHz CSIPHY enabled:0
[    1.086745] max96714 2-002a: Error invalid reg-width 0 for reg 0x0010
[    1.307710] max96714_probe success
[    1.307717] flork max96714_kthread start 
[    1.307924] [Flork]:96714 GMSL2 link state change! [0] -> [1]
[    1.307959] lt9211 2-0029: #####111 find panel error: -517
[    1.308126] [Flork]:96714 Video pipe link state change! [0] -> [1]
[    1.308325] [Flork]:96717 0X000D = 0xc8!
[    1.308329] [Flork] ERROR :96717 DEV ID incorrect!
[    1.308642] panel-lvds panel-lvds: /panel-lvds: problems parsing panel-timing (-2)
[    1.316265] panel-lvds: probe of panel-lvds failed with error -2
[    1.316372] twl 2025-03-04 panel_simple_probe 
[    1.316376] twl test panel 
[    1.355597] davinci_mdio 8000f00.mdio: davinci mdio revision 9.7, bus freq 1000000
[    1.356688] davinci_mdio 8000f00.mdio: phy[0]: device 8000f00.mdio:00, driver unknown
[    1.357175] Registered IR keymap rc-nec-terratec-cinergy-xs
[    1.357255] rc rc0: gpio_ir_recv as /devices/platform/gpio-ir-receiver/rc/rc0
[    1.357361] input: gpio_ir_recv as /devices/platform/gpio-ir-receiver/rc/rc0/input0
[    1.358350] lt9211 2-0029: panel found
[    1.358366] lt9211 2-0029: bridge found
[    1.503591] twl reset chip
[    1.525130] lt9211 2-0029: Chip ID: 0x21 0x03 0xe1
[    1.526477] LT9211C chip id:0x21 0x03 0xe1
[    1.526483] lt9211 2-0029: ########################### 2
[    1.526489] lt9211 2-0029: DRM bridge lt9211 probed
[    1.526603] twl LT9211C_Main start 
[    1.528791] [drm] Initialized tidss 1.0.0 20180215 for 30200000.dss on minor 0
[    1.530517] tidss 30200000.dss: [drm] fb0: tidssdrmfb frame buffer device
[    1.533272] debugfs: Directory 'pd:182' with parent 'pm_genpd' already present!
[    1.540619] debugfs: Directory 'pd:182' with parent 'pm_genpd' already present!
[    1.548720] clk: Disabling unused clocks
[    1.552915] ALSA device list:
[    1.552923]   No soundcards found.
[    1.631615] RxState = 0x02
[    1.631634] TxState = 0x02
[    1.636633] RxState = 0x03
[    1.642329] TTLRx Input Mode:    RGB888
[    1.642338] RxState = 0x04
[    1.825949] EXT4-fs (mmcblk1p2): recovery complete
[    1.835152] EXT4-fs (mmcblk1p2): mounted filesystem 391702fc-cf91-4f81-afcd-ec5ecaef14e3 r/w with ordered data mode. Quota mode: none.
[    1.835241] VFS: Mounted root (ext4 filesystem) on device 179:2.
[    1.835848] devtmpfs: mounted
[    1.836475] Freeing unused kernel memory: 2432K
[    1.836558] Run /sbin/init as init process
[    1.836561]   with arguments:
[    1.836564]     /sbin/init
[    1.836567]   with environment:
[    1.836569]     HOME=/
[    1.836572]     TERM=linux
[    2.048090] Rx Pll Cal done and Lock
[    2.049005] vmax, vmin = 0x10 0x00
[    2.049464] vcoset = 0x08
[    2.051535] uci fit = 0x08
[    2.051538] ucihact, ucivact = 1920 720
[    2.058863] self timing set:
[    2.058872] hfp, hs, hbp, hact, htotal = 72 12 64 1920 2068
[    2.058879] vfp, vs, vbp, vact, vtotal = 15 6 18 720 759
[    2.164075] TTL input sync_polarity = 0x0f
[    2.165826] sync mode enable, generate de
[    2.166693] RxState = 0x05
[    2.184069] FM CLOCK DET Stable
[    2.186680] TTL input dclock: 92997
[    2.186687] ==============
[    2.186690] ==============
[    2.186693] ==============
[    2.300085] ************video check phase: 0x09
[    2.300104] sync_polarity = 0x0f
[    2.300108] hfp, hs, hbp, hact, htotal = 72 12 64 1920 2068
[    2.300114] vfp, vs, vbp, vact, vtotal = 15 6 18 720 759
[    2.300120] RxState = 0x06
[    2.301813] TxState = 0x03
[    2.316077] FM CLOCK DET Stable
[    2.321936] temp is: 0x02
[    2.360497] Tx Pll Lock
[    2.360510] TxState = 0x04
[    2.362755] systemd[1]: System time before build time, advancing clock.
[    2.450073] NET: Registered PF_INET6 protocol family
[    2.450905] Segment Routing with IPv6
[    2.450929] In-situ OAM (IOAM) with IPv6
[    2.508842] systemd[1]: systemd 255.4^ running in system mode (+PAM -AUDIT -SELINUX -APPARMOR +IMA -SMACK +SECCOMP -GCRYPT -GNUTLS -OPENSSL +ACL +BLKID -CURL -ELFUTILS -FIDO2 -IDN2 -IDN -IPTC +KMOD -LIBCRYPTSETUP +LIBFDISK -PCRE2 -PWQUALITY -P11KIT -QRENCODE -TPM2 -BZIP2 -LZ4 -XZ -ZLIB +ZSTD -BPF_FRAMEWORK -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
[    2.508875] systemd[1]: Detected architecture arm64.
[    2.537610] systemd[1]: Hostname set to <am62axx-evm>.
[    2.637190] systemd-sysv-generator[116]: SysV service '/etc/init.d/edgeai-launcher.sh' lacks a native systemd unit file. ~ Automatically generating a unit file for compatibility. Please update package to include a native systemd unit file, in order to make it safe, robust and future-proof. ! This compatibility logic is deprecated, expect removal soon. !
[    2.849274] systemd[1]: /usr/lib/systemd/system/bt-enable.service:9: Standard output type syslog is obsolete, automatically updating to journal. Please update your unit file, and consider removing the setting altogether.
[    2.943006] systemd[1]: /etc/systemd/system/sync-clocks.service:11: Standard output type syslog is obsolete, automatically updating to journal. Please update your unit file, and consider removing the setting altogether.
[    3.012592] systemd[1]: Queued start job for default target Graphical Interface.
[    3.044979] systemd[1]: Created slice Slice /system/getty.
[    3.061834] systemd[1]: Created slice Slice /system/modprobe.
[    3.081764] systemd[1]: Created slice Slice /system/serial-getty.
[    3.097224] systemd[1]: Created slice User and Session Slice.
[    3.111997] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.
[    3.127895] systemd[1]: Started Forward Password Requests to Wall Directory Watch.
[    3.143763] systemd[1]: Expecting device /dev/ttyS2...
[    3.155799] systemd[1]: Reached target Path Units.
[    3.167718] systemd[1]: Reached target Remote File Systems.
[    3.183700] systemd[1]: Reached target Slice Units.
[    3.199800] systemd[1]: Reached target Swaps.
[    3.247379] systemd[1]: Listening on RPCbind Server Activation Socket.
[    3.263918] systemd[1]: Reached target RPC Port Mapper.
[    3.286692] systemd[1]: Listening on Process Core Dump Socket.
[    3.304082] systemd[1]: Listening on initctl Compatibility Named Pipe.
[    3.320698] systemd[1]: Listening on Journal Audit Socket.
[    3.336316] systemd[1]: Listening on Journal Socket (/dev/log).
[    3.352316] systemd[1]: Listening on Journal Socket.
[    3.364469] systemd[1]: Listening on Network Service Netlink Socket.
[    3.382653] systemd[1]: Listening on udev Control Socket.
[    3.400255] systemd[1]: Listening on udev Kernel Socket.
[    3.416278] systemd[1]: Listening on User Database Manager Socket.
[    3.422192] LVDS Output PortA&B
[    3.430607] LVDS MODE : SYNC
[    3.432306] Data Format: VESA
[    3.432314] ColorSpace: RGB
[    3.432317] ColorDepth: 8Bit
[    3.448056] LVDS Tx Video Out
[    3.448076] TxState = 0x05
[    3.448083] LT9211C_Main tx rx all ready. 
[    3.448086] LT9211C_Main exit. 
[    3.459938] systemd[1]: Mounting Huge Pages File System...
[    3.475845] systemd[1]: Mounting POSIX Message Queue File System...
[    3.495754] systemd[1]: Mounting Kernel Debug File System...
[    3.508247] systemd[1]: Kernel Trace File System was skipped because of an unmet condition check (ConditionPathExists=/sys/kernel/tracing).
[    3.513808] systemd[1]: Mounting Temporary Directory /tmp...
[    3.528991] systemd[1]: Starting Create List of Static Device Nodes...
[    3.547991] systemd[1]: Starting Load Kernel Module configfs...
[    3.563922] systemd[1]: Starting Load Kernel Module drm...
[    3.583955] systemd[1]: Starting Load Kernel Module fuse...
[    3.605393] systemd[1]: Starting Start psplash boot splash screen...
[    3.628481] systemd[1]: Starting RPC Bind...
[    3.628618] fuse: init (API version 7.39)
[    3.644112] systemd[1]: File System Check on Root Device was skipped because of an unmet condition check (ConditionPathIsReadWrite=!/).
[    3.651364] systemd[1]: Starting Journal Service...
[    3.699146] systemd[1]: Starting Load Kernel Modules...
[    3.716515] systemd[1]: Starting Generate network units from Kernel command line...
[    3.738357] systemd[1]: Starting Remount Root and Kernel File Systems...
[    3.746282] systemd-journald[134]: Collecting audit messages is enabled.
[    3.760451] systemd[1]: Starting Coldplug All udev Devices...
[    3.783077] systemd[1]: Started Start psplash boot splash screen.
[    3.805622] systemd[1]: Started RPC Bind.
[    3.820544] systemd[1]: Mounted Huge Pages File System.
[    3.828557] EXT4-fs (mmcblk1p2): re-mounted 391702fc-cf91-4f81-afcd-ec5ecaef14e3 r/w. Quota mode: none.
[    3.841890] systemd[1]: Started Journal Service.
[    4.215456] systemd-journald[134]: Received client request to flush runtime journal.
[    4.510115] audit: type=1334 audit(1709054766.144:2): prog-id=6 op=LOAD
[    4.510289] audit: type=1334 audit(1709054766.144:3): prog-id=7 op=LOAD
[    4.721094] audit: type=1334 audit(1709054766.356:4): prog-id=8 op=LOAD
[    4.767242] audit: type=1334 audit(1709054766.400:5): prog-id=9 op=LOAD
[    4.844797] audit: type=1334 audit(1709054766.480:6): prog-id=10 op=LOAD
[    4.845236] audit: type=1334 audit(1709054766.480:7): prog-id=11 op=LOAD
[    4.845492] audit: type=1334 audit(1709054766.480:8): prog-id=12 op=LOAD
[    5.526106] CAN device driver interface
[    5.558028] m_can_platform 20701000.can: m_can device registered (irq=541, version=32)
[    5.643033] tas2505-codec 2-0018: tas2505_i2c_probe: clock-source = BCLK
[    5.689577] k3-dsp-rproc 7e000000.dsp: assigned reserved memory node c7x-dma-memory@99800000
[    5.691446] k3-dsp-rproc 7e000000.dsp: configured DSP for IPC-only mode
[    5.692149] vdec 30210000.video-codec: error -ENXIO: IRQ index 0 not found
[    5.697749] remoteproc remoteproc0: 7e000000.dsp is available
[    5.699120] remoteproc remoteproc0: attaching to 7e000000.dsp
[    5.699319] vdec 30210000.video-codec: failed to get irq resource, falling back to polling
[    5.735575] audit: type=1334 audit(1761700852.745:9): prog-id=13 op=LOAD
[    5.738243] vdec 30210000.video-codec: Added wave5 driver with caps: 'ENCODE' 'DECODE'
[    5.738267] vdec 30210000.video-codec: Product Code:      0x521c
[    5.738275] vdec 30210000.video-codec: Firmware Revision: 320127
[    5.745693] platform 79000000.r5f: configured R5F for IPC-only mode
[    5.745854] platform 79000000.r5f: assigned reserved memory node r5f-dma-memory@9b800000
[    5.746415] remoteproc remoteproc1: 79000000.r5f is available
[    5.746504] remoteproc remoteproc1: attaching to 79000000.r5f
[    5.754699] k3-dsp-rproc 7e000000.dsp: DSP initialized in IPC-only mode
[    5.754741] rproc-virtio rproc-virtio.8.auto: assigned reserved memory node c7x-dma-memory@99800000
[    5.755358] virtio_rpmsg_bus virtio0: rpmsg host is online
[    5.755411] rproc-virtio rproc-virtio.8.auto: registered virtio0 (type 7)
[    5.755422] remoteproc remoteproc0: remote processor 7e000000.dsp is now attached
[    5.756217] virtio_rpmsg_bus virtio0: creating channel rpmsg_chrdev addr 0xd
[    5.766787] platform 79000000.r5f: R5F core initialized in IPC-only mode
[    5.768640] rproc-virtio rproc-virtio.9.auto: assigned reserved memory node r5f-dma-memory@9b800000
[    5.769357] virtio_rpmsg_bus virtio1: rpmsg host is online
[    5.769415] virtio_rpmsg_bus virtio1: creating channel ti.ipc4.ping-pong addr 0xd
[    5.769425] rproc-virtio rproc-virtio.9.auto: registered virtio1 (type 7)
[    5.769435] remoteproc remoteproc1: remote processor 79000000.r5f is now attached
[    5.773610] virtio_rpmsg_bus virtio1: creating channel rpmsg_chrdev addr 0xe
[    5.905485] platform 78000000.r5f: R5F core may have been powered on by a different host, programmed state (0) != actual state (1)
[    5.906462] platform 78000000.r5f: configured R5F for IPC-only mode
[    5.906621] platform 78000000.r5f: assigned reserved memory node r5f-dma-memory@9c800000
[    5.909658] remoteproc remoteproc2: 78000000.r5f is available
[    5.909748] remoteproc remoteproc2: attaching to 78000000.r5f
[    5.919009] platform 78000000.r5f: R5F core initialized in IPC-only mode
[    5.919051] rproc-virtio rproc-virtio.10.auto: assigned reserved memory node r5f-dma-memory@9c800000
[    5.919735] virtio_rpmsg_bus virtio2: rpmsg host is online
[    5.919778] rproc-virtio rproc-virtio.10.auto: registered virtio2 (type 7)
[    5.919787] remoteproc remoteproc2: remote processor 78000000.r5f is now attached
[    5.920389] virtio_rpmsg_bus virtio2: creating channel rpmsg_chrdev addr 0xd
[    5.920571] virtio_rpmsg_bus virtio2: creating channel rpmsg_chrdev addr 0x15
[    5.920695] virtio_rpmsg_bus virtio0: creating channel rpmsg_chrdev addr 0x15
[    5.923217] virtio_rpmsg_bus virtio0: creating channel ti.ipc4.ping-pong addr 0xe
[    5.928694] virtio_rpmsg_bus virtio2: creating channel ti.ipc4.ping-pong addr 0xe
[    5.928901] virtio_rpmsg_bus virtio2: msg received with no recipient
[    5.929125] virtio_rpmsg_bus virtio0: msg received with no recipient
[    5.992523] tas2505-codec 2-0018: tas2505_codec_probe:tas2505_codec_probe
[    5.997140] e5010 fd20000.jpeg-encoder: Device registered as /dev/video2
[    6.001865] tas2505-codec 2-0018: ASoC: error at soc_component_write_no_lock on tas2505-codec.2-0018 for register: [0x00000001] -121
[    6.028217] tas2505-codec 2-0018: ASoC: error at soc_component_write_no_lock on tas2505-codec.2-0018 for register: [0x00000004] -121
[    6.043078] tas2505-codec 2-0018: ASoC: error at soc_component_write_no_lock on tas2505-codec.2-0018 for register: [0x00000005] -121
[    6.053009] dbus-broker-lau[389]: memfd_create() called without MFD_EXEC or MFD_NOEXEC_SEAL set
[    6.058675] tas2505-codec 2-0018: ASoC: error at soc_component_write_no_lock on tas2505-codec.2-0018 for register: [0x00000006] -121
[    6.071966] tas2505-codec 2-0018: ASoC: error at soc_component_write_no_lock on tas2505-codec.2-0018 for register: [0x00000007] -121
[    6.091269] tas2505-codec 2-0018: ASoC: error at soc_component_write_no_lock on tas2505-codec.2-0018 for register: [0x00000008] -121
[    6.128639] tas2505-codec 2-0018: ASoC: error at soc_component_write_no_lock on tas2505-codec.2-0018 for register: [0x0000003c] -121
[    6.161522] tas2505-codec 2-0018: ASoC: error at soc_component_write_no_lock on tas2505-codec.2-0018 for register: [0x0000003f] -121
[    6.175899] tas2505-codec 2-0018: ASoC: error at soc_component_write_no_lock on tas2505-codec.2-0018 for register: [0x00000040] -121
[    6.189813] tas2505-codec 2-0018: ASoC: error at soc_component_write_no_lock on tas2505-codec.2-0018 for register: [0x00000041] -121
[    6.191878] m_can_platform 20701000.can main_mcan0: renamed from can0
[    6.205471] tas2505-codec 2-0018: ASoC: error at soc_component_write_no_lock on tas2505-codec.2-0018 for register: [0x00000000] -121
[    6.223697] tas2505-codec 2-0018: ASoC: error at soc_component_write_no_lock on tas2505-codec.2-0018 for register: [0x00000081] -121
[    6.269297] tas2505-codec 2-0018: ASoC: error at soc_component_write_no_lock on tas2505-codec.2-0018 for register: [0x0000008a] -121
[    6.311662] audit: type=1334 audit(1761700853.321:10): prog-id=14 op=LOAD
[    6.311696] audit: type=1334 audit(1761700853.321:11): prog-id=15 op=LOAD
[    6.311706] audit: type=1334 audit(1761700853.321:12): prog-id=16 op=LOAD
[    6.341488] tas2505-codec 2-0018: ASoC: error at soc_component_write_no_lock on tas2505-codec.2-0018 for register: [0x0000008c] -121
[    6.353621] tas2505-codec 2-0018: ASoC: error at soc_component_write_no_lock on tas2505-codec.2-0018 for register: [0x00000096] -121
[    6.366982] tas2505-codec 2-0018: ASoC: error at soc_component_write_no_lock on tas2505-codec.2-0018 for register: [0x00000098] -121
[    6.381325] tas2505-codec 2-0018: ASoC: error at soc_component_write_no_lock on tas2505-codec.2-0018 for register: [0x00000089] -121
[    6.395683] tas2505-codec 2-0018: ASoC: error at soc_component_write_no_lock on tas2505-codec.2-0018 for register: [0x00000090] -121
[    6.407832] tas2505-codec 2-0018: ASoC: error at soc_component_write_no_lock on tas2505-codec.2-0018 for register: [0x000000ae] -121
[    6.420722] tas2505-codec 2-0018: ASoC: error at soc_component_write_no_lock on tas2505-codec.2-0018 for register: [0x000000b0] -121
[    6.433851] tas2505-codec 2-0018: ASoC: error at soc_component_write_no_lock on tas2505-codec.2-0018 for register: [0x000000ad] -121
[    6.453264] tas2505-codec 2-0018: ASoC: error at soc_component_write_no_lock on tas2505-codec.2-0018 for register: [0x0000001b] -121
[    6.479439] tas2505-codec 2-0018: ASoC: error at snd_soc_component_update_bits on tas2505-codec.2-0018 for register: [0x000000ad] -121
[    6.494092] tas2505-codec 2-0018: ASoC: error at snd_soc_component_update_bits on tas2505-codec.2-0018 for register: [0x0000003f] -121
[    6.499523] cdns-csi2rx 30101000.csi-bridge: Probed CSI2RX with 4/4 lanes, 4 streams, external D-PHY
[    6.599001] audit: type=1334 audit(1761700853.609:13): prog-id=17 op=LOAD
[    7.153461] tas2505-codec 2-0018: ASoC: error at soc_component_write_no_lock on tas2505-codec.2-0018 for register: [0x00000041] -121
[    7.165670] tas2505-codec 2-0018: ASoC: error at soc_component_write_no_lock on tas2505-codec.2-0018 for register: [0x000000ae] -121
[    7.207768] cfg80211: Loading compiled-in X.509 certificates for regulatory database
[    7.223885] Loaded X.509 cert 'sforshee: 00b28ddf47aef9cea7'
[    7.224624] Loaded X.509 cert 'wens: 61c038651aabdcf94bd0ac7ff06c7248db18c600'
[    7.362522] EXT4-fs (mmcblk0p1): mounted filesystem 6178f84d-13e8-45bf-890c-a219a0607b8a r/w with ordered data mode. Quota mode: none.
[    7.365449] tas2505-codec 2-0018: ASoC: error at soc_component_write_no_lock on tas2505-codec.2-0018 for register: [0x00000041] -121
[    7.379315] tas2505-codec 2-0018: ASoC: error at soc_component_write_no_lock on tas2505-codec.2-0018 for register: [0x000000ae] -121
[    7.421708] am65-cpsw-nuss 8000000.ethernet eth0: PHY [8000f00.mdio:00] driver [Generic PHY] (irq=POLL)
[    7.421738] am65-cpsw-nuss 8000000.ethernet eth0: configuring for phy/rmii link mode
[    9.472305] am65-cpsw-nuss 8000000.ethernet eth0: Link is Up - 100Mbps/Full - flow control rx/tx
[    9.581088] audit: type=1334 audit(1761700856.593:14): prog-id=18 op=LOAD
[    9.581668] audit: type=1334 audit(1761700856.593:15): prog-id=19 op=LOAD
[    9.581822] audit: type=1334 audit(1761700856.593:16): prog-id=20 op=LOAD
[   39.804021] audit: type=1334 audit(1761700886.817:17): prog-id=20 op=UNLOAD
[   39.804048] audit: type=1334 audit(1761700886.817:18): prog-id=19 op=UNLOAD
[   39.804057] audit: type=1334 audit(1761700886.817:19): prog-id=18 op=UNLOAD
[   39.938640] audit: type=1006 audit(1761700886.949:20): pid=995 uid=0 old-auid=4294967295 auid=0 tty=(none) old-ses=4294967295 ses=1 res=1
[   39.938668] audit: type=1300 audit(1761700886.949:20): arch=c00000b7 syscall=64 success=yes exit=1 a0=8 a1=ffffd7fcbcf8 a2=1 a3=1 items=0 ppid=1 pid=995 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=1 comm="(systemd)" exe="/usr/lib/systemd/systemd-executor" key=(null)
[   39.938682] audit: type=1327 audit(1761700886.949:20): proctitle="(systemd)"
[   39.984214] audit: type=1334 audit(1761700886.997:21): prog-id=21 op=LOAD
[   39.984246] audit: type=1300 audit(1761700886.997:21): arch=c00000b7 syscall=280 success=yes exit=8 a0=5 a1=ffffc2a4a478 a2=90 a3=0 items=0 ppid=1 pid=995 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=1 comm="systemd" exe="/usr/lib/systemd/systemd" key=(null)
[   39.984257] audit: type=1327 audit(1761700886.997:21): proctitle="(systemd)"
[   39.984270] audit: type=1334 audit(1761700886.997:22): prog-id=21 op=UNLOAD
[   45.742191] max 96714 get format
[   45.742819] max96712 stream on
[   45.742827] max96714 2-002a: stream on!!!!
[   48.722820] tas2505-codec 2-0018: tas2505_hw_params: bit width = 16
[   48.722901] tas2505-codec 2-0018: tas2505_setup_pll: tas2505_setup_pll 
[   48.722910] tas2505-codec 2-0018: [0] mclksrc.rate (48000) vs rate (44100)
[   48.722919] tas2505-codec 2-0018: [1] mclksrc.rate (44100) vs rate (44100)
[   48.723162] tas2505-codec 2-0018: ASoC: error at soc_component_write_no_lock on tas2505-codec.2-0018 for register: [0x00000005] -121
[   48.740015] tas2505-codec 2-0018: ASoC: error at soc_component_write_no_lock on tas2505-codec.2-0018 for register: [0x00000006] -121
[   48.762018] tas2505-codec 2-0018: ASoC: error at soc_component_write_no_lock on tas2505-codec.2-0018 for register: [0x00000007] -121
[   48.779104] tas2505-codec 2-0018: ASoC: error at soc_component_write_no_lock on tas2505-codec.2-0018 for register: [0x00000008] -121
[   48.812646] tas2505-codec 2-0018: ASoC: error at soc_component_write_no_lock on tas2505-codec.2-0018 for register: [0x0000000b] -121
[   48.829591] tas2505-codec 2-0018: ASoC: error at soc_component_write_no_lock on tas2505-codec.2-0018 for register: [0x0000000c] -121
[   48.847809] tas2505-codec 2-0018: ASoC: error at soc_component_write_no_lock on tas2505-codec.2-0018 for register: [0x0000000d] -121
[   48.864466] tas2505-codec 2-0018: ASoC: error at soc_component_write_no_lock on tas2505-codec.2-0018 for register: [0x0000000e] -121
[   48.881845] tas2505-codec 2-0018: ASoC: error at snd_soc_component_update_bits on tas2505-codec.2-0018 for register: [0x0000001e] -121
[   48.899779] davinci-mcasp 2b10000.audio-controller: Sample-rate is off by 400 PPM
[   48.901211] tas2505-codec 2-0018: ASoC: error at snd_soc_component_update_bits on tas2505-codec.2-0018 for register: [0x00000082] -121
[   48.917040] tas2505-codec 2-0018: tas2505_set_bias_level:power on
[   48.917176] tas2505-codec 2-0018: ASoC: error at snd_soc_component_update_bits on tas2505-codec.2-0018 for register: [0x0000003f] -121
[   48.934850] tas2505-codec 2-0018: ASoC: error at snd_soc_component_update_bits on tas2505-codec.2-0018 for register: [0x000000ad] -121
[   48.953312] tas2505-codec 2-0018: ASoC: error at soc_component_write_no_lock on tas2505-codec.2-0018 for register: [0x000000ae] -121
[   91.037914] tidss 30200000.dss: CRTC1 SYNC LOST: (irq 800)
[   97.838353] tidss 30200000.dss: CRTC1 SYNC LOST: (irq 800)
[  143.737739] tidss 30200000.dss: CRTC1 SYNC LOST: (irq 800)
[  155.736322] virtio_rpmsg_bus virtio2: msg received with no recipient
[  155.736354] virtio_rpmsg_bus virtio2: msg received with no recipient
[  155.736361] virtio_rpmsg_bus virtio2: msg received with no recipient
[  155.736367] virtio_rpmsg_bus virtio2: msg received with no recipient
[  155.736372] virtio_rpmsg_bus virtio2: msg received with no recipient
[  155.736378] virtio_rpmsg_bus virtio2: msg received with no recipient
[  156.071866] tas2505-codec 2-0018: ASoC: error at snd_soc_component_update_bits on tas2505-codec.2-0018 for register: [0x00000040] -121
[  156.084055] tas2505-codec 2-0018: ASoC: error at snd_soc_component_update_bits on tas2505-codec.2-0018 for register: [0x0000003f] -121
[  156.096356] tas2505-codec 2-0018: ASoC: error at snd_soc_component_update_bits on tas2505-codec.2-0018 for register: [0x000000ad] -121
[  156.108682] tas2505-codec 2-0018: tas2505_set_bias_level:SND_SOC_BIAS_STANDBY
[  156.108767] tas2505-codec 2-0018: ASoC: error at snd_soc_component_update_bits on tas2505-codec.2-0018 for register: [0x00000081] -121
[  156.120929] tas2505-codec 2-0018: ASoC: error at snd_soc_component_update_bits on tas2505-codec.2-0018 for register: [0x00000082] -121
[  156.133009] tas2505-codec 2-0018: tas2505_set_bias_level:power off
[  156.166928] max96712 stream off
[  156.166947] max96714 2-002a: stream off!!!!

Here is the top and htop output

// HERE IS THE TOP OUTPUT
op - 01:27:29 up 5 min,  2 users,  load average: 2.10, 1.57, 0.75
Tasks: 151 total,   1 running, 150 sleeping,   0 stopped,   0 zombie
%Cpu(s): 40.1 us,  8.5 sy,  0.0 ni, 48.2 id,  0.0 wa,  2.5 hi,  0.8 si,  0.0 st
MiB Mem :   3319.5 total,   2867.5 free,    378.9 used,    220.1 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used.   2940.6 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   1094 root      20   0 1730968 102960  25336 S 189.4   3.0   0:49.19 media_main
    406 root     -51   0       0      0      0 S   2.3   0.0   0:03.37 irq/15-mbox-r5-0
    383 root      20   0       0      0      0 S   1.0   0.0   0:01.23 vpu_irq_thread
   1127 root      20   0    7436   4480   2432 R   1.0   0.1   0:00.17 top
   
// HERE IS THE HTOP OUTPUT

    0[###################################**********************                                  58.1%] Tasks: 37, 54 thr, 99 kthr; 0 running
    1[########################################**********                                         51.3%] Load average: 3.54 2.42 1.20
    2[###########################################********                                        52.2%] Uptime: 00:07:38
    3[#######################################****************                                    56.3%]
  Mem[|||||||#@$$$$$$                                                                       240M/3.24G]
  Swp[                                                                                           0K/0K]

  [Main] [I/O]
    PID USER       PRI  NI  VIRT   RES   SHR S  CPU%-MEM%   TIME+  Command
   1173 root        20   0 1692M 98.4M 25208 S  42.9  3.0  0:06.95 /root/media_main
   1166 root        20   0 1688M 98.4M 25208 R  37.8  3.0  0:06.27 /root/media_main
   1168 root        20   0 1690M 98.4M 25208 S  33.3  3.0  0:05.46 /root/media_main
   1161 root        20   0 1690M 98.4M 25208 R  20.5  3.0  0:03.31 /root/media_main
   1163 root        20   0 1690M 98.4M 25208 S  10.3  3.0  0:01.65 /root/media_main
   1172 root        20   0 1690M 98.4M 25208 S   9.6  3.0  0:01.55 /root/media_main
   1167 root        20   0 1688M 98.4M 25208 S   9.0  3.0  0:01.60 /root/media_main
   1154 root        20   0 1690M 98.4M 25208 S   7.1  3.0  0:01.22 /root/media_main
   1165 root        20   0 1690M 98.4M 25208 S   5.8  3.0  0:01.05 /root/media_main
   1174 root        20   0 1690M 98.4M 25208 R   5.1  3.0  0:00.72 /root/media_main
   1151 root        20   0 1690M 98.4M 25208 S   3.8  3.0  0:00.65 /root/media_main
   1176 root        20   0  5632  3968  2432 R   3.8  0.1  0:00.59 htop
   1169 root        20   0 1690M 98.4M 25208 S   3.2  3.0  0:00.52 /root/media_main
   1162 root        20   0 1690M 98.4M 25208 S   1.9  3.0  0:00.34 /root/media_main
      1 root        20   0 19044 10572  7628 S   0.0  0.3  0:02.03 /sbin/init

When I attempted to increase CPU load using stress-ng, it appears that the frequency of screen tearing indeed increased.

root@am62axx-evm:~# stress-ng --cpu 4 --cpu-load 50 --timeout 200
stress-ng: info:  [1199] setting to a 3 mins, 20 secs run per stressor
stress-ng: info:  [1199] dispatching hogs: 4 cpu
stress-ng: info:  [1200] cpu: for stable load results, select a specific cpu stress method with --cpu-method other than 'all'
stress-ng: info:  [1201] cpu: for stable load results, select a specific cpu stress method with --cpu-method other than 'all'
stress-ng: info:  [1202] cpu: for stable load results, select a specific cpu stress method with --cpu-method other than 'all'
stress-ng: info:  [1203] cpu: for stable load results, select a specific cpu stress method with --cpu-method other than 'all'
[ 1211.978176] tidss 30200000.dss: CRTC1 SYNC LOST: (irq 800)
[ 1214.708312] tidss 30200000.dss: CRTC1 SYNC LOST: (irq 800)
[ 1216.592337] tidss 30200000.dss: CRTC1 SYNC LOST: (irq 800)
[ 1218.457278] tidss 30200000.dss: CRTC1 SYNC LOST: (irq 800)
[ 1219.549533] tidss 30200000.dss: CRTC1 SYNC LOST: (irq 800)
[ 1219.657815] tidss 30200000.dss: CRTC1 SYNC LOST: (irq 800)
[ 1222.005762] tidss 30200000.dss: CRTC1 SYNC LOST: (irq 800)
[ 1224.826556] tidss 30200000.dss: CRTC1 SYNC LOST: (irq 800)
[ 1225.277437] tidss 30200000.dss: CRTC1 SYNC LOST: (irq 800)
[ 1226.068489] tidss 30200000.dss: CRTC1 SYNC LOST: (irq 800)
[ 1229.010577] tidss 30200000.dss: CRTC1 SYNC LOST: (irq 800)
[ 1233.397963] tidss 30200000.dss: CRTC1 SYNC LOST: (irq 800)
[ 1237.191036] tidss 30200000.dss: CRTC1 SYNC LOST: (irq 800)
[ 1237.586709] tidss 30200000.dss: CRTC1 SYNC LOST: (irq 800)
[ 1239.383986] tidss 30200000.dss: CRTC1 SYNC LOST: (irq 800)
[ 1239.905909] tidss 30200000.dss: CRTC1 SYNC LOST: (irq 800)
[ 1244.388372] tidss 30200000.dss: CRTC1 SYNC LOST: (irq 800)
[ 1247.917870] tidss 30200000.dss: CRTC1 SYNC LOST: (irq 800)
[ 1250.397427] tidss 30200000.dss: CRTC1 SYNC LOST: (irq 800)
[ 1251.154941] tidss 30200000.dss: CRTC1 SYNC LOST: (irq 800)
[ 1258.820905] tidss 30200000.dss: CRTC1 SYNC LOST: (irq 800)
[ 1259.202269] tidss 30200000.dss: CRTC1 SYNC LOST: (irq 800)
[ 1262.570151] tidss 30200000.dss: CRTC1 SYNC LOST: (irq 800)
[ 1264.364950] tidss 30200000.dss: CRTC1 SYNC LOST: (irq 800)
^C^Cstress-ng: info:  [1199] skipped: 0
stress-ng: info:  [1199] passed: 4: cpu (4)
stress-ng: info:  [1199] failed: 0
stress-ng: info:  [1199] metrics untrustworthy: 0
stress-ng: info:  [1199] successful run completed in 55.53 secs
root@am62axx-evm:~# [ 1283.763220] tidss 30200000.dss: CRTC1 SYNC LOST: (irq 800)
[ 1358.403009] tidss 30200000.dss: CRTC1 SYNC LOST: (irq 800)
[ 1421.702536] tidss 30200000.dss: CRTC1 SYNC LOST: (irq 800)

I'm suspecting this might be related to an incomplete update process. Next, I plan to verify by creating a complete 10.1 file system directly on the SD card. However, since setting up the entire imaging pipeline in the 10.1 SDK (such as the MAX96714 driver and device tree) would be time-consuming, I'd prefer to continue testing on this current 10.0.0.08 file system. Are there any benchmark tests available that could help rule out factors like our own algorithm models?

Jason

0 Reese Grimsley 4 months ago in reply to Jason Zeng

TI__Mastermind 18176 points

Hi Jason,

These are good findings, especially that CPU loading has an impact. I am wondering if this could be related to IRQ balancing.

Is there a process running the background for 'irqbalance'? This is an application that splits interrupt-handling across multiple cores, as otherwise they may all run on the same core. If you are based on a tiny yocto image, then perhaps this application / service is not running.

root@am62axx-evm:/opt/edgeai-gst-apps# ps -ax | grep irqbalance
    456 ?        Ssl    0:01 /usr/sbin/irqbalance --foreground
## May also be an `irqbalanced` service shown in systemctl console

If this is not running, then please try running as `irqbalance -f` in a separate process and see if the application is still encountering the same issues.

We can also see more about interrupts from running `cat /proc/interrupts` -- please supply a printout of this before and after your application has run for a short period.

It might also help to run the `irqtop` application to see any runtime behavior. Please supply a printout of this as well.

0 Reese Grimsley 4 months ago in reply to Reese Grimsley

TI__Mastermind 18176 points

Hi Jason,

These are good findings, especially that CPU loading has an impact. I am wondering if this could be related to IRQ balancing.

root@am62axx-evm:/opt/edgeai-gst-apps# ps -ax | grep irqbalance
    456 ?        Ssl    0:01 /usr/sbin/irqbalance --foreground
## May also be an `irqbalanced` service shown in systemctl console

If this is not running, then please try running as `irqbalance -f` in a separate process and see if the application is still encountering the same issues.

We can also see more about interrupts from running `cat /proc/interrupts` -- please supply a printout of this before and after your application has run for a short period.

It might also help to run the `irqtop` application to see any runtime behavior. Please supply a printout of this as well.

0 Jason Zeng 4 months ago in reply to Reese Grimsley

Intellectual 375 points

Hi Reese

I can see that irqbalance is running in the background, and the issue can still be reproduced under these circumstances.

root@am62axx-evm:~# ps -ef | grep "irqbalance"
root         462       1  0 00:38 ?        00:00:00 /usr/sbin/irqbalance --foreground
root        1069    1057  0 00:47 pts/0    00:00:00 grep irqbalance

Here is the interrupts printout before the application runs.

data-mode="text">root@am62axx-evm:~# cat /proc/interrupts CPU2 CPU3 8187 21300 4264 GICv3 30 Level arch_timer 0 0 GICv3 131 Level pinctrl 0 0 GICv3 108 Level mbox-r5-0 0 0 GICv3 109 Level mbox-c7x-0 0 0 GICv3 140 Level mbox-mcu-r5-0 0 0 GICv3 23 Level arm-pmu 0 0 GICv3 66 Level 4d000000.mailbox thr_012 0 0 GICv3 193 Level 20000000.i2c 0 0 GICv3 194 Level 20010000.i2c 0 0 GICv3 195 Level 20020000.i2c 0 0 MSI-INTA 1713152 Level 485c0100.dma-controller chan0 0 0 MSI-INTA 1714176 Edge 485c0100.dma-controller chan1 0 0 MSI-INTA 1714688 Level 485c0100.dma-controller chan1 0 0 MSI-INTA 1715718 Edge 485c0100.dma-controller chan2 0 0 MSI-INTA 1716230 Level 485c0100.dma-controller chan2 0 0 MSI-INTA 1970707 Level 8000000.ethernet-tx0 0 1 MSI-INTA 1970715 Level 8000000.ethernet-tx1 0 0 MSI-INTA 1970723 Level 8000000.ethernet-tx2 0 6 MSI-INTA 1970731 Level 8000000.ethernet-tx3 0 0 MSI-INTA 1970739 Level 8000000.ethernet-tx4 0 0 MSI-INTA 1970747 Level 8000000.ethernet-tx5 0 27 5 MSI-INTA 1970755 Level 8000000.ethernet-tx6 2 0 MSI-INTA 1970763 Level 8000000.ethernet-tx7 0 0 MSI-INTA 1970779 Level 485c0000.dma-controller chan2 374 7 MSI-INTA 1971731 Level 8000000.ethernet 0 0 MSI-INTA 1971755 Level 485c0000.dma-controller chan0 0 0 MSI-INTA 1971756 Level 485c0000.dma-controller chan1 0 0 GICv3 210 Level 2800000.serial 0 0 pinctrl 456 Edge 2800000.serial:wakeup 0 0 GICv3 212 Level 2820000.serial 0 0 GICv3 171 Level fc40000.spi 0 0 GICv3 134 Level 8000000.ethernet 0 0 GICv3 220 Level xhci-hcd:usb1 0 0 GICv3 258 Level xhci-hcd:usb2 0 0 GICv3 132 Level 2b1f0000.rtc 0 0 GICv3 165 Level mmc0 0 0 GICv3 115 Level mmc1 0 0 GPIO 0 Edge -davinci_gpio tps6594-1-0x48 0 0 GPIO 16 Edge -davinci_gpio gpio-ir-recv-irq 0 0 tps6594-1-0x48 0 Edge buck1_ov 0 0 tps6594-1-0x48 1 Edge buck1_uv 0 0 tps6594-1-0x48 2 Edge buck1_sc 0 0 tps6594-1-0x48 3 Edge buck1_ilim 0 0 tps6594-1-0x48 4 Edge buck2_ov 0 0 tps6594-1-0x48 5 Edge buck2_uv 0 0 tps6594-1-0x48 6 Edge buck2_sc 0 0 tps6594-1-0x48 7 Edge buck2_ilim 0 0 tps6594-1-0x48 8 Edge buck3_ov 0 0 tps6594-1-0x48 9 Edge buck3_uv 0 0 tps6594-1-0x48 10 Edge buck3_sc 0 0 tps6594-1-0x48 11 Edge buck3_ilim 0 0 tps6594-1-0x48 12 Edge buck4_ov 0 0 tps6594-1-0x48 13 Edge buck4_uv 0 0 tps6594-1-0x48 14 Edge buck4_sc 0 0 tps6594-1-0x48 15 Edge buck4_ilim 0 0 tps6594-1-0x48 16 Edge buck5_ov 0 0 tps6594-1-0x48 17 Edge buck5_uv 0 0 tps6594-1-0x48 18 Edge buck5_sc 0 0 tps6594-1-0x48 19 Edge buck5_ilim 0 0 tps6594-1-0x48 20 Edge ldo1_ov 0 0 tps6594-1-0x48 21 Edge ldo1_uv 0 0 tps6594-1-0x48 22 Edge ldo1_sc 0 0 tps6594-1-0x48 23 Edge ldo1_ilim 0 0 tps6594-1-0x48 24 Edge ldo2_ov 0 0 tps6594-1-0x48 25 Edge ldo2_uv 0 0 tps6594-1-0x48 26 Edge ldo2_sc 0 0 tps6594-1-0x48 27 Edge ldo2_ilim 0 0 tps6594-1-0x48 28 Edge ldo3_ov 0 0 tps6594-1-0x48 29 Edge ldo3_uv 0 0 tps6594-1-0x48 30 Edge ldo3_sc 0 0 tps6594-1-0x48 31 Edge ldo3_ilim 0 0 tps6594-1-0x48 32 Edge ldo4_ov 0 0 tps6594-1-0x48 33 Edge ldo4_uv 0 0 tps6594-1-0x48 34 Edge ldo4_sc 0 0 tps6594-1-0x48 35 Edge ldo4_ilim 0 0 tps6594-1-0x48 36 Edge vcca_ov 0 0 tps6594-1-0x48 37 Edge vcca_uv 0 0 tps6594-1-0x48 55 Edge npwron_start 0 0 tps6594-1-0x48 56 Edge enable 0 0 tps6594-1-0x48 57 Edge fsd 0 0 tps6594-1-0x48 58 Edge soft_reboot 0 0 tps6594-1-0x48 59 Edge bist_pass 0 0 tps6594-1-0x48 60 Edge ext_clk 0 0 tps6594-1-0x48 61 Edge twarn 0 0 tps6594-1-0x48 62 Edge tsd_ord 0 0 tps6594-1-0x48 63 Edge bist_fail 0 0 tps6594-1-0x48 64 Edge reg_crc_err 0 0 tps6594-1-0x48 65 Edge recov_cnt 0 0 tps6594-1-0x48 66 Edge spmi_err 0 0 tps6594-1-0x48 67 Edge npwron_long 0 0 tps6594-1-0x48 68 Edge nint_readback 0 0 tps6594-1-0x48 69 Edge nrstout_readback 0 0 tps6594-1-0x48 70 Edge tsd_imm 0 0 tps6594-1-0x48 71 Edge vcca_ovp 0 0 tps6594-1-0x48 72 Edge pfsm_err 0 0 tps6594-1-0x48 73 Edge imm_shutdown 0 0 tps6594-1-0x48 74 Edge ord_shutdown 0 0 tps6594-1-0x48 75 Edge mcu_pwr_err 0 0 tps6594-1-0x48 76 Edge soc_pwr_err 0 0 tps6594-1-0x48 77 Edge comm_frm_err 0 0 tps6594-1-0x48 78 Edge comm_crc_err 0 0 tps6594-1-0x48 79 Edge comm_adr_err 0 0 tps6594-1-0x48 82 Edge en_drv_readback 0 0 tps6594-1-0x48 83 Edge nrstout_soc_readback 0 0 tps6594-1-0x48 84 Edge esm_soc_pin 0 0 tps6594-1-0x48 85 Edge esm_soc_fail 0 0 tps6594-1-0x48 86 Edge esm_soc_rst 0 0 tps6594-1-0x48 88 Edge alarm 0 0 GICv3 116 Level tidss 0 0 GICv3 269 Level 2b10000.audio-controller_rx 0 0 GICv3 270 Level 2b10000.audio-controller_tx 0 0 GICv3 130 Level e5010 694 613 Rescheduling interrupts 4860 4836 Function call interrupts 0 0 CPU stop interrupts 0 0 CPU stop (for crash dump) interrupts 0 0 Timer broadcast interrupts 0 0 IRQ work interrupts 0 0 CPU wake-up interrupts

And here is the printout after the application has run and the DSS error message has appeared.

root@am62axx-evm:~# cat /proc/interrupts
           CPU0       CPU1       CPU2       CPU3
 11:      27676      35958      36648      20028     GICv3  30 Level     arch_timer
 14:          0          0          0          0     GICv3 131 Level     pinctrl
 15:      38420          0          0          0     GICv3 108 Level     mbox-r5-0
 16:       4088          0          0          0     GICv3 109 Level     mbox-c7x-0
 17:          4          0          0          0     GICv3 140 Level     mbox-mcu-r5-0
 18:          0          0          0          0     GICv3  23 Level     arm-pmu
 19:       1582          0          0          0     GICv3  66 Level     4d000000.mailbox thr_012
 27:       3611          0          0          0     GICv3 193 Level     20000000.i2c
 28:          7          0          0          0     GICv3 194 Level     20010000.i2c
 29:       6399          0          0          0     GICv3 195 Level     20020000.i2c
 30:          0          0          0          0  MSI-INTA 1713152 Level     485c0100.dma-controller chan0
 48:          0          0          0          0  MSI-INTA 1714176 Edge      485c0100.dma-controller chan1
 60:          0          0          0          0  MSI-INTA 1714688 Level     485c0100.dma-controller chan1
 78:          0          0          0          0  MSI-INTA 1715718 Edge      485c0100.dma-controller chan2
 96:          0          0          0          0  MSI-INTA 1716230 Level     485c0100.dma-controller chan2
118:         14          0          0          0  MSI-INTA 1970707 Level     8000000.ethernet-tx0
126:          0          0          0          2  MSI-INTA 1970715 Level     8000000.ethernet-tx1
134:         44          0          0          0  MSI-INTA 1970723 Level     8000000.ethernet-tx2
142:          0          0          0          6  MSI-INTA 1970731 Level     8000000.ethernet-tx3
150:          0          4          0          0  MSI-INTA 1970739 Level     8000000.ethernet-tx4
158:          0          4          0          0  MSI-INTA 1970747 Level     8000000.ethernet-tx5
166:          6          0        142         43  MSI-INTA 1970755 Level     8000000.ethernet-tx6
174:          0          0          2          0  MSI-INTA 1970763 Level     8000000.ethernet-tx7
190:          0          0          0          0  MSI-INTA 1970779 Level     485c0000.dma-controller chan2
208:        319        744        649        446  MSI-INTA 1971731 Level     8000000.ethernet
232:          0          0          0          0  MSI-INTA 1971755 Level     485c0000.dma-controller chan0
233:          0          0          0          0  MSI-INTA 1971756 Level     485c0000.dma-controller chan1
240:          0          0          0          0  MSI-INTA 13045248 Edge      4e230000.dma-controller chan0
241:          0          0          0          0  MSI-INTA 13045249 Edge      4e230000.dma-controller chan1
242:          0          0          0          0  MSI-INTA 13045250 Edge      4e230000.dma-controller chan2
243:          0          0          0          0  MSI-INTA 13045251 Edge      4e230000.dma-controller chan3
244:          0          0          0          0  MSI-INTA 13045252 Edge      4e230000.dma-controller chan4
245:          0          0          0          0  MSI-INTA 13045253 Edge      4e230000.dma-controller chan5
246:       3059          0          0          0  MSI-INTA 13045760 Level     4e230000.dma-controller chan0
247:          0          0          0          0  MSI-INTA 13045761 Level     4e230000.dma-controller chan1
248:          0          0          0          0  MSI-INTA 13045762 Level     4e230000.dma-controller chan2
249:          0          0          0          0  MSI-INTA 13045763 Level     4e230000.dma-controller chan3
250:          0          0          0          0  MSI-INTA 13045764 Level     4e230000.dma-controller chan4
251:          0          0          0          0  MSI-INTA 13045765 Level     4e230000.dma-controller chan5
252:        873          0          0          0     GICv3 210 Level     2800000.serial
253:          0          0          0          0   pinctrl 456 Edge      2800000.serial:wakeup
254:          0          0          0          0     GICv3 212 Level     2820000.serial
255:          0          0          0          0     GICv3 171 Level     fc40000.spi
257:          0          0          0          0     GICv3 134 Level     8000000.ethernet
267:          0          0          0          0     GICv3 220 Level     xhci-hcd:usb1
268:          0          0          0          0     GICv3 258 Level     xhci-hcd:usb2
269:          0          0          0          0     GICv3 132 Level     2b1f0000.rtc
270:        598          0          0          0     GICv3 165 Level     mmc0
271:      14134          0          0          0     GICv3 115 Level     mmc1
274:          1          0          0          0      GPIO   0 Edge    -davinci_gpio  tps6594-1-0x48
416:          0          0          0          0      GPIO  16 Edge    -davinci_gpio  gpio-ir-recv-irq
452:          0          0          0          0  tps6594-1-0x48   0 Edge      buck1_ov
453:          0          0          0          0  tps6594-1-0x48   1 Edge      buck1_uv
454:          0          0          0          0  tps6594-1-0x48   2 Edge      buck1_sc
455:          0          0          0          1  tps6594-1-0x48   3 Edge      buck1_ilim
456:          0          0          0          0  tps6594-1-0x48   4 Edge      buck2_ov
457:          0          0          0          0  tps6594-1-0x48   5 Edge      buck2_uv
458:          0          0          0          0  tps6594-1-0x48   6 Edge      buck2_sc
459:          0          0          0          0  tps6594-1-0x48   7 Edge      buck2_ilim
460:          0          0          0          0  tps6594-1-0x48   8 Edge      buck3_ov
461:          0          0          0          0  tps6594-1-0x48   9 Edge      buck3_uv
462:          0          0          0          0  tps6594-1-0x48  10 Edge      buck3_sc
463:          0          0          0          0  tps6594-1-0x48  11 Edge      buck3_ilim
464:          0          0          0          0  tps6594-1-0x48  12 Edge      buck4_ov
465:          0          0          0          0  tps6594-1-0x48  13 Edge      buck4_uv
466:          0          0          0          0  tps6594-1-0x48  14 Edge      buck4_sc
467:          0          0          0          0  tps6594-1-0x48  15 Edge      buck4_ilim
468:          0          0          0          0  tps6594-1-0x48  16 Edge      buck5_ov
469:          0          0          0          0  tps6594-1-0x48  17 Edge      buck5_uv
470:          0          0          0          0  tps6594-1-0x48  18 Edge      buck5_sc
471:          0          0          0          0  tps6594-1-0x48  19 Edge      buck5_ilim
472:          0          0          0          0  tps6594-1-0x48  20 Edge      ldo1_ov
473:          0          0          0          0  tps6594-1-0x48  21 Edge      ldo1_uv
474:          0          0          0          0  tps6594-1-0x48  22 Edge      ldo1_sc
475:          0          0          0          0  tps6594-1-0x48  23 Edge      ldo1_ilim
476:          0          0          0          0  tps6594-1-0x48  24 Edge      ldo2_ov
477:          0          0          0          0  tps6594-1-0x48  25 Edge      ldo2_uv
478:          0          0          0          0  tps6594-1-0x48  26 Edge      ldo2_sc
479:          0          0          0          0  tps6594-1-0x48  27 Edge      ldo2_ilim
480:          0          0          0          0  tps6594-1-0x48  28 Edge      ldo3_ov
481:          0          0          0          0  tps6594-1-0x48  29 Edge      ldo3_uv
482:          0          0          0          0  tps6594-1-0x48  30 Edge      ldo3_sc
483:          0          0          0          0  tps6594-1-0x48  31 Edge      ldo3_ilim
484:          0          0          0          0  tps6594-1-0x48  32 Edge      ldo4_ov
485:          0          0          0          0  tps6594-1-0x48  33 Edge      ldo4_uv
486:          0          0          0          0  tps6594-1-0x48  34 Edge      ldo4_sc
487:          0          0          0          0  tps6594-1-0x48  35 Edge      ldo4_ilim
488:          0          0          0          0  tps6594-1-0x48  36 Edge      vcca_ov
489:          0          0          0          0  tps6594-1-0x48  37 Edge      vcca_uv
507:          0          0          0          0  tps6594-1-0x48  55 Edge      npwron_start
508:          0          0          0          0  tps6594-1-0x48  56 Edge      enable
509:          0          0          0          0  tps6594-1-0x48  57 Edge      fsd
510:          0          0          0          0  tps6594-1-0x48  58 Edge      soft_reboot
511:          0          0          0          0  tps6594-1-0x48  59 Edge      bist_pass
512:          0          0          0          0  tps6594-1-0x48  60 Edge      ext_clk
513:          0          0          0          0  tps6594-1-0x48  61 Edge      twarn
514:          0          0          0          0  tps6594-1-0x48  62 Edge      tsd_ord
515:          0          0          0          0  tps6594-1-0x48  63 Edge      bist_fail
516:          0          0          0          0  tps6594-1-0x48  64 Edge      reg_crc_err
517:          0          0          0          0  tps6594-1-0x48  65 Edge      recov_cnt
518:          0          0          0          0  tps6594-1-0x48  66 Edge      spmi_err
519:          0          0          0          0  tps6594-1-0x48  67 Edge      npwron_long
520:          0          0          0          0  tps6594-1-0x48  68 Edge      nint_readback
521:          0          0          0          0  tps6594-1-0x48  69 Edge      nrstout_readback
522:          0          0          0          0  tps6594-1-0x48  70 Edge      tsd_imm
523:          0          0          0          0  tps6594-1-0x48  71 Edge      vcca_ovp
524:          0          0          0          0  tps6594-1-0x48  72 Edge      pfsm_err
525:          0          0          0          0  tps6594-1-0x48  73 Edge      imm_shutdown
526:          0          0          0          0  tps6594-1-0x48  74 Edge      ord_shutdown
527:          0          0          0          0  tps6594-1-0x48  75 Edge      mcu_pwr_err
528:          0          0          0          0  tps6594-1-0x48  76 Edge      soc_pwr_err
529:          0          0          0          0  tps6594-1-0x48  77 Edge      comm_frm_err
530:          0          0          0          0  tps6594-1-0x48  78 Edge      comm_crc_err
531:          0          0          0          0  tps6594-1-0x48  79 Edge      comm_adr_err
532:          0          0          0          0  tps6594-1-0x48  82 Edge      en_drv_readback
533:          0          0          0          0  tps6594-1-0x48  83 Edge      nrstout_soc_readback
534:          0          0          0          0  tps6594-1-0x48  84 Edge      esm_soc_pin
535:          0          0          0          0  tps6594-1-0x48  85 Edge      esm_soc_fail
536:          0          0          0          0  tps6594-1-0x48  86 Edge      esm_soc_rst
538:          0          0          0          0  tps6594-1-0x48  88 Edge      alarm
540:       4221          0          0          0     GICv3 116 Level     tidss
545:          0          0          0          0     GICv3 269 Level     2b10000.audio-controller_rx
546:          0          0          0          0     GICv3 270 Level     2b10000.audio-controller_tx
547:          0          0          0          0     GICv3 130 Level     e5010
IPI0:       931       1438       1324       1255       Rescheduling interrupts
IPI1:     17176      25847      26162      26110       Function call interrupts
IPI2:         0          0          0          0       CPU stop interrupts
IPI3:         0          0          0          0       CPU stop (for crash dump) interrupts
IPI4:         0          0          0          0       Timer broadcast interrupts
IPI5:         3          0          0          0       IRQ work interrupts
IPI6:         0          0          0          0       CPU wake-up interrupts
Err:          0

Jason

0 Reese Grimsley 4 months ago in reply to Jason Zeng

TI__Mastermind 18176 points

Hi Jason,

Okay, irqbalance was already running.. not an easy workaround here, then. I believe IRQ balance does not impact all interrupts, but only certain ones. TI-specific ones like mbox-c7x-0 are still pinned to core 0. Still, I see your overall CPU utilization is far less than 80% across the cores, so I would not have expected so much contention that there are frequently missed interrupts or signaling deadlines.

You may try pinning some of the IRQ's, mainly mbox-c7x-0 and tidss from /proc/irq/N/smp_affinity to be 2 (or any other hex values 2 to D such that bit 0 is not 1).

'N' is the irq number, which shows from /proc/interrupts. 540 corresponds to tidss in your printout
You may need to turn off irqbalance during this

Jason Zeng said:
I'm suspecting this might be related to an incomplete update process.

If you ran the update script in edgeai-tidl-tools, the migration process should be complete. This will replace C7x firmware and a few libraries under /usr/lib. These are mainly for TIDL components and libtivision_apps (which contains TI OpenVX functions; critical for many pieces of the stack).

This is a reach, but I am curious if libtivision_apps could remain as the 10.0 original version and only update the other TIDL libraries to 10.1. tivision_apps is generally more stable release-to-release. I would treat this as an experiment rather than a recommended workaround.

We could approach trying to reduce the CPU utilization, but I think this is a wrong path to take. It may be worth looking at SDK 10.1.

Perhaps you can try using an encoded video file in lieu of the actual camera sensor driver to simplify this experiment. That would remove a few plugins from the pipeline like tiovxisp and add video-decode, so not a perfect 1:1 comparison, but perhaps close enough to validate whether this issue will occur there too when you add your 3 appsinks.

0 Jason Zeng 4 months ago in reply to Reese Grimsley

Intellectual 375 points

Hi Reese

root@am62axx-evm:~# cat /proc/interrupts | grep tidss
540:        303          0          0          0     GICv3 116 Level     tidss
root@am62axx-evm:~# cat /proc/interrupts | grep mbox-c7x-0
 16:          6          0          0          0     GICv3 109 Level     mbox-c7x-0
root@am62axx-evm:~# echo 2 > /proc/irq/540/smp_affinity
root@am62axx-evm:~# cat /proc/irq/540/smp_affinity
2
root@am62axx-evm:~# echo 2 > /proc/irq/16/smp_affinity
root@am62axx-evm:~# cat /proc/irq/16/smp_affinity
2

After setting the irq, the DSS error problem still persists.

During this period, I will start to migrate and use the SDK version 10.1 to verify if the same issue exists in version 10.1 as well.I will post the results after verification. The entire process will take approximately 2 to 3 days.

Jason

0 Reese Grimsley 4 months ago in reply to Jason Zeng

TI__Mastermind 18176 points

Hi Jason,

Yes, please proceed with that effort. If the IRQ balance aspect did not work, then my series of recommendations is running thin. Only reducing the model size (or latency by some other means) remains in my mind here.

Do you have access to the ti-firmware-builder? If the issue persists in 10.1 SDK, then I have a change I would suggest implementing there in the mcu_plus_sdk package. I have recently learned 10.1 SDK had a non-optimal cache setting for the C7x that might impact performance in some models.

This was using the optimal setting for 10.0, but- we may have used non-optimal setting for the 10.1 version w/ backward compatibility provided via TIDL tooling. Rebuilding this backwards-compatible firmware otherwise is a bit challenging -- it requires specific source code release from TI

BR,
Reese

0 Jason Zeng 4 months ago in reply to Reese Grimsley

Intellectual 375 points

Hi Reese

I recently tested the media runtime of version 10.1 and found that the algorithm failed to run, with the following error.Lately I've been investigating solutions to this issue, which may be related to our algorithm lib.

    73.373062 s:  VX_ZONE_ERROR: [ownContextSendCmd:912] Command ack message returned failure cmd_status: -1
    73.373106 s:  VX_ZONE_ERROR: [ownNodeKernelInit:604] Target kernel, TIVX_CMD_NODE_CREATE failed for node tidl_node
    73.373128 s:  VX_ZONE_ERROR: [ownNodeKernelInit:605] Please be sure the target callbacks have been registered for this core
    73.373139 s:  VX_ZONE_ERROR: [ownNodeKernelInit:606] If the target callbacks have been registered, please ensure no errors are occurring within the create callback of this kernel
    73.373154 s:  VX_ZONE_ERROR: [ownGraphNodeKernelInit:690] kernel init for node 1, kernel com.ti.tidl:1:3 ... failed !!!
    73.373199 s:  VX_ZONE_ERROR: [ graph_83 ] Node kernel init failed
    73.373210 s:  VX_ZONE_ERR[   68.481751] max 96714 get format
OR: [ graph_83 ] Graph verify failed
[TIOVX_MODULES][ERROR] 791[   68.488014] max96712 stream on
: tiovx_modules_verify_graph: Graph Verify failed
 [Leve[   68.495939] max96714 2-002a: stream on!!!!
l]:Error,[FILE]:/home/zyh/tiovx/edgeai-tiovx-apps-develop/alg/src/alglib.cpp [Func]:algInit [Line]:300 [Info]:tiovx_modules_link_pads failed!

I do have access to TI Firmware Builder

Does this means that using the 11.1 SDK version is more recommended? Additionally, I would like to inquire about the scheduled release date for the next SDK version update.

Jason

0 Reese Grimsley 4 months ago in reply to Jason Zeng

TI__Mastermind 18176 points

Hi Jason,

When you run this test, please run the /opt/vx_app_arm_remote_log.out in the background so we see more TIOVX logging.

Are you using identical artifacts to what you were using with 10.0 and the updated firmware? You may need to recompile against true 10.1 tools like 10_01_00_04. If there is a line in the vx_app_arm_remote_log.out like the following, then you will need to recompile the model (all settings can be identical otherwise; only need to point to a separate tidl_tools directory)

VX_ZONE_ERROR:[tivxKernelTIDLCreate:910] Network version - 0x20240215, Expected version - 0x20240401

[1] relevant FAQ
The model artifacts take a conversative stance to versioning, even for such similar versions as 10_01_03_00 and 10_01_04_00

I will hold off on recommending you port to 11.1 SDK. It is stable and performing well, but would be larger change on your side. Let's ensure the issue isn't a versioning detail on the artifacts

11.2 SDK timeline is December 2025 at the earliest

[1] [FAQ] EDGE-AI-STUDIO: Is SDK version important for Edge AI and TI Deep learning (TIDL) with C7x on AM6xA SoCs [AM62A, AM67A, AM68A, AM68PA, AM69A]?

0 Jason Zeng 4 months ago in reply to Reese Grimsley

Intellectual 375 points

Hi Reese

Sorry for the late ,I've been busy with other tasks these past few days. Here's the output I got after running /opt/vx_app_arm_remote_log.out– there are indeed prints indicating a Network version mismatch.

root@am62axx-evm:/opt/edgeai-gst-apps#  /opt/vx_app_arm_remote_log.out
[MCU1_0]      0.033625 s: CIO: Init ... Done !!!
[MCU1_0]      0.033663 s: APP: Init ... !!!
[MCU1_0]      0.033696 s: MEM: Init ... !!!
[MCU1_0]      0.033712 s: MEM: Created heap (DDR_LOCAL_MEM, id=0, flags=0x00000004) @ af000000 of size 16777216 bytes !!!
[MCU1_0]      0.033763 s: MEM: Init ... Done !!!
[MCU1_0]      0.033779 s: IPC: Init ... !!!
[MCU1_0]      0.033795 s: IPC: 3 CPUs participating in IPC !!!
[MCU1_0]      0.034092 s: IPC: Waiting for HLOS to be ready ... !!!
[MCU1_0]      0.038436 s: #Sciserver Version: v2023.11.0.0REL.MCUSDK.MM.NN.PP.bb
[MCU1_0]      0.041209 s: ##RM_PM_HAL Version: vMM.NN.PP
[MCU1_0]      0.044156 s: ##Starting Sciserver..... PASSED
[MCU1_0]     13.131625 s: IPC: HLOS is ready !!!
[MCU1_0]     13.131709 s: IPC: Init ... Done !!!
[MCU1_0]     13.131734 s: APP: Syncing with 2 CPUs ... !!!
[MCU1_0]     13.131757 s: APP: Syncing with 2 CPUs ... Done !!!
[MCU1_0]     13.131777 s: REMOTE_SERVICE: Init ... !!!
[MCU1_0]     13.131853 s: REMOTE_SERVICE: Init ... Done !!!
[MCU1_0]     13.131879 s: FVID2: Init ... !!!
[MCU1_0]     13.131905 s: FVID2: Init ... Done !!!
[MCU1_0]     13.131921 s: VHWA: VPAC Init ... !!!
[MCU1_0]     13.131936 s: SCICLIENT: Sciclient_pmSetModuleState module=219 state=2
[MCU1_0]     13.132022 s: SCICLIENT: Sciclient_pmSetModuleState success
[MCU1_0]     13.132143 s: VHWA: LDC Init ... !!!
[MCU1_0]     13.132238 s: VHWA: LDC Init ... Done !!!
[MCU1_0]     13.132262 s: VHWA: MSC Init ... !!!
[MCU1_0]     13.132863 s: VHWA: MSC Init ... Done !!!
[MCU1_0]     13.132882 s: VHWA: VISS Init ... !!!
[MCU1_0]     13.133539 s: VHWA: VISS Init ... Done !!!
[MCU1_0]     13.133568 s: VHWA: VPAC Init ... Done !!!
[MCU1_0]     13.133595 s:  VX_ZONE_INFO: Globally Enabled VX_ZONE_ERROR
[MCU1_0]     13.133618 s:  VX_ZONE_INFO: Globally Enabled VX_ZONE_WARNING
[MCU1_0]     13.133640 s:  VX_ZONE_INFO: Globally Enabled VX_ZONE_INFO
[MCU1_0]     13.134277 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:162] registered kernel com.ti.capture.scalar_sink on target MCU1-0
[MCU1_0]     13.134388 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:162] registered kernel com.ti.capture.scalar_source on target MCU1-0
[MCU1_0]     13.134471 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:162] registered kernel com.ti.capture.scalar_sink2 on target MCU1-0
[MCU1_0]     13.134563 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:162] registered kernel com.ti.capture.scalar_source2 on target MCU1-0
[MCU1_0]     13.134652 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:162] registered kernel com.ti.capture.scalar_intermediate on target MCU1-0
[MCU1_0]     13.134745 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:162] registered kernel com.ti.test_kernels.scalar_intermediate_2 on target MCU1-0
[MCU1_0]     13.134800 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:162] registered kernel com.ti.test_kernels.scalar_source_error on target MCU1-0
[MCU1_0]     13.134855 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:162] registered kernel com.ti.test_kernels.scalar_source_obj_array on target MCU1-0
[MCU1_0]     13.134909 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:162] registered kernel com.ti.test_kernels.scalar_sink_obj_array on target MCU1-0
[MCU1_0]     13.134964 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:162] registered kernel com.ti.test_kernels.pyramid_intermediate on target MCU1-0
[MCU1_0]     13.135018 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:162] registered kernel com.ti.test_kernels.pyramid_source on target MCU1-0
[MCU1_0]     13.135071 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:162] registered kernel com.ti.test_kernels.pyramid_sink on target MCU1-0
[MCU1_0]     13.135180 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:162] registered kernel com.ti.test_kernels.test_target on target MCU1-0
[MCU1_0]     13.135238 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:162] registered kernel com.ti.capture.image_intermediate on target MCU1-0
[MCU1_0]     13.135294 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:162] registered kernel com.ti.test_kernels.cmd_timeout_test on target MCU1-0
[MCU1_0]     13.135461 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:162] registered kernel com.ti.test_kernels.tiovx_overhead on target MCU1-0
[MCU1_0]     13.135580 s:  VX_ZONE_INFO: [tivxPlatformCreateTargetId:134] Added target MCU1-0
[MCU1_0]     13.135658 s:  VX_ZONE_INFO: [tivxPlatformCreateTargetId:134] Added target VPAC_LDC1
[MCU1_0]     13.135731 s:  VX_ZONE_INFO: [tivxPlatformCreateTargetId:134] Added target VPAC_MSC1
[MCU1_0]     13.135807 s:  VX_ZONE_INFO: [tivxPlatformCreateTargetId:134] Added target VPAC_MSC2
[MCU1_0]     13.135886 s:  VX_ZONE_INFO: [tivxPlatformCreateTargetId:134] Added target VPAC_VISS1
[MCU1_0]     13.135918 s:  VX_ZONE_INFO: [tivxInitLocal:126] Initialization Done !!!
[MCU1_0]     13.135943 s:  VX_ZONE_INFO: Globally Disabled VX_ZONE_INFO
[MCU1_0]     13.135962 s: APP: OpenVX Target kernel init ... !!!
[MCU1_0]     13.140465 s: APP: OpenVX Target kernel init ... Done !!!
[MCU1_0]     13.140491 s: VISS REMOTE SERVICE: Init ... !!!
[MCU1_0]     13.140538 s: VISS REMOTE SERVICE: Init ... Done !!!
[MCU1_0]     13.140557 s: APP: Init ... Done !!!
[MCU1_0]     13.140574 s: APP: Run ... !!!
[MCU1_0]     13.140587 s: IPC: Starting echo test ...
[MCU1_0]     13.140710 s: APP: Run ... Done !!!
[MCU1_0]     13.141468 s: IPC: Echo status: a530-0[.] r5f0-0[s] c75ss0[P]
[C7x_1 ]      3.046190 s: CIO: Init ... Done !!!
[C7x_1 ]      3.046209 s: APP: Init ... !!!
[C7x_1 ]      3.046221 s: SCICLIENT: Init ... !!!
[C7x_1 ]      3.046283 s: SCICLIENT: DMSC FW version [10.1.8--v10.01.08 (Fiery Fox)]
[C7x_1 ]      3.046303 s: SCICLIENT: DMSC FW revision 0xa
[C7x_1 ]      3.046317 s: SCICLIENT: DMSC FW ABI revision 4.0
[C7x_1 ]      3.046332 s: SCICLIENT: Init ... Done !!!
[C7x_1 ]      3.046345 s: UDMA: Init ... !!!
[C7x_1 ]      3.046391 s: UDMA: Init ... Done !!!
[C7x_1 ]      3.046405 s: MEM: Init ... !!!
[C7x_1 ]      3.046418 s: MEM: Created heap (DDR_LOCAL_MEM, id=0, flags=0x00000004) @ b2000000 of size 117440512 bytes !!!
[C7x_1 ]      3.046446 s: MEM: Init ... Done !!!
[C7x_1 ]      3.046458 s: IPC: Init ... !!!
[C7x_1 ]      3.046470 s: IPC: 3 CPUs participating in IPC !!!
[C7x_1 ]      3.046673 s: IPC: Waiting for HLOS to be ready ... !!!
[C7x_1 ]     12.403333 s: IPC: HLOS is ready !!!
[C7x_1 ]     12.403408 s: IPC: Init ... Done !!!
[C7x_1 ]     12.403422 s: APP: Syncing with 2 CPUs ... !!!
[C7x_1 ]     13.131758 s: APP: Syncing with 2 CPUs ... Done !!!
[C7x_1 ]     13.131773 s: REMOTE_SERVICE: Init ... !!!
[C7x_1 ]     13.132396 s: REMOTE_SERVICE: Init ... Done !!!
[C7x_1 ]     13.132428 s:  VX_ZONE_INFO: Globally Enabled VX_ZONE_ERROR
[C7x_1 ]     13.132450 s:  VX_ZONE_INFO: Globally Enabled VX_ZONE_WARNING
[C7x_1 ]     13.132471 s:  VX_ZONE_INFO: Globally Enabled VX_ZONE_INFO
[C7x_1 ]     13.133155 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:162] registered kernel vx_tutorial_graph.phase_rgb on target DSP_C7-1
[C7x_1 ]     13.133207 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:162] registered kernel com.ti.capture.scalar_sink on target DSP_C7-1
[C7x_1 ]     13.133255 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:162] registered kernel com.ti.capture.scalar_source on target DSP_C7-1
[C7x_1 ]     13.133311 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:162] registered kernel com.ti.capture.scalar_sink2 on target DSP_C7-1
[C7x_1 ]     13.133361 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:162] registered kernel com.ti.capture.scalar_source2 on target DSP_C7-1
[C7x_1 ]     13.133407 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:162] registered kernel com.ti.capture.scalar_intermediate on target DSP_C7-1
[C7x_1 ]     13.133454 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:162] registered kernel com.ti.test_kernels.scalar_intermediate_2 on target DSP_C7-1
[C7x_1 ]     13.133502 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:162] registered kernel com.ti.test_kernels.scalar_source_error on target DSP_C7-1
[C7x_1 ]     13.133550 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:162] registered kernel com.ti.test_kernels.scalar_source_obj_array on target DSP_C7-1
[C7x_1 ]     13.133599 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:162] registered kernel com.ti.test_kernels.scalar_sink_obj_array on target DSP_C7-1
[C7x_1 ]     13.133647 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:162] registered kernel com.ti.test_kernels.pyramid_intermediate on target DSP_C7-1
[C7x_1 ]     13.133696 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:162] registered kernel com.ti.test_kernels.pyramid_source on target DSP_C7-1
[C7x_1 ]     13.133742 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:162] registered kernel com.ti.test_kernels.pyramid_sink on target DSP_C7-1
[C7x_1 ]     13.133789 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:162] registered kernel com.ti.test_kernels.test_target on target DSP_C7-1
[C7x_1 ]     13.133837 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:162] registered kernel com.ti.capture.image_intermediate on target DSP_C7-1
[C7x_1 ]     13.133885 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:162] registered kernel com.ti.test_kernels.test_target on target DSP_C7-1
[C7x_1 ]     13.133933 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:162] registered kernel com.ti.test_kernels.tiovx_overhead on target DSP_C7-1
[C7x_1 ]     13.134059 s:  VX_ZONE_INFO: [tivxPlatformCreateTargetId:134] Added target DSP_C7-1
[C7x_1 ]     13.134153 s:  VX_ZONE_INFO: [tivxPlatformCreateTargetId:134] Added target DSP_C7-1_PRI_2
[C7x_1 ]     13.134249 s:  VX_ZONE_INFO: [tivxPlatformCreateTargetId:134] Added target DSP_C7-1_PRI_3
[C7x_1 ]     13.134350 s:  VX_ZONE_INFO: [tivxPlatformCreateTargetId:134] Added target DSP_C7-1_PRI_4
[C7x_1 ]     13.134445 s:  VX_ZONE_INFO: [tivxPlatformCreateTargetId:134] Added target DSP_C7-1_PRI_5
[C7x_1 ]     13.134539 s:  VX_ZONE_INFO: [tivxPlatformCreateTargetId:134] Added target DSP_C7-1_PRI_6
[C7x_1 ]     13.134634 s:  VX_ZONE_INFO: [tivxPlatformCreateTargetId:134] Added target DSP_C7-1_PRI_7
[C7x_1 ]     13.134728 s:  VX_ZONE_INFO: [tivxPlatformCreateTargetId:134] Added target DSP_C7-1_PRI_8
[C7x_1 ]     13.134756 s:  VX_ZONE_INFO: [tivxInitLocal:126] Initialization Done !!!
[C7x_1 ]     13.134777 s:  VX_ZONE_INFO: Globally Disabled VX_ZONE_INFO
[C7x_1 ]     13.134792 s: APP: OpenVX Target kernel init ... !!!
[C7x_1 ]     13.135013 s: APP: OpenVX Target kernel init ... Done !!!
[C7x_1 ]     13.135028 s: APP: Init ... Done !!!
[C7x_1 ]     13.135041 s: APP: Run ... !!!
[C7x_1 ]     13.135052 s: IPC: Starting echo test ...
[C7x_1 ]     13.135150 s: APP: Run ... Done !!!
[C7x_1 ]     13.141554 s: IPC: Echo status: a530-0[.] r5f0-0[P] c75ss0[s]
[C7x_1 ]    104.441908 s:  VX_ZONE_ERROR: [tivxKernelTIDLCreate:960] Network version - 0x20250131, Expected version - 0x20241120
[C7x_1 ]    233.075785 s:  VX_ZONE_ERROR: [tivxKernelTIDLCreate:960] Network version - 0x20250131, Expected version - 0x20241120
[C7x_1 ]    358.912103 s:  VX_ZONE_ERROR: [tivxKernelTIDLCreate:960] Network version - 0x20250131, Expected version - 0x20241120

Does this imply that I need to compile the model in a complete 10.1 environment?

Jason

0 Reese Grimsley 4 months ago in reply to Jason Zeng

TI__Mastermind 18176 points

Hi Jason,

No problem on response time. This is the output I was expecting.

You are on the 10.1 SDK now, yes? I suppose that you compiled with 10_01_03_00 edgeai-tidl-tools, based on what I can learn from tags in arm-tidl. The 'expected version' aligns with 10_01_00_01 / 10_01_02_00, and 10_01_04_00 and yours with 10_01_03_00. [1,2,3]

Yes, this implies you need to compile with 10_01_04_00 tidl_tools version to target 10.1 SDK. Looks like compiling for 10_01_03 targeting updated 10.0 SDK does not produce artifacts that work with 10.1 SDK as well.

[1] https://git.ti.com/cgit/processor-sdk-vision/arm-tidl

[2] https://git.ti.com/cgit/processor-sdk-vision/arm-tidl/tree/rt/inc/itidl_ti.h?h=REL.PSDK.ANALYTICS.10.01.00.04#n2019

[3] https://git.ti.com/cgit/processor-sdk-vision/arm-tidl/tree/tiovx_kernels/tidl/dsp/vx_tidl_target.c?h=REL.PSDK.ANALYTICS.10.01.00.04#n955

BR,

Reese

0 Jason Zeng 4 months ago in reply to Reese Grimsley

Intellectual 375 points

Hi Reese,

Currently, I have re-established the drawing process using version 10.01.00.05, and asked the algorithm team members to export the algorithm model and libraries using the tidl version 10_01_00_04. However, this error still persists. Could you please try to reproduce this issue on your end? Based on my current testing, the problem occurs whenever the algorithm is running with high CPU usage.

[  229.334824] tidss 30200000.dss: CRTC1 SYNC LOST: (irq 200800)
[  229.340686] tidss 30200000.dss: Plane1 underflow (irq 200800)
root@am62axx-evm:/opt/edgeai-gst-apps# ps -ef | grep "irqbalance"
root         402       1  0 02:06 ?        00:00:00 /usr/sbin/irqbalance --foreground
root        1617    1274  0 02:12 ttyS2    00:00:00 grep irqbalance

BR,

Jason

0 Jason Zeng 4 months ago in reply to Reese Grimsley

Intellectual 375 points

Hi Reese

Here's a simple reproduction method: Open multiple telnetd terminals, run the following application on one terminal and execute the gstreamer command for video output on another terminal to reproduce the issue.

gst-launch-1.0 videotestsrc  pattern=0 is-live=true ! \
video/x-raw,width=1920,height=1080,format=NV12,framerate=10/1 ! \
kmssink driver-name=tidss sync=true can-scale=false

here is the demo , model and test picture

test_demo.tar.gz

The compressed package contains an edgeai-tiovx-ti_libtestapplication, two algorithm model binary files, and an input test image. The run command example is as follows:

./edgeai-tiovx-ti_libtest /root/BSD_10_01_w352h60_io_1.bin /root/BSD_10_01_w352h60_net.bin testv.png output.png 352 608

Screen tearing can be observed after simultaneously running the algorithm and video output, which will help your team pinpoint the issue.

SDK version ： 10.01.00.05

Jason

0 Reese Grimsley 4 months ago in reply to Jason Zeng

TI__Mastermind 18176 points

Hi Jason,

I think I've reproduced this, but I wasn't able to run your application due to a missing library for libalgtiovxbase.so.0.1.0

On my side, I run the test pattern pipeline. When adding high CPU load, there is no dropped frames or sync lost. When I add high DDR load, I do see dropped frames, but no sync lost or CRTC1 SYNC LOST messages

Now when I turn off the stress-ng commands that were creating a load and run a heavy model (semantic segmentation, deeplabv3), I do see this error that you are mentioning. Interestingly, there are no actual frames being dropped here, but the screen tearing is present. Other heavy models have a similar effect

Altering IRQ affinity for tidss (display subsystem) to avoid CPU0 (where the other cores like C7x have interrupts serviced) does not seem to impact this issue. I will need to log this with our development team to investigate more deeply. This needs to be root caused.

I ran a quick test one 11.1 SDK as well -- this issue is present there too under the same conditions.

Lightweight models do not cause this effect, which is interesting because heavier models will have fewer IRQ's passed per unit time versus faster models.

So in summary, this is present on multiple SDK versions and we have reproduced this. The root cause will require some effort. It is important to know that this effect was not otherwise present in the 10.0 SDK until 10.1 TIDL was incorporated

BR,
Reese

0 Jason Zeng 4 months ago in reply to Reese Grimsley

Intellectual 375 points

Hi Reese

Thank you for the detailed investigation. We look forward to you and your development team resolving this issue.

Meanwhile, I apologize for forgetting to provide the so file earlier. Here it is.

Your forum doesn't support direct uploads of .so.0.1.0 files, as it will trigger an error. Therefore, I've compressed it into an archive.

libalgtiovxbase.so.0.1.rar

Jason

0 Reese Grimsley 4 months ago in reply to Jason Zeng

TI__Mastermind 18176 points

Thanks Jason. I think I have the information needed for our development team to investigate more deeply.

They've suggested this is related to the QoS for DDR going to the display-subsystem may need to be changed so priority is increased (I'm still seeking clarity on how this is implemented). They've noted that the issue is probably a result of high DDR traffic on C7xMMA for the TIDL model, and this may be getting different QoS vs. A53 DDR traffic (or at least what stress-ng DDR load creates).

Smaller models with more internal memory reuse (i.e. lower DDR utilization) will mitigate this issue, but I have concern that it does not eliminate it entirely, especially if you have bursty periods of DDR utilization during your application.

I appreciate your patience here.

BR,,
Reese

0 Reese Grimsley 3 months ago in reply to Reese Grimsley

TI__Mastermind 18176 points

Hi Jason,

Wanted to give an update, although we are in a holiday week here in the US

I've taken another look at the original 10.0 SDK vs. the patched version with TIDL 10.1.3. I actually see the same type of issue on 10.0, albeit more rarely and only with more DDR-heavy models (higher resolutions will generally come with higher DDR load)

We've encountered a similar issue on another SoC (TDA4VH), in which C7x traffic was causing display starvation and the same kind of screen tearing effect. The engineers who resolved that issue confirmed that it was indeed coming from QoS / CoS prioritization of DDR traffic. This does explain why only DDR traffic from C7x causes such screen tearing, whereas even higher DDR loads from A53 does not.

I would consider this issue effectively root-caused, so now the task is to implement and provide a fix for this. That fix is expected to be in u-boot within the SDK around the board-support/ti-u-boot-2024.04+git/drivers/ram/k3-ddrss portion, so when that is ready, it will entail a rebuild of u-boot components.

Thank you for the continued patience here.

0 Jason Zeng 3 months ago in reply to Reese Grimsley

Intellectual 375 points

Hi Reese

Thank you for the update, and I truly appreciate you responding to my message during your holiday. I've noted the situation and look forward to the resolution.

I noticed that you mentioned updating in U-boot to fix this issue. Could it also be resolved by starting through SBL? In our application, we start through SBL and have reduced U-boot to achieve a faster startup time.

Hope you enjoy your break!

Jason

0 Reese Grimsley 3 months ago in reply to Jason Zeng

TI__Mastermind 18176 points

Hi Jason,

It is that low-level configuration of DDR traffic prioritization that would need to change. SBL and u-boot may differ in their implementation, but I would assume that these use the same configurations by default.

Thus far, the registers (from AM62A TRM section 14.5.5) and functional description (9.1 and 9.1.3.2 Class of Service (CoS)) are suggesting that default SDK is not prioritizing any particular interface's DDR traffic, based on the MMR register values I query at runtime. I will need to work with someone with more familiarity of the low-level configs here.

BR,
Reese

+1 Reese Grimsley 3 months ago in reply to Reese Grimsley

TI__Mastermind 18176 points

Hi Jason,

More news in a positive direction. I have something for you to try:

devmem2 0x0f300030 w 0x11111111 #set default priority of DDR traffic slightly lower

You can run this command at runtime to change the memory mapped register value. The mention for u-boot and SBL is still relevant such that it is done at startup, but I've learned this is not required.

When I run this command before running the testvideosrc gstreamer pipeline and a DDR heavy TIDL model that showed this sync lost error, I no longer see the linux kernel messages for sync-lost, nor do I see screen tearing effects.

I will call this a workaround, but I'd like to dig a bit deeper here. That command above sets the V2A_LPT_DEF_PRI_MAP_REG register (section 14.5.5.7.1 of the AM62A TRM) to treat map all DDR traffic to slightly below the highest priority (0 is highest, 7 is lowest). This register translates priorities from one domain (VBUSM) to the DDR controller's domain priorities.

I have noted that the value 0x11111111 (default is 0x0) give correct behavior. Once I use 0x00000111, the erroneous behavior returns. I would like to have a more precise configuration here, so we are only changing DDR prioritization for the necessary channels / cores. There are additional registers for configuring a set of "Route ID's" to a particular range of priorities, which would be the preferred solution.

I will also note that I have not seen any change in the TIDL AI performance, but I am also not loading various cores with loads such that C7x would get lower priority treatment (which could very well reduce TIDL performance)

Can you try the above command, run you application, and confirm if the issue persists or not?

Edit: I did a bit more testing. I find that the deprioritizing C7x traffic does not effect the original error, but deprioritizing some of the A53 traffic does.

devmem2 0x0f300024 w 0xf0000000 # change route 1 to include all A53 writes. 
### Can replace value above with 0xf000f010 to include A53 reads, but only writes seem to cause original issue
devmem2 0x0f300034 w 0x11111111 # downgrade route 1 priorities from 0x0's
devmem2 0x0f300030 w 0x0       # change default priorities back to 0x0's (highest)

This will allow C7x to remain at the high priority for DDR traffic. Reducing A53 priority may have other performance impact on the rest of the application, including TIDL (copying data to/from DDR_SHARED memory region is initiated by A53 -- this is how we share input/output tensors with the C7x through TI OpenVX)

BR,
Reese

0 Jason Zeng 3 months ago in reply to Reese Grimsley

Intellectual 375 points

Hi Reese

After I run these command, the screen tearing effects is no more exits!

devmem2 0x0f300024 w 0xf0000000 # change route 1 to include all A53 writes. 
devmem2 0x0f300034 w 0x11111111 # downgrade route 1 priorities from 0x0's
devmem2 0x0f300030 w 0x0        # change default priorities back to 0x0's (highest)

However, when I used the stress-ngcommand to increase the CPU load to 70%, a new problem emerged: the video frames became disordered. Please refer to the attached video for details.I believe this issue is not caused by the DDR modification, as I also observed the same problem without entering any commands.

BR,

Jason

0 Reese Grimsley 3 months ago in reply to Jason Zeng

TI__Mastermind 18176 points

Hi Jason,

Great, thanks for confirming!

Jason Zeng said:
stress-ngcommand to increase the CPU load to 70%, a new problem emerged

Okay, understood. Looks like frame skips / replays. I would suggest making a new thread for this issue -- we have solved the root of this one

For your full application, will you see CPU utilization this high? Does your application show frame-skips as well?

0 Jason Zeng 3 months ago in reply to Reese Grimsley

Intellectual 375 points

Hi Reese

OK, I make a new thread for this AM62A7: Frame skips when high CPU usage - Processors forum - Processors - TI E2E support forums

In my application, the CPU usage is usually not so high. However, this is still a concerning point. After all, this issue might not occur easily under low CPU usage but is not completely absent either.So being able to solve this issue would be the best outcome.

Jason