Because of the Thanksgiving holiday in the U.S., TI E2E™ design support forum responses may be delayed the week of November 23.

This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

AM62A7: Intermittent Kernel Panic and Boot Failure on Custom AM62A74 Board

Part Number: AM62A7
Other Parts Discussed in Thread: SK-AM62A-LP, , SYSCONFIG, PROCESSOR-SDK-AM62A

Dear Sir,

We are currently developing a custom hardware platform based on the AM62A74 SoC. The hardware has been designed with reference to the TI evaluation kit and according to our system architecture.

We have received the fabricated boards and initially we were able to successfully download the firmware, boot the system, and run the application as expected.

However, we are now facing an intermittent boot issue on some of the boards. We are observing unexpected kernel panics during the boot process, while some other boards with the same hardware design and software image are working normally.

The issue is intermittent and difficult for us to identify the root cause. We would appreciate your support in analyzing this issue.

We would specifically like your guidance on the following:

  1. Could this issue be related to the software image, Linux kernel version, U-Boot version, or other software components?
  2. Could you please review the attached boot/kernel panic logs and help us identify the possible root cause?
  3. Could this behavior be related to DDR initialization, memory timing, power sequencing, or hardware-level variations?
  4. Are there any specific AM62A74 hardware checks, boot logs, debug registers, or diagnostic procedures that you recommend for troubleshooting intermittent kernel panics?
  5. Is there any recommended TI software release, SDK version, kernel version, or bootloader version that we should use for our custom AM62A74 board?
  6. Since some boards are working correctly while others are experiencing the issue, could you please advise us on the best way to distinguish between a hardware issue and a software/image-related issue?older_boot_with_latest_kernel_2.txt 

We have attached the relevant kernel panic and boot logs for your reference.

Could you please analyze the logs and provide your recommendations on the possible root cause and the next steps for debugging?

Your support in resolving this issue would be highly appreciated.

Best Regards,
Samudralankaiah

  • Hello Samudra,

    Could this issue be related to the software image, Linux kernel version, U-Boot version, or other software components?

    Since the same firmware image works fine on some boards and panics on others, this strongly points toward a board-level  variation. Rather than a generic kernel/U-Boot bug, a genuine SW bug would normally reproduce consistently across all boards running the identical image. That said, marginal timing issues in DDR/boot configuration can make software "expose" a hardware weakness intermittently, so both angles are worth checking.

    These are some typical hardware-side root causes for this exact pattern:

    • DDR PHY/controller timing margins— border-line signal integrity, layout deviations, or component tolerance from board to board (e.g., DDR IBIS/DDR4 or LPDDR4 layout mismatches vs. TI's layout guidelines). LPDDR4 designs must copy the SK-AM62A-LP layout exactly (trace geometry, stack-up, spacing) to guarantee margin. Any board-to-board PCB fabrication variation (via a different fab run, impedance tolerance, etc.) can push some boards outside DDR training margin — causing intermittent training/DDR-init failures that look like early kernel panics on a subset of boards.
    • Power sequencing / rail stability— MCU_PORz (cold reset) deasserted before supplies are fully ramped and stable, or PMIC rail timing variance between boards.
    • Crystal/oscillator start-up time variance— if MCU_OSC0 hasn't stabilized before boot mode pins are latched

    PCB/component tolerance in DDR routing, decoupling cap placement (per PDN target impedance tables), or backup/boot mode pin float/mis-termination.

    • Marginal solder joints or component placement differences across fabrication lots.

    Could this behavior be related to DDR initialization, memory timing, power sequencing, or hardware-level variations?

    DDR configuration/timing on AM62A7 is generated via TI's SysConfig DDR tool (`k3-am62x-ddr-config.dtsi`) and consumed by U-Boot's wakeup-SPL (running on the R5 core) before Linux boots, so a marginal DDR config would typically panic very early, often even before kernel logs appear, or show up as random memory corruption/kernel panics under load. Our recommendation for custom board bring-up is to de-tune DDR to the slowest supported speed first (667 MHz / 1334 MT/s for DDR4 with DLL on, similarly conservative settings for LPDDR4) to rule out signal integrity margins, then incrementally increase speed once stability is confirmed on all boards. It's also common for new boards to show intermittent issues specifically at optimized/full DDR speed— de-tuning is the standard TI-recommended diagnostic step.

    For the Power sequencing, can you confirm that MCU_PORz (cold reset) is released only after all processor supplies have ramped AND the recommended reset-hold delay has elapsed for crystal/oscillator stabilization. You can review this in the AM62A7 datasheet in section 6.11.3.1 Reset Timing.

    Can you please check board-to-board variance in PMIC output timing/voltage accuracy — small tolerances in supply ramp/sequencing between boards from the same batch can cause exactly this "some boards work, some don't" symptom.

    Are there any specific AM62A74 hardware checks, boot logs, debug registers, or diagnostic procedures that you recommend for troubleshooting intermittent kernel panics?
    • What you can do is to compare U-Boot/SPL logs from a working vs. failing board side-by-side:
      • UART console, not just kernel log
      • Check DDR training/init messages, timing values, and any warnings.
    • Enable TIFS/SYSFW trace via MAIN_UART1 to get low-level firmware boot trace before Linux even starts, very useful for isolating DDR/power failures that occur before kernel logs are printed.
    • Check DDR calibration/training pass/fail status in the R5 SPL log (look for "DDR" training messages before "Starting ATF") — capture full, unmangled logs.
    • Swap SD/eMMC images between a good and a bad board to confirm the failure follows the board, not the media/image.
    • Enable kernel debug traces (menuconfig → Kernel Hacking) and rebuild with debug symbols to get a more useful panic decode. Please review the Linux Board Porting on AM62x Devices application note. Section 7.1 Kernel Debug Traces may help you. Here is a link to it:

    Linux Board Porting on AM62x Devices

    • Run the board with a detuned/minimal U-Boot DDR + peripheral configuration (lower DDR frequency, SD card at legacy 25MHz/3.3V) to see if the panic disappears. This isolates whether it's a DDR/SI margin issue.
    • Use a JTAG/DebugSS to capture more detail on the exact failure point.
    • Measure VCC_CORE, VDD_MCU_0V85 (or equivalent) and all critical rails with a scope/multimeter on both a "good" and a "bad" board, and compare against datasheet limits.
    • Check PMIC configuration/strap pins for correct pull-up rail selection.
    • Run i2cdetect on the failing boards to see whether the pca953x GPIO expander (and any PMIC on I2C) responds at all — compare address visibility between good/bad boards.
    • Capture full dmesg /regulator framework logs (cat /sys/kernel/debug/regulator/regulator_summary if you can reach a shell, or add regulator_dummy/ verbose regulator debug via kernel cmdline) to see exactly which regulator supply is stuck.

    Is there any recommended TI software release, SDK version, kernel version, or bootloader version that we should use for our custom AM62A74 board?

    Use the latest PROCESSOR-SDK-AM62A unless you have a specific compatibility reason to stay on an older version.

    Since some boards are working correctly while others are experiencing the issue, could you please advise us on the best way to distinguish between a hardware issue and a software/image-related issue?older_boot_with_latest_kernel_2.txt 

    A few important anomalies stand out in the log that you provided:

    • Two different bootloader/firmware builds appear in the same capture
      • First SPL: U-Boot SPL 2023.04-ti-gf9b96c67473 (Mar 19 2024), SYSFW 9.2.7
      • Second SPL: U-Boot SPL 2023.04-dirty (Jul 28 2025), SYSFW 9.3.7

    This means the log mixes two different boot attempts with different U-Boot builds and different TI System Firmware versions. If different boards (or even the same board across power cycles) are running different SYSFW/U-Boot versions, that alone can produce "intermittent" and board-dependent behavior.

    First step: confirm every board is flashed with the exact same, single boot image (tiboot3.bin/tispl.bin/u-boot.img) and the same SYSFW build.

    • I2C GPIO expander (pca953x) probe failures

    pca953x 1-0023: failed writing register: -121

    pca953x 1-0023: probe with driver pca953x failed with error -121

    pca953x 1-0022: failed writing register: -121

    Error -121 is -EREMOTEIO, i.e., the I2C bus got a NACK/no response from the device. This is a classic symptom of:

    • Marginal I2C signal integrity (pull-ups, bus loading, trace length)
    • The I/O expander not powered/sequenced yet when the driver probes
    • A hardware assembly issue (bad solder joint, wrong address strap) that could easily vary board-to-board — this is a strong candidate for your "some boards work, some don't" symptom.
    • Ethernet PHY not detected

    Could not get PHY for ethernet@8000000port@1: addr 0

    am65_cpsw_nuss_port ethernet@8000000: phy_connect() failed

    Again board-dependent— could be PHY strap/power/reset timing variation.

    • Boot appears to stall waiting on a regulator dependency

    platform fa00000... probe pending: platform: supplier regulator-3 not ready

    This is the last line of your log, the capture doesn't actually show a kernel panic/oops. It shows the boot stalling waiting for a regulator to become ready. This pattern (regulator supplier not ready → dependent device stuck in "probe deferred" indefinitely) is frequently caused by:

    • A PMIC rail that hasn't stabilized / sequenced correctly
    • A regulator that depends on GPIO control (e.g., load switch, LDO enable) that is marginal on some boards
    • Device tree regulator dependency chains that are sensitive to slight voltage/timing variation between boards

    Best Regards,

    Borislav Lazarkov

  • Dear Borislav,

    Thank you very much for your detailed analysis and recommendations.

    We have followed your suggestions and would like to share an update on the AM62A7 DVRU board bring-up and boot stability.

    Current Status

    • We have addressed the initial kernel panic and boot instability issues observed during the board bring-up.
    • The board is currently booting consistently with 667 MHz LPDDR4 configuration.
    • We have completed an automated reboot validation with 100 consecutive successful boots, with no kernel panics or memory corruption observed during this test.
    • We also identified and fixed some Device Tree configuration issues that were contributing to the observed problems.

    DDR Configuration

    As recommended, we first reduced the DDR operating frequency to establish a stable baseline.

    We generated the DVRU-specific 4 GB LPDDR4 configuration using TI SysConfig and validated the configuration at 667 MHz. With this configuration, the board is currently stable during repeated boot testing.

    We are using this 667 MHz configuration as our current baseline to eliminate DDR timing/margin-related issues.

    SD Card

    We also updated the SD card power and configuration to avoid the previously observed bus corruption and data-abort issues.

    Remaining Kernel Panic Issue

    Although the board is now stable during boot and has passed 100 consecutive reboot cycles, we would like to clarify that the kernel panic issue is still present intermittently on all of our boards.

    The failure is not limited to a particular board. It can occur intermittently while the system is executing an application, rather than consistently during the boot process.

    Therefore, we would like to continue investigating this as a possible DDR/memory stability, peripheral, power, or software/Device Tree interaction issue.

    Our current approach is:

    1. First validate the system at the lower DDR frequency of 667 MHz.
    2. Fix and validate the Device Tree configuration issues.
    3. Establish 667 MHz as the stable baseline.
    4. Incrementally increase the LPDDR4 frequency and validate system stability at each step.
    5. Monitor whether the intermittent kernel panic reappears at a particular DDR frequency or during specific application workloads.

    Could you please advise us on the next recommended validation/debug steps to identify the root cause of the remaining intermittent kernel panic?

    In particular, we would appreciate your guidance on:

    • Recommended DDR stress/memory tests for validating LPDDR4 stability on the AM62A7.
    • Any TI-recommended tests to identify DDR timing or signal-integrity margin issues.
    • Recommended kernel debug configuration or tracing to capture more information when the panic occurs during application execution.
    • Any specific AM62A7 registers, DDR PHY status, or System Firmware/TIFS logs that we should capture when the failure occurs.
    • Whether there are any additional power-rail or reset checks that you recommend during the application workload.
    • Any recommended TI Processor SDK / Linux kernel / U-Boot version that we should use as the baseline for this validation.

    We can also provide the latest kernel panic logs and complete boot logs from the failing condition if required.

    Thank you again for your continued support and guidance. We would appreciate your recommendations on the next steps to systematically isolate and resolve the remaining intermittent kernel panic issue.

    Best Regards,
    Samudralankaiah.

  • Hello Samudralankaiah,

    Please share the latest kernel panic logs and full boot logs from the failing condition.

    Recommended DDR stress/memory tests for validating LPDDR4 stability on the AM62A7.

    Run these concurrently with your actual application (not instead of it), since the panic is workload-triggered:

    • memtester — pattern-based read/write stress on a large chunk of free RAM (memtester 2048M 20). This is good for catching stuck-bit / pattern-sensitive faults.
    • stressapptest (Google's stressapptest -M <MB> -s <seconds> -m <threads> --pause_delay 0) — specifically designed to catch intermittent DRAM errors under sustained bandwidth load; widely used for exactly this kind of "rare corruption under load" investigation.
    • stress-ng --vm and --cache — combine CPU+cache+memory pressure to mimic your app's contention pattern.
    • Concurrent A53 (Linux) + DSP/C7x/R5F (RTOS firmwares) memory access if your application uses the C7x DSP/accelerators on AM62A7— cross-core DDR contention is a common trigger for intermittent corruption that pure Linux-side memtest won't catch. If your workload uses the vision/AI accelerators, be sure your stress test also drives those simultaneously.
    • Run each test for multiple hours to overnight, in a loop, with timestamped logging, since your panic is intermittent.

    Fix and validate the Device Tree configuration issues.
    • Vary ambient temperature (cold start vs. after prolonged thermal soak under load)— DDR margin often shrinks as die temp rises, and this is a classic cause of load-dependent (not boot-time) panics.
    • Vary input voltage/rail loading— run the stress test while also driving other high-current peripherals (USB, display, camera, DSP) simultaneously with DDR stress, since power droop under combined load is a common trigger that boot-only testing won't reveal.
    • Re-run the same 667 MHz-vs-higher-frequency comparison under sustained load, not just boot— a frequency that survives 100 boots may still fail after 30 minutes of DDR-heavy activity once self-heating sets in.
    • If you have scope access: capture VDD_DDR and VDDQ rail ripple/droop while the application runs (not just at idle/boot)— transient droop during peak DDR activity is a good hardware-margin indicator.
    • Since you've already generated your DDR config via SysConfig, also double check whether write/read leveling margins were flagged as marginal in the SysConfig DDR tool output. It reports margin numbers when you generate the timing.

    Recommended kernel debug configuration or tracing to capture more information when the panic occurs during application execution.
    • Enable panic_on_oops=1 and set kernel.panic=<N> for auto-reboot, but more importantly enable pstore/ramoops so panic logs survive the reboot:
      • Reserve a DDR region in the device tree for ramoops (compatible = "ramoops"), so dmesg /panic backtrace from the crash is preserved in /sys/fs/pstore/ after reboot — critical for an intermittent, unattended failure you can't catch live on a serial console.
    • Enable kdump/kexec if storage/RAM allows, to get a full vmcore for offline analysis with crash.
    • ftrace — run trace-cmd record -e sched\* -e irq\* -e printk continuously with a ring buffer, so you get the last N events leading to the panic (function tracer/irqsoff tracer especially useful if the panic correlates with IRQ latency).
    • ignore_loglevel and earlycon kept enabled on the serial console so you don't lose early panic messages.
    • If it's an application-triggered panic, also capture which subsystem is in the backtrace (DDR/EDAC, DMA, IOMMU, driver-specific) — that'll immediately tell you if it's memory-related or a specific peripheral driver.

    Any specific AM62A7 registers, DDR PHY status, or System Firmware/TIFS logs that we should capture when the failure occurs.
    • EDAC (Error Detection and Correction), if enabled in your DDR controller config, will log correctable/uncorrectable ECC events to dmesg / /sys/devices/system/edac/— if your LPDDR4 doesn't have ECC, this won't apply, but worth confirming your part/config.
    • DDRSS controller status/training registers— the AM62Ax TRM documents the DDRSS (DDR Subsystem) register space including PHY general status registers (PGSR0/PGSR1-equivalent) that report calibration/training pass status and any post-training error flags. Dump these via devmem2 / busybox devmem at the DDRSS base address post-failure if you can catch the system in a hung/pre-panic state.
    • TIFS/SYSFW logs on the boot UART (before Linux hands off)— re-review the SYSFW abort/trace log if the panic is severe enough to also affect the secure firmware; SYSFW trace can be decoded with TI's k3-image-gen sysfw-trace-parser tool if you capture the trace buffer.
    • PMIC status/fault registers— many PMICs used on AM62A EVM/DVRU boards (TPS6594 family) have fault/OV/UV/OT status registers readable over I2C— read these immediately after a panic (or continuously log them) to check whether a rail briefly dropped/spiked right before the crash.

    Whether there are any additional power-rail or reset checks that you recommend during the application workload.
    • Continuously log PMIC rail voltages and temperature (via I2C polling script, e.g., every 100 ms) throughout your stress run, correlated with a timestamp, so you can line up any dip/spike with the exact panic time.
    • Watch for PMIC reset/warm-reset events — check if the SoC's reset cause register (RESET_CAUSE or WKUP_CTRL reset status in AM62Ax TRM) shows a brown-out or watchdog reset rather than a true kernel panic — this would strongly indicate a power-rail droop under load rather than a memory/software fault.
    • Verify your decoupling and bulk capacitance near the DDR rails and the main VDD_CORE rail are adequate for the current transients your application draws.
    • Check the PMIC's power-good / interrupt (nINT) line— if it toggles during the workload, that may be a hardware sign of a rail issue rather than pure software.

    Any recommended TI Processor SDK / Linux kernel / U-Boot version that we should use as the baseline for this validation.

    Since your log showed mixed U-Boot/SYSFW builds earlier, this is still the most important housekeeping item:

    • Standardize all boards on one official TI Processor SDK Linux release for AM62A (pick the latest stable/LTS release available for AM62A7 from TI's software download page— check the release notes for the exact version marked "Long Term Support" as of your build date, since TI periodically updates the LTS designation).
    • Confirm the U-Boot, kernel, and SYSFW/TIFS versions are all sourced from the same SDK release, don't mix a newer kernel with an older SYSFW or vice versa, as was seen in your earlier logs.

    Best Regards,

    Borislav Lazarkov

  • Hi Sir,

    Please find attached the kernel log related to the kernel panic issue.

    Thank you very much for your detailed explanation and valuable guidance.

    We have started running both the memory test and stress-ng continuously. We will share the complete logs with you tomorrow.

    We also have a small clarification regarding the frequency settings. We are currently setting the F1 and F2 values to 698 MHz. However, in the DTSI file, the value appears to be reduced by half. Could you please clarify whether 698 MHz represents the actual operating frequency or the data rate?

    Also, the F2 value is not editable in the tool; the tool is automatically taking the same value as F1.

    We are a little confused about the difference between frequency and data rate in this context. Could you please clarify this and advise us on the correct values to configure?

    Thank you once again for your valuable guidance and support.

    Best Regards,
    Samudralankaiah

    U-Boot SPL 2023.04-dirty (Aug 14 2026 - 17:08:13 +0530)
    SYSFW ABI: 3.1 (firmware rev 0x0009 '9.2.7--v09.02.07 (Kool Koala)')
    am62a_init: board_init_f done
    SPL initial stack usage: 17064 bytes
    am62a_init: spl_boot_device: devstat = 0x243 bootmedia = 0x8 bootindex = 0
    Trying to boot from MMC2
    am62a_init: spl_boot_device: devstat = 0x243 bootmedia = 0x8 bootindex = 0
    Authentication passed
    am62a_init: spl_boot_device: devstat = 0x243 bootmedia = 0x8 bootindex = 0
    Authentication passed
    am62a_init: spl_boot_device: devstat = 0x243 bootmedia = 0x8 bootindex = 0
    Authentication passed
    am62a_init: spl_boot_device: devstat = 0x243 bootmedia = 0x8 bootindex = 0
    Authentication passed
    am62a_init: spl_boot_device: devstat = 0x243 bootmedia = 0x8 bootindex = 0
    Authentication passed
    Starting ATF on ARM64 core...
    
    NOTICE:  BL31: v2.10.0(release):v2.10.0-367-g00f1ec6b87-dirty
    NOTICE:  BL31: Built : 16:09:05, Feb  9 2024
    
    U-Boot SPL 2023.04-dirty (Aug 14 2026 - 17:08:01 +0530)
    SYSFW ABI: 3.1 (firmware rev 0x0009 '9.2.7--v09.02.07 (Kool Koala)')
    am62a_init: board_init_f done
    am62a_init: spl_boot_device: devstat = 0x243 bootmedia = 0x8 bootindex = 0
    Trying to boot from MMC2
    am62a_init: spl_boot_device: devstat = 0x243 bootmedia = 0x8 bootindex = 0
    Authentication passed
    am62a_init: spl_boot_device: devstat = 0x243 bootmedia = 0x8 bootindex = 0
    Authentication passed
    
    
    U-Boot 2023.04-dirty (Aug 14 2026 - 17:08:01 +0530)
    
    SoC:   AM62AX SR1.0 HS-FS
    Model: Texas Instruments AM62A7 SK
    DRAM:  2 GiB (effective 4 GiB)
    Core:  58 devices, 28 uclasses, devicetree: separate
    MMC:   mmc@fa10000: 0, mmc@fa00000: 1
    Loading Environment from nowhere... OK
    In:    serial@2800000
    Out:   serial@2800000
    Err:   serial@2800000
    Net:   Could not get PHY for ethernet@8000000port@1: addr 0
    am65_cpsw_nuss_port ethernet@8000000port@1: phy_connect() failed
    No ethernet found.
    
    Hit any key to stop autoboot:  0 
    switch to partitions #0, OK
    mmc1 is current device
    SD/MMC found on device 1
    Failed to load 'boot.scr'
    675 bytes read in 23 ms (28.3 KiB/s)
    Loaded env from uEnv.txt
    Importing environment from mmc1 ...
    ## Error: "main_cpsw0_qsgmii_phyinit" not defined
    19376640 bytes read in 827 ms (22.3 MiB/s)
    58214 bytes read in 27 ms (2.1 MiB/s)
    Working FDT set to 88000000
    3441 bytes read in 25 ms (133.8 KiB/s)
    ## Flattened Device Tree blob at 88000000
       Booting using the fdt blob at 0x88000000
    Working FDT set to 88000000
       Loading Device Tree to 000000008feee000, end 000000008fffffff ... OK
    Working FDT set to 8feee000
    
    Starting kernel ...
    
    [    0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd034]
    [    0.000000] Linux version 6.1.80-ti-g2e423244f8c0 (oe-user@oe-host) (aarch64-oe-linux-gcc (GCC) 11.4.0, GNU ld (GNU Binutils) 2.38.20220708) #1 SMP PREEMPT Wed Mar 20 14:43:33 UTC 2024
    [    0.000000] Machine model: Texas Instruments AM62A7 SK
    [    0.000000] earlycon: ns16550a0 at MMIO32 0x0000000002800000 (options '')
    [    0.000000] printk: bootconsole [ns16550a0] enabled
    [    0.000000] efi: UEFI not found.
    [    0.000000] Reserved memory: created CMA memory pool at 0x00000000c0000000, size 576 MiB
    [    0.000000] OF: reserved mem: initialized node linux,cma, compatible id shared-dma-pool
    [    0.000000] Reserved memory: created DMA memory pool at 0x0000000099800000, size 1 MiB
    [    0.000000] OF: reserved mem: initialized node c7x-dma-memory@99800000, compatible id shared-dma-pool
    [    0.000000] Reserved memory: created DMA memory pool at 0x0000000099900000, size 30 MiB
    [    0.000000] OF: reserved mem: initialized node c7x-memory@99900000, compatible id shared-dma-pool
    [    0.000000] Reserved memory: created DMA memory pool at 0x000000009b800000, size 1 MiB
    [    0.000000] OF: reserved mem: initialized node r5f-dma-memory@9b800000, compatible id shared-dma-pool
    [    0.000000] Reserved memory: created DMA memory pool at 0x000000009b900000, size 15 MiB
    [    0.000000] OF: reserved mem: initialized node r5f-dma-memory@9b900000, compatible id shared-dma-pool
    [    0.000000] Reserved memory: created DMA memory pool at 0x000000009c800000, size 1 MiB
    [    0.000000] OF: reserved mem: initialized node r5f-dma-memory@9c800000, compatible id shared-dma-pool
    [    0.000000] Reserved memory: created DMA memory pool at 0x000000009c900000, size 30 MiB
    [    0.000000] OF: reserved mem: initialized node r5f-dma-memory@9c900000, compatible id shared-dma-pool
    [    0.000000] Reserved memory: created DMA memory pool at 0x00000000a1000000, size 32 MiB
    [    0.000000] OF: reserved mem: initialized node edgeai-dma-memory@a1000000, compatible id shared-dma-pool
    [    0.000000] OF: reserved mem: initialized node edgeai_shared-memories, compatible id dma-heap-carveout
    [    0.000000] Reserved memory: created DMA memory pool at 0x00000000ae000000, size 288 MiB
    [    0.000000] OF: reserved mem: initialized node edgeai-core-heap-memory@ae000000, compatible id shared-dma-pool
    [    0.000000] Zone ranges:
    [    0.000000]   DMA      [mem 0x0000000080000000-0x00000000ffffffff]
    [    0.000000]   DMA32    empty
    [    0.000000]   Normal   [mem 0x0000000100000000-0x00000008ffffffff]
    [    0.000000] Movable zone start for each node
    [    0.000000] Early memory node ranges
    [    0.000000]   node   0: [mem 0x0000000080000000-0x00000000997fffff]
    [    0.000000]   node   0: [mem 0x0000000099800000-0x000000009b7fefff]
    [    0.000000]   node   0: [mem 0x000000009b800000-0x000000009e6fffff]
    [    0.000000]   node   0: [mem 0x000000009e700000-0x000000009e77ffff]
    [    0.000000]   node   0: [mem 0x000000009e780000-0x00000000a2ffffff]
    [    0.000000]   node   0: [mem 0x00000000a3000000-0x00000000adffffff]
    [    0.000000]   node   0: [mem 0x00000000ae000000-0x00000000bfffffff]
    [    0.000000]   node   0: [mem 0x00000000c0000000-0x00000000ffffffff]
    [    0.000000]   node   0: [mem 0x0000000880000000-0x00000008ffffffff]
    [    0.000000] Initmem setup node 0 [mem 0x0000000080000000-0x00000008ffffffff]
    [    0.000000] On node 0, zone DMA: 1 pages in unavailable ranges
    [    0.000000] psci: probing for conduit method from DT.
    [    0.000000] psci: PSCIv1.1 detected in firmware.
    [    0.000000] psci: Using standard PSCI v0.2 function IDs
    [    0.000000] psci: Trusted OS migration not required
    [    0.000000] psci: SMC Calling Convention v1.4
    [    0.000000] percpu: Embedded 20 pages/cpu s41064 r8192 d32664 u81920
    [    0.000000] Detected VIPT I-cache on CPU0
    [    0.000000] CPU features: detected: GIC system register CPU interface
    [    0.000000] CPU features: detected: ARM erratum 845719
    [    0.000000] alternatives: applying boot alternatives
    [    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 1032191
    [    0.000000] Kernel command line: console=ttyS2,115200n8 earlycon=ns16550a,mmio32,0x02800000 mtdparts=spi-nand0:512k(ospi_nand.tiboot3),2m(ospi_nand.tispl),4m(ospi_nand.u-boot),256k(ospi_nand.env),256k(ospi_nand.env.backup),98048k@32m(ospi_nand.rootfs),256k@130816k(ospi_nand.phypattern) root=PARTUUID=076c4a2a-02 rw rootfstype=ext4 rootwait
    [    0.000000] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes, linear)
    [    0.000000] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)
    [    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
    [    0.000000] software IO TLB: area num 4.
    [    0.000000] software IO TLB: mapped [mem 0x00000000fbfff000-0x00000000fffff000] (64MB)
    [    0.000000] Memory: 2808820K/4194300K available (11712K kernel code, 1258K rwdata, 3812K rodata, 1984K init, 438K bss, 795656K reserved, 589824K cma-reserved)
    [    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
    [    0.000000] rcu: Preemptible hierarchical RCU implementation.
    [    0.000000] rcu: 	RCU event tracing is enabled.
    [    0.000000] rcu: 	RCU restricting CPUs from NR_CPUS=256 to nr_cpu_ids=4.
    [    0.000000] 	Trampoline variant of Tasks RCU enabled.
    [    0.000000] 	Tracing variant of Tasks RCU enabled.
    [    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
    [    0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4
    [    0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
    [    0.000000] GICv3: GIC: Using split EOI/Deactivate mode
    [    0.000000] GICv3: 256 SPIs implemented
    [    0.000000] GICv3: 0 Extended SPIs implemented
    [    0.000000] Root IRQ handler: gic_handle_irq
    [    0.000000] GICv3: GICv3 features: 16 PPIs
    [    0.000000] GICv3: CPU0: found redistributor 0 region 0:0x0000000001880000
    [    0.000000] ITS [mem 0x01820000-0x0182ffff]
    [    0.000000] GIC: enabling workaround for ITS: Socionext Synquacer pre-ITS
    [    0.000000] ITS@0x0000000001820000: Devices Table too large, reduce ids 20->19
    [    0.000000] ITS@0x0000000001820000: allocated 524288 Devices @880800000 (flat, esz 8, psz 64K, shr 0)
    [    0.000000] ITS: using cache flushing for cmd queue
    [    0.000000] GICv3: using LPI property table @0x0000000880040000
    [    0.000000] GIC: using cache flushing for LPI property table
    [    0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000880050000
    [    0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.
    [    0.000000] arch_timer: cp15 timer(s) running at 200.00MHz (phys).
    [    0.000000] clocksource: arch_sys_counter: mask: 0x3ffffffffffffff max_cycles: 0x2e2049d3e8, max_idle_ns: 440795210634 ns
    [    0.000001] sched_clock: 58 bits at 200MHz, resolution 5ns, wraps every 4398046511102ns
    [    0.008505] Console: colour dummy device 80x25
    [    0.013093] Calibrating delay loop (skipped), value calculated using timer frequency.. 400.00 BogoMIPS (lpj=800000)
    [    0.023775] pid_max: default: 32768 minimum: 301
    [    0.028540] LSM: Security Framework initializing
    [    0.033360] Mount-cache hash table entries: 8192 (order: 4, 65536 bytes, linear)
    [    0.040940] Mountpoint-cache hash table entries: 8192 (order: 4, 65536 bytes, linear)
    [    0.050389] cblist_init_generic: Setting adjustable number of callback queues.
    [    0.057819] cblist_init_generic: Setting shift to 2 and lim to 1.
    [    0.064110] cblist_init_generic: Setting adjustable number of callback queues.
    [    0.071506] cblist_init_generic: Setting shift to 2 and lim to 1.
    [    0.077876] rcu: Hierarchical SRCU implementation.
    [    0.082784] rcu: 	Max phase no-delay instances is 1000.
    [    0.088341] Platform MSI: msi-controller@1820000 domain created
    [    0.094594] PCI/MSI: /bus@f0000/interrupt-controller@1800000/msi-controller@1820000 domain created
    [    0.103996] EFI services will not be available.
    [    0.108849] smp: Bringing up secondary CPUs ...
    [    0.114054] Detected VIPT I-cache on CPU1
    [    0.114140] GICv3: CPU1: found redistributor 1 region 0:0x00000000018a0000
    [    0.114156] GICv3: CPU1: using allocated LPI pending table @0x0000000880060000
    [    0.114201] CPU1: Booted secondary processor 0x0000000001 [0x410fd034]
    [    0.114810] Detected VIPT I-cache on CPU2
    [    0.114873] GICv3: CPU2: found redistributor 2 region 0:0x00000000018c0000
    [    0.114887] GICv3: CPU2: using allocated LPI pending table @0x0000000880070000
    [    0.114916] CPU2: Booted secondary processor 0x0000000002 [0x410fd034]
    [    0.115456] Detected VIPT I-cache on CPU3
    [    0.115523] GICv3: CPU3: found redistributor 3 region 0:0x00000000018e0000
    [    0.115537] GICv3: CPU3: using allocated LPI pending table @0x0000000880080000
    [    0.115563] CPU3: Booted secondary processor 0x0000000003 [0x410fd034]
    [    0.115627] smp: Brought up 1 node, 4 CPUs
    [    0.195363] SMP: Total of 4 processors activated.
    [    0.200177] CPU features: detected: 32-bit EL0 Support
    [    0.205446] CPU features: detected: CRC32 instructions
    [    0.210752] CPU: All CPU(s) started at EL2
    [    0.214949] alternatives: applying system-wide alternatives
    [    0.222106] devtmpfs: initialized
    [    0.233628] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
    [    0.243618] futex hash table entries: 1024 (order: 4, 65536 bytes, linear)
    [    0.267215] pinctrl core: initialized pinctrl subsystem
    [    0.271646] Unable to handle kernel NULL pointer dereference at virtual address 00000000000000c0
    [    0.272574] Internal error: Oops - Undefined instruction: 0000000002000000 [#1] PREEMPT SMP
    [    0.281544] Mem abort info:
    [    0.290075] Modules linked in:
    [    0.292918]   ESR = 0x0000000096000004
    [    0.292920] 
    [    0.292926] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 6.1.80-ti-g2e423244f8c0 #1
    [    0.296033]   EC = 0x25: DABT (current EL), IL = 32 bits
    [    0.299855] Hardware name: Texas Instruments AM62A7 SK (DT)
    [    0.301366]   SET = 0, FnV = 0
    [    0.308921] pstate: 600000c5 (nZCv daIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
    [    0.314340]   EA = 0, S1PTW = 0
    [    0.320028] pc : irq_enter_rcu+0x38/0x70
    [    0.323139]   FSC = 0x04: level 0 translation fault
    [    0.330248] lr : el1_interrupt+0x28/0x70
    [    0.333448] Data abort info:
    [    0.337447] sp : ffff80000932b920
    [    0.342423]   ISV = 0, ISS = 0x00000004
    [    0.346422] x29: ffff80000932b920
    [    0.349356]   CM = 0, WnR = 0
    [    0.352734]  x28: ffff000800108000
    [    0.356644] [00000000000000c0] user address but active_mm is swapper
    [    0.360021]  x27: 000000000000003b
    [    0.376467] x26: 0000000000000000 x25: ffff800009285028 x24: ffff80000932bb97
    [    0.383766] x23: 0000000060000005 x22: ffff8000080af9c4 x21: ffff80000932bab0
    [    0.391064] x20: ffff80000853f0c0 x19: ffff80000932b960 x18: ffffffffffffffff
    [    0.398362] x17: 000000000000001c x16: 000000001b881694 x15: ffff80008932ba27
    [    0.405660] x14: 0000000000000000 x13: 6d65747379736275 x12: 73206c7274636e69
    [    0.412958] x11: 702064657a696c61 x10: 6974696e69203a65 x9 : 7274636e69702064
    [    0.420256] x8 : 657a696c61697469 x7 : 205d353132373632 x6 : 322e30202020205b
    [    0.427554] x5 : 0000000000000000 x4 : 0000000000000000 x3 : 00000000ffffffff
    [    0.434852] x2 : 0000000000000040 x1 : 0000000000b10000 x0 : ffff000800108000
    [    0.442151] Call trace:
    [    0.444644]  irq_enter_rcu+0x38/0x70
    [    0.448296]  el1_interrupt+0x28/0x70
    [    0.451948]  el1h_64_irq_handler+0x18/0x2c
    [    0.456134]  el1h_64_irq+0x64/0x68
    [    0.459608]  console_emit_next_record.constprop.0+0x190/0x28c
    [    0.465484]  console_unlock+0x18c/0x21c
    [    0.469403]  vprintk_emit+0x1b4/0x2c4
    [    0.473144]  vprintk_default+0x38/0x4c
    [    0.476974]  vprintk+0xa4/0xc0
    [    0.480093]  _printk+0x60/0x8c
    [    0.483212]  pinctrl_init+0x20/0xc4
    [    0.486777]  do_one_initcall+0x54/0x1d0
    [    0.490694]  kernel_init_freeable+0x218/0x284
    [    0.495145]  kernel_init+0x24/0x130
    [    0.498710]  ret_from_fork+0x10/0x20
    [    0.502364] Code: 12183c00 7140401f 540041e0 52a01621 (d5387b00) 
    [    0.508591] ---[ end trace 0000000000000000 ]---
    [    0.513307] Kernel panic - not syncing: Oops - Undefined instruction: Fatal exception in interrupt
    [    0.522464] SMP: stopping secondary CPUs
    [    1.601998] SMP: failed to stop secondary CPUs 0,3
    [    1.606894] ---[ end Kernel panic - not syncing: Oops - Undefined instruction: Fatal exception in interrupt ]---
    
    

  • Hello Samudralankaiah,

    We also have a small clarification regarding the frequency settings. We are currently setting the F1 and F2 values to 698 MHz. However, in the DTSI file, the value appears to be reduced by half. Could you please clarify whether 698 MHz represents the actual operating frequency or the data rate?

    Data Rate = 2 × Clock Frequency

    This is because LPDDR4 transfers data on both the rising and falling edge of the clock— hence "Double Data Rate." In the AM62A7 datasheet in Table 6-2. Device Operating Performance Points is stated the exact relationship for the DDR PLL:

    "The DDR PLL output, which sources DDR0_CK0 and DDR0_CK0_n, is typically defined in units of frequency. So the 'DDR PLL Bypass' transaction rate is equal to 2x the DDR PLL output frequency when operating in bypass mode."

    So when you enter 698 MHz in the SysConfig tool's F1/F2 field:

    • 698 MHz is the actual DDR clock frequency (the real toggling rate of DDR_CK/CK_n on the PCB)
    • The effective data rate is 2 × 698 = 1396 MT/s
    • The DTSI ends up storing ~349 MHz (half of 698) because many of the DDR controller/PHY timing register fields and PLL divider calculations in the Denali IP are defined relative to a half-rate reference clock internally (this is normal/expected behavior of the DDR controller's internal clocking architecture, not a bug or a mistake in your configuration)

    Also, the F2 value is not editable in the tool; the tool is automatically taking the same value as F1.

    This is expected tool behavior when your design uses a single DDR operating point. F1 and F2 represent two selectable frequency sets in the DDR controller (used for frequency-switching designs, e.g., low-power vs. high-performance DDR modes). If your SysConfig project is configured for a single fixed DDR frequency, the tool automatically mirrors F1 into F2 since there's no second frequency point to define— this is normal, not a bug, and does not need to be manually edited.

    From this log, this panic happens during kernel boot (early kernel_init_freeable → pinctrl_init initcall), not during application execution. That's worth flagging since it's a different failure signature than the "intermittent during app execution" panics you described, it may be a separate/additional manifestation of the same underlying issue.

    The log shows:

    Unable to handle kernel NULL pointer dereference at virtual address 00000000000000c0
    Internal error: Oops - Undefined instruction ...
    pc : irq_enter_rcu+0x38/0x70
    lr : el1_interrupt+0x28/0x70
    Call trace:
     irq_enter_rcu+0x38/0x70
     el1_interrupt+0x28/0x70
     ...
     vprintk_emit -> console_unlock -> pinctrl_init -> do_one_initcall -> kernel_init_freeable
    

     This is a very significant finding, for this reason:

    • The fault occurred inside irq_enter_rcu, which is deep, generic core-kernel IRQ-entry/RCU bookkeeping code — completely unrelated to pinctrl_init itself. It only appears in this trace because an IRQ happened to fire while pinctrl_init() was calling printk().
    • This pattern — a fault in unrelated," heavily-used core kernel code, that is intermittent and not reproducible at a fixed location — is a classic fingerprint of memory corruption: a bit flip or corrupted read/write to DRAM (or occasionally L1/L2 cache) that hit a kernel data structure (in this case, likely an RCU-related per-CPU or IRQ context pointer) purely by chance, causing a crash somewhere essentially random in the code.

     

    What you can do as next steps are:

    • Remove app execution from the loop for now, and instead just run memtester / stressapptest continuously through many boot cycles at 667 MHz to see if you can reproduce this exact kind of random-location panic without any application running at all. If it reproduces under pure memory stress with no app, that's may confirm this is DDR/hardware-margin related, not software.
    • If you have kdump/ramoops (pstore) enabled, capture a few more of these crashes and check whether the faulting PC/address is different each time versus always the exact same address.
    • Also check dmesg for any earlier EDAC/ECC warnings before the panic, if EDAC is enabled in your kernel config, any correctable-error logs leading up to the panic would be a smoking gun.
    • Since this is happening even at your reduced, "stable" 667 MHz baseline, it further supports checking whether this is not purely a DDR frequency/margin issue, but possibly power-rail droop under transient load— since a brief VDD_CORE or DDR VDDQ dip during a heavy I/O burst (like an IRQ storm or console/printk burst, which is exactly when this fault occurred) could momentarily corrupt an in-flight memory transaction without failing DDR training itself.
    • Please continue the overnight memtester/stress-ng run and share results tomorrow as planned, but also specifically capture whether pure memory-stress-only boots (no application) can reproduce this same random-fault pattern. That single data point will most efficiently confirm hardware/DDR vs. application-software as the root cause.

     Best Regards,

    Borislav Lazarkov

  • Dear Sir,

    Please find attached the ZIP file containing the test case execution results.

    As part of the testing, we monitored the system temperature and PMIC status while loading and stressing the LPDDR4 memory. We performed memory tests using Memtester, including Walking Zeros, Walking Ones, and other test patterns, with a memory capacity of 2.5 GB, to verify memory stability and overall system behavior under stress conditions.

    Kindly review the attached test results and let me know whether we are proceeding in the right direction. Please also let us know if any additional tests or information are required from our side.

    Thank you for your valuable guidance and support.

    Best Regards,
    SamudralankaiahStress_test_logs.zip

  • Dear Sir,

    Can I try increasing the LPDDR4 driver strength by adjusting the impedance settings? By default, the impedance is set to 40 Ω.

    Could you please recommend suitable impedance values for increasing the drive strength? We would like to try the recommended settings and then perform testing at the previously failed frequency(1866MHz to check the memory stability.

    Please suggest the appropriate impedance values and any recommended range for this adjustment.

    I have attached a snapshot for your reference.

    Thank you for your guidance and support.

    Best Regards,
    Samudralankaiah

  • Hello Samudralankaiah,

    "Drive Strength: Pull-Down (PDDS)" is the LPDDR4 PHY output driver impedance, currently set to RZQ/6 (40Ω). This value follows the standard JEDEC LPDDR4 drive-strength calibration steps, which are all defined as fractions of RZQ (the external ZQ calibration resistor, 240Ω on AM62A designs per the board design guide). The selectable steps in the tool are fixed, discrete values tied to these RZQ ratios — you can't type in an arbitrary number; you pick from the supported list (e.g., RZQ/5 = 48Ω, RZQ/6 = 40Ω, RZQ/7 = 34Ω, and so on, depending on what your specific LPDDR4 IBIS/AC timing model supports). Increasing "drive strength" means decreasing the ohm value, not increasing it. A lower-impedance driver can push more current onto the trace for the same voltage swing — i.e., higher drive strength = lower Ω. So going from 40Ω (RZQ/6) toward 34Ω (RZQ/7) would be the direction to try if you want to strengthen the drive. The baseline drive impedance and ODT settings for attached memory (LPDDR4) are derived from the signal integrity (SI) simulations performed on the SK or EVM. The recommendation is to perform simulation on the custom board design to finalize the values, as the configuration values can be different compared to the SK/EVM schematic implementation.

    What I could recommend to do is:

    •  Increasing drive strength without re-simulating can just as easily make signal integrity worse (overshoot/undershoot, ringing, ISI) if your trace lengths/topology were tuned around the 40Ω default.
    • That said, as an empirical debug step (not a production fix), trying one step down from default — RZQ/7 (34Ω) — is a reasonable, low-risk experiment to see if it changes the failure signature at 1866 MHz. I would not recommend jumping multiple steps without simulation, since overly strong drive can overshoot and cause the same kind of intermittent bit-error symptoms you're already chasing.
    • Since you're testing at 1866 MHz specifically because it previously failed, I'd suggest changing only the drive strength variable at a time (keep everything else — VREF, ODT, termination settings — at your current baseline) so any change in stability can be attributed cleanly to the drive-strength adjustment.

    Best regards,

    Borislav Lazarkov

  • Dear Sir,

    Thank you very much for your valuable guidance and support.

    Could you please review the test cases we have performed and let us know whether they are appropriate for validating the LPDDR4 memory stability? We would also appreciate your guidance on whether we are proceeding in the right direction with the current testing approach.

    If you recommend any additional test cases or specific parameters that we should monitor, please let us know. We will proceed with the recommended tests accordingly.

    Thank you once again for your time and valuable guidance.

    Best Regards,
    Samudralankaiah

  • Dear Sir,

    Please note that the below options are available in the SysConfig tool for configuring the LPDDR4 drive strength.

    We are using SysConfig version 1.28.0+4696. The available drive strength options are:

    • RFU
    • RZQ/1 (240 Ω)
    • RZQ/2 (120 Ω)
    • RZQ/3 (80 Ω)
    • RZQ/4 (60 Ω)
    • RZQ/5 (48 Ω)
    • RZQ/6 (40 Ω)

    These are the only drive strength/impedance options available in the current SysConfig version.

    Could you please verify these options and advise us which impedance value you recommend for increasing the LPDDR4 drive strength? We would like to use the recommended setting and then retest the previously failed frequency to check the memory stability.

    Your guidance on the appropriate setting would be highly appreciated.

    Thank you very much for your continued support and guidance.

    Best Regards,
    Samudralankaiah

  • Hello Samudralankaiah,

    I have written to my colleague for additional support.
    Please expect 2-3 days delay in his response.
    Thank you for your patience!
    Best Regards,

    Borislav Lazarkov

  • Hi Samudra,

    before you start changing drive strength and/or termination values, I'd like to review your DDR configuration.

    -Can you provide the .dtsi and .syscfg files from the DDR register configuration tool for both the 667MHz configuration and the max frequency configuration. you have been running.

    -can you provide the DDR part number and/or datasheet?

    -what is the full part number for AM62A you are using.  Some speed grades of the device only support max 1600MHz DDR clock

    Regards,

    James

  • Dear James,

    Thank you for your email and for your guidance.

    Please find the requested details below:

    • SoC Part Number: AM62A74AUMSIAMBRQ1
    • LPDDR4 Part Number: MT53E1G32D2FW-046 AUT

    Please find attached the DTSI files for both the 667 MHz and 1600 MHz working configurations, as well as the maximum-frequency configuration that is currently failing. I have also attached all the filesDDR_Configurations.zip for your reference.

    Please review the attached files and let us know if you need any additional information or configuration files from our side.

    Thank you for your continued support and guidance.

    Best Regards,
    Samudralankaiah

  • Hello Samudralankaiah,

    Please expect a delay in James's  response.

    Thank you for your patience!

    Best Regards,

    Borislav Lazarkov

  • Samudra, the SoC part number reveals the LPDDR4 max speed grade is 1600MHz (3200MTs).  You will not be able to run the device with LPDDR4 frequency of 1866MHz.

    Excerpts from the AM62A datasheet

    Regards,

    James

  • Dear James and Borislav,

    Thank you so much for your detailed explanation and guidance. It helped me a lot.

    However, we noticed that the evaluation board is working with the LPDDR4 at 1866 MHz, which is why we wanted to investigate this further. It is possible that the evaluation board is using a different SoC speed grade or memory part.

    I will cross-check the exact SoC and LPDDR4 part numbers used on the evaluation board and compare them with our device. I will get back to you once I have verified the details.

    Thank you once again for your valuable support and guidance. I would also like to thank Borislav Lazarkov for his support.

    Could you please also verify the test cases we shared and provide your feedback? Your guidance has been very helpful for us in narrowing down the issue.

    Thank you very much, James and Borislav, for your continued support.

    Best Regards,
    Samudra