J721S2XSOMXEVM: TDA4AL: DDR: memtester failed

hyeonseong Bae

Prodigy 110 points

Part Number: J721S2XSOMXEVM

Tool/software:

Hi Ti Expert,

We encountered intermittent memory issues while running the APP, as shown in the attached file.

Consequently, we ran memtester, and the result indicates an issue with the Block Sequential test.

How should we handle this situation?

Should we consider sending the board for repair?

Best Regards,

Hyeonseong.

Fullscreen memory_issue1.txt Download

root@j721s2-evm:~# [ 1428.249398] Unable to handle kernel paging request at virtual address ffff800b7e190220
[ 1428.249865] Unable to handle kernel paging request at virtual address ffff1800080539ec
[ 1428.257330] Kernel panic - not syncing: stack-protector: Kernel stack is corrupted in: console_unlock+0x20c/0x210
[ 1428.257334] SMP: stopping secondary CPUs
[ 1428.265204] Mem abort info:
[ 1428.275437]   ESR = 0x0000000086000004
[ 1428.279344]   EC = 0x21: IABT (current EL), IL = 32 bits
[ 1428.282125]   SET = 0, FnV = 0
[ 1428.285858]   EA = 0, S1PTW = 0
[ 1428.291151]   FSC = 0x04: level 0 translation fault
[ 1428.294192] swapper pgtable: 4k pages, 48-bit VAs, pgdp=000000008303b000
[ 1428.297320] [ffff1800080539ec] pgd=0000000000000000, p4d=0000000000000000
[ 1428.315628] Internal error: Oops: 0000000086000004 [#1] PREEMPT SMP
[ 1428.321876] Modules linked in: overlay cfg80211 bluetooth ecdh_generic ecc rfkill cryptodev(O) peak_usb can_raw can panel_edp ti_am335x_adc kfifo_buf rpmsg_ctrl rpmsg_char omap_rng cdns3 cdns_usb_common crct10dif_ce display_connector phy_can_transceiver ti_k3_r5_remoteproc wave5 ti_sn65dsi86 videobuf2_dma_contig videobuf2_memops v4l2_mem2mem videobuf2_v4l2 videobuf2_common cdns_mhdp8546 drm_dp_aux_bus ti_am335x_tscadc videodev pvrsrvkm(O) drm_display_helper ti_k3_dsp_remoteproc virtio_rpmsg_bus rpmsg_ns ti_k3_common tidss drm_dma_helper drm_kms_helper mc sa2ul cfbfillrect syscopyarea cfbimgblt sysfillrect sysimgblt fb_sys_fops cfbcopyarea cdns_dphy_rx cdns3_ti optee_rng cdns_dphy m_can_platform m_can can_dev pwm_tiehrpwm rng_core fuse drm drm_panel_orientation_quirks ipv6
[ 1428.390193] CPU: 0 PID: 1054 Comm: sv_vx_app_aptiv Not tainted 6.1.33-g8f7f371be2 #1
[ 1428.397917] Hardware name: Texas Instruments J721S2 EVM (DT)
[ 1428.403558] pstate: 400000c5 (nZcv daIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[ 1428.410500] pc : 0xffff1800080539ec
[ 1428.413976] lr : 0xffff1800080539ec
[ 1428.417450] sp : ffff80000b4c3c80
[ 1428.420749] x29: ffff10000b4c3980 x28: ffff000835759c80 x27: ffff00082d92fbe0
[ 1428.427866] x26: 0000000000000002 x25: 0000000000000055 x24: ffff00000b4c2368
[ 1428.434983] x23: 000080000000000a x22: ffff000835759c50 x21: 0000900000000000
[ 1428.442100] x20: ffff80083575f280 x19: 0000b0000000bc0b x18: ffff80000b4c3d68
[ 1428.449218] x17: 0000000000000000 x16: ffff800008da5e10 x15: 0000000000000000
[ 1428.456334] x14: ffff000836dbdc0c x13: 1fffe00106db7b81 x12: ffff80000b4c3d68
[ 1428.463450] x11: 0000000000000000 x10: ffff000836dbdc08 x9 : 0000000000000001
[ 1428.470567] x8 : 0000000000000001 x7 : 0000000000000090 x6 : 0000000000000250
[ 1428.477685] x5 : ffffffffffffd8f8 x4 : 0000000000000000 x3 : ffffffffffffffff
[ 1428.484801] x2 : ffff800b750a6000 x1 : 0000000000000000 x0 : ffff00082e4a73c0
[ 1428.491920] Call trace:
[ 1428.494354]  0xffff1800080539ec
[ 1428.497485] Code: bad PC value
[ 1428.500525] ---[ end trace 0000000000000000 ]---
[ 1428.505126] note: sv_vx_app_aptiv[1054] exited with irqs disabled
[ 1429.324837] SMP: failed to stop secondary CPUs 0-1
[ 1429.329619] Kernel Offset: disabled
[ 1429.333091] CPU features: 0x20000,20028084,0000420b
[ 1429.337952] Memory Limit: none
[ 1429.340994] ---[ end Kernel panic - not syncing: s-ack}prooector: eernul s ack`is uorrupte: inn coosolm_un+ock;0x2xc/0x210 ]---

Fullscreen memory_issue2.txt Download

root@j721s2-evm:~# [ 1428.249398] Unable to handle kernel paging request at virtual address ffff800b7e190220
[ 1428.249865] Unable to handle kernel paging request at virtual address ffff1800080539ec
[ 1428.257330] Kernel panic - not syncing: stack-protector: Kernel stack is corrupted in: console_unlock+0x20c/0x210
[ 1428.257334] SMP: stopping secondary CPUs
[ 1428.265204] Mem abort info:
[ 1428.275437]   ESR = 0x0000000086000004
[ 1428.279344]   EC = 0x21: IABT (current EL), IL = 32 bits
[ 1428.282125]   SET = 0, FnV = 0
[ 1428.285858]   EA = 0, S1PTW = 0
[ 1428.291151]   FSC = 0x04: level 0 translation fault
[ 1428.294192] swapper pgtable: 4k pages, 48-bit VAs, pgdp=000000008303b000
[ 1428.297320] [ffff1800080539ec] pgd=0000000000000000, p4d=0000000000000000
[ 1428.315628] Internal error: Oops: 0000000086000004 [#1] PREEMPT SMP
[ 1428.321876] Modules linked in: overlay cfg80211 bluetooth ecdh_generic ecc rfkill cryptodev(O) peak_usb can_raw can panel_edp ti_am335x_adc kfifo_buf rpmsg_ctrl rpmsg_char omap_rng cdns3 cdns_usb_common crct10dif_ce display_connector phy_can_transceiver ti_k3_r5_remoteproc wave5 ti_sn65dsi86 videobuf2_dma_contig videobuf2_memops v4l2_mem2mem videobuf2_v4l2 videobuf2_common cdns_mhdp8546 drm_dp_aux_bus ti_am335x_tscadc videodev pvrsrvkm(O) drm_display_helper ti_k3_dsp_remoteproc virtio_rpmsg_bus rpmsg_ns ti_k3_common tidss drm_dma_helper drm_kms_helper mc sa2ul cfbfillrect syscopyarea cfbimgblt sysfillrect sysimgblt fb_sys_fops cfbcopyarea cdns_dphy_rx cdns3_ti optee_rng cdns_dphy m_can_platform m_can can_dev pwm_tiehrpwm rng_core fuse drm drm_panel_orientation_quirks ipv6
[ 1428.390193] CPU: 0 PID: 1054 Comm: sv_vx_app_aptiv Not tainted 6.1.33-g8f7f371be2 #1
[ 1428.397917] Hardware name: Texas Instruments J721S2 EVM (DT)
[ 1428.403558] pstate: 400000c5 (nZcv daIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[ 1428.410500] pc : 0xffff1800080539ec
[ 1428.413976] lr : 0xffff1800080539ec
[ 1428.417450] sp : ffff80000b4c3c80
[ 1428.420749] x29: ffff10000b4c3980 x28: ffff000835759c80 x27: ffff00082d92fbe0
[ 1428.427866] x26: 0000000000000002 x25: 0000000000000055 x24: ffff00000b4c2368
[ 1428.434983] x23: 000080000000000a x22: ffff000835759c50 x21: 0000900000000000
[ 1428.442100] x20: ffff80083575f280 x19: 0000b0000000bc0b x18: ffff80000b4c3d68
[ 1428.449218] x17: 0000000000000000 x16: ffff800008da5e10 x15: 0000000000000000
[ 1428.456334] x14: ffff000836dbdc0c x13: 1fffe00106db7b81 x12: ffff80000b4c3d68
[ 1428.463450] x11: 0000000000000000 x10: ffff000836dbdc08 x9 : 0000000000000001
[ 1428.470567] x8 : 0000000000000001 x7 : 0000000000000090 x6 : 0000000000000250
[ 1428.477685] x5 : ffffffffffffd8f8 x4 : 0000000000000000 x3 : ffffffffffffffff
[ 1428.484801] x2 : ffff800b750a6000 x1 : 0000000000000000 x0 : ffff00082e4a73c0
[ 1428.491920] Call trace:
[ 1428.494354]  0xffff1800080539ec
[ 1428.497485] Code: bad PC value
[ 1428.500525] ---[ end trace 0000000000000000 ]---
[ 1428.505126] note: sv_vx_app_aptiv[1054] exited with irqs disabled
[ 1429.324837] SMP: failed to stop secondary CPUs 0-1
[ 1429.329619] Kernel Offset: disabled
[ 1429.333091] CPU features: 0x20000,20028084,0000420b
[ 1429.337952] Memory Limit: none
[ 1429.340994] ---[ end Kernel panic - not syncing: s-ack}prooector: eernul s ack`is uorrupte: inn coosolm_un+ock;0x2xc/0x210 ]---

Fullscreen 3480.memtester.txt Download

login as: root
root@10.0.50.100's password:
# cd /jffs2/
# date && ./memtester 900M
Thu Jul 14 07:45:55 UTC 2022
memtester version 4.3.0 (32-bit)
Copyright (C) 2001-2012 Charles Cazabon.
Licensed under the GNU General Public License version 2 (only).

pagesize is 4096
pagesizemask is 0xfffff000
want 900MB (943718400 bytes)
got  900MB (943718400 bytes), trying mlock ...locked.
Loop 1:
  Stuck Address       : testing  11FAILURE: possible bad address line at offset 0x0f586000.
Skipping to next test...
  Random Value        : ok
  Compare XOR         : ok
  Compare SUB         : ok
  Compare MUL         : ok
  Compare DIV         : ok
  Compare OR          : ok
  Compare AND         : ok
  Sequential Increment: ok
  Solid Bits          : testing   3FAILURE: 0xff00ff00 != 0x00000000 at offset 0x03ee6010.
FAILURE: 0x00ff00ff != 0xffffffff at offset 0x03ee6014.
FAILURE: 0xff00ff00 != 0x00000000 at offset 0x03ee6018.
  Block Sequential    : setting   0


Hi TI expert,

We use memtester to test some boards failed.

over 1 year ago

0 Kevin S over 1 year ago

TI__Mastermind 42270 points

Hi,

Does the issue go away if the LPDDR4 frequency is reduced?
What memory is being used in the system?
Can you provide the configuration files being used to configure the DDRSS? (ex: DTSI file and filled-in XLS tool)
How many systems/boards does the issue occur on? (raw count and percentage)
Are there any temperature dependencies (does the issue occur at room temperature vs. hot or cold?)

Regards,
Kevin

0 hyeonseong Bae over 1 year ago in reply to Kevin S

Prodigy 110 points

Hi Kevin,

We have not reduced the LPDDR4 frequency yet. Could you please advise how much we should reduce the frequency and which configuration files need to be modified?
We are using the system's LPDDR4 DDR memory.
The attached file k3-j721s2-rtos-memory-map.dtsi is the one we are using. Please let me know if you need any additional files.
The issue occurs on only one board out of all the ones we are using. We have run memtester on other boards without encountering any issues.
The issue occurs even at room temperature.

Fullscreen k3-j721s2-rtos-memory-map.dtsi.txt Download

 
 /* 
  * IMPORTANT NOTE: Follow the below instructions to apply the updated memory map to the linux dtsi file,
  * 
  * 1. Copy the contents of the generated k3-{SOC}-rtos-memory-map.dtsi file and replace the
  *      reserved_memory section of the k3-{SOC}-rtos-memory-map.dtsi file that is located here:
  *      PSDK_LINUX_PATH/board-support/linux*/arch/arm64/boot/dts/ti
  * 
  * 2. Rebuild the dtb and dtbo from the PSDK_LINUX_PATH directory with the following:
  *      make linux-dtbs
  * 
  * 3. Install the dtb and dtbo to the rootfs/boot folder on the SD card from the same directory with the following:
  *      sudo make linux-dtbs_install; sync
  * 
   */


	vision_apps_mcu_r5fss0_core0_dma_memory_region: vision-apps-r5f-dma-memory@a0000000 {
		compatible = "shared-dma-pool";
		reg = <0x00 0xa0000000 0x00 0x00100000>;
		no-map;
	};
	vision_apps_mcu_r5fss0_core0_memory_region: vision-apps-r5f-memory@a0100000 {
		compatible = "shared-dma-pool";
		reg = <0x00 0xa0100000 0x00 0x00f00000>;
		no-map;
	};
	vision_apps_mcu_r5fss0_core1_dma_memory_region: vision-apps-r5f-dma-memory@a1000000 {
		compatible = "shared-dma-pool";
		reg = <0x00 0xa1000000 0x00 0x00100000>;
		no-map;
	};
	vision_apps_mcu_r5fss0_core1_memory_region: vision-apps-r5f-memory@a1100000 {
		compatible = "shared-dma-pool";
		reg = <0x00 0xa1100000 0x00 0x00f00000>;
		no-map;
	};
	vision_apps_main_r5fss0_core0_dma_memory_region: vision-apps-r5f-dma-memory@a2000000 {
		compatible = "shared-dma-pool";
		reg = <0x00 0xa2000000 0x00 0x00100000>;
		no-map;
	};
	vision_apps_main_r5fss0_core0_memory_region: vision-apps-r5f-memory@a2100000 {
		compatible = "shared-dma-pool";
		reg = <0x00 0xa2100000 0x00 0x01f00000>;
		no-map;
	};
	vision_apps_main_r5fss0_core1_dma_memory_region: vision-apps-r5f-dma-memory@a4000000 {
		compatible = "shared-dma-pool";
		reg = <0x00 0xa4000000 0x00 0x00100000>;
		no-map;
	};
	vision_apps_main_r5fss0_core1_memory_region: vision-apps-r5f-memory@a4100000 {
		compatible = "shared-dma-pool";
		reg = <0x00 0xa4100000 0x00 0x01f00000>;
		no-map;
	};
	vision_apps_main_r5fss1_core0_dma_memory_region: vision-apps-r5f-dma-memory@a6000000 {
		compatible = "shared-dma-pool";
		reg = <0x00 0xa6000000 0x00 0x00100000>;
		no-map;
	};
	vision_apps_main_r5fss1_core0_memory_region: vision-apps-r5f-memory@a6100000 {
		compatible = "shared-dma-pool";
		reg = <0x00 0xa6100000 0x00 0x00f00000>;
		no-map;
	};
	vision_apps_main_r5fss1_core1_dma_memory_region: vision-apps-r5f-dma-memory@a7000000 {
		compatible = "shared-dma-pool";
		reg = <0x00 0xa7000000 0x00 0x00100000>;
		no-map;
	};
	vision_apps_main_r5fss1_core1_memory_region: vision-apps-r5f-memory@a7100000 {
		compatible = "shared-dma-pool";
		reg = <0x00 0xa7100000 0x00 0x00f00000>;
		no-map;
	};
	vision_apps_rtos_ipc_memory_region: vision-apps-rtos-ipc-memory-region@a8000000 {
		compatible = "shared-dma-pool";
		reg = <0x00 0xa8000000 0x00 0x02000000>;
		no-map;
	};
	vision_apps_memory_region: vision-apps-dma-memory@aa000000 {
		compatible = "shared-dma-pool";
		reg = <0x00 0xaa000000 0x00 0x06000000>;
		no-map;
	};
	vision_apps_c71_0_dma_memory_region: vision-apps-c71-dma-memory@b0000000 {
		compatible = "shared-dma-pool";
		reg = <0x00 0xb0000000 0x00 0x00100000>;
		no-map;
	};
	vision_apps_c71_0_memory_region: vision-apps-c71_0-memory@b0100000 {
		compatible = "shared-dma-pool";
		reg = <0x00 0xb0100000 0x00 0x05f00000>;
		no-map;
	};
	vision_apps_c71_1_dma_memory_region: vision-apps-c71_1-dma-memory@b6000000 {
		compatible = "shared-dma-pool";
		reg = <0x00 0xb6000000 0x00 0x00100000>;
		no-map;
	};
	vision_apps_c71_1_memory_region: vision-apps-c71_1-memory@b6100000 {
		compatible = "shared-dma-pool";
		reg = <0x00 0xb6100000 0x00 0x01f00000>;
		no-map;
	};
	vision_apps_core_heaps_lo: vision-apps-core-heap-memory-lo@b8000000 {
		compatible = "shared-dma-pool";
		reg = <0x00 0xb8000000 0x00 0x04000000>;
		no-map;
	};
	c7x_ddr_heaps_hi: c7x_ddr_heaps_hi-apps-core-heap-memory-hi@880000000 {
		compatible = "shared-dma-pool";
		reg = <0x08 0x80000000 0x00 0x2c000000>;
		no-map;
	};
	vision_apps_shared_region: vision_apps_shared-memories {
		compatible = "dma-heap-carveout";
		reg = <0x09 0x00000000 0x00 0x20000000>;
	};

Regards,

Hyeonseong

Processors

Processors forum

J721S2XSOMXEVM: TDA4AL: DDR: memtester failed