AM625: DDR4 slow performance of the DDR4 at address 0x880000000

I also give a try to the configuration withData Bus Width (per device set to 8 instead of 16 but performance are the same. 

Hi Alexis,

can you post the patches you have performed in the device tree?  Are you able to successfully access all 4GB, but just the performance is slow when accessing the upper 2GB at 0x8_8000_0000?  If so, i don't think there is a problem with the DDR configuration, but rather an issue with the MMU region (eg, you may not have that region setup the same way as the lower 2GB. Let me loop in someone from our software team as well.

Regards,

James 

Hi James,

Here are the patches apply to u-boot and kernel device-tree.

From 872df5574a780bfdf76f59f8f1f6a1dc9cfa0035 Mon Sep 17 00:00:00 2001
From: Alexis LOGNON <alexis.lognon@actiatelecom.fr>
Date: Tue, 7 Nov 2023 10:03:11 +0000
Subject: [PATCH] Set 4GB of memory in DTS

---
 arch/arm64/boot/dts/ti/k3-am62x-sk-common.dtsi | 13 ++++++++++---
 1 file changed, 10 insertions(+), 3 deletions(-)

diff --git a/arch/arm64/boot/dts/ti/k3-am62x-sk-common.dtsi b/arch/arm64/boot/dts/ti/k3-am62x-sk-common.dtsi
index 6c6782e33668..3f3c598904a9 100644
--- a/arch/arm64/boot/dts/ti/k3-am62x-sk-common.dtsi
+++ b/arch/arm64/boot/dts/ti/k3-am62x-sk-common.dtsi
@@ -92,9 +92,9 @@
 
 	memory@80000000 {
 		device_type = "memory";
-		/* 2G RAM */
-		reg = <0x00000000 0x80000000 0x00000000 0x80000000>;
-
+	        /* 4G RAM */
+		reg = <0x00000000 0x80000000 0x00000000 0x80000000>,
+			<0x00000008 0x80000000 0x00000000 0x80000000>;
 	};
 
 	reserved-memory {
@@ -148,6 +148,13 @@
 			alignment = <0x1000>;
 			no-map;
 		};
+
+		upper_2gb: upper-2gb@880000000 {
+			reg = <0x08 0x80000000 0x00 0x10000000>;
+			alignment = <0x1000>;
+			no-map;
+		};
+
 	};
 
 	vmain_pd: regulator-0 {

uboot-Add-support-of-4GB-in-u-boot.patch.txt

Regards,

Alexis.

So are you able to access the full 4GB, but only the performance is slow in the upper 2GB?  How are you performing the measurements?  

Regards,

James

Hi James,

I am able to read and write using memtester on lnux and mtest on u-boot.

For the test I execute the command mtest with a range of 512MB and I measure the time with a chronometer.

By the way, is it possible to make a memory test using jtag script under CCS?

JTAG scripts are not a good way to perform memory tests as they would be too slow to execute.  memtester and mtest are not really intended to perform performance related tests.  what is the command you are using for each?  I assume you are specifying a start address?

Regards,

James

Alexis, can you try using bw_mem for the performance tests, like this:

bw_mem -P 4 512M bcopy

Regards,

James

Hello James,

Here is the result of the command : bw_mem for the AM62x-EMV-SK and our board.

I use the SDK 08.06.00.42 with thinlinux image with ethboot enabled.

I use the same device tree for both with 1 memory range of 2GB starting at 0x80000000.

bw_mem -P 4 XM bcopy    CUSTOM        EVM
100M                                  1259.52       1261.83
200M                                  1161.01       1175.17
210M                                  1260.46       1237.51
220M                      220MZ            1296.05       1285.49

When a size greater than 220M is used I got the same error for both boards, even if boot the EVM from SDCard.

bw_mem -P 4 230M bcopy
[ 2298.301317] bw_mem invoked oom-killer: gfp_mask=0x400dc0(GFP_KERNEL_ACCOUNT|__GFP_ZERO), order=0, oom_score_adj=0
[ 2298.321714] CPU: 1 PID: 961 Comm: bw_mem Tainted: G D O 5.10.168-g2c23e6c538 #1
[ 2298.330172] Hardware name: Texas Instruments AM625 SK (DT)
[ 2298.335682] Call trace:
[ 2298.338186] dump_backtrace+0x0/0x1b0
[ 2298.341911] show_stack+0x18/0x38
[ 2298.345284] dump_stack+0xe8/0x124
[ 2298.348722] dump_header+0x48/0x19c
[ 2298.352259] oom_kill_process+0x224/0x228
[ 2298.356317] out_of_memory+0xe4/0x330
[ 2298.360018] __alloc_pages_slowpath.constprop.0+0x958/0x9f8
[ 2298.365614] __alloc_pages_nodemask+0x1ec/0x240
[ 2298.370180] __pte_alloc+0x30/0x160
[ 2298.373700] handle_mm_fault+0x980/0xff0
[ 2298.377638] do_page_fault+0x130/0x3b0
[ 2298.381413] do_translation_fault+0x58/0x68
[ 2298.385625] do_mem_abort+0x40/0xa0
[ 2298.389151] el0_da+0x28/0x38
[ 2298.392154] el0_sync_handler+0x88/0xb8
[ 2298.396021] el0_sync+0x180/0x1c0
[ 2298.407622] Mem-Info:
[ 2298.413490] active_anon:197 inactive_anon:471760 isolated_anon:0
[ 2298.413490] active_file:12 inactive_file:13 isolated_file:0
[ 2298.413490] unevictable:0 dirty:0 writeback:0
[ 2298.413490] slab_reclaimable:2700 slab_unreclaimable:6472
[ 2298.413490] mapped:292 shmem:2280 pagetables:1228 bounce:0
[ 2298.413490] free:6006 free_pcp:0 free_cma:533
[ 2298.450282] Node 0 active_anon:788kB inactive_anon:1887040kB active_file:48kB inactive_file:52kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:1168kB dirty:0kB writeback:0kB shmem:9120kB shmems
[ 2298.485577] DMA free:24024kB min:22528kB low:28160kB high:33792kB reserved_highatomic:0KB active_anon:788kB inactive_anon:1886200kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:B
[ 2298.520505] lowmem_reserve[]: 0 0 0 0
[ 2298.537717] DMA: 105*4kB (UMEC) 51*8kB (UEC) 145*16kB (UEC) 70*32kB (UMEC) 36*64kB (UMEC) 37*128kB (UME) 26*256kB (UMEC) 4*512kB (UME) 2*1024kB (UC) 0*2048kB 0*4096kB = 23180kB
[ 2298.557609] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[ 2298.568146] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=32768kB
[ 2298.578693] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[ 2298.589138] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=64kB
[ 2298.600519] 2313 total pagecache pages
[ 2298.604494] 0 pages in swap cache
[ 2298.611351] Swap cache stats: add 0, delete 0, find 0/0
[ 2298.617009] Free swap = 0kB
[ 2298.623463] Total swap = 0kB
[ 2298.626495] 524288 pages RAM
[ 2298.629405] 0 pages HighMem/MovableOnly
[ 2298.635032] 28776 pages reserved
[ 2298.638365] 131072 pages cma reserved
[ 2298.643731] 0 pages hwpoisoned
[ 2298.646912] Tasks state (memory values in pages):
[ 2298.651757] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
[ 2298.660619] [ 192] 999 192 920 57 40960 0 0 rpcbind
[ 2298.669102] [ 194] 0 194 4829 920 61440 0 -250 systemd-journal
[ 2298.678257] [ 212] 0 212 3428 279 49152 0 -1000 systemd-udevd
[ 2298.687252] [ 234] 992 234 20267 102 57344 0 0 systemd-timesyn
[ 2298.696382] [ 409] 997 409 1114 111 49152 0 -900 dbus-daemon
[ 2298.705148] [ 438] 0 438 633 24 40960 0 0 tee-supplicant
[ 2298.714155] [ 439] 0 439 1299 17 40960 0 0 modprobe
[ 2298.722726] [ 442] 994 442 1891 107 49152 0 0 systemd-network
[ 2298.731912] [ 459] 0 459 1747 100 49152 0 0 systemd-logind
[ 2298.741029] [ 483] 993 483 1781 92 49152 0 0 systemd-resolve
[ 2298.750187] [ 490] 996 490 1208 78 45056 0 0 avahi-daemon
[ 2298.759099] [ 491] 0 491 376751 11264 331776 0 -500 dockerd
[ 2298.767559] [ 492] 998 492 783 142 40960 0 0 rpc.statd
[ 2298.776204] [ 495] 0 495 509 26 36864 0 0 agetty
[ 2298.784584] [ 496] 996 496 1175 60 45056 0 0 avahi-daemon
[ 2298.793478] [ 497] 0 497 1133 99 49152 0 0 login
[ 2298.801751] [ 509] 0 509 369148 6929 253952 0 -500 containerd
[ 2298.810521] [ 662] 0 662 2138 176 49152 0 0 systemd
[ 2298.819024] [ 663] 0 663 2671 407 57344 0 0 (sd-pam)
[ 2298.827567] [ 668] 0 668 1110 325 45056 0 0 sh
[ 2298.835785] [ 686] 0 686 696 21 40960 0 0 telnetd
[ 2298.844334] [ 706] 0 706 777 33 45056 0 0 netserver
[ 2298.853026] [ 958] 0 958 754 23 40960 0 0 bw_mem
[ 2298.861424] [ 960] 0 960 113062 112338 937984 0 0 bw_mem
[ 2298.869783] [ 961] 0 961 113062 111279 933888 0 0 bw_mem
[ 2298.878110] [ 962] 0 962 113062 112338 937984 0 0 bw_mem
[ 2298.886402] [ 963] 0 963 113062 112338 937984 0 0 bw_mem
[ 2298.894699] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-0.slice/session-c1.scope,task=bw_mem,pid=960,uid=0
[ 2298.910580] Out of memory: Killed process 960 (bw_mem) total-vm:452248kB, anon-rss:449344kB, file-rss:4kB, shmem-rss:0kB, UID:0 pgtables:916kB oom_score_adj:0

Our board works with the same device tree, but we are using 2 DDR chip of 1 or 2 GB and connected each one to the 8bit data bus. I wonder why the test is working because AM62x-EVM-SK is using a 16bit data bus connected to 1 chip of 2GB ?

Regards,

Alexis.

Hi Alexis,

Alexis LOGNON said:

When a size greater than 220M is used I got the same error for both boards, even if boot the EVM from SDCard.

Kernel fails to allocate such big memory for the test. Let's use 200MB in the performance test for now.

Here is one way to check the DDR performance between address 0x8000_0000 and 0x8_8000_0000:

Prepare two different kernel builds, one has 2GB DDR configuration in kernel DTS (no "<0x00000008 0x80000000 0x00000000 0x80000000>;" in memory <reg> node), the other has 4GB DDR configuration.

In each case, run command "bw_mem -P 4 200M bcopy" multiple times, such as 10 times, then compare the average between the two cases.

Hi Alexis,

Bin Liu said:

Prepare two different kernel builds, one has 2GB DDR configuration in kernel DTS (no "<0x00000008 0x80000000 0x00000000 0x80000000>;" in memory <reg> node), the other has 4GB DDR configuration.

Please note this has to be two different U-Boot builds, with different DDR size configuration in U-Boot DTS.

I just learned from my experiment that U-Boot overrides kernel DDR size configuration base on the memory info in U-Boot. So the DDR size change has to be done in U-Boot, not kernel DTS.

Hi Bin,

I patch the device tree for memory changes in u-boot and in the kernel. So there are aligned with the same mapping.

Could you help me to configue memory to only one memory block at 0x88000 0000.

I would like to know if the issue is due to the memory address or with the 2 DDR chip used.

If u-boot and Linux can boot using this base address we could validate if the issue comes from the handling of 2 DDR component.

Do you think you can give me the configuration for u-boot and Linux.

Regards,

Alexis.

Hello Alexis

Bin is now out of office till beginning of the year. Responses will be delayed. 

Regards

Mukul 

Alexis, just to answer the hw question, i don't think you have a hardware issue.  If you can access all 4GB of the memory *uniquely* with u-boot commands, then your hardware and DDR configuration should be fine. 

I think Bin is suggesting to have 2 different builds, one allocating 2GB from 0x8000_0000 to 0xFFFFFFFF, and another with 4GB (0x8000_0000 to 0xFFFFFFFF, and 0x8_8000_0000 to 0x8_FFFF_FFFF), and then running the performance tests.

Regards,

James