AM6422: Self-test of DDR ECC DED function will cause External Abort and System Crash

Hello Lina,

I am looking at your queries.

Please expect a reply in one or two days.

Regards,

S.Anil.

Hello Lina,

Can you please share the below details to better assist you ?

Are you doing error injection tests from the A53 core or the R5F core?

Have you taken these codes (ESM modules) from MCU+SDK or PROCESSOR SDK or written them yourself?

Are you using the below example to inject errors?

C:\ti\mcu_plus_sdk_am64x_09_00_00_31\examples\drivers\ddr\ddr_ecc_test_main_esm\am64x-evm

Regards,

S.Anil.

Thank you for your reply.

My code is released as an ELF file, booted by u-boot, running on A53 core. Therefore, most of the code is similar to u-boot, whether it is ESM driver or DDRSS Self-test driver. The Self-test driver is to perform an error injection.

ESM driver is written by myself, confirmed valid during DDR ECC SEC self-test. Some refer to drivers/misc/ke_esm.c from u-boot, others refer to AM64x TRM.

The process of injecting errors refers to cmd/ti/ddrss.c from u-boot, because there is a significant gap between the software architecture of the SDK and my requirements. I have tried to refer to the SDK process for encoding and implementing error injection operations, but I was unable to successfully trigger the interrupt signal in that case.

In my understanding, the triggering process for SEC and DED should be the same, except that the number of data bits that need to be flipped and the corresponding interrupt signal are different. Will this lead to any differences in code implementation?

Sorry, due to a click error, I accidentally selected the issue has been resolved, although it still exists now...

Since I have encapsulated many interfaces for compatibility, let me briefly describe the key points. If you have any questions about any of them, we can discuss them in detail later.

1. In DDRSS Self-test driver, enable ESM sign 6 with Level Error Interrupt, then enable ESM sign 69 with Level Error Interrupt and High Priority. The only difference between these two is sign number and whether High Priority is enabled. The rest of the parts are executed using the same function, so there should be no difference.

2. Calling self test function in DDRSS Self-test driver:

a. Transfer one bit error to an address, for example, flip 0x00100000 at 0x90000010:

i. Alloc the memory of that physical address with cache-disabled;

ii. Disable DDR ECC by reset DDRSS_ECC_R0_STR_ADDR_REG to 0xFFFF and DDRSS_ECC_R0_END_ADDR_REG to 0x0000;(Refer to u-boot)

iii. Read the value of that address, filp it by xor;

iv. Restore those ADDR_REG to enable DDR ECC;

v. Read the value of that address once;

vi. Check DDRSS_V2A_INIT_RAW_REG, ECC1BERR should be detected;

vii. Restore that memory.

b. Transfer two bit error to an address, for example, flip 0x00010010 at 0x90000020:

i - iv. Same to the above;

v. Read the value of that address, then crashed.(It shouldn't be the reason for that specific address, because when I tested SEC here, the results were good)

3. Need to perform a system reset and then reinitialize 1.

4. Load another version which skip driver self check and test in self check code:

a. Trigger an one bit error to an address and wait for interrupt:

i - v. Same to the first time mentioned above;

vi. After a slight wait, the pre prepared callback function was successfully called;

vii. Restore that memory.

b. Trigger an two bit error to an address and wait for interrupt:

i - iv. Same to the first time mentioned above;

v. Read the value of that address, then crashed.

lina zhang said:

3. Need to perform a system reset and then reinitialize 1.

4. Load another version which skip driver self check and test in self check code:

Hello lina,

I could not understand why you are performing these steps, and one more thing: you are running which OS on A53.

Apart from these, your test procedure is okay, and we are also doing the same thing in R5F core.

Is it possible to check if ESM module initialization is properly configured or not?

Already, the same example is available at the below link.

C:\ti\mcu_plus_sdk_am64x_09_00_00_31\examples\drivers\ddr\ddr_ecc_test_main_esm\am64x-evm

Regards,

S.Anil.

Due to 2.b.v. crashed, I rigorously added a restart operation to 3.

Step 2 is to detect any issues within DDRSS Self-test driver, by actively reading registers. Step 4 is to confirm that the callback function can be called normally. This is the difference.

I am running our own designed OS, which can also be considered as a small ELF application booted through u-boot.

Swargam Anil said:

and we are also doing the same thing in R5F core.

Do you mean you have never tested DDR ECC on A53? Is it possible that certain characteristics in R5 may cause this issue not to occur in R5, but only in A53?

Since the detection of SEC is always successful, I tend to believe that the ESM driver I wrote has no issues. At least it will not cause this type of crash on 2.b.v. or 4.b.v.

lina zhang said:

Do you mean you have never tested DDR ECC on A53? Is it possible that certain characteristics in R5 may cause this issue not to occur in R5, but only in A53?

Yes lina, 

I did not test the DDR example on A53, but I have tested the same example injecting errors in single bits and dual bits on DDR memories on R5F cores, which is working properly.

Is it possible to share your code? From my side, I can check what the difference is between your code and our code (R5F) and will provide suggestions.

Regards,

S.Anil.

The Original question has the same phenomenon with my code when running on A53. Can you check that code(u-boot from TI SDK 8.6) and compare it with your code to provide any suggestions?

Any suggestion about that code?

Hello lina,

Sorry for the delayed response.

It is very difficult to check the DDR ECC example on the A53 core since we don't have any such examples on A53. If you share your code, I can check at my side what the difference is between A53 and R5F and will help you. Sorry for the inconvenience.

If it is not okay to share code in this Thread , then please share code via mail or use private messages in the thread. so that it can be shared between us.

Regards,

S.Anil.

Due to some copyright restrictions, my code is not suitable for sharing to external networks. I hope we can all refer to the publicly available code for discussion as much as possible.

I noticed that am64xx-evm-linux-sdk-src(link here) contains an u-boot, which includes a cmd/ti/ddrss.c(copied below) file that describes how to perform DDR ECC testing in u-boot environment. This code should be able to run on A53, right? If so, it should be a suitable example for us to discuss together.

// SPDX-License-Identifier: GPL-2.0+
/*
 * DDRSS: DDR 1-bit inline ECC test command
 *
 * Copyright (C) 2022 Texas Instruments Incorporated - http://www.ti.com/
 */

#include <asm/io.h>
#include <asm/cache.h>
#include <common.h>
#include <command.h>
#include <cpu_func.h>
#include <linux/bitops.h>

DECLARE_GLOBAL_DATA_PTR;

#define K3_DDRSS_MAX_ECC_REGIONS	3

#define DDRSS_BASE			0x0f300000 /* AM62/AM64 */
#define DDRSS_V2A_CTL_REG		0x0020
#define DDRSS_V2A_INT_RAW_REG		0x00a0
#define DDRSS_V2A_INT_STAT_REG		0x00a4
#define DDRSS_V2A_INT_ECC1BERR		BIT(3)
#define DDRSS_V2A_INT_ECC2BERR		BIT(4)
#define DDRSS_V2A_INT_ECCM1BERR		BIT(5)
#define DDRSS_ECC_CTRL_REG		0x0120
#define DDRSS_ECC_CTRL_REG_ECC_EN	BIT(0)
#define DDRSS_ECC_CTRL_REG_RMW_EN	BIT(1)
#define DDRSS_ECC_CTRL_REG_ECC_CK	BIT(2)
#define DDRSS_ECC_CTRL_REG_WR_ALLOC	BIT(4)
#define DDRSS_ECC_R0_STR_ADDR_REG	0x0130
#define DDRSS_ECC_Rx_STR_ADDR_REG(x)	(0x0130 + ((x) * 8))
#define DDRSS_ECC_Rx_END_ADDR_REG(x)	(0x0134 + ((x) * 8))
#define DDRSS_ECC_1B_ERR_CNT_REG	0x0150
#define DDRSS_ECC_1B_ERR_THRSH_REG	0x0154
#define DDRSS_ECC_1B_ERR_ADR_REG	0x0158
#define DDRSS_ECC_1B_ERR_MSK_LOG_REG	0x015c

static inline u32 ddrss_read(u32 reg)
{
	return readl((unsigned long)(DDRSS_BASE + reg));
}

static inline void ddrss_write(u32 value, u32 reg)
{
	writel(value, (unsigned long)(DDRSS_BASE + reg));
}

/* ddrss_check_ecc_status()
 * Report the ECC state after test. Check/clear the interrupt
 * status register, dump the ECC err counters and ECC error offset.
 */
static void ddrss_check_ecc_status(void)
{
	u32 ecc_1b_err_cnt, v2a_int_raw, ecc_1b_err_msk;
	phys_addr_t ecc_1b_err_adr;

	v2a_int_raw = ddrss_read(DDRSS_V2A_INT_RAW_REG);

	/* 1-bit correctable */
	if (v2a_int_raw & DDRSS_V2A_INT_ECC1BERR) {
		puts("\tECC test: DDR ECC 1-bit error\n");

		/* Dump the 1-bit counter and reset it, as we want a
		 * new interrupt to be generated when above the error
		 * threshold
		 */
		ecc_1b_err_cnt = ddrss_read(DDRSS_ECC_1B_ERR_CNT_REG);
		if (ecc_1b_err_cnt) {
			printf("\tECC test: 1-bit ECC err count: %u\n",
			       ecc_1b_err_cnt & 0xffff);
			ddrss_write(1, DDRSS_ECC_1B_ERR_CNT_REG);
		}

		/* ECC fault addresses are also recorded in a 2-word deep
		 * FIFO. Calculate and report the 8-byte range of the error
		 */
		ecc_1b_err_adr = ddrss_read(DDRSS_ECC_1B_ERR_ADR_REG);
		ecc_1b_err_msk = ddrss_read(DDRSS_ECC_1B_ERR_MSK_LOG_REG);
		if (ecc_1b_err_msk) {
			/* AM64/AM62:
			 * The address of the ecc error is 16-byte aligned.
			 * Each bit in 4 bit mask represents 8 bytes ECC quanta
			 * that has the 1-bit error
			 */
			ecc_1b_err_msk &= 0xf;
			ecc_1b_err_adr <<= 4;
			ecc_1b_err_adr += (fls(ecc_1b_err_msk) - 1) * 8;
			printf("\tECC test: 1-bit error in [0x%llx:0x%llx]\n",
			       ecc_1b_err_adr, ecc_1b_err_adr + 8);
			/* Pop the top of the addr/mask FIFOs */
			ddrss_write(1, DDRSS_ECC_1B_ERR_ADR_REG);
			ddrss_write(1, DDRSS_ECC_1B_ERR_MSK_LOG_REG);
		}
		ddrss_write(DDRSS_V2A_INT_ECC1BERR, DDRSS_V2A_INT_STAT_REG);
	}

	/* 2-bit uncorrectable */
	if (v2a_int_raw & DDRSS_V2A_INT_ECC2BERR) {
		puts("\tECC test: DDR ECC 2-bit error\n");
		ddrss_write(DDRSS_V2A_INT_ECC2BERR, DDRSS_V2A_INT_STAT_REG);
	}

	/* multiple 1-bit errors (uncorrectable) in multpile words */
	if (v2a_int_raw & DDRSS_V2A_INT_ECCM1BERR) {
		puts("\tECC test: DDR ECC multi 1-bit errors\n");
		ddrss_write(DDRSS_V2A_INT_ECCM1BERR, DDRSS_V2A_INT_STAT_REG);
	}
}

/* ddrss_memory_ecc_err()
 * Simulate an ECC error - change a 32b word at address in an ECC enabled
 * range. This removes the tested address from the ECC checks, changes a
 * word, and then restores the ECC range as configured by k3_ddrss in R5 SPL.
 */
static int ddrss_memory_ecc_err(u32 addr, u32 ecc_err, int range)
{
	u32 ecc_start_addr, ecc_end_addr, ecc_temp_addr;
	u32 val1, val2, val3;

	/* Flush and disable dcache */
	flush_dcache_all();
	dcache_disable();

	/* Setup a threshold for 1-bit errors to generate interrupt */
	ddrss_write(1, DDRSS_ECC_1B_ERR_THRSH_REG);

	puts("Testing DDR ECC:\n");
	/* Get the Rx range configuration */
	ecc_start_addr = ddrss_read(DDRSS_ECC_Rx_STR_ADDR_REG(range));
	ecc_end_addr = ddrss_read(DDRSS_ECC_Rx_END_ADDR_REG(range));

	/* Calculate the end of the Rx ECC region up to the tested address */
	ecc_temp_addr = (addr - gd->ram_base) >> 16;
	if (ecc_temp_addr)
		ecc_temp_addr--;
	puts("\tECC test: Disabling DDR ECC ...\n");
	/* Set the new range */
	ddrss_write(ecc_temp_addr, DDRSS_ECC_Rx_END_ADDR_REG(range));
	/* ECC is still on in a single block. The range is disabled if start > end */
	if (ecc_temp_addr == ecc_start_addr)
		ddrss_write(ecc_temp_addr + 1, DDRSS_ECC_Rx_STR_ADDR_REG(range));

	/* Flip some bits, one bit preferably, but let's allow more */
	addr &= ~3;
	val1 = readl((unsigned long)addr);
	val2 = val1 ^ ecc_err;
	writel(val2, (unsigned long)addr);
	val3 = readl((unsigned long)addr);

	/* Re-enable the ECC checks for the R0 region */
	ddrss_write(ecc_end_addr, DDRSS_ECC_Rx_END_ADDR_REG(range));
	ddrss_write(ecc_start_addr, DDRSS_ECC_Rx_STR_ADDR_REG(range));
	/* Make sure the ECC range is restored before doing anything else */
	mb();

	printf("\tECC test: addr 0x%x, read data 0x%x, written data 0x%x, "
	       "err pattern: 0x%x, read after write data 0x%x\n",
	       addr, val1, val2, ecc_err, val3);

	puts("\tECC test: Enabled DDR ECC ...\n");
	/* Read again from the address. This creates an ECC 1-bit error
	 * condition, and returns the corrected value
	 */
	val1 = readl((unsigned long)addr);
	printf("\tECC test: addr 0x%x, read data 0x%x\n", addr, val1);

	/* Set threshold for 1-bit errors to 0 to disable the interrupt */
	ddrss_write(0, DDRSS_ECC_1B_ERR_THRSH_REG);
	/* Report the ECC status */
	ddrss_check_ecc_status();

	dcache_enable();

	return 0;
}

/* ddrss_addr_in_range()
 * Check if the address is valid and within any configured ECC R[0-2] range.
 */
static int ddrss_addr_in_range(phys_addr_t addr)
{
	phys_addr_t ecc_start_addr, ecc_end_addr;
	int i;

	if (addr < gd->ram_base || addr >= gd->ram_top)
		return -1;

	addr -= gd->ram_base;
	/* Find if any of the enabled ECC Rx ranges contains the address.
	 * Only a single range (R0) is usually enabled by k3-ddrss in R5 SPL
	 * for the full memory.
	 */
	for (i = 0; i < K3_DDRSS_MAX_ECC_REGIONS; i++) {
		ecc_start_addr = ddrss_read(DDRSS_ECC_Rx_STR_ADDR_REG(i));
		ecc_end_addr = ddrss_read(DDRSS_ECC_Rx_END_ADDR_REG(i));
		/* Not configured/disabled */
		if (ecc_start_addr >= ecc_end_addr)
			continue;
		ecc_start_addr <<= 16;
		ecc_end_addr = (ecc_end_addr << 16) + 0xffff;

		if (addr >= ecc_start_addr && addr <= ecc_end_addr)
			return i;
	}

	return -1;
}

/* ddrss_is_ecc_enabled()
 * Report if ECC is enabled.
 */
static int ddrss_is_ecc_enabled(void)
{
	u32 ecc_ctrl = ddrss_read(DDRSS_ECC_CTRL_REG);

	/* Assume ECC is enabled only if all bits set by k3_ddrss are set */
	return (ecc_ctrl & (DDRSS_ECC_CTRL_REG_ECC_EN |
			    DDRSS_ECC_CTRL_REG_RMW_EN |
			    DDRSS_ECC_CTRL_REG_WR_ALLOC |
			    DDRSS_ECC_CTRL_REG_ECC_CK));
}

static int do_ddrss_test(struct cmd_tbl *cmdtp, int flag, int argc,
			 char *const argv[])
{
	u32 start_addr, ecc_err;
	int range;

	if (!(argc == 4 && (strncmp(argv[1], "ecc_err", 8) == 0)))
		return cmd_usage(cmdtp);

	if (!ddrss_is_ecc_enabled()) {
		puts("ECC not enabled. Please enable ECC any try again\n");
		return CMD_RET_FAILURE;
	}

	start_addr = simple_strtoul(argv[2], NULL, 16);
	ecc_err = simple_strtoul(argv[3], NULL, 16);

	range = ddrss_addr_in_range(start_addr);
	if (range < 0) {
		printf("Address 0x%x not in any configured ECC range\n",
		       start_addr);
		return CMD_RET_FAILURE;
	}

	ddrss_memory_ecc_err(start_addr, ecc_err, range);
	return 0;
}

U_BOOT_CMD(ddrss, 5, 1, do_ddrss_test,
	   "DDRSS test",
	   "ecc_err <addr in hex> <bit_err in hex> - generate bit errors\n"
	   "	in DDR data at <addr>, the command will read a 32-bit data\n"
	   "	from <addr>, and write (data ^ bit_err) back to <addr>\n"
);

Executing the code in A53 will also encounter the same problem as I do now: SEC's test case can run normally, but DED's test case will result in an External Abort. We have also discussed before that the testing process for SEC and DED should be the same, so what is the reason for this discrepancy?

Hello lina,

Thanks for the sharing code and please allow me one or two days to get back to you .

Regards,

S.Anil.

Hello lina,

Can you please confirm if you are injecting an error for which data, either code memory or data memory?

Regards,

S.Anil,

lina zhang said:

b. Transfer two bit error to an address, for example, flip 0x00010010 at 0x90000020:

I think the address 0x9000,0020 should be a memory of DDR (mapped from 0x8000,0000), and it should be a piece of free memory.

Hello Lina,

Actually, after compilation of your code, you will get a map file in which you can just see that the 0x9000,0020 memory belongs to code memory or data

memory.

Recently, I have seen that one customer also faces similar issues when they inject an error in code memory, soc enters in to exception state . So, I asked

them to inject single-bit error, so they don't get any exceptions.

Here also, I am suspecting the same issue with you since you are corrupting errors in the code.

memory, and then next time the CPU is not able to process the corrupted data , hence you may get an exception.

Regards,

S.Anil.

Hi Anil,

I am supporting this customer from field side, single bit error test is verified successfully based on UBOOT driver, what they want to test now is  DED, any suggestion on DED test as show above?

-Thomas

I don't think I'm using the code memory. I have tested the memory of 0x9000,0000 ~ 0x9001,0000 by mtest, which should include the operation of writing memory, but there is no exception. Additionally, I tested the "ddrss ecc_err" command at addresses 0x9000,0000, 0x9800,0000, 0xa000,0000, and 0xa800,0000, but the results were all the same: SEC was okay but DED crashed. I believe that there shouldn't be such a large amount of code memory in u-boot.

Hello Lina,

Can you please confirm which exception you are getting? In that exception, there is a while (1) condition, so you are not able to come out of that state.

So,try to come out of the while(1) loop condition manually change condition from the debugger, and next see if you are able to get an ESM interrupt or not,

and check the ESM_RAW Registers status as well to monitor ESM interrupt status. With these data, I can check with SDL experts on this topic. As of now,

your code seems to be fine, and I don't have any thoughts about further debugging.

Regards,

S.Anil.

The problem is that I don't know where the while(1) is... I have tried to find the string "Unhandled" in u-boot, but cannot find any code that matches or is similar to "Unhandled External Abort". One of the issues with this post is to figure out where the exception happened, and why it happened.

Here is the Boot Flow when using u-boot: 3.1.1.1. General Information — Processor SDK AM64X Documentation

The table there shows some code that runs on R5. So I'm quite worried and curious, what does the "External" in the screenshot represent? Can you provide anything about this?

Hi Anil, 

"I have verified on R5F, and we are also following the same method, which means that injecting errors is just flipping two bits in the target memory."

What is your inject error address in R5F example? We are trying to dig out any difference between R5 example and above customized uboot code (MMU, ESM, DDR address, DED initialization steps ) , If you have any clues/findings on R5F code, please share with us.

-Thomas

Hello Thomas,

Can you please see the below code, which is mainly used to inject a dual-bit error?

And injecting error address location is

DDR_ECC_TEST_ADDR =  0x9000 0100;

/* Clear any residual ECC errors */
DDR_clearECCError (DDR_ECC_ERR_ALL);

gTest_Addr = (uint32_t *) (DDR_ECC_TEST_ADDR);

CacheP_wbInv ((void *)gTest_Addr, 4, CacheP_TYPE_ALL);
/* Read reference value */
testVal = gTest_Addr[0];
/* flip 2 bits */
testVal2 = testVal ^ 0x00101000u;
/* Calculate translated address */
translatedMemPtr = (volatile uint32_t *)(DDRGetTranslatedAddress ((uintptr_t)gTest_Addr));

/* Generating a 2b ECC error */
/* NOTE: The following section should NOT be useed in actual application */
/* ================================================================================ */

/* Temporarily disable ECC */
DDR_enableInlineECC (0);

/* Now corrupt the value */
*(translatedMemPtr) = testVal2;

/* Make sure the values are written back */
CacheP_wbInv ((void *)translatedMemPtr, 4, CacheP_TYPE_ALL);

/* Enable back ECC */
DDR_enableInlineECC (1);

/* ================================================================================ */

/* Invalidate cache */
CacheP_inv ((void *)gTest_Addr, 4, CacheP_TYPE_ALL);

/* Read value to trigger error */
testVal2 = gTest_Addr[0];

DebugP_log ("Waiting on Dual bit error detection Interrupt...\r\n");

while ((gDedTestPass == FALSE) && (waitCount++ < 100u))
{
ClockP_usleep (100);
}

if (gDedTestPass == TRUE)
{
DebugP_log ("2b ECC error detected\r\n");
status = SystemP_SUCCESS;
}
else
{
DebugP_logError ("2b Inline ECC Test failed timedout ...\r\n");
status = SystemP_FAILURE;
}

/* Restore original value */
gTest_Addr[0] = testVal;

Here, I am attaching an example project for your reference.

ddr_ecc_test_main_esm_am64x-evm_r5fss0-0_nortos_ti-arm-clang.zip

Regards,

S.Anil.

I checked your R5F code and customer code, and find the difference on inject and test steps  are on cache disable and enable sequency, in your code, R5F cache is always enabled and manually snoop operation,   but in customer code (uboot code), there is cache disable -> ecc error injection -> read error address ->cache enable, to keep the test is apple to apple, is it possible to disable cache to have a try?

As the crash happed immediately, it is hard to check DDR16SS0_DDRSS_DRAM_ECC_UNCORR_ERR_LVL_0 event status which is mapped to ESM0_LVL_IN_69.

Hello Thomas,

I can do this testing and let you know the status. I have seen your email to speed up this issue on your boot. I need to take help from Linux experts to further

debug the issue.

Regards,

S.Anil.

HI  Anil,

Any update from team?

-Thomas

Hello Thomas,

Internally checking with the team, how can we proceed further with this implementation on the A53 core?

I can update the status next week as to how we are proceeding further. As you know, this implementation is available on R5F and not on the A53 core.

Regards,

S.Anil.

Hi  Anil,

Ping you on any update?

Thanks!

-Thomas

Thomas,

Really sorry, I have not gotten any update on this from Development. Here, test code should be implemented on A53, and our expert is RTOS.

So, taking  more time how to proceed this one . I can check Anshu/Mukul will update status by eod.

Regards,

S.Anil.

Anil,

Any update?ping again.

-Thomas

Thomas,

Status updated by Mukul from Mail .

Regards,

S.Anil.