AM62A7-Q1: ECC error after enabling inline DDR ECC

What is the full part number of the AM62A device you are using?  You need to check the speed grade of the processor, because some speed grades only have a max of 1600MHz for DDR

Refer to the following sections in the datasheet:

Regards,

James

Hi James

We are using speed grade V, which theoretically supports 1866MHz.

Best regards!

XUE Fadong

Ok, that confirms you can run the DDR interface up to 1866MHz. 

Please try without changing these parameters:

DDRSS.lpddr4.config_dram_tREFIpb_ns = 488;
DDRSS.lpddr4.config_dram_tREFIab_ns = 3906;
DDRSS.lpddr4.config_dram_tRASmax_ns = 35154;

These parameters automatically change when choosing an operating temp range. 

As a second experiment, try running at 1600MHz.

When you said you are getting ECC interrupt, are you getting correctable ECC errors, or uncorrectable ones? 

Regards,

James

Hi James

Our parameter configuration is as follows:

And we tested enabling DDR ECC at 1600MHz and encountered ECC errors.

When an ECC interrupt occurs, it can be a correctable ECC error or an uncorrectable error, both of which have been encountered.

Best regards!

XUE Fadong

Please try with the attached configuration.  This has just minimal changes in the config.  

/cfs-file/__key/communityserver-discussions-components-files/791/ddr_5F00_cfg_5F00_simplified.zip

When you ran memtester with ECC disabled, how many loops did you run?  And did you run across your operating temp range, especially at high temp?  What you are encountering seems to be an infrequent marginality issue, which may be revealed in a memtester run at high temps.

Can you also use https://www.ti.com/tool/download/DDR-MARGIN-FW/1.9.0 (instructions are in the tool).  This is a virtual eye tool which will give you an idea of the marginality in your design.

Regards,

James

Hi James

We conducted tests based on the configuration file you provided

/cfs-file/__key/communityserver-discussions-components-files/791/ddr_5F00_cfg_5F00_simplified.zip

However, an illegal access interrupt occurred; the cause is currently unknown.


We also conducted eye diagram testing on this; the test results are as follows.
Teye_micron_2GB_1600.pdf


We also tested the configuration currently in use (1865 MHz); the eye diagram is shown below:
Teye_micron_2GB_1865.pdf
Based on this eye diagram, the margins for DQ26 through DQ29 appear to be very low. Could this be the cause of the DDR ECC errors? How should we go about optimizing this?

When the ECC error occurred, the SoC temperature was approximately 70+ degrees Celsius.

Furthermore, based on our current configuration, simply changing the frequency to 1600 MHz also results in ECC errors.

Best regards!

XUE Fadong

Hi XUE,

The Teye_micron_2GB_1600.pdf show much wider eyes on each of the bits than Teye_micron_2GB_1865.pdf.  Although it is difficult to determine if this is because of the frequency difference or a configuration difference.  

I would recommend continuing to debug with the configuration i gave you, and try to figure out the illegal access interrupt.  It looks like is that something to do with an ESM event, although i don't know why you would his this error with the DDR config i gave you, it should not have mattered.  Maybe you can disable this event for now or somehow determine why you are getting this interrupt.  If you can provide a full log, maybe our software team can help.

Regards,

James

Hi James

We conducted tests based on the configuration file you provided and disabled other ESM interrupts; the result was that the R-core hung, regardless of whether DDR ECC was enabled.

The R-core hangs unless `tREFlab` is changed to 3906. We checked the DDR MR4 register and found it is currently in 1× Refresh mode, whereas we had set `tREFlab` for 4× Refresh. Could this discrepancy be the cause of the issue?


DDR ECC enablement tests were conducted with only the tREFlab parameter modified to 3906, yet ECC errors still occurred.

Best regards!

XUE Fadong

Hi XUE,

the fact that you can get things working by reducing the refresh rate indicates that there maybe some bandwidth limitation that may be hit with a high refresh rate.

Try using the configuration below, which is configured for 85C max, this will keep the refresh rate at 1x

/cfs-file/__key/communityserver-discussions-components-files/791/ddr_5F00_config_5F00_simplified_5F00_1600MHz_5F00_85C.zip

If the board boots with this config, try the memtester tests.  If you can get the tests to pass, then at least we have a baseline configuration that works with ECC enabled.  

Then we can work on increasing frequency.  Will you application need to run greater than 85C?  

Regards,

James

Hi James

Reducing the frequency does not solve the problem; it merely reduces the probability.
/cfs-file/__key/communityserver-discussions-components-files/791/ddr_5F00_config_5F00_simplified_5F00_1600MHz_5F00_85C.zip
We still encounter ECC errors with this configuration, and even though we are continuously cooling the SoC, the interrupts occur at temperatures below 60°C.
Our product has high temperature requirements and needs to be able to operate at 105 degrees.

Are there any other configurations we can modify and test? Currently, we do not have a configuration that avoids ECC errors.

Best regards!

XUE Fadong

Hi XUE,

let me ask a few more questions to get some context:

-it appears you are using SBL to boot into linux.  What is this log from?  It doesn't look like a linux log.  Is this some monitoring log running from R5?

-This snippet is from the u-boot dtb, but that is irrelevant since you are using SBL.  Can you show the change you made in the kernel dtb?

Fadong Xue said:

3. Update dtb in u-boot-spl & u-boot & kernel

-When the ECC occurs, can you show the log in linux?  The memtester log you posted doesn't show any memtester errors.

-Does linux crash when the ECC errors occur? 

-What is your system doing when the ECC errors occur?  For example, are you just running memtester when the errors occur?  Are you performing any application level operations?  For example, any data transfers with other peripherals in the system?  How many processor cores are executing at the same time?   

-you indicated a temperature sensitivity.  So do you ramp the temperature and start seeing failures?  Or are you booting and running at a certain temperature?

-you mentioned you were getting both correctable and uncorrectable errors.  Are you able to boot and run for some time before hitting the uncorrectable errors?  Or do these errors occur during boot and you system just never can get through the boot stage?

Regards,

James

Hi Jamms

1. We boot using the SBL, and that log originates from the R-core. We have enabled ESM ECC fault detection on the R-core; a reset signal is generated when an ECC fault is triggered.
2. The modifications in the kernel are similar to those in U-Boot, limiting the total memory size.

3. The Linux system exhibits no abnormalities when an ECC error occurs.
4. The Linux system did not crash.
5. Our application running on the A-core utilizes shared memory for inter-core communication with the R-core. We have observed that all memory regions reporting DDR ECC errors are located within this specific shared memory area. What could be the cause of this?
6. We also encounter this issue at around 60°C; temperature does not appear to be the definitive cause.
7. The system operates normally for a period of time before an ECC error occurs.

Best regards!

XUE Fadong

Can you give us more details on the scenario which causes these errors.  What type of communication are you performing between A53 and R5?  What type of data is being shared?  Is there any handshaking or semaphores to ensure A53 and R5 are not accessing the same region at the same time?  What is the nature of the shared data (element size, block size, structures, single data words, etc)?

Regards,

James

Just a follow up here, we'd like to reproduce this failure on our EVM if possible.  If you have an EVM, we can really accelerate the debug if you can reproduce this on an EVM, or give us enough information about the failure scenario so we can try to reproduce it.

Regards,

James