AM5729: Unhandled fault: Alignment Exception

Jared Silbermann

Part Number: AM5729

Tool/software:

Hi,

I am running a bunch of software on an AM5729 custom board under Linux SDK 6.03. I am having intermittent issues (and seemingly random issues) that manifests in errors as such:

[ 104.776326] Alignment trap: not handling instruction ed927b00 at [<b4f57638>]
[ 104.784006] Unhandled fault: alignment exception (0x001) at 0xea2edebf
[ 104.790913] pgd = ceb88000
[ 104.793778] [ea2edebf] *pgd=00000000

These errors come from different applications at different times and do not seem to follow any logical trend as to when they occur. Sometimes I can run the software stack for hours without seeing the issue and other times it happens almost immediately when the applications begin.

I am a bit stuck on how to proceed and was hoping to get some advice. Some things I have done so far:
- Run applications using gdb to ensure is not a software fault in terms of memory management causing these issues. The fact that different applicatons cause the problem (it is not always the same one) seems to rule out this

- Run the am57xx-ddr.dss to verify that the changes I have made from the EMIF Tool have successfully gone through via u-boot - they seem to be as expected

- Run memtester for an extended period of time - no problems

- Used stressapptest to test the system under duress. Everytime it is run I see issues such as - this is the red flag that I keep coming back to which seems to suggest there is a problem with my memory setup, however, I cannot seem to find any issues

Log: CrcCopyPage CRC mismatch aaaaaaaa01aaaaaaaa01aaffffff5700aaffffff57f4 != aaaaaaaa01aaaaaaaa01aaffffff5700aaffffff5700, but no miscompares found. Retrying with fresh data.
Process Error: CPU 0(0x2) CrcCopyPage CRC mismatch aaaaaaaa01aaaaaaaa01aaffffff5700aaffffff57f4 != aaaaaaaa01aaaaaaaa01aaffffff5700aaffffff5700, but no miscompares found on second pass.
Report Error: miscompare : DIMM Unknown : 1 : 4s
Hardware Error: miscompare on CPU 0(0x2) at 0x968e0000(0x2e60d0:DIMM Unknown): read:0x5555555555555555, reread:0x5555555555555555 expected:0x0000000000000000

I am currently running out of things to try so any advice / recommendations on how to proceed to track down this issue would be greatly appreciated.
Thanks,
Jared

21 days ago

0 Josue Zamitiz-Ayala 20 days ago

TI__Mastermind 20844 points

Hello Jared,

I will confer internally and get back to you in a couple of days.

-Josue

0 Josue Zamitiz-Ayala 20 days ago in reply to Josue Zamitiz-Ayala

TI__Mastermind 20844 points

Jared,

See the following existing threads: https://e2e.ti.com/support/processors-group/processors/f/processors-forum/41140/alignment-trap-error-on-dm365/143351#143351

This seems like it is an issue from the CPU reading mis-aligned memory.

Are you using the toolchain recommended by our release notes?

Another issue where this was observed by a different customer: https://e2e.ti.com/support/processors-group/processors/f/processors-forum/322543/alignment-issues-after-upgrading-to-linaro-4-7

-Josue

0 Jared Silbermann 20 days ago in reply to Josue Zamitiz-Ayala

Prodigy 155 points

Hi Josue,

I am using arm-linux-gnueabihf-gcc. Regarding those threads, If the same software was crashing everytime in the same place I would expect it to be a specific software fault. It also doesn't rule out the issues I am having using stressapptest.

I guess my main concern at the moment is it is a fault with the memory setup / configuration on our custom board. Can you recommend a way that I can comprehensively validate the memory setup on my custom board? From what I have read memtester is a start but does not rule out all problems.

0 Josue Zamitiz-Ayala 19 days ago in reply to Jared Silbermann

TI__Mastermind 20844 points

Hello Jared,

I am still looking at this internally. Thank you for your patience. You are right that this seems like the DDR is not correctly calibrated: See fedevel.com/.../arm-ddr3-memory-testing

I will assign this to or HW DDR engineer for guidance.

-Josue

0 Jared Silbermann 13 days ago in reply to Josue Zamitiz-Ayala

Prodigy 155 points

Hi Josue,

Have you got any update from the HW DDR engineer. Thanks

0 Josue Zamitiz-Ayala 13 days ago in reply to Jared Silbermann

TI__Mastermind 20844 points

Jared,

The thread has been assigned to them and they will answer whenever they get a chance.

In the meantime I suggest you go over the all the values you input into the EMIF tool and make sure you followed all the directions correctly:

See the following FAQ: https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1000823/faq-processor-sdk-am57x-steps-to-modify-ddr-ram-configurations-for-custom-boards

-Josue

0 Kevin S 12 days ago in reply to Josue Zamitiz-Ayala

TI__Mastermind 32135 points

Hi,

Are you observing this issue on a single board, or multiple boards? (how many boards have been tested)? If you have only tested on a single board, can you try a few additional boards and observe if you see the same issue?

Checking the impact of frequency / voltage can be a good debug step. For voltage, you could try increasing / decreasing by ~ 30 mV. For instance, if the failure improves (or gets worse) when changing the CPU voltage, then it would tell us that the issue is less likely DDR related. The DDR interface is generally sensitive to the vdd_core supply. (though the DDR IO voltage should also be checked).

Regards,
Kevin

0 Jared Silbermann 5 days ago in reply to Kevin S

Prodigy 155 points

Hi Kevin,

This issue is appearing on all boards tested so far. Changing the frequency / voltage seems to have no noticeable effect.

Processors

Processors forum

AM5729: Unhandled fault: Alignment Exception