AM3894: Implications of DDR3 Slave Ratio Search Program failing to converge / reporting 0.

We recently updated a board design that uses the AM3894 and are trying to put a bow on the DQS RAM leveling values for the new design and DDR3 memory.  Our latest exercise was intended as a re-affirmation of the leveling values that were determined using the Slave Ratio Search Program with the prototype spin of the latest design.  However, we are running into issues with the DQS leveling process on the released hardware.

The issue at hand is what we understand to be a failure to converge on an acceptable value and a output result of 0.  We encounter this frequently for multiple DQS leveling values.  We have made some refinements to the parameters used to calculate our seed values, but the consequent seed value changes were minor and the results have not changed.  When we do happen to get a "clean run", the resulting outputs are not appreciably different than the values we determined using the prototype boards.  Since we don't know specifically what is occurring while the test is running, we are not sure what the implications are.  Our concern is that this behavior is indicative of a broader issue that may present itself as memory failures during runtime, for example an unacceptable error rate at or near the entered seed value, but our memory testing results so far show no memory errors.

Is our concern here valid?  What might cause the program to fail to converge and report a 0 on a functional memory bus?  Are there any resources we can use to get smarter on the specific process this program uses to tune the memory?

For some history and additional context: 

We ran a prototype spin and characterized the DQS read and write leveling values using the instructions and resources included in our archived copy of the "DM816x C6A816x AM389x DDR3 Init" processors wiki page.  We updated these values in our Uboot/OS and ran memory testing across our required ambient temperature range.  No errors were observed during memory testing, so we released the board.  Months later we received the released boards and immediately began memory testing at temperature using the same Uboot/OS settings.  Memory tests passed and we were happy.  In an effort to document the process of determining the DQS leveling values we re-ran the Slave Ratio Search Program on the released boards, but we are running into frequent issues.  We did move back to the prototype board to verify there hadn't been a change in any settings, and the process ran without issues for several runs.  There were no changes to the memory routing or PCBA materials between the two board revisions.

Five prototype boards were used to determine the DQS leveling values used today.  The Slave Ratio Search Program was run multiple times at multiple temperatures on each board, and the values used are the averages of this data set.  The results were consistent across runs, across boards, and changed little over temperature.  At least 10 of the released boards have successfully ran persistent memory tests for 60+ hours uninterrupted without failures.  Issues have been observed on 5 out of 5 released boards while running the Slave Ratio Search Program, 2 of which were part of the 10 boards used for memory testing.

An example of the output we're seeing is shown below:

Enter 0 for DDR Controller 0 & 1 for DDR Controller 1
0
DDR START ADDR=0x80000000

Enter the Seed Read DQS Gate Ratio Value in Hex to search the RD DQS Gate Window
1EC

Enter the Seed Read DQS Ratio Value in Hex to search the RD DQS Ratio Window
40

Enter the Seed Write DQS Ratio Value in Hex to search the Write DQS Ratio Window
EC
RD DQS RATIO MINIMUM VALUE DIDN'T CONVERGE
RD DQS RATIO MINIMUM VALUE DIDN'T CONVERGE
RD DQS GATE RATIO MINIMUM VALUE DIDN'T CONVERGE
RD DQS GATE RATIO MINIMUM VALUE DIDN'T CONVERGE
RD DQS RATIO MINIMUM VALUE DIDN'T CONVERGE
RD DQS RATIO MINIMUM VALUE DIDN'T CONVERGE
RD DQS RATIO MINIMUM VALUE DIDN'T CONVERGE
RD DQS RATIO MINIMUM VALUE DIDN'T CONVERGE
*********************************************************
Byte level Slave Ratio Search Program Values
*********************************************************
BYTE0 BYTE1 BYTE2 BYTE3
*********************************************************
Read DQS MAX 0 48 20000019 2000001a
Read DQS MIN 0 37 3c 0
Read DQS OPT 0 3f 1000002a 1000000d
*********************************************************
Read DQS GATE MAX 0 1f2 b9 bb
Read DQS GATE MIN 0 1db ab b0
Read DQS GATE OPT 0 1e6 b2 b5
*********************************************************
Write DQS MAX f1 f2 a0 0
Write DQS MIN 0 ea 88 90
Write DQS OPT 78 ee 94 80000048

===== END OF TEST =====

--- snip ---

Enter 0 for DDR Controller 0 & 1 for DDR Controller 1
0
DDR START ADDR=0x80000000

Enter the Seed Read DQS Gate Ratio Value in Hex to search the RD DQS Gate Window
1A2

Enter the Seed Read DQS Ratio Value in Hex to search the RD DQS Ratio Window
40

Enter the Seed Write DQS Ratio Value in Hex to search the Write DQS Ratio Window
C5
RD DQS RATIO MINIMUM VALUE DIDN'T CONVERGE
RD DQS RATIO MINIMUM VALUE DIDN'T CONVERGE
RD DQS RATIO MINIMUM VALUE DIDN'T CONVERGE
RD DQS RATIO MINIMUM VALUE DIDN'T CONVERGE
*********************************************************
Byte level Slave Ratio Search Program Values
*********************************************************
BYTE0 BYTE1 BYTE2 BYTE3
*********************************************************
Read DQS MAX 0 35 32 33
Read DQS MIN 0 2e 2d 2a
Read DQS OPT 0 31 2f 2e
*********************************************************
Read DQS GATE MAX 0 13b 13c 13e
Read DQS GATE MIN 0 12f 12b 137
Read DQS GATE OPT 0 135 133 13a
*********************************************************
Write DQS MAX 0 200000b4 200000b5 200000ba
Write DQS MIN c0 bc c4 c2
Write DQS OPT 80000060 100000b8 100000bc 100000be

===== END OF TEST =====

--- snip ---

Enter 0 for DDR Controller 0 & 1 for DDR Controller 1
0
DDR START ADDR=0x80000000

Enter the Seed Read DQS Gate Ratio Value in Hex to search the RD DQS Gate Window
182

Enter the Seed Read DQS Ratio Value in Hex to search the RD DQS Ratio Window
40

Enter the Seed Write DQS Ratio Value in Hex to search the Write DQS Ratio Window
98
RD DQS RATIO MINIMUM VALUE DIDN'T CONVERGE
RD DQS RATIO MINIMUM VALUE DIDN'T CONVERGE
RD DQS RATIO MINIMUM VALUE DIDN'T CONVERGE
RD DQS RATIO MINIMUM VALUE DIDN'T CONVERGE
RD DQS RATIO MINIMUM VALUE DIDN'T CONVERGE
RD DQS RATIO MINIMUM VALUE DIDN'T CONVERGE
RD DQS RATIO MINIMUM VALUE DIDN'T CONVERGE
RD DQS RATIO MINIMUM VALUE DIDN'T CONVERGE
RD DQS RATIO MINIMUM VALUE DIDN'T CONVERGE
RD DQS RATIO MINIMUM VALUE DIDN'T CONVERGE
RD DQS RATIO MINIMUM VALUE DIDN'T CONVERGE
RD DQS RATIO MINIMUM VALUE DIDN'T CONVERGE
RD DQS GATE RATIO MINIMUM VALUE DIDN'T CONVERGE
RD DQS GATE RATIO MINIMUM VALUE DIDN'T CONVERGE
RD DQS GATE RATIO MINIMUM VALUE DIDN'T CONVERGE
RD DQS GATE RATIO MINIMUM VALUE DIDN'T CONVERGE
*********************************************************
Byte level Slave Ratio Search Program Values
*********************************************************
BYTE0 BYTE1 BYTE2 BYTE3
*********************************************************
Read DQS MAX 0 0 0 44
Read DQS MIN 0 0 0 0
Read DQS OPT 0 0 0 22
*********************************************************
Read DQS GATE MAX 0 0 0 194
Read DQS GATE MIN 0 0 0 180
Read DQS GATE OPT 0 0 0 18a
*********************************************************
Write DQS MAX 0 a8 9b a0
Write DQS MIN 0 0 0 96
Write DQS OPT 0 54 4d 9b

===== END OF TEST =====