DM36x IPNC crash / hang with no error message - how to debug ?

Typically this type of issue is root-caused to either issues with power supply integrity or DDR2 configuration/integrity. Check the following:

1. Check that there aren't any droops on neither the Core supply, 1.8v supply or 3.3v supply.  Check this with a high speed scope near the pin of the processor

2. Have the data sheet DDR2 guidelines been used?  Does the DDR2 routin been checked to fully comply with he data sheet?

3. Have you ensured that the EMIF timings have been properly programmed for the type of DDR2 you're using on your board?

Also, prior to this batch, did you have any boards working fine?  What exact device part number are you using?

Hi,

Which encoder is running with IPNC? is it H264 or MPEG4? or both?. when it is hung is linux console is responsive or non-responsive? means is whole system got struct?(you have a told no response to confirm just asking again).

Can you run with CE_DEBUG=3 and send us the log? what is the version of the codec you are using?

Thanks,

Veeranna

One more item: make sure you're outputting properly to the right console (i.e. console=ttyO0) and that there's no typos in your bootargs.

Ivan, thanks very much for your reply. In response to your questions:

1. Good suggestion, we will check this. The power and main boards are a little bit close together which makes probing difficult until we substitute a flexible ribbon connector instead of the rigid inter-board connector, but we'll do that so that we have full probing access to at least one of the board sets.

2. Our camera design followed the IPNC design exactly for the DDR2, including the routing of that area of the PCB, so this part should be met. We'll double check that the original IPNC design follows all the Samsung and TI DDR2 guidelines.

3. We did not change any timing, it is all exactly as found in the IPNC design. The BOM for IPNC DM368 specifies Samsung K4T1G164QE-HCE7 and we're using the same part which is now called K4T1G164QF-BCE7. (Apparently there was a mask change from E step to F step, which should not affect anything that we are aware of)

4. Prior to this batch we had just a few prototypes that were not tested continuously so we don't have previous data.

On this lot, the TI part number is TMS320DM368ZCEDF. Other numbers appearing on the chip are: 29A6K6W, and 570, GI

For these tests, we run H264 only. Correct, when it is hung, the console no longer responds. Since we have no problem with the console prior to hanging, we are definitely using the correct console and UART parameters. It is interesting that the Ethernet link and activity LEDs continue to function, so the Davicom Ethernet chip is not hung.

How do we use CE_DEBUG=3 ? Is this a kernel parameter in the boot args or compiled in the code ? We would be glad to send the log.... we tried it with different versions of IPNC software with similar results. Do you prefer boot log from MontaVista 2.6.18 (IPNC DM368 IMX036 version 3.010) or 2.6.37 (Raghu's new V5 with IMX036 SPI Kernel Patch) ? Or is MT5 version preferred over IMX036 ?

As issue coming on all the version, you can send the log with latest version. To run with CE_DEBUG=3, you need to pass this as command line argument while running the .out.

root@login>> CE_DEBUG=3 ./av_server.out ------ (remaining options)

Veeranna, thanks very much for your followup.

Well, we spent the weekend trying to narrow down the problem, with no success. Attached are 2 boot logs, one from running your binary with MT9P031 sensor and one with our compiled version of V5 software that has no modifications except for Raghu's spi_kernel_patch, selecting IMX036 sensor, and adding the CE_DEBUG=3 case just for the H264 D1 case.

When running your binary on our board, there is an error message about the MT9P031 sensor but it still works. When we make the required modifications to the code the error message disappears. Also, If we use the IMX036 sensor with Raghu's modifications then it works.

The reason why we added the CE_DEBUG=3 only to the case for D1 resolution is so that we can show you a clean boot of the default use case (720P) then verify in the GUI that the sensor works, and use the GUI to select D1 which then also activates the CE_DEBUG=3. But then we let it run for many hours and it didn't hang, so it's not clear if adding the CE_DEBUG=3 slows down the processing and avoids the hang, or if the D1 resolution doesn't cause the hang. We'll let it run for a day to see what happens.

To recap the situation again: our camera boards are very similar in construction to the IPNC DM368 design, in fact so close that we can run IMX036 version binary without modification and we can run the MT5 binary if we modify just a couple of bit positions. We built a manufacturing lot and some cameras seem to work fine, but a few of them hang with no error message. On a few boards this happens relatively quickly (after a few hours) while on other boards it may take a day or 2. Other boards were tested and can run for weeks.

So we're trying the CE_DEBUG=3 to see if we can figure out where it hangs so that we can find out what component on the board may be responsible. The DM368 is running at the default 432-340 speed and the DDR is the correct matching speed grade (HCE7: 400/800 Mhz).

In the boot logs we started out by running the diagnostics to show that the DDR test passes, and we paused in uBoot to print out the environment variables, and then after the bootup was complete we also issued the dmesg command to show kernel logs and also used the ps / top commands to show running processes.

Please let us know if we can help with more logs. We will also continue to double-check the stability of all power supplies with a high speed scope.

Thanks for the log and detailed description.

From the log we wanted check hang is happening inside codec or outside the codec. But hang is not happening with debug prints, so its still not clear. It gives one information that  slowing down the system is avoiding the hang.

Are you doing any resolution change in your usecase?. And also can you try with some prints in application side instead of CE_DEBUG?

Thanks for your continued followup, we really appreciate it as we put all production on hold while trying to find this issue.

What we found so far:

- some of the boards from this manufacturing lot have been working continuously for several weeks, with different resolutions, different versions of software, even different image sensor configuration

- the problem boards hang after a few hours or at most 1 day.

- it is not likely a temperature or connection problem of any kind. Pushing the reset on the debug board, without any kind of touching of the main board, makes the camera restart immediately and work for another few hours after the hang, and attempts at mild heating of the board, slight bending or tapping have not caused the hang

- 24 Mhz crystal is spot on. Loading caps were not 100% correct, but correcting them did not make a difference

- At D1 resolution the problem does not occur even after waiting 2 days. (Maybe it takes longer, we didn't try)

- At 720P (30 fps / 4000 kpbs) resolution the problem also doesn't occur after several days, except when we turn on the time / date display. Then the camera hangs after a couple of hours

- using IMX036 sensor with 5.0 software at full resolution of 3MP with 4000 kbps the problem does occur even without time / date display

- since using IMX036 or MT9P sensor does not make a difference, the problem does not appear related to SPI

- Appro 2A or TI 2A made no difference. With auto iris or fixed Iris made no difference. PoE or 12V power supply made no difference.

- no audio was used for any of the experiments, no settings other than default except just the specific resolutions as listed above, and with or without date/time display as listed above

- many repeated diagnostic program cycles with DDR test did not produce the hang or an error message

- replacing Davicom Ethernet chip with a new one made no difference

- in general, the board build quality and solder quality as observed under inspection microscope looks flawless. All PCBs were electrically tested prior to assembly.

- the only interesting fact so far: when the camera hangs and is restarted with a reset, the RTC time is lost. But diagnostic program shows RTC working fine

Question for you:

Does the fact that the RTC time reverts to 1/1/1970 have any significance about what might be causing the problem ? Please note that at no time do we ever interrupt the power, and also our camera has the exact same battery backup circuit with Elna supercap as in the original IPNC design.

We will proceed to check every power supply with a 500 Mhz / 1 gigasample scope, with 1 Ghz probes, and compare overshoot/undershoot to the IPNC test report.

>>- At 720P (30 fps / 4000 kpbs) resolution the problem also doesn't occur after several days, except when we turn on the time / date display. Then the camera hangs after a couple of hours

>>- using IMX036 sensor with 5.0 software at full resolution of 3MP with 4000 kbps the problem does occur even without time / date display

It seems like this is not a DDR2-related problem if it can run for a long time w/o failure at 720P, or 3MP.  What happens if you disconnect the RTC circuit to help isolate?

>>Does the fact that the RTC time reverts to 1/1/1970 have any significance about what might be causing the problem ? Please note that at no time do we ever interrupt the power, and also our camera has the exact same battery backup circuit with Elna supercap as in the original IPNC design.

Hmm...are you using an external RTC?  Or the on-chip RTC?

Ivan, thanks for the followup. We use the same on-chip RTC circuit as the circuit used in the DM368 IPNC. It is certainly worth trying to disable it.... so we take out the 32.768Khz crystal and ground the pins ? An oscilloscope on the crystal showed a very accurate 32.768Khz waveform with the Seiko crystal that we used.

Glad to know that it is unlikely to be a DDR2 issue.

By the way in a previous message it was suggested that we should use a high speed scope to verify the voltages. But when we did that, we found significantly more ripple than what is shown in the IPNC DM368 test report, until we saw that the IPNC DM368 test report did the measurement of the ripple with bandwidth limiting turned on. Once we used bandwidth limiting we got comparable ripple results.

Ivan, yes, we use the same exact circuit and power supply as the Appro IPNC design. Also, for debugging, we tried the same binaries as the earlier version Appro DM368 IMX036 software, and same results. Now we are trying mostly the latest version 5.0 but with Raghu's modifications for IMX036. So unless the standard release of IPNC DM368 software has a bug that somehow is activated more frequently in one of our boards, it is unlikely to be a pure software issue.

So, we will continue to check the power supplies in detail, and thank you for your suggestion to focus on that.

Ivan, thanks very much for your suggestion (and second reminder) to use a high speed scope. We've definitely found a problem, and it's been several hours since fixing it and so far it works. Not sure if this was the last problem or the specific problem that has been giving us this trouble, but the design certainly works better now than before the troubleshooting.

Right now, it looks like the specific modes that caused or didn't cause the failures had nothing to do with the software, other than just consuming different amounts of power on the DM368 and the rest of the board circuitry.

The IPNC DM368 testing report shows ripple measurements with bandwidth limitation. Fortunately your suggestion to allow > 200 Mhz allowed us to capture some pretty nasty transients. Now it's the opposite, we wonder how some of the other boards are even working!

Again, thanks so much for your detailed response, which helped us improve the design. Hopefully we found the last problem but if not, we know how to check. Best Regards