AM3874/DM8148 HDVPSS fails on its firmware loading after a watchdog reset occurs

Hi Maynard,

How exactly do you trigger the Watchdog reset?

When I run the saWatchdog executable available at {EZSDK}/example-applications/linux-driver-examples-psp04.04.00.01/watchdog/bin/ folder on the DM814x/AM387x EVM, the HDVPSS firmware is loaded properly, I do not observe any issue.

root@dm814x-evm:~# ./saWatchdog 12

Current timeout value is : 60 seconds

New timeout value is : 10 seconds
Current timeout value is : 10 seconds
Write succeeded


U-Boot 2010.06 (Dec 26 2012 - 18:17:23)

TI8148-GP rev 2.1

ARM clk: 600MHz
DDR clk: 400MHz

..............................................

Loading HDVPSS Firmware
FIRMWARE: I2cInit will be done by M3
FIRMWARE: Memory map bin file not passed
Usage : firmware_loader <Processor Id> <Location of Firmware> <start|stop> [-mmap <memory_map_file>] [-i2c <0|1>]
===Mandatory arguments===
<Processor Id>         0: DSP, 1: Video-M3, 2: Vpss-M3
<Location of Firmware> firmware binary file
<start|stop>           to start/stop the firmware
===Optional arguments===
-mmap                  input memory map bin file name
-i2c                   0: i2c init not done by M3, 1(default): i2c init done by M3
FIRMWARE: isI2cInitRequiredOnM3: 1
FIRMWARE: Default memory configuration is used
Firmware Loader debugging not configured
Default FL_DEBUG: warning
Allowed FL_DEBUG levels: error, warning, info, debug, log
MemCfg: DCMM (Dynamically Configurable Memory Map) Version :  2.1.2.1
FIRMWARE: 2 start Successful

Can you try with this saWatchdog executable? Do you have the same "firmware load" issue?

Refer also to the below wiki page:

http://processors.wiki.ti.com/index.php/TI81XX_PSP_WDT_Driver_User_Guide#Sample_Applications

BR
Pavel

Hi Pavel,

We are using the busybox version of the watchdog application.  One way to trigger was to just terminate the watchdog (busybox) application.  The other method was to trigger a linux kernel hang.  Both triggers will result in our issue.

I did try the saWatchdog application you suggested and it does encounter the same issue we have where the watchdog reset will create a failure when loading the HDVPSS firmware.

Regards,
Maynard

Maynard,

Maynard Cabiente said:

I did try the saWatchdog application you suggested and it does encounter the same issue we have where the watchdog reset will create a failure when loading the HDVPSS firmware.

Did you try saWatchdog on DM814x/AM387x EVM or on AM387x custom board? What version of the EZSDK you are using?

Regards,
Pavel

Hi Pavel,

I tried it both on the DM814x EVM board and our own AM3874 custom board.  Both boards have the same issue.  We based the software on EZSDK 5.04..00.11.

We did change the memory mapping on both the DM814x EVM and our own AM3874 custom board to only have 512 MB of DDR3 and only using EMIF0.  But, I don't believe this can cause an issue as it has been working for us for over 6 months now.  This issue is only encountered when the watchdog reset occurs.

We also modified prcm_config as the AM3874 chip does not contain the HDVICP2 module.

Regards,
Maynard
 

Maynard,

Maynard Cabiente said:

For the DM8168 boards, the WD_OUT pin is gated to the PORn (power-on reset) pin of the DM8168 board.  So, in this case, HDVPSS will probably get reset correctly and the problem will not happen.

Maynard Cabiente said:

For the DM8148 boards, RSTOUTn_WD_OUT is not gated to the PORn pin of the DM8148 board.  We just followed this part of the circuitry in our AM3874 board.

Do we need to follow the DM8168 EVM board watchdog reset circuitry and connect it to the PORn pin to make this work properly? 

No, I do not think you should follow the DM8168 EVM watchdog reset circuitry for DM814x/AM387x board. The DM816x device has silicon bug, described in the DM816x Silicon Errata: http://www.ti.com/lit/er/sprz329d/sprz329d.pdf

Maynard,

Maynard Cabiente said:

We did change the memory mapping on both the DM814x EVM and our own AM3874 custom board to only have 512 MB of DDR3 and only using EMIF0

Maynard Cabiente said:

We also modified prcm_config as the AM3874 chip does not contain the HDVICP2 module.

I suspect that this might be an issue, if not done properly. Please have a look in the below E2E threads:

http://e2e.ti.com/support/dsp/davinci_digital_media_processors/f/716/t/239149.aspx

http://e2e.ti.com/support/dsp/davinci_digital_media_processors/f/716/p/254918/892158.aspx

http://e2e.ti.com/support/embedded/bios/f/355/t/265428.aspx

http://e2e.ti.com/support/embedded/linux/f/354/t/195025.aspx

http://e2e.ti.com/support/dsp/davinci_digital_media_processors/f/716/p/226016/797378.aspx

http://e2e.ti.com/support/dsp/davinci_digital_media_processors/f/716/p/186702/838821.aspx

Regards,
Pavel

Maynard,

This wiki page looks also related:

http://processors.wiki.ti.com/index.php/SysLink_FAQs#Ipc_control.28STARTCALLBACK.29_timeouts_on_TI81XX_device_when_only_the_M3_Core_1_is_used

Regards,
Pavel

Hi Pavel,

Thanks for the links.  I look at most of it.  But, the last thread:  

http://e2e.ti.com/support/dsp/davinci_digital_media_processors/f/716/p/186702/838821.aspx

is the one that prompts me to ask my initial question.  If you look at this thread and the post by Sivaramkrishna Neeruganti on Jan 17 2013 01:19 AM, he had a similar issue on the DM814x/AM387x board when the watchdog reset occurs.  Unfortunately, there was no more reply to his watchdog reset issue.  Thus, it lead me to the DM816x EVM board hardware work-around for the watchdog.

We have no issue with the firmware loader for HDVICP2 and HDVPSS on power cycle, reboot, or push-button reset.  We only have this issue when the watchdog reset occurs.  This looks like there is something in the HDVPSS module that requires a (cold) reset as a push-button reset will fix this issue.  The push-button reset is actually connected to the POR pin.

Regards,
Maynard
 

Maynard,

As I stated, the watchdog reset (triggered from the saWatchdog application) is working fine on my own DM814x/AM387x EVM rev 2.1, the HDVPSS firmware is reloaded fine, without need to execute POR reset.

What is the revision of your EVM?

The reported error (FIRMWARE: Ipc_CONTROLCMD_STARTCALLBACK Error: ProcMgr status 0xffffffff) is indication for IPC failure, and the reason for this failure may be not the watchdog reset. I will further investigate this.

Regards,
Pavel

Hi Pavel,

Our DM814x EVM board is Rev. D.

During my testing today, using the DM814x EVM board, I went back to almost the original 5.04.00.11 EZSDK that I used for NFS booting.  I run the saWatchdog to let the watchdog reset.  After the watchdog reset, it tried to boot and then hang in the same place where my issue is.  I actually did 2 tests.

Test 1:
1. Booting from NFS using version of 5.04.00.11 EZSDK.
2. Run saWatchdog 12.
3. Watchdog reset occurs.
4. On boot-up to NFS again, encountered the same issue (Ipc_CONTROLCMD_STARTCALLBACK Error: ProcMgr status 0xffffffff)

Test 2: 
1. Booting from NFS using version 5.04.00.11 EZSDK
2. Run saWatchdog 12.
3. Watchdog reset occurs.
4. On reset, board boots directly from the SD card with the original EZSDK version that is shipped with the board.  I believe it was 5.03.xx.xx.
5. During the HDVPSS firmware loading, it got stuck and never got out.  I needed a power cycle to get out of this hang.  Here's the last message I received from the board when running from SD card after a watchdog reset.

 Loading HDVPSS Firmware

FIRMWARE: Memory map bin file not passed
Usage : firmware_loader <Processor Id> <Location of Firmware> <start|stop> [Location of Mem map bin file]
FIRMWARE: Default memory configuration is used
MemCfg: DCMM (Dynamically Configurable Memory Map) Version : 2.1.1.1

So, I don't know if our DM814x EVM board is broken and somehow all our custom AM3874 board is mimicking the breakage as you and another TI person is saying that watchdog always work in your system.

Regards,
Maynard
 

Maynard,

What is the silicon revision of your device? It is reported in u-boot. My DM814x device is  TI8148-GP rev 2.1

I made a test with pure EZSDK 5.04.00.11 and it works fine, the HDVPSS firmware was reloaded successfully after the saWatchdog reset. But I am using SD card. I have created the SD card with the following approach:

host: $ sudo ti-ezsdk_dm814x-evm_5_04_00_11/bin/mksdboot.sh --device /dev/sdb --sdk ti-ezsdk_dm814x-evm_5_04_00_11/

Can you try with booting (and rebooting) from SD card with pure EZSDK 5.04?

Best regards,
Pavel

Dear All,

I have some similar problem with our DM8148 based platform.

I would like to know if someone else can reproduce this problem and, hopefully, find a solution together.

I made some changes to u-boot to do a software cold reset if the last boot was caused by a warm reset (wdt or software warm reset):

diff --git a/arch/arm/cpu/arm_cortexa8/start.S b/arch/arm/cpu/arm_cortexa8/start.S

index 423b01b..4935650 100644
--- a/arch/arm/cpu/arm_cortexa8/start.S
+++ b/arch/arm/cpu/arm_cortexa8/start.S
@@ -31,6 +31,7 @@

#include <config.h>
#include <version.h>
+#include <asm/arch/cpu.h>

.globl _start
_start: b reset
@@ -98,6 +99,10 @@ FIQ_STACK_START:
/*
* the actual reset code
*/
+
+prm_device_rsttst:
+ .word PRM_DEVICE_RSTTST
+

reset:
/*
@@ -107,6 +112,15 @@ reset:
bic r0, r0, #0x1f
orr r0, r0, #0xd3
msr cpsr,r0
+
+ ldr r0, prm_device_rsttst
+ ldr r1, [r0]
+ str r1, [r0]
+ cmp r1, #2
+ beq reset_cpu
+ cmp r1, #8
+ beq reset_cpu
+

#if (CONFIG_OMAP34XX)
/* Copy vectors to mask ROM indirect addr */

This solve the HDVPSS boot issue but I can still hang the CPU if I do the following:

* boot from NAND with kernel via TFTP and root file system over NFS (however the same apply, for example with NOR SPI boot)

* to save power, I call (from Linux driver)

clk_disable(clk->clk);

where clk is "iss_dpll_ck"

* trigger a WDT reset

/usr/share/ti/ti-psp-examples/saWatchdog 12

By looking inside the CPU with a JTAG debugger I can see that u-boot has never been called (I'm sure of this because I add an infinite loop at reset label) and bootrom hang in 0x2008C infinite loop (undefined instruction handler)

Can anybody reproduce this?

Is this a know "bug" of bootrom or maybe I'm doing something wrong by disabling iss pll? (the latter is not a real issue, because WDT should help in any situation, wrong code may lead to erroneous iss_dpll_ck disabling even if it's not directly called..)

Best Regards,

Andrea

Hi, we get the same problems. By the way, your code won't compile as you haven't defined PRM_DEVICE_RSTTST.

Did you mean to use 0x4818 00A8?

Ralph

Hi Ralph,

I observed that saWatchdog is not working fine when u-boot and linux is used from the arago-project.org. But when using u-boot and linux from the EZSDK 5.05.02.00, saWatchdog is working correct. You can then patch the EZSDK5.05.02.00/PSP04.04.00.01 with the patches from the arago-project.org

Regards,
Pavel

_Ralph_ said:

Did you mean to use 0x4818 00A8?

Ralph

Yes

#define PRCM_BASE        0x48180000

#define PRM_DEVICE_RSTTST                        (PRCM_BASE + 0x00A8)

BTW I'm currently using the arago-project.org u-boot/kernel. The latest commit from where I start my branch was v2.6.37_TI81XXPSP_04.04.00.02.patch1

Regards,

Andrea