• State Resolved
  • Locked Locked
  • Replies 56 replies
  • Answers 1 answer
  • Subscribers 99 subscribers
  • Views 4325 views
  • Users 0 members are here
Support feedback
Options
Options
Related

This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

AM4378: Debugging SPL on custom board

Tanner Hunsucker73
Expert 2190 points
Part Number: AM4378
Other Parts Discussed in Thread: TPS65218D0, , SN74LVC2G241, TPS65218

Hello,

I am trying to get a custom board up and running, but when I try to boot it up nothing is printed to the console. So I hooked up a JTAG emulator and I am now trying to debug the SPL code in CCS. I'm not able to set breakpoints in the C code, but I can step through the assembly code, and from what I can tell it looks like it dies when it tries to execute a "memset" call. The specific assembly command that fails is:

FB00129E   blx   memset

Could this be caused by the EMIF registers not being configured correctly or is more likely caused by something else? Please let me know your thoughts.

  • 0
    TI__Guru 55345 points

    Hello Tanner,

    Your board doesn't have an EEPROM right? Depending on that is it AM437x EMV or AM437x SK based board, please apply the patch from this or this post and check whether the board will start to print something on the console.

    Best regards,
    Kemal

  • 0 in reply to Kemal R. Shakir
    Expert 2190 points
    Hi Kemal,

    No, it does not have an EEPROM, but I do have the board name being hard coded already, and I have the function "ti_i2c_eeprom_get" commented out in my board_detect.c file.
  • 0 in reply to Tanner Hunsucker73
    TI__Guru 55345 points
    What DDR and PMIC you have used in your design?
  • 0 in reply to Kemal R. Shakir
    Expert 2190 points
    PMIC: TI TPS65218D0
    DDR3, Winbond W632GG6NB-12
  • 0 in reply to Tanner Hunsucker73
    TI__Guru 55345 points
    Have you setup the proper DDR timing configurations with the EMIF tool?
  • 0 in reply to Kemal R. Shakir
    Expert 2190 points

    Yes I have. Here is the EMIF spreadsheet that we filled out and then the corresponding values we entered into the board.c file.

    .sdram_config = 0x61A00A32,
    .ref_ctrl = 0x00000C30,
    .sdram_tim1 = 0xEAAAD4DB,
    .sdram_tim2 = 0x266B7FDA,
    .sdram_tim3 = 0x5F7F82BF,
    .read_idle_ctrl = 0x00050000,
    .zq_config = 0x500742C6,
    .temp_alert_config = 0x0,
    .emif_ddr_phy_ctlr_1 = 0x00048007,
    .emif_ddr_ext_phy_ctrl_1 = 0x00040100,
    .emif_ddr_ext_phy_ctrl_2 = 0x0,
    .emif_ddr_ext_phy_ctrl_3 = 0x0,
    .emif_ddr_ext_phy_ctrl_4 = 0x0,
    .emif_ddr_ext_phy_ctrl_5 = 0x0,
    .emif_rd_wr_exec_thresh = 0x80000405,
    .emif_prio_class_serv_map = 0x80000001,
    .emif_connect_id_serv_1_map = 0x80000094,
    .emif_connect_id_serv_2_map = 0x00000000,
    .emif_cos_config = 0x000FFFFF

    5611.SPRAC70_AM437x_EMIF_Configuration_Tool_V20.xlsx

  • 0 in reply to Tanner Hunsucker73
    TI__Guru 55345 points
    Are you expecting the prints on UART0?
  • 0 in reply to Tanner Hunsucker73
    Expert 2190 points
    It seems to me like something with DDR might not be configured correctly, because the memset function fails. So I tried setting CONFIG_USE_ARCH_MEMSET to false, and the execution got past the part it usually dies at, and made it into the board_init_r function. But then it failed at the first memory access, at the line:

    gd->bd = &bdata;

    This leads me to think that something with DDR is not right, do you think that is a correct assumption? If so, how should I go about solving this? Could you take a look at the EMIF configurations that I sent you and see if you believe they are set up correctly?
  • 0 in reply to Tanner Hunsucker73
    TI__Guru 55345 points
    Hello Tanner,

    It is difficult to compare the addresses and values in mind and decide whether they are correct or not. Can you attach the changes you have made so far? Maybe I can find there something useful there.

    cd <Processor SDK>/board-support/u-boot-<version>/
    git diff > u-boot-changes.log

    Best regards,
    Kemal
  • 0 in reply to Kemal R. Shakir
    Expert 2190 points
    u-boot-changes.log
    diff --git a/board/ti/am43xx/board.c b/board/ti/am43xx/board.c
    index d8a024f..58a6d62 100644
    --- a/board/ti/am43xx/board.c
    +++ b/board/ti/am43xx/board.c
    @@ -232,20 +232,20 @@ const struct emif_regs ddr3_emif_regs_400Mhz_beta = {

    /* EMIF DDR3 Configurations are different for production AM43X GP EVMs */
    const struct emif_regs ddr3_emif_regs_400Mhz_production = {
    - .sdram_config = 0x638413B2,
    + .sdram_config = 0x61A00A32,
    .ref_ctrl = 0x00000C30,
    .sdram_tim1 = 0xEAAAD4DB,
    .sdram_tim2 = 0x266B7FDA,
    - .sdram_tim3 = 0x107F8678,
    + .sdram_tim3 = 0x5F7F82BF,
    .read_idle_ctrl = 0x00050000,
    - .zq_config = 0x50074BE4,
    + .zq_config = 0x500742C6,
    .temp_alert_config = 0x0,
    - .emif_ddr_phy_ctlr_1 = 0x0E004008,
    - .emif_ddr_ext_phy_ctrl_1 = 0x08020080,
    - .emif_ddr_ext_phy_ctrl_2 = 0x00000066,
    - .emif_ddr_ext_phy_ctrl_3 = 0x00000091,
    - .emif_ddr_ext_phy_ctrl_4 = 0x000000B9,
    - .emif_ddr_ext_phy_ctrl_5 = 0x000000E6,
    + .emif_ddr_phy_ctlr_1 = 0x00048007,
    + .emif_ddr_ext_phy_ctrl_1 = 0x00040100,
    + .emif_ddr_ext_phy_ctrl_2 = 0x0,
    + .emif_ddr_ext_phy_ctrl_3 = 0x0,
    + .emif_ddr_ext_phy_ctrl_4 = 0x0,
    + .emif_ddr_ext_phy_ctrl_5 = 0x0,
    .emif_rd_wr_exec_thresh = 0x80000405,
    .emif_prio_class_serv_map = 0x80000001,
    .emif_connect_id_serv_1_map = 0x80000094,
    diff --git a/board/ti/common/board_detect.c b/board/ti/common/board_detect.c
    index 6414958..afb54dc 100644
    --- a/board/ti/common/board_detect.c
    +++ b/board/ti/common/board_detect.c
    @@ -186,13 +186,14 @@ int __maybe_unused ti_i2c_eeprom_am_get(int bus_addr, int dev_addr)
    ep->serial[0] = 0x0;
    ep->config[0] = 0x0;

    - rc = ti_i2c_eeprom_get(bus_addr, dev_addr, TI_EEPROM_HEADER_MAGIC,
    + /*rc = ti_i2c_eeprom_get(bus_addr, dev_addr, TI_EEPROM_HEADER_MAGIC,
    sizeof(am_ep), (uint8_t *)&am_ep);
    if (rc)
    - return rc;
    + return rc;*/

    ep->header = am_ep.header;
    - strlcpy(ep->name, am_ep.name, TI_EEPROM_HDR_NAME_LEN + 1);
    + //strlcpy(ep->name, am_ep.name, TI_EEPROM_HDR_NAME_LEN + 1);
    + strlcpy(ep->name, "AM43__GP", TI_EEPROM_HDR_NAME_LEN + 1);
    ti_eeprom_string_cleanup(ep->name);

    /* BeagleBone Green '1' eeprom, board_rev: 0x1a 0x00 0x00 0x00 */
    diff --git a/configs/am43xx_evm_defconfig b/configs/am43xx_evm_defconfig
    index 88b4a9d..1193a12 100644
    --- a/configs/am43xx_evm_defconfig
    +++ b/configs/am43xx_evm_defconfig
    @@ -82,3 +82,4 @@ CONFIG_SPL_USB_GADGET_SUPPORT=y
    CONFIG_SPL_NET_VCI_STRING="AM43xx U-Boot SPL"
    CONFIG_SPL_NET_SUPPORT=y
    CONFIG_SPL_ETH_SUPPORT=y
    +CONFIG_USE_ARCH_MEMSET=n
  • 0 in reply to Tanner Hunsucker73
    TI__Guru 55345 points

    Will you able to execute these recommended steps with JTAG on that stage?

  • 0 in reply to Kemal R. Shakir
    Expert 2190 points
    I'm not really sure what step 2 is referring to (Verify MMU is Off in bottom line of CCS (otherwise that can mess with addresses)), but when I open up the memory browser at address 0x80000000, the values are red and they keep changing, which it looks like is not good?
  • 0 in reply to Tanner Hunsucker73
    Expert 2190 points
    I now see the "MMU Off" at the bottom of CCS, so that shouldn't be affecting things.
  • 0 in reply to Tanner Hunsucker73
    TI__Guru*** 125430 points

    Tanner Hunsucker73 said:
    I'm not really sure what step 2 is referring to (Verify MMU is Off in bottom line of CCS (otherwise that can mess with addresses)), but when I open up the memory browser at address 0x80000000, the values are red and they keep changing, which it looks like is not good?

    Yes, this is a DDR issue.  Thanks for noting the MMU is off.  Please download the following diagnostic script:
    Directions on how to run the script can be found here:
    You will get a *.txt and *.csv output from that script.  Please zip and post them.
  • 0 in reply to Brad Griffis
    TI__Guru*** 125430 points
    On a related note, there are some patches needed in u-boot to get things working properly when trying to use the output of the spreadsheet. Take a look at the 6 patches I added in this private tree:

    git.ti.com/.../am437x_hw_leveling

    At a bare minimum, be sure to apply this one specific patch:

    git.ti.com/.../92538daea23db37377841ae9176a59c4f9177098

    The other patches primarily relate to suspend/resume.
  • 0 in reply to Tanner Hunsucker73
    TI__Guru*** 125430 points
    Take a look in the *.txt file. Here's an excerpt:

    DDR PHY: EXT_PHY_CTRL_1 = 0x08020080
    * ERROR! Mismatch between PHY_INVERT_CLKOUT and EXT_PHY_CTRL_1 values!
    More info: e2e.ti.com/.../2466872
    Expected value is 0x00040100 when PHY_INVERT_CLKOUT=1.

    In short, you're missing the specific patch I mentioned previously (though in fairness I mentioned it after I had told you to collect these outputs). So please apply that patch and see if that fixes things.
  • 0 in reply to Brad Griffis
    Expert 2190 points

    Okay so I applied the 6 patches and reran the test, and the error was cleared. Here are the new output files:2086.output_files.tar.gz

  • 0 in reply to Tanner Hunsucker73
    TI__Guru*** 125430 points
    What do you see when you refresh 0x80000000? Is it working properly (i.e. not changing)? Trying poking some values into there: 0xDEADBEEF, 0xFFFFFFFF, 0x00000000, 0xAAAAAAAA, 0x55555555. How do things change? Are there certain byte lanes that seem to be having issues? Certain bits? Totally random data?
  • 0 in reply to Brad Griffis
    TI__Guru*** 125430 points
    PS. Your first dump showed a DQS gate leveling timeout. That looks to have been fixed in the second dump. So you're moving in the right direction.
  • 0 in reply to Brad Griffis
    Expert 2190 points
    When I look at 0x80000000 now, none of the values are changing. I can also poke different values and things still don't change, so it seems like things are working now?
  • 0 in reply to Tanner Hunsucker73
    TI__Guru*** 125430 points
    Sounds like the DDR is fixed. There might be other issues still, but the DDR is always a big one. So do you get to a u-boot prompt now?
  • 0 in reply to Brad Griffis
    Expert 2190 points
    Okay cool - yes, it does look like DDR is fixed, memset commands are now working. I still don't get a u-boot prompt though... Looks like I have some more debugging to do.

    Just out of curiosity, why were those DDR patches needed? Is that something that is fixed in a newer SDK?
  • 0 in reply to Tanner Hunsucker73
    TI__Guru*** 125430 points
    Are you trying to just make some modifications to TI's AM437x board file? If so, keep in mind that we are leveraging data stored inside the EEPROM to determine whether the board is an EVM, IDK, Starter Kit, etc. If you don't have an EEPROM that is programmed to emulate the TI board you copied, then it will fail silently. I find a quick easy fix for board bring-up is to go to board/ti/am43xx/board.h where various board_is_xyz() functions are defined and to just hard-code the return value for the one that your board matches to 1, and the rest to 0.
  • 0 in reply to Brad Griffis
    TI__Guru*** 125430 points
    Those patches are in the process of being upstreamed and integrated into an SDK. I think SDK 6.00 will be the first SDK to have it all properly configured. The issue was that the EMIF spreadsheet didn't exist back when the code was originally written. As a result it didn't fully and properly handle the case of INVERT_CLKOUT=1. Also, in the process of fixing some of those issues, I realized the code wasn't even taking advantage of hardware leveling. So that was a pretty major effort to enable in u-boot as it had repercussions in the kernel too, especially relating to suspend/resume robustness. So it has taken a long time to get everything fully validated across all the different boards and making sure we didn't break anything related to power management.
  • 0 in reply to Brad Griffis
    Expert 2190 points
    Yes, at first that is what I'm trying to do. I'm just trying to get to the u-boot prompt using the AM437x board.c file with some slight modifications. I had been using Kemal's suggestion to bypassing the EEPROM check, and I just tried it with your suggestion as well, but still nothing is being printed.

    Is there anything else you can think of that I could try?
  • 0 in reply to Tanner Hunsucker73
    TI__Guru*** 125430 points
    How are you loading SPL? Are you booting from SD card (i.e. MLO file)?
  • 0 in reply to Brad Griffis
    Expert 2190 points
    Okay so I just got a u-boot prompt when running the SPL through JTAG. But when I try to boot without being connected to JTAG nothing is printed.

    I am trying to boot from an MLO file on the SD card.

    Does the SPL need to be loaded separately?
  • 0 in reply to Tanner Hunsucker73
    TI__Guru*** 125430 points
    If you connect to the Cortex A9 after a failed boot, where is the program counter? Is DDR configured?
  • 0 in reply to Brad Griffis
    Expert 2190 points
    I'm trying to connect after a failed boot and I am getting an error from the emulator and it is not allowing me to connect. The error says:

    IcePick_D_0: Error connecting to the target: (Error -233 @ 0x0) The JTAG IR and DR scan-paths cannot circulate bits, they may be broken. An attempt to scan the JTAG scan-path has failed. The target's JTAG scan-path appears to be broken with a stuck-at-ones or stuck-at-zero fault. (Emulation package 7.0.100.0)
  • 0 in reply to Tanner Hunsucker73
    TI__Guru*** 125430 points
    Make this change in u-boot and try again:

    diff --git a/arch/arm/cpu/armv7/start.S b/arch/arm/cpu/armv7/start.S
    index 7eee54ba70..d3c0382885 100644
    --- a/arch/arm/cpu/armv7/start.S
    +++ b/arch/arm/cpu/armv7/start.S
    @@ -36,6 +36,8 @@
    #endif

    reset:
    + /* Wait for debugger to attach */
    + b reset
    /* Allow the board to save important registers */
    b save_boot_params
    save_boot_params_ret:

    Assuming that the boot ROM has succeeded in copying MLO into internal RAM, the CPU should be in a good state and should be spinning in this loop. At this point you can:

    1. Make sure you don't have any gel files as part of your CCS configuration. In general if you select AM4378 (the chip) rather than the board, there won't be any associated gel files.
    2. Go to Run -> Load -> Load Symbols (make sure you load symbols, not program or else you will be overwriting the code) and select your u-boot-spl executable. At this point you should have symbolic debug. This is easier if you're debugging from the same Linux PC that you used to build it.
    3. Force the PC to the instruction following your spin loop.
    4. Now you can step through the code to see where things are going haywire.
  • 0 in reply to Brad Griffis
    Expert 2190 points
    I'm still getting the same error when trying to connect...
  • 0 in reply to Tanner Hunsucker73
    Expert 2190 points
    If I close out minicom though and then connect, I can see the execution is in the loop I just created.
  • 0 in reply to Tanner Hunsucker73
    TI__Guru*** 125430 points

    Can you use a different PC for the console? We need to figure out why your UART conflicts with your JTAG. Is it a software issue, eg are you connecting to a UART port that is required by CCS? Or do you have some kind of hardware conflict where the UART is trampling on ur JTAG chain?

  • 0 in reply to Brad Griffis
    Expert 2190 points
    Is it possible to use a Windows PC for the console?
  • 0 in reply to Tanner Hunsucker73
    TI__Guru*** 125430 points

    Tanner Hunsucker73 said:
    Is it possible to use a Windows PC for the console?

    Absolutely.  In fact, that's usually how I do my work.  I use Tera Term in Windows (though I've seen other people use putty, etc.).  Most of the time I just SSH into my Linux box for doing builds, etc.
  • 0 in reply to Brad Griffis
    Expert 2190 points

    Okay cool. So I just tried using Tera Term on a separate PC and still go the same error. And then when I closed Tera Term it was able to connect. Does that mean there is some sort of conflict between UART and JTAG?
  • 0 in reply to Tanner Hunsucker73
    TI__Guru*** 125430 points
    Yes, sounds like some kind of hardware issue. In general I would expect UART and JTAG to be fully independent. If you want to post some schematic snippets of your UART and JTAG I will take a look. Otherwise you will need to dig into it on your side to figure out what's happening.
  • 0 in reply to Brad Griffis
    Expert 2190 points
    I looked at the schematic but I didn't see anything that looks like the two are conflicting. The guy who did the PCB design is out of the office right now so I'm waiting for him to get back to ask him about this. Is this issue something that would prevent the board from booting or just prevent me from debugging?

    On a side note, when I load U-boot through JTAG and run it, it is printing:

    MMC: no card present
    ** Bad device mmc 0 **

    Could that be why booting is not working, because it can't find the SD card?
  • 0 in reply to Tanner Hunsucker73
    TI__Guru*** 125430 points
    You said that you could boot the image from SD card that had the spin loop right? Can you do that again please?

    Once you power up the board (i.e. such that it is spinning at the entry point), I have another script that I'd like you to run:

    git.ti.com/.../am43xx-boot.dss

    Perhaps something in the output from this script will give us a clue about what's happening.
  • 0 in reply to Brad Griffis
    Expert 2190 points

    I think the script ran successfully, I got this output file:

    Fullscreen am43xx-boot-analysis_2019-03-20_124244.txt Download 
    CONTROL: device_id = 0x2b98c02f
  * AM43xx family
  * Silicon Revision 1.2

PRM_DEVICE: PRCM_PRM_RSTST = 0x00000001
  * Bit 0 : GLOBAL_COLD_RST

CONTROL: control_status = 0x06000302
  * Bits 26 (SYSBOOT18=1) and 5 (SYSBOOT5=0): Route 25MHz EXTCLK to CLKOUT2
  * Bits 23:22 (SYSBOOT15:14=0): 19.2 MHz
  * Boot Sequence : SPI -> USB_MS (USB1) -> MMC0 -> USB_CL (USB0)

Boot Error Counter 1 = 0x00000001
Boot Error Counter 2 = 0x00000001
Boot Error Counter 3 = 0x00000000
Boot Error Counter 4 = 0x00000000

ROM: tracing vector, word 1 = 0x0010009e
  * Bit 1  : [General] Entered main function
  * Bit 2  : [General] Running after the cold reset
  * Bit 3  : [Boot] Main booting routine entered
  * Bit 4  : [Memory Boot] Memory booting started
  * Bit 7  : [Boot] Header found
  * Bit 20 : [MMC Configuration Header] CHSETTINGS found

ROM: tracing vector, word 2 = 0x00013000
  * Bit 12 : [Memory Boot] Memory booting trial 0
  * Bit 13 : [Memory Boot] Memory booting trial 1
  * Bit 16 : [Memory Boot] Execute GP image

ROM: tracing vector, word 3 = 0x02000050
  * Bit 4  : [Memory Boot] Memory booting device MMCSD0
  * Bit 6  : [Memory Boot] Memory booting device SPI
  * Bit 25 : Reserved

ROM: tracing vector, word 4 = 0x020d1e00
  * Bit 9  : [Memory Boot] MMC card in Ready state (CMD1 complete)
  * Bit 10 : [Memory Boot] Data read from the MMC card
  * Bit 11 : [Memory Boot USB_MS/MMC] Master Boot Record (MBR) found
  * Bit 12 : [Memory Boot USB_MS/MMC] Active Partition found
  * Bit 16 : [Memory Boot USB_MS] Device protocol supported
  * Bit 18 : [Memory Boot SPI] SPI configuration completed
  * Bit 19 : [Memory Boot SPI] SPI Read initialized
  * Bit 25 : [Memory Boot] Reserved

ROM: tracing vector, word 5 = 0x00000000

Cortex A9 Program Counter = 0x808002e8

  • 0 in reply to Brad Griffis
    TI__Guru*** 125430 points
    Are you using a 19.2 MHz clock as the main input to your AM437x? The EVM uses 24 MHz and most people usually copy the TI design so I wanted to be sure. Also, I was curious about the boot mode you selected: SPI -> USB_MS (USB1) -> MMC0 -> USB_CL (USB0). Since you're booting from MMC0, was there a specific reason you chose that mode?
  • 0 in reply to Brad Griffis
    Expert 2190 points

    Okay, to answer all of your questions...

    Yes, we are using the 19.2 MHz clock - it is what we have used in the past so we just stuck with it.

    We selected that boot mode because we were unsure if we would be able to boot from the SD card, or if we might have to boot from SPI.

    We can change either of those settings if you think they could be causing issues.

    I also am attaching a couple snippets of the schematic so you can look at the UART and JTAG to see if anything looks out of the ordinary. (For the Debug port, we clipped pin 3 so that it is no longer connected to 3.3 V).

    schematic_snippets.zip

  • 0 in reply to Tanner Hunsucker73
    TI__Guru*** 125430 points

    Tanner Hunsucker73 said:
    we clipped pin 3 so that it is no longer connected to 3.3 V).

    This comment has me really confused:

    • Pin 3 is TDI, not the 3.3V pin.
    • Pin 6  (3rd pin on the "even" side) is normally the one that's clipped.  That's to ensure that you don't plug the JTAG header on backward.  Please double check your JTAG probe to make sure there's a corresponding "plug" in the hole for Pin 6 to enforce that you plug the header onto the board correctly.
    • Pin 5 is 3.3V.  That's the 3rd pin on the "odd" side.

    Can you clarify on your comment?  The 3.3V signal on Pin 5 (VTREF) is used by the JTAG probe to know if your board is powered up.  Otherwise it might damage the IC by driving signals into an unpowered device.  So you really need to have this signal...  Which JTAG probe are you using?  XDS100v2?  XDS110, etc.?

  • 0 in reply to Brad Griffis
    TI__Guru*** 125430 points
    Looking at your UART connection, do you have a buffer anywhere? It looks like UART0 connects directly to some kind of header. Is that correct? If that is the case, do you have some kind of external logic that prevents you from driving signals into the UART when the board is unpowered? Normally UART signals are high, so if you have 3.3V signals sitting on the UART pins while the AM437x is turned off, you're going to destroy those pins...
  • 0 in reply to Brad Griffis
    Expert 2190 points
    Yes, it connects directly to a header. And that's the header I was referring to when I said that we clipped pin 3.
  • 0 in reply to Tanner Hunsucker73
    Expert 2190 points
    Pin 6 is clipped on the JTAG header like normal, but we clipped pin 3 on the UART header as well because it was not clipped at first, and when we connected to the serial cable it caused the processor to get hot.
  • 0 in reply to Tanner Hunsucker73
    TI__Guru*** 125430 points

    Oh, sorry, when you said "Debug port" I was thinking the JTAG.  I see the UART is labeled "Debug Port".  Now I'm with you...  This UART port looks problematic to me.  I think you're going to end up damaging a lot of boards this way.  In fact, you may have already damaged some of your boards.  In most designs you'll see that we use a sn74lvc2g241 to provide isolation in the case that the processor is powered down.  Here's an example from the BeagleBone Black:

  • 0 in reply to Brad Griffis
    Expert 2190 points
    Okay, I see... So how serious of a problem is this?