Linux/AM5728: Unable to debug U-boot

Tool/software: Linux

Hello, I know that I've made several of these posts lately, but I keep finding that they get shuffled to the bottom of the heap as I am spending more time on the problem, especially since I've replied several times to a suggestion and not heard a reply.  I'm sorry for that.  However, I'd like to collate all my issues into this post.

Previous Post:

Related post by my colleague:  

EDIT: added link to debug steps from the Wiki that I've been following, and to fix that "from the above link" that doesn't exist :) 

My method:

1. power on board with no SD card/image in memory, follow the JTAG configuration from the above link
2. load u-boot-spl.bin via tools->load memory in CCS7. Specifically load it to 0x40300000.
3. load the u-boot-spl ELF symbols
4. change the PC to 0x40300000 and run u-boot-spl
5. get this message 
   
6. Pause debugger, load u-boot via run->load program

• From here I cannot single step, every time I try the PC goes directly to 0x0000000C.  If it matters, this is the error that CCS gives, as well as the core registers
  
• enabling the DEBUG macro in am57xx-evm.h does not present any debugging information
• printf does not print anything
• Doing the above with an AM572x EVM with a plain copy of PSDK 4 results in the same behaviour, which may rule out my previous hypothesis that the EMIF/LISA configs for our custom board are incorrect.

When I place u-boot.img and MLO in the boot partition on an SD card with nothing in the rootfs partition, the EVM will reach the U-Boot console where I can get bdinfo

I would use these offsets for our custom board, but it only has 256MB of RAM, which these offsets are clearly beyond, not ot mention that U-Boot relocates itself to the top of the SDRAM stack.  Likewise, I have also actually tried using that relocaddr value, with no luck.  Another option has been to try and find gd->relocaddr by debugging SPL, but so far that has been unsuccessful.  Attempting the same method of putting the custom board's MLO and u-boot.img into an SD Card and booting has resulted in nothing being printed from the UART.

Our previous hypothesis was that the EMIF/LISA were misconfigured, but we're fairly certain that they're fine.  Regardless, here are our configurations (the LISA config is not what the SPRAC36 workbook gives, as we've found that to be incorrect).  Our desired RAM setup is one continuous 256MB block of DDR3 mapped only to EMIF2.  We are hardware levelling, and I can provide the datasheet upon request.

static const struct dmm_lisa_map_regs beagle_x15_lisa_regs = {
.dmm_lisa_map_0 = 0x0,
.dmm_lisa_map_1 = 0x0,
.dmm_lisa_map_2 = 0x0,
.dmm_lisa_map_3 = 0x80400200,
.is_ma_present = 0x1
};

static const struct emif_regs beagle_x15_emif2_ddr3_532mhz_emif_regs = {
.sdram_config_init = 0x61855A32,
.sdram_config = 0x61855A32,
.sdram_config2 = 0x00000000,
.ref_ctrl = 0x000040F1,
.ref_ctrl_final = 0x00001035,
.sdram_tim1 = 0xcccf36ab,
.sdram_tim2 = 0x30457fda,
.sdram_tim3 = 0x407f83a8,
.read_idle_ctrl = 0x00050000,
.zq_config = 0x5007190b,
.temp_alert_config = 0x00000000,
.emif_ddr_phy_ctlr_1_init = 0x0024400B,
.emif_ddr_phy_ctlr_1 = 0x0e24400b,
.emif_ddr_ext_phy_ctrl_1 = 0x10040100,
.emif_ddr_ext_phy_ctrl_2 = 0x0,
.emif_ddr_ext_phy_ctrl_3 = 0x0,
.emif_ddr_ext_phy_ctrl_4 = 0x0,
.emif_ddr_ext_phy_ctrl_5 = 0x0,
.emif_rd_wr_lvl_rmp_win = 0x00000000,
.emif_rd_wr_lvl_rmp_ctl = 0x80000000, //enabling hardware leveling
.emif_rd_wr_lvl_ctl = 0x00000000, // hardware leveling. TRM page 3457
.emif_rd_wr_exec_thresh = 0x00000305
};

static const u32 beagle_x15_emif2_ddr3_ext_phy_ctrl_const_regs[] = {
0x10040100, //EMIF_EXT_PHY_CONTROL_1
0x0, //EMIF_EXT_PHY_CONTROL_2
0x0, //EMIF_EXT_PHY_CONTROL_3
0x0, //EMIF_EXT_PHY_CONTROL_4
0x0, //EMIF_EXT_PHY_CONTROL_5
0x0, //EMIF_EXT_PHY_CONTROL_6
0x0, //EMIF_EXT_PHY_CONTROL_7
0x0, //EMIF_EXT_PHY_CONTROL_8
0x0, //EMIF_EXT_PHY_CONTROL_ 9
0x0, //EMIF_EXT_PHY_CONTROL_10
0x0, //EMIF_EXT_PHY_CONTROL_11
0x0, //EMIF_EXT_PHY_CONTROL_12
0x0, //EMIF_EXT_PHY_CONTROL_13
0x0, //EMIF_EXT_PHY_CONTROL_14
0x0, //EMIF_EXT_PHY_CONTROL_15
0x0, //EMIF_EXT_PHY_CONTROL_16
0x0, //EMIF_EXT_PHY_CONTROL_17
0x0, //EMIF_EXT_PHY_CONTROL_18
0x0, //EMIF_EXT_PHY_CONTROL_19
0x0, //EMIF_EXT_PHY_CONTROL_20
0x0, //EMIF_EXT_PHY_CONTROL_21
0x0, //EMIF_EXT_PHY_CONTROL_22
0x0, //EMIF_EXT_PHY_CONTROL_23
0x40010080, //EMIF_EXT_PHY_CONTROL_24
0x08102040, //EMIF_EXT_PHY_CONTROL_25
0x0, //EMIF_EXT_PHY_CONTROL_26
0x0, //EMIF_EXT_PHY_CONTROL_27
0x0, //EMIF_EXT_PHY_CONTROL_28
0x0, //EMIF_EXT_PHY_CONTROL_29
0x0, //EMIF_EXT_PHY_CONTROL_30
0x0, //EMIF_EXT_PHY_CONTROL_31
0x0, //EMIF_EXT_PHY_CONTROL_32
0x0, //EMIF_EXT_PHY_CONTROL_33
0x0, //EMIF_EXT_PHY_CONTROL_34
0x0 //EMIF_EXT_PHY_CONTROL_35
};

Since we've been able to view memory in the Memory window in CCS after u-boot-spl has finished running, we believe that we are in the DDR3 RAM, also due to the fact that we can load U-Boot proper, which is a 4.4MB file, larger than the cache on the AM5728.  

Investigating whether this is related to the device tree did not result in much, considering that the EVM, which has a known working device tree, behaved in the same way as our custom board did when debugged by JTAG.  At first I figured that it could have been that U-Boot could have been looking for a dtb file in the filesystem, but the device tree binary gets compiled into the U-Boot binary anyway.  The most major difference in our custom board compared to the EVM is that the custom board uses MMC1 and MMC3 both as SD Card cages.  MMC1 is the unit that the boot SD would be loaded into. 

So my running hypothesis, the confirmation of which is made difficult by the lack of proper printf debugging capabilities, is that the device tree for the custom board does not work somehow.  The evidence for this would be that I can get to the U-Boot console from the EVM when just MLO and u-boot.img are present, but receive nothing when the same is done with the custom board.  The evidence that refutes this however is that our custom board uses the same UART as the EVM, and can print messages during the SPL.  I feel that since the SPL debugging init in preloader_console_init() is done without the device tree, that evidence is shakey at best.


EDIT Aug 21 2017: I realized that I was making a simple mistake with the PSDK create-sdcard.sh script by mistaking the prebuilt location for the u-boot development directory, and loaded my most recent u-boot.img and MLO into the boot partition on the SD card by hand.  I now am receiving debug information from the custom board via UART!  However, my board hangs at the following:

INSERT: table 4031cd14, filled 60/521 rv 81f13e0c ==> name="netboot" value="echo Booting from network ...; setenv autoload no; dhcp; run netloadimage; run netloadfdt; run netargs; bootz ${loadaddr} - ${fdtaddr}"
INSERT: free(data = 81f0ff88)
INSERT: done
81f1a8c0
81f1a900
<2, 0, 1024>
81f1ad00
Failed to mount ext2 filesystem...
spl_load_image_ext: ext4fs mount err - 0
Jumping to U-Boot
loaded - jumping to U-Boot...
image entry point: 0x80800000

It appears that U-Boot is trying to load my boot partition as EXT2, when it is in fact a FAT partition.  My rootfs partition is EXT4, but right now it is emtpy.  When I try this with the EVM, it works fine, and jumps to U-Boot normally.

EDIT Aug 21 2017 #2: another problem, when I boot my custom board, I'm seeing this message regarding the device trees.  Of note is that i have edited the am57xx-evm-reva3.dts file for my custom board's device tree, merely updating the MMC settings. 

gc - clustnum: 19931, startsect: 22167
81f0ed00
81f0ed40
Size: 920128, got: 481728
image: dst=80800000, data_offset=6f0, size=75990
No matching DT out of these options:
am57xx-beagle-x15
am57xx-beagle-x15-revb1
am57xx-evm-reva3
am572x-idk
am571x-idk
malloc_simple: size=4, ptr=ed4c, limit=100000: 81f0ed48
81f0ed80
81f0edc0
reading uboot.env
81f0ee00
81f0f000
81f0f040
VFAT Support enabled
FAT32, fat_sect: 32, fatlength: 1103
Rootdir begins at cluster: 2, sector: 2238, offset: 117c00
Data begins at: 2236
Sector size: 512, cluster size: 1
FAT read(sect=2238, cnt:1), clust_size=1, DIRENTSPERBLOCK=16
81f0fc40
RootMismatch: |mlo||
Rootvfatname: |u-boot.img|
RootMismatch: |u-boot.img|u-boot.img|
RootMismatch: |u-boot.img||
RootDentname == NULL - 6

** Unable to read "uboot.env" from mmc0:1 **
Using default environment

In the defconfig, I've changed the default device tree to CONFIG_DEFAULT_DEVICE_TREE="am57xx-evm-reva3"

EDIT Aug 22 2017: After commenting out CONFIG_FIT=y in /configs/am57xx_evm_defconfig, as per 

I've found that U-Boot tries to brute-force its way through the device tree.  I've edited or stripped the parts of the EVM device tree that we're not using, to no results.  Attached is a debug log from that: putty_g3_stripped_dt.log