This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

Video Demo Kernel Lockup with 2 GByte Memory Map



Video Demo Kernel Lockup with 2 GByte Memory Map

Distribution:

DVR RDK 03.50.00.05 (customized)

Hardware:

Spectrum Digital DM8168 EVM Rev. J

Host OS:

Ubuntu 10.04

Issue overview:

When we attempt to use the 2 GByte memory map provided with the RDK at 'dvr_rdk/mcfw/src_bios6/cfg/ti816x/config_2G.bld', the video demos shipped with the RDK cause the kernel to lockup. Only a hard power cycle is able to bring the system back into a usable state.

Steps to reproduce:

Please see the whole report for detailed steps to reproduce.

5807.index.html

Log output:

Please see the whole report with log outputs for both 1GByte and 2 GByte maps.

5807.index.html

  • Is any other demo working on 2G configuration like demo option 1 or demo option 4 ?

    Also is this the last line of the log in the failing case:

    [m3vpss ]  45484: CAPTURE: VIP0 PortA capture mode is [ 8-bit, Non-mux Embedded Sync] !!!

    I would expect some other msg like M3 s/w or h/w exception if it just stopped execution.

    Also for building the kernel make sure you use the method as documented in the PSP user guide

    http://processors.wiki.ti.com/index.php/DM816x_AM389x_PSP_User_Guide#Auto_detection_of_Kernel_Load_Address_and_Run_Time_RAM_Base_Determination

     

    Cmd to rebuild uImage@${RDK_LINUX_BASE_ADDR}:

    mkimage -A arm -O linux -T kernel -C none -a ${RDK_LINUX_LOAD_ADDR} -e ${RDK_LINUX_LOAD_ADDR} -n 'Linux-2.6.37' -d arch/arm/boot/zImage uImage

     

     

     

    • Is any other demo working on 2G configuration like demo option 1 or demo option 4 ?
      • In the 2G configuration there are errors for all 4 use cases.
        • Usecase 1: Assertion error
        • Usecase 2: Assertion error
        • Usecase 3: Kernel lockup
        • Usecase 4: Kernel lockup
      • In the 1G configuration there are errors for 2 of the 4 use cases.Full logs for all usecases on both memory maps have been included in this report: 7624.2gb_mem_map_issue.html
        • Usecase 1: Assertion error
        • Usecase 2: Assertion error
        • Usecase 3: No errors
        • Usecase 4: No errors
    • Also is this the last line of the log in the failing case:
      • This is the last line of output. These logs were copied exactly from the console output. The kernel is unresponsive after this line and no further input or output is possible.
      • Full logs have been attached: 7624.2gb_mem_map_issue.html
    • Also for building the kernel make sure you use the method as documented in the PSP user guide
      • I have read through the page you referenced above. I've attached some selected output from the build log for you to look at. I've found the kernel loads properly at the appropriate address and is consistent with the build system of the RDK. Perhaps I'm mistaken on this though but the output of the build logs indicate all is OK. Also the kernel boots properly.
      • Please see the build log section of the report: 7624.2gb_mem_map_issue.html


  • You seem to be using binaries built for DVR board on the EVM. I think the capture asserts are due to that. Regarding the kernel loclup check from uboot if you are really able to access entire 2GB by first writing different patterns at 128M offsets and then reading them back. If this fails thens uboot DMM_LISA settings may be wrong and causing the issues for 2GB memory configuration.ALso ensure when you run on EVM you do make sys_all after selecting the BOARD TYPE as EVM correctly so that the uboot and kernel are built for EVM and not DVR.

  • We have run the U-Boot driven 'mtest' command on select memory sections:

    • mtest 0x81000000 0xa1000000 0xaa55aa55 1
      • Pattern AA55AA55  Writing...  Reading...Tested 1 iteration(s) with 0 errors.
    • mtest 0xa1000000 0xc0000000 0xaa55aa55 1
      • Pattern AA55AA55  Writing...  Reading...Tested 1 iteration(s) with 0 errors.
    • mtest 0xc1000000 0xe1000000 0xaa55aa55 1
      • Pattern AA55AA55  Writing...  Reading...Tested 1 iteration(s) with 0 errors.
    • mtest 0xe1000000 0xffff0000 0xaa55aa55 1
      • Pattern AA55AA55  Writing...  Reading...Tested 1 iteration(s) with 0 errors.
    • mtest 0xffff0000 0xffffffff 0xaa55aa55 1
      • Causes U-Boot to hang and become unrecoverable.
    • mtest 0x80000000 0x81000000 0xaa55aa55 1
      • Causes U-Boot to hang and become unrecoverable.
    • mtest 0xc0000000 0xc1000000 0xaa55aa55 1
      • Causes U-Boot to hang and become unrecoverable.

    Aside from the memory portions which cause U-Boot to become unrecoverable, the memory is functional. Do you have any ideas on why the 'mtest' command hangs when it is run on certain sections of memory? We have another standalone application which validates the full SDRAM address space and it is programmed into the Cortex A8 through an XDS200 JTAG programmer and run through CCS. This has successfully validated the full 2GB address space without any hangs or lockups.

    We have validated the DMM_LISA register settings according to the wiki page (http://processors.wiki.ti.com/index.php/EZSDK_Memory_Map#Changing_Memory_Map_For_512MB_DM816x_Board). Our current U-Boot build is setup with values matching those given on this page for DM8168 2 GB total SDRAM.

    With regards to the board configuration, we used the DVR configuration over the EVM configuration due to issues with NAND and UBIFS on the EVM configuration.

    We have our own video chain application which makes use of the video capture interface and encode/decode functionality. It is able to run without errors or hangs on the 1 GB memory map. We see the same failure point in our application as the RDK demo when we switch to the 2 GB memory map. Another project at our company has used the DM8168 using the DVR configuration and they have not reported issues with the video system using the 1 GB memory map.

    Are the DVR and the EVM incompatible in some way that would affect the video system? I can attempt to re-build and try the demos on the EVM configuration. I will post the reports once this is complete.

  • mtest will not catch the issue if you have issue with LISA mapping. I want to know if the Chip select  or DMM LISA settings are wrong causing different EMIF address to get reflected on same physical memory. Confirmation that this is not the case requires first writing distinct values to 2GB and then reading 2GB back and confirming values are unchanged.

    The fact that kernel is hanging on first buffer allocation indicates SharedRegion buffers are physically overlaid over memory allocated to Linux kernel .

    The mtest hang for certain address range is probably because uboot itself occupies some DDR address range and would get corrupted if mtest is done on those addresses..

    EVM and DVR are different build options and based on actual board you are running the binaries choose the correct option. There are differences in uboot/kernel/i2c addresses etc between EVM and DVR configuration and you should never use binaries built for one configuration on a different board.

     

  • This information is helpful and we will continue to debug according to your recommendations. To summarize the validation required for DMM LISA settings:

    1. Write (0x55555555) to each address in the full 2 GB address space (0x80000000 -> 0xFFFFFFFF)
    2. Once all values have been written, move back to the start of the 2 GB address space and for each address:
      1. Validate the address holds 0x55555555
      2. Perform a bitwise NOT on the address value
      3. Validate the address value is 0xAAAAAAAA
      4. Advance to the next address.

    The test above will detect address aliasing. We will be moving forward with this today. If you see issues with this testing process with regards to validating the DMM LISA settings please let us know.

  • We have found the source of the issue, which was related to the DVR build configuration vs. the EVM build configuration. U-Boot and Linux were built using the DVR configuration, which assumes use of the DVR hardware. The DVR is built with 2 GBytes of DDR, whereas the EVM is built with only 1 GByte of DDR. By default, the Linux memory map in the DVR configuration only uses the first GByte of DDR, thus no issues arose when we ran the DVR Linux build on the EVM.

    Attempting to use the 2 GByte memory map on the EVM obviously causes a failure as half of this memory space is not physically present. When we ported our build onto our custom hardware (which has 2 GBytes of DDR) everything worked as expected and no memory issues were present.