• State TI Thinks Resolved
  • Locked Locked
  • Replies 20 replies
  • Answers 2 answers
  • Subscribers 98 subscribers
  • Views 513 views
  • Users 0 members are here
Support feedback
Options
Options
Related

This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

AM5728: bad reads from eMMC hardware boot partitions

Dave Owens
Prodigy 30 points
Part Number: AM5728


Tool/software:

I recently started using the hardware boot partitions on eMMC. Reads from /dev/mmcblk1boot[0|1] typically return the expected data.  Occasionally however, a read will return data not from /dev/mmcblk1boot0 but from /dev/mmcblk1 instead.  Using hexdump, I can see that the offset specified is correct, just the data comes from the wrong device node.  An example of this occurring is below.  I tried running hexdump with strace to compare good vs bad reads and the only meaningful difference between the two is the data returned.  I can see the device node /dev/mmcblk1boot0 is being opened in both.  This same eMMC part is used on another, non-AM57 target and it does not exhibit the same behavior. 

My Linux kernel version is 6.1.38 and I have CONFIG_MMC_OMAP and CONFIG_MMC_OMAP_HS compiled in.  Any thoughts on what might be cause the occasional bad reads?

Good read:

hexdump -C /dev/mmcblk1boot0 -s 0x3FC000 -n 20

003fc000 4b d8 fa 7e 42 4f 4f 54 5f 41 5f 4c 45 46 54 3d |K..~BOOT_A_LEFT=|
003fc010 33 00 42 4f |3.BO|
003fc014

Bad read:

hexdump -C /dev/mmcblk1boot0 -s 0x3FC000 -n 20

003fc000 51 b9 9a 36 f0 56 e7 c4 d3 36 2b 2d 2c d2 e5 bd |Q..6.V...6+-,...|
003fc010 66 8a 00 98 |f...|
003fc014

Read from same offset of /dev/mmcblk1

hexdump -C /dev/mmcblk1 -s 0x3FC000 -n 20

003fc000 51 b9 9a 36 f0 56 e7 c4 d3 36 2b 2d 2c d2 e5 bd |Q..6.V...6+-,...|
003fc010 66 8a 00 98 |f...|
003fc014

Thanks,

Dave

  • 0
    TI__Mastermind 33235 points

    Hi Dave,

    Will need to confer with our MMC engineer. Please expect some delay in response. I do not have anything off the top of my head.

    A couple of questions..

    • Does this happen in more than one device?

    • Is the device custom or a TI evm?
    • I am assuming you are using the latest SDK 9.2/3, correct?

    -Josue

  • 0 in reply to Josue Zamitiz-Ayala
    Prodigy 30 points

    Hi Josue,

    This happens on every device I've tested.  The device is custom and we use yocto to build the image with Linux kernel 6.1.38 from the mainline repository.  Please let me know if I can provide any additional information.

    Thanks for your response.

    -Dave

  • 0 in reply to Dave Owens
    TI__Mastermind 33235 points

    Do you have a TI EVM? Is this repeatable on a TI EVM?

    -Josue

  • 0 in reply to Josue Zamitiz-Ayala
    Prodigy 30 points

    Unfortunately, I do not.

    -Dave

  • 0 in reply to Dave Owens
    TI__Mastermind 33235 points

    Dave,

    Is this for eMMC boot or for other purposes?

    Do you have a testcase/ procedure readily available?

    -Josue

  • 0 in reply to Josue Zamitiz-Ayala
    Prodigy 30 points

    Yes, I am using both hardware boot partitions for eMMC boot and booting itself has never been a problem.  This issue was discovered when using fw_printenv from libubootenv in userspace to print the contents of u-boot env.  As for a testcase, the read failure can be observed using any application to read from /dev/mmcblk1boot0.  On my system, non-zero data exists in /dev/mmcblk1 at address 0.  If I use hexdump to print 1000 bytes of data from /dev/mmcblk1boot0 starting at offset 0, eventually, after several reads, the data at address 0 from /dev/mmcblk1 will be returned.  Sometimes I have to wait several seconds between calls to hexdump for the issue to appear.  Other times it happens right away.

    -Dave

  • 0 in reply to Dave Owens
    TI__Mastermind 33235 points

    Dave,

    I will have to run some tests locally. Please be advised that emmc boot is not validated by our SW releases therefore it is not supported under our standard support. There will be some delay in testing. I appreciate your patience.

    I will attempt to test this by the end of the week.

    -Josue

  • 0 in reply to Josue Zamitiz-Ayala
    Prodigy 30 points

    Thanks.  If it helps, the issue doesn't have anything to do with emmc boot.  The system is fully booted and the bad reads occur in Linux.

  • 0 in reply to Dave Owens
    TI__Mastermind 33235 points

    Understood.

    Thank you Dave.

  • 0
    Prodigy 30 points

    Update on the issue.  Linux kernel 5.15 does not have the problem.  I'll post an update after bisecting to narrow down the cause of the problem

  • 0 in reply to Dave Owens
    TI__Mastermind 33235 points

    That's interesting. That might eliminate the possibility of it being a HW problem.

  • 0 in reply to Josue Zamitiz-Ayala
    TI__Mastermind 33235 points

    Dave, 

    I believe I reproduced the issue locally but only upon reboot.  Once the correct data is read, it has not happened again. Made a script to read 100s of times and never received the wrong data once it came through. Not sure why its happening at boot time. 

    To clarify I am using SDK 9.3 - Linux 6.1

    Arago 2023.10 am57xx-evm -

am57xx-evm login: ***************************************************************
***************************************************************
NOTICE: This file system contains the following GPL-3.0 packages:
        adwaita-icon-theme-symbolic
        autoconf
        bash-dev
        bash
        bc
        binutils
        cifs-utils
        coreutils-stdbuf
        coreutils
        cpio
        cpp-symlinks
        cpp
        dosfstools
        elfutils
        g++-symlinks
        g++
        gawk
        gcc-symlinks
        gcc
        gdb
        gdbc6x
        gdbserver
        gettext
        glmark2
        gnu-config
        gzip
        hidapi
        less
        libasm1
        libatomic-dev
        libatomic1
        libbfd
        libdebuginfod1
        libdw1
        libelf1
        libgcc-s-dev
        libgcc1
        libgdbm-compat4
        libgdbm-dev
        libgdbm6
        libgettextlib
        libgettextsrc
        libgmp10
        libgomp1
        libidn2-0
        libmpc3
        libmpfr6
        libopcodes
        libqt5charts-examples
        libqt5charts-plugins
        libqt5charts-qmlplugins
        libqt5charts5
        libqt5sensors-plugins
        libqt5sensors-qmlplugins
        libqt5sensors5
        libqt5serialport-examples
        libqt5serialport-plugins
        libqt5serialport-qmlplugins
        libqt5serialport5
        libqt5svg-examples
        libqt5svg-plugins
        libqt5svg-qmlplugins
        libqt5svg5
        libqt5virtualkeyboard-plugins
        libqt5virtualkeyboard-qmlplugins
        libqt5virtualkeyboard5
        libqt5webchannel-plugins
        libqt5webchannel-qmlplugins
        libqt5webchannel5
        libreadline-dev
        libreadline8
        libstdc++-dev
        libstdc++6
        libunistring2
        m4-dev
        m4
        make
        nettle
        parted
        pdm-anomaly-detection
        piglit
        qt3d-plugins
        qt3d-qmlplugins
        qt3d
        qtbase-examples
        qtbase-plugins
        qtbase-qmlplugins
        qtbase
        qtconnectivity-plugins
        qtconnectivity-qmlplugins
        qtconnectivity
        qtdeclarative-plugins
        qtdeclarative-qmlplugins
        qtdeclarative-tools
        qtdeclarative
        qtgraphicaleffects-qmlplugins
        qtlocation-examples
        qtlocation-plugins
        qtlocation-qmlplugins
        qtlocation
        qtmultimedia-examples
        qtmultimedia-plugins
        qtmultimedia-qmlplugins
        qtmultimedia
        qtquics-qmlplugins.control
        qtquics2-plugins.control
        qtquics2-qmlplugins.control
        qtquics2.control
        qtscript-examples
        qtscript-plugins
        qtscript-qmlplugins
        qtscript
        qtwayland-examples
        qtwayland-plugins
        qtwayland-qmlplugins
        qtwayland
        tar
        which

If you do not wish to distribute GPL-3.0 components please remove
the above packages prior to distribution.  This can be done using
the opkg remove command.  i.e.:
    opkg remove <package>
Where <package> is the name printed in the list above

NOTE: If the package is a dependency of another package you
      will be notified of the dependent packages.  You should
      use the --force-removal-of-dependent-packages option to
      also remove the dependent packages as well
***************************************************************
***************************************************************

am57xx-evm login: root
root@am57xx-evm:~# 
root@am57xx-evm:~# hexdump -C /dev/mmcblk1boot0 -n 40
00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
root@am57xx-evm:~# ./mmc_read.sh 
Starting mmc_read script/n
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
00000000  44 45 41 44 42 45 45 46  0a 00 00 00 00 00 00 00  |DEADBEEF........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000020  00 00 00 00 00 00 00 00                           |........|
00000028
Finished script/n
root@am57xx-evm:~#

    The way I am doing it is by making one read.. waiting a couple of seconds and then I get all correct data..

    Not sure what is happening. 

    Is your system doing this randomly? Seems like it just takes the system a second to pull in the correct data.

     -Josue

  • 0 in reply to Josue Zamitiz-Ayala
    Prodigy 30 points

    Delaying 0-2 seconds between reads seems to work for me.  If I wait 20 seconds between reads I start to see problems.  Would you be able to run your loop test as you did before and then wait 20 seconds before trying another couple reads.  This reproduces the problem for me pretty reliably.

    -Dave

  • 0 in reply to Dave Owens
    TI__Mastermind 33235 points

    Dave,

    I am out of office today. Will be back tomorrow.

    -Josue

  • 0 in reply to Josue Zamitiz-Ayala
    TI__Mastermind 33235 points

    Dave,

    I did not have time today but I will hopefully get to this by Friday.

    -Josue

  • 0
    Prodigy 30 points

    The issue appears to be from this commit in v5.16.  I'm going through each commit in the merge now to see which one causes the regression on my system.  Perhaps I'm missing a Kconfig or device tree setting required by the updates?

    commit 8a73c77c809a7342497b78a2b4555aa40506af94
    Merge: 316b7eaa932d 348ecd61770f
    Author: Linus Torvalds <torvalds@linux-foundation.org>
    Date: Mon Nov 1 18:55:12 2021 -0700

    Merge tag 'mmc-v5.16' of git://git.kernel.org/pub/scm/linux/kernel/git/ulfh/mmc

  • 0
    Prodigy 30 points

    This commit caused the regression.  3edf588e7fe00e90d1dc7fb9e599861b2c2cf442

  • 0 in reply to Dave Owens
    TI__Mastermind 33235 points

    Dave,

    Awesome find! Internally we were suspecting it was a PM issue.

    Does reverting this "Aggressive PM" patch give you the expected behaviour?

    -Josue

  • 0 in reply to Josue Zamitiz-Ayala
    Prodigy 30 points

    Yes, reverting the aggressive PM patch does resolve things.  I stumbled onto a kernel mailing list post about the same patch set but a different problem and unfortunately, the patch to fix things mentioned in the original post does not fix this particular issue.  Only reverting the PM commit does so far.

    Here's a link to the original PM fix, which contains a link back to the original discussion:  lore.kernel.org/.../

  • 0 in reply to Dave Owens
    TI__Mastermind 33235 points

    Thanks Dave! Will keep this in mind. I am glad you were able to resolve your particular issue.

    -Josue