This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

AM5728: eMMC boot fails

Part Number: AM5728
Other Parts Discussed in Thread: DRA752, CSD

Hello,

For our products  we use AM5728 and Silicon Motion eMMC chips. Recently we run into an issue where some of the devices fails to boot from the eMMC chips.

This devices, used to boot fine before and after a few power cycles, they just stop booting.

With further investigation, we found out that the SoC is not able to load the first stage bootloader and can't proceed with the booting.

I have attached an SD card and tried to boot from it, and was able to boot from the SD card. I boot from SD card and stay at U-boot, and tried different commands to see how on board eMMC chip resonds. Below are my findings.

U-Boot# mmc list
OMAP SD/MMC: 0 (SD)
OMAP SD/MMC: 1 (eMMC)

U-Boot# mmc dev 1
mmc_read_data: timedout waiting for status!
omap_hsmmc_send_cmd: timedout waiting on cmd inhibit to clear

U-Boot# ext4ls mmc 1:3
mmc_read_data: timedout waiting for status!
omap_hsmmc_send_cmd: timedout waiting on cmd inhibit to clear
** No partition table - mmc 1 **

Tried Re-partitioning of the eMMC chip:

Device:1

U-Boot# setenv uuid_gpt_disk df335b83-b69b-4731-aa6d-8f161c8dea02;
U-Boot# setenv uuid_gpt_spl ba00fed3-bf1e-4e58-96c2-ca5488e60861;
U-Boot# setenv uuid_gpt_uboot ba00fed3-bf1e-4e58-96c2-ca5488e60862;
U-Boot# setenv uuid_gpt_boot ba00fed3-bf1e-4e58-96c2-ca5488e60863;
U-Boot# setenv boot_size '16'
U-Boot# setenv boot_size_hex '0x10'
U-Boot# setenv rootfs_size @rootfs_size@;
U-Boot# setenv overlays_size '512'
U-Boot# setenv spl_part "name=spl,start=128KiB,uuid=${uuid_gpt_spl},size=128KiB";
U-Boot# setenv uboot_part "name=uboot,start=384KiB,uuid=${uuid_gpt_uboot},size=768KiB";
U-Boot# setenv boot_part "name=boot,start=2MiB,uuid=${uuid_gpt_boot},size=${boot_size}MiB";
U-Boot# setenv partitions "uuid_disk=${uuid_gpt_disk};${spl_part};${uboot_part};${boot_part}";
U-Boot# gpt write mmc 1 ${partitions};
mmc_read_data: timedout waiting for status!
omap_hsmmc_send_cmd: timedout waiting on cmd inhibit to clear
do_gpt: mmc dev 1 NOT available

Device:2

U-Boot# setenv uuid_gpt_disk df335b83-b69b-4731-aa6d-8f161c8dea02;
U-Boot# setenv uuid_gpt_spl ba00fed3-bf1e-4e58-96c2-ca5488e60861;
U-Boot# setenv uuid_gpt_uboot ba00fed3-bf1e-4e58-96c2-ca5488e60862;
U-Boot# setenv uuid_gpt_boot ba00fed3-bf1e-4e58-96c2-ca5488e60863;
U-Boot# setenv boot_size '16'
U-Boot# setenv boot_size_hex '0x10'
U-Boot# setenv rootfs_size @rootfs_size@;
U-Boot# setenv overlays_size '512'
U-Boot# setenv spl_part "name=spl,start=128KiB,uuid=${uuid_gpt_spl},size=128KiB";
U-Boot# setenv uboot_part "name=uboot,start=384KiB,uuid=${uuid_gpt_uboot},size=768KiB";
U-Boot# setenv boot_part "name=boot,start=2MiB,uuid=${uuid_gpt_boot},size=${boot_size}MiB";
U-Boot# setenv partitions "uuid_disk=${uuid_gpt_disk};${spl_part};${uboot_part};${boot_part}";
U-Boot# gpt write mmc 1 ${partitions};
Writing GPT: omap_hsmmc_send_cmd: timedout waiting on cmd inhibit to clear
** Can't write to device 1 **
** Can't write to device 1 **
error!

On some of the boards we also see below errors at Uboot when I stay in U-boot using SD card boot (Device 1 is the onboard eMMC chip and partition 3 and 4 are ext4 based):

U-Boot# ext4ls mmc 1:3
** Bad device mmc 1 **

U-Boot# ext4ls mmc 1:4
** Bad device mmc 1 **

________________________________________________________________________________________________________________________________

Then, I booted my system through the SD card and checked the kernel logs, if the kernel was able to initialize the eMMC chip, and I found below logs:

[    4.925068] mmc1: error -110 whilst initialising MMC card
[    4.956095] omap_hsmmc 480d1000.mmc: card claims to support voltages below defined range

We are not able to understand the root cause of this issue, since the board used to boot and work fine.

I would really appreciate a response soon.

Thanks,

Divyeshkumar M

  • When you power cycle, are you using the eMMC (read/write data)? I'm not sure that matters since it sounds like something is corrupted in the extcsd.

    Steve K.
  • Hi Steve,

    Thanks for the reply. Yes we do write some of our user data at the shutdown from RAM to a partition of the eMMC. We restore those data at system start from eMMC back to RAM.

    We also highly doubt it to be a chip corruption issue, but the thing that confuses me is, our filesystem is read-only and we only use one partition to R/W data at power cycles. How does that corrupt the first stage boot-loader or the whole chip?

    If the particular partition gets corrupted due to the I/O at power cycles, that makes a lot more sense.

    Do you see any reason why it will corrupt whole chip or may corrupt the partition table of the eMMC?

    For some reason I also doubt that this could be a bad soldering or bad chip itself. Because the failure happens too quickly after the first utilization.

    We were using a Micron eMMC previously and we never faced such failures with those chips. And the new Silicon Motion chips we're using now are eligible for drop-in replacement and that's how we have used it. We have also tested some samples with the SiMo chips for almost 1900 power cycles with file I/O on main rootfs.

    What kind of issue you think it is, S/W, H/W or production?

    Again I really appreciate your help with the issue.

    Thanks,
    Divyeshkumar
  • In u-boot, when you boot from SD and see

    mmc_read_data: timedout waiting for status!
    omap_hsmmc_send_cmd: timedout waiting on cmd inhibit to clear

    On the eMMC, can you use md.l to dump the controller registers? It sounds like possibly a timeout is not long enough. The -110 error in Linux is a timeout error.

    Steve K.

  • Hi Steve thanks for the response.

    I can't select dev 1 as the default mmc device and hence can't run the md.l command on the devices that show no boot signs at all. I keep seeing below output:

    U-Boot# mmc dev 1 

    mmc_read_data: timedout waiting for status! 
    omap_hsmmc_send_cmd: timedout waiting on cmd inhibit to clear

     

    _________________________________________________________________________________________

    I had one unit where I was able to boot till U-boot from eMMC and then everytime the hash verification for the kernel image fails. Below is the output:

    U-Boot SPL 2015.07 (May 25 2017 - 08:50:13)
    DRA752 ES2.0


    U-Boot 2015.07 (May 25 2017 - 08:50:13 -0400)

    CPU  : DRA752 ES2.0
    Board: Raymond GDM
    I2C:   ready
    DRAM:  2 GiB
    MMC:   OMAP SD/MMC: 0, OMAP SD/MMC: 1
    Card did not respond to voltage select!
    ** Bad device mmc 0 **
    Using default environment

    Net:   Net Initialization Skipped
    No ethernet found.
    Hit any key to stop autoboot:  0  
    Card did not respond to voltage select!
    Card did not respond to voltage select!
    6670 bytes read in 3 ms (2.1 MiB/s)
    Running bootscript from mmc
    ## Executing script at 82000000
    Bootscript executing...
    Using eMMC
    attempt: 1  
    1 bytes read in 3 ms (0 Bytes/s)
    rootfs_part: 1
    1 bytes read in 3 ms (0 Bytes/s)
    boot_retries: 9
    attempt: 2  
    rootfs_part: 0
    1 bytes read in 2 ms (0 Bytes/s)
    boot_retries: 2
    File System is consistent
    file found deleting
    update journal finished
    File System is consistent
    update journal finished
    1 bytes written in 123 ms (0 Bytes/s)
    3970384 bytes read in 233 ms (16.3 MiB/s)
    GUID Partition Table Entry Array CRC is wrong: 0xebb5260f != 0x5f2916ca
    get_partition_info_efi: *** ERROR: Invalid GPT ***
    get_partition_info_efi: ***        Using Backup GPT ***
    bootargs=console=ttyO2,115200n8 quiet vt.global_cursor_default=0 root=PARTUUID=d3fe00ba-1ebf-584e-96c2-ca5488e60864 ro rootfstype=ext4 rootwait
    gpio: pin 58 (gpio 58) value is 1
    ## Loading kernel from FIT Image at 82000000 ...
      Trying 'kernel@1' kernel subimage
        Description:  Linux kernel
        Type:         Kernel Image
        Compression:  uncompressed
        Data Start:   0x82000104
        Data Size:    3769168 Bytes = 3.6 MiB
        Architecture: ARM
        OS:           Linux
        Load Address: 0x80008000
        Entry Point:  0x80008000
        Hash algo:    sha1
        Hash value:   7162a13bc1d4e35a9055f2793ff69221f0e4755a
      Verifying Hash Integrity ... sha1 error!
    Bad hash value for 'hash@1' hash node in 'kernel@1' image node
    Bad Data Hash
    ERROR: can't get kernel image!
    ERROR: No bootable partition
    U-Boot#

    I have re-flashed the whole eMMC chip still it will stuck with the same issue. I don't know if this will give you any clue or this could be initial stage of eMMC boot failure.

    I have tried running md .l in this unit and the result is below:

    U-Boot# md .l
    00000000:data abort
    pc : [<fff8ff6a>]          lr : [<fff72b5b>]
    reloc pc : [<8082af6a>]    lr : [<8080db5b>]
    sp : fef54a60  ip : 00000014     fp : 00000000
    r10: 00000000  r9 : fef54ef0     r8 : 00000004
    r7 : 00000001  r6 : fffa1564     r5 : fef54ad4  r4 : 00000001
    r3 : 00000020  r2 : fef54cfc     r1 : fef54cfc  r0 : fef54ad4
    Flags: Nzcv  IRQs off  FIQs off  Mode SVC_32
    Resetting CPU ...

    resetting ...

    After running the md.l command, device is not able to find kernel image and shows below output: 

    U-Boot SPL 2015.07 (May 25 2017 - 08:50:13) 
    DRA752 ES2.0 


    U-Boot 2015.07 (May 25 2017 - 08:50:13 -0400) 

    CPU  : DRA752 ES2.0 
    Board: Raymond GDM 
    I2C:   ready 
    DRAM:  2 GiB 
    MMC:   OMAP SD/MMC: 0, OMAP SD/MMC: 1 
    Card did not respond to voltage select! 
    ** Bad device mmc 0 ** 
    Using default environment 

    Net:   Net Initialization Skipped 
    No ethernet found. 
    Hit any key to stop autoboot:  0  
    Card did not respond to voltage select! 
    Card did not respond to voltage select! 
    6670 bytes read in 3 ms (2.1 MiB/s) 
    Running bootscript from mmc 
    ## Executing script at 82000000 
    Bad data crc 
    U-Boot#  

    ______________________________________________________________________________________

    We also have some units which failed to boot properly for a few times and after re-programming the eMMC or opening the enclosure they started working fine again.

    The time-out thing could also be because the eMMC is dead and not responding, correct?

    What else could be wrong?

    Again I really appreciate your cooperation with this issue Steve.

    Thanks,

    Divyeshkumar

  • You have to put the address of the controller registers after md.l so the command would be
    md.l 0x481d8110

    Press enter and then press enter again. This will print out all the controller registers.

    Steve K.
  • The controller register for the unit that does not show any boot signs are:

    U-Boot# mmc dev 1
    mmc_read_data: timedout waiting for status!
    omap_hsmmc_send_cmd: timedout waiting on cmd inhibit to clear
    switch to partitions #0, OK
    mmc1(part 0) is current device


    U-Boot# md.l 0x481d8110
    481d8110:data abort
    pc : [<fff8ffa6>]          lr : [<fff72b5b>]
    reloc pc : [<8082afa6>]    lr : [<8080db5b>]
    sp : fef54a60  ip : 00000000     fp : 481d8110
    r10: 481d8110  r9 : fef54ef0     r8 : 00000004
    r7 : 00000001  r6 : fffa1567     r5 : fef54ad5  r4 : 00000001
    r3 : 0000002a  r2 : fef54cfc     r1 : fef54cfc  r0 : fef54ad4
    Flags: nzCv  IRQs off  FIQs off  Mode SVC_32
    Resetting CPU ...

    resetting ...


    The controller register for the device where I am getting Kernel Hash mismatch issue:

    U-Boot SPL 2015.07 (May 25 2017 - 08:50:13)
    DRA752 ES2.0


    U-Boot 2015.07 (May 25 2017 - 08:50:13 -0400)

    CPU  : DRA752 ES2.0
    Board: Raymond GDM
    I2C:   ready
    DRAM:  2 GiB
    MMC:   OMAP SD/MMC: 0, OMAP SD/MMC: 1
    Card did not respond to voltage select!
    ** Bad device mmc 0 **
    Using default environment

    Net:   Net Initialization Skipped
    No ethernet found.
    Hit any key to stop autoboot:  0  
    Card did not respond to voltage select!
    Card did not respond to voltage select!
    6670 bytes read in 2 ms (3.2 MiB/s)
    Running bootscript from mmc
    ## Executing script at 82000000
    Bootscript executing...
    Using eMMC
    attempt: 1  
    1 bytes read in 3 ms (0 Bytes/s)
    rootfs_part: 1
    1 bytes read in 3 ms (0 Bytes/s)
    boot_retries: 9
    attempt: 2  
    rootfs_part: 0
    1 bytes read in 2 ms (0 Bytes/s)
    boot_retries: 3
    File System is consistent
    file found deleting
    update journal finished
    File System is consistent
    update journal finished
    1 bytes written in 146 ms (0 Bytes/s)
    3970384 bytes read in 233 ms (16.3 MiB/s)
    bootargs=console=ttyO2,115200n8 quiet vt.global_cursor_default=0 root=PARTUUID=d3fe00ba-1ebf-584e-96c2-ca5488e60864 ro rootfstype=ext4 rootwait
    gpio: pin 58 (gpio 58) value is 1
    ## Loading kernel from FIT Image at 82000000 ...
      Trying 'kernel@1' kernel subimage
        Description:  Linux kernel
        Type:         Kernel Image
        Compression:  uncompressed
        Data Start:   0x82000104
        Data Size:    3769168 Bytes = 3.6 MiB
        Architecture: ARM
        OS:           Linux
        Load Address: 0x80008000
        Entry Point:  0x80008000
        Hash algo:    sha1
        Hash value:   7162a13bc1d4e35a9055f2793ff69221f0e4755a
      Verifying Hash Integrity ... sha1 error!
    Bad hash value for 'hash@1' hash node in 'kernel@1' image node
    Bad Data Hash
    ERROR: can't get kernel image!
    ERROR: No bootable partition


    U-Boot# md.l 0x481d8110
    481d8110:data abort
    pc : [<fff8ff5a>]          lr : [<fff72b5b>]
    reloc pc : [<8082af5a>]    lr : [<8080db5b>]
    sp : fef54a60  ip : 00000000     fp : 481d8110
    r10: 481d8110  r9 : fef54ef0     r8 : 00000004
    r7 : 00000001  r6 : fffa1564     r5 : fef54ad4  r4 : 00000001
    r3 : 481d8110  r2 : fef54cfc     r1 : fef54cfc  r0 : fef54ad4
    Flags: nzcv  IRQs off  FIQs off  Mode SVC_32
    Resetting CPU ...

    resetting ...

    __________________________________________________________________________________


    I have also checked the TRM for AM5728 and if you're talking about the MMC register mapping for MMC2 below is the output:

    For the device that shows bad hash for Kernel:

    U-Boot# md.l 0x480B4000  
    480b4000: 40200303 0000004b 00000000 00000000    .. @K...........
    480b4010: 00000028 00000000 00000000 00000000    (...............
    480b4020: 00000000 00000000 00000000 00000000    ................
    480b4030: 00000000 00000000 00000000 00000000    ................
    480b4040: 00000000 00000000 00000000 00000000    ................
    480b4050: 00000000 00000000 00000000 00000000    ................
    480b4060: 00000000 00000000 00000000 00000000    ................
    480b4070: 00000000 00000000 00000000 00000000    ................
    480b4080: 00000000 00000000 00000000 00000000    ................
    480b4090: 00000000 00000000 00000000 00000000    ................
    480b40a0: 00000000 00000000 00000000 00000000    ................
    480b40b0: 00000000 00000000 00000000 00000000    ................
    480b40c0: 00000000 00000000 00000000 00000000    ................
    480b40d0: 00000000 00000000 00000000 00000000    ................
    480b40e0: 00000000 00000000 00000000 00000000    ................
    480b40f0: 00000000 00000000 00000000 00000000    ................
    U-Boot#  
    480b4100: 00000000 00000000 00000000 00000000    ................
    480b4110: 00002015 00000001 00000000 00000000    . ..............
    480b4120: 00000000 00000000 00008000 00000600    ................
    480b4130: 00000000 80000000 00000000 00000000    ................
    480b4140: 00000000 00000000 00000000 00000000    ................
    480b4150: 00000000 00000000 00000000 00000000    ................
    480b4160: 00000000 00000000 00000000 00000000    ................
    480b4170: 00000000 00000000 00000000 00000000    ................
    480b4180: 00000000 00000000 00000000 00000000    ................
    480b4190: 00000000 00000000 00000000 00000000    ................
    480b41a0: 00000000 00000000 00000000 00000000    ................
    480b41b0: 00000000 00000000 00000000 00000000    ................
    480b41c0: 00000000 00000000 00000000 00000000    ................
    480b41d0: 00000000 00000000 00000000 00000000    ................
    480b41e0: 00000000 00000000 00000000 00000000    ................
    480b41f0: 00000000 00000000 00000000 00000000    ................
    U-Boot#  
    480b4200: 00000000 00000200 00000000 0c1b0000    ................
    480b4210: 00000b00 ffffffef 0f5903ff d0270132    ..........Y.2.'.
    480b4220: 00000000 01f00000 00000d02 000e0087    ................
    480b4230: 20008002 307f0033 00000000 00000000    ... 3..0........
    480b4240: 26e90080 00000f77 00000000 00000000    ...&w...........
    480b4250: 00000000 00000000 00000000 00000000    ................
    480b4260: 000401e0 00040002 00010002 00020000    ................
    480b4270: 00000000 00000000 00000000 00000000    ................
    480b4280: 00000000 00000000 00000000 00000000    ................
    480b4290: 00000000 00000000 00000000 00000000    ................
    480b42a0: 00000000 00000000 00000000 00000000    ................
    480b42b0: 00000000 00000000 00000000 00000000    ................
    480b42c0: 00000000 00000000 00000000 00000000    ................
    480b42d0: 00000000 00000000 00000000 00000000    ................
    480b42e0: 00000000 00000000 00000000 00000000    ................
    480b42f0: 00000000 00000000 00000000 33020000    ...............3
    U-Boot#  
    480b4300: 00000000 00000000 00000000 00000000    ................
    480b4310: 00000000 00000000 00000000 00000000    ................
    480b4320: 00000000 00000000 00000000 00000000    ................
    480b4330: 00000000 00000000 00000000 00000000    ................
    480b4340: 00000000 00000000 00000000 00000000    ................
    480b4350: 00000000 00000000 00000000 00000000    ................
    480b4360: 00000000 00000000 00000000 00000000    ................
    480b4370: 00000000 00000000 00000000 00000000    ................
    480b4380: 00000000 00000000 00000000 00000000    ................
    480b4390: 00000000 00000000 00000000 00000000    ................
    480b43a0: 00000000 00000000 00000000 00000000    ................
    480b43b0: 00000000 00000000 00000000 00000000    ................
    480b43c0: 00000000 00000000 00000000 00000000    ................
    480b43d0: 00000000 00000000 00000000 00000000    ................
    480b43e0: 00000000 00000000 00000000 00000000    ................
    480b43f0: 00000000 00000000 00000000 00000000    ................
    U-Boot#  
    480b4400: 40200303 0000004b 00000000 00000000    .. @K...........
    480b4410: 00000028 00000000 00000000 00000000    (...............
    480b4420: 00000000 00000000 00000000 00000000    ................
    480b4430: 00000000 00000000 00000000 00000000    ................

     

    For the Device that shows no sign of boot:

    U-Boot# md.l 0x480B4000
    480b4000: 40200303 0000004b 00000000 00000000 .. @K...........
    480b4010: 00000028 00000000 00000000 00000000 (...............
    480b4020: 00000000 00000000 00000000 00000000 ................
    480b4030: 00000000 00000000 00000000 00000000 ................
    480b4040: 00000000 00000000 00000000 00000000 ................
    480b4050: 00000000 00000000 00000000 00000000 ................
    480b4060: 00000000 00000000 00000000 00000000 ................
    480b4070: 00000000 00000000 00000000 00000000 ................
    480b4080: 00000000 00000000 00000000 00000000 ................
    480b4090: 00000000 00000000 00000000 00000000 ................
    480b40a0: 00000000 00000000 00000000 00000000 ................
    480b40b0: 00000000 00000000 00000000 00000000 ................
    480b40c0: 00000000 00000000 00000000 00000000 ................
    480b40d0: 00000000 00000000 00000000 00000000 ................
    480b40e0: 00000000 00000000 00000000 00000000 ................
    480b40f0: 00000000 00000000 00000000 00000000 ................
    U-Boot#
    480b4100: 00000000 00000000 00000000 00000000 ................
    480b4110: 00002015 00000001 00000000 00000000 . ..............
    480b4120: 00000000 00000000 00008000 00000600 ................
    480b4130: 00000000 80000000 00000000 00000000 ................
    480b4140: 00000000 00000000 00000000 00000000 ................
    480b4150: 00000000 00000000 00000000 00000000 ................
    480b4160: 00000000 00000000 00000000 00000000 ................
    480b4170: 00000000 00000000 00000000 00000000 ................
    480b4180: 00000000 00000000 00000000 00000000 ................
    480b4190: 00000000 00000000 00000000 00000000 ................
    480b41a0: 00000000 00000000 00000000 00000000 ................
    480b41b0: 00000000 00000000 00000000 00000000 ................
    480b41c0: 00000000 00000000 00000000 00000000 ................
    480b41d0: 00000000 00000000 00000000 00000000 ................
    480b41e0: 00000000 00000000 00000000 00000000 ................
    480b41f0: 00000000 00000000 00000000 00000000 ................
    U-Boot#
    480b4200: 00000000 00000000 00000000 00000000 ................
    480b4210: 00000000 00000000 00000000 00000000 ................
    480b4220: 00000000 01f00000 00000000 00000000 ................
    480b4230: 00000000 00000000 00000000 00000000 ................
    480b4240: 20e90080 00000f77 00000000 00000000 ... w...........
    480b4250: 00000000 00000000 00000000 00000000 ................
    480b4260: 000401e0 00040002 00010002 00020000 ................
    480b4270: 00000000 00000000 00000000 00000000 ................
    480b4280: 00000000 00000000 00000000 00000000 ................
    480b4290: 00000000 00000000 00000000 00000000 ................
    480b42a0: 00000000 00000000 00000000 00000000 ................
    480b42b0: 00000000 00000000 00000000 00000000 ................
    480b42c0: 00000000 00000000 00000000 00000000 ................
    480b42d0: 00000000 00000000 00000000 00000000 ................
    480b42e0: 00000000 00000000 00000000 00000000 ................
    480b42f0: 00000000 00000000 00000000 33020000 ...............3
    U-Boot#
    480b4300: 00000000 00000000 00000000 00000000 ................
    480b4310: 00000000 00000000 00000000 00000000 ................
    480b4320: 00000000 00000000 00000000 00000000 ................
    480b4330: 00000000 00000000 00000000 00000000 ................
    480b4340: 00000000 00000000 00000000 00000000 ................
    480b4350: 00000000 00000000 00000000 00000000 ................
    480b4360: 00000000 00000000 00000000 00000000 ................
    480b4370: 00000000 00000000 00000000 00000000 ................
    480b4380: 00000000 00000000 00000000 00000000 ................
    480b4390: 00000000 00000000 00000000 00000000 ................
    480b43a0: 00000000 00000000 00000000 00000000 ................
    480b43b0: 00000000 00000000 00000000 00000000 ................
    480b43c0: 00000000 00000000 00000000 00000000 ................
    480b43d0: 00000000 00000000 00000000 00000000 ................
    480b43e0: 00000000 00000000 00000000 00000000 ................
    480b43f0: 00000000 00000000 00000000 00000000 ................
    U-Boot#
    480b4400: 40200303 0000004b 00000000 00000000 .. @K...........
    480b4410: 00000028 00000000 00000000 00000000 (...............
    480b4420: 00000000 00000000 00000000 00000000 ................
    480b4430: 00000000 00000000 00000000 00000000 ................

     If we compare both of the units, below is something different in both:

    The unit that shows bad hash for kernel:

    480b4200: 00000000 00000200 00000000 0c1b0000 ................
    480b4210: 00000b00 ffffffef 0f5903ff d0270132 ..........Y.2.'.
    480b4220: 00000000 01f00000 00000d02 000e0087 ................
    480b4230: 20008002 307f0033 00000000 00000000 ... 3..0........
    480b4240: 26e90080 00000f77 00000000 00000000 ...&w...........

    The unit that shows no boot sign:

    480b4200: 00000000 00000000 00000000 00000000 ................
    480b4210: 00000000 00000000 00000000 00000000 ................
    480b4220: 00000000 01f00000 00000000 00000000 ................
    480b4230: 00000000 00000000 00000000 00000000 ................
    480b4240: 20e90080 00000f77 00000000 00000000 ... w...........

     


    Let me know if you need more info.

    Thanks,

    Divyeshkumar

  • Hello Steve,

    Do we have any update?

    Thanks,
    Divyeshkumar
  • Hello Divyeshkumar,

    I am still analyzing this.

    Steve K.

  • Hello Steve,

    OK, Please update me with your findings.

    Thanks,
    Divyeshkumar
  • For the bad hash, can you test without any hash? For the other board, nothing looks unusual. I'm still looking at it.

    Steve K.
  • Hi Steve,

    Thanks for the response, Can you please suggest how to skip hash check for kernel?

    Thanks,
    Divyeshkumar
  • Look in the u-boot/doc/uImage.FIT directory. Three files to look at are verified-boot.txt, signature.txt, and beaglebone_vboot.txt. They explain how to create the hash. I was thinking just don't create the hash and don't sign it. This is part of how you build your code.

    Steve K.
  • That would be a reprogramming of the eMMC right? While I work on getting the hash less image, do you think of anything else going bad in here?

    We just had some feedback from Silicom Motion, they tested two of the bad boards' eMMC and they claim the eMMC chips are healthy. They also mentioned that the host stops sending the commands to the eMMC chip. Does that give any clue to you?

    Thanks,
    Divyeshkumar
  • Can you find out at what point the host stops sending commands to the eMMC? I would guess after these messages
    mmc_read_data: timedout waiting for status!
    omap_hsmmc_send_cmd: timedout waiting on cmd inhibit to clear

    Can you see if commands are sent before those messages? Maybe add some printf to the driver? This is in u-boot.

    What happens in you increase MAX_RETRY_MS to 30 from 20?

    Steve K.
  • Silicon Motion found the "host does not send command", from the logs of the bad board's eMMC chip that does not boot. They don't use SD card and test the eMMC from u-boot.

    I also checked our u-boot source, and the MAX_RETRY_MS is set to 1 seconds currently. See below:

    /* If we fail after 1 second wait, something is really bad */
    #define MAX_RETRY_MS 1000

    I won't be able to re-program the eMMC chip as I can't communicate with the chip, so I won't be able to test the modified u-boot.

    Also why do we have all 0's below in the bad board:

    480b4200: 00000000 00000000 00000000 00000000 ................
    480b4210: 00000000 00000000 00000000 00000000 ................
    480b4220: 00000000 01f00000 00000000 00000000 ................
    480b4230: 00000000 00000000 00000000 00000000 ................
    480b4240: 20e90080 00000f77 00000000 00000000 ... w...........


    Do you think it is something related to the timings?

    Thanks,
    Divyeshkumar

  • It could be timings since we are seeing timeouts. Where did the message "host does not send command" come from? I can't find it in u-boot or the kernel.

    Steve K.
  • Oh that's the statement Silicon Motion made. Sorry I should not have quoted that.

    Steve, if you also think this could be a timing issue? Can that be due to H/W?

    Thanks,
    Divyeshkumar
  • Hi Steve,

    Do we have an update?

    Thanks,
    Divyeshkumar
  • In your earlier post with the register dump, I am puzzled why all the zeroes. Probably because there was a timeout and the eMMC was not initialized properly. It should look more like this
    => md.l 0x480b4200
    480b4200: 00000000 00000200 0070ffff 113a0011 ..........p...:.
    480b4210: 00000900 ffffffef 0f5903ff 00000b00 ..........Y.....
    480b4220: 00000000 01f00000 00000d00 000e0107 ................
    480b4230: 20008000 327f0033 00000000 00070000 ... 3..2........
    480b4240: 22e90080 00000f77 00000000 00000000 ..."w...........

    How did Silicon Motion test the eMMC? Do you or Silicon Motion have an analyzer so that you can capture the command and response to and from the eMMC on your board booting u-boot? I would like to see what command is failing.

    Steve K.
  • Hi Steve,

    I have queried your questions to Silicon Motion and will let you know what they respond.

    I did notice all zeros in the controller registers, can you think of other reasons why  will that happen?

    Thanks,

    Divyeshkumar

  • Hi Steve,

    Here is the command trace for unit that will NOT boot:

    Fail.xlsx

    Here is the command trace from working unit:

    Pass.xlsx

    Let me know if you need more info.

    Thanks,

    Divyeshkumar

  • Hi Steve,

    I have analyzed the files above and noticed that,

    In file Fail.xlsx, line #513 and in file Pass.xlsx, line #515, why host controller is asking to read two different memory region?

    The other test Silicon Motion did was, they program the failed board's eMMC chip byte by byte using their proprietary tools and the board started booting fine. Prior to programming the board, they confirmed that the image stored in the working board and the bad board were different.


    Thanks,
    Divyeshkumar

  • Hi Steve,

    Have you looked at the command trace?

    Thanks,

    Divyeshkumar

  • I am still analyzing them.

    Steve K.
  • OK thanks.

    We performed some other exercise that may give you some clues:

    1. We asked Silicon Motion to clone the eMMC image from the not booting board and we loaded that image on an SD card and could able to boot fine from the SD card.
    2. Silicon Motion used that same failed boot eMMC image and reloaded the same eMMC chip with that image and same not booting device was able to boot fine from it.

    This confirms that the data on the eMMC chip was not corrupted and there was something wrong with the chip before Silicon Motion reloaded the failed image.

    I used SD card and tried to communicate with the eMMC chip on a failed board and below are some output:

    [ 2.337728] mmc1: Enabling HPI failed
    [ 2.341593] mmc1: new high speed MMC card at address 0001
    [ 2.348010] mmcblk1: mmc1:0001 M MOR 128 MiB
    [ 2.353063] mmcblk1boot0: mmc1:0001 M MOR partition 1 2.00 MiB
    [ 2.359655] mmcblk1boot1: mmc1:0001 M MOR partition 2 2.00 MiB

    [ 3.764755] mmcblk1: error -110 transferring data, sector 0, nr 8, cmd response 0x900, card status 0xb00
    [ 3.774735] mmcblk1: retrying using single block read
    [ 5.178289] mmcblk1: error -110 transferring data, sector 0, nr 8, cmd response 0x900, card status 0x0
    [ 5.188164] blk_update_request: I/O error, dev mmcblk1, sector 0
    [ 6.592719] mmcblk1: error -110 transferring data, sector 1, nr 7, cmd response 0x900, card status 0x0
    [ 6.602577] blk_update_request: I/O error, dev mmcblk1, sector 1
    [ 8.007129] mmcblk1: error -110 transferring data, sector 2, nr 6, cmd response 0x900, card status 0x0
    [ 8.016909] blk_update_request: I/O error, dev mmcblk1, sector 2
    [ 9.421472] mmcblk1: error -110 transferring data, sector 3, nr 5, cmd response 0x900, card status 0x0
    [ 9.431268] blk_update_request: I/O error, dev mmcblk1, sector 3
    [ 10.835819] mmcblk1: error -110 transferring data, sector 4, nr 4, cmd response 0x900, card status 0x0
    [ 10.845631] blk_update_request: I/O error, dev mmcblk1, sector 4
    [ 12.250182] mmcblk1: error -110 transferring data, sector 5, nr 3, cmd response 0x900, card status 0x0
    [ 12.259974] blk_update_request: I/O error, dev mmcblk1, sector 5
    [ 13.664528] mmcblk1: error -110 transferring data, sector 6, nr 2, cmd response 0x900, card status 0x0
    [ 13.674334] blk_update_request: I/O error, dev mmcblk1, sector 6
    [ 15.078919] mmcblk1: error -110 transferring data, sector 7, nr 1, cmd response 0x900, card status 0x0
    [ 15.088707] blk_update_request: I/O error, dev mmcblk1, sector 7
    [ 15.095075] Buffer I/O error on dev mmcblk1, logical block 0, async page read
    [ 16.500930] mmcblk1: error -110 transferring data, sector 0, nr 8, cmd response 0x900, card status 0xb00
    [ 16.510891] mmcblk1: retrying using single block read
    [ 17.914443] mmcblk1: error -110 transferring data, sector 0, nr 8, cmd response 0x900, card status 0x0
    [ 17.924248] blk_update_request: I/O error, dev mmcblk1, sector 0
    [ 19.328810] mmcblk1: error -110 transferring data, sector 1, nr 7, cmd response 0x900, card status 0x0
    [ 19.338612] blk_update_request: I/O error, dev mmcblk1, sector 1
    [ 20.743160] mmcblk1: error -110 transferring data, sector 2, nr 6, cmd response 0x900, card status 0x0
    [ 20.752972] blk_update_request: I/O error, dev mmcblk1, sector 2
    [ 22.157521] mmcblk1: error -110 transferring data, sector 3, nr 5, cmd response 0x900, card status 0x0
    [ 22.167348] blk_update_request: I/O error, dev mmcblk1, sector 3
    [ 23.571931] mmcblk1: error -110 transferring data, sector 4, nr 4, cmd response 0x900, card status 0x0
    [ 23.581714] blk_update_request: I/O error, dev mmcblk1, sector 4
    [ 24.986268] mmcblk1: error -110 transferring data, sector 5, nr 3, cmd response 0x900, card status 0x0
    [ 24.996064] blk_update_request: I/O error, dev mmcblk1, sector 5
    [ 26.400610] mmcblk1: error -110 transferring data, sector 6, nr 2, cmd response 0x900, card status 0x0
    [ 26.410445] blk_update_request: I/O error, dev mmcblk1, sector 6
    [ 27.815003] mmcblk1: error -110 transferring data, sector 7, nr 1, cmd response 0x900, card status 0x0
    [ 27.824805] blk_update_request: I/O error, dev mmcblk1, sector 7
    [ 27.831124] Buffer I/O error on dev mmcblk1, logical block 0, async page read
    [ 27.838685] mmcblk1: unable to read partition table
    [ 27.854586] omap_hsmmc 480d1000.mmc: card claims to support voltages below defined range

    [ 32.131121] mmcblk1: error -110 transferring data, sector 262016, nr 8, cmd response 0x900, card status 0xb00
    [ 32.141819] mmcblk1: retrying using single block read

    [ 44.915661] mmcblk1: error -110 transferring data, sector 262016, nr 8, cmd response 0x900, card status 0xb00
    [ 44.926301] mmcblk1: retrying using single block read
    [ 46.329961] mmcblk1: error -110 transferring data, sector 262016, nr 8, cmd response 0x900, card status 0x0
    [ 46.340241] blk_update_request: I/O error, dev mmcblk1, sector 262016
    [ 47.746226] mmcblk1: error -110 transferring data, sector 262017, nr 7, cmd response 0x900, card status 0x0
    [ 47.756485] blk_update_request: I/O error, dev mmcblk1, sector 262017
    [ 49.161537] mmcblk1: error -110 transferring data, sector 262018, nr 6, cmd response 0x900, card status 0x0
    [ 49.171788] blk_update_request: I/O error, dev mmcblk1, sector 262018
    [ 50.578198] mmcblk1: error -110 transferring data, sector 262019, nr 5, cmd response 0x900, card status 0x0
    [ 50.588455] blk_update_request: I/O error, dev mmcblk1, sector 262019
    [ 51.993643] mmcblk1: error -110 transferring data, sector 262020, nr 4, cmd response 0x900, card status 0x0
    [ 52.003930] blk_update_request: I/O error, dev mmcblk1, sector 262020
    [ 53.410478] mmcblk1: error -110 transferring data, sector 262021, nr 3, cmd response 0x900, card status 0x0
    [ 53.420720] blk_update_request: I/O error, dev mmcblk1, sector 262021
    [ 54.825986] mmcblk1: error -110 transferring data, sector 262022, nr 2, cmd response 0x900, card status 0x0
    [ 54.836690] blk_update_request: I/O error, dev mmcblk1, sector 262022

    If you want, I can provide you a complete boot log.

    Thanks,
    Divyeshkumar
  • Yes, can you post a complete boot log? Please attach it as a file. Thank you.

    Steve K.

  • Hi Steve,

    Here's the complete boot log: 

    [    0.000000] Booting Linux on physical CPU 0x0
    [    0.000000] Initializing cgroup subsys cpuset
    [    0.000000] Initializing cgroup subsys cpu
    [    0.000000] Initializing cgroup subsys cpuacct
    [    0.000000] Linux version 4.4.52-g0b07074fe6 (dmaisuria@USNYGRPCLNX09855) (gcc version 5.2.0 (Sourcery CodeBench 2015.12-104) ) #1 SMP PREEMPT Thu Jan 11 14:34:43 EST 2018
    [    0.000000] CPU: ARMv7 Processor [412fc0f2] revision 2 (ARMv7), cr=30c5387d
    [    0.000000] CPU: PIPT / VIPT nonaliasing data cache, PIPT instruction cache
    [    0.000000] Machine model: TI AM5728 Raymond-GDM
    [    0.000000] bootconsole [earlycon0] enabled
    [    0.000000] Reserved memory: created CMA memory pool at 0x0000000095800000, size 56 MiB
    [    0.000000] Reserved memory: initialized node ipu2_cma@95800000, compatible id shared-dma-pool
    [    0.000000] Reserved memory: created CMA memory pool at 0x0000000099000000, size 64 MiB
    [    0.000000] Reserved memory: initialized node dsp1_cma@99000000, compatible id shared-dma-pool
    [    0.000000] Reserved memory: created CMA memory pool at 0x000000009d000000, size 32 MiB
    [    0.000000] Reserved memory: initialized node ipu1_cma@9d000000, compatible id shared-dma-pool
    [    0.000000] Reserved memory: created CMA memory pool at 0x000000009f000000, size 8 MiB
    [    0.000000] Reserved memory: initialized node dsp2_cma@9f000000, compatible id shared-dma-pool
    [    0.000000] cma: Reserved 24 MiB at 0x00000000fe400000
    [    0.000000] Forcing write-allocate cache policy for SMP
    [    0.000000] Memory policy: Data cache writealloc
    [    0.000000] OMAP4: Map 0x00000000ffd00000 to fe600000 for dram barrier
    [    0.000000] On node 0 totalpages: 474368
    [    0.000000] free_area_init_node: node 0, pgdat c0a7aa80, node_mem_map eeda2000
    [    0.000000]   DMA zone: 1728 pages used for memmap
    [    0.000000]   DMA zone: 0 pages reserved
    [    0.000000]   DMA zone: 147456 pages, LIFO batch:31
    [    0.000000]   HighMem zone: 326912 pages, LIFO batch:31
    [    0.000000] DRA752 ES2.0
    [    0.000000] PERCPU: Embedded 11 pages/cpu @eed0b000 s16104 r8192 d20760 u45056
    [    0.000000] pcpu-alloc: s16104 r8192 d20760 u45056 alloc=11*4096
    [    0.000000] pcpu-alloc: [0] 0 [0] 1 
    [    0.000000] Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 472640
    [    0.000000] Kernel command line: console=ttyO2,115200n8 vt.global_cursor_default=0 root=/dev/mmcblk0p4 ro rootfstype=ext4 rootwait earlyprintk
    [    0.000000] PID hash table entries: 4096 (order: 2, 16384 bytes)
    [    0.000000] Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
    [    0.000000] Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
    [    0.000000] Memory: 1673636K/1897472K available (6973K kernel code, 480K rwdata, 2860K rodata, 452K init, 6059K bss, 35420K reserved, 188416K cma-reserved, 1283072K highmem)
    [    0.000000] Virtual kernel memory layout:
                       vector  : 0xffff0000 - 0xffff1000   (   4 kB)
                       fixmap  : 0xffc00000 - 0xfff00000   (3072 kB)
                       vmalloc : 0xf0800000 - 0xff800000   ( 240 MB)
                       lowmem  : 0xc0000000 - 0xf0000000   ( 768 MB)
                       pkmap   : 0xbfe00000 - 0xc0000000   (   2 MB)
                       modules : 0xbf000000 - 0xbfe00000   (  14 MB)
                         .text : 0xc0008000 - 0xc09a29e8   (9835 kB)
                         .init : 0xc09a3000 - 0xc0a14000   ( 452 kB)
                         .data : 0xc0a14000 - 0xc0a8c248   ( 481 kB)
                          .bss : 0xc0a8e000 - 0xc1078c18   (6060 kB)
    [    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
    [    0.000000] Preemptible hierarchical RCU implementation.
    [    0.000000]  Build-time adjustment of leaf fanout to 32.
    [    0.000000] NR_IRQS:16 nr_irqs:16 16
    [    0.000000] ti_dt_clocks_register: failed to lookup clock node gmac_gmii_ref_clk_div
    [    0.000000] OMAP clockevent source: timer1 at 32786 Hz
    [    0.000000] Architected cp15 timer(s) running at 6.14MHz (virt).
    [    0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0x16af5adb9, max_idle_ns: 440795202250 ns
    [    0.000004] sched_clock: 56 bits at 6MHz, resolution 162ns, wraps every 4398046511023ns
    [    0.008306] Switching to timer-based delay loop, resolution 162ns
    [    0.014939] clocksource: 32k_counter: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 58327039986419 ns
    [    0.025199] OMAP clocksource: 32k_counter at 32768 Hz
    [    0.031001] Console: colour dummy device 80x30
    [    0.035665] Lock dependency validator: Copyright (c) 2006 Red Hat, Inc., Ingo Molnar
    [    0.043705] ... MAX_LOCKDEP_SUBCLASSES:  8
    [    0.047998] ... MAX_LOCK_DEPTH:          48
    [    0.052382] ... MAX_LOCKDEP_KEYS:        8191
    [    0.056944] ... CLASSHASH_SIZE:          4096
    [    0.061505] ... MAX_LOCKDEP_ENTRIES:     32768
    [    0.066155] ... MAX_LOCKDEP_CHAINS:      65536
    [    0.070805] ... CHAINHASH_SIZE:          32768
    [    0.075456]  memory used by lock dependency info: 5151 kB
    [    0.081087]  per task-struct memory footprint: 1536 bytes
    [    0.086729] Calibrating delay loop (skipped), value calculated using timer frequency.. 12.29 BogoMIPS (lpj=6147)
    [    0.097281] pid_max: default: 32768 minimum: 301
    [    0.102309] Mount-cache hash table entries: 2048 (order: 1, 8192 bytes)
    [    0.109196] Mountpoint-cache hash table entries: 2048 (order: 1, 8192 bytes)
    [    0.117757] Initializing cgroup subsys io
    [    0.122004] Initializing cgroup subsys memory
    [    0.126601] Initializing cgroup subsys devices
    [    0.131281] Initializing cgroup subsys freezer
    [    0.135959] Initializing cgroup subsys perf_event
    [    0.140890] Initializing cgroup subsys pids
    [    0.145311] CPU: Testing write buffer coherency: ok
    [    0.150935] /cpus/cpu@0 missing clock-frequency property
    [    0.156499] /cpus/cpu@1 missing clock-frequency property
    [    0.162049] CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
    [    0.167979] Setting up static identity map for 0x80008380 - 0x800083e0
    [    0.186312] CPU1: thread -1, cpu 1, socket 0, mpidr 80000001
    [    0.186461] Brought up 2 CPUs
    [    0.195507] SMP: Total of 2 processors activated (24.58 BogoMIPS).
    [    0.201942] CPU: All CPU(s) started in SVC mode.
    [    0.207718] devtmpfs: initialized
    [    0.277181] VFP support v0.3: implementor 41 architecture 4 part 30 variant f rev 0
    [    0.286903] omap_hwmod: l3_main_2 using broken dt data from ocp
    [    0.601771] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns
    [    0.611885] futex hash table entries: 512 (order: 3, 32768 bytes)
    [    0.623784] pinctrl core: initialized pinctrl subsystem
    [    0.631428] NET: Registered protocol family 16
    [    0.637664] DMA: preallocated 256 KiB pool for atomic coherent allocations
    [    0.649889] cpuidle: using governor ladder
    [    0.658244] cpuidle: using governor menu
    [    0.675066] gpiochip_add: registered GPIOs 0 to 31 on device: gpio
    [    0.676002] OMAP GPIO hardware version 0.1
    [    0.681362] gpiochip_add: registered GPIOs 32 to 63 on device: gpio
    [    0.682935] gpiochip_add: registered GPIOs 64 to 95 on device: gpio
    [    0.684489] gpiochip_add: registered GPIOs 96 to 127 on device: gpio
    [    0.686035] gpiochip_add: registered GPIOs 128 to 159 on device: gpio
    [    0.687591] gpiochip_add: registered GPIOs 160 to 191 on device: gpio
    [    0.689124] gpiochip_add: registered GPIOs 192 to 223 on device: gpio
    [    0.690689] gpiochip_add: registered GPIOs 224 to 255 on device: gpio
    [    0.692189] irq: no irq domain found for /ocp/l4@4a000000/scm@2000/pinmux@1400 !
    [    0.701311] irq: no irq domain found for /ocp/l4@4a000000/scm@2000/pinmux@1400 !
    [    0.709964] irq: no irq domain found for /ocp/l4@4a000000/scm@2000/pinmux@1400 !
    [    0.718192] irq: no irq domain found for /ocp/l4@4a000000/scm@2000/pinmux@1400 !
    [    0.763273] hw-breakpoint: found 5 (+1 reserved) breakpoint and 4 watchpoint registers.
    [    0.771592] hw-breakpoint: maximum watchpoint size is 8 bytes.
    [    0.778267] omap4_sram_init:Unable to allocate sram needed to handle errata I688
    [    0.785966] omap4_sram_init:Unable to get sram pool needed to handle errata I688
    [    0.794850] OMAP DMA hardware revision 0.0
    [    0.839233] omap-dma-engine 4a056000.dma-controller: OMAP DMA engine driver (LinkedList1/2/3 supported)
    [    0.850818] edma 43300000.edma: memcpy is disabled
    [    0.866207] edma 43300000.edma: TI EDMA DMA engine driver
    [    0.872517] reg-fixed-voltage fixedregulator@2: could not find pctldev for node /ocp/l4@4a000000/scm@2000/pinmux@1400/wlan_pins_default, deferring probe
    [    0.886730] of_get_named_gpiod_flags: can't parse 'gpio' property of node '/fixedregulator-vdd_3v3[0]'
    [    0.887153] of_get_named_gpiod_flags: can't parse 'gpio' property of node '/fixedregulator-aic_dvdd[0]'
    [    0.887498] reg-fixed-voltage fixedregulator-sw_5v: could not find pctldev for node /ocp/l4@4a000000/scm@2000/pinmux@1400/sw_5v_en_default, deferring probe
    [    0.906387] omap-iommu 40d01000.mmu: 40d01000.mmu registered
    [    0.912578] omap-iommu 40d02000.mmu: 40d02000.mmu registered
    [    0.918744] omap-iommu 58882000.mmu: 58882000.mmu registered
    [    0.924934] omap-iommu 55082000.mmu: 55082000.mmu registered
    [    0.931257] omap-iommu 41501000.mmu: 41501000.mmu registered
    [    0.937452] omap-iommu 41502000.mmu: 41502000.mmu registered
    [    0.947741] pca953x 0-0021: could not find pctldev for node /ocp/l4@4a000000/scm@2000/pinmux@1400/tca6416_pins_default, deferring probe
    [    0.960719] palmas 0-0058: Irq flag is 0x00000008
    [    0.980851] palmas 0-0058: Muxing GPIO 2f, PWM 0, LED 0
    [    1.018919] gpiochip_find_base: found new base at 504
    [    1.019122] gpiochip_add: registered GPIOs 504 to 511 on device: 48070000.i2c:tps659038@58:tps659038_gpio
    [    1.019630] omap_i2c 48070000.i2c: bus 0 rev0.12 at 400 kHz
    [    1.025545] omap_i2c 48060000.i2c: could not find pctldev for node /ocp/l4@4a000000/scm@2000/pinmux@1400/i2c3_pins_default, deferring probe
    [    1.038545] omap_i2c 4807a000.i2c: could not find pctldev for node /ocp/l4@4a000000/scm@2000/pinmux@1400/i2c4_pins_default, deferring probe
    [    1.051752] media: Linux media interface: v0.10
    [    1.056577] Linux video capture interface: v2.00
    [    1.061464] pps_core: LinuxPPS API ver. 1 registered
    [    1.066655] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
    [    1.076152] PTP clock support registered
    [    1.080334] EDAC MC: Ver: 3.0.0
    [    1.085124] omap-mailbox 48840000.mailbox: omap mailbox rev 0x400
    [    1.091981] omap-mailbox 48842000.mailbox: omap mailbox rev 0x400
    [    1.098878] Advanced Linux Sound Architecture Driver Initialized.
    [    1.106412] Bluetooth: Core ver 2.21
    [    1.110223] NET: Registered protocol family 31
    [    1.114880] Bluetooth: HCI device and connection manager initialized
    [    1.121529] Bluetooth: HCI socket layer initialized
    [    1.126642] Bluetooth: L2CAP socket layer initialized
    [    1.132000] Bluetooth: SCO socket layer initialized
    [    1.137868] clocksource: Switched to clocksource arch_sys_counter
    [    1.165790] NET: Registered protocol family 2
    [    1.171455] TCP established hash table entries: 8192 (order: 3, 32768 bytes)
    [    1.178909] TCP bind hash table entries: 8192 (order: 6, 294912 bytes)
    [    1.187215] TCP: Hash tables configured (established 8192 bind 8192)
    [    1.193943] UDP hash table entries: 512 (order: 3, 40960 bytes)
    [    1.200321] UDP-Lite hash table entries: 512 (order: 3, 40960 bytes)
    [    1.207512] NET: Registered protocol family 1
    [    1.212769] RPC: Registered named UNIX socket transport module.
    [    1.218991] RPC: Registered udp transport module.
    [    1.223949] RPC: Registered tcp transport module.
    [    1.228878] RPC: Registered tcp NFSv4.1 backchannel transport module.
    [    1.235601] PCI: CLS 0 bytes, default 64
    [    1.237269] hw perfevents: enabled with armv7_cortex_a15 PMU driver, 7 counters available
    [    1.268533] squashfs: version 4.0 (2009/01/31) Phillip Lougher
    [    1.276457] NFS: Registering the id_resolver key type
    [    1.281906] Key type id_resolver registered
    [    1.286301] Key type id_legacy registered
    [    1.290612] ntfs: driver 2.1.32 [Flags: R/O].
    [    1.306453] bounce: pool size: 64 pages
    [    1.310879] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248)
    [    1.318609] io scheduler noop registered
    [    1.322753] io scheduler deadline registered
    [    1.327293] io scheduler cfq registered (default)
    [    1.340521] pinctrl-single 4a003400.pinmux: 282 pins at pa fc003400 size 1128
    [    1.355974] pwm-backlight backlight: GPIO lookup for consumer enable
    [    1.355985] pwm-backlight backlight: using device tree for GPIO lookup
    [    1.355998] of_get_named_gpiod_flags: can't parse 'enable-gpios' property of node '/backlight[0]'
    [    1.356010] of_get_named_gpiod_flags: can't parse 'enable-gpio' property of node '/backlight[0]'
    [    1.356019] pwm-backlight backlight: using lookup tables for GPIO lookup
    [    1.356066] pwm-backlight backlight: lookup for GPIO enable failed
    [    1.356111] backlight supply power not found, using dummy regulator
    [    1.370058] omap_uart 4806a000.serial: no wakeirq for uart0
    [    1.375884] omap_uart 4806a000.serial: RS485 Probe
    [    1.380906] of_get_named_gpiod_flags: can't parse 'rts-gpio' property of node '/ocp/serial@4806a000[0]'
    [    1.380918] of_get_named_gpiod_flags: can't parse 'de-gpio' property of node '/ocp/serial@4806a000[0]'
    [    1.381239] 4806a000.serial: ttyO0 at MMIO 0x4806a000 (irq = 299, base_baud = 3000000) is a OMAP UART0
    [    1.391579] omap_uart 48020000.serial: RS485 Probe
    [    1.396608] of_get_named_gpiod_flags: can't parse 'rts-gpio' property of node '/ocp/serial@48020000[0]'
    [    1.396619] of_get_named_gpiod_flags: can't parse 'de-gpio' property of node '/ocp/serial@48020000[0]'
    [    1.396733] 48020000.serial: ttyO2 at MMIO 0x48020000 (irq = 300, base_baud = 3000000) is a OMAP UART2
    [    1.406608] console [ttyO2] enabled
    [    1.413939] bootconsole [earlycon0] disabled
    [    1.423590] omap_uart 4806e000.serial: no wakeirq for uart3
    [    1.429449] omap_uart 4806e000.serial: RS485 Probe
    [    1.434509] of_get_named_gpiod_flags: parsed 'rts-gpio' property of node '/ocp/serial@4806e000[0]' - status (0)
    [    1.434576] of_get_named_gpiod_flags: parsed 'de-gpio' property of node '/ocp/serial@4806e000[0]' - status (0)
    [    1.434601] omap_uart 4806e000.serial: RS485 for uart3
    [    1.440147] 4806e000.serial: ttyO3 at MMIO 0x4806e000 (irq = 301, base_baud = 3000000) is a OMAP UART3
    [    1.450565] omap_uart 48066000.serial: no wakeirq for uart4
    [    1.456424] omap_uart 48066000.serial: RS485 Probe
    [    1.461456] of_get_named_gpiod_flags: can't parse 'rts-gpio' property of node '/ocp/serial@48066000[0]'
    [    1.461467] of_get_named_gpiod_flags: can't parse 'de-gpio' property of node '/ocp/serial@48066000[0]'
    [    1.461633] 48066000.serial: ttyO4 at MMIO 0x48066000 (irq = 302, base_baud = 3000000) is a OMAP UART4
    [    1.472099] omap_uart 48068000.serial: RS485 Probe
    [    1.477139] of_get_named_gpiod_flags: can't parse 'rts-gpio' property of node '/ocp/serial@48068000[0]'
    [    1.477150] of_get_named_gpiod_flags: can't parse 'de-gpio' property of node '/ocp/serial@48068000[0]'
    [    1.477324] 48068000.serial: ttyO5 at MMIO 0x48068000 (irq = 303, base_baud = 3000000) is a OMAP UART5
    [    1.487749] omap_uart 48420000.serial: no wakeirq for uart6
    [    1.493602] omap_uart 48420000.serial: RS485 Probe
    [    1.498661] of_get_named_gpiod_flags: parsed 'rts-gpio' property of node '/ocp/serial@48420000[0]' - status (0)
    [    1.498722] of_get_named_gpiod_flags: parsed 'de-gpio' property of node '/ocp/serial@48420000[0]' - status (0)
    [    1.498746] omap_uart 48420000.serial: RS485 for uart6
    [    1.504293] 48420000.serial: ttyO6 at MMIO 0x48420000 (irq = 304, base_baud = 3000000) is a OMAP UART6
    [    1.514717] omap_uart 48422000.serial: RS485 Probe
    [    1.519762] of_get_named_gpiod_flags: can't parse 'rts-gpio' property of node '/ocp/serial@48422000[0]'
    [    1.519773] of_get_named_gpiod_flags: can't parse 'de-gpio' property of node '/ocp/serial@48422000[0]'
    [    1.519957] 48422000.serial: ttyO7 at MMIO 0x48422000 (irq = 305, base_baud = 3000000) is a OMAP UART7
    [    1.530405] omap_uart 4ae2b000.serial: RS485 Probe
    [    1.535450] of_get_named_gpiod_flags: can't parse 'rts-gpio' property of node '/ocp/serial@4ae2b000[0]'
    [    1.535461] of_get_named_gpiod_flags: can't parse 'de-gpio' property of node '/ocp/serial@4ae2b000[0]'
    [    1.535633] 4ae2b000.serial: ttyO9 at MMIO 0x4ae2b000 (irq = 306, base_baud = 3000000) is a OMAP UART9
    [    1.546283] [drm] Initialized drm 1.1.0 20060810
    [    1.554427] OMAP DSS rev 6.1
    [    1.558508] omapdss_dss 58000000.dss: bound 58001000.dispc (ops dispc_component_ops)
    [    1.567654] omapdss_dss 58000000.dss: bound 58040000.encoder (ops hdmi5_component_ops)
    [    1.577479] of_get_named_gpiod_flags: parsed 'hpd-gpios' property of node '/connector@0[0]' - status (0)
    [    1.578272] panel-dpi display: GPIO lookup for consumer enable
    [    1.578283] panel-dpi display: using device tree for GPIO lookup
    [    1.578318] of_get_named_gpiod_flags: parsed 'enable-gpios' property of node '/display[0]' - status (0)
    [    1.591177] loop: module loaded
    [    1.600558] at25 spi1.0: 512 Byte at25 eeprom, pagesize 1
    [    1.610396] libphy: Fixed MDIO Bus: probed
    [    1.615799] tun: Universal TUN/TAP device driver, 1.6
    [    1.621104] tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
    [    1.627937] CAN device driver interface
    [    1.633548] c_can_platform 4ae3c000.can: c_can_platform device registered (regs=fce3c000, irq=351)
    [    1.644098] c_can_platform 48480000.can: c_can_platform device registered (regs=fa480000, irq=352)
    [    1.655552] PPP generic driver version 2.4.2
    [    1.660390] PPP BSD Compression module registered
    [    1.665349] PPP Deflate Compression module registered
    [    1.670670] PPP MPPE Compression module registered
    [    1.675716] NET: Registered protocol family 24
    [    1.682067] mousedev: PS/2 mouse device common for all mice
    [    1.688695] -->tsc2013_init
    [    1.691639] tsc2013_init-->i2c_add_driver
    [    1.698102] rtc rtc0: bq32k: dev (254:0)
    [    1.698125] bq32k 0-0068: rtc core: registered bq32k as rtc0
    [    1.704901] i2c /dev entries driver
    [    1.713561] Bluetooth: HCI UART driver ver 2.3
    [    1.718233] Bluetooth: HCI UART protocol H4 registered
    [    1.723634] Bluetooth: HCI UART protocol BCSP registered
    [    1.730743] omap_hsmmc 4809c000.mmc: GPIO lookup for consumer cd
    [    1.730754] omap_hsmmc 4809c000.mmc: using device tree for GPIO lookup
    [    1.730789] of_get_named_gpiod_flags: parsed 'cd-gpios' property of node '/ocp/mmc@4809c000[0]' - status (0)
    [    1.730807] omap_hsmmc 4809c000.mmc: Got CD GPIO
    [    1.735657] omap_hsmmc 4809c000.mmc: GPIO lookup for consumer wp
    [    1.735667] omap_hsmmc 4809c000.mmc: using device tree for GPIO lookup
    [    1.735677] of_get_named_gpiod_flags: can't parse 'wp-gpios' property of node '/ocp/mmc@4809c000[0]'
    [    1.735688] of_get_named_gpiod_flags: can't parse 'wp-gpio' property of node '/ocp/mmc@4809c000[0]'
    [    1.735697] omap_hsmmc 4809c000.mmc: using lookup tables for GPIO lookup
    [    1.735707] omap_hsmmc 4809c000.mmc: lookup for GPIO wp failed
    [    1.736032] omap_hsmmc 4809c000.mmc: no pinctrl state for sdr104 mode
    [    1.742792] omap_hsmmc 4809c000.mmc: no pinctrl state for ddr50 mode
    [    1.749461] omap_hsmmc 4809c000.mmc: no pinctrl state for sdr50 mode
    [    1.756126] omap_hsmmc 4809c000.mmc: no pinctrl state for sdr25 mode
    [    1.762784] omap_hsmmc 4809c000.mmc: no pinctrl state for sdr12 mode
    [    1.794527] omap_hsmmc 480b4000.mmc: GPIO lookup for consumer cd
    [    1.794538] omap_hsmmc 480b4000.mmc: using device tree for GPIO lookup
    [    1.794550] of_get_named_gpiod_flags: can't parse 'cd-gpios' property of node '/ocp/mmc@480b4000[0]'
    [    1.794561] of_get_named_gpiod_flags: can't parse 'cd-gpio' property of node '/ocp/mmc@480b4000[0]'
    [    1.794570] omap_hsmmc 480b4000.mmc: using lookup tables for GPIO lookup
    [    1.794581] omap_hsmmc 480b4000.mmc: lookup for GPIO cd failed
    [    1.794593] omap_hsmmc 480b4000.mmc: GPIO lookup for consumer wp
    [    1.794601] omap_hsmmc 480b4000.mmc: using device tree for GPIO lookup
    [    1.794612] of_get_named_gpiod_flags: can't parse 'wp-gpios' property of node '/ocp/mmc@480b4000[0]'
    [    1.794622] of_get_named_gpiod_flags: can't parse 'wp-gpio' property of node '/ocp/mmc@480b4000[0]'
    [    1.794630] omap_hsmmc 480b4000.mmc: using lookup tables for GPIO lookup
    [    1.794640] omap_hsmmc 480b4000.mmc: lookup for GPIO wp failed
    [    1.794910] omap_hsmmc 480b4000.mmc: no pinctrl state for sdr25 mode
    [    1.801583] omap_hsmmc 480b4000.mmc: no pinctrl state for sdr12 mode
    [    1.808436] vdd_3v3: supplied by regen1
    [    1.835191] omap_hsmmc 480d1000.mmc: GPIO lookup for consumer cd
    [    1.835201] omap_hsmmc 480d1000.mmc: using device tree for GPIO lookup
    [    1.835213] of_get_named_gpiod_flags: can't parse 'cd-gpios' property of node '/ocp/mmc@480d1000[0]'
    [    1.835223] of_get_named_gpiod_flags: can't parse 'cd-gpio' property of node '/ocp/mmc@480d1000[0]'
    [    1.835232] omap_hsmmc 480d1000.mmc: using lookup tables for GPIO lookup
    [    1.835242] omap_hsmmc 480d1000.mmc: lookup for GPIO cd failed
    [    1.835253] omap_hsmmc 480d1000.mmc: GPIO lookup for consumer wp
    [    1.835262] omap_hsmmc 480d1000.mmc: using device tree for GPIO lookup
    [    1.835272] of_get_named_gpiod_flags: can't parse 'wp-gpios' property of node '/ocp/mmc@480d1000[0]'
    [    1.835282] of_get_named_gpiod_flags: can't parse 'wp-gpio' property of node '/ocp/mmc@480d1000[0]'
    [    1.835290] omap_hsmmc 480d1000.mmc: using lookup tables for GPIO lookup
    [    1.835300] omap_hsmmc 480d1000.mmc: lookup for GPIO wp failed
    [    1.835541] omap_hsmmc 480d1000.mmc: no pinctrl state for sdr25 mode
    [    1.842211] omap_hsmmc 480d1000.mmc: no pinctrl state for sdr12 mode
    [    1.848882] omap_hsmmc 480d1000.mmc: no pinctrl state for hs mode
    [    1.861643] ledtrig-cpu: registered to indicate activity on CPUs
    [    1.878630] of_get_named_gpiod_flags: can't parse 'gpio-reset' property of node '/ocp/i2c@48070000/tlv320aic3104@18[0]'
    [    1.878667] of_get_named_gpiod_flags: parsed 'gpio-mute' property of node '/ocp/i2c@48070000/tlv320aic3104@18[0]' - status (-517)
    [    1.882804] NET: Registered protocol family 10
    [    1.889028] sit: IPv6 over IPv4 tunneling driver
    [    1.895025] NET: Registered protocol family 17
    [    1.899706] can: controller area network core (rev 20120528 abi 9)
    [    1.906519] NET: Registered protocol family 29
    [    1.911216] can: raw protocol (rev 20120528)
    [    1.915740] can: broadcast manager protocol (rev 20120528 t)
    [    1.921697] can: netlink gateway (rev 20130117) max_hops=1
    [    1.927868] Bluetooth: RFCOMM TTY layer initialized
    [    1.933008] Bluetooth: RFCOMM socket layer initialized
    [    1.938414] Bluetooth: RFCOMM ver 1.11
    [    1.942371] Bluetooth: BNEP (Ethernet Emulation) ver 1.3
    [    1.947946] Bluetooth: BNEP filters: protocol multicast
    [    1.953448] Bluetooth: BNEP socket layer initialized
    [    1.958673] Bluetooth: HIDP (Human Interface Emulation) ver 1.2
    [    1.964899] Bluetooth: HIDP socket layer initialized
    [    1.970722] Key type dns_resolver registered
    [    1.975345] omap_voltage_late_init: Voltage driver support not added
    [    1.982896] Adding alias for supply vdd,cpu0 -> vdd,4a003b20.oppdm
    [    1.989386] Adding alias for supply vbb,cpu0 -> vbb,4a003b20.oppdm
    [    1.996652] Adding alias for supply vdd,cpu0 -> vdd,4a003b20.oppdm
    [    2.003190] Adding alias for supply vbb,cpu0 -> vbb,4a003b20.oppdm
    [    2.013505] Power Management for TI OMAP4+ devices.
    [    2.018870] Registering SWP/SWPB emulation handler
    [    2.026457] dmm 4e000000.dmm: initialized all PAT entries
    [    2.038533] [drm] Supports vblank timestamp caching Rev 2 (21.10.2013).
    [    2.045472] [drm] No driver support for vblank timestamp query.
    [    2.053666] [drm] Enabling DMM ywrap scrolling
    [    2.095409] mmc0: host does not support reading read-only switch, assuming write-enable
    [    2.097351] mmc0: Problem switching card into high-speed mode!
    [    2.097439] mmc0: new SDHC card at address 0001
    [    2.098282] Console: switching to colour frame buffer device 60x34
    [    2.099328] mmcblk0: mmc0:0001 SD32G 29.0 GiB 
    [    2.110643] GPT:Primary header thinks Alt. header is not at the end of the disk.
    [    2.110646] GPT:15269887 != 60751871
    [    2.110647] GPT:Alternate GPT header not at the end of the disk.
    [    2.110650] GPT:15269887 != 60751871
    [    2.110651] GPT: Use GNU Parted to correct GPT errors.
    [    2.110672]  mmcblk0: p1 p2 p3 p4 p5 p6 p7
    [    2.160902] omapdrm omapdrm.0: fb0: omapdrm frame buffer device
    [    2.172500] [drm] Initialized omapdrm 1.0.0 20110917 on minor 0
    [    2.180275] of_get_named_gpiod_flags: parsed 'gpio' property of node '/fixedregulator@2[0]' - status (0)
    [    2.180701] of_get_named_gpiod_flags: parsed 'gpio' property of node '/fixedregulator-sw_5v[0]' - status (0)
    [    2.181643] gpiochip_find_base: found new base at 488
    [    2.181774] gpiochip_add: registered GPIOs 488 to 503 on device: tca6416
    [    2.183227] omap_i2c 48060000.i2c: bus 2 rev0.12 at 100 kHz
    [    2.202739] tmp102 3-0048: initialized
    [    2.206942] -->tsc2013_probe
    [    2.210019] of_get_named_gpiod_flags: can't parse 'gpios' property of node '/ocp/i2c@4807a000/tsc2013@4b[0]'
    [    2.210026] tsc2013 3-004b: GPIO not specified in DT (of_get_gpio returned -2)
    [    2.217613] tsc2013_probe-->tsc2013_hw_init
    [    2.221998] -->tsc2013_hw_init
    [    2.225198] tsc2013_probe-->tsc2013_sw_init
    [    2.229736] tsc2013_i2c_write_register : CFG[62] = 0xa58c
    [    2.235544] tsc2013_i2c_write_register : CFG[6a] = 0x0
    [    2.241084] tsc2013_i2c_write_register : CFG[72] = 0xb81c
    [    2.247073] input: tsc2013_ts as /devices/virtual/input/input0
    [    2.253691] omap_i2c 4807a000.i2c: bus 3 rev0.12 at 400 kHz
    [    2.259985] omap_hsmmc 480d1000.mmc: GPIO lookup for consumer cd
    [    2.259992] omap_hsmmc 480d1000.mmc: using device tree for GPIO lookup
    [    2.260000] of_get_named_gpiod_flags: can't parse 'cd-gpios' property of node '/ocp/mmc@480d1000[0]'
    [    2.260007] of_get_named_gpiod_flags: can't parse 'cd-gpio' property of node '/ocp/mmc@480d1000[0]'
    [    2.260013] omap_hsmmc 480d1000.mmc: using lookup tables for GPIO lookup
    [    2.260021] omap_hsmmc 480d1000.mmc: lookup for GPIO cd failed
    [    2.260029] omap_hsmmc 480d1000.mmc: GPIO lookup for consumer wp
    [    2.260035] omap_hsmmc 480d1000.mmc: using device tree for GPIO lookup
    [    2.260042] of_get_named_gpiod_flags: can't parse 'wp-gpios' property of node '/ocp/mmc@480d1000[0]'
    [    2.260048] of_get_named_gpiod_flags: can't parse 'wp-gpio' property of node '/ocp/mmc@480d1000[0]'
    [    2.260054] omap_hsmmc 480d1000.mmc: using lookup tables for GPIO lookup
    [    2.260061] omap_hsmmc 480d1000.mmc: lookup for GPIO wp failed
    [    2.260221] omap_hsmmc 480d1000.mmc: no pinctrl state for sdr25 mode
    [    2.266897] omap_hsmmc 480d1000.mmc: no pinctrl state for sdr12 mode
    [    2.273555] omap_hsmmc 480d1000.mmc: no pinctrl state for hs mode
    [    2.337728] mmc1: Enabling HPI failed
    [    2.341593] mmc1: new high speed MMC card at address 0001
    [    2.348010] mmcblk1: mmc1:0001 M MOR 128 MiB 
    [    2.353063] mmcblk1boot0: mmc1:0001 M MOR partition 1 2.00 MiB
    [    2.359655] mmcblk1boot1: mmc1:0001 M MOR partition 2 2.00 MiB
    [    2.404328] of_get_named_gpiod_flags: can't parse 'gpio-reset' property of node '/ocp/i2c@48070000/tlv320aic3104@18[0]'
    [    2.404362] of_get_named_gpiod_flags: parsed 'gpio-mute' property of node '/ocp/i2c@48070000/tlv320aic3104@18[0]' - status (0)
    [    2.404896] aic_dvdd_fixed: supplied by vdd_3v3
    [    2.410545] of_get_named_gpiod_flags: can't parse 'simple-audio-card,hp-det-gpio' property of node '/sound@0[0]'
    [    2.410556] of_get_named_gpiod_flags: can't parse 'simple-audio-card,mic-det-gpio' property of node '/sound@0[0]'
    [    2.412345] asoc-simple-card sound@0: tlv320aic3x-hifi <-> 4846c000.mcasp mapping ok
    [    2.423265] of_get_named_gpiod_flags: parsed 'gpios' property of node '/gpio-keys/left-key[0]' - status (0)
    [    2.423703] input: gpio-keys as /devices/platform/gpio-keys/input/input1
    [    2.431667] bq32k 0-0068: setting system clock to 2018-06-08 13:02:01 UTC (1528462921)
    [    2.468647] aic_dvdd_fixed: disabling
    [    2.472758] ldousb: disabling
    [    2.476595] ALSA device list:
    [    2.479722]   #0: Raymond-GDM
    [    3.764755] mmcblk1: error -110 transferring data, sector 0, nr 8, cmd response 0x900, card status 0xb00
    [    3.774735] mmcblk1: retrying using single block read
    [    5.178289] mmcblk1: error -110 transferring data, sector 0, nr 8, cmd response 0x900, card status 0x0
    [    5.188164] blk_update_request: I/O error, dev mmcblk1, sector 0
    [    6.592719] mmcblk1: error -110 transferring data, sector 1, nr 7, cmd response 0x900, card status 0x0
    [    6.602577] blk_update_request: I/O error, dev mmcblk1, sector 1
    [    8.007129] mmcblk1: error -110 transferring data, sector 2, nr 6, cmd response 0x900, card status 0x0
    [    8.016909] blk_update_request: I/O error, dev mmcblk1, sector 2
    [    9.421472] mmcblk1: error -110 transferring data, sector 3, nr 5, cmd response 0x900, card status 0x0
    [    9.431268] blk_update_request: I/O error, dev mmcblk1, sector 3
    [   10.835819] mmcblk1: error -110 transferring data, sector 4, nr 4, cmd response 0x900, card status 0x0
    [   10.845631] blk_update_request: I/O error, dev mmcblk1, sector 4
    [   12.250182] mmcblk1: error -110 transferring data, sector 5, nr 3, cmd response 0x900, card status 0x0
    [   12.259974] blk_update_request: I/O error, dev mmcblk1, sector 5
    [   13.664528] mmcblk1: error -110 transferring data, sector 6, nr 2, cmd response 0x900, card status 0x0
    [   13.674334] blk_update_request: I/O error, dev mmcblk1, sector 6
    [   15.078919] mmcblk1: error -110 transferring data, sector 7, nr 1, cmd response 0x900, card status 0x0
    [   15.088707] blk_update_request: I/O error, dev mmcblk1, sector 7
    [   15.095075] Buffer I/O error on dev mmcblk1, logical block 0, async page read
    [   16.500930] mmcblk1: error -110 transferring data, sector 0, nr 8, cmd response 0x900, card status 0xb00
    [   16.510891] mmcblk1: retrying using single block read
    [   17.914443] mmcblk1: error -110 transferring data, sector 0, nr 8, cmd response 0x900, card status 0x0
    [   17.924248] blk_update_request: I/O error, dev mmcblk1, sector 0
    [   19.328810] mmcblk1: error -110 transferring data, sector 1, nr 7, cmd response 0x900, card status 0x0
    [   19.338612] blk_update_request: I/O error, dev mmcblk1, sector 1
    [   20.743160] mmcblk1: error -110 transferring data, sector 2, nr 6, cmd response 0x900, card status 0x0
    [   20.752972] blk_update_request: I/O error, dev mmcblk1, sector 2
    [   22.157521] mmcblk1: error -110 transferring data, sector 3, nr 5, cmd response 0x900, card status 0x0
    [   22.167348] blk_update_request: I/O error, dev mmcblk1, sector 3
    [   23.571931] mmcblk1: error -110 transferring data, sector 4, nr 4, cmd response 0x900, card status 0x0
    [   23.581714] blk_update_request: I/O error, dev mmcblk1, sector 4
    [   24.986268] mmcblk1: error -110 transferring data, sector 5, nr 3, cmd response 0x900, card status 0x0
    [   24.996064] blk_update_request: I/O error, dev mmcblk1, sector 5
    [   26.400610] mmcblk1: error -110 transferring data, sector 6, nr 2, cmd response 0x900, card status 0x0
    [   26.410445] blk_update_request: I/O error, dev mmcblk1, sector 6
    [   27.815003] mmcblk1: error -110 transferring data, sector 7, nr 1, cmd response 0x900, card status 0x0
    [   27.824805] blk_update_request: I/O error, dev mmcblk1, sector 7
    [   27.831124] Buffer I/O error on dev mmcblk1, logical block 0, async page read
    [   27.838685]  mmcblk1: unable to read partition table
    [   27.854586] omap_hsmmc 480d1000.mmc: card claims to support voltages below defined range
    [   27.866993] EXT4-fs (mmcblk0p4): mounted filesystem with ordered data mode. Opts: (null)
    [   27.875575] VFS: Mounted root (ext4 filesystem) readonly on device 179:4.
    [   27.877055] mmc2: queuing unknown CIS tuple 0x91 (3 bytes)
    [   27.880615] mmc2: new SDIO card at address 0001
    [   27.898205] devtmpfs: mounted
    [   27.901691] Freeing unused kernel memory: 452K (c09a3000 - c0a14000)
    [   27.908374] This architecture does not have kernel memory protection.
    [   28.458652] random: systemd: uninitialized urandom read (16 bytes read, 57 bits of entropy available)
    [   28.478580] random: systemd: uninitialized urandom read (16 bytes read, 57 bits of entropy available)
    [   28.516218] random: systemd: uninitialized urandom read (16 bytes read, 57 bits of entropy available)
    [   28.574293] random: systemd-gpt-aut: uninitialized urandom read (16 bytes read, 58 bits of entropy available)
    [   28.585131] random: systemd-gpt-aut: uninitialized urandom read (16 bytes read, 58 bits of entropy available)
    [   28.720389] random: systemd: uninitialized urandom read (16 bytes read, 61 bits of entropy available)
    [   28.733268] random: systemd: uninitialized urandom read (16 bytes read, 61 bits of entropy available)
    [   28.743234] random: systemd: uninitialized urandom read (16 bytes read, 61 bits of entropy available)
    [   28.753376] random: systemd: uninitialized urandom read (16 bytes read, 61 bits of entropy available)
    [   28.779237] random: systemd: uninitialized urandom read (16 bytes read, 61 bits of entropy available)
    [   29.195774] CMEMK module: reference Linux version 4.4.52
    [   29.204541] cmemk initialized
    [   29.251900] EXT4-fs (mmcblk0p4): re-mounted. Opts: (null)
    [   29.283438] Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
    [   30.361032] omap-rproc 58820000.ipu: assigned reserved memory node ipu1_cma@9d000000
    [   30.395956]  remoteproc0: 58820000.ipu is available
    [   30.407470]  remoteproc0: Note: remoteproc is still under development and considered experimental.
    [   30.420741]  remoteproc0: THE BINARY FORMAT IS NOT YET FINALIZED, and backward compatibility isn't yet guaranteed.
    [   30.434452] omap-rproc 55020000.ipu: assigned reserved memory node ipu2_cma@95800000
    [   30.442690]  remoteproc0: Direct firmware load for dra7-ipu1-fw.xem4 failed with error -2
    [   30.451337]  remoteproc0: failed to load dra7-ipu1-fw.xem4
    [   30.461832]  remoteproc1: 55020000.ipu is available
    [   30.471883]  remoteproc1: Note: remoteproc is still under development and considered experimental.
    [   30.493084]  remoteproc1: THE BINARY FORMAT IS NOT YET FINALIZED, and backward compatibility isn't yet guaranteed.
    [   30.526725]  remoteproc1: Direct firmware load for dra7-ipu2-fw.xem4 failed with error -2
    [   30.537036] omap-rproc 40800000.dsp: assigned reserved memory node dsp1_cma@99000000
    [   30.545195]  remoteproc1: failed to load dra7-ipu2-fw.xem4
    [   30.551260]  remoteproc2: 40800000.dsp is available
    [   30.557147]  remoteproc2: Note: remoteproc is still under development and considered experimental.
    [   30.567179]  remoteproc2: THE BINARY FORMAT IS NOT YET FINALIZED, and backward compatibility isn't yet guaranteed.
    [   30.580100]  remoteproc2: Direct firmware load for dra7-dsp1-fw.xe66 failed with error -2
    [   30.593172] omap_rng 48090000.rng: OMAP Random Number Generator ver. 20
    [   30.601044] omap-rproc 41000000.dsp: assigned reserved memory node dsp2_cma@9f000000
    [   30.611626]  remoteproc2: failed to load dra7-dsp1-fw.xe66
    [   30.620555]  remoteproc3: 41000000.dsp is available
    [   30.622144] of_get_named_gpiod_flags: parsed 'gpio-pdb' property of node '/ocp/i2c@4807a000/iblue@76[0]' - status (0)
    [   30.622162] of_get_named_gpiod_flags: parsed 'gpio-bisten' property of node '/ocp/i2c@4807a000/iblue@76[0]' - status (0)
    [   30.622178] of_get_named_gpiod_flags: parsed 'gpio-pwren' property of node '/ocp/i2c@4807a000/iblue@76[0]' - status (0)
    [   30.629113]  remoteproc3: Note: remoteproc is still under development and considered experimental.
    [   30.640223]  remoteproc3: THE BINARY FORMAT IS NOT YET FINALIZED, and backward compatibility isn't yet guaranteed.
    [   30.653638]  remoteproc3: Direct firmware load for dra7-dsp2-fw.xe66 failed with error -2
    [   30.666800]  remoteproc3: failed to load dra7-dsp2-fw.xe66
    [   30.705699] omap-des 480a5000.des: OMAP DES hw accel rev: 2.2
    [   30.723678] iblue 3-0076: Reg 0x00: 0xFFFFFF87
    [   30.729272] iblue 3-0076: Reg 0x01: 0xFFFFFF87
    [   30.734949] alg: skcipher: setkey failed on test 5 for ecb-des-omap: flags=100
    [   30.753759] alg: skcipher-ddst: setkey failed on test 5 for ecb-des-omap: flags=100
    [   30.754275] iblue 3-0076: Reg 0x02: 0xFFFFFF87
    [   30.766649] iblue 3-0076: Reg 0x03: 0xFFFFFF87
    [   30.773563] iblue 3-0076: Reg 0x04: 0xFFFFFF87
    [   30.779868] alg: skcipher-ddst: setkey failed on test 5 for ecb-des-omap: flags=100
    [   30.782998] iblue 3-0076: iblue 3-0076 sensor driver register.
    [   30.822931] rtc rtc1: 48838000.rtc: dev (254:1)
    [   30.822944] omap_rtc 48838000.rtc: rtc core: registered 48838000.rtc as rtc1
    [   30.920724] vip 48970000.vip: loading firmware vpdma-1b8.bin
    [   31.227073] vip 48970000.vip: VPDMA firmware loaded
    [   31.232325] vip1-s0: Port A: Using subdev iblue 3-0076 for capture
    [   31.239961] vip1-s0: device registered as video0
    [   31.244977] vip1-s1: Port A enabled but no endpoints found
    [   31.258546] SCSI subsystem initialized
    [   31.300575] omap_wdt: OMAP Watchdog Timer Rev 0x01: initial timeout 60 sec
    [   31.323122] omap-sham 4b101000.sham: hw accel on OMAP rev 4.3
    [   31.445399] omap-hdmi-audio omap-hdmi-audio.0.auto: snd-soc-dummy-dai <-> 58040000.encoder mapping ok
    [   31.494366] libata version 3.00 loaded.
    [   31.589202] (stk) :sysfs entries created
    [   31.599500] EXT4-fs (mmcblk0p3): mounted filesystem with ordered data mode. Opts: (null)
    [   31.624879] extcon-usb-gpio extcon_usb2: GPIO lookup for consumer id
    [   31.624891] extcon-usb-gpio extcon_usb2: using device tree for GPIO lookup
    [   31.624901] of_get_named_gpiod_flags: can't parse 'id-gpios' property of node '/extcon_usb2[0]'
    [   31.624909] of_get_named_gpiod_flags: can't parse 'id-gpio' property of node '/extcon_usb2[0]'
    [   31.624916] extcon-usb-gpio extcon_usb2: using lookup tables for GPIO lookup
    [   31.624925] extcon-usb-gpio extcon_usb2: lookup for GPIO id failed
    [   31.624933] extcon-usb-gpio extcon_usb2: GPIO lookup for consumer vbus
    [   31.624939] extcon-usb-gpio extcon_usb2: using device tree for GPIO lookup
    [   31.624947] of_get_named_gpiod_flags: can't parse 'vbus-gpios' property of node '/extcon_usb2[0]'
    [   31.625012] of_get_named_gpiod_flags: parsed 'vbus-gpio' property of node '/extcon_usb2[0]' - status (0)
    [   31.832335] omap-aes 4b500000.aes: OMAP AES hw accel rev: 3.3
    [   31.864968] omap-aes 4b700000.aes: OMAP AES hw accel rev: 3.3
    [   31.874645] [drm] Initialized pvr 1.14.3699939 20110701 on minor 1
    [   31.881939] ahci 4a140000.sata: forcing port_map 0x0 -> 0x1
    [   31.889458] ahci 4a140000.sata: AHCI 0001.0300 32 slots 1 ports 3 Gbps 0x1 impl platform mode
    [   31.899084] ahci 4a140000.sata: flags: 64bit ncq sntf pm led clo only pmp pio slum part ccc apst 
    [   31.911139] scsi host0: ahci
    [   31.914831] ata1: SATA max UDMA/133 mmio [mem 0x4a140000-0x4a1410ff] port 0x100 irq 345
    [   32.131121] mmcblk1: error -110 transferring data, sector 262016, nr 8, cmd response 0x900, card status 0xb00
    [   32.141819] mmcblk1: retrying using single block read
    [   32.230902] ata1: SATA link down (SStatus 0 SControl 300)
    [   32.495494] usbcore: registered new interface driver usbfs
    [   32.521393] usbcore: registered new interface driver hub
    [   32.521693] usbcore: registered new device driver usb
    [   32.554229] xhci-hcd xhci-hcd.1.auto: xHCI Host Controller
    [   32.560099] xhci-hcd xhci-hcd.1.auto: new USB bus registered, assigned bus number 1
    [   32.569403] xhci-hcd xhci-hcd.1.auto: hcc params 0x0220f04c hci version 0x100 quirks 0x00210010
    [   32.582342] xhci-hcd xhci-hcd.1.auto: irq 455, io mem 0x48890000
    [   32.589422] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
    [   32.600568] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
    [   32.610951] usb usb1: Product: xHCI Host Controller
    [   32.618812] usb usb1: Manufacturer: Linux 4.4.52-g0b07074fe6 xhci-hcd
    [   32.625676] usb usb1: SerialNumber: xhci-hcd.1.auto
    [   32.634432] hub 1-0:1.0: USB hub found
    [   32.641836] hub 1-0:1.0: 1 port detected
    [   32.647174] (stc):  chnl_id list empty :4 
    [   32.647174] (stk) : st_kim_start
    [   32.666573] xhci-hcd xhci-hcd.1.auto: xHCI Host Controller
    [   32.688653] xhci-hcd xhci-hcd.1.auto: new USB bus registered, assigned bus number 2
    [   32.756556] (stk) :ldisc_install = 1
    [   32.763526] usb usb2: We don't know the algorithms for LPM for this host, disabling LPM.
    [   32.778102] usb usb2: New USB device found, idVendor=1d6b, idProduct=0003
    [   32.785269] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
    [   32.793934] usb usb2: Product: xHCI Host Controller
    [   32.799061] usb usb2: Manufacturer: Linux 4.4.52-g0b07074fe6 xhci-hcd
    [   32.805895] usb usb2: SerialNumber: xhci-hcd.1.auto
    [   32.812541] hub 2-0:1.0: USB hub found
    [   32.820093] hub 2-0:1.0: 1 port detected
    [   32.839970] (stc): st_tty_open 
    [   32.843178] (stk) :line discipline installed
    [   32.849027] (stk) :ti-connectivity/TIInit_11.8.32.bts
    [   32.863915] (stk) :change remote baud rate command in firmware
    [   32.870042] (stk) :skipping the wait event of change remote baud
    [   32.927957] random: nonblocking pool is initialized
    [   33.071303] (stk) :sleep command in scr
    [   33.142763] EXT4-fs (mmcblk0p7): mounted filesystem with ordered data mode. Opts: (null)
    [   33.169200] EXT4-fs (mmcblk0p6): mounted filesystem with ordered data mode. Opts: (null)
    [   33.309487] systemd-journald[141]: Received request to flush runtime journal from PID 1
    [   33.545486] mmcblk1: error -110 transferring data, sector 262016, nr 8, cmd response 0x900, card status 0x0
    [   33.575822] blk_update_request: I/O error, dev mmcblk1, sector 262016
    [   33.584738] (stk) :sleep command in scr
    [   33.829112] cfg80211: World regulatory domain updated:
    [   33.836471] cfg80211:  DFS Master region: unset
    [   33.848871] cfg80211:   (start_freq - end_freq @ bandwidth), (max_antenna_gain, max_eirp), (dfs_cac_time)
    [   33.878889] cfg80211:   (2402000 KHz - 2472000 KHz @ 40000 KHz), (N/A, 2000 mBm), (N/A)
    [   33.912938] cfg80211:   (2457000 KHz - 2482000 KHz @ 40000 KHz), (N/A, 2000 mBm), (N/A)
    [   33.937898] cfg80211:   (2474000 KHz - 2494000 KHz @ 20000 KHz), (N/A, 2000 mBm), (N/A)
    [   33.983901] cfg80211:   (5170000 KHz - 5250000 KHz @ 80000 KHz, 160000 KHz AUTO), (N/A, 2000 mBm), (N/A)
    [   34.023884] cfg80211:   (5250000 KHz - 5330000 KHz @ 80000 KHz, 160000 KHz AUTO), (N/A, 2000 mBm), (0 s)
    [   34.054985] cfg80211:   (5490000 KHz - 5730000 KHz @ 160000 KHz), (N/A, 2000 mBm), (0 s)
    [   34.080551] cfg80211:   (5735000 KHz - 5835000 KHz @ 80000 KHz), (N/A, 2000 mBm), (N/A)
    [   34.102959] cfg80211:   (57240000 KHz - 63720000 KHz @ 2160000 KHz), (N/A, 0 mBm), (N/A)
    [   34.160933] wl18xx_driver wl18xx.2.auto: Direct firmware load for ti-connectivity/wl1271-nvs.bin failed with error -2
    [   34.486983] usb 2-1: new SuperSpeed USB device number 2 using xhci-hcd
    [   34.505207] usb 2-1: New USB device found, idVendor=0451, idProduct=8140
    [   34.514041] usb 2-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
    [   34.539713] hub 2-1:1.0: USB hub found
    [   34.543810] hub 2-1:1.0: 4 ports detected
    [   34.549215] wlcore: wl18xx HW: 183x or 180x, PG 2.2 (ROM 0x11)
    [   34.555351] usb 1-1: new high-speed USB device number 2 using xhci-hcd
    [   34.571996] wlcore: loaded
    [   34.577559] (stc): add_channel_to_table: id 4
    [   34.582176] (stc): add_channel_to_table: id 2
    [   34.586774] (stc): add_channel_to_table: id 3
    [   34.680298] PVR_K: UM DDK-(3699939) and KM DDK-(3699939) match. [ OK ]
    [   34.728294] usb 1-1: New USB device found, idVendor=0451, idProduct=8142
    [   34.745180] usb 1-1: New USB device strings: Mfr=0, Product=0, SerialNumber=1
    [   34.755620] usb 1-1: SerialNumber: 060718616BFB
    [   34.811913] hub 1-1:1.0: USB hub found
    [   34.817647] hub 1-1:1.0: 4 ports detected
    [   35.001285] mmcblk1: error -110 transferring data, sector 262017, nr 7, cmd response 0x900, card status 0x0
    [   35.013121] blk_update_request: I/O error, dev mmcblk1, sector 262017
    [   35.390597] EXT4-fs (mmcblk0p4): re-mounted. Opts: (null)
    [   35.503193] EXT4-fs (mmcblk0p4): re-mounted. Opts: (null)
    [   35.829403] c_can_platform 4ae3c000.can can0: setting BTR=1c09 BRPE=0000
    [   35.898029] c_can_platform 48480000.can can1: setting BTR=1c09 BRPE=0000
    [   36.148427] EXT4-fs (mmcblk0p4): re-mounted. Opts: (null)
    [   36.320624] EXT4-fs (mmcblk0p4): re-mounted. Opts: (null)
    [   36.418181] mmcblk1: error -110 transferring data, sector 262018, nr 6, cmd response 0x900, card status 0x0
    [   36.429934] blk_update_request: I/O error, dev mmcblk1, sector 262018
    [   37.836682] mmcblk1: error -110 transferring data, sector 262019, nr 5, cmd response 0x900, card status 0x0
    [   37.846953] blk_update_request: I/O error, dev mmcblk1, sector 262019
    [   39.253181] mmcblk1: error -110 transferring data, sector 262020, nr 4, cmd response 0x900, card status 0x0
    [   39.263429] blk_update_request: I/O error, dev mmcblk1, sector 262020
    [   39.430895] palmas-gpadc 48070000.i2c:tps659038@58:gpadc: conversion not completed
    [   39.438870] palmas-gpadc 48070000.i2c:tps659038@58:gpadc: ADC start conversion failed
    [   40.668579] mmcblk1: error -110 transferring data, sector 262021, nr 3, cmd response 0x900, card status 0x0
    [   40.678992] blk_update_request: I/O error, dev mmcblk1, sector 262021
    [   42.084026] mmcblk1: error -110 transferring data, sector 262022, nr 2, cmd response 0x900, card status 0x0
    [   42.094276] blk_update_request: I/O error, dev mmcblk1, sector 262022
    [   43.499309] mmcblk1: error -110 transferring data, sector 262023, nr 1, cmd response 0x900, card status 0x0
    [   43.509553] blk_update_request: I/O error, dev mmcblk1, sector 262023
    [   43.746142] wlcore: PHY firmware version: Rev 8.2.0.0.240
    [   43.861142] wlcore: firmware booted (Rev 8.9.0.0.75)
    [   43.902197] IPv6: ADDRCONF(NETDEV_UP): wlan0: link is not ready
    [   44.034459] (stc): remove_channel_from_table: id 3
    [   44.039553] (stc): remove_channel_from_table: id 2
    [   44.044641] (stc): remove_channel_from_table: id 4
    [   44.049758] (stc):  all chnl_ids unregistered 
    [   44.054273] (stk) :ldisc_install = 0
    [   44.058410] (stc): st_tty_close 
    [   44.915661] mmcblk1: error -110 transferring data, sector 262016, nr 8, cmd response 0x900, card status 0xb00
    [   44.926301] mmcblk1: retrying using single block read
    [   46.329961] mmcblk1: error -110 transferring data, sector 262016, nr 8, cmd response 0x900, card status 0x0
    [   46.340241] blk_update_request: I/O error, dev mmcblk1, sector 262016
    [   47.746226] mmcblk1: error -110 transferring data, sector 262017, nr 7, cmd response 0x900, card status 0x0
    [   47.756485] blk_update_request: I/O error, dev mmcblk1, sector 262017
    [   49.161537] mmcblk1: error -110 transferring data, sector 262018, nr 6, cmd response 0x900, card status 0x0
    [   49.171788] blk_update_request: I/O error, dev mmcblk1, sector 262018
    [   50.578198] mmcblk1: error -110 transferring data, sector 262019, nr 5, cmd response 0x900, card status 0x0
    [   50.588455] blk_update_request: I/O error, dev mmcblk1, sector 262019
    [   51.993643] mmcblk1: error -110 transferring data, sector 262020, nr 4, cmd response 0x900, card status 0x0
    [   52.003930] blk_update_request: I/O error, dev mmcblk1, sector 262020
    [   53.410478] mmcblk1: error -110 transferring data, sector 262021, nr 3, cmd response 0x900, card status 0x0
    [   53.420720] blk_update_request: I/O error, dev mmcblk1, sector 262021
    [   54.825986] mmcblk1: error -110 transferring data, sector 262022, nr 2, cmd response 0x900, card status 0x0
    [   54.836690] blk_update_request: I/O error, dev mmcblk1, sector 262022
    [   55.363318] gpio-189 (sysfs): gpiod_request: status -16
    [   55.363332] export_store: status -16
    [   55.364040] gpio-194 (sysfs): gpiod_request: status -16
    [   55.364051] export_store: status -16
    [   55.364742] gpio-193 (sysfs): gpiod_request: status -16
    [   55.364752] export_store: status -16
    [   55.388016] gpio-498 (sysfs): gpiod_request: status -16
    [   55.388031] export_store: status -16
    [   56.273046] mmcblk1: error -110 transferring data, sector 262023, nr 1, cmd response 0x900, card status 0x0
    [   56.283734] blk_update_request: I/O error, dev mmcblk1, sector 262023
    [   56.292864] Buffer I/O error on dev mmcblk1, logical block 32752, async page read
    [   56.320346] blk_update_request: I/O error, dev mmcblk1boot0, sector 3968
    [   56.329347] blk_update_request: I/O error, dev mmcblk1boot0, sector 3968
    [   56.336454] Buffer I/O error on dev mmcblk1boot0, logical block 496, async page read
    [   56.349643] blk_update_request: I/O error, dev mmcblk1boot1, sector 3968
    [   56.358142] blk_update_request: I/O error, dev mmcblk1boot1, sector 3968
    [   56.365176] Buffer I/O error on dev mmcblk1boot1, logical block 496, async page read
    root@raymond-gdm:~# 
    root@raymond-gdm:~# 
    root@raymond-gdm:~# [  OK  ] Stopped target Remote File Systems.
    [  OK  ] Stopped target Multi-User System.
    [  112.697959] wlcore: down
             Stopping WPA supplicant daemon (interface- and nl80211 driver-specific version)...
    [  OK  ] Stopped target Remote File Systems (Pre).
             Stopping Avahi mDNS/DNS-SD Stack...
             Stopping Raypower service...
             Stopping Telephony service...
    [  OK  ] Stopped target Sound Card.
             Stopping Kernel Logging Service...
             Stopping Periodic Command Scheduler...
    [  OK  ] Stopped Flush Journal to Persistent Storage.
             Stopping Login Service...
             Stopping System Logging Service...
             Stopping Camera LED service...
    [  OK  ] Stopped target Timers.
             Unmounting /mnt/data...
    [  OK  ] Stopped target Login Prompts.
    [  OK  ] Stopped Serial Getty on getty.
    [  OK  ] Stopped target Bluetooth.
             Stopping Serial Getty on ttyO5...
    [  OK  ] Stopped target Network.
             Stopping Connection service...
             Stopping Bluetooth service...
             [  112.789578] wlcore: down
    Stopping Serial Getty on ttyO2...
    [  OK  ] Stopped Periodic Command Scheduler.
    [  OK  ] Stopped Bluetooth service.
    [  OK  ] Stopped Serial Getty on ttyO2.
    [  OK  ] Stopped Avahi mDNS/DNS-SD Stack.
    [  OK  ] Stopped Kernel Logging Service.
    [  OK  ] Stopped Camera LED service.
    [  OK  ] Stopped Telephony service.
    [  OK  ] Stopped System Logging Service.
    [  OK  ] Stopped Login Service.
    [  OK  ] Stopped Serial Getty on ttyO5.
    [  OK  ] Stopped Raypower service.
    [  OK  ] Stopped Connection service.
    [  OK  ] Stopped WPA supplicant daemon (inte...l80211 driver-specific version).
    [  OK  ] Unmounted /mnt/data.
             Stopping User Mode Init Manager for TI shared transport...
             Stopping Raydrive service...
             Stopping D-Bus System Message Bus...
    [  OK  ] Removed slice system-serial\x2dgetty.slice.
    [  OK  ] Stopped D-Bus System Message Bus.
    [  OK  ] Stopped User Mode Init Manager for TI shared transport.
    [  OK  ] Stopped Raydrive service.
    [  OK  ] Stopped Runs raydrive.sh every hour and at powerup.
    [  OK  ] Stopped MGC System Update Bookkeeping.
    [  OK  ] Stopped MGC overlay service.
    [  OK  ] Stopped MGC Application Update Overlays.
             Unmounting /mnt/overlays...
    [  OK  ] Unmounted /mnt/overlays.
    [  OK  ] Stopped MGC First Boot System Initializer.
    [  OK  ] Stopped target Basic System.
    [  OK  ] Stopped target Slices.
    [  OK  ] Removed slice User and Session Slice.
    [  OK  ] Stopped target Sockets.
    [  OK  ] Closed Avahi mDNS/DNS-SD Stack Activation Socket.
    [  OK  ] Closed sshd.socket.
    [  OK  ] Closed RPCbind Server Activation Socket.
    [  OK  ] Closed Syslog Socket.
    [  OK  ] Closed D-Bus System Message Bus Socket.
    [  OK  ] Stopped target System Initialization.
    [  OK  ] Stopped Apply Kernel Variables.
    [  OK  ] Stopped Load Kernel Modules.
    [  OK  ] Stopped target Swap.
             Stopping Load/Save RF Kill Switch Status of rfkill1...
             Stopping Load/Save Screen Backlight...htness of backlight:backlight...
    [  OK  ] Stopped Setup Virtual Console.
    [  OK  ] Stopped target Paths.
    [  OK  ] Stopped Dispatch Password Requests to Console Directory Watch.
    [  OK  ] Stopped Forward Password Requests to Wall Directory Watch.
             Unmounting /mnt/boot...
    [  OK  ] Stopped Load/Save RF Kill Switch Status of rfkill1.
    [  OK  ] Stopped Load/Save Screen Backlight Brightness of backlight:backlight.
    [  OK  ] Unmounted /mnt/boot.
    [  OK  ] Stopped Create Volatile Files and Directories.
    [  OK  ] Stopped target Local File Systems.
             Unmounting /opt...                                                            +-----------------------------+
             Unmounting /etc/machine-id...                                                 |                             |
             Unmounting /lib/firmware...                                                   |  Cannot open /dev/ttyUSB0!  |
             Unmounting /var/lib/opkg...                                                   |                             |
             Unmounting /var/lib/connman...                                                +-----------------------------+
             Stopping Bind mount volatile /media...
             Unmounting /home/root...
             Unmounting /etc/resolv.conf...
             Unmounting /media...
             Unmounting /etc/pulse...
             Unmounting /tmp/data...
             Stopping Bind mount volatile /home/root...
             Stopping Bind mount volatile /etc/resolv.conf...
    [  OK  ] Removed slice system-systemd\x2dbacklight.slice.
    [  OK  ] Removed slice system-systemd\x2drfkill.slice.
    [  OK  ] Unmounted /opt.
    [  OK  ] Unmounted /etc/machine-id.
    [  OK  ] Unmounted /lib/firmware.
    [  OK  ] Unmounted /var/lib/opkg.
    [  OK  ] Unmounted /var/lib/connman.
    [  OK  ] Stopped Bind mount volatile /media.
    [  OK  ] Unmounted /home/root.
    [  OK  ] Unmounted /etc/resolv.conf.
    [  OK  ] Unmounted /media.
    [  OK  ] Unmounted /etc/pulse.
    [  OK  ] Unmounted /tmp/data.
    [  OK  ] Stopped Bind mount volatile /home/root.
    [  OK  ] Stopped Bind mount volatile /etc/resolv.conf.
             Unmounting Temporary Directory...
             Unmounting /var/volatile...
    [  OK  ] Unmounted Temporary Directory.
    [  OK  ] Unmounted /var/volatile.
    [  OK  ] Stopped target Local File Systems (Pre).
    [  OK  ] Stopped Remount Root and Kernel File Systems.
    [  OK  ] Stopped Create Static Device Nodes in /dev.
    [  OK  ] Reached target Shutdown.
    [  OK  ] Reached target Unmount All Filesystems.
    [  OK  ] Reached target Final Step.
             Starting Reboot...
    [  114.211115] watchdog watchdog0: watchdog did not stop!
    [  114.255164] systemd-journald[141]: Received SIGTERM from PID 1 (systemd-shutdow).
    [  114.291514] EXT4-fs (mmcblk0p4): re-mounted. Opts: (null)
    [  114.298195] EXT4-fs (mmcblk0p4): re-mounted. Opts: (null)
    [  114.334748] reboot: Restarting system 
    

    Furthermore, I modified the Uboot and enabled debug mode and MMC trace. I also increased the time-outs from 1 to 10 seconds and loaded that Uboot on an SD card and booted from it. I took 6 failed boards and 4 working boards and tried "mmc dev 1" command in Uboot to select the eMMC device. I tested this to check the communication between host controller and eMMC chip's controller. I found some odd behaviors in the failed boards. I have zipped all the outputs and attaching it below:

    eMMC_uboot_debug_06112018.zip

    Here is the directory tree for the ZIP:

    uboot_debug

             |--------------> Not_Working 

             |                           |-------------> 1,2,3,4_ODD,5,6 (THIS ARE FAILED BOARDS)

    |                                                           |--------------------> 1,2.3 (ATTEMPTS TO SELECT DEVICE ON THE PARTICULAR FAILED BOARD)

    |

    |---------------> Working

                                |-------------> 1,2.3,4 (THIS ARE WORKING BOARDS)

    |-------------------------> 1 (ATTEMPTS TO SELECT DEVICE ON THE PARTICULAR WORKING BOARD)

    If you see in this logs, for the failed boards, the eMMC takes more than 1 seconds to compete the "mmc dev 1" command. Sometimes, the eMMC never come to the operational state even after 10 second time-out. A few times, the eMMC does not respond with correct CSD and CID registers. Failed board #4 has some odd behavior on it, it did not respond to first try and then it did on the second try and I was able to boot from that eMMC. I tried again and then it stopped come to the operational mode.

    If you'd compare the logs from the failed boards to the working board, you'll have the idea of what's different. Another thing to notice here is, the chip's datasheet mentioned that OCR register will always respond 0x00FF8080 for capacity up to 2GB and 0x40FF8080 for capacity more than 2GB. We have an 8GB chip, yet in some failed boards the eMMC response with 0x00FF8080, and all the working boards response with 0x40FF8080.

    I hope this is a handful information to understand what's going on. I think the eMMC chip's controller registers are getting corrupted for some reason.

    Please let me know if you need more info.

    Thanks,

    Divyeshkumar

  • Can you take a board that fails, remove the Silicon Motion eMMC, and replace with a Micron one that worked?

    Steve K.
  • Hi Steve,

    We performed that test.

    We took the failed board, removed the Silicon Motion chip from it and replace it with a new Silicon Motion chip and the board started booting fine.

    To further investigate the data corruption, Silicon Motion cloned the data from the failed chip using their MMC debugger and reloaded the same data on the same eMMC chip and the chips started booting.

    Thanks,
    Divyeshkumar
  • Hi Steve,

    We are also facing similar issue where devices used to boot fine before and after a few power cycles, they just stop booting.
    The device gets stuck at
    "mmc1: error -110 whilst initialising MMC card "
    But once reset is pressed it boots normally. Sometimes we need to press reset twice or thrice to boot normally.

    Can this could be a timing issue? Can that be due to H/W?
    How can it be resolved?
  • Hi Steve,

    Do we have any updates?

    Thanks,

    Divyeshkumar

  • Hi Steve,

    With further investigation on our end, we found out that the Silicon Motion eMMC chips are getting damaged internally due to some electrical defect either on the mother board or the eMMC chip. Actual root cause has not been identified yet. Our electrical group headed the investigation further. I really appreciate all the support you showed for the issue. Thank you so much.

    BR,

    Divyeshkumar