This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

SK-AM62: Cloning SD to eMMC - running CQE recovery - Experimentation!

Part Number: SK-AM62
Other Parts Discussed in Thread: AM625

I did some experimentations to see if it's possible to clone my SD root fs to the eMMC root fs.

Why? Because I needed some software that do not exist on the base image for eMMC, as the default image did not work for my EVM.

Good news: It's possible.

Bad news: A single non-essential folder is causing issues.

After the copy process runs for a while and gets to the folder /usr/bin/, it suddenly fails with "running CQE recovery".

After some digging, I found out that the specific file/folder it fails at is /usr/bin/SGX, which seems to be non-essential.

By excluding that folder, pretty much all other files and folder can be copied.

TI might need to have a look at why this is happening, as I suspect that this may cause issues elsewhere as well.

Here is the error report:

[ 307.010385] ------------[ cut here ]------------
[ 307.010400] mmc0: cqhci: spurious TCN for tag 29
[ 307.010467] WARNING: CPU: 0 PID: 171 at drivers/mmc/host/cqhci.c:742 cqhci_irq+0x318/0x4a0
[ 307.010494] Modules linked in: xt_conntrack xt_addrtype iptable_filter br_netfilter bridge stp llc overlay xfrm_user xfrm_algo md5 des_generic libdes cbc iptable_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_tables x_tables usb_storage wl18xx wlcore xhci_plat_hcd rpmsg_char xhci_hcd mac80211 usbcore cfg80211 rfkill libarc4 dwc3 udc_core usb_common cdns_csi2rx v4l2_fwnode pru_rproc irq_pruss_intc crct10dif_ce snd_soc_simple_card snd_soc_simple_card_utils wlcore_sdio ti_k3_r5_remoteproc pvrsrvkm(O) virtio_rpmsg_bus dwc3_am62 ti_k3_m4_remoteproc rti_wdt tps6598x sa2ul j721e_csi2rx sha512_generic videobuf2_dma_contig typec mcrc authenc videobuf2_memops snd_soc_tlv320aic3x roles videobuf2_v4l2 cdns_dphy pruss videobuf2_common sch_fq_codel cryptodev(O) ipv6
[ 307.010676] CPU: 0 PID: 171 Comm: irq/33-mmc0 Tainted: G O 5.10.168-rt83-gc1a1291911 #1
[ 307.010684] Hardware name: Texas Instruments AM625 SK (DT)
[ 307.010689] pstate: 40000005 (nZcv daif -PAN -UAO -TCO BTYPE=--)
[ 307.010696] pc : cqhci_irq+0x318/0x4a0
[ 307.010704] lr : cqhci_irq+0x318/0x4a0
[ 307.010710] sp : ffff80001215bc20
[ 307.010713] x29: ffff80001215bc20 x28: ffff00000008b600
[ 307.010719] x27: ffff000001e58280 x26: 0000000000000001
[ 307.010726] x25: ffff800010e2fe28 x24: ffff000001e58298
[ 307.010732] x23: ffff8000111e81eb x22: ffff000001e40f00
[ 307.010739] x21: ffff00000008b000 x20: 0000000000000002
[ 307.010745] x19: 000000000000001d x18: 0000000000000001
[ 307.010751] x17: ffff000003404080 x16: 0000000000000000
[ 307.010757] x15: ffff000001e41460 x14: ffffffffffffffff
[ 307.010764] x13: ffff800011210716 x12: ffff800011210712
[ 307.010770] x11: 0000000000000000 x10: ffff8000110fbb88
[ 307.010776] x9 : 00000000fffffffe x8 : 000000000000001d
[ 307.010782] x7 : 6972757073203a69 x6 : ffff80001215ba70
[ 307.010788] x5 : ffff00007fb91b38 x4 : 0000000000000000
[ 307.010795] x3 : 0000000000000027 x2 : 0000000100000000
[ 307.010800] x1 : a18286a2f8620300 x0 : 0000000000000000
[ 307.010808] Call trace:
[ 307.010811] cqhci_irq+0x318/0x4a0
[ 307.010818] sdhci_am654_cqhci_irq+0x58/0x88
[ 307.010826] sdhci_irq+0xb0/0xe88
[ 307.010831] irq_forced_thread_fn+0x3c/0xd0
[ 307.010841] irq_thread+0x194/0x2b8
[ 307.010848] kthread+0x174/0x198
[ 307.010856] ret_from_fork+0x10/0x30
[ 307.010866] ---[ end trace 0000000000000002 ]---
[ 308.011083] sched: RT throttling activated
[ 308.052258] mmc0: running CQE recovery
[ 308.119928] mmc0: running CQE recovery
[ 308.128711] mmc0: running CQE recovery
[ 308.137441] mmc0: running CQE recovery
[ 308.146787] mmc0: running CQE recovery
[ 308.151439] blk_update_request: I/O error, dev mmcblk0, sector 7905280 op 0x1:(WRITE) flags 0x4000 phys_seg 12 prio class 0
[ 308.155088] mmc0: running CQE recovery
[ 308.159686] mmc0: running CQE recovery
[ 308.164370] mmc0: running CQE recovery
[ 308.165412] blk_update_request: I/O error, dev mmcblk0, sector 7906304 op 0x1:(WRITE) flags 0x4000 phys_seg 12 prio class 0
[ 308.165431] EXT4-fs warning (device mmcblk0p1): ext4_end_bio:347: I/O error 10 writing to inode 263539 starting block 988416)
[ 308.165506] Buffer I/O error on device mmcblk0p1, logical block 987136
[ 308.165525] Buffer I/O error on device mmcblk0p1, logical block 987137
[ 308.165532] Buffer I/O error on device mmcblk0p1, logical block 987138
[ 308.165538] Buffer I/O error on device mmcblk0p1, logical block 987139
[ 308.165545] Buffer I/O error on device mmcblk0p1, logical block 987140
[ 308.165552] Buffer I/O error on device mmcblk0p1, logical block 987141
[ 308.165559] Buffer I/O error on device mmcblk0p1, logical block 987142
[ 308.165565] Buffer I/O error on device mmcblk0p1, logical block 987143
[ 308.165571] Buffer I/O error on device mmcblk0p1, logical block 987144
[ 308.165577] Buffer I/O error on device mmcblk0p1, logical block 987145
[ 308.167416] mmc0: running CQE recovery
[ 308.194730] mmc0: running CQE recovery
[ 308.205534] mmc0: running CQE recovery
[ 308.220503] mmc0: running CQE recovery
[ 308.227335] mmc0: running CQE recovery
[ 308.232723] mmc0: running CQE recovery
[ 369.702527] mmc0: cqhci: timeout for tag 0
[ 369.702553] mmc0: cqhci: ============ CQHCI REGISTER DUMP ===========
[ 369.702558] mmc0: cqhci: Caps: 0x000030c8 | Version: 0x00000510
[ 369.702562] mmc0: cqhci: Config: 0x00000101 | Control: 0x00000000
[ 369.702566] mmc0: cqhci: Int stat: 0x00000000 | Int enab: 0x00000006
[ 369.702570] mmc0: cqhci: Int sig: 0x00000006 | Int Coal: 0x00000000
[ 369.702575] mmc0: cqhci: TDL base: 0x8039f000 | TDL up32: 0x00000000
[ 369.702579] mmc0: cqhci: Doorbell: 0xffffffff | TCN: 0x00000000
[ 369.702583] mmc0: cqhci: Dev queue: 0x00000000 | Dev Pend: 0x00000000
[ 369.702587] mmc0: cqhci: Task clr: 0x00000000 | SSC1: 0x00011000
[ 369.702592] mmc0: cqhci: SSC2: 0x00000001 | DCMD rsp: 0x00000000
[ 369.702596] mmc0: cqhci: RED mask: 0xfdf9a080 | TERRI: 0x992f1a2c
[ 369.702600] mmc0: cqhci: Resp idx: 0x0000002f | Resp arg: 0x00000900
[ 369.702605] mmc0: sdhci: ============ SDHCI REGISTER DUMP ===========
[ 369.702608] mmc0: sdhci: Sys addr: 0x00000400 | Version: 0x00001004
[ 369.702612] mmc0: sdhci: Blk size: 0x00007080 | Blk cnt: 0x00000000
[ 369.702617] mmc0: sdhci: Argument: 0x00000000 | Trn mode: 0x00000013
[ 369.702621] mmc0: sdhci: Present: 0x01ff00f0 | Host ctl: 0x0000003c
[ 369.702625] mmc0: sdhci: Power: 0x0000000f | Blk gap: 0x00000080
[ 369.702629] mmc0: sdhci: Wake-up: 0x00000000 | Clock: 0x00000007
[ 369.702633] mmc0: sdhci: Timeout: 0x0000000e | Int stat: 0x00000000
[ 369.702637] mmc0: sdhci: Int enab: 0x02ff4000 | Sig enab: 0x02ff4000
[ 369.702642] mmc0: sdhci: ACmd stat: 0x00000000 | Slot int: 0x00000000
[ 369.702646] mmc0: sdhci: Caps: 0x3decc801 | Caps_1: 0x18002407
[ 369.702650] mmc0: sdhci: Cmd: 0x0000153a | Max curr: 0x00000000
[ 369.702655] mmc0: sdhci: Resp[0]: 0x00000900 | Resp[1]: 0xffc003ff
[ 369.702660] mmc0: sdhci: Resp[2]: 0x328f5903 | Resp[3]: 0x00d07f01
[ 369.702664] mmc0: sdhci: Host ctl2: 0x0000000b
[ 369.702667] mmc0: sdhci: ADMA Err: 0x00000000 | ADMA Ptr: 0x0000000081d3920c
[ 369.702672] mmc0: sdhci: ============================================
[ 369.702739] mmc0: running CQE recovery
[ 369.706310] mmc0: running CQE recovery
[ 369.723708] mmc0: running CQE recovery
[ 369.731967] mmc0: running CQE recovery
[ 369.739242] mmc0: running CQE recovery
[ 369.744389] mmc0: running CQE recovery
[ 369.750655] mmc0: running CQE recovery
[ 369.755435] blk_update_request: I/O error, dev mmcblk0, sector 7921664 op 0x1:(WRITE) flags 0x4000 phys_seg 57 prio class 0
[ 369.757609] mmc0: running CQE recovery
[ 431.144140] mmc0: cqhci: timeout for tag 0
[ 431.144161] mmc0: cqhci: ============ CQHCI REGISTER DUMP ===========
[ 431.144166] mmc0: cqhci: Caps: 0x000030c8 | Version: 0x00000510
[ 431.144171] mmc0: cqhci: Config: 0x00000101 | Control: 0x00000000
[ 431.144175] mmc0: cqhci: Int stat: 0x00000000 | Int enab: 0x00000006
[ 431.144180] mmc0: cqhci: Int sig: 0x00000006 | Int Coal: 0x00000000
[ 431.144184] mmc0: cqhci: TDL base: 0x8039f000 | TDL up32: 0x00000000
[ 431.144188] mmc0: cqhci: Doorbell: 0xffffffff | TCN: 0x00000000
[ 431.144192] mmc0: cqhci: Dev queue: 0x00000000 | Dev Pend: 0x00000000
[ 431.144196] mmc0: cqhci: Task clr: 0x00000000 | SSC1: 0x00011000
[ 431.144201] mmc0: cqhci: SSC2: 0x00000001 | DCMD rsp: 0x00000000
[ 431.144205] mmc0: cqhci: RED mask: 0xfdf9a080 | TERRI: 0x972f1a2c
[ 431.144209] mmc0: cqhci: Resp idx: 0x0000002d | Resp arg: 0x00000d00
[ 431.144214] mmc0: sdhci: ============ SDHCI REGISTER DUMP ===========
[ 431.144218] mmc0: sdhci: Sys addr: 0x00000400 | Version: 0x00001004
[ 431.144221] mmc0: sdhci: Blk size: 0x00007080 | Blk cnt: 0x00000000
[ 431.144226] mmc0: sdhci: Argument: 0x00000000 | Trn mode: 0x00000013
[ 431.144230] mmc0: sdhci: Present: 0x01ff00f0 | Host ctl: 0x0000003c
[ 431.144234] mmc0: sdhci: Power: 0x0000000f | Blk gap: 0x00000080
[ 431.144238] mmc0: sdhci: Wake-up: 0x00000000 | Clock: 0x00000007
[ 431.144242] mmc0: sdhci: Timeout: 0x0000000e | Int stat: 0x00000000
[ 431.144246] mmc0: sdhci: Int enab: 0x02ff4000 | Sig enab: 0x02ff4000
[ 431.144251] mmc0: sdhci: ACmd stat: 0x00000000 | Slot int: 0x00000000
[ 431.144255] mmc0: sdhci: Caps: 0x3decc801 | Caps_1: 0x18002407
[ 431.144259] mmc0: sdhci: Cmd: 0x0000153a | Max curr: 0x00000000
[ 431.144264] mmc0: sdhci: Resp[0]: 0x00000900 | Resp[1]: 0xffc003ff
[ 431.144268] mmc0: sdhci: Resp[2]: 0x328f5903 | Resp[3]: 0x00d07f01
[ 431.144272] mmc0: sdhci: Host ctl2: 0x0000000b
[ 431.144276] mmc0: sdhci: ADMA Err: 0x00000000 | ADMA Ptr: 0x0000000081d3920c
[ 431.144281] mmc0: sdhci: ============================================
[ 431.144352] mmc0: running CQE recovery
[ 431.150646] mmc0: running CQE recovery
[ 431.159309] mmc0: running CQE recovery
[ 431.179801] mmc0: running CQE recovery
[ 431.189564] mmc0: running CQE recovery
[ 431.197302] mmc0: running CQE recovery
[ 431.202457] blk_update_request: I/O error, dev mmcblk0, sector 7923712 op 0x1:(WRITE) flags 0x0 phys_seg 26 prio class 0
[ 431.202480] EXT4-fs warning (device mmcblk0p1): ext4_end_bio:347: I/O error 10 writing to inode 263542 starting block 990569)
[ 431.211906] mmc0: running CQE recovery
[ 431.214005] blk_update_request: I/O error, dev mmcblk0, sector 7897088 op 0x1:(WRITE) flags 0x4000 phys_seg 18 prio class 0
[ 431.218905] mmc0: running CQE recovery
[ 431.231286] mmc0: running CQE recovery
[ 431.236904] mmc0: running CQE recovery
[ 431.247441] mmc0: running CQE recovery
[ 431.249066] blk_update_request: I/O error, dev mmcblk0, sector 7891968 op 0x1:(WRITE) flags 0x0 phys_seg 29 prio class 0
[ 431.259243] mmc0: running CQE recovery
[ 431.274697] mmc0: running CQE recovery
[ 431.279502] blk_update_request: I/O error, dev mmcblk0, sector 7872512 op 0x1:(WRITE) flags 0x4000 phys_seg 51 prio class 0
[ 431.280906] mmc0: running CQE recovery
[ 431.287251] mmc0: running CQE recovery
[ 431.289749] mmc0: running CQE recovery
[ 431.291546] mmc0: running CQE recovery
[ 431.293275] mmc0: running CQE recovery
[ 492.585752] mmc0: cqhci: timeout for tag 0
[ 492.585778] mmc0: cqhci: ============ CQHCI REGISTER DUMP ===========
[ 492.585783] mmc0: cqhci: Caps: 0x000030c8 | Version: 0x00000510
[ 492.585787] mmc0: cqhci: Config: 0x00000101 | Control: 0x00000000
[ 492.585792] mmc0: cqhci: Int stat: 0x00000000 | Int enab: 0x00000006
[ 492.585796] mmc0: cqhci: Int sig: 0x00000006 | Int Coal: 0x00000000
[ 492.585800] mmc0: cqhci: TDL base: 0x8039f000 | TDL up32: 0x00000000
[ 492.585805] mmc0: cqhci: Doorbell: 0xffffffff | TCN: 0x00000000
[ 492.585809] mmc0: cqhci: Dev queue: 0x00000000 | Dev Pend: 0x00000000
[ 492.585813] mmc0: cqhci: Task clr: 0x00000000 | SSC1: 0x00011000
[ 492.585817] mmc0: cqhci: SSC2: 0x00000001 | DCMD rsp: 0x00000000
[ 492.585821] mmc0: cqhci: RED mask: 0xfdf9a080 | TERRI: 0x992f1a2c
[ 492.585825] mmc0: cqhci: Resp idx: 0x0000002f | Resp arg: 0x00000900
[ 492.585830] mmc0: sdhci: ============ SDHCI REGISTER DUMP ===========
[ 492.585834] mmc0: sdhci: Sys addr: 0x000000f8 | Version: 0x00001004
[ 492.585838] mmc0: sdhci: Blk size: 0x00007080 | Blk cnt: 0x00000000
[ 492.585842] mmc0: sdhci: Argument: 0x00000000 | Trn mode: 0x00000013
[ 492.585846] mmc0: sdhci: Present: 0x01ff00f0 | Host ctl: 0x0000003c
[ 492.585850] mmc0: sdhci: Power: 0x0000000f | Blk gap: 0x00000080
[ 492.585854] mmc0: sdhci: Wake-up: 0x00000000 | Clock: 0x00000007
[ 492.585858] mmc0: sdhci: Timeout: 0x0000000e | Int stat: 0x00000000
[ 492.585863] mmc0: sdhci: Int enab: 0x02ff4000 | Sig enab: 0x02ff4000
[ 492.585867] mmc0: sdhci: ACmd stat: 0x00000000 | Slot int: 0x00000000
[ 492.585871] mmc0: sdhci: Caps: 0x3decc801 | Caps_1: 0x18002407
[ 492.585876] mmc0: sdhci: Cmd: 0x0000153a | Max curr: 0x00000000
[ 492.585880] mmc0: sdhci: Resp[0]: 0x00000900 | Resp[1]: 0xffc003ff
[ 492.585884] mmc0: sdhci: Resp[2]: 0x328f5903 | Resp[3]: 0x00d07f01
[ 492.585888] mmc0: sdhci: Host ctl2: 0x0000000b
[ 492.585892] mmc0: sdhci: ADMA Err: 0x00000000 | ADMA Ptr: 0x0000000081d3920c
[ 492.585897] mmc0: sdhci: ============================================
[ 492.585966] mmc0: running CQE recovery
[ 492.595157] mmc0: running CQE recovery
[ 492.601432] blk_update_request: I/O error, dev mmcblk0, sector 7930880 op 0x1:(WRITE) flags 0x4000 phys_seg 36 prio class 0
[ 492.612344] mmc0: running CQE recovery
[ 492.618655] mmc0: running CQE recovery
[ 492.619710] mmc0: running CQE recovery
[ 492.624589] blk_update_request: I/O error, dev mmcblk0, sector 7917568 op 0x1:(WRITE) flags 0x4000 phys_seg 66 prio class 0
[ 492.624990] mmc0: running CQE recovery
[ 492.626599] mmc0: running CQE recovery
[ 492.632042] mmc0: running CQE recovery
[ 492.634112] mmc0: running CQE recovery
[ 492.639436] blk_update_request: I/O error, dev mmcblk0, sector 7880704 op 0x1:(WRITE) flags 0x4000 phys_seg 57 prio class 0
[ 492.640897] mmc0: running CQE recovery
[ 492.646480] mmc0: running CQE recovery
[ 492.653432] mmc0: running CQE recovery

  • After some digging, I found out that the specific file/folder it fails at is /usr/bin/SGX, which seems to be non-essential.

    This looks like your system has issues reading the data from this specific folder. You can isolate the issue by removing the "copy to eMMC" from the equation, and just try to recursively read & discard all files in that subfolder, like this:

    root@am62xx-evm:/usr/bin# grep -r dummy-str1ng SGX/
    root@am62xx-evm:/usr/bin#

    If all goes well it should look like my code snippet. However I suspect you'll encounter the same SD card read issues, even with that simply "dummy read folder recursively" command. If that's the case please confirm your SDK version. And if not SDK v8.6, please try the latest AM62x SDK (currently v8.6).

    Regards, Andreas

  • Hi Andreas, Thanks for the tip.

    I ran the suggested command while booted to the SD card.

    The error did NOT happen.

    It is an eMMC issue and not an SD card issue.

    How do I find out which SDK version I have installed on the SD card?

  • It is an eMMC issue and not an SD card issue.

    Yeah the 'mmc0' errors in your log would also align with this theory.

    How do I find out which SDK version I have installed on the SD card?

    The easiest might be to just to re-program the image based on the latest SDK downloads, and thensee if you can re-create the issues with that.

    Regards, Andreas