DRA750: The file system of Samsung eMMC(HS200) is broken

Hi, Experts

SoC: DRA750

SDK VERSION: 3.2

Kernel version: Linux-4.4.84

Our customer has Samsung eMMC issue in their system based on DRA750. List below is the detail offered by our customer. Thank you for your help!

======================================================================================================================

We have received abnormal products and the error status is analyzed as bellow.

• Basic info of this issue:

Reproduce in desk/debug env: none

Reproduce in real products: 2%

Reproduce situation: system startup/or running.

Issue Details : the file-system cannot boot, because the key-files(showing in bellow) is lost or corrupted. But the kernel is boot.

• Basic info of our products:

SDK VERSION: 3.2

Kernel version: Linux-4.4.84

MMC driver modification by customer: none.

• Products information of KLM8G1GEUF-B04P-EMMC-8GB-Ver5.1-3.3V-8(as referred as Samsung emmc)
  • meta section (flash type SLC):
    • emmc frameware，
    • map tab: address map for logic and physical address
    • （user cannot access those sections）
  • User data section(flash type:mlc): we are using this section for kernel/rootfs storage.

• The mmc controller is working at HS200 mode but the I/O voltage is 3.3v which against specifications (6.6.2.2 section of Embedded Multi-Media Card (e•MMC) Electrical Standard (5.1), JESD84-B51).

• We have double checked with the mmc driver and found that the driver has been switch the mode to 1.8v with register 0x13c(the address offset 0x13c is from the source code, the actual address might be 0x3c. the register name is AC12) bit19(search from the source for AC12_V1V8_SIGEN). The question is, why the the voltage is not 1.8v?

• files of /opt/user/dms or /opt/user/lib/init (/opt is mounted with /dev/mmcblk0p2 and library & program is programmed there.) is broken. And the kernel error log is bellow.

[   36.242869] EXT4-fs error (device mmcblk0p2): ext4_mb_generate_buddy:758: group 1, block bitmap and bg descriptor inconsistent: 387 vs 390 free clusters

[   36.260867] EXT4-fs error (device mmcblk0p2): ext4_mb_generate_buddy:758: group 2, block bitmap and bg descriptor inconsistent: 8189 vs 8192 free clusters

[   36.285184] EXT4-fs error (device mmcblk0p2): ext4_mb_generate_buddy:758: group 3, block bitmap and bg descriptor inconsistent: 3776 vs 3779 free clusters

[   36.300090] EXT4-fs error (device mmcblk0p2): ext4_mb_generate_buddy:758: group 4, block bitmap and bg descriptor inconsistent: 1459 vs 1462 free clusters

[   36.315835] EXT4-fs error (device mmcblk0p2): ext4_mb_generate_buddy:758: group 5, block bitmap and bg descriptor inconsistent: 7752 vs 7755 free clusters

[   36.330445] EXT4-fs error (device mmcblk0p2): ext4_mb_generate_buddy:758: group 6, block bitmap and bg descriptor inconsistent: 8189 vs 8192 free clusters

[   36.344824] EXT4-fs error (device mmcblk0p2): ext4_mb_generate_buddy:758: group 7, block bitmap and bg descriptor inconsistent: 1565 vs 1568 free clusters

[   36.360756] EXT4-fs error (device mmcblk0p2): ext4_mb_generate_buddy:758: group 8, block bitmap and bg descriptor inconsistent: 917 vs 920 free clusters

[   36.375191] EXT4-fs error (device mmcblk0p2): ext4_mb_generate_buddy:758: group 9, block bitmap and bg descriptor inconsistent: 1786 vs 1789 free clusters

[   36.390123] EXT4-fs error (device mmcblk0p2): ext4_mb_generate_buddy:758: group 10, block bitmap and bg descriptor inconsistent: 877 vs 880 free clusters

 

[   56.762658] virtio_rpmsg_bus virtio2: msg received with no recipient

[[LPCan] destroySystemCan

[libcan] Release inUsd pool: 3

[   56.802772] virtio_rpmsg_bus virtio2: msg received with no recipient

[libcan] destroy CAN channel:90200 type:Physics endpoint:Client at pool[3] DONE !!![   56.810417] EXT4-fs error (device mmcblk0p3): ext4_mb_generate_buddy:758: group 1,

[libcan] destroySystemCan DONE !!!

block bitmap and bg descriptor inconsistent: 14509 vs 14510 free clusters

[   56.836192] JBD2: Spotted dirty metadata buffer (dev = mmcblk0p3, blocknr = 0). There's a risk of filesystem corruption in case of system crash.

Requesting system rebootesses

[   58.718159] reboot: Restarting system

• We tried to update the system and the kernel panic is showing as bellow:

 [  234.138843] Unable to handle kernel paging request at virtual address ffffffec

[  234.146104] pgd = ee195d80

[  234.148821] [ffffffec] *pgd=80000080007003, *pmd=affa6003, *pte=00000000

[  234.155578] Internal error: Oops: 207 [#1] PREEMPT SMP ARM

[  234.161085] Modules linked in:

[  234.164161] CPU: 0 PID: 1222 Comm: tar Not tainted 4.4.84+ #1

[  234.169929] Hardware name: Generic DRA74X (Flattened Device Tree)

[  234.176049] task: eee24980 ti: e8162000 task.ti: e8162000

[  234.181476] PC is at iommu_fault_handler+0x18/0x14c

[  234.186375] LR is at handle_irq_event_percpu+0xb4/0x160

[  234.191622] pc : [<c02a15ec>]    lr : [<c0072968>]    psr: a0030193

[  234.191622] sp : e8163b00  ip : e8163b40  fp : e8163b3c

[  234.203148] r10: c06c8b67  r9 : eee2ca80  r8 : 00000169

[  234.208394] r7 : 00000000  r6 : 00000000  r5 : ee5f9a10  r4 : ee61a980

[  234.214947] r3 : c02a15d4  r2 : c05b8cbc  r1 : ee5f9a10  r0 : 00000000

[  234.221501] Flags: NzCv  IRQs off  FIQs on  Mode SVC_32  ISA ARM  Segment user

[  234.228752] Control: 30c5387d  Table: ae195d80  DAC: fffffffd

[  234.234520] Process tar (pid: 1222, stack limit = 0xe8162210)

[  234.240289] Stack: (0xe8163b00 to 0xe8164000)

[  234.244665] 3b00: eee74f00 ffffffff eee249c8 337f0133 00000400 e8163b20 ee61a980 eee2cae0

[  234.252877] 3b20: 00000000 00000000 00000169 eee2ca80 e8163b84 e8163b40 c0072968 c02a15e0

[  234.261088] 3b40: e7cd5680 c06863a4 e8163bc4 c05b8cd0 c05b8cbc c05b8c94 c0063778 eee2ca80

[  234.269300] 3b60: eee2cae0 c06b0ca4 00000000 eec08000 00000000 000006b4 e8163ba4 e8163b88

[  234.277511] 3b80: c0072a54 c00728c0 eee2ca80 eee2cae0 c06b0ca4 00000000 e8163bc4 e8163ba8

[  234.285723] 3ba0: c0075d98 c0072a20 c068640c 00000169 00000000 00000000 e8163bd4 e8163bc8

[  234.293935] 3bc0: c0071f94 c0075ce4 e8163bfc e8163bd8 c007226c c0071f74 c068c7d4 fa21200c

[  234.302146] 3be0: e8163c28 fa212000 fa213000 00000000 e8163c24 e8163c00 c0009490 c0072214

[  234.310357] 3c00: c0189e1c a0030013 ffffffff e8163c5c 000a794c 00000000 e8163c8c e8163c28

[  234.318569] 3c20: c0013cc0 c000945c eef42000 0000148b 00000001 eef41c00 eef41800 00000002

[  234.326781] 3c40: 00000001 0000148b 000a794c 00000000 000006b4 e8163c8c e8163c90 e8163c78

[  234.334993] 3c60: c0189f48 c0189e1c a0030013 ffffffff 00000001 00000000 eef41800 00000002

[  234.343205] 3c80: e8163cb4 e8163c90 c0189f48 c0189df4 00001000 00000000 e7e8c0b0 e7e8c0b0

[  234.351416] 3ca0: 000a794c 00000000 e8163cd4 e8163cb8 c016c75c c0189f30 00000000 00001000

[  234.359629] 3cc0: 00000007 e7e8c0b0 e8163cfc e8163cd8 c01228f8 c016c730 e7e8c0b0 efeeda80

[  234.367839] 3ce0: 000a794c 00000000 000a794c 00000000 e8163d3c e8163d00 c012b7e8 c01228cc

[  234.376051] 3d00: 000006b4 000006b4 efeeda80 00000000 00000000 e7e8c0b0 00000000 000006b4

[  234.384262] 3d20: e7e8c17c ee84d800 e57cca80 000006b4 e8163d94 e8163d40 c016b5f8 c012b714

[  234.392473] 3d40: 000006b4 000006b4 efeeda80 00000000 ffeff000 ffefe000 000a8000 00000000

[  234.400684] 3d60: 000a794c 00000000 c0225e3c c016b4f0 e8163ef4 e7e8c17c 000006b4 0000094c

[  234.408896] 3d80: c04c3c9c 000006b4 e8163df4 e8163d98 c00b7ec4 c016b4fc 000006b4 000006b4

[  234.417108] 3da0: efeeda80 00000000 00000000 e8162000 00000000 e57cca80 000a794c 00000000

[  234.425319] 3dc0: efeeda80 00000000 000000ea 00000000 e57cca80 e8163f08 e7e8c17c e8163ef4

[  234.433530] 3de0: e7e8c118 00000000 e8163e44 e8163df8 c00b9b38 c00b7dac e8163e34 e8163e08

[  234.441742] 3e00: 000000ea 07a7dd4c 00000001 ee849148 e8163e60 00000000 ee614198 00000000

[  234.449954] 3e20: e8163f08 e7e8c0b0 e8163ef4 e57cca80 e7e8c118 00001000 e8163edc e8163e48

[  234.458166] 3e40: c0161070 c00b99a0 e8163e84 e8163e58 c01161a4 c00fe50c 000000ea 07a7dd4c

[  234.466378] 3e60: 000000ea 07a7dd4c 00001000 ee62ee30 eeff7d00 00000000 e8163edc 00000000

[  234.474589] 3e80: c0103f10 c01160fc 00020002 eeff7d18 ea40a780 e8162000 e8163ef4 c04bf918

[  234.482801] 3ea0: 00000000 00000000 00000006 ee62ee00 00001000 e57cca80 00000000 00001000

[  234.491012] 3ec0: e8163f80 000a794c 00000000 00000000 e8163f4c e8163ee0 c00fc138 c0160e40

[  234.499223] 3ee0: 00001000 c006a8cc ea40a780 bee17998 00001000 00000001 00000000 00001000

[  234.507434] 3f00: e8163eec 00000001 e57cca80 00000000 000a794c 00000000 00000000 e8163e7c

[  234.515647] 3f20: 00000000 00000000 e57cca80 00001000 bee17998 e8163f80 c000fac4 e8162000

[  234.523859] 3f40: e8163f7c e8163f50 c00fc924 c00fc090 00000000 00000000 e8163f7c e57cca80

[  234.532071] 3f60: e57cca80 bee17998 00001000 c000fac4 e8163fa4 e8163f80 c00fd194 c00fc898

[  234.540283] 3f80: 000a794c 00000000 00000000 0000004a bee17998 00000004 00000000 e8163fa8

[  234.548494] 3fa0: c000f920 c00fd154 00000000 0000004a 0000004a bee17998 00001000 ffffffff

[  234.556705] 3fc0: 00000000 0000004a bee17998 00000004 000c8e34 00000000 ffffffff 00001000

[  234.564917] 3fe0: 00000000 bee1795c 0001bac7 b6e33556 60030030 0000004a 8152900b 38299c89

[  234.573124] Backtrace:

[  234.575592] [<c02a15d4>] (iommu_fault_handler) from [<c0072968>] (handle_irq_event_percpu+0xb4/0x160)

[  234.584848]  r9:eee2ca80 r8:00000169 r7:00000000 r6:00000000 r5:eee2cae0 r4:ee61a980

[  234.592666] [<c00728b4>] (handle_irq_event_percpu) from [<c0072a54>] (handle_irq_event+0x40/0x64)

[  234.601573]  r10:000006b4 r9:00000000 r8:eec08000 r7:00000000 r6:c06b0ca4 r5:eee2cae0

[  234.609474]  r4:eee2ca80

[  234.612026] [<c0072a14>] (handle_irq_event) from [<c0075d98>] (handle_fasteoi_irq+0xc0/0x194)

[  234.620584]  r7:00000000 r6:c06b0ca4 r5:eee2cae0 r4:eee2ca80

[  234.626304] [<c0075cd8>] (handle_fasteoi_irq) from [<c0071f94>] (generic_handle_irq+0x2c/0x3c)

[  234.634949]  r7:00000000 r6:00000000 r5:00000169 r4:c068640c

[  234.640666] [<c0071f68>] (generic_handle_irq) from [<c007226c>] (__handle_domain_irq+0x64/0xbc)

[  234.649405] [<c0072208>] (__handle_domain_irq) from [<c0009490>] (gic_handle_irq+0x40/0x7c)

[  234.657789]  r9:00000000 r8:fa213000 r7:fa212000 r6:e8163c28 r5:fa21200c r4:c068c7d4

[  234.665604] [<c0009450>] (gic_handle_irq) from [<c0013cc0>] (__irq_svc+0x40/0x74)

[  234.673117] Exception stack(0xe8163c28 to 0xe8163c70)

[  234.678189] 3c20:                   eef42000 0000148b 00000001 eef41c00 eef41800 00000002

[  234.686400] 3c40: 00000001 0000148b 000a794c 00000000 000006b4 e8163c8c e8163c90 e8163c78

[  234.694611] 3c60: c0189f48 c0189e1c a0030013 ffffffff

[  234.699679]  r9:00000000 r8:000a794c r7:e8163c5c r6:ffffffff r5:a0030013 r4:c0189e1c

[  234.707502] [<c0189de8>] (ext4_journal_check_start) from [<c0189f48>] (__ext4_journal_start_sb+0x24/0x84)

[  234.717106]  r5:00000002 r4:eef41800

[  234.720716] [<c0189f24>] (__ext4_journal_start_sb) from [<c016c75c>] (ext4_dirty_inode+0x38/0x68)

[  234.729622]  r7:00000000 r6:000a794c r5:e7e8c0b0 r4:e7e8c0b0

[  234.735341] [<c016c724>] (ext4_dirty_inode) from [<c01228f8>] (__mark_inode_dirty+0x38/0x244)

[  234.743898]  r5:e7e8c0b0 r4:00000007

[  234.747504] [<c01228c0>] (__mark_inode_dirty) from [<c012b7e8>] (generic_write_end+0xe0/0x14c)

[  234.756150]  r9:00000000 r8:000a794c r7:00000000 r6:000a794c r5:efeeda80 r4:e7e8c0b0

[  234.763967] [<c012b708>] (generic_write_end) from [<c016b5f8>] (ext4_da_write_end+0x108/0x27c)

[  234.772612]  r10:000006b4 r9:e57cca80 r8:ee84d800 r7:e7e8c17c r6:000006b4 r5:00000000

[  234.780512]  r4:e7e8c0b0

[  234.783068] [<c016b4f0>] (ext4_da_write_end) from [<c00b7ec4>] (generic_perform_write+0x124/0x1bc)

[  234.792061]  r10:000006b4 r9:c04c3c9c r8:0000094c r7:000006b4 r6:e7e8c17c r5:e8163ef4

[  234.799962]  r4:c016b4f0

[  234.802515] [<c00b7da0>] (generic_perform_write) from [<c00b9b38>] (__generic_file_write_iter+0x1a4/0x200)

[  234.812206]  r10:00000000 r9:e7e8c118 r8:e8163ef4 r7:e7e8c17c r6:e8163f08 r5:e57cca80

[  234.820105]  r4:00000000

[  234.822659] [<c00b9994>] (__generic_file_write_iter) from [<c0161070>] (ext4_file_write_iter+0x23c/0x4f0)

[  234.832262]  r10:00001000 r9:e7e8c118 r8:e57cca80 r7:e8163ef4 r6:e7e8c0b0 r5:e8163f08

[  234.840161]  r4:00000000

[  234.842716] [<c0160e34>] (ext4_file_write_iter) from [<c00fc138>] (__vfs_write+0xb4/0xe4)

[  234.850925]  r10:00000000 r9:00000000 r8:000a794c r7:e8163f80 r6:00001000 r5:00000000

[  234.858824]  r4:e57cca80

[  234.861376] [<c00fc084>] (__vfs_write) from [<c00fc924>] (vfs_write+0x98/0x174)

[  234.868713]  r9:e8162000 r8:c000fac4 r7:e8163f80 r6:bee17998 r5:00001000 r4:e57cca80

[  234.876528] [<c00fc88c>] (vfs_write) from [<c00fd194>] (SyS_write+0x4c/0xa8)

[  234.883604]  r8:c000fac4 r7:00001000 r6:bee17998 r5:e57cca80 r4:e57cca80

[  234.890375] [<c00fd148>] (SyS_write) from [<c000f920>] (ret_fast_syscall+0x0/0x34)

[  234.897974]  r7:00000004 r6:bee17998 r5:0000004a r4:00000000

[  234.903690] Code: e24cb004 e24dd018 e1a05001 e5910010 (e5103014)

[  234.909811] ---[ end trace bdbbaabf06960aa5 ]---

[  234.914445] Kernel panic - not syncing: Fatal exception in interrupt

[  234.920834] CPU1: stopping

[  234.923559] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G      D         4.4.84+ #1

[  234.930812] Hardware name: Generic DRA74X (Flattened Device Tree)

[  234.936928] Backtrace:

[  234.939402] [<c0012fcc>] (dump_backtrace) from [<c00131c8>] (show_stack+0x18/0x1c)

[  234.947002]  r7:eec99f48 r6:20080193 r5:00000000 r4:c06b0964

[  234.952728] [<c00131b0>] (show_stack) from [<c0215fe8>] (dump_stack+0x8c/0xa0)

[  234.959986] [<c0215f5c>] (dump_stack) from [<c00155c0>] (handle_IPI+0x184/0x198)

[  234.967411]  r7:eec99f48 r6:00000000 r5:00000001 r4:c068640c

[  234.973128] [<c001543c>] (handle_IPI) from [<c00094c8>] (gic_handle_irq+0x78/0x7c)

[  234.980727]  r7:fa212000 r6:eec99f48 r5:fa21200c r4:c068c7d4

[  234.986441] [<c0009450>] (gic_handle_irq) from [<c0013cc0>] (__irq_svc+0x40/0x74)

[  234.993955] Exception stack(0xeec99f48 to 0xeec99f90)

[  234.999027] 9f40:                   00000000 ef338388 fe600000 00000000 eec98000 c068c4ac

[  235.007241] 9f60: c0685364 c04b8fa4 eec99fb8 c06c8b65 c068c4f8 eec99fa4 eec99f84 eec99f98

[  235.015452] 9f80: c0026300 c0010440 60080013 ffffffff

[  235.020521]  r9:c06c8b65 r8:eec99fb8 r7:eec99f7c r6:ffffffff r5:60080013 r4:c0010440

[  235.028342] [<c0010418>] (arch_cpu_idle) from [<c0069678>] (default_idle_call+0x28/0x34)

[  235.036470] [<c0069650>] (default_idle_call) from [<c00697a8>] (cpu_startup_entry+0x124/0x190)

[  235.045121] [<c0069684>] (cpu_startup_entry) from [<c00151a8>] (secondary_start_kernel+0x16c/0x178)

[  235.054203]  r7:c06ca288

[  235.056754] [<c001503c>] (secondary_start_kernel) from [<8000956c>] (0x8000956c)

[  235.064178]  r7:c06ca288 r6:30c0387d r5:00000000 r4:aec47380

[  235.069897] ---[ end Kernel panic - not syncing: Fatal exception in interrupt

• Aiming at the problems raised by the software department,the hardware department measured the clk and data waveforms of Samsung eMMC and Micron eMMC

• The clock frequency of Samsung eMMC is about 192MHz,and the clk and data waveforms are seriously distorted

• Clock waveform of Samsung eMMC

• Data waveform of Samsung eMMC

• Data waveform eye diagram of Samsung eMMC

• Clock waveform of Micron eMMC

• Clock waveform eye diagram of Micron eMMC

• Data waveform of Micron eMMC

• Data waveform eye diagram of Micron eMMC

• In the data sheet of Samsung eMMC,the maximum speed of 3V I/O port is 52MHz

• Description of bus speed mode of DRA750

• HS200 mode selection flow diagram

• The bus speed mode set by the software is HS200 mode,eMMC will first detect the level of Vccq pin to determine whether HS200 mode is selected.Because the I/O port uses 3.3V level,it cannot meet the operating conditions of HS200 mode.However,the clock speed of DRA750 and Samsung eMMC is automatically negotiated to 192MHz.

1. Why the HS200 mode is still negotiated when the operation conditions are not met