[AM335x] System Crash issue after removing sdcard during data writing to SDCard

yimin bai1

Expert 1495 points

Other Parts Discussed in Thread: AM3352

Hi Sir

We met system crashed issue by doing below steps in AM335x customized board

Software: SDK 6.0 (kernel 3.2)

1. booting from MMC0 interface

2. insert SDcard into the MMC2 interface of AM3352 MB.

3. write some files into sdcard

4. we remove SDcard from MB during data writing into SDcard.

5. system crashed and showed below message

root@CRV:#

root@CRV:# /sdcard/# umount /sdcard/;mount -o sync /dev/mmcblk1p1 /sdcard/;cp android.rar

[ 213.895972] MMC irq err status:0x00208002

[ 213.900452] mmcblk1: error -84 transferring data, sector 723, nr 1, cmd response 0x900, card status 0x0

[ 213.930785] MMC irq err status:0x00018000

[ 215.016175] end_request: I/O error, dev mmcblk1, sector 723

[ 215.022054] Buffer I/O error on device mmcblk1p1, logical block 722

[ 215.028648] lost page write due to I/O error on mmcblk1p1

[ 215.061118] MMC irq err status:0x00018000

[ 215.080633] MMC irq err status:0x00018000

[ 215.084940] mmcblk1: error -110 sending status command, retrying

[ 215.091317] MMC irq err status:0x00018000

[ 215.095718] mmcblk1: error -110 sending status command, retrying

[ 215.102094] MMC irq err status:0x00018000

[ 215.116536] mmcblk1: error -110 sending status command, aborting

[ 215.122871] ****** mmc_blk_cmd_recovery: entered if loop

[ 215.147311] ****** mmc_blk_cmd_recovery: return ERR_ABORT

[ 215.153026] end_request: I/O error, dev mmcblk1, sector 723

[ 215.159124] FAT-fs (mmcblk1p1): FAT read failed (blocknr 722)

[ 215.184313] MMC irq err status:0x20018000

[ 236.124100] BUG: soft lockup - CPU#0 stuck for 22s! [mmcqd/1:1622]

[ 236.130606] Modules linked in: m25p80

[ 236.134473]

[ 236.136040] Pid: 1622, comm: mmcqd/1

[ 236.140996] CPU: 0 Not tainted (3.2.0-gef4c37c-dirty #379)

[ 236.147151] PC is at __do_softirq+0x60/0x158

[ 236.151647] LR is at __do_softirq+0x48/0x158

[ 236.156143] pc : [<c0046b94>] lr : [<c0046b7c>] psr: 20000113

[ 236.156151] sp : cf613d18 ip : cf613d18 fp : cf613d54

[ 236.168230] r10: 00000000 r9 : ce37b80c r8 : 00000000

[ 236.173728] r7 : c064a804 r6 : 00000000 r5 : 00000044 r4 : 00000002

[ 236.180599] r3 : cf613d18 r2 : 00000000 r1 : ffffffe1 r0 : 00000000

[ 236.187473] Flags: nzCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment kernel

[ 236.195168] Control: 10c5387d Table: 8e408019 DAC: 00000015

[ 236.201212] Backtrace:

[ 236.203599]

[ 236.205375] [<c00181dc>] (dump_backtrace+0x0/0x10c) from [<c042aeb8>] (dump_stack+0x18/0x1c)

[ 236.214257] r6:00000016 r5:c05fae28 r4:cf613cd0 r3:00000002

[ 236.220250] [<c042aea0>] (dump_stack+0x0/0x1c) from [<c0015fd0>] (show_regs+0x50/0x54)

[ 236.228602] [<c0015f80>] (show_regs+0x0/0x54) from [<c0078138>] (watchdog_timer_fn+0x114/0x14c)

[ 236.237756] r4:c05fae20 r3:00000002

[ 236.241549] [<c0078024>] (watchdog_timer_fn+0x0/0x14c) from [<c005fa30>] (__run_hrtimer.isra.21+0x60/0xf4)

[ 236.251723] [<c005f9d0>] (__run_hrtimer.isra.21+0x0/0xf4) from [<c00602cc>] (hrtimer_interrupt+0xf8/0x27c)

[ 236.261885] r6:c05fa5e0 r5:00000036 r4:f9d9527c r3:00000036

[ 236.267884] [<c00601d4>] (hrtimer_interrupt+0x0/0x27c) from [<c0022d84>] (omap2_gp_timer_interrupt+0x34/0x3c)

[ 236.278335] [<c0022d50>] (omap2_gp_timer_interrupt+0x0/0x3c) from [<c0078ab4>] (handle_irq_event_percpu+0x54/0x1c0)

[ 236.289334] [<c0078a60>] (handle_irq_event_percpu+0x0/0x1c0) from [<c0078c78>] (handle_irq_event+0x58/0x80)

[ 236.299604] [<c0078c20>] (handle_irq_event+0x0/0x80) from [<c007ad34>] (handle_level_irq+0x90/0x12c)

[ 236.309218] r5:00000044 r4:c05fcc20

[ 236.313002] [<c007aca4>] (handle_level_irq+0x0/0x12c) from [<c0078394>] (generic_handle_irq+0x2c/0x40)

[ 236.322799] r4:c061393c r3:c007aca4

[ 236.326596] [<c0078368>] (generic_handle_irq+0x0/0x40) from [<c001599c>] (handle_IRQ+0x38/0x8c)

[ 236.335765] [<c0015964>] (handle_IRQ+0x0/0x8c) from [<c00087b8>] (omap3_intc_handle_irq+0x80/0x88)

[ 236.345195] r6:fa200000 r5:cf613cd0 r4:00000004 r3:00000044

[ 236.351188] [<c0008738>] (omap3_intc_handle_irq+0x0/0x88) from [<c00146c0>] (__irq_svc+0x40/0x70)

[ 236.360529] Exception stack(0xcf613cd0 to 0xcf613d18)

[ 236.365850] 3cc0: 00000000 ffffffe1 00000000 cf613d18

[ 236.374464] 3ce0: 00000002 00000044 00000000 c064a804 00000000 ce37b80c 00000000 cf613d54

Did anyone meet above issue before ? please advise and thanks for your help

over 10 years ago

0 Ivan Matrakov over 10 years ago

TI__Expert 4475 points

Hi yimin,

Are you sure that the /dev/mmcblk1p1 is sdcard. Usually this device is eMMC.

BR
Ivan

0 yimin bai1 over 10 years ago in reply to Ivan Matrakov

Expert 1495 points

Hi Ivan

Yes.

BTW, attached file is the detailed log message

Fullscreen 2350.CardRemove.txt Download

[  380.493611] card move
[  408.061923] Backtrace: 
[  408.064564] [<c00181dc>] (dump_backtrace+0x0/0x10c) from [<c042c068>] (dump_stack+0x18/0x1c)
[  408.073452]  r6:00000017 r5:c05fce28 r4:c05fce20 r3:00000000
[  408.079458] [<c042c050>] (dump_stack+0x0/0x1c) from [<c00780f8>] (watchdog_timer_fn+0xd4/0x150)
[  408.088635] [<c0078024>] (watchdog_timer_fn+0x0/0x150) from [<c005fa30>] (__run_hrtimer.isra.21+0x60/0xf4)
[  408.098812] [<c005f9d0>] (__run_hrtimer.isra.21+0x0/0xf4) from [<c00602cc>] (hrtimer_interrupt+0xf8/0x27c)
[  408.108974]  r6:c05fc5e0 r5:0000005f r4:02458025 r3:0000005f
[  408.114975] [<c00601d4>] (hrtimer_interrupt+0x0/0x27c) from [<c0022d84>] (omap2_gp_timer_interrupt+0x34/0x3c)
[  408.125428] [<c0022d50>] (omap2_gp_timer_interrupt+0x0/0x3c) from [<c0078ab8>] (handle_irq_event_percpu+0x54/0x1c0)
[  408.136427] [<c0078a64>] (handle_irq_event_percpu+0x0/0x1c0) from [<c0078c7c>] (handle_irq_event+0x58/0x80)
[  408.146697] [<c0078c24>] (handle_irq_event+0x0/0x80) from [<c007ad38>] (handle_level_irq+0x90/0x12c)
[  408.156313]  r5:00000044 r4:c05fec20
[  408.160096] [<c007aca8>] (handle_level_irq+0x0/0x12c) from [<c0078398>] (generic_handle_irq+0x2c/0x40)
[  408.169895]  r4:c061593c r3:c007aca8
[  408.173693] [<c007836c>] (generic_handle_irq+0x0/0x40) from [<c001599c>] (handle_IRQ+0x38/0x8c)
[  408.182863] [<c0015964>] (handle_IRQ+0x0/0x8c) from [<c00087b8>] (omap3_intc_handle_irq+0x80/0x88)
[  408.192294]  r6:fa200000 r5:cf6c5c20 r4:00000004 r3:00000044
[  408.198289] [<c0008738>] (omap3_intc_handle_irq+0x0/0x88) from [<c00146c0>] (__irq_svc+0x40/0x70)
[  408.207633] Exception stack(0xcf6c5c20 to 0xcf6c5c68)
[  408.212958] 5c20: 00000000 fffffde1 00000000 cf6c5c68 00000202 00000044 00000000 c064ce44
[  408.221574] 5c40: 00000000 cf43640c 00000000 cf6c5ca4 cf6c5c68 cf6c5c68 c0046b7c c0046b94
[  408.230182] 5c60: 20000113 ffffffff
[  408.233851]  r7:cf6c5c54 r6:ffffffff r5:20000113 r4:c0046b94
[  408.239846] [<c0046b34>] (__do_softirq+0x0/0x158) from [<c00470c8>] (irq_exit+0x94/0x9c)
[  408.248371] [<c0047034>] (irq_exit+0x0/0x9c) from [<c00159a0>] (handle_IRQ+0x3c/0x8c)
[  408.256614]  r4:c061593c r3:c007aca8
[  408.260396] [<c0015964>] (handle_IRQ+0x0/0x8c) from [<c00087b8>] (omap3_intc_handle_irq+0x80/0x88)
[  408.269824]  r6:fa200000 r5:cf6c5d00 r4:00000004 r3:00000044
[  408.275818] [<c0008738>] (omap3_intc_handle_irq+0x0/0x88) from [<c00146c0>] (__irq_svc+0x40/0x70)
[  408.285159] Exception stack(0xcf6c5d00 to 0xcf6c5d48)
[  408.290483] 5d00: cf6c5d70 cf6c5d68 cf6c5d68 00000001 cf6c5d70 cf6c5d68 c0016bbc 00000002
[  408.299100] 5d20: 00000000 cf43640c 00000000 cf6c5d64 cf6c5d48 cf6c5d48 c0017fcc c0017fd8
[  408.307711] 5d40: 60000013 ffffffff
[  408.311380]  r7:cf6c5d34 r6:ffffffff r5:60000013 r4:c0017fd8
[  408.317374] [<c0017fa8>] (walk_stackframe+0x0/0x40) from [<c0016c28>] (return_address+0x3c/0x5c)
[  408.326623]  r6:c05fa678 r5:00000001 r4:cf6c4000 r3:c0016bec
[  408.332616] [<c0016bec>] (return_address+0x0/0x5c) from [<c0038a74>] (add_preempt_count+0x94/0xd0)
[  408.342062] [<c00389e0>] (add_preempt_count+0x0/0xd0) from [<c042f838>] (__schedule+0x30/0x3f0)
[  408.351219]  r5:cf6c4000 r4:c05fa678
[  408.355004] [<c042f808>] (__schedule+0x0/0x3f0) from [<c04300ec>] (schedule+0x50/0x68)
[  408.363351] [<c043009c>] (schedule+0x0/0x68) from [<c0430520>] (schedule_timeout+0x174/0x1e8)
[  408.372337] [<c04303ac>] (schedule_timeout+0x0/0x1e8) from [<c042fe04>] (wait_for_common+0x10c/0x194)
[  408.382055] [<c042fcf8>] (wait_for_common+0x0/0x194) from [<c042fea4>] (wait_for_completion+0x18/0x1c)
[  408.391869] [<c042fe8c>] (wait_for_completion+0x0/0x1c) from [<c02df0b4>] (mmc_wait_for_req_done+0x80/0xcc)
[  408.402136] [<c02df034>] (mmc_wait_for_req_done+0x0/0xcc) from [<c02dfd94>] (mmc_start_req+0x5c/0x128)
[  408.411950] [<c02dfd38>] (mmc_start_req+0x0/0x128) from [<c02ed094>] (mmc_blk_issue_rw_rq+0x9c/0x4c4)
[  408.421668] [<c02ecff8>] (mmc_blk_issue_rw_rq+0x0/0x4c4) from [<c02ed794>] (mmc_blk_issue_rq+0x2d8/0x524)
[  408.431751] [<c02ed4bc>] (mmc_blk_issue_rq+0x0/0x524) from [<c02edd80>] (mmc_queue_thread+0x60/0x11c)
[  408.441481] [<c02edd20>] (mmc_queue_thread+0x0/0x11c) from [<c005bb40>] (kthread+0x8c/0x94)
[  408.450294] [<c005bab4>] (kthread+0x0/0x94) from [<c004460c>] (do_exit+0x0/0x6dc)
[  408.458171]  r6:c004460c r5:c005bab4 r4:cf035cc4
[  408.463055] BUG: soft lockup - CPU#0 stuck for 23s! [mmcqd/1:1969]
[  408.469559] Modules linked in: m25p80
[  408.473423] 
[  408.474989] Pid: 1969, comm:              mmcqd/1
[  408.479943] CPU: 0    Not tainted  (3.2.0-g47e50a4-dirty #301)
[  408.486083] PC is at __do_softirq+0x60/0x158
[  408.490578] LR is at __do_softirq+0x48/0x158
[  408.495076] pc : [<c0046b94>]    lr : [<c0046b7c>]    psr: 20000113
[  408.495084] sp : cf6c5c68  ip : cf6c5c68  fp : cf6c5ca4
[  408.507164] r10: 00000000  r9 : cf43640c  r8 : 00000000
[  408.512663] r7 : c064ce44  r6 : 00000000  r5 : 00000044  r4 : 00000202
[  408.519534] r3 : cf6c5c68  r2 : 00000000  r1 : fffffde1  r0 : 00000000
[  408.526407] Flags: nzCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment kernel
[  408.534102] Control: 10c5387d  Table: 8f54c019  DAC: 00000015
[  408.540146] Backtrace: 
[  408.542734] [<c00181dc>] (dump_backtrace+0x0/0x10c) from [<c042c068>] (dump_stack+0x18/0x1c)
[  408.551616]  r6:00000017 r5:c05fce28 r4:cf6c5c20 r3:00000002
[  408.557609] [<c042c050>] (dump_stack+0x0/0x1c) from [<c0015fd0>] (show_regs+0x50/0x54)
[  408.565956] [<c0015f80>] (show_regs+0x0/0x54) from [<c007813c>] (watchdog_timer_fn+0x118/0x150)
[  408.575114]  r4:c05fce20 r3:00000002
[  408.578900] [<c0078024>] (watchdog_timer_fn+0x0/0x150) from [<c005fa30>] (__run_hrtimer.isra.21+0x60/0xf4)
[  408.589076] [<c005f9d0>] (__run_hrtimer.isra.21+0x0/0xf4) from [<c00602cc>] (hrtimer_interrupt+0xf8/0x27c)
[  408.599240]  r6:c05fc5e0 r5:0000005f r4:02458025 r3:0000005f
[  408.605236] [<c00601d4>] (hrtimer_interrupt+0x0/0x27c) from [<c0022d84>] (omap2_gp_timer_interrupt+0x34/0x3c)
[  408.615686] [<c0022d50>] (omap2_gp_timer_interrupt+0x0/0x3c) from [<c0078ab8>] (handle_irq_event_percpu+0x54/0x1c0)
[  408.626683] [<c0078a64>] (handle_irq_event_percpu+0x0/0x1c0) from [<c0078c7c>] (handle_irq_event+0x58/0x80)
[  408.636949] [<c0078c24>] (handle_irq_event+0x0/0x80) from [<c007ad38>] (handle_level_irq+0x90/0x12c)
[  408.646561]  r5:00000044 r4:c05fec20
[  408.650345] [<c007aca8>] (handle_level_irq+0x0/0x12c) from [<c0078398>] (generic_handle_irq+0x2c/0x40)
[  408.660139]  r4:c061593c r3:c007aca8
[  408.663923] [<c007836c>] (generic_handle_irq+0x0/0x40) from [<c001599c>] (handle_IRQ+0x38/0x8c)
[  408.673088] [<c0015964>] (handle_IRQ+0x0/0x8c) from [<c00087b8>] (omap3_intc_handle_irq+0x80/0x88)
[  408.682515]  r6:fa200000 r5:cf6c5c20 r4:00000004 r3:00000044
[  408.688505] [<c0008738>] (omap3_intc_handle_irq+0x0/0x88) from [<c00146c0>] (__irq_svc+0x40/0x70)
[  408.697844] Exception stack(0xcf6c5c20 to 0xcf6c5c68)
[  408.703166] 5c20: 00000000 fffffde1 00000000 cf6c5c68 00000202 00000044 00000000 c064ce44
[  408.711779] 5c40: 00000000 cf43640c 00000000 cf6c5ca4 cf6c5c68 cf6c5c68 c0046b7c c0046b94
[  408.720387] 5c60: 20000113 ffffffff
[  408.724056]  r7:cf6c5c54 r6:ffffffff r5:20000113 r4:c0046b94
[  408.730046] [<c0046b34>] (__do_softirq+0x0/0x158) from [<c00470c8>] (irq_exit+0x94/0x9c)
[  408.738571] [<c0047034>] (irq_exit+0x0/0x9c) from [<c00159a0>] (handle_IRQ+0x3c/0x8c)
[  408.746812]  r4:c061593c r3:c007aca8
[  408.750595] [<c0015964>] (handle_IRQ+0x0/0x8c) from [<c00087b8>] (omap3_intc_handle_irq+0x80/0x88)
[  408.760023]  r6:fa200000 r5:cf6c5d00 r4:00000004 r3:00000044
[  408.766015] [<c0008738>] (omap3_intc_handle_irq+0x0/0x88) from [<c00146c0>] (__irq_svc+0x40/0x70)
[  408.775354] Exception stack(0xcf6c5d00 to 0xcf6c5d48)
[  408.780677] 5d00: cf6c5d70 cf6c5d68 cf6c5d68 00000001 cf6c5d70 cf6c5d68 c0016bbc 00000002
[  408.789290] 5d20: 00000000 cf43640c 00000000 cf6c5d64 cf6c5d48 cf6c5d48 c0017fcc c0017fd8
[  408.797897] 5d40: 60000013 ffffffff
[  408.801564]  r7:cf6c5d34 r6:ffffffff r5:60000013 r4:c0017fd8
[  408.807555] [<c0017fa8>] (walk_stackframe+0x0/0x40) from [<c0016c28>] (return_address+0x3c/0x5c)
[  408.816803]  r6:c05fa678 r5:00000001 r4:cf6c4000 r3:c0016bec
[  408.822794] [<c0016bec>] (return_address+0x0/0x5c) from [<c0038a74>] (add_preempt_count+0x94/0xd0)
[  408.832234] [<c00389e0>] (add_preempt_count+0x0/0xd0) from [<c042f838>] (__schedule+0x30/0x3f0)
[  408.841391]  r5:cf6c4000 r4:c05fa678
[  408.845176] [<c042f808>] (__schedule+0x0/0x3f0) from [<c04300ec>] (schedule+0x50/0x68)
[  408.853521] [<c043009c>] (schedule+0x0/0x68) from [<c0430520>] (schedule_timeout+0x174/0x1e8)
[  408.862508] [<c04303ac>] (schedule_timeout+0x0/0x1e8) from [<c042fe04>] (wait_for_common+0x10c/0x194)
[  408.872225] [<c042fcf8>] (wait_for_common+0x0/0x194) from [<c042fea4>] (wait_for_completion+0x18/0x1c)
[  408.882035] [<c042fe8c>] (wait_for_completion+0x0/0x1c) from [<c02df0b4>] (mmc_wait_for_req_done+0x80/0xcc) ***********************
[  408.892301] [<c02df034>] (mmc_wait_for_req_done+0x0/0xcc) from [<c02dfd94>] (mmc_start_req+0x5c/0x128)
[  408.902109] [<c02dfd38>] (mmc_start_req+0x0/0x128) from [<c02ed094>] (mmc_blk_issue_rw_rq+0x9c/0x4c4)
[  408.911825] [<c02ecff8>] (mmc_blk_issue_rw_rq+0x0/0x4c4) from [<c02ed794>] (mmc_blk_issue_rq+0x2d8/0x524)
[  408.921907] [<c02ed4bc>] (mmc_blk_issue_rq+0x0/0x524) from [<c02edd80>] (mmc_queue_thread+0x60/0x11c)
[  408.931623] [<c02edd20>] (mmc_queue_thread+0x0/0x11c) from [<c005bb40>] (kthread+0x8c/0x94)
[  408.940422] [<c005bab4>] (kthread+0x0/0x94) from [<c004460c>] (do_exit+0x0/0x6dc)
[  408.948296]  r6:c004460c r5:c005bab4 r4:cf035cc4

If you have any suggestion, please advise.

0 yimin bai1 over 10 years ago in reply to yimin bai1

Expert 1495 points

Hi Sir

We added some print message in mmc_blk_cmd_recovery function of drivers/mmc/card/block.c

/* We couldn't get a response from the card. Give up. */

if (err) {

printk("****** mmc_blk_cmd_recovery: entered if loop\n");

/* Check if the card is removed */

if (mmc_detect_card_removed(card->host))

{

printk("****** mmc_blk_cmd_recovery: return ERR_NOMEDIUM\n");

return ERR_NOMEDIUM;

}

printk("****** mmc_blk_cmd_recovery: return ERR_ABORT\n");

return ERR_ABORT;

}

1. System works well after removing sdcard and the message shows as below

root@CRV:# /sdcard/# umount /sdcard/;mount -o sync /dev/mmcblk1p1 /sdcard/;cp android.rar

[ 114.103183] omap_hsmmc omap_hsmmc.2: IRQ Status is 1

[ 119.818189] MMC irq err status:0x00208002

[ 120.903986] MMC irq err status:0x00018000

[ 120.908287] mmcblk1: error -110 sending status command, retrying

[ 120.923102] MMC irq err status:0x00018000

[ 120.927433] mmcblk1: error -110 sending status command, retrying

[ 120.933815] MMC irq err status:0x00018000

[ 120.938222] mmcblk1: error -110 sending status command, aborting

[ 120.953359] ****** mmc_blk_cmd_recovery: entered if loop

[ 120.967629] MMC irq err status:0x00018000

[ 120.971954] MMC irq err status:0x00018000

[ 120.976419] MMC irq err status:0x00018000

[ 120.980726] MMC irq err status:0x00018000[

[ 120.985025] ****** mmc_blk_cmd_recovery: return ERR_NOMEDIUM

cp: write error: Input/output error

[ 120.998884] mmc1: card e624 removed

2. System will crash after removing sdcard and the message shows as below

[ 215.116536] mmcblk1: error -110 sending status command, aborting

[ 215.122871] ****** mmc_blk_cmd_recovery: entered if loop

[ 215.147311] ****** mmc_blk_cmd_recovery: return ERR_ABORT

[ 215.153026] end_request: I/O error, dev mmcblk1, sector 723

[ 215.159124] FAT-fs (mmcblk1p1): FAT read failed (blocknr 722)

[ 215.184313] MMC irq err status:0x20018000

[ 236.124100] BUG: soft lockup - CPU#0 stuck for 22s! [mmcqd/1:1622]

[ 236.130606] Modules linked in: m25p80

After checking , this issue will happen with below conditions

1. mmc_detect_card_removed function returns ERR_ABORT after removing sdcard.

2. mmc1: card e624 removed message does not show up

Please advise and thanks in advance.

Processors

Processors forum

[AM335x] System Crash issue after removing sdcard during data writing to SDCard