• State Resolved
  • Locked Locked
  • Replies 6 replies
  • Subscribers 98 subscribers
  • Views 346 views
  • Users 0 members are here
Support feedback
Options
Options
Related

This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

AM623: Crashing when unloading/loading camera modules

Lyndon Amsdon
Prodigy 75 points
Part Number: AM623
Other Parts Discussed in Thread: SK-AM62

Setup: Default Version 09.00.00.03 SDK Image booting from SDCARD on SK-AM62 EVM and Digilent Pcam5 OV5640 Image sensor. Device Overlay for camera is loaded via u-boot.

I wish to unload/load the camera driver module ov5640.ko - which isn't possible (says module in use) unless the j721e_csi2rx is unloaded first. 

rmmod j721e_csi2rx
rmmod ov5640
insmod /lib/modules/6.1.33-g40c32565ca/kernel/drivers/media/i2c/ov5640.ko
insmod /lib/modules/6.1.33-g40c32565ca/kernel/drivers/media/platform/ti/j721e-csi2rx/j721e-csi2rx.ko

Is there some other camera related modules that should be unloaded/loaded. Is there a specific order this needs to be done in? It doesn't seem to happen on the first unload/load, only the second attempt.

These are the crashes, whch sometimes vary:

****

Nothing - complete lock up of the UART console

*****

[ 189.858267] ti-sci 44043000.system-controller: Mbox timedout in resp(caller: sci_clk_determine_rate+0x64/0xe4)
[ 189.868325] ti-sci 44043000.system-controller: Mbox send fail -110
[ 189.874528] ti-sci-clk 44043000.system-controller:clock-controller: determine-rate failed for dev=135, clk=0, ret=-110
[ 190.914251] ti-sci 44043000.system-controller: Mbox timedout in resp(caller: sci_clk_determine_rate+0x64/0xe4)
[ 190.924278] ti-sci 44043000.system-controller: Mbox send fail -110
[ 190.930464] ti-sci-clk 44043000.system-controller:clock-controller: determine-rate failed for dev=135, clk=0, ret=-110
[ 191.970250] ti-sci 44043000.system-controller: Mbox timedout in resp(caller: sci_clk_determine_rate+0x64/0xe4)
[ 191.980272] ti-sci 44043000.system-controller: Mbox send fail -110
[ 191.986457] ti-sci-clk 44043000.system-controller:clock-controller: determine-rate failed for dev=135, clk=0, ret=-110
[ 191.997151] cpu cpu0: _opp_config_clk_single: failed to set clock rate: -22
[ 192.004115] cpufreq: __target_index: Failed to change cpu frequency: -22
[ 193.026261] ti-sci 44043000.system-controller: Mbox timedout in resp(caller: sci_clk_determine_rate+0x64/0xe4)
[ 193.036291] ti-sci 44043000.system-controller: Mbox send fail -110
[ 193.042476] ti-sci-clk 44043000.system-controller:clock-controller: determine-rate failed for dev=135, clk=0, ret=-110
[ 194.082251] ti-sci 44043000.system-controller: Mbox timedout in resp(caller: sci_clk_determine_rate+0x64/0xe4)
[ 194.092272] ti-sci 44043000.system-controller: Mbox send fail -110
[ 194.098456] ti-sci-clk 44043000.system-controller:clock-controller: determine-rate failed for dev=135, clk=0, ret=-110
[ 195.138250] ti-sci 44043000.system-controller: Mbox timedout in resp(caller: sci_clk_determine_rate+0x64/0xe4)
[ 195.148270] ti-sci 44043000.system-controller: Mbox send fail -110
[ 195.154459] ti-sci-clk 44043000.system-controller:clock-controller: determine-rate failed for dev=135, clk=0, ret=-110
[ 195.165152] cpu cpu0: _opp_config_clk_single: failed to set clock rate: -22
[ 195.172114] cpufreq: __target_index: Failed to change cpu frequency: -22


*****


[ 2735.658621] ti-sci 44043000.system-controller: Mbox timedout in resp(caller: ti_sci_cmd_put_device+0x18/0x24)
[ 2735.668611] ti-sci 44043000.system-controller: Mbox send fail -110
[ 2755.246589] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 2755.252707] rcu: 0-...0: (1 GPs behind) idle=5d4c/1/0x4000000000000000 softirq=44506/44507 fqs=1049
[ 2755.261833] rcu: 2-...0: (2 GPs behind) idle=c36c/1/0x4000000000000000 softirq=35186/35187 fqs=1049
[ 2755.270956] rcu: 3-...0: (8 ticks this GP) idle=6a24/1/0x4000000000000000 softirq=46240/46241 fqs=1049
[ 2755.280338] rcu: Tasks blocked on level-0 rcu_node (CPUs 0-3): P143/1:b..l
[ 2755.287305] (detected by 1, t=5252 jiffies, g=102293, q=105 ncpus=4)
[ 2755.293735] Task dump for CPU 0:
[ 2755.296955] task:insmod state:R running task stack:0 pid:6448 ppid:5487 flags:0x00000206
[ 2755.306863] Call trace:
[ 2755.309304] __switch_to+0xe4/0x160
[ 2755.312807] lock_timer_base+0x64/0xa4
[ 2755.316557] del_timer_sync+0x4c/0xd0
[ 2755.320215] schedule_timeout+0x88/0xf0
[ 2755.324051] wait_for_completion_timeout+0x74/0x144
[ 2755.328925] tx_tick+0x44/0xa4
[ 2755.331982] Task dump for CPU 2:
[ 2755.335200] task:systemd-udevd state:R running task stack:0 pid:194 ppid:1 flags:0x00000a02
[ 2755.345106] Call trace:
[ 2755.347544] __switch_to+0xe4/0x160
[ 2755.351030] 0xffff000004abb900
[ 2755.354168] Task dump for CPU 3:
[ 2755.357386] task:kworker/3:1H state:R running task stack:0 pid:137 ppid:2 flags:0x0000000a
[ 2755.367295] Workqueue: kblockd blk_mq_run_work_fn
[ 2755.372000] Call trace:
[ 2755.374438] __switch_to+0xe4/0x160
[ 2755.377924] __blk_mq_run_hw_queue+0x80/0xbc
[ 2755.382189] blk_mq_run_work_fn+0x20/0x30
[ 2755.386194] process_one_work+0x1d0/0x320
[ 2755.390199] worker_thread+0x14c/0x444
[ 2755.393942] kthread+0x10c/0x110
[ 2755.397170] ret_from_fork+0x10/0x20
[ 2755.400745] task:jbd2/mmcblk1p2- state:R running task stack:0 pid:143 ppid:2 flags:0x00000008
[ 2755.410649] Call trace:
[ 2755.413087] __switch_to+0xe4/0x160
[ 2755.416573] __schedule+0x22c/0x660
[ 2755.420058] preempt_schedule_notrace+0x4c/0x70
[ 2755.424583] blk_mq_sched_insert_requests+0x104/0x250
[ 2755.429634] blk_mq_flush_plug_list+0xd8/0x240
[ 2755.434074] __blk_flush_plug+0xf0/0x140
[ 2755.437996] blk_finish_plug+0x38/0xf0
[ 2755.441742] jbd2_journal_commit_transaction+0xa2c/0x1710
[ 2755.447137] kjournald2+0xb8/0x254
[ 2755.450538] kthread+0x10c/0x110
[ 2755.453765] ret_from_fork+0x10/0x20

*****

  • 0
    TI__Guru 59125 points

    Hello Lyndon,

    Lyndon Amsdon said:
    Is there some other camera related modules that should be unloaded/loaded.

    There is another module in the camera capture path: kernel/drivers/media/platform/cadence/cdns-csi2rx.ko.

    Please try the following order:

      rmmod j721e_csi2rx
  rmmod cdns_csi2rx
  rmmod ov5640
  insmod /lib/modules/6.1.33-g40c32565ca/kernel/drivers/media/i2c/ov5640.ko
  insmod /lib/modules/6.1.33-g40c32565ca/kernel/drivers/media/platform/cadence/cdns-csi2rx.ko
  insmod /lib/modules/6.1.33-g40c32565ca/kernel/drivers/media/platform/ti/j721e-csi2rx/j721e-csi2rx.ko

    Regards,

    Jianzhong

  • 0 in reply to Jianzhong Xu
    Prodigy 75 points

    The suggestion seems to work, but if you have used the camera using yavta with this command: 

    yavta -c5 -Fcapture -s 640x480 -f UYVY /dev/video0

    Then it will fail when loading in the cdns-csi2rx module:


    root@am62xx-evm:~# insmod /lib/modules/6.1.33-g40c32565ca/kernel/drivers/media/platform/cadence/cdns-csi2rx.ko

    <Hangs/Crash - no response on debug uart console>

  • 0 in reply to Lyndon Amsdon
    TI__Guru 59125 points

    Hi Lyndon,

    Not sure if I understood you. Did you mean yavta was hanging? If that's the case, that must be caused by the low power mode of the system.

    Please try the following and then run yavta:

    root@am62xx-evm:~# echo on > /sys/devices/platform/bus@f0000/20020000.i2c/i2c-2/i2c-4/4-003c/power/control

    Regards,

    Jianzhong

  • 0 in reply to Jianzhong Xu
    Prodigy 75 points

    Hi,

    The hang/crash happened when reloading cdns-csi2rx.ko. Try the following commands below and it will hang.

      rmmod j721e_csi2rx
  rmmod cdns_csi2rx
  rmmod ov5640
  insmod /lib/modules/6.1.33-g40c32565ca/kernel/drivers/media/i2c/ov5640.ko
  insmod /lib/modules/6.1.33-g40c32565ca/kernel/drivers/media/platform/cadence/cdns-csi2rx.ko
  insmod /lib/modules/6.1.33-g40c32565ca/kernel/drivers/media/platform/ti/j721e-csi2rx/j721e-csi2rx.ko
  
  yavta -c5 -Fcapture -s 640x480 -f UYVY /dev/video0
  
  rmmod j721e_csi2rx
  rmmod cdns_csi2rx
  rmmod ov5640
  insmod /lib/modules/6.1.33-g40c32565ca/kernel/drivers/media/i2c/ov5640.ko
  insmod /lib/modules/6.1.33-g40c32565ca/kernel/drivers/media/platform/cadence/cdns-csi2rx.ko
  insmod /lib/modules/6.1.33-g40c32565ca/kernel/drivers/media/platform/ti/j721e-csi2rx/j721e-csi2rx.ko

  • 0 in reply to Lyndon Amsdon
    TI__Guru 59125 points

    Ok. Thanks for providing the detailed log. I was able to reproduce the problem.

    Let me investigate this and get back to you as soon as I can.

    Regards,

    Jianzhong

  • +1 in reply to Jianzhong Xu
    TI__Guru 59125 points

    Hi Lyndon,

    Thanks again for reporting this issue. We've root caused the problem. Please try a temporary fix in this patch: https://e2e.ti.com/cfs-file/__key/communityserver-discussions-components-files/791/0001_2D00_Testing_2D00_workaround_2D00_for_2D00_issues_2D00_with_2D00_runtime_2D00_loading_2D00_m.patch

    We'll be doing more testing and provide a permanent fix through CI/CD snapshot.

    Regards,

    Jianzhong