• State TI Thinks Resolved
  • Locked Locked
  • Replies 20 replies
  • Answers 3 answers
  • Subscribers 102 subscribers
  • Views 1081 views
  • Users 0 members are here
Support feedback
Options
Options
Similar topics

This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

Linux/DRA746: Weston Crashes in over night Test with Video playback and navigation running

Ikshwaku Chauhan
Expert 2630 points
Part Number: DRA746

Tool/software: Linux

Hello Team, 

We use custom h/w and software based on DRA746 and VSDK 3.04

During overnight testing where we are doing continuous Video playback with Navigation is active in background and weston crashes,  before weston crashes we observed HMI was hung and shown some colored lines  on HMI(will share the Image through portal).

Weston is crashing with following backtrace:

(gdb) bt
#0 0xb6d9e18c in raise (sig=sig@entry=5) at ../sysdeps/unix/sysv/linux/pt-raise.c:36
#1 0x0002d014 in on_caught_signal (s=<optimized out>, siginfo=<optimized out>, context=<optimized out>) at /usr/src/debug/weston/1.11.0-r0.arago14/weston-1.11.0/src/main.c:383
#2 <signal handler called>
#3 0xb6f4d33c in wl_list_insert_list (list=list@entry=0xb6d88c3c, other=0xb6d88ba0 <main_arena+1016>) at /usr/src/debug/wayland/1.11.0-r0/wayland-1.11.0/src/wayland-util.c:106
#4 0x000195d4 in surface_stash_subsurface_views (surface=0xb6d88978 <main_arena+464>) at /usr/src/debug/weston/1.11.0-r0.arago14/weston-1.11.0/src/compositor.c:2201
#5 0x0001dd3c in weston_compositor_build_view_list (compositor=0x48ef0) at /usr/src/debug/weston/1.11.0-r0.arago14/weston-1.11.0/src/compositor.c:2302
#6 0x0001dbb0 in weston_view_destroy (view=0xbbb10) at /usr/src/debug/weston/1.11.0-r0.arago14/weston-1.11.0/src/compositor.c:1923
#7 0xb5545800 in ivi_view_destroy (ivi_view=0x9e488) at /usr/src/debug/weston/1.11.0-r0.arago14/weston-1.11.0/ivi-shell/ivi-layout.c:156
#8 0xb5547824 in ivi_layout_surface_destroy (ivisurf=0xe2d80) at /usr/src/debug/weston/1.11.0-r0.arago14/weston-1.11.0/ivi-shell/ivi-layout.c:244
#9 0xb5549454 in layout_surface_cleanup (ivisurf=0xcd0d8) at /usr/src/debug/weston/1.11.0-r0.arago14/weston-1.11.0/ivi-shell/ivi-shell.c:152
#10 0x0001ded4 in wl_signal_emit (data=0xe5288, signal=0xe528c)
at /home/buildserver/work/jenkins/var/lib/jenkins/workspace/MMT2020_ADV_RELEASE_APPLINUX_R4-3/project/elina-distro/build-cpm-mmt-2020/tmp/sysroots/mmt2020-a880/usr/include/wayland-server-core.h:330
#11 weston_surface_destroy (surface=0xe5288) at /usr/src/debug/weston/1.11.0-r0.arago14/weston-1.11.0/src/compositor.c:1952
#12 0xb6f47dfc in destroy_resource (element=0xbc490, data=<optimized out>) at /usr/src/debug/wayland/1.11.0-r0/wayland-1.11.0/src/wayland-server.c:571
#13 0xb6f4d7a4 in for_each_helper (entries=0xbc1f0, entries=0xbc1f0, data=0xbef2e6ec, func=0xb6f47d88 <destroy_resource>) at /usr/src/debug/wayland/1.11.0-r0/wayland-1.11.0/src/wayland-util.c:372
#14 wl_map_for_each (map=map@entry=0xbc1f0, func=0xb6f47d88 <destroy_resource>, data=0xbef2e6ec, data@entry=0xbef2e6e4) at /usr/src/debug/wayland/1.11.0-r0/wayland-1.11.0/src/wayland-util.c:378
#15 0xb6f488cc in wl_client_destroy (client=client@entry=0xbc1d0) at /usr/src/debug/wayland/1.11.0-r0/wayland-1.11.0/src/wayland-server.c:709
#16 0xb6f48994 in wl_client_connection_data (fd=<optimized out>, mask=<optimized out>, data=0xbc1d0) at /usr/src/debug/wayland/1.11.0-r0/wayland-1.11.0/src/wayland-server.c:248
#17 0xb6f4ac88 in wl_event_loop_dispatch (loop=0x46650, timeout=timeout@entry=-1) at /usr/src/debug/wayland/1.11.0-r0/wayland-1.11.0/src/event-loop.c:421
#18 0xb6f49060 in wl_display_run (display=display@entry=0x46608) at /usr/src/debug/wayland/1.11.0-r0/wayland-1.11.0/src/wayland-server.c:1051
#19 0x0001849c in main (argc=-1091376748, argv=<optimized out>) at /usr/src/debug/weston/1.11.0-r0.arago14/weston-1.11.0/src/main.c:1454

Also, we are observing following  kernal logs:

mmt2020-a880 4,4088,456704620,-;------------[ cut here ]------------
mmt2020-a880 4,4089,456704636,-;WARNING: CPU: 0 PID: 1147 at /home/buildserver/work/jenkins/var/lib/jenkins/workspace/MMT2020_ADV_RELEASE_APPLINUX_R4-3/project/elina-distro/build-cpm-mmt-2020/tmp/work-shared/mmt2020-a880/kernel-source/mm/page_alloc.c:6827 free_contig_range+0xa4/0xa8()
mmt2020-a880 4,4090,456704645,-;375 pages are still in use!
mmt2020-a880 4,4091,456704652,-;Modules linked in: cputop(O) tntfs(PO) texfat(PO) usb_f_ncm u_ether usb_f_fs sd8xxx mlan libcomposite configfs cfg80211 cryptoloop loop ti_fpd3_serdes smsc95xx smsc75xx asix usbnet mii usb_storage snd_soc_simple_card snd_soc_bt_sco snd_soc_mmt2020_mcasp snd_soc_edma snd_soc_omap dspipc snd_soc_omap_harman_dsp snd_soc_omap_dsp snd_soc_omap_dsp_pcm dwc3 udc_core snd_soc_core dwc3_omap snd_pcm_dmaengine xhci_plat_hcd xhci_hcd snd_pcm snd_timer snd soundcore usbcore usb_common memcache(O) rpmsg_proto pvrsrvkm(O) silabs_dabplugin omap_remoteproc rpmsg_rpc virtio_rpmsg_bus remoteproc virtio virtio_ring omap_mailbox [last unloaded: cfg80211]
mmt2020-a880 4,4092,456705557,-;CPU: 0 PID: 1147 Comm: weston Tainted: P W O 4.4.84 #Rel_Elina_J6_MMT_19031A
mmt2020-a880 4,4093,456705567,-;Hardware name: Generic DRA74X (Flattened Device Tree)
mmt2020-a880 4,4094,456705574,-;Backtrace:
mmt2020-a880 4,4095,456705591,-;[<c0015170>] (dump_backtrace) from [<c00153c0>] (show_stack+0x20/0x24)
mmt2020-a880 4,4096,456705599,-; r7:600f0013 r6:c09b9c04 r5:c09b9c04 r4:00000000
mmt2020-a880 4,4097,456705630,-;[<c00153a0>] (show_stack) from [<c0365ff4>] (dump_stack+0x90/0xa4)
mmt2020-a880 4,4098,456705646,-;[<c0365f64>] (dump_stack) from [<c0038990>] (warn_slowpath_common+0x94/0xc4)
mmt2020-a880 4,4099,456705654,-; r7:c0133570 r6:00001aab r5:00000009 r4:ede69b7c
mmt2020-a880 4,4100,456705684,-;[<c00388fc>] (warn_slowpath_common) from [<c0038a10>] (warn_slowpath_fmt+0x50/0x6c)
mmt2020-a880 4,4101,456705692,-; r8:c0a41580 r7:c0977850 r6:00000177 r5:000fa777 r4:c097448c
mmt2020-a880 4,4102,456705727,-;[<c00389c4>] (warn_slowpath_fmt) from [<c0133570>] (free_contig_range+0xa4/0xa8)
mmt2020-a880 4,4103,456705735,-; r3:00000177 r2:c080ee0c
mmt2020-a880 4,4104,456705750,-; r4:00000000
mmt2020-a880 4,4105,456705766,-;[<c01334cc>] (free_contig_range) from [<c0178b14>] (cma_release+0x9c/0x194)
mmt2020-a880 4,4106,456705774,-; r9:ef316210 r8:01135800 r7:00000177 r6:c0a42ecc r5:ed1fb800 r4:000fa600
mmt2020-a880 4,4107,456705810,-;[<c0178a78>] (cma_release) from [<c0464684>] (dma_release_from_contiguous+0x40/0x44)
mmt2020-a880 4,4108,456705818,-; r9:ef316210 r8:01135800 r7:c097448c r6:00000001 r5:ed1fb800 r4:00177000
mmt2020-a880 4,4109,456705856,-;[<c0464644>] (dma_release_from_contiguous) from [<c001e92c>] (__arm_dma_free+0x144/0x230)
mmt2020-a880 4,4110,456705869,-;[<c001e7e8>] (__arm_dma_free) from [<c001ea70>] (arm_dma_free+0x28/0x30)
mmt2020-a880 4,4111,456705878,-; r10:00000014 r9:c001ea48 r8:fa600000 r7:ed1fb800 r6:00177000 r5:ef316210
mmt2020-a880 4,4112,456705911,-; r4:e60b5700
mmt2020-a880 4,4113,456705931,-;[<c001ea48>] (arm_dma_free) from [<c0434e20>] (v_gem_free_object+0x114/0x14c)
mmt2020-a880 4,4114,456705947,-;[<c0434d0c>] (v_gem_free_object) from [<c041377c>] (drm_gem_object_free+0x40/0x58)
mmt2020-a880 4,4115,456705956,-; r9:c0413a78 r8:ee275100 r7:00000001 r6:eea69000 r5:e60b5700 r4:eea69000
mmt2020-a880 4,4116,456705995,-;[<c041373c>] (drm_gem_object_free) from [<c0413934>] (drm_gem_object_handle_unreference_unlocked+0xf8/0x12c)
mmt2020-a880 4,4117,456706004,-; r5:eea69034 r4:e60b5700
mmt2020-a880 4,4118,456706030,-;[<c041383c>] (drm_gem_object_handle_unreference_unlocked) from [<c0413ad8>] (drm_gem_object_release_handle+0x60/0x78)
mmt2020-a880 4,4119,456706038,-; r5:ee275100 r4:e60b5700
mmt2020-a880 4,4120,456706065,-;[<c0413a78>] (drm_gem_object_release_handle) from [<c0366b18>] (idr_for_each+0xc0/0x10c)
mmt2020-a880 4,4121,456706073,-; r7:00000001 r6:000000ff r5:ede69d04 r4:00000000
mmt2020-a880 4,4122,456706106,-;[<c0366a58>] (idr_for_each) from [<c0414908>] (drm_gem_release+0x2c/0x38)
mmt2020-a880 4,4123,456706114,-; r10:ee27517c r9:eea69034 r8:00000100 r7:00000200 r6:ee275180 r5:eea69000
mmt2020-a880 4,4124,456706147,-; r4:ee275120
mmt2020-a880 4,4125,456706165,-;[<c04148dc>] (drm_gem_release) from [<c041355c>] (drm_release+0x3fc/0x4c0)
mmt2020-a880 4,4126,456706174,-; r5:eea69000 r4:ee275100
mmt2020-a880 4,4127,456706197,-;[<c0413160>] (drm_release) from [<c0182534>] (__fput+0x90/0x1e0)
mmt2020-a880 4,4128,456706208,-; r10:ee2751c8 r9:00000008 r8:00000000 r7:eec27000 r6:ef307d10 r5:eea16348
mmt2020-a880 4,4129,456706244,-; r4:ee2751c0
mmt2020-a880 4,4130,456706264,-;[<c01824a4>] (__fput) from [<c01826f4>] (____fput+0x18/0x1c)
mmt2020-a880 4,4131,456706275,-; r10:ede68000 r9:00000000 r8:0418004f r7:ee84b500 r6:ee3cb580 r5:c09f165c
mmt2020-a880 4,4132,456706308,-; r4:ee84b968
mmt2020-a880 4,4133,456706327,-;[<c01826dc>] (____fput) from [<c0055bbc>] (task_work_run+0xa4/0xd8)
mmt2020-a880 4,4134,456706346,-;[<c0055b18>] (task_work_run) from [<c003b6f8>] (do_exit+0x348/0xac8)
mmt2020-a880 4,4135,456706354,-; r7:ede69e24 r6:ede58afc r5:ee0ad6d4 r4:ee84b500
mmt2020-a880 4,4136,456706388,-;[<c003b3b0>] (do_exit) from [<c003bf10>] (do_group_exit+0x4c/0xcc)
mmt2020-a880 4,4137,456706396,-; r7:b6dd0188
mmt2020-a880 4,4138,456706415,-;[<c003bec4>] (do_group_exit) from [<c0047ca4>] (get_signal+0x2b0/0x6ec)
mmt2020-a880 4,4139,456706423,-; r7:b6dd0188 r6:ede69ed8 r5:00000005 r4:ee103b10
mmt2020-a880 4,4140,456706456,-;[<c00479f4>] (get_signal) from [<c0014528>] (do_signal+0xe8/0x3f4)
mmt2020-a880 4,4141,456706464,-; r10:00000000 r9:00000000 r8:b6dd018c r7:b6dd0188 r6:ede69ec4 r5:c097448c
mmt2020-a880 4,4142,456706493,-; r4:ede69fb0
mmt2020-a880 4,4143,456706511,-;[<c0014440>] (do_signal) from [<c0014a0c>] (do_work_pending+0xa8/0xc0)
mmt2020-a880 4,4144,456706519,-; r10:00000000 r9:ede68000 r8:c0010e44 r7:0000010c r6:ede69fb0 r5:c0010e44
mmt2020-a880 4,4145,456706549,-; r4:ede68010
mmt2020-a880 4,4146,456706566,-;[<c0014964>] (do_work_pending) from [<c0010cec>] (slow_work_pending+0xc/0x20)
mmt2020-a880 4,4147,456706574,-; r7:0000010c r6:b6dbacc0 r5:b6f80a58 r4:00000002
mmt2020-a880 4,4148,456706738,-;---[ end trace 102261866c07be86 ]---
mmt2020-a880 4,4149,456706833,-;------------[ cut here ]------------

These logs are coming continuously when issue happens.

Could you please help us in debugging this issue.

Regards,

Ikshwaku 

  • 0
    TI__Mastermind 30480 points

    Hi Ikshwaku,

    Are you seeing the issue after increasing the CMA from 128MB to 192MB?

    How log it took to reproduce the issue? Is the back trace generated same every time it happens?

    Did you see this issue with PSDKLA3.x (i,e without M4 display server)?

    Thanks

    Ram

  • 0 in reply to Ramprasad
    Expert 2630 points
    Hello Ram,

    We are facing this issue, for 7 inch display and here we are using 128MB as CMA. The other issue where we increased CAM from 128MB to 192MB is for 12.7 inch display.

    How log it took to reproduce the issue? Is the back trace generated same every time it happens?
    --> This issue happens in overnight test and it is some times issue not always. once or twice we are able to reproduce this issue within 2-3 hrs, but yes mostly occurs in overnight testing.

    Did you see this issue with PSDKLA3.x (i,e without M4 display server)?
    --> all over setups are with M4 configurations.

    Regards,
    Ikshwaku
  • 0 in reply to Ikshwaku Chauhan
    Expert 2630 points
    Hello Ram,

    Any update for this issue.??

    Regards,
    Ikshwaku
  • 0 in reply to Ikshwaku Chauhan
    TI__Mastermind 30480 points

    Hi Ikshwaku,

    I have discussed internally on this issue. The traces you  observed are normally observed sometimes but currption on HMi is may be due to some memory corruption in vision SDK.

    I will update more on this on Monday.

    Ram

  • 0 in reply to Ramprasad
    TI__Mastermind 30480 points
    Hi Ikshwaku,
    We didn't get any clue what can cause this corruption of HMI screen.
    Can you please try to get these details?
    Once this issue happens, please collect dmesg logs and vision-sdk logs if any
    Are you able to execute video playback again when this issue happens?

    Thanks
    Ram
  • 0 in reply to Ramprasad
    Expert 2630 points

    Hello Ram,

    Issue is reproduce again and this time issues is reproduced with just Navigation.

    Once this issue happens, please collect dmesg logs and vision-sdk logs if any
    --> sharing the logs through portal(dmesg , meminfo, weston crash backtrace).

    Are you able to execute video playback again when this issue happens?
    --> when issue happens weston is crashed, so video playback is not possible. And this time issue happens just with navigation running.

    I have observed that when issue happens Free CMA is only 512KB.


    Regards,
    Ikshwaku

  • 0 in reply to Ikshwaku Chauhan
    TI__Mastermind 30480 points

    Hi Ikshwaku,
    We will analyze all the logs you shared. Can you try to get answer to these questions?
    1) Can you check if CMA entry in /proc/meminfo keeps on decreasing when navigation application is executed?
    2) Can you check running only gstreamer video playback overnight reproduces the issue? check if /proc/meminfo entry for CMA decreases over time.

    Weston allocates CMA memory for 3 buffers during weston launch and it won't ask for any other memory. So need to narrow down
    if navigation application is leaking any memory. In gstreamer also all allocations happen upfront.

    Thanks
    Ram

  • 0 in reply to Ramprasad
    Expert 2630 points
    Hello Ram,

    1) Can you check if CMA entry in /proc/meminfo keeps on decreasing when navigation application is executed?
    --> When we run Navigation app then we observed sometime CAM is reducing to "0 MB" as well and system is not crashing, but operations on HMI is not smooth after that.

    For gstreamer will check the behavior update you .

    Regards,
    Ikshwaku
  • 0 in reply to Ikshwaku Chauhan
    Expert 2630 points
    Hello Ram,

    One more update:

    Weston is not crashed but is exited with the SIGABRT signal and Systemd has sent SIGABRT to Weston because weston is not sending the watchdog ping to Systemd. It seems weston is hanged.

    Any inputs from you related to CMA reducing to "0 MB" and system is not crashing.

    Regards,
    Ikshwaku
  • 0 in reply to Ikshwaku Chauhan
    TI__Mastermind 30480 points
    Hi Ikshwaku,
    Can you keep checking dri/0/gem when you launch navigation application,

    cat /sys/kernel/debug/dri/0/gem
    and check if this entry list is growing.

    Thanks
    Ram
  • 0 in reply to Ramprasad
    Expert 2630 points

    Before navigation starts:
    root@mmt2020-a880:~# cat /sys/kernel/debug/dri/0/gem
    All Objects:
    eebc5f40: 0 ( 3) 00010000 0xf7e00000 1536000
    ee3f3640: 0 ( 3) 00010177 0xf8000000 1536000
    ee36e400: 0 ( 3) 000102ee 0xf8200000 1536000
    eea77ac0: 0 ( 3) 00010465 0xf8400000 1536000
    ee012e80: 0 ( 3) 000105dc 0xf8600000 1536000
    ef183040: 0 ( 3) 00010753 0xf8800000 1536000
    ee213e80: 0 ( 3) 000108ca 0xf8a00000 1536000
    ee3a0dc0: 0 ( 3) 00010a41 0xf8c00000 1536000
    ee3a0d00: 0 ( 3) 00010bb8 0xf8e00000 1536000
    ee3a0c40: 0 ( 3) 00010d2f 0xf9000000 1536000
    ee3df7c0: 0 ( 3) 00010ea6 0xf9200000 1536000
    ee3a9ac0: 0 ( 3) 0001101d 0xf9400000 1536000
    ee3a9400: 0 ( 3) 00011194 0xf9600000 1536000
    ee3a9880: 0 ( 3) 0001130b 0xf9800000 1536000
    ee25ce80: 0 ( 3) 00011482 0xf9a00000 1536000
    eebfcb80: 3 ( 3) 00011770 0xf9e00000 1536000
    ed542d00: 2 ( 3) 000118e7 0xfa000000 1536000
    e73de1c0: 1 ( 3) 000115f9 0xf9c00000 1536000
    e5482700: 4 ( 3) 00011a5e 0xfa200000 1536000
    e7069280: 0 ( 2) 00011bd9 0xfa400000 1536000
    e4f4eac0: 5 ( 3) 00011d50 0xfa600000 1536000
    e71cc4c0: 0 ( 2) 00011ec7 0xfa800000 1536000
    e3c4f100: 6 ( 3) 0001203e 0xfaa00000 1536000
    Total 23 objects, 35328000 bytes

    After Navigation starts:
    --------------------------------
    root@mmt2020-a880:~# cat /sys/kernel/debug/dri/0/gem
    All Objects:
    eebc5f40: 0 ( 3) 00010000 0xf7e00000 1536000
    ee3f3640: 0 ( 3) 00010177 0xf8000000 1536000
    ee36e400: 0 ( 3) 000102ee 0xf8200000 1536000
    eea77ac0: 0 ( 3) 00010465 0xf8400000 1536000
    ee012e80: 0 ( 3) 000105dc 0xf8600000 1536000
    ef183040: 0 ( 3) 00010753 0xf8800000 1536000
    ee213e80: 0 ( 3) 000108ca 0xf8a00000 1536000
    ee3a0dc0: 0 ( 3) 00010a41 0xf8c00000 1536000
    ee3a0d00: 0 ( 3) 00010bb8 0xf8e00000 1536000
    ee3a0c40: 0 ( 3) 00010d2f 0xf9000000 1536000
    ee3df7c0: 0 ( 3) 00010ea6 0xf9200000 1536000
    ee3a9ac0: 0 ( 3) 0001101d 0xf9400000 1536000
    ee3a9400: 0 ( 3) 00011194 0xf9600000 1536000
    ee3a9880: 0 ( 3) 0001130b 0xf9800000 1536000
    ee25ce80: 0 ( 3) 00011482 0xf9a00000 1536000
    eebfcb80: 3 ( 3) 00011770 0xf9e00000 1536000
    ed542d00: 2 ( 3) 000118e7 0xfa000000 1536000
    e73de1c0: 1 ( 3) 000115f9 0xf9c00000 1536000
    e5482700: 4 ( 3) 00011a5e 0xfa200000 1536000
    e7069280: 0 ( 2) 00011bd9 0xfa400000 1536000
    e4f4eac0: 5 ( 3) 00011d50 0xfa600000 1536000
    e71cc4c0: 0 ( 2) 00011ec7 0xfa800000 1536000
    e3c4f100: 6 ( 3) 0001203e 0xfaa00000 1536000
    e4d46dc0: 9 ( 3) 000121b5 0xfac00000 1536000
    e6b2fd00: 7 ( 3) 0001232c 0xfae00000 1536000
    eebc5040: 10 ( 3) 000124a3 0xfb000000 720896
    e9115580: 8 ( 3) 00012553 0xfb100000 720896
    Total 27 objects, 39841792 bytes
    root@mmt2020-a880:~#

    Monitored for 20 mins, but it is not growing.


    Regards,
    Ikshwaku

  • 0 in reply to Ikshwaku Chauhan
    TI__Mastermind 30480 points
    Hi Ikshwaku,
    Can you try running this command when CmaFree goes down over time?

    echo 1 > /proc/sys/vm/drop_caches
    and check if memory is reclaimed.

    Thanks
    Ram
  • 0 in reply to Ramprasad
    Expert 2630 points
    Hello Ram,

    We have executed "echo 1 > /proc/sys/vm/drop_caches" when CmaFree: 12752 kB
    and after executing this CMA increased "CmaFree: 68716 kB".

    Regards,
    Ikshwaku
  • 0 in reply to Ikshwaku Chauhan
    TI__Mastermind 30480 points
    Hi Ikshwaku,
    Thanks for checking this.
    Can you create a script to execute this for every once hour or two hour and give a over night run?

    while true
    do
    sleep 3600
    echo 1 > /proc/sys/vm/drop_caches
    done

    and check overnight . I want to confirm if CMA memory exhausting is causing the issue or not.

    I am able to see playback working even if CMAFree is 0kb

    Thanks
    Ram
  • 0 in reply to Ramprasad
    Expert 2630 points
    Hello Ram,

    We have run the experiment overnight and system is running fine. But, here we are not sure that issue actually hit or not, because this issue occurs sometime.

    Regards,
    Ikshwaku
  • 0 in reply to Ikshwaku Chauhan
    Expert 2630 points
    Hello Ram,

    Any update for this issue?

    Regards,
    Ikshwaku
  • 0 in reply to Ikshwaku Chauhan
    TI__Mastermind 30480 points
    Hi Ikshwaku,
    What is the observation with drop_cache regularly? does it resolve the issue?
    From our last discussion, this is the limitation of Linux kernel that CMA can not be controlled to be allocated for other processes unless it is device based CMA allocation

    Thanks
    Ram
  • 0 in reply to Ramprasad
    Expert 2630 points
    Hello Ram,


    What is the observation with drop_cache regularly? does it resolve the issue?
    --> We have run the experiment overnight and system is running fine. But, here we are not sure that issue actually hit or not, because this issue occurs sometime.

    CMA reducing to Zero. That we agreed as the limitation of CMA.
    But of this issue we see that Navigation output is becoming blur when issue happens weston crashes. I have already shared that image.
    Sharing again.

    As per out last discussion with (Subahjit and you) following logs are not harmful, But we getting these logs only when issue happens otherwise these logs are not observed.

    mmt2020-a880 4,4094,456705574,-;Backtrace:
    mmt2020-a880 4,4095,456705591,-;[<c0015170>] (dump_backtrace) from [<c00153c0>] (show_stack+0x20/0x24)
    mmt2020-a880 4,4096,456705599,-; r7:600f0013 r6:c09b9c04 r5:c09b9c04 r4:00000000
    mmt2020-a880 4,4097,456705630,-;[<c00153a0>] (show_stack) from [<c0365ff4>] (dump_stack+0x90/0xa4)
    mmt2020-a880 4,4098,456705646,-;[<c0365f64>] (dump_stack) from [<c0038990>] (warn_slowpath_common+0x94/0xc4)
    mmt2020-a880 4,4099,456705654,-; r7:c0133570 r6:00001aab r5:00000009 r4:ede69b7c
    mmt2020-a880 4,4100,456705684,-;[<c00388fc>] (warn_slowpath_common) from [<c0038a10>] (warn_slowpath_fmt+0x50/0x6c)
    mmt2020-a880 4,4101,456705692,-; r8:c0a41580 r7:c0977850 r6:00000177 r5:000fa777 r4:c097448c
    mmt2020-a880 4,4102,456705727,-;[<c00389c4>] (warn_slowpath_fmt) from [<c0133570>] (free_contig_range+0xa4/0xa8)
    mmt2020-a880 4,4103,456705735,-; r3:00000177 r2:c080ee0c
    mmt2020-a880 4,4104,456705750,-; r4:00000000
    mmt2020-a880 4,4105,456705766,-;[<c01334cc>] (free_contig_range) from [<c0178b14>] (cma_release+0x9c/0x194)
    mmt2020-a880 4,4106,456705774,-; r9:ef316210 r8:01135800 r7:00000177 r6:c0a42ecc r5:ed1fb800 r4:000fa600
    mmt2020-a880 4,4107,456705810,-;[<c0178a78>] (cma_release) from [<c0464684>] (dma_release_from_contiguous+0x40/0x44)
    mmt2020-a880 4,4108,456705818,-; r9:ef316210 r8:01135800 r7:c097448c r6:00000001 r5:ed1fb800 r4:00177000
    mmt2020-a880 4,4109,456705856,-;[<c0464644>] (dma_release_from_contiguous) from [<c001e92c>] (__arm_dma_free+0x144/0x230)
    mmt2020-a880 4,4110,456705869,-;[<c001e7e8>] (__arm_dma_free) from [<c001ea70>] (arm_dma_free+0x28/0x30)
    mmt2020-a880 4,4111,456705878,-; r10:00000014 r9:c001ea48 r8:fa600000 r7:ed1fb800 r6:00177000 r5:ef316210
    mmt2020-a880 4,4112,456705911,-; r4:e60b5700


    Regards,
    Ikshwaku
  • 0 in reply to Ikshwaku Chauhan
    TI__Mastermind 30480 points
    Hi Ikshwaku,
    Discussed with Subhajit. Here also CMA is from general pool and kernel can not control the CMA, drop_cache is the work around in this case.
  • 0 in reply to Ramprasad
    TI__Mastermind 30480 points
    Hi Ikshwaku,
    The rootcause of the issue is CMA pool is being used by all processes and can not be controlled. so some allocation may fail due to non-availability of CMA and may lead to the traces you mentioned.

    So occationally dropping the cache will make sure this won't happen.

    Thanks
    Ram