This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

[DM8148]: GPU Compositing OOM Killer



Hello TI masters,


I am running an application capturing video frame and displaying it with the GPU thanks to GPU composition daemon (http://processors.wiki.ti.com/index.php/GPU_Compositing). I am also encoding this same video frame and streaming over Ethernet to watch the video on a laptop.

Strangely, when I run the application for 3 hours (more or less, it is not always the same time), it crashes indicating that oom-killer killed composition process. Therefore my main application also crashes...

From 0sec to ~3hours, the application (named "demo") works perfectly well.

Here I have the crash log with OOM-killer indication. As indicated, it ran out of memory but I just don't get why it take so much time before it crashes.

Does someone have any idea about this issue ?


Thank you very much for your help.


OOM-KILLER LOG:

root@dm814x-evm:~/CES# demo invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
Backtrace:
[<c004abd0>] (dump_backtrace+0x0/0x110) from [<c03d7f7c>] (dump_stack+0x18/0x1c)
 r7:00000000 r6:00000000 r5:c2afadc0 r4:000201da
[<c03d7f64>] (dump_stack+0x0/0x1c) from [<c00a510c>] (dump_header+0x60/0x138)
[<c00a50ac>] (dump_header+0x0/0x138) from [<c00a5468>] (oom_kill_process+0x50/0x1f8)
 r8:c17fe000 r7:00000041 r6:00000000 r5:c2afadc0 r4:000201da
[<c00a5418>] (oom_kill_process+0x0/0x1f8) from [<c00a5874>] (out_of_memory+0x264/0x2e0)
[<c00a5610>] (out_of_memory+0x0/0x2e0) from [<c00a8a98>] (__alloc_pages_nodemask+0x430/0x51c)
[<c00a8668>] (__alloc_pages_nodemask+0x0/0x51c) from [<c00aa418>] (__do_page_cache_readahead+0x9c/0x1e8)
[<c00aa37c>] (__do_page_cache_readahead+0x0/0x1e8) from [<c00aa590>] (ra_submit+0x2c/0x34)
[<c00aa564>] (ra_submit+0x0/0x34) from [<c00a33c4>] (filemap_fault+0x170/0x3b0)
[<c00a3254>] (filemap_fault+0x0/0x3b0) from [<c00b6900>] (__do_fault+0x58/0x3dc)
[<c00b68a8>] (__do_fault+0x0/0x3dc) from [<c00b739c>] (handle_mm_fault+0x318/0xaa0)
[<c00b7084>] (handle_mm_fault+0x0/0xaa0) from [<c03dbef8>] (do_page_fault+0x114/0x20c)
[<c03dbde4>] (do_page_fault+0x0/0x20c) from [<c003c208>] (do_PrefetchAbort+0x3c/0x9c)
[<c003c1cc>] (do_PrefetchAbort+0x0/0x9c) from [<c03da424>] (ret_from_exception+0x0/0x10)
Exception stack(0xc17fffb0 to 0xc17ffff8)
ffa0:                                     00000000 00000000 5b8624f4 00000000
ffc0: 0012abd8 0012ac50 00000000 000000a2 5b861bf4 00000000 00124078 5b861cf4
ffe0: 00000000 5b861bb0 4026e1f4 4022fea4 60000010 ffffffff
 r8:5b861bf4 r7:000000a2 r6:00000000 r5:0012ac50 r4:ffffffff
Mem-info:
Normal per-cpu:
CPU    0: hi:   18, btch:   3 usd:   6
active_anon:622 inactive_anon:33 isolated_anon:0
 active_file:187 inactive_file:421 isolated_file:0
 unevictable:0 dirty:0 writeback:0 unstable:0
 free:279 slab_reclaimable:342 slab_unreclaimable:7888
 mapped:1560 shmem:37 pagetables:299 bounce:0
Normal free:1116kB min:1120kB low:1400kB high:1680kB active_anon:2488kB inactive_anon:132kB active_file:748kB inactive_file:1684kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:78848kB mlocked:0kB dirty:0kB writeback:0kB mapped:6240kB shmem:148kB slab_reclaimable:1368kB slab_unreclaimable:31552kB kernel_stack:632kB pagetables:1196kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0
Normal: 7*4kB 0*8kB 6*16kB 5*32kB 3*64kB 1*128kB 2*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1116kB
645 total pagecache pages
32768 pages of RAM
412 free pages
14495 reserved pages
8045 slab pages
2265 pages shared
0 pages swap cached
[ pid ]   uid  tgid total_vm      rss cpu oom_adj oom_score_adj name
[   71]     0    71      537       53   0     -17         -1000 udevd
[ 1215]    42  1215      864       56   0       0             0 dbus-daemon
[ 1226]     0  1226      560       30   0       0             0 dropbear
[ 1231]     0  1231      734       21   0       0             0 telnetd
[ 1236]     0  1236      631       29   0       0             0 netserver
[ 1241]     0  1241      750       44   0       0             0 syslogd
[ 1243]     0  1243      734       27   0       0             0 klogd
[ 1248]     0  1248      573       75   0       0             0 thttpd
[ 1358]     0  1358    23186     1739   0       0             0 composition
[ 1380]     0  1380      629       53   0       0             0 login
[ 1381]     0  1381      492       31   0       0             0 getty
[ 1383]     0  1383      818       70   0       0             0 sh
[ 1386]     0  1386   120744      201   0       0             0 demo
Out of memory: Kill process 1358 (composition) score 65 or sacrifice child
Killed process 1358 (composition) total-vm:92744kB, anon-rss:664kB, file-rss:6292kB

[1] + Broken pipe                ./demo -r 1

  • Hello,

    You could check are your demo cause a memory leak. Probably this is why it crashes after a while(~3 hours) with out of memory error.

    Best Regards,

    Margarita

  • Hello Margarita,

    I did some tests on demo application. The major problem is coming from the named pipe writing operation. I am writing over the named pipe (for display, config_data = 0) at each captured frame which means 60 times per second. I am also writing a new configuration (config_data=1 of composition daemon) every 10 captured frame.

    Is it too much for the pipe to handle ? Is there any flush operation that I need to do after a writing/reading operation on the pipe ? The weird thing is that it can handle 2h30 of writing operations but then stops working...


    Thank you for your reply .


    Dylan

  • Hello Dylan,

    I am not a GPU compositors expert. I will try to involve someone to helps here.

    Best Regards,

    Margarita