[DSP2:02204] mca: base: components_open: Looking for btl components [DSP2:02204] mca: base: components_open: opening btl components [DSP2:02204] mca: base: components_open: found loaded component hlink [DSP2:02204] BTL_HLINK TIMPIDBG: hlink_component_register!!! [DSP2:02204] This is EVM, using hl0 only! [DSP2:02204] mca: base: components_open: component hlink register function successful [DSP2:02204] BTL_HLINK TIMPIDBG: hlink_component_open!!! [DSP2:02204] BTL_HLINK BTL HLINK start of HYPLNKINITCFG: 0xb6a70130 [DSP2:02204] BTL_HLINK [0x21400000] [DSP2:02204] BTL_HLINK [0x40000000] [DSP2:02204] BTL_HLINK [0x21400100] [DSP2:02204] BTL_HLINK [0x28000000] [DSP2:02204] BTL_HLINK [(nil)] [DSP2:02204] BTL_HLINK [(nil)] [DSP2:02204] BTL_HLINK [(nil)] [DSP2:02204] BTL_HLINK [(nil)] [DSP2:02204] BTL_HLINK BTL HLINK end of HYPLNKINITCFG [DSP2:02204] BTL_HLINK: CMEM_init OK! [DSP2:02204] mca: base: components_open: component hlink open function successful [DSP2:02204] mca: base: components_open: found loaded component self [DSP2:02204] mca: base: components_open: component self has no register function [DSP2:02204] mca: base: components_open: component self open function successful [DSP1:02191] mca: base: components_open: Looking for btl components [DSP2:02204] select: initializing btl component hlink [DSP2:02204] BTL_HLINK TIMPIDBG: hlink_component_init!!! [DSP2:02204] BTL_HLINK shmem open successfull!! [DSP2:02204] BTL_HLINK: CMEM physAddr: 22000000 (to a2000000) userAddr:0xb5989000 [DSP2:02204] BTL_HLINK shmem MSMC0 mmap successfull!! [DSP2:02204] BTL_HLINK shmem MSMC0 mmap successfull!! [DSP2:02204] BTL_HLINK attempt HyperLink0 then HyperLink1 [DSP2:02204] BTL_HLINK hyplnk0 attempt opening [DSP2:02204] BTL_HLINK hyplnk0 open successfull!! [DSP2:02204] BTL_HLINK hyplnk1 attempt opening [DSP1:02191] mca: base: components_open: opening btl components [DSP1:02191] mca: base: components_open: found loaded component hlink [DSP1:02191] BTL_HLINK TIMPIDBG: hlink_component_register!!! [DSP1:02191] This is EVM, using hl0 only! [DSP1:02191] mca: base: components_open: component hlink register function successful [DSP1:02191] BTL_HLINK TIMPIDBG: hlink_component_open!!! [DSP1:02191] BTL_HLINK BTL HLINK start of HYPLNKINITCFG: 0xb6a1a130 [DSP1:02191] BTL_HLINK [0x21400000] [DSP1:02191] BTL_HLINK [0x40000000] [DSP1:02191] BTL_HLINK [0x21400100] [DSP1:02191] BTL_HLINK [0x28000000] [DSP1:02191] BTL_HLINK [(nil)] [DSP1:02191] BTL_HLINK [(nil)] [DSP1:02191] BTL_HLINK [(nil)] [DSP1:02191] BTL_HLINK [(nil)] [DSP1:02191] BTL_HLINK BTL HLINK end of HYPLNKINITCFG [DSP1:02191] BTL_HLINK: CMEM_init OK! [DSP1:02191] mca: base: components_open: component hlink open function successful [DSP1:02191] mca: base: components_open: found loaded component self [DSP1:02191] mca: base: components_open: component self has no register function [DSP1:02191] mca: base: components_open: component self open function successful [DSP1:02191] select: initializing btl component hlink [DSP1:02191] BTL_HLINK TIMPIDBG: hlink_component_init!!! [DSP1:02191] BTL_HLINK shmem open successfull!! [DSP1:02191] BTL_HLINK: CMEM physAddr: 22000000 (to a2000000) userAddr:0xb5933000 [DSP1:02191] BTL_HLINK shmem MSMC0 mmap successfull!! [DSP1:02191] BTL_HLINK shmem MSMC0 mmap successfull!! [DSP1:02191] BTL_HLINK attempt HyperLink0 then HyperLink1 [DSP1:02191] BTL_HLINK hyplnk0 attempt opening [DSP2:02204] BTL_HLINK hyplnk1 open failed [DSP2:02204] BTL_HLINK hyplnk0=0x77cf0 hyplnk1=(nil) [DSP2:02204] BTL_HLINK hyplnk0 MSMC mmap successfull!! [DSP2:02204] BTL_HLINK hyplnk0 DDR mmap successfull!! [DSP2:02204] BTL_HLINK BTL HLINK turned ON (hl0=1 hl1=0)!!! [DSP2:02204] select: init of component hlink returned success [DSP2:02204] select: initializing btl component self [DSP2:02204] select: init of component self returned success [DSP1:02191] BTL_HLINK hyplnk0 open failed [DSP1:02191] BTL_HLINK hyplnk1 attempt opening [DSP1:02191] BTL_HLINK hyplnk1 open successfull!! [DSP1:02191] BTL_HLINK hyplnk0=(nil) hyplnk1=0x76f38 [DSP1:02191] BTL_HLINK hyplnk1 MSMC mmap successfull!! [DSP1:02191] BTL_HLINK hyplnk1 DDR mmap successfull!! [DSP1:02191] BTL_HLINK BTL HLINK turned ON (hl0=0 hl1=1)!!! [DSP1:02191] select: init of component hlink returned success [DSP1:02191] select: initializing btl component self [DSP1:02191] select: init of component self returned success [DSP1:02191] TIMPIDBG 0 rmt_msgbase=0x9dd0b000 loc_msgbase=0xb6133000 [DSP1:02191] TIMPIDBG B hlink_module_add_procs(0) remote hostname=DSP1 (over HL0:NOTUSED HL1:DSP2, bridge:NOTUSED) rem_rank[0] jobid=c5460001 [DSP1:02191] TIMPIDBG B hlink_module_add_procs(1) remote hostname=DSP2 (over HL0:NOTUSED HL1:DSP2, bridge:NOTUSED) rem_rank[1] jobid=c5460001 [DSP2:02204] TIMPIDBG 0 rmt_msgbase=0xa5d69000 loc_msgbase=0xb5d89000 [DSP2:02204] TIMPIDBG B hlink_module_add_procs(0) remote hostname=DSP1 (over HL0:DSP1 HL1:NOTUSED, bridge:NOTUSED) rem_rank[0] jobid=c5460001 [DSP2:02204] HLINK0: TIMPIDBG[0, nlocal 0] hlink_module_add_procs, I can talk to rank: (0) on DSP1 [DSP2:02204] TIMPIDBG B hlink_module_add_procs(1) remote hostname=DSP2 (over HL0:DSP1 HL1:NOTUSED, bridge:NOTUSED) rem_rank[1] jobid=c5460001 [DSP2:02204] TIMPIDBG hlink_module_add_procs, RANKS: my_rank=1 nprocs=2 nlocal_procs=1 endpoint_count=2 [DSP1:02191] HLINK1: TIMPIDBG[1, nlocal 0] hlink_module_add_procs, I can talk to rank: (1) on DSP2 [DSP1:02191] TIMPIDBG hlink_module_add_procs, RANKS: my_rank=0 nprocs=2 nlocal_procs=1 endpoint_count=2 [DSP1:02191] TIMPIDBG: hlink_module_prepare_src EAGER: 130048 MAX: 131072 (btl=0xb6ac72fc) [DSP1:02191] BTL HLINK fragmentInitialized (btl=0xb6ac72fc) frag=0x7f900 [DSP1:02191] BTL HLINK fragmentInitialized (btl=0xb6ac72fc) frag=0x7f980 [DSP1:02191] BTL HLINK fragmentInitialized (btl=0xb6ac72fc) frag=0x7fa00 [DSP1:02191] BTL HLINK fragmentInitialized (btl=0xb6ac72fc) frag=0x7fa80 [DSP1:02191] BTL HLINK fragmentInitialized (btl=0xb6ac72fc) frag=0x7fb00 [DSP1:02191] BTL HLINK fragmentInitialized (btl=0xb6ac72fc) frag=0x7fb80 [DSP1:02191] BTL HLINK fragmentInitialized (btl=0xb6ac72fc) frag=0x7fc00 [DSP1:02191] BTL HLINK fragmentInitialized (btl=0xb6ac72fc) frag=0x7fc80 [DSP1:02191] BTL HLINK fragmentInitialized (btl=0xb6ac72fc) frag=0x7fd00 [DSP1:02191] BTL HLINK fragmentInitialized (btl=0xb6ac72fc) frag=0x7fd80 [DSP1:02191] BTL HLINK fragmentInitialized (btl=0xb6ac72fc) frag=0x7fe00 [DSP1:02191] BTL HLINK fragmentInitialized (btl=0xb6ac72fc) frag=0x7fe80 [DSP1:02191] BTL HLINK fragmentInitialized (btl=0xb6ac72fc) frag=0x7ff00 [DSP1:02191] BTL HLINK fragmentInitialized (btl=0xb6ac72fc) frag=0x7ff80 [DSP1:02191] BTL HLINK fragmentInitialized (btl=0xb6ac72fc) frag=0x80000 [DSP1:02191] BTL HLINK fragmentInitialized (btl=0xb6ac72fc) frag=0x80080 [DSP1:02191] BTL HLINK fragmentInitialized (btl=0xb6ac72fc) frag=0x80100 [DSP1:02191] BTL HLINK fragmentInitialized (btl=0xb6ac72fc) frag=0x80180 [DSP1:02191] BTL HLINK fragmentInitialized (btl=0xb6ac72fc) frag=0x80200 [DSP1:02191] BTL HLINK fragmentInitialized (btl=0xb6ac72fc) frag=0x80280 [DSP1:02191] BTL HLINK fragmentInitialized (btl=0xb6ac72fc) frag=0x80300 [DSP1:02191] BTL HLINK fragmentInitialized (btl=0xb6ac72fc) frag=0x80380 [DSP1:02191] BTL HLINK fragmentInitialized (btl=0xb6ac72fc) frag=0x80400 [DSP1:02191] BTL HLINK fragmentInitialized (btl=0xb6ac72fc) frag=0x80480 [DSP1:02191] BTL HLINK fragmentInitialized (btl=0xb6ac72fc) frag=0x80500 [DSP1:02191] BTL HLINK fragmentInitialized (btl=0xb6ac72fc) frag=0x80580 [DSP1:02191] BTL HLINK fragmentInitialized (btl=0xb6ac72fc) frag=0x80600 [DSP1:02191] BTL HLINK fragmentInitialized (btl=0xb6ac72fc) frag=0x80680 [DSP1:02191] BTL HLINK fragmentInitialized (btl=0xb6ac72fc) frag=0x80700 [DSP1:02191] BTL HLINK fragmentInitialized (btl=0xb6ac72fc) frag=0x80780 [DSP1:02191] BTL HLINK fragmentInitialized (btl=0xb6ac72fc) frag=0x80800 [DSP1:02191] BTL HLINK fragmentInitialized (btl=0xb6ac72fc) frag=0x80880 [DSP1:02191] TIMPIDBG: hlink_module_send frag=0x80880 (base=0x9d8b848c size=28014)!!! [DSP1:02191] TIMPIDBG: hlink_module_send FIFO_WRITE: nprocs 1(0)->0, frag_hdr=0x9d8b8480 eager_limit=130048 [DSP1:02191] TMPIDBG: rmt_ep 0 send (hyplnk_iface1) (bridging 0) (use_edma 1 seg_len 28014 eager_limit 130048) [DSP1:02191] BTL_HL[ 122.505225] mpirun invoked oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=0 [ 122.512698] CPU: 3 PID: 2189 Comm: mpirun Tainted: G O 3.10.72-gca334de #1 [ 122.520431] [] (unwind_backtrace+0x0/0xf8) from [] (show_stack+0x10/0x14) [ 122.528947] [] (show_stack+0x10/0x14) from [] (dump_header.isra.14+0x78/0x198) [ 122.531444] syslog-ng: page allocation failure: order:0, mode:0x201db [ 122.531449] CPU: 0 PID: 2046 Comm: syslog-ng Tainted: G O 3.10.72-gca334de #1 [ 122.531464] [] (unwind_backtrace+0x0/0xf8) from [] (show_stack+0x10/0x14) [ 122.531475] [] (show_stack+0x10/0x14) from [] (warn_alloc_failed+0xa8/0x120) [ 122.531485] [] (warn_alloc_failed+0xa8/0x120) from [] (__alloc_pages_nodemask+0x614/0x8c8) [ 122.531492] [] (__alloc_pages_nodemask+0x614/0x8c8) from [] (filemap_fault+0x1c8/0x410) [ 122.535890] 3367 reserved pages [ 122.544463] 4665 total pagecache pages [ 122.549902] 139264 pages of RAM [ 122.549905] 3367 reserved pages [ 122.551925] 4665 total pagecache pages [ 122.557679] 139264 pages of RAM [ 122.557682] 3367 reserved pages [ 122.561911] 4665 total pagecache pages [ 122.567090] 139264 pages of RAM [ 122.567092] 3367 reserved pages [ 122.571902] 4665 total pagecache pages [ 122.576763] 139264 pages of RAM [ 122.576765] 3367 reserved pages [ 122.576767] 2235 slab pages [ 122.576769] 263437 pages shared [ 122.576770] 0 pages swap cached [ 122.581730] syslog-ng: page allocation failure: order:0, mode:0x201db [ 122.581734] CPU: 0 PID: 2046 Comm: syslog-ng Tainted: G O 3.10.72-gca334de #1 [ 122.581747] [] (unwind_backtrace+0x0/0xf8) from [] (show_stack+0x10/0x14) [ 122.581756] [] (show_stack+0x10/0x14) from [] (warn_alloc_failed+0xa8/0x120) [ 122.581765] [] (warn_alloc_failed+0xa8/0x120) from [] (__alloc_pages_nodemask+0x614/0x8c8) [ 122.581772] [] (__alloc_pages_nodemask+0x614/0x8c8) from [] (filemap_fault+0x1c8/0x410) [ 122.581779] [] (filemap_fault+0x1c8/0x410) from [] (__do_fault+0x68/0x560) [ 122.581786] [] (__do_fault+0x68/0x560) from [] (handle_pte_fault+0x90/0x1164) [ 122.581792] [] (handle_pte_fault+0x90/0x1164) from [] (handle_mm_fault+0xec/0x154) [ 122.581801] [] (handle_mm_fault+0xec/0x154) from [] (do_page_fault.part.7+0x1d8/0x36c) [ 122.581809] [] (do_page_fault.part.7+0x1d8/0x36c) from [] (do_page_fault+0x60/0x6c) [ 122.581816] [] (do_page_fault+0x60/0x6c) from [] (do_PrefetchAbort+0x2c/0x94) [ 122.581822] [] (do_PrefetchAbort+0x2c/0x94) from [] (ret_from_exception+0x0/0x10) [ 122.581824] Exception stack(0xdd65dfb0 to 0xdd65dff8) [ 122.581828] dfa0: 00000001 bed63b98 00000008 0010efc8 [ 122.581833] dfc0: bed63b98 00000455 bed63c48 000000fc bed63c20 00013800 000003e8 bed63c44 [ 122.581837] dfe0: 00000000 bed63b94 b6eedbdc b6cd054c 600f0010 ffffffff [ 122.581838] Mem-info: [ 122.581841] DMA per-cpu: [ 122.581844] CPU 0: hi: 186, btch: 31 usd: 39 [ 122.581847] CPU 1: hi: 186, btch: 31 usd: 153 [ 122.581849] CPU 2: hi: 186, btch: 31 usd: 177 [ 122.581852] CPU 3: hi: 186, btch: 31 usd: 65 [ 122.581860] active_anon:123198 inactive_anon:3876 isolated_anon:0 [ 122.581860] active_file:25 inactive_file:9 isolated_file:32 [ 122.581860] unevictable:0 dirty:0 writeback:0 unstable:0 [ 122.581860] free:703 slab_reclaimable:902 slab_unreclaimable:2876 [ 122.581860] mapped:48 shmem:4573 pagetables:787 bounce:0 [ 122.581860] free_cma:9 [ 122.581870] DMA free:2812kB min:2900kB low:3624kB high:4348kB active_anon:492792kB inactive_anon:15504kB active_file:100kB inactive_file:36kB unevictable:0kB isolated(anon):0kB isolated(file):128kB prs [ 122.581877] lowmem_reserve[]: 0 0 0 0 [ 122.581899] DMA: 1*4kB (R) 0*8kB 0*16kB 1*32kB (R) 0*64kB 0*128kB 1*256kB (R) 1*512kB (R) 0*1024kB 1*2048kB (R) 0*4096kB = 2852kB [ 122.581900] 4665 total pagecache pages [ 122.586573] 139264 pages of RAM [ 122.586574] 1473 free pages [ 122.586575] 3367 reserved pages [ 122.586577] 2235 slab pages [ 122.586579] 263437 pages shared [ 122.586580] 0 pages swap cached [ 122.591718] syslog-ng: page allocation failure: order:0, mode:0x201db [ 122.591723] CPU: 0 PID: 2046 Comm: syslog-ng Tainted: G O 3.10.72-gca334de #1 [ 122.591736] [] (unwind_backtrace+0x0/0xf8) from [] (show_stack+0x10/0x14) [ 122.591746] [] (show_stack+0x10/0x14) from [] (warn_alloc_failed+0xa8/0x120) [ 122.591754] [] (warn_alloc_failed+0xa8/0x120) from [] (__alloc_pages_nodemask+0x614/0x8c8) [ 122.591761] [] (__alloc_pages_nodemask+0x614/0x8c8) from [] (filemap_fault+0x1c8/0x410) [ 122.591768] [] (filemap_fault+0x1c8/0x410) from [] (__do_fault+0x68/0x560) [ 122.591775] [] (__do_fault+0x68/0x560) from [] (handle_pte_fault+0x90/0x1164) [ 122.591781] [] (handle_pte_fault+0x90/0x1164) from [] (handle_mm_fault+0xec/0x154) [ 122.591789] [] (handle_mm_fault+0xec/0x154) from [] (do_page_fault.part.7+0x1d8/0x36c) [ 122.591797] [] (do_page_fault.part.7+0x1d8/0x36c) from [] (do_page_fault+0x60/0x6c) [ 122.591804] [] (do_page_fault+0x60/0x6c) from [] (do_PrefetchAbort+0x2c/0x94) [ 122.591810] [] (do_PrefetchAbort+0x2c/0x94) from [] (ret_from_exception+0x0/0x10) [ 122.591812] Exception stack(0xdd65dfb0 to 0xdd65dff8) [ 122.591816] dfa0: 00000001 bed63b98 00000008 0010efc8 [ 122.591821] dfc0: bed63b98 00000455 bed63c48 000000fc bed63c20 00013800 000003e8 bed63c44 [ 122.591824] dfe0: 00000000 bed63b94 b6eedbdc b6cd054c 600f0010 ffffffff [ 122.591826] Mem-info: [ 122.591828] DMA per-cpu: [ 122.591831] CPU 0: hi: 186, btch: 31 usd: 39 [ 122.591833] CPU 1: hi: 186, btch: 31 usd: 153 [ 122.591836] CPU 2: hi: 186, btch: 31 usd: 177 [ 122.591838] CPU 3: hi: 186, btch: 31 usd: 65 [ 122.591846] active_anon:123198 inactive_anon:3876 isolated_anon:0 [ 122.591846] active_file:25 inactive_file:9 isolated_file:32 [ 122.591846] unevictable:0 dirty:0 writeback:0 unstable:0 [ 122.591846] free:703 slab_reclaimable:902 slab_unreclaimable:2876 [ 122.591846] mapped:48 shmem:4573 pagetables:787 bounce:0 [ 122.591846] free_cma:9 [ 122.591857] DMA free:2812kB min:2900kB low:3624kB high:4348kB active_anon:492792kB inactive_anon:15504kB active_file:100kB inactive_file:36kB unevictable:0kB isolated(anon):0kB isolated(file):128kB prs [ 122.591864] lowmem_reserve[]: 0 0 0 0 [ 122.591887] DMA: 1*4kB (R) 0*8kB 0*16kB 1*32kB (R) 0*64kB 0*128kB 1*256kB (R) 1*512kB (R) 0*1024kB 1*2048kB (R) 0*4096kB = 2852kB [ 122.591889] 4665 total pagecache pages [ 122.596425] 139264 pages of RAM [ 122.596427] 1473 free pages [ 122.596428] 3367 reserved pages [ 122.596430] 2235 slab pages [ 122.596431] 263437 pages shared [ 122.596433] 0 pages swap cached [ 122.601709] syslog-ng: page allocation failure: order:0, mode:0x201db [ 122.601714] CPU: 0 PID: 2046 Comm: syslog-ng Tainted: G O 3.10.72-gca334de #1 [ 122.601728] [] (unwind_backtrace+0x0/0xf8) from [] (show_stack+0x10/0x14) [ 122.601738] [] (show_stack+0x10/0x14) from [] (warn_alloc_failed+0xa8/0x120) [ 122.601748] [] (warn_alloc_failed+0xa8/0x120) from [] (__alloc_pages_nodemask+0x614/0x8c8) [ 122.601755] [] (__alloc_pages_nodemask+0x614/0x8c8) from [] (filemap_fault+0x1c8/0x410) [ 122.601763] [] (filemap_fault+0x1c8/0x410) from [] (__do_fault+0x68/0x560) [ 122.601769] [] (__do_fault+0x68/0x560) from [] (handle_pte_fault+0x90/0x1164) [ 122.601776] [] (handle_pte_fault+0x90/0x1164) from [] (handle_mm_fault+0xec/0x154) [ 122.601784] [] (handle_mm_fault+0xec/0x154) from [] (do_page_fault.part.7+0x1d8/0x36c) [ 122.601792] [] (do_page_fault.part.7+0x1d8/0x36c) from [] (do_page_fault+0x60/0x6c) [ 122.601799] [] (do_page_fault+0x60/0x6c) from [] (do_PrefetchAbort+0x2c/0x94) [ 122.601805] [] (do_PrefetchAbort+0x2c/0x94) from [] (ret_from_exception+0x0/0x10) [ 122.601808] Exception stack(0xdd65dfb0 to 0xdd65dff8) [ 122.601812] dfa0: 00000001 bed63b98 00000008 0010efc8 [ 122.601816] dfc0: bed63b98 00000455 bed63c48 000000fc bed63c20 00013800 000003e8 bed63c44 [ 122.601820] dfe0: 00000000 bed63b94 b6eedbdc b6cd054c 600f0010 ffffffff [ 122.601822] Mem-info: [ 122.601823] DMA per-cpu: [ 122.601826] CPU 0: hi: 186, btch: 31 usd: 39 [ 122.601828] CPU 1: hi: 186, btch: 31 usd: 153 [ 122.601831] CPU 2: hi: 186, btch: 31 usd: 177 [ 122.601834] CPU 3: hi: 186, btch: 31 usd: 65 [ 122.601842] active_anon:123198 inactive_anon:3876 isolated_anon:0 [ 122.601842] active_file:25 inactive_file:9 isolated_file:32 [ 122.601842] unevictable:0 dirty:0 writeback:0 unstable:0 [ 122.601842] free:703 slab_reclaimable:902 slab_unreclaimable:2876 [ 122.601842] mapped:48 shmem:4573 pagetables:787 bounce:0 [ 122.601842] free_cma:9 [ 122.601854] DMA free:2812kB min:2900kB low:3624kB high:4348kB active_anon:492792kB inactive_anon:15504kB active_file:100kB inactive_file:36kB unevictable:0kB isolated(anon):0kB isolated(file):128kB prs [ 122.601860] lowmem_reserve[]: 0 0 0 0 [ 122.601883] DMA: 1*4kB (R) 0*8kB 0*16kB 1*32kB (R) 0*64kB 0*128kB 1*256kB (R) 1*512kB (R) 0*1024kB 1*2048kB (R) 0*4096kB = 2852kB [ 122.601884] 4665 total pagecache pages [ 122.606313] 139264 pages of RAM [ 122.606314] 1473 free pages [ 122.606316] 3367 reserved pages [ 122.606317] 2235 slab pages [ 122.606318] 263437 pages shared [ 122.606320] 0 pages swap cached [ 122.611713] syslog-ng: page allocation failure: order:0, mode:0x201db [ 122.611717] CPU: 0 PID: 2046 Comm: syslog-ng Tainted: G O 3.10.72-gca334de #1 [ 122.611730] [] (unwind_backtrace+0x0/0xf8) from [] (show_stack+0x10/0x14) [ 122.611740] [] (show_stack+0x10/0x14) from [] (warn_alloc_failed+0xa8/0x120) [ 122.611748] [] (warn_alloc_failed+0xa8/0x120) from [] (__alloc_pages_nodemask+0x614/0x8c8) [ 122.611755] [] (__alloc_pages_nodemask+0x614/0x8c8) from [] (filemap_fault+0x1c8/0x410) [ 122.611761] [] (filemap_fault+0x1c8/0x410) from [] (__do_fault+0x68/0x560) [ 122.611768] [] (__do_fault+0x68/0x560) from [] (handle_pte_fault+0x90/0x1164) [ 122.611774] [] (handle_pte_fault+0x90/0x1164) from [] (handle_mm_fault+0xec/0x154) [ 122.611782] [] (handle_mm_fault+0xec/0x154) from [] (do_page_fault.part.7+0x1d8/0x36c) [ 122.611790] [] (do_page_fault.part.7+0x1d8/0x36c) from [] (do_page_fault+0x60/0x6c) [ 122.611797] [] (do_page_fault+0x60/0x6c) from [] (do_PrefetchAbort+0x2c/0x94) [ 122.611803] [] (do_PrefetchAbort+0x2c/0x94) from [] (ret_from_exception+0x0/0x10) [ 122.611806] Exception stack(0xdd65dfb0 to 0xdd65dff8) [ 122.611809] dfa0: 00000001 bed63b98 00000008 0010efc8 [ 122.611814] dfc0: bed63b98 00000455 bed63c48 000000fc bed63c20 00013800 000003e8 bed63c44 [ 122.611818] dfe0: 00000000 bed63b94 b6eedbdc b6cd054c 600f0010 ffffffff [ 122.611820] Mem-info: [ 122.611821] DMA per-cpu: [ 122.611824] CPU 0: hi: 186, btch: 31 usd: 39 [ 122.611827] CPU 1: hi: 186, btch: 31 usd: 153 [ 122.611829] CPU 2: hi: 186, btch: 31 usd: 177 [ 122.611831] CPU 3: hi: 186, btch: 31 usd: 65 [ 122.611840] active_anon:123198 inactive_anon:3876 isolated_anon:0 [ 122.611840] active_file:25 inactive_file:9 isolated_file:32 [ 122.611840] unevictable:0 dirty:0 writeback:0 unstable:0 [ 122.611840] free:703 slab_reclaimable:902 slab_unreclaimable:2876 [ 122.611840] mapped:48 shmem:4573 pagetables:787 bounce:0 [ 122.611840] free_cma:9 [ 122.611850] DMA free:2812kB min:2900kB low:3624kB high:4348kB active_anon:492792kB inactive_anon:15504kB active_file:100kB inactive_file:36kB unevictable:0kB isolated(anon):0kB isolated(file):128kB prs [ 122.611857] lowmem_reserve[]: 0 0 0 0 [ 122.611879] DMA: 1*4kB (R) 0*8kB 0*16kB 1*32kB (R) 0*64kB 0*128kB 1*256kB (R) 1*512kB (R) 0*1024kB 1*2048kB (R) 0*4096kB = 2852kB [ 122.611881] 4665 total pagecache pages [ 122.616280] 139264 pages of RAM [ 122.616281] 1473 free pages [ 122.616282] 3367 reserved pages [ 122.616284] 2235 slab pages [ 122.616285] 263437 pages shared [ 122.616288] 0 pages swap cached [ 122.621731] syslog-ng: page allocation failure: order:0, mode:0x201db [ 122.621736] CPU: 0 PID: 2046 Comm: syslog-ng Tainted: G O 3.10.72-gca334de #1 [ 122.621750] [] (unwind_backtrace+0x0/0xf8) from [] (show_stack+0x10/0x14) [ 122.621759] [] (show_stack+0x10/0x14) from [] (warn_alloc_failed+0xa8/0x120) [ 122.621767] [] (warn_alloc_failed+0xa8/0x120) from [] (__alloc_pages_nodemask+0x614/0x8c8) [ 122.621774] [] (__alloc_pages_nodemask+0x614/0x8c8) from [] (filemap_fault+0x1c8/0x410) [ 122.621781] [] (filemap_fault+0x1c8/0x410) from [] (__do_fault+0x68/0x560) [ 122.621787] [] (__do_fault+0x68/0x560) from [] (handle_pte_fault+0x90/0x1164) [ 122.621794] [] (handle_pte_fault+0x90/0x1164) from [] (handle_mm_fault+0xec/0x154) [ 122.621803] [] (handle_mm_fault+0xec/0x154) from [] (do_page_fault.part.7+0x1d8/0x36c) [ 122.621812] [] (do_page_fault.part.7+0x1d8/0x36c) from [] (do_page_fault+0x60/0x6c) [ 122.621818] [] (do_page_fault+0x60/0x6c) from [] (do_PrefetchAbort+0x2c/0x94) [ 122.621825] [] (do_PrefetchAbort+0x2c/0x94) from [] (ret_from_exception+0x0/0x10) [ 122.621828] Exception stack(0xdd65dfb0 to 0xdd65dff8) [ 122.621832] dfa0: 00000001 bed63b98 00000008 0010efc8 [ 122.621836] dfc0: bed63b98 00000455 bed63c48 000000fc bed63c20 00013800 000003e8 bed63c44 [ 122.621840] dfe0: 00000000 bed63b94 b6eedbdc b6cd054c 600f0010 ffffffff [ 122.621842] Mem-info: [ 122.621843] DMA per-cpu: [ 122.621846] CPU 0: hi: 186, btch: 31 usd: 39 [ 122.621849] CPU 1: hi: 186, btch: 31 usd: 153 [ 122.621851] CPU 2: hi: 186, btch: 31 usd: 177 [ 122.621853] CPU 3: hi: 186, btch: 31 usd: 65 [ 122.621861] active_anon:123198 inactive_anon:3876 isolated_anon:0 [ 122.621861] active_file:25 inactive_file:9 isolated_file:32 [ 122.621861] unevictable:0 dirty:0 writeback:0 unstable:0 [ 122.621861] free:703 slab_reclaimable:902 slab_unreclaimable:2876 [ 122.621861] mapped:48 shmem:4573 pagetables:787 bounce:0 [ 122.621861] free_cma:9 [ 122.621872] DMA free:2812kB min:2900kB low:3624kB high:4348kB active_anon:492792kB inactive_anon:15504kB active_file:100kB inactive_file:36kB unevictable:0kB isolated(anon):0kB isolated(file):128kB prs [ 122.621879] lowmem_reserve[]: 0 0 0 0 [ 122.621901] DMA: 1*4kB (R) 0*8kB 0*16kB 1*32kB (R) 0*64kB 0*128kB 1*256kB (R) 1*512kB (R) 0*1024kB 1*2048kB (R) 0*4096kB = 2852kB [ 122.621902] 4665 total pagecache pages [ 122.626290] 139264 pages of RAM [ 122.626291] 1473 free pages [ 122.626292] 3367 reserved pages [ 122.626294] 2235 slab pages [ 122.626295] 263437 pages shared [ 122.626296] 0 pages swap cached [ 124.022116] [] (dump_header.isra.14+0x78/0x198) from [] (oom_kill_process+0x368/0x3dc) [ 124.031742] [] (oom_kill_process+0x368/0x3dc) from [] (out_of_memory+0x2a4/0x2f8) [ 124.040936] [] (out_of_memory+0x2a4/0x2f8) from [] (__alloc_pages_nodemask+0x89c/0x8c8) [ 124.050650] [] (__alloc_pages_nodemask+0x89c/0x8c8) from [] (handle_pte_fault+0xab4/0x1164) [ 124.060709] [] (handle_pte_fault+0xab4/0x1164) from [] (handle_mm_fault+0xec/0x154) [ 124.070069] [] (handle_mm_fault+0xec/0x154) from [] (do_page_fault.part.7+0x1d8/0x36c) [ 124.079694] [] (do_page_fault.part.7+0x1d8/0x36c) from [] (do_page_fault+0x60/0x6c) [ 124.089059] [] (do_page_fault+0x60/0x6c) from [] (do_DataAbort+0x2c/0x94) [ 124.097561] [] (do_DataAbort+0x2c/0x94) from [] (__dabt_usr+0x34/0x40) [ 124.105796] Exception stack(0xdf9edfb0 to 0xdf9edff8) [ 124.110838] dfa0: 00002021 0001c179 380a1e88 00000000 [ 124.118980] dfc0: 00000000 b6e344c8 b6e344f8 3809fe70 000001ff 0001e198 0003d368 b6e34500 [ 124.127131] dfe0: b6e344c8 bed32640 b6dab351 b6da952e 00000030 ffffffff [ 124.133727] Mem-info: [ 124.135986] DMA per-cpu: [ 124.138507] CPU 0: hi: 186, btch: 31 usd: 39 [ 124.143280] CPU 1: hi: 186, btch: 31 usd: 153 [ 124.148046] CPU 2: hi: 186, btch: 31 usd: 177 [ 124.152817] CPU 3: hi: 186, btch: 31 usd: 65 [ 124.157589] active_anon:123209 inactive_anon:3876 isolated_anon:0 [ 124.157589] active_file:19 inactive_file:11 isolated_file:32 [ 124.157589] unevictable:0 dirty:0 writeback:0 unstable:0 [ 124.157589] free:721 slab_reclaimable:902 slab_unreclaimable:2876 [ 124.157589] mapped:59 shmem:4573 pagetables:795 bounce:0 [ 124.157589] free_cma:0 [ 124.188662] DMA free:2884kB min:2900kB low:3624kB high:4348kB active_anon:492836kB inactive_anon:15504kB active_file:76kB inactive_file:44kB unevictable:0kB isolated(anon):0kB isolated(file):128kB preo [ 124.230269] lowmem_reserve[]: 0 0 0 0 [ 124.233949] DMA: 1*4kB (R) 0*8kB 0*16kB 2*32kB (UR) 0*64kB 0*128kB 1*256kB (R) 1*512kB (R) 0*1024kB 1*2048kB (R) 0*4096kB = 2884kB [ 124.245757] 4608 total pagecache pages [ 124.254280] 139264 pages of RAM [ 124.257403] 1531 free pages [ 124.260176] 3367 reserved pages [ 124.263304] 2235 slab pages [ 124.266078] 263245 pages shared [ 124.269198] 0 pages swap cached [ 124.272328] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [ 124.280194] [ 1347] 0 1347 661 102 4 0 -1000 udevd [ 124.288085] [ 1961] 997 1961 588 38 4 0 0 dbus-daemon [ 124.296494] [ 1983] 0 1983 35826 62 69 0 0 mpmsrv [ 124.304471] [ 1999] 0 1999 472 41 4 0 0 rmServer.out [ 124.312965] [ 2003] 0 2003 467 16 3 0 0 telnetd [ 124.321027] [ 2006] 0 2006 376 22 3 0 0 lad_tci6638 [ 124.329425] [ 2013] 0 2013 660 104 4 0 -1000 udevd [ 124.337312] [ 2014] 0 2014 660 104 4 0 -1000 udevd [ 124.345200] [ 2018] 999 2018 483 48 4 0 0 rpcbind [ 124.353265] [ 2031] 998 2031 419 34 3 0 0 rpc.statd [ 124.361500] [ 2037] 0 2037 536 29 4 0 0 dropbear [ 124.369641] [ 2045] 0 2045 863 42 4 0 0 syslog-ng [ 124.377873] [ 2046] 0 2046 1591 219 6 0 0 syslog-ng [ 124.386105] [ 2182] 0 2182 657 63 4 0 0 login [ 124.393991] [ 2183] 0 2183 652 59 4 0 0 sh [ 124.401619] [ 2184] 0 2184 652 82 4 0 0 bash [ 124.409414] [ 2189] 0 2189 230529 121206 452 0 0 mpirun [ 124.417388] [ 2190] 0 2190 681 46 3 0 0 ssh [ 124.425106] [ 2191] 0 2191 105251 299 209 0 0 nbody [ 124.432993] Out of memory: Kill process 2189 (mpirun) score 868 or sacrifice child [ 124.440535] Killed process 2191 (nbody) total-vm:421004kB, anon-rss:1196kB, file-rss:0kB -------------------------------------------------------------------------- mpirun noticed that process rank 0 with PID 2191 on node DSP1 exited on signal 9 (Killed). -------------------------------------------------------------------------- INK send EDMA_TRANSFER (to a2800000 from a2000000, size=28160! 00000, size=28160! [DSP1:02191] BTL_HLINK no room in FIFO (send)! [DSP1:02191] BTL_HLINK no room in FIFO (send)! [DSP1:02191] BTL_HLINK no room in FIFO (send)! [DSP1:02191] BTL_HLINK no room in FIFO (send)! [DSP1:02191] BTL_HLINK no room in FIFO (send)! [DSP1:02191] BTL_HLINK no room in FIFO (send)! [DSP1:02191] BTL_HLINK no room in FIFO (send)! [DSP1:02191] BTL_HLINK no room in FIFO (send)! [DSP1:02191] BTL_HLINK no room in FIFO (send)! [DSP1:02191] BTL_HLINK no room in FIFO (send)! [DSP1:02191] BTL_HLINK no room in FIFO (send)! [DSP1:02191] BTL_HLINK no room in FIFO (send)! [DSP1:02191] BTL_HLINK no room in FIFO (send)! [DSP1:02191] BTL_HLINK no room in FIFO (send)! [DSP1:02191] BTL_HLINK no room in FIFO (send)! [DSP1:02191] BTL_HLINK no room in FIFO (send)! [DSP1:02191] BTL_HLINK no room in FIFO (send)! [DSP1:02191] BTL_HLINK no room in FIFO (send)! [DSP1:02191] BTL_HLINK no room in FIFO (send)! [DSP1:02191] BTL_HLINK no room in FIFO (send)! [DSP1:02191] BTL_HLINK no room in FIFO (send)! [DSP1:02191] BTL_HLINK no room in FIFO (send)! [DSP1:02191] BTL_HLINK no room in FIFO (send)! [DSP1:02191] BTL_HLINK no room in FIFO (send)! [DSP1:02191] BTL_HLINK no room in FIFO (send)! [DSP1:02191] BTL_HLINK no room in FIFO (send)! [DSP1:02191] BTL_HLINK no room in FIFO (send)! [DSP1:02191] BTL_HLINK no room in FIFO (send)! [DSP1:02191] BTL_HLINK no room in FIFO (send)! [DSP1:02191] BTL_HLINK no room in FIFO (send)! [DSP1:02191] BTL_HLINK no room in FIFO (send)! [DSP1:02191] BTL_HLINK no room in FIFO (send)! [DSP1:02191] BTL_HLINK no room in FIFO (send)! [DSP1:02191] BTL_HLINK no room in FIFO (send)! [DSP1:02191] BTL_HLINK no room in FIFO (send)! [DSP1:02191] BTL_HLINK no room in FIFO (send)! [DSP1:02191] BTL_HLINK no room in FIFO (send)! [DSP1:02191] BTL_HLINK no room in FIFO (send)! [DSP1:02191] BTL_HLINK no room in FIFO (send)! [DSP1:02191] BTL_HLINK no room in FIFO (send)! [DSP1:02191] BTL_HLINK no room in FIFO (send)! [DSP1:02191] BTL_HLINK no room in FIFO (send)! [DSP1:02191] BTL_HLINK no room in FIFO (send)! [DSP1:02191] BTL_HLINK no room in FIFO (send)! [DSP1:02191] BTL_HLINK no room in FIFO (send)! [DSP1:02191] BTL_HLINK no room in FIFO (send)! [DSP1:02191] BTL_HLINK no room in FIFO (send)! [DSP1:02191] BTL_HLINK no room in FIFO (send)! [DSP1:02191] BTL_HLINK no room in FIFO (send)! [DSP1:02191] BTL_HLINK no room in FIFO (send)! [DSP1:02191] BTL_HLINK no room in FIFO (send)! [DSP1:02191] BTL_HLINK no room in FIFO (send)! [DSP1:02191] BTL_HLINK no room in FIFO (send)! [DSP1:02191] BTL_HLINK no room in FIFO (send)! [DSP1:02191] BTL_HLINK no room in FIFO (send)! [DSP1:02191] BTL_HLINK no room in FIFO (send)! [DSP1:02191] BTL_HLINK no room in FIFO (send)! [DSP1:02191] BTL_HLINK no room in FIFO (send)! [DSP1:02191] BTL_HLINK no room in FIFO (send)! [DSP1:02191] BTL_HLINK no room in FIFO (send)! [DSP1:02191] BTL_HLINK no room in FIFO (send)! [DSP1:02191] BTL_HLINK no room in FIFO (send)! [DSP1:02191] BTL_HLINK no room in FIFO (send)! [DSP1:02191] BTL_HLINK no room in FIFO (send)! [DSP1:02191] BTL_HLINK no room in FIFO (send)! [DSP1:02191] BTL_HLINK no room in FIFO (send)! [DSP1:02191] BTL_HLINK no room in FIFO (send)! [DSP1:02191] BTL_HLINK no room in FIFO (send)! [DSP1:02191] BTL_HLINK no room in FIFO (send)! [DSP1:02191] BTL_HLINK no room in FIFO (send)! [DSP1:02191] BTL_HLINK no room in FIFO (send)! [DSP1:02191] BTL_HLINK no room in FIFO (send)! [DSP1:02191] BTL_HLINK no room in FIFO (send)! [DSP1:02191] BTL_HLINK no room in FIFO (send)! [DSP1:02191] BTL_HLINK no room in FIFO (send)! [DSP1:02191] BTL_HLINK no room in FIFO (send)! [DSP1:02191] BTL_HLINK no room in FIFO (send)! [DSP1:02191] BTL_HLINK no room in FIFO (send)! [DSP1:02191] BTL_HLINK no room in FIFO (send)!