About the stability of executing the application built by c6runapp on Beagle Board xm

Huan Hou

Other Parts Discussed in Thread: TMS320C6747

hi, all

I'm trying to debug why running the application built by C6run on Beagle board can make the Linux OS crash. Below is my test process.

First ssh connect to the beagle board, then execute the algorithm once and quit the ssh connection. Each time the execution of the ARM Linux application built by C6runapp-cc is like this "ssh root@${REMOTEADDR} \"cd ${REMOTE_DIR}; ${RUN_CMD}; \"", here the RUN_CMD is to run the application. The error message from the kernel is like this

[ 7770.725952] sh invoked oom-killer: gfp_mask=0xd0, order=2, oom_adj=0
[ 7770.732391] [<c003ab54>] (unwind_backtrace+0x0/0xdc) from [<c00b75d0>] (dump_header+0x5c/0x154)
[ 7770.741180] [<c00b75d0>] (dump_header+0x5c/0x154) from [<c00b7704>] (oom_kill_process+0x3c/0xdc)
[ 7770.750305] [<c00b7704>] (oom_kill_process+0x3c/0xdc) from [<c00b7d50>] (__out_of_memory+0x188/0x1ac)
[ 7770.759582] [<c00b7d50>] (__out_of_memory+0x188/0x1ac) from [<c00b7e08>] (out_of_memory+0x94/0xec)
[ 7770.768615] [<c00b7e08>] (out_of_memory+0x94/0xec) from [<c00badb4>] (__alloc_pages_nodemask+0x3e4/0x55c)
[ 7770.778259] [<c00badb4>] (__alloc_pages_nodemask+0x3e4/0x55c) from [<c00baf3c>] (__get_free_pages+0x10/0x3c)
[ 7770.788330] [<c00baf3c>] (__get_free_pages+0x10/0x3c) from [<c003cb98>] (get_pgd_slow+0x14/0xe0)
[ 7770.797180] [<c003cb98>] (get_pgd_slow+0x14/0xe0) from [<c006322c>] (mm_init+0x90/0xcc)
[ 7770.805236] [<c006322c>] (mm_init+0x90/0xcc) from [<c00637f8>] (dup_mm+0x64/0x524)
[ 7770.812866] [<c00637f8>] (dup_mm+0x64/0x524) from [<c006451c>] (copy_process+0x824/0xea8)
[ 7770.821105] [<c006451c>] (copy_process+0x824/0xea8) from [<c0064cfc>] (do_fork+0x15c/0x3ac)
[ 7770.829528] [<c0064cfc>] (do_fork+0x15c/0x3ac) from [<c0035f40>] (ret_fast_syscall+0x0/0x2c)
[ 7770.838043] Mem-info:
[ 7770.840332] Normal per-cpu:
[ 7770.843139] CPU    0: hi:   18, btch:   3 usd:   0
[ 7770.848144] active_anon:208 inactive_anon:412 isolated_anon:0
[ 7770.848144]  active_file:0 inactive_file:23 isolated_file:0
[ 7770.848175]  unevictable:0 dirty:0 writeback:0 unstable:0
[ 7770.848175]  free:1847 slab_reclaimable:386 slab_unreclaimable:947
[ 7770.848175]  mapped:16 shmem:59 pagetables:58 bounce:0
[ 7770.876342] Normal free:7388kB min:1140kB low:1424kB high:1708kB active_anon:832kB inactive_anon:1648kB active_file:0kB inactive_file:92kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:81280kB mlocked:0kB dirty:0kB writeback:0kB mapped:64kB shmem:236kB slab_reclaimable:1544kB slab_unreclaimable:3788kB kernel_stack:424kB pagetables:232kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[ 7770.914672] lowmem_reserve[]: 0 0
[ 7770.918029] Normal: 761*4kB 507*8kB 16*16kB 1*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 7388kB
[ 7770.928833] 82 total pagecache pages
[ 7770.932434] 0 pages in swap cache
[ 7770.935760] Swap cache stats: add 0, delete 0, find 0/0
[ 7770.941284] Free swap  = 0kB
[ 7770.944183] Total swap = 0kB
[ 7770.949279] 20480 pages of RAM
[ 7770.952423] 2704 free pages
[ 7770.955230] 10318 reserved pages
[ 7770.958465] 1333 slab pages
[ 7770.961273] 63 pages shared
[ 7770.964080] 0 pages swap cached
[ 7770.967254] Out of memory: kill process 10087 (sh) score 988 or a child
[ 7770.973907] Killed process 10089 (enc_c64_beagle_)

Firstly I doubt if the network connection cause this problem, but if I run an ARM-only application, it seems the Linux OS will not crash even I run the application by thousands of times which cost more than one day. But if I replace the ARM-only application with the APP built by C6run, the OS will crash with above error after one or two hours, is there any clue for solution or is it a C6run known bug?

The application is built by following steps, first, using cgt7.2.2 to build a library contains the core algorithm running on DSP side and then build the executive application using c6run. And the C6run version is 0.94.05.06, which is the default version for dvsdk 4.01.00.09.

Thanks a lot

over 13 years ago

0 Andrei Hertz over 12 years ago

Prodigy 10 points

Hi. I have a similar application running on my beagleboard XM rev C and I'm facing exactly the same problem for some months now.

Does anyone from TI know about this issue ?

First I thought this problem was due to a bug in the CMEM or C6run tool and therefore I recompiled the dvsdk 4.01.00.09 using the newer version of the C6run tool (0.98.03.03) and a newer version of the linuxutils (3.21.00.04). The system runs more stable than before but still crashes after running for aprox. 5 hours.

Basically it starts calling the oom-killer at some point until my ssh server is killed and I cannot access the board anymore.

Anyone else from the community has seen this issue before?

All the best,

Andrei Hertz.

0 kuyniu iyniuyn over 12 years ago in reply to Andrei Hertz

Prodigy 120 points

Hi, I just want to add that I also am having this problem, with a TI C6746 (TMS320C6747 board). It runs for a certain amount of time before invoking the oom killer. Could there be a memory leak in the CMEM or DSPLINK kernel modules?

0 Ramsey over 12 years ago in reply to kuyniu iyniuyn

TI__Genius 12025 points

Kuyniu,

Looks like you are running on a single core DSP chip. Are you running Linux has the OS? What is DSPLink talking to? Are you connected over PCI bus to another device? Please give us a little more detail.

Thanks
~Ramsey

0 kuyniu iyniuyn over 12 years ago in reply to Ramsey

Prodigy 120 points

Hi Ramsey

Yes, linux is running on the ARM mcu and I am using C6Run to run our actual program on the C6747 DSP. So there is an .axf file which is executed on the ARM, which sends the DSP code and other files to the DSP using DSPLink. CMEM is used to handle the shared memory pool between the two devices. My understanding is that this is a standard C6Run set up.

The linux distribution is Montavista linux (bundled with the CD included in the box).

Everything works at the beginning, but over multiple executions of the executable, eventually the oom-killer is invoked due to low memory.

0 Ramsey over 12 years ago in reply to kuyniu iyniuyn

TI__Genius 12025 points

Kuyniu,

There might be a memory leak in DSPLink. However, if the application does not free up all resources, then it might look like a DSPLink memory leak but it is really the application. Make sure the application is releasing all resources (i.e. close message queues, free messages, etc).

Look at /proc/meminfo before, during and after running your DSP program. Look at the VmallocUsed entry. It this value does not return to the start point, then there is a memory leak in either application or DSPLink.

Another test is to uninstall and reinstall the dsplinkk.ko module between DSP runs. This should reclaim any leaked memory.

If your results point at DSPLink, then you will need to debug the driver. Try tracking VmallocUsed from the driver.

~Ramsey

0 kuyniu iyniuyn over 12 years ago in reply to Ramsey

Prodigy 120 points

Hi Ramsey

Thanks for your response. I can confirm that every time I run the C6Run executable, a new line is added to vmallocinfo (about 19 memory pages). If I then run cat /proc/meminfo, I can see that vmallocused has gone up by about 80kb.

This happens with a test executable that I created, that simply mallocs and frees memory. So there is no memory leak in that test program.

The weird thing is that "rmmod dsplinkk; insmod dsplinkk" doesn't seem to free the memory either.

Processors

Processors forum

About the stability of executing the application built by c6runapp on Beagle Board xm