• State TI Thinks Resolved
  • Locked Locked
  • Replies 9 replies
  • Answers 1 answer
  • Subscribers 101 subscribers
  • Views 2652 views
  • Users 0 members are here
Support feedback
Options
Options
Similar topics

This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

Linux/TMS320DM8148: Kernel panic on heavy UDP load

Farrah Rashid
Intellectual 340 points
Part Number: TMS320DM8148

Tool/software: Linux

I'm using DVR_RDK 4.1 to receive uncompressed video data.  The network stack size is set as follows:

  • sysctl -w net.core.rmem_max=33554432
  • sysctl -w net.core.wmem_max=33554432
  • sysctl -w net.core.rmem_default=33554432
  • sysctl -w net.core.wmem_default=33554432
  • sysctl -w net.ipv4.udp_mem='4096 87380 33554432'
  • sysctl -w net.ipv4.route.flush=1

Each packet is 1300 bytes (UDP) and each stream is transmitted at a rate of 12800 packets per second.  It works with 2 streams but I get a page allocation failure if I add a third one.  How do I fix this?

vidapp: page allocation failure. order:0, mode:0x4020
Backtrace:
[<c004cfc4>] (dump_backtrace+0x0/0x110) from [<c035b48c>] (dump_stack+0x18/0x1c)
 r6:c04b5ee0 r5:00000000 r4:00004020 r3:60000193
[<c035b474>] (dump_stack+0x0/0x1c) from [<c00a95c0>] (__alloc_pages_nodemask+0x4fc/0x560)
[<c00a90c4>] (__alloc_pages_nodemask+0x0/0x560) from [<c00c67cc>] (new_slab+0x7c/0x200)
[<c00c6750>] (new_slab+0x0/0x200) from [<c00c6e44>] (__slab_alloc.clone.66+0x114/0x1e0)
 r8:c02cde60 r7:00000020 r6:00000000 r5:c4802400 r4:00000000
r3:003fffff
[<c00c6d30>] (__slab_alloc.clone.66+0x0/0x1e0) from [<c00c8360>] (__kmalloc_track_caller+0x84/0xc8)
 r8:c02cde60 r7:00000000 r6:a0000113 r5:00000020 r4:c4802400
r3:c05f2060
[<c00c82dc>] (__kmalloc_track_caller+0x0/0xc8) from [<c02cd7e0>] (__alloc_skb+0x58/0xe8)
 r8:c02cde60 r7:00000020 r6:000006c0 r5:c4802100 r4:c3e96180
r3:c05f2018
[<c02cd788>] (__alloc_skb+0x0/0xe8) from [<c02cde60>] (__netdev_alloc_skb+0x24/0x4c)
[<c02cde3c>] (__netdev_alloc_skb+0x0/0x4c) from [<c0274f88>] (cpsw_rx_handler+0xc0/0x13c)
 r4:c4916000 r3:00000003
[<c0274ec8>] (cpsw_rx_handler+0x0/0x13c) from [<c0270f10>] (__cpdma_chan_free+0x88/0x8c)
 r7:00020000 r6:c3977f40 r5:c3978380 r4:60000113
[<c0270e88>] (__cpdma_chan_free+0x0/0x8c) from [<c0271010>] (__cpdma_chan_process+0xfc/0x110)
[<c0270f14>] (__cpdma_chan_process+0x0/0x110) from [<c0271054>] (cpdma_chan_process+0x30/0x54)
 r7:00000000 r6:00000040 r5:00000001 r4:c3978380
[<c0271024>] (cpdma_chan_process+0x0/0x54) from [<c0274078>] (cpsw_poll+0x34/0xa0)
 r6:00000040 r5:c4916370 r4:c4916360 r3:c0274044
[<c0274044>] (cpsw_poll+0x0/0xa0) from [<c02d4f7c>] (net_rx_action+0x6c/0x15c)
 r8:00000040 r7:0000010e r6:00000001 r5:c04b5f40 r4:c4916370
r3:c0274044
[<c02d4f10>] (net_rx_action+0x0/0x15c) from [<c0076348>] (__do_softirq+0x84/0x114)
[<c00762c4>] (__do_softirq+0x0/0x114) from [<c0076738>] (irq_exit+0x48/0x98)
[<c00766f0>] (irq_exit+0x0/0x98) from [<c003f07c>] (asm_do_IRQ+0x7c/0x9c)
[<c003f000>] (asm_do_IRQ+0x0/0x9c) from [<c035d67c>] (__irq_usr+0x3c/0xa0)
Exception stack(0xc1fe9fb0 to 0xc1fe9ff8)
9fa0:                                     00000001 00000514 00000514 00000001
9fc0: 50ec84d8 50ec8490 4013f550 00000152 003d0f00 4014b3d8 00000000 50ec7e24
9fe0: 50ec7e0c 50ec78e8 00010cf0 000106a0 60000010 ffffffff
 r5:fa200000 r4:ffffffff
Mem-info:
Normal per-cpu:
CPU    0: hi:   18, btch:   3 usd:   3
active_anon:1083 inactive_anon:22 isolated_anon:0
 active_file:344 inactive_file:680 isolated_file:32
 unevictable:0 dirty:0 writeback:0 unstable:0
 free:104 slab_reclaimable:343 slab_unreclaimable:5107
 mapped:477 shmem:50 pagetables:111 bounce:0
Normal free:416kB min:1112kB low:1388kB high:1668kB active_anon:4332kB inactive_anon:88kB active_file:1376kB inactive_file:2720kB unevictable:0kB isolated(anon):0kB isolated(file):128kB present:77824kB mlocked:0kB dirty:0kB writeback:0kB mapped:1908kB shmem:200kB slab_reclaimable:1372kB slab_unreclaimable:20428kB kernel_stack:512kB pagetables:444kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:32 all_unreclaimable? no
lowmem_reserve[]: 0 0 0
Normal: 52*4kB 12*8kB 1*16kB 1*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 416kB
1092 total pagecache pages
32768 pages of RAM
224 free pages
14558 reserved pages
2344 slab pages
1777 pages shared
0 pages swap cached
SLUB: Unable to allocate memory on node -1 (gfp=0x20)
  cache: kmalloc-2048, object size: 2048, buffer size: 2048, default order: 2, min order: 0
  node 0: slabs: 1028, objs: 8134, free: 0

  • 0
    Intellectual 340 points

    Seems like it's a known but in the cpsw driver but I can't find a patch.  Here's the rest of the trace:

    ------------[ cut here ]------------
    WARNING: at drivers/net/cpsw.c:725 cpsw_rx_handler+0x134/0x140()
    Modules linked in: vpss osa_kermod syslink
    Backtrace:
    [<c004dbf4>] (dump_backtrace+0x0/0x110) from [<c0368b24>] (dump_stack+0x18/0x1c)
     r7:00000000 r6:c02817b0 r5:c044bb99 r4:000002d5
    [<c0368b0c>] (dump_stack+0x0/0x1c) from [<c0072914>] (warn_slowpath_common+0x54/0x6c)
    [<c00728c0>] (warn_slowpath_common+0x0/0x6c) from [<c0072950>] (warn_slowpath_null+0x24/0x2c)
     r9:0000000a r8:0000053e r7:00000000 r6:00000000 r5:c42a4600
    r4:c4916000
    [<c007292c>] (warn_slowpath_null+0x0/0x2c) from [<c02817b0>] (cpsw_rx_handler+0x134/0x140)
    [<c028167c>] (cpsw_rx_handler+0x0/0x140) from [<c027bb44>] (__cpdma_chan_free+0x88/0x8c)
     r8:00020000 r7:c42a4600 r6:c3996380 r5:c3995f40 r4:60000113
    [<c027babc>] (__cpdma_chan_free+0x0/0x8c) from [<c027bc54>] (__cpdma_chan_process+0x10c/0x124)
    [<c027bb48>] (__cpdma_chan_process+0x0/0x124) from [<c027bc9c>] (cpdma_chan_process+0x30/0x50)
    [<c027bc6c>] (cpdma_chan_process+0x0/0x50) from [<c02805ac>] (cpsw_poll+0x34/0xa0)
     r7:00000001 r6:00000040 r5:c4916360 r4:00000000
    [<c0280578>] (cpsw_poll+0x0/0xa0) from [<c02e14c4>] (net_rx_action+0x58/0x154)
     r9:0000000a r8:0000012a r7:00000001 r6:00000002 r5:00000040
    r4:c4916370
    [<c02e146c>] (net_rx_action+0x0/0x154) from [<c0077aa0>] (__do_softirq+0x80/0x108)
    [<c0077a20>] (__do_softirq+0x0/0x108) from [<c0077b70>] (irq_exit+0x48/0x94)
    [<c0077b28>] (irq_exit+0x0/0x94) from [<c003f080>] (asm_do_IRQ+0x80/0xa0)
    [<c003f000>] (asm_do_IRQ+0x0/0xa0) from [<c036ad7c>] (__irq_usr+0x3c/0xa0)
    Exception stack(0xc1c7ffb0 to 0xc1c7fff8)
    ffa0:                                     505ba8f8 001be604 0000022c 00000140
    ffc0: 505bb4d8 505bb490 40178550 00000152 003d0f00 401843d8 00000000 505ba8ac
    ffe0: 505babc0 505ba888 000121ac 00009ee8 20000010 ffffffff
     r5:fa200000 r4:ffffffff
    ---[ end trace 524c8480b22df6e7 ]---

  • 0 in reply to Farrah Rashid
    TI__Guru**** 170625 points
    Hi Farrah,

    Make sure you have in your code base all the ethernet/cpsw related patches from the below git tree:

    arago-project.org/.../

    See also the below wiki pages:

    processors.wiki.ti.com/.../TI81XX_PSP_04.04.00.02_Feature_Performance_Guide
    processors.wiki.ti.com/.../TI81XX_UDP_Performance_Improvement

    Regards,
    Pavel
  • 0 in reply to Pavel Botev
    Intellectual 340 points
    The kernel is 2.6.37 and already has the patches from the git tree.

    I've tried enabling interrupt pacing and increasing the network size queue and socket buffer queue but I still get the same failure.

    I haven't tried moving the DMA descriptors from internal BD Ram to DDR location. How do I do this?
  • 0 in reply to Farrah Rashid
    TI__Guru**** 170625 points
    Farrah,

    You can explore the latest CPSW driver and check if you can back port it in your code base:

    ti-processor-sdk-linux-am335x-evm-03.03.00.04/board-support/linux-4.4.41/drivers/net/ethernet/ti/cpsw.c

    For how to transfer DMA descriptors to external DDR3 memory, check DM814x TRM.

    Regards,
    Pavel
  • 0
    TI__Guru**** 170625 points

    Farrah Rashid said:

    Each packet is 1300 bytes (UDP) and each stream is transmitted at a rate of 12800 packets per second.  It works with 2 streams but I get a page allocation failure if I add a third one.  How do I fix this?

    vidapp: page allocation failure. order:0, mode:0x4020

    This might be also low memory problem, not performance problem. You can try to solve this providing more memory to linux kernel through the boot args. See the below pointers for more info:

    linux-dm81xx/mm/page_alloc.c
    linux-dm81xx/Documentation/fault-injection/fault-injection.txt

    Regards,
    Pavel

  • 0 in reply to Pavel Botev
    Intellectual 340 points
    I tried increasing the memory by increasing mem=128M to mem=256M but it caused the system not to boot properly. I suspect it is a low memory problem but expect that the Linux kernel would drop udp packets if it couldn't allocate space for them???
  • 0 in reply to Farrah Rashid
    TI__Guru**** 170625 points
    Farrah,

    UDP packets will drop if the receive buffer get full. But in your case you can not allocate memory (for receive buffer or for something else related to the 3rd UDP stream). For UDP packets drop, you should allocate receive buffer successful, then this buffer should get full too fast for your handler.

    May be you allocate too much space (very big buffers for 1st and 2nd stream) thus no space left for 3rd stream. Do you have any packet drop in 1st and/or 2nd stream? Can you try to decrease the memory used by 1st and 2nd stream?

    Regards,
    Pavel
  • 0 in reply to Pavel Botev
    Intellectual 340 points

    Hi Pavel,

    The design of the video application is based on the one provided with the DM8148 development kit i.e. new packets are allocated memory and assembled into a frame, then copied into the a memory buffer preallocated for display.  I tried changing the application so that the socket just listens on UDP port but does not process any of the data (the application has no UDP or display buffers allocated) but I still get a kernel dump.

    Farrah