This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

OMAP-L138/Linux: Swapping code out on GPP

Part Number: OMAP-L138
Other Parts Discussed in Thread: DA8XX, DSD1791

Tool/software: Linux

We are using:
OMAP-L138 processor on a CriticalLink MityDSP-L138F board, with DSPLink MDK 2012-08-10
The GPP software runs on modified DaVinci Linux (DspLink) 1.65.0.3

We use DSPLink's NOTIFY_notify() to send a shared memory buffer address from the DSP to the GPP, which the GPP reads data from. Sometimes there are long delays or hangups in the registered Notify callback in the GPP code. The delays don't seem to be related to what the callback function is doing - they occur even if essentially all code is removed from the callback function - but the delays on the GPP side cause buffer corruption problems because the DSP is not delayed and the DSP begins to overwrite ping-pong buffered data before it can be read by the GPP. Can these delays in the GPP be related to the code running on the GPP being swapped out by Linux? Questions:
1) We see that the DSP seems to "send an interrupt" to the GPP with a call to DSPLink's HAL_intSend(). Is the callback on the GPP being called from code running in the context of an interrupt handler on the GPP?
2) The code that calls the callback simply calls through a function pointer to the callback function which is coded in our GPP source code and which is running on embedded Linux on the MityDSP-L138F board. Linux sometimes swaps executing code out. We see the buffer overwrite error approximately once an hour. Is it possible that the callback is sometimes being swapped out, so that it has to be swapped back in when the callback function is called, and something in the OS occasionally causes long delays in the swapping process? Is it possible to prevent the callback and library code from ever being swapped out? Is there some other possible cause of the delays in callback execution on the GPP?
3) Approximately 210,000 bytes of data is being sent from the DSP to the GPP per second, in 6K buffers allocated in shared memory (shared between DSP and GPP). We use only two buffers (ping-pong fashion) to send the data to the GPP. We can increase the number of shared memory buffers the DSP writes its data to increase the time after which an overwrite would occur, but DSP/GPP shared memory is limited. Given the problem described, is there an alternative solution that would prevent buffer overwrites by the DSP code?

  • Hi,

    Sorry for the additional questions, but since the linux you use is quite old (and with limited support), can you share the repository from which you've downloaded it as well as the GPP repository?
    What is the Kernel version & GPP version?

    Can you share the errors you see on the console & the full bootlog of your board?

    Best Regards,
    Yordan
  • Hi Yordan,

    Thank you! I am not sure what you mean by "GPP repository" or "GPP version". The GPP is a chip. What software do you mean, please? If you mean the DSPLink software, it is DSPLink version 1.65.00.03. We got it in a .tar file from:
    software-dl.ti.com/dsps/dsps_public_sw/sdo_sb/targetcontent/DSPLink/index.html

    Some of the DSPLink software runs on the GPP (ARM chip). For example, this is how we start the DSP from our GPP application, using DSPLink:

    void startDSP(char* app)
    {
     lpDspApp = new tcDspApp(); <-- DSPLink class
     printf("Loading file %s\n", app);
     lpDspApp->LoadApp(app);  <-- DSPLink call

     lpMessageInbound_cmd = new tcIPCInbound((char*)"GPPMSGQ1"); <- DSPLink
     if( NULL != lpMessageInbound_cmd) {
      lpMessageInbound_cmd->Register(handleInboundMessage, (void*)NULL)   <-- DSPLink
      lpMessageInbound_cmd->Initialize();
     }

     lpMessageOutbound = new tcIPCOutbound((char*)"DSPMSGQ0"); <-- DSPLink class

     NOTIFY_register(0,0,7, (FnNotifyCbck)notifyCallBack, NULL); <-- DSPLink
     NOTIFY_register(0,0,8, (FnNotifyCbck)commandCallback, NULL); <-- DSPLink

     while( !gbDone) // wait for DSP to finish initialization
      ;
    }

    This is the callback function in the GPP side that gets delayed:

    void notifyCallBack( Uint32 eventNo, Pvoid arg, Pvoid info )
    {
     unsigned int *dspAddr = (unsigned int *)info;

     if( dspAddr != 0 ){
      lpDspApp->ReadMemory( dspAddr, EVENT_TABLE_SIZE * sizeof(Event), eventData)

      // NOTE to Yordan - I put calls to clock_gettime() inside
      // dataSocketSend() and commented out everything
      // else in the function that could be taking more than a
      // few CPU instructions. The elapsed time could still be
      // extensive. I also timed the ReadMemory() call above;
      // it was very fast, relatively speaking, and doesn't
      // seem to be related to the occasional lengthy delay.

      // NOTE to Yordan - also - the time that I am measuring is *inside* the callback, so

      // it is not just a delay in the ISR on the GPP being triggered or the call to the callback function

      // being delayed - it seem as though the CPU stops executing the code inside the callback

      // function for a while. That's why I'm wondering about swapping.
      dataSocketSend( eventData, EVENT_TABLE_SIZE * sizeof(Event) );
     }
    }

    Additional info - we are sending and receiving across TCP/IP from our GPP application to the system the board is plugged into. The delay problem occurs even when the external application isn't running, i.e. when there is no TCP/IP communication going on. If it is related to sockets, then if the callback is inside an interrupt I would think the callback would not be held up no matter what the Linux TCP/IP is doing - that's part of the reason why I was asking about the interrupt context of the callback on the GPP.

    What do you mean by "the full bootlog"? I saw a URL that says the boot log is available from 'dmesg'. I am pasting the output I get from 'dmesg' at the bottom here. It looks like it starts from the beginning of a boot. Is that what you were looking for?

    There aren't any errors that appear on the console when we see this delay in the GPP. The software we wrote that runs on the GPP works, but in the callback function from Notify, if I use the Linux clock_gettime() function to obtain the time at the start and end of the callback, usually the elapsed time between those two calls is very short, but sometimes it is very long, I have seen over 1.0 seconds. The callback is called about 30 times a second on average, calls spaced fairly evenly apart, but the problem which is a long delay occurs on average around once an hour, but sometimes not until several hours after booting the board up. When the delay occurs, the data we get in buffers we get from our DSP software is simultaneously corrupted due to the buffers being overwritten in the DSP while the data is in the process of being sent in the GPP because the GPP suddenly took a break and didn't keep up with the data stream. So, no error in the console from the problem, just some debug code in our app that reports when our data has been corrupted, which would not be of help. (We see the data corruption occurring even when we do not call clock_gettime() to measure elapsed time in the callback, so getting the elapsed time is not itself causing the problem.) But do you mean errors in the console during bootup? There is no console active during bootup, though I can arrange for one to be active if that is what you are looking for - please let me know.

    It looks like what I said was the Linux version number was actually the DSPLink version; sorry about that!

    We are using Linux kernel 3.2.0:
    uname -a gives: "Linux mityomapl138 3.2.0 #1 PREEMPT Fri Aug 10 12:29:30 EDT 2012 armv5tejl GNU/Linux".
    Described here:
    support.criticallink.com/.../Linux_Kernel
    Git repo: git://support.criticallink.com/home/git/linux-davinci.git

    We are using the Angstrom linux distro:
    "cat /etc/issue" yields "The Angstrom Distribution; Angstrom v2012.05 - kernel"
    We do not download it from a repository - we copy it, along with the Linux kernel, from a "master" SD card, onto another SD card we insert into the board. We will try to find out where it originally came from, but the person who would have obtained it in 2012 may not be available. I see this repo referenced on a URL from 2016:

    git clone git://github.com/Angstrom-distribution/setup-scripts.git

    but I am not sure if you can get the v2012.05 version from that.

    Following my signature is the dmesg output - please let me know what you mean by the GPP version, etc. and I will try to give whatever other information I can.

    Thank you!
    Jim

    Linux version 3.2.0 (root@mitydsp) (gcc version 4.5.4 20120305 (prerelease) (GCC) ) #1 PREEMPT Fri Aug 10 12:29:30 EDT 2012
    CPU: ARM926EJ-S [41069265] revision 5 (ARMv5TEJ), cr=00053177
    CPU: VIVT data cache, VIVT instruction cache
    Machine: MityDSP-L138/MityARM-1808
    Ignoring unrecognised tag 0x42000101
    Memory policy: ECC disabled, Data cache writethrough
    DaVinci da850/omap-l138/am18x variant 0x1
    On node 0 totalpages: 24576
    free_area_init_node: node 0, pgdat c05bd310, node_mem_map c05eb000
      DMA zone: 192 pages used for memmap
      DMA zone: 0 pages reserved
      DMA zone: 24384 pages, LIFO batch:3
    pcpu-alloc: s0 r0 d32768 u32768 alloc=1*32768
    pcpu-alloc: [0] 0
    Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 24384
    Kernel command line: mem=96M console=ttyS1,115200n8 mtdparts=nand:128M(rootfs),-(userfs) root=/dev/mmcblk0p1 rw,rootwait ip=172.16.134.126
    PID hash table entries: 512 (order: -1, 2048 bytes)
    Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)
    Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)
    Memory: 96MB = 96MB total
    Memory: 91336k/91336k available, 6968k reserved, 0K highmem
    Virtual kernel memory layout:
        vector  : 0xffff0000 - 0xffff1000   (   4 kB)
        fixmap  : 0xfff00000 - 0xfffe0000   ( 896 kB)
        vmalloc : 0xc6800000 - 0xfea00000   ( 898 MB)
        lowmem  : 0xc0000000 - 0xc6000000   (  96 MB)
        modules : 0xbf000000 - 0xc0000000   (  16 MB)
          .text : 0xc0008000 - 0xc05347b4   (5298 kB)
          .init : 0xc0535000 - 0xc055d000   ( 160 kB)
          .data : 0xc055e000 - 0xc05bdd80   ( 384 kB)
           .bss : 0xc05bdda4 - 0xc05ea7b0   ( 179 kB)
    NR_IRQS:245
    Console: colour dummy device 80x30
    Calibrating delay loop... 148.88 BogoMIPS (lpj=744448)
    pid_max: default: 32768 minimum: 301
    Mount-cache hash table entries: 512
    CPU: Testing write buffer coherency: ok
    devtmpfs: initialized
    DaVinci: 144 gpio irqs
    print_constraints: dummy:
    NET: Registered protocol family 16
    baseboard_pre_init: Entered
    mityomapl138_setup_nand: using 8 bit data
    baseboard_init [IndustrialIO]...
    bio: create slab <bio-0> at 0
    SCSI subsystem initialized
    libata version 3.00 loaded.
    usbcore: registered new interface driver usbfs
    usbcore: registered new interface driver hub
    usbcore: registered new device driver usb
    print_constraints: VDCDC1: 1150 <--> 1350 mV at 1200 mV
    print_constraints: VDCDC2: 1800 mV
    print_constraints: VDCDC3: 1200 mV
    print_constraints: LDO1: 1800 mV
    print_constraints: LDO2: 2500 <--> 3300 mV at 3300 mV
    Advanced Linux Sound Architecture Driver Version 1.0.24.
    Switching to clocksource timer0_1
    musb-hdrc: version 6.0, ?dma?, otg (peripheral+host)
    Waiting for USB PHY clock good...
    musb-hdrc: ConfigData=0x06 (UTMI-8, dyn FIFOs, SoftConn)
    musb-hdrc: MHDRC RTL version 1.800
    musb-hdrc: setup fifo_mode 2
    musb-hdrc: 7/9 max ep, 2624/4096 memory
    musb-hdrc musb-hdrc: USB OTG mode controller at fee00000 using PIO, IRQ 58
    NET: Registered protocol family 2
    IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
    TCP established hash table entries: 4096 (order: 3, 32768 bytes)
    TCP bind hash table entries: 4096 (order: 2, 16384 bytes)
    TCP: Hash tables configured (established 4096 bind 4096)
    TCP reno registered
    UDP hash table entries: 256 (order: 0, 4096 bytes)
    UDP-Lite hash table entries: 256 (order: 0, 4096 bytes)
    NET: Registered protocol family 1
    RPC: Registered named UNIX socket transport module.
    RPC: Registered udp transport module.
    RPC: Registered tcp transport module.
    RPC: Registered tcp NFSv4.1 backchannel transport module.
    JFFS2 version 2.2. (NAND) © 2001-2006 Red Hat, Inc.
    msgmni has been set to 178
    io scheduler noop registered (default)
    start plist test
    end plist test
    Serial: 8250/16550 driver, 3 ports, IRQ sharing disabled
    serial8250.0: ttyS0 at MMIO 0x1c42000 (irq = 25) is a 16550A
    serial8250.0: ttyS1 at MMIO 0x1d0c000 (irq = 53) is a 16550A
    console [ttyS1] enabled
    serial8250.0: ttyS2 at MMIO 0x1d0d000 (irq = 61) is a 16550A
    brd: module loaded
    at24 1-0050: 256 byte 24c02 EEPROM, read-only, 0 bytes/write
    MityOMAPL138: Found MAC = 40:d8:55:17:c3:3d
    MityOMAPL138: Part Number = L138-FI-236-RL
    ONFI flash detected
    ONFI param page 0 valid
    NAND device: Manufacturer ID: 0x2c, Chip ID: 0xdc (Micron MT29F4G08ABADAWP)
    Bad block table found at page 262080, version 0x01
    Bad block table found at page 262016, version 0x01
    Creating 2 MTD partitions on "davinci_nand.1":
    0x000000000000-0x000008000000 : "rootfs"
    0x000008000000-0x000020000000 : "homefs"
    davinci_nand davinci_nand.1: controller rev. 2.5
    spi_davinci spi_davinci.1: DMA: supported
    spi_davinci spi_davinci.1: DMA: RX channel: 18, TX channel: 19, event queue: 0
    spi_davinci spi_davinci.1: registered master spi1
    spi spi1.0: setup mode 0, 8 bits/w, 30000000 Hz max --> 0
    m25p80 spi1.0: m25p64-nonjedec (8192 Kbytes)
    Creating 8 MTD partitions on "m25p80":
    0x000000000000-0x000000010000 : "ubl"
    0x000000010000-0x000000090000 : "u-boot"
    0x000000090000-0x0000000a0000 : "u-boot-env"
    0x0000000a0000-0x0000000b0000 : "periph-config"
    No LCD configured
    MII PHY configured
    0x0000000b0000-0x000000100000 : "reserved"
    0x000000100000-0x000000400000 : "kernel"
    0x000000400000-0x000000600000 : "fpga"
    0x000000600000-0x000000800000 : "spare"
    spi_davinci spi_davinci.1: registered child spi1.0
    spi spi1.1: setup mode 0, 8 bits/w, 10000000 Hz max --> 0
    spi_davinci spi_davinci.1: registered child spi1.1
    spi spi1.2: setup mode 0, 8 bits/w, 1000000 Hz max --> 0
    spi_davinci spi_davinci.1: registered child spi1.2
    spi_davinci spi_davinci.1: Controller at 0xfef0e000
    CAN device driver interface
    mcp251x spi1.1: setup mode 0, 8 bits/w, 10000000 Hz max --> 0
    mcp251x spi1.1: probed
    davinci_mdio davinci_mdio.0: davinci mdio revision 1.5
    davinci_mdio davinci_mdio.0: detected phy mask fffffff7
    davinci_mdio.0: probed
    davinci_mdio davinci_mdio.0: phy[3]: device 0:03, driver unknown
    ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
    ohci ohci.0: DA8xx OHCI
    ohci ohci.0: new USB bus registered, assigned bus number 1
    Waiting for USB PHY clock good...
    ohci ohci.0: irq 59, io mem 0x01e25000
    hub 1-0:1.0: USB hub found
    hub 1-0:1.0: 1 port detected
    Initializing USB Mass Storage driver...
    usbcore: registered new interface driver usb-storage
    USB Mass Storage support registered.
    mousedev: PS/2 mouse device common for all mice
    omap_rtc omap_rtc: rtc core: registered omap_rtc as rtc0
    omap_rtc: RTC power up reset detected
    omap_rtc: already running
    i2c /dev entries driver
    cpuidle: using governor ladder
    cpuidle: using governor menu
    davinci_mmc davinci_mmc.0: Using DMA, 4-bit mode
    usbcore: registered new interface driver usbhid
    usbhid: USB HID core driver
    mmc0: new high speed SDHC card at address aaaa
    mmcblk0: mmc0:aaaa SS04G 3.69 GiB
     mmcblk0: p1
    dsd1791 spi1.2: Failed to add route LLOUT->Line Out
    asoc: dsd1791 <-> davinci-mcasp.0 mapping ok
    ALSA device list:
      #0: MityDSP-L138 INDIO
    TCP cubic registered
    NET: Registered protocol family 17
    can: controller area network core (rev 20090105 abi 8)
    NET: Registered protocol family 29
    can: raw protocol (rev 20090105)
    regulator_init_complete: LDO2: incomplete constraints, leaving on
    regulator_init_complete: LDO1: incomplete constraints, leaving on
    regulator_init_complete: VDCDC3: incomplete constraints, leaving on
    regulator_init_complete: VDCDC2: incomplete constraints, leaving on
    regulator_init_complete: VDCDC1: incomplete constraints, leaving on
    console [netcon0] enabled
    netconsole: network logging started
    omap_rtc omap_rtc: setting system clock to 2001-10-26 12:04:10 UTC (1004097850)
    davinci_mdio davinci_mdio.0: resetting idled controller
    net eth0: attached PHY driver [Generic PHY] (mii_bus:phy_addr=0:03, id=40005201)
    PHY: 0:03 - Link is Up - 100/Full
    IP-Config: Guessing netmask 255.255.0.0
    IP-Config: Complete:
         device=eth0, addr=172.16.134.126, mask=255.255.0.0, gw=255.255.255.255,
         host=172.16.134.126, domain=, nis-domain=(none),
         bootserver=255.255.255.255, rootserver=255.255.255.255, rootpath=
    kjournald starting.  Commit interval 5 seconds
    EXT3-fs (mmcblk0p1): using internal journal
    ext3_orphan_cleanup: deleting unreferenced inode 96963
    EXT3-fs (mmcblk0p1): 1 orphan inode deleted
    EXT3-fs (mmcblk0p1): recovery complete
    EXT3-fs (mmcblk0p1): mounted filesystem with writeback data mode
    VFS: Mounted root (ext3 filesystem) on device 179:1.
    devtmpfs: mounted
    Freeing init memory: 160K
    NET: Registered protocol family 10
    Disabling lock debugging due to kernel taint
    >udevd[1088]: starting version 182
    DSPLINK Module (1.65.00.03) created on Date: Aug 10 2012 Time: 16:33:03
    Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
    svc: failed to register lockdv1 RPC service (errno 97).
    eth0: no IPv6 routers present
    fpga fpga_ctrl: loading the fpga_ctrl module.
    fpga fpga_ctrl: Entering fpga_ctrl_enum_devices
    fpga fpga_ctrl: Found Device ID 00-Base Module (01.01) at C687C000
    fpga fpga_ctrl: Found Device ID 250-Unknown (01.00) at C687C380
    fpga fpga_ctrl: Device ID 250-Unknown (01.00) at C687C380 -- managed by DSP
    fpga fpga_ctrl: Found Device ID 251-Unknown (01.00) at C687C400
    fpga fpga_ctrl: Device ID 251-Unknown (01.00) at C687C400 -- managed by DSP

  • Hello Jim,

    DSPLink is no longer supported on these forums and is EOL as indicated here:
    processors.wiki.ti.com/.../Category:DSPLink

    The currently supported IPC software option is IPC 3.x that comes bundled with Processor SDK, which can be downloaded here:
    www.ti.com/.../PROCESSOR-SDK-OMAPL138

    Regards,
    Sahin