• State TI Thinks Resolved
  • Locked Locked
  • Replies 7 replies
  • Answers 2 answers
  • Subscribers 98 subscribers
  • Views 1006 views
  • Users 0 members are here
Support feedback
Options
Options
Related

This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

Original question:

Linux/TDA2: Cannot EDMA transfer between OSA_memAllocSR() and aligned_alloc() allocated buffers

Linux/TDA2: Cannot write to SSD from OSA_memAllocSR() buffers with O_DIRECT flag

Nick Cage
Intellectual 730 points

Part Number: TDA2

Tool/software: Linux

Hello, I'm using Processor SDK 3.03, building for custom board with TDA2 chip.

The usecase that we have fills 4 input buffers in NullSrc link from a file on SD card and forwards them to a link that we are using to write those buffers to an SSD. We have successfully written to 4 files opened without O_DIRECT flag using write() API. However, the CPU load is very high which is resulting in frame drops and we attributed that to heavy file system caching we suspect is being done in the background.

The next thing we did was try to open files with O_DIRECT flag on but there was nothing written to the files and the "dstat" shows no disk activity. We looked into it and found out that the write() is not being done at all, returning -1 instead. The errno error was "Bad address". We know that O_DIRECT usage requires proper buffer alignment but we made sure that they are aligned in several ways (printing the address to console as well as using this nifty trick here: http://www.titov.net/2006/01/02/using-o_largefile-or-o_direct-on-linux ) but to no avail, it keeps returning -1.

I just stumbled upon this post:  https://e2e.ti.com/support/dsp/davinci_digital_media_processors/f/716/t/242361#pi320098=3. It has something to do with the SR buffers being allocated in CMEM pool that the kernel cannot see or something like that. Is there any way to overlap a CMEM pool and the memory that kernel can see? Or is there something else entirely?

Thank you in advance.

Regards,

Nick

  • 0
    TI__Genius 14650 points
    Hi Nick,

    Is the write going through without O_DIRECT?

    Regards
    Shravan
  • 0 in reply to Shravan Karthik
    Intellectual 730 points
    Hi Shravan,

    Yes, I mentioned it in the original post.

    Regards,
    Nick
  • 0 in reply to Nick Cage
    TI__Genius 14650 points
    Hi Nick,

    Can you try the below changes and let me know if it works.

    1. Update the following entries in apps/build/tda2px/mem_segment_definition_linux.xs in VSDK
    SR1_BUFF_ECC_ASIL_SIZE = 1*MB;
    SR1_BUFF_ECC_QM_SIZE = 1*MB;
    SR1_BUFF_NON_ECC_ASIL_SIZE = 1*MB;


    2. Add the below line to links_fw/src/hlos/osa/src/osa_mem.c in VSDK
    "#define OSA_BEBUG_MEM"

    3. When you're writing to the file using o_direct, please print the physical and virtual address of the buffer being written.

    Once all these changes are made, please perform a clean build, and send over the logs.

    Regards
    Shravan
  • 0 in reply to Shravan Karthik
    Intellectual 730 points

    Hi Shravan,

    Unfortunately it did not work. Please find the log in the attachment.

    Buffer addresses are:

    [HOST] [HOST ] 261.854328 s: Buffer 0 virtual address = 0xa1fb3000
    [HOST] [HOST ] 261.854328 s: Buffer 0 physical address = 0x84500000
    [HOST] [HOST ] 261.857561 s: Buffer 1 virtual address = 0xa21ae000
    [HOST] [HOST ] 261.857561 s: Buffer 1 physical address = 0x846fb000
    [HOST] [HOST ] 261.860855 s: Buffer 2 virtual address = 0xa23a9000
    [HOST] [HOST ] 261.860855 s: Buffer 2 physical address = 0x848f6000
    [HOST] [HOST ] 261.864149 s: Buffer 3 virtual address = 0xa25a4000
    [HOST] [HOST ] 261.864149 s: Buffer 3 physical address = 0x84af1000

    I obtained the physical addresses using OSA_memVirt2Phys() API.

    Regards,

    Nick

    log.log

  • 0 in reply to Nick Cage
    TI__Genius 14650 points
    Hi Nick,

    Instead of using O_DIRECT, can you write to a pre-allocated file using fallocate. Allocate a file (of less than 2GB) using fallocate and try to write to the file without the O_DIRECT flag. You should see a performance bump (throughput should increase, CPU load must come down too).

    root# rm -rf test1.yuv
    root# fallocate -l 1G test1.yuv

    Can you please try this and let us know your observations?
    Writing using O_DIRECT from SR-1 isn't feasible (even though we mmap the region, and mark it as shared). SR-1 is analogous to I/O memory and providing this access to kernel mapped pages is a hack, and shouldn't be allowed.

    Regards
    Shravan
  • 0 in reply to Shravan Karthik
    Intellectual 730 points
    Hi Shravan,

    We have tried what you suggested and the load is still very high (~90%) and there are frame drops.

    What we would like to know is where does dma_alloc_coherent() allocate memory from. We have not been able to write to SSD with O_DIRECT flag from a buffer allocated via that API, too. Is that memory allocated from the same segment from which OSA_memAllocSR() allocates memory?

    Regards,
    Nick
  • 0 in reply to Nick Cage
    TI__Genius 14650 points
    Hi Nick,

    The dma_alloc_coherent memory allocates memory from the CMA pool if no device is specified.
    The OSA_memAllocSR sends a call from Linux to TI-RTOS (running on M4) to allocate memory from SR. TI-RTOS allocates memory and passes a physical address to Linux and the underlying drivers convert the physical address to a file-descriptor / virtual address. So when OSA_memAllocSR is called the allocation happens from TI-RTOS and not from Linux.

    The reason why both of these fail when writing with O_DIRECT is because, both are I/O memory. These memory regions are marked as reserved which prevents the kernel from using these memory regions. So there has to be a copy from user-space (where there is an explicit call to map the reserved region) to kernel space, and this is what increases the load.

    To enable O_DIRECT with I/O memory will involve I/O mem directly writing to kernel space, which is a security flaw.

    Regards
    Shravan