AM62A7: Performance Discrepancy Between DMA Heap and Aligned malloc Memory Access

Vishvajeet Godse

Part Number: AM62A7

Hi TI-Team

We are currently attempting to read frame buffer data from shared memory at address 0xa1800000 on the A53 processor, utilizing the open(/dev/mem,...) API. From the DMR5 side, the frame buffer data is written at intervals of approximately 33ms for DMS frames and 200ms for OMS frames.

The A53 reads from the same shared memory location and supplies this data to the perception library.However, after feeding frames to the perception library, we observe frame drops and a reduction in output FPS.Our debugging efforts revealed that when using aligned malloc, the expected performance is achieved. Specifically, reading a single frame from shared memory via DMA heap takes approximately 300ms, whereas accessing heap memory allocated with aligned malloc takes around 5ms.

Please note that the shared memory region 0xa1800000 is configured as cacheable, as the no_map property is not included in its region definition inside dts file. We seek your guidance to understand why the dma_heap_carveout area exhibits significantly higher read latency compared to heap memory allocated with aligned malloc, despite both being heap-based memory regions.Your insights will be valuable in optimizing our memory access strategy.Thank you for your support.

Best regards,

Vishvajeet

6 days ago

0 Hari Nagalla 5 days ago

TI__Mastermind 27270 points

I wonder if your usage of opening with /dev/mem and them mmap? is rending the buffer un-cached even though you have not provided 'no_map' property in the device tree. Can, you share your test code? And what is the buffer size you used for measuring the read latencies? In general aligned mallocs can give a little bit of performance lift when the buffers are allocated on cache line boundary.

0 Vishvajeet Godse 2 days ago in reply to Hari Nagalla

Expert 1370 points

Hi Hari,

Thanks for reply.

We are accessing shared memory using below code line-

fd = open( "/dev/mem", O_RDWR | O_SYNC );

after that we are mapping it using mmap API

map_base = mmap( nullptr,map_size,PROT_READ | PROT_WRITE, MAP_SHARED, fd,static_cast<long int>( target & ~( static_cast<uint64_t>( pagesize ) - 1U ) ) );

We have attached below text file for the test code which is used to read edgeAI shared memory and heap memory created using alligned_malloc-

Note- we are reading level 0 dms image of size 1638400 bytes[DMS_L0_size 1600 * 1024].

Test code and test logs-

Fullscreen testcode.txt Download

//================================ DMS L0 shared-mem vs heap-mem read latency benchmark ==========================//
        /* Measures are limited to the first 1000 DMS frames to avoid log flooding.
         * Both paths use a volatile byte-by-byte loop so the compiler cannot elide
         * the reads as dead code.  The XOR accumulator (u8Sink) is also volatile so
         * each load must be committed.  clock_gettime(CLOCK_MONOTONIC) gives
         * sub-microsecond resolution, far better than gettimeofday truncated to ms. */
#if 1
        {
            static uint32_t s_l0BenchCount = 0U;
            s_l0BenchCount++;
            if ( s_l0BenchCount <= 1000U )
            {
                /* Shared-memory base pointer for DMS L0 */
                const volatile uint8_t* const shmPtr =
                    static_cast<const volatile uint8_t*>(
                        static_cast<const void*>(
                            static_cast<const char*>( localBuffAddr )
                            + ( frameData[FrameIdxInfo.QueueCnt].l0_Frame_Addr - DMR5_START_ADDR ) ) );

                /* Heap pointer — slot.l0 was just populated from shared mem above */
                const volatile uint8_t* const heapPtr =
                    static_cast<const volatile uint8_t*>( static_cast<const void*>( slot.l0 ) );

                volatile uint8_t u8Sink = 0U; /* Accumulator prevents dead-code elimination */
                struct timespec tsStart, tsEnd;
                int64_t elapsedNs;

                /* --- Shared memory sequential read --- */
                ( void )clock_gettime( CLOCK_MONOTONIC, &tsStart );
                for ( uint32_t i = 0U; i < DMS_L0_size; i++ )
                {
                    u8Sink = static_cast<uint8_t>( shmPtr[i] );
                }
                ( void )clock_gettime( CLOCK_MONOTONIC, &tsEnd );
                elapsedNs = ( static_cast<int64_t>( tsEnd.tv_sec - tsStart.tv_sec ) * 1000000000LL )
                          + ( static_cast<int64_t>( tsEnd.tv_nsec ) - static_cast<int64_t>( tsStart.tv_nsec ) );
                const int64_t shmReadUs = elapsedNs / 1000LL;

                /* --- Heap memory sequential read --- */
                ( void )clock_gettime( CLOCK_MONOTONIC, &tsStart );
                for ( uint32_t i = 0U; i < DMS_L0_size; i++ )
                {
                    u8Sink = static_cast<uint8_t>( heapPtr[i] );
                }
                ( void )clock_gettime( CLOCK_MONOTONIC, &tsEnd );
                elapsedNs = ( static_cast<int64_t>( tsEnd.tv_sec - tsStart.tv_sec ) * 1000000000LL )
                          + ( static_cast<int64_t>( tsEnd.tv_nsec ) - static_cast<int64_t>( tsStart.tv_nsec ) );
                const int64_t heapReadUs = elapsedNs / 1000LL;

                printf( "[L0 ReadBench #%u] DMS_L0 (%u bytes): ShmRead=%lld us  HeapRead=%lld us  Diff(shm-heap)=%lld us\n",
                        s_l0BenchCount,
                        DMS_L0_size,
                        static_cast<long long>( shmReadUs ),
                        static_cast<long long>( heapReadUs ),
                        static_cast<long long>( shmReadUs - heapReadUs ) );
            }
        }
#endif
        //==========================================================================================================//

Can TI confirm the memory attributes applied when a reserved-memory shared DDR region is accessed through /dev/mem + mmap() on Linux A53?

Is there a possibility that the resulting mapping is non-cacheable or strongly ordered despite the reserved-memory region being defined without the no-map property?

Regards,

Vishvajeet

0 Hari Nagalla 16 hours ago in reply to Vishvajeet Godse

TI__Mastermind 27270 points

Can you try without the O_SYNC flag in mmap call? The O_SYNC flag ensures to flush the page caches immediately on writes. But it should not affect the reads as such.

Processors

Processors forum

AM62A7: Performance Discrepancy Between DMA Heap and Aligned malloc Memory Access