AM62A7: Performance Discrepancy Between DMA Heap and Aligned malloc Memory Access

Vishvajeet Godse

Part Number: AM62A7

Hi TI-Team

We are currently attempting to read frame buffer data from shared memory at address 0xa1800000 on the A53 processor, utilizing the open(/dev/mem,...) API. From the DMR5 side, the frame buffer data is written at intervals of approximately 33ms for DMS frames and 200ms for OMS frames.

The A53 reads from the same shared memory location and supplies this data to the perception library.However, after feeding frames to the perception library, we observe frame drops and a reduction in output FPS.Our debugging efforts revealed that when using aligned malloc, the expected performance is achieved. Specifically, reading a single frame from shared memory via DMA heap takes approximately 300ms, whereas accessing heap memory allocated with aligned malloc takes around 5ms.

Please note that the shared memory region 0xa1800000 is configured as cacheable, as the no_map property is not included in its region definition inside dts file. We seek your guidance to understand why the dma_heap_carveout area exhibits significantly higher read latency compared to heap memory allocated with aligned malloc, despite both being heap-based memory regions.Your insights will be valuable in optimizing our memory access strategy.Thank you for your support.

Best regards,

Vishvajeet

2 months ago

0 Hari Nagalla 2 months ago

TI__Mastermind 27470 points

I wonder if your usage of opening with /dev/mem and them mmap? is rending the buffer un-cached even though you have not provided 'no_map' property in the device tree. Can, you share your test code? And what is the buffer size you used for measuring the read latencies? In general aligned mallocs can give a little bit of performance lift when the buffers are allocated on cache line boundary.

0 Vishvajeet Godse 2 months ago in reply to Hari Nagalla

Expert 1400 points

Hi Hari,

Thanks for reply.

We are accessing shared memory using below code line-

fd = open( "/dev/mem", O_RDWR | O_SYNC );

after that we are mapping it using mmap API

map_base = mmap( nullptr,map_size,PROT_READ | PROT_WRITE, MAP_SHARED, fd,static_cast<long int>( target & ~( static_cast<uint64_t>( pagesize ) - 1U ) ) );

We have attached below text file for the test code which is used to read edgeAI shared memory and heap memory created using alligned_malloc-

Note- we are reading level 0 dms image of size 1638400 bytes[DMS_L0_size 1600 * 1024].

Test code and test logs-

Fullscreen testcode.txt Download

//================================ DMS L0 shared-mem vs heap-mem read latency benchmark ==========================//
        /* Measures are limited to the first 1000 DMS frames to avoid log flooding.
         * Both paths use a volatile byte-by-byte loop so the compiler cannot elide
         * the reads as dead code.  The XOR accumulator (u8Sink) is also volatile so
         * each load must be committed.  clock_gettime(CLOCK_MONOTONIC) gives
         * sub-microsecond resolution, far better than gettimeofday truncated to ms. */
#if 1
        {
            static uint32_t s_l0BenchCount = 0U;
            s_l0BenchCount++;
            if ( s_l0BenchCount <= 1000U )
            {
                /* Shared-memory base pointer for DMS L0 */
                const volatile uint8_t* const shmPtr =
                    static_cast<const volatile uint8_t*>(
                        static_cast<const void*>(
                            static_cast<const char*>( localBuffAddr )
                            + ( frameData[FrameIdxInfo.QueueCnt].l0_Frame_Addr - DMR5_START_ADDR ) ) );

                /* Heap pointer — slot.l0 was just populated from shared mem above */
                const volatile uint8_t* const heapPtr =
                    static_cast<const volatile uint8_t*>( static_cast<const void*>( slot.l0 ) );

                volatile uint8_t u8Sink = 0U; /* Accumulator prevents dead-code elimination */
                struct timespec tsStart, tsEnd;
                int64_t elapsedNs;

                /* --- Shared memory sequential read --- */
                ( void )clock_gettime( CLOCK_MONOTONIC, &tsStart );
                for ( uint32_t i = 0U; i < DMS_L0_size; i++ )
                {
                    u8Sink = static_cast<uint8_t>( shmPtr[i] );
                }
                ( void )clock_gettime( CLOCK_MONOTONIC, &tsEnd );
                elapsedNs = ( static_cast<int64_t>( tsEnd.tv_sec - tsStart.tv_sec ) * 1000000000LL )
                          + ( static_cast<int64_t>( tsEnd.tv_nsec ) - static_cast<int64_t>( tsStart.tv_nsec ) );
                const int64_t shmReadUs = elapsedNs / 1000LL;

                /* --- Heap memory sequential read --- */
                ( void )clock_gettime( CLOCK_MONOTONIC, &tsStart );
                for ( uint32_t i = 0U; i < DMS_L0_size; i++ )
                {
                    u8Sink = static_cast<uint8_t>( heapPtr[i] );
                }
                ( void )clock_gettime( CLOCK_MONOTONIC, &tsEnd );
                elapsedNs = ( static_cast<int64_t>( tsEnd.tv_sec - tsStart.tv_sec ) * 1000000000LL )
                          + ( static_cast<int64_t>( tsEnd.tv_nsec ) - static_cast<int64_t>( tsStart.tv_nsec ) );
                const int64_t heapReadUs = elapsedNs / 1000LL;

                printf( "[L0 ReadBench #%u] DMS_L0 (%u bytes): ShmRead=%lld us  HeapRead=%lld us  Diff(shm-heap)=%lld us\n",
                        s_l0BenchCount,
                        DMS_L0_size,
                        static_cast<long long>( shmReadUs ),
                        static_cast<long long>( heapReadUs ),
                        static_cast<long long>( shmReadUs - heapReadUs ) );
            }
        }
#endif
        //==========================================================================================================//

Can TI confirm the memory attributes applied when a reserved-memory shared DDR region is accessed through /dev/mem + mmap() on Linux A53?

Is there a possibility that the resulting mapping is non-cacheable or strongly ordered despite the reserved-memory region being defined without the no-map property?

Regards,

Vishvajeet

0 Hari Nagalla 2 months ago in reply to Vishvajeet Godse

TI__Mastermind 27470 points

Can you try without the O_SYNC flag in mmap call? The O_SYNC flag ensures to flush the page caches immediately on writes. But it should not affect the reads as such.

0 Vishvajeet Godse 2 months ago in reply to Hari Nagalla

Expert 1400 points

Hi Hari Nagalla (214817),

Thank you for your response.

With the suggested changes, we have observed a significant improvement in the shared memory read latency. The shared memory read time has been reduced from approximately 300 ms to 5,761 µs, and is now nearly comparable to the expected memory access time, as shown in the log snapshot below.

With this change, the overall frame rate has increased to approximately 34 FPS. However, after processing and pushing around 100 frames to the SM module, we consistently encounter a segmentation fault.

Could you please help us analyze the possible cause of this issue and suggest any additional debugging steps or recommendations?

Thank you for your support.

Best regards,
Vishvajeet

+1 Hari Nagalla 2 months ago in reply to Vishvajeet Godse

TI__Mastermind 27470 points

Good to know, it improved the performance. I am not familiar with DMS/OMS applications and so can't guide you much. How ever, i suggest couple of things to check : a) Do, you see this segmentation fault only when O_SYNC flag is removed? If so, it hints that there may be some data coherency issue with these DMA buffers cached. Check kernel logs (dmesg) to see if any further details of this seg fault is available, like PC or data access error etc.

0 Vishvajeet Godse 2 months ago in reply to Hari Nagalla

Expert 1400 points

Hi Hari Nagalla ,

Thanks for your response.

We observe the segmentation fault only when the O_SYNC flag is removed. We also checked the dmesg logs, but apart from rpmsg_failure due to a reset, we do not see any more additional information (Please see attached dmesg logs for more information).

Could you please suggest any additional debugging steps or tools that would help narrow down the root cause? For example, are there any specific kernel configurations, memory validation checks, cache-coherency diagnostics, or tracing mechanisms that you would recommend for investigating this issue?

Any guidance would be greatly appreciated.

Regards,

Vishvajeet

0 Hari Nagalla 2 months ago in reply to Vishvajeet Godse

TI__Mastermind 27470 points

I was suggesting data coherency might be one reason for your segmentation faults as you are exchanging and updating buffers across the cores. But, it could any other things too. I suggest to review the thread buffer exchange protocol and how they are used. May be you need to log buffer addresses and see the validity of the buffers when segmentation fault occurs. Refer to any generic linux web content for debugging segmentation faults. https://www.doulos.com/knowhow/arm-embedded/embedded-linux-debugging-user-space-seg-faults/

0 Vishvajeet Godse 2 months ago in reply to Hari Nagalla

Expert 1400 points

Hi Hari Nagalla

Thanks for support, with the suggested solution issue is resolved now.

Regards,

Vishvajeet

Processors

Processors forum

AM62A7: Performance Discrepancy Between DMA Heap and Aligned malloc Memory Access