PCIe EDMA3 parallel transmit, EVM AM572x, TI-RTOS

I have asked the RTOS team to comment. They will post directly here.

github.com/.../evmAM572x_PCIe - my projects is here
linux driver for device in directory named "am572x_driver"

llya,

The host runs Linux and EP runs RTOS. Is host a Linux machine or an embedded Linux system? How the PCIE interface physically connnected? How the PCIE clock connected? Does the Linux host and AM572x EVM has its own seperate PCIE clock or the clock is sourced from Linux to AM572x EVM?

Does the host provide the PCIE enumeration? what is the established PCIE link speed? GEN1 or GEN2? Is the PCIE link stable (stay in L0 always) during traffic?

On AM572x side, are you running on A15 or C66x or M4? Given there are only two cores, how do you run the 8 EDMA transfer? Are they in parallel or one by one?

For the data missing:

For the OB writing, do you have any buffer allocated in Linux side and found out data was partially landed?

For the OB reading, AM572x side has some partial data in the buffer, is this the problem? The reading is from 0x2000_1000 region into OCMC, DDR?

Regards, Eric

Host is a linux machine(PC). I use this connector to connect EP to RC. The clock is not connected. There are only tx/rx.  The Linux host and AM572x EVM has its own seperate clock.

The host provide PCIe enumeration. It doesn't matter, what the link speed I use, but I use GEN1 for accuracy. The PCIe link is stable. The PCIe core in EVM AM572x doesn't generate an error interrupts during traffic. 

On AM572x side I am running on A15. A buffer consists of a blocks. In the EDMA channel params aCnt is block size, bCnt is block count in a buffer. I send a first block of a buffers one by one, and other blocks send in the EDMA Channel TCC CallBack function by call the EDMA3_DRV_enableTransfer function with trig_mode_manual. 

Yes, I have allocated buffers on Host side and AM572x side in DDR memory. Allocated buffer memory aligned to 0x1000 (4K) in AM572x side.

OB Regions allign to 4K.

If I use 4 channels with blockSize(bCnt) = 1536, it is broken too. But if I use 4 channels with blockSize = 2048 it is not broken.

If I use 8 channels it always broken, while I not allign OB Region addrs to 1M. Numbers of broken buffers is random. Only first part of buffer breaks.

I linked picture which illustrate EP to RC transmition.

Do you have test setup between two TI EVM? The Linux machine uses DSS clock and EVM uses seperate fixed clock, I am not sure if any issue under burst EDMA traffic. As you have 8 channels, is that correct at anytime you only have 1 channel in transfer? That is the ending of one channel triggers the transfer of the next channel? There is no 8 channels transferring at the same time. In the EDMA OPT field, do you use transfer completion or intermediate transfer completion to trigger an interrupt?

Regards, Eric

 

I don't have any TI EVM. If you send it me with free price, it will be very good :-)
If I have only 1 channel in transfer it work correct. No, I don't wait while one channel send buffer to start the next channel. I send blocks(arrays) one by one.
I use transfer completion and intermediate transfer completion to trigger an interrupt.

Very simplified code to show the logic:

uint32_t leftBlockCount[8]; /* There are left blocks count */
uint32_t edmaChannelIds[8]; /* edma channel ID by channel number[0-7]*/

void edmaChannelCallback(uint32_t tcc, EDMA3_RM_TccStatus status, void *appData) 
{ 
    uint32_t chNum = (uint32_t)appData;
    
    switch( status ) 
    { 
        case EDMA3_RM_XFER_COMPLETE: 
            /* Transfer completed successfully */ 
            leftBlockCount[chNum]--;
            if(leftBlockCount[chNum] > 0)
                EDMA3_DRV_enableTransfer(edmaHandle, edmaChannelIds[chNum], EDMA3_DRV_TRIG_MODE_MANUAL); 
            else
                printf("Channel %d is complete\r\n", chNum);
        break; 
    
        case EDMA3_RM_E_CC_DMA_EVT_MISS: 
        case EDMA3_RM_E_CC_QDMA_EVT_MISS: 
            printf("Event miss\r\n");
        break; 
    
        default: 
        break; 
    }
}
    
void edmaChannelTest()
{
    uint32_t chNum = 0;
    for(chNum = 0; chNum < 8; chNum++) {
        leftBlockCount[chNum] = 100000; 
        EDMA3_DRV_enableTransfer(edmaHandle, edmaChannelIds[chNum], EDMA3_DRV_TRIG_MODE_MANUAL);
    }
}

We met similar unfinished transfer in the past when multiple transfers in parallel. The way to make it works:
- evenly spread all transfers/channel across different transfer controllers (TC)
- For different TC, assign a different priorities (by default they are all the same: highest priority)

Refer to TRM: EDMA_TPCC_QUEPRI, use different priorities for TC0 and TC1.

Regards, Eric

Sorry for delay. It didn't help me, but speed increased.