[AM3359] NDK and DMA

Hi Steve, thank you for your support.

These are interrupt numbers and priorities:

EDMA for fpga transfers is programmed this way:

	EDMA3Init(SOC_EDMA30CC_0_REGS, EVT_QUEUE_NUM_GPMC);

       EDMA3CrossBarChannelMap(SOC_CONTROL_REGS, EDMA3_CHA_XDMA_EVENT,  EDMA3_CHA_DMA_EVENT);

       /* Clean-up the contents of structure variable. */
       for(index = 0; index < sizeof(paramSet); index++)
       {
             p[index] = 0;
       }
       paramSet.srcAddr    = (unsigned int) (AF14_FPGA_TEST_DPRAM) ;
       paramSet.destAddr   = (unsigned int) my_fpga_buffer_destination;
       paramSet.aCnt       = WORD_SIZE;				/*2*/
       paramSet.bCnt       = WORDS_PER_SAMPLE; 		/*48*/
       paramSet.cCnt       = SAMPLED_POINTS_PER_CIG;/*128*/
       paramSet.srcBIdx    = paramSet.aCnt;
       paramSet.destBIdx   = paramSet.cCnt * paramSet.aCnt;
       paramSet.srcCIdx    = 0;
       paramSet.destCIdx   = paramSet.aCnt;
       paramSet.linkAddr   = 0xFFFFu;  //no re-arm
       paramSet.bCntReload = 0;

       paramSet.opt = 0x4;

       tccNum = EDMA3_CHA_DMA_EVENT;
       paramSet.opt |= ((tccNum << EDMA3CC_OPT_TCC_SHIFT) & EDMA3CC_OPT_TCC);
       paramSet.opt |= (1 << EDMA3CC_OPT_TCINTEN_SHIFT);

       EDMA3SetPaRAM(SOC_EDMA30CC_0_REGS, EDMA3_CHA_DMA_EVENT, &paramSet);
       EDMA3EnableEvtIntr(SOC_EDMA30CC_0_REGS, EDMA3_CHA_DMA_EVENT);

       EDMA3EnableTransfer(SOC_EDMA30CC_0_REGS, EDMA3_CHA_DMA_EVENT, EDMA3_TRIG_MODE_EVENT);

    EDMA3RequestChannel(SOC_EDMA30CC_0_REGS, EDMA3_CHANNEL_TYPE_DMA,
    					EDMA3_CHA_DMA_EVENT, EDMA3_CHA_DMA_EVENT,
                        EVT_QUEUE_NUM_GPMC);

    /* Registering Callback Function for TX*/
	EDMA_RegisterCallback( &callback_edma_transfer_ok, EDMA3_CHA_DMA_EVENT );

    /* Registering Callback Function for TX*/
	EDMA_RegisterErrorCallback( &callback_edma_transfer_error, EDMA3_CHA_DMA_EVENT );

EMAC dma configuration is done inside CPSW drivers bundled with AM335x library software.

Steve, I've an update.

When NDK hangs, no CPSW RX interrupt is fired.

ROV shows that RX HWI is enabled, with no pending requests. It's a nonsense, because ethernet cable is plugged and tons of broadcast packets should be processed by NDK:

So, I've exported INTC and CPSW registers when the stack is running and when it's not:

7585.NDK-DMA_problem.txt

5383.NDK-DMA_ok.txt

Hope this will be useful.

Eugenio,

I've been looking at the registers that differ in value between those 2 files you attached above.  Unfortunately, the AM3359 Technical Reference Manual (TRM) does not provide any description for many of the registers, so it is hard to tell what they are for (and if the values indicate a problem).

However, I do see one register value that stands out - I see that the CPSW_WR_C0_RX_STAT register changed value from 0x1 to 0x0.

I think this means that the RX interrupt is disabled (which would agree with what you are seeing, RX interrupt for the CPSW not firing when this problem comes up:

Can you try writing 0x1 to this register when you see the problem?  Does it re-enable the CPSW RX interrupt and allow data to be received?

Another thing to check (I didn't find it in your register dumps) - can you check the value of the RX statistics register?

Steve

Hi Steve, I've run through some tests.

CPSW_WR_C0_RX_STAT is a read only register. Reading TRM, I can argue that if this bit is zero, could be because:

1. corresponding RX INT is disabled
2. no RX interrupt are issued by "subsystem core 0" RX (in AM3359 only core 0 is implemented in silicon)

So, I looked forward trying to figure out if interrupt are enabled or whatsoever other abnormal condition. Subsystem core 0 gathers several interrupt sources and forward one interrupt pulse to CPU. One of these sources is CPSW_DMA controller.

Here are the main differences in CPSW_DMA registers (left=working, right=not working):

CPSW_CPDMA_RX_INTSTAT_RAW is zero when not working. This means that no RX_INT requests are generated from CPSW_DMA, so it's not a matter of RX_INT enabled or disabled.

CPSW_CPDMA_DMA_INTSTAT_RAW is 1 when not working: this could mean that a pending DMA cycle is stuck somewhere, and no one is servicing data packets coming through FIFOs. 

I've tried to manually write 1's in 

• CPSW_CPDMA_RX_INTMASK_SET
• CPSW_CPDMA_DMA_INTMASK_SET

with no success.

So, I searched inside INTC controller, with no evidence of CPSW RX interrupt disabled. I've noticed out that spurious interrupt register is 0x44 when not working, while is 0x01 when working: what does this could mean?

As you requested, here are the STATS registers (not mapped in CCS6 debugger windows, so only plain memory view is available):

I want to overemphasize that the issue comes only when DMA is transferring data from a peripheral to DDR. Is there any method to set priorities between DMA and CPSW_DMA? I could run some tests to understand where the problem is.

Hi Prasad, I've tried your suggestions:

1. modified .cfg file: 

Global.memRawPageCount = 384;

Global.pktNumFrameBufs = 768;

2. modified cpsw_ethdriver.c, line 38, from 4 to 16:

 #define CPSW_PKT_MAX 16 // was 4

No success: problem is still there.

Meanwhile I've ported NDK to another application, and the same problem occurred.

So I've tried to figure out what is common in these application: both have an hardware interrupt with high priority that preempt everything else. This ISR is *very* time critical, and may steal all CPU time for hundreds of microseconds.

During this time all interrupts are enabled, but they cannot preempt the time critical ISR routine. So, if CPSW receives ethernet packets (from a network heavily loaded with broadcasts), some internal queue will fill; NDK cannot handle this situation and will stuck forever.

If the CPU is connected directly to the host (direct cable connection, or via a switch isolated from main network), no problem occurs.

So, this is the situation: I've a time critical routine that cannot be preempted; CPSW should drop packets if CPU cannnot serve its RX interrupts. How this could be handled with NDK CPSW drivers?

For your convenience, the same applications with lwIP don't have this problem. This is CPSW RX interrupt handling scheme with lwIP stack: 

- every packet RX ISR first disables CPSW interrupt with Hwi_disableInterrupt(41).

- ISR post a sempahore that will wake up a packet servicing task (with lowest priority)

- Packet servicing task is called, (maybe milliseconds after RX packet ISR was fired, if hard realtime stuff is being executed) and data is passed to lwIP stack.

- lwIP stack does its magic and then acknowledge CPSW interrupt with CPSWCPDMAEndOfIntVectorWrite(cpswinst->cpdma_base, CPSW_EOI_RX_PULSE)

- the task enables back again CPSW interrupts Hwi_enableInterrupt(41), and will pend itself on the semaphore.

Could this be the difference? Is possibile to do the same in NDK drivers?

/* -------------------------------------------------------------------------- */
/*
** Task for Core 0 Receive
*/
Void CPSWCore0Rx_Task(UArg a0, UArg a1)
{
	while(1)
    {
    	Semaphore_pend(semCPSWCore0Rx, BIOS_WAIT_FOREVER);
        lwIPRxIntHandler(0);
        Hwi_enableInterrupt(41);
    }
}

/* -------------------------------------------------------------------------- */
/*
** Interrupt Handler for Core 0 Receive interrupt
*/
static void CPSWCore0RxIsr(void)
{
    Semaphore_post(semCPSWCore0Rx);
    Hwi_disableInterrupt(41);
}

Hello Eugenio,

From your description there are two possibilities for CPDMA receive hang in your example.

1. Host error  - This occurs when you submit wrong buffer descriptors to the CPDMA. As this is non-recoverable CPDMA will get stuck till reseted. In the register details you shared earlier host error is not set but still you can confirm.

2. Misqueued packet condition

 This is when software and hardware were accessing last descriptor simultaneously. If CPDMA is stuck and you dont have host error most likely it is misqueued packet condition. 

You can check RX0_HDP  registers of receive channel to see if content is zero. Also check RX0_CP for last buffer descriptor processed and see next descriptor is submitted to hardware (ownership bit set). If this is true then try restarting receive operation manually by writing descriptor address to HDP register.

Let me know if any clarifications needed. Thanks.

Regards,

Prasad

Hi Prasad. I've tried to dig into CPSW registers, but I gave up after a while.

I then modified ethernetip-adapter sample from TI to reproduce the issue on the ICEv2 demoboard:

1881.ndk_hangs.zip

Default IP is 192.168.1.200. 

I've configuerd an hardware timer to generate an interrupt that every 500us waste 350us of CPU time. Its priority is 3, so way higher than every ohter interrupt in the system. You can see its activity through RUN led on ICEv2 (next to ethernet_0 rj45 connector).

If the ethernet port (ethernet_0 in ICEv2) is connected to a network with heavy broadcast traffic, after some seconds/minutes NDK will stop respoding to ping requests. 

The same application with CPU isolated from the network (ie: direct cable connection with the host PC) will run forever.

Could you try it?

Hello,

Due to unavailability of ICEv2 demoboard I couldn't try reproducing the issue with your application. From your description it looks like it is misqueud packet condition.

I have asked internally for expert help for running on demoboard. Will let you know.

Regards,

Prasad

Hi Prasad.

Got news?

We are still stuck in this issue, and no further developing is possibile until this is resolved.

Hello Eugenio,

Sorry for delay in resolving this issue. Steven Connell is checking if we can arrange this board. Will update you accordingly.

Hi Eugenio,

Could you confirm what target SW you are using?

Do you use the Industrial SDK? if yes which version?
http://www.ti.com/tool/sysbiossdk-ind-sitara

Have you updated the NDK version and SYSBIOS version originally provided in the Industrial SDK?
If yes please provide the version number.

Are you using the industrial protocols beside the CPSW?

Thanks!

Anthony

Software: ethernetIP adapter demo application, found in am335x_sysbios_ind_sdk examples.

industrial SDK version: am335x_sysbios_ind_sdk_01.01.01.01

Compiler: TI5.2.5

NDK: 2.22.3.20

SYS/BIOS: 6.45.0.19

XDC: 3.31.2.38

I'm not using any industrial protocol in this demo (ie: EIPMain() commented out).

Here the application: it runs on ICEv2 demoboard, configured to run ethernetip_demo app:

1667.ndk_hangs.zip

The issue is present only when the application runs through CPSW; no issue when ICSS is used instead of CPSW.

Can you reproduce the problem?

Eugenio,

It seems that you have mixed different SW components. The default components in the SDK are:
http://processors.wiki.ti.com/index.php/AM335x_SYSBIOS_Industrial_SDK_01.01.01_Release_Notes#What_Is_SupportedNote that only the SW provided in a given SDK are system tested together.

I would advise to at least update to the latest SDK 01.01.03 and use the components mentioned (SYBIOS 6.41, NDK 2.24, .etc):
http://software-dl.ti.com/sitara_indus/esd/AM335x_SYSBIOS_Industrial_SDK/latest/index_FDS.html
http://processors.wiki.ti.com/index.php/AM335x_SYSBIOS_Industrial_SDK_01.01.03_Release_Notes

Can you please try to reproduce the issue with SDK 01.01.03 using the default SW components provided?
Thanks in advance,

Note that moving forward the processor RTOS SDK (+ specific add-on for IND protocols) for AM335x will replace the AM335x SYSBIOS IND SDK.
This SW should be available sometimes in 2017.
If you are not using any Industrial protocols in your application you can already use the processor RTOS SDK.

A.

Got news?

Still in troubles...

Hi Eugenio,

I checked with some collegues and it seems that :
- You use the NDK TCP IP stack with an old version of the CPSW Ethernet drivers.
- The CPSW side drivers provided on the IND SDK 1.x and IND SDK 2.x are not maintained anymore.

The focus of the INDUSTRIAL SDKs (1.x, 2.x) is the Ethernet based protocols running on the PRU-ICSS side, not CPSW.
The CPSW side SW that was once added is not maintained.

- For the CPSW the maintained SW branch is the CPSW drivers part of the processor RTOS SDK (PROCESSOR-SDK-RTOS-AM335X  03.x):
http://www.ti.com/tool/PROCESSOR-SDK-AM335
The is an Ethernet NIMU drivers for the CPSW:
http://processors.wiki.ti.com/index.php/Processor_SDK_RTOS_NDK

Today only standard Ethernet is supported in the processor RTOS SDK.

Moving forward there will be add-on for the Industrial protocols (Profinet, ethercat, ..etc) to be used on the processor RTOS SDK for the PRU-ICSS.
But keep in mind that the CPSW SW (NDK and low level drivers) will be provide just for standard Ethernet.

Anthony

Hi Eugenio,

Make sure to read the new note about SW maintenance/migration that has been added at:
http://www.ti.com/tool/sysbiossdk-ind-sitara

Best regards,

Anthony