AM6442: Behaviour of PRU_ICSSG after running out of RX ressources when receiving new ethernet frames

Hi Sven Severus,

Thanks for your query.

What is the speed frames are coming on AM64x-ICSSG ports ?

Are you getting same results with https://www.ti.com/tool/download/INDUSTRIAL-COMMUNICATIONS-SDK-AM64X ?

Best Regards

Ashwani

Thank you, Ashwani, for your reply.
The speed frames are received from wire is very very slow: one or a few frames per minute, while link is established at 1Gbit/s.
Frankly speaking I am a little bit surprised by this question, because I did not suspect any relationship or dependency between the behaviour of the PRU_ICSSG after running out of RX ressources (because of the host processor halted by debugger) and the speed of incoming frames...
On the other hand we actually observe frame loss in normal operation (this is when the host processor is not halted and working normally) when stressing the system with high traffic loads with significantly more than 1000 received frames per second.
With INDUSTRIAL-COMMUNICATIONS-SDK-AM64X we did not gain any experience till now. Do you expect a different behaviour?
Thanks and best regards, Sven

Sven Severus said:

How does the PRU_ICSSG deal with received frames

Once a frame is incoming, the widget collects and writes it into the MSMC queue. MSMC queue will overflow, if the packets are not pushed over the PSIL DMA. 

NRT_HOST_EGRESS_Q_PRE_OVERFLOW_PASTATID diagnostic counter is expected to increment.

This PASTAT is slice independent (it will increment for both slices)

Best Regards

Ashwani

Sven Severus said:

Does the PRU_ICSSG notify the host processor in some way when such a situation has occurred? And if yes, how exactly?

No, we don't notify the host processor.

Best Regards

Ashwani

Thank you for your answer, Ashwani - and sorry for the delay of my response: in Germany october 3rd is the national holiday, so I was not in office.

We made the following experiment: We got ENET/LWIP/ICSSG application software running, and 'ping [EVM-IP] -i.001 -f' works perfectly without any frame loss.
Then we halt the host processor by debugger. From that moment ping reports requests not being answered (as expected).
We can view the PA statistics in the debugger by displaying the shared memory contents starting at address 0x300a7000 in the host processor's address space). We actually see the PA statistics counter for received bytes and received frames continueously incrementing (for far more than 10.000 frames after halting the host processor). Exactly what we expect.
However, the counter at byte offset 0xf0, this is NRT_HOST_EGRESS_Q_PRE_OVERFLOW_PASTATID, remains at count zero, although - according to my understanding of your post - many overflows of the MSMC queue must have occurred after halting the host processor!
Do you have any explanation for this behaviour?
In our opinion an increment of an overflow counter would be a perfect and completely sufficient indication for overflows, but only when we can rely upon...

Best regards, Sven

Hi Sven,

To reproduce the issue you are seeing on my setup. I build enet_lwip_icssg example. Load the binary and pause the r5f_0_0 core. 

You are expecting 0x300a70f0 to be non-zero as host is unreachable for ping response. correct?

Best Regards

Ashwani

Yes, exactly! The display of my debugger is nearly identical to yours...

Thanks Sven for confirmation.

I have raised a software bug regarding this. waiting for timeline for the fix.

Will provide you update as soon as I will get form dev team.

Best Regards

Ashwani

Thank you very much, Ashwani.

For us it would be an important bugfix. Our application makes high demands on diagnosis of frame loss...

Best regards, Sven

Dear Ashwani,

the last days we made some progress in understanding how PRU_ICSSG and UDMA work and interact:

(1) The PRU_ICSSG pushes all the frames it has received to the UDMA Rx channel, regardless of the speed, the host CPU retrieves them from there.
If the host CPU is slow (or halted by debugger) the ring of the Rx channel gets filled up, and the next attempt of the PRU_ICSSG to push a received frame will fail. In this case ("descriptor starvation") a special drop count attached to the Rx channel is incremented (register DMASS0_PKTDMA_0_RCHANRT_dcnt_j,
address 4a800404h+formula).
In fact we have observed this register counting up, in particular after halting the host CPU this register increments whenever the PRU_ICSSG tries to push a received frame to the Rx channel. However, the PRU_ICSSG seems not to care for these problems and just keeps on working normally.

(2) Indeed even the PRU_ICSSG can encounter receive problems in its own domain: When receiving many large frames in very short time the PRU_ICSSG seems to fail to prevent occasional overflows of its own Rx FIFO. In fact we have observed the PA statistics counter NRT_HOST_EGRESS_Q_PRE_OVERFLOW counting up under very high traffic conditions (and meanwhile we believe this counter works properly, and there is no bug at all).

What is your opinion? Can you confirm our view?

We found out, that it does not need very high traffic conditions for descriptor starvation to occur, not even 20.000 received frames per second are quite enough.
Can you give an advice what measures we can take in MCU+-SDK software to improve the performance of the host cpu in order to minimize occurrences of descriptor starvation?

Thanks in advance for your answers.

Best regards, Sven

Thanks Seven for updates.

As I am on vacation, please expect my response by end of next week.

Best Regards

Ashwani

Hi Seven,

Can you please check from https://www.ti.com/tool/download/MCU-PLUS-SDK-AM64X/09.01.00.41, ?
dev team confirmed that NRT_HOST_EGRESS_Q_EXP_OVERFLOW (0x300a70f4) incrementing in case we pause R5F. 

Best Regards

Ashwani

Dear Ashwani,

thank you very much for your last posting.
I have three questions:
(1)
The only information we could find about "NRT_HOST_EGRESS_Q_EXP_OVERFLOW" was included in the source file source/networking/enet/core/src/per/firmware/icssg/fw_mem_map.h in the 09.01.00.41 release of MCU+ package:
There we found the two symbols NRT_HOST_EGRESS_Q_EXP_OVERFLOW_MAC_SLICE0_PASTATID and NRT_HOST_EGRESS_Q_EXP_OVERFLOW_MAC_SLICE1_PASTATID, with values 0x0260 and 0x0264 respectively.
When my understanding is correct, that the value of each symbol is the byte offset of the corresponding counter inside the PA statistics address range, then the two counters are located at address 0x300a7260 and 0x300a7264 respectively.
Is the address you have specified, "0x300a70f4", a typo? I could not find any reference for a counter located at this address...
(2)
Can you give us a link to any documentation, that conatins specification about what counters exist in the PA statistics address range, and (even more important) what exactly are the events, counted by each individual counter?
Unfortunately the TRM for AM64x/AM243x (Chapter 6.4, PRU_ICSSG) does not give any information in detail about this...
(3)
Can you confirm (or correct), what I wrote under (1) and (2) in my posting from Oct 26th?

Best regards and thanks in advance, Sven

Hi Seven,

Thanks for follow-up.

I will check and get back to you.

Best Regards

Ashwani

Hi Seven,

your understanding is correct only.

We have missed some PA STATS registers in documentation.

Best Regards

Ashwani

Dear Ashwani,
thank you very much, your statement is a very helpful information for us.
Meanwhile we have the ENET/LWIP/ICSSG-application from the 09.01.00.41 release of the MCU+ package running on the EVM64x.
OK so far...
But after halting the host processor - in contrast to the statement of your dev team - we do _NOT_ see any counting up of a statistics counter located at 0x300a70f4 or 0x300a7260 or 0x300a7264, although the PRU keeps on receiving frames: we see the drop count of the Rx channel (DMASS0_PKTDMA_0_RCHANRT_dcnt_j) counting up, indicating occurences of descriptor starvation anytime the PRU tries to deliver a newly received packet to the RX channel...
Maybe the confirmation of your dev team was wrong?
Best regards, Sven