SRIO NWRITE "Unavailable Outbound Credit"

Try NWRITE_R whihc will wait for a resonpose from Slave before transmitting again. let me know if it helps.

Thanks,

Arun.

Hi Arun,

ArunMani said:

Try NWRITE_R whihc will wait for a resonpose from Slave before transmitting again. let me know if it helps.

Yes, I appreciate I could use NWRITE_R instead of NWRITE and believe that using NWRITE_R would stop this problem because the LSU will remain busy for much longer.

Ours is a simple SRIO system, a DSP and FPGA connected directly together. The 2 chips are physically very close.

I was hoping/expecting that there would be no (or at least very few) errors on such a system and that NWRITE_R would be unnecessary. Using NWRITE_R makes my communication infrastructure more complex and slower.

I was surprised by the way LSU_REG6 BSY behaves and wondered if anyone could explain and/or justify its behaviour.

Thanks,

Matt

Hey Matt

How'd you resolve this issue. We are interfacing to an Altera IP SRIO core and having the same problem when the core tries to do flow control. The flow control works for awhile, but then the LSU_REG6 says COMPLETION_CODE = "Transaction complet, packet not sent due to unabilable outbound credit at given priority". Arg.

Cheers
Eddie

Hi Eddie,

Eddie said:

How'd you resolve this issue.

I have not resolved this issue to my satisfaction!

I always delay for at least 800 CPU cycles after the completion status indicates not busy and that seems to work for me.

I've been meaning to have another look at this but I haven't done anything about it since writing the original post.

Matt

Matt,

The outbound credit issue depends on a number of factors.  For example, maybe the endpoint or switch you are connected to can not keep up with the packet rate, buffers back up and eventually fill the TX buffers on the DSP.  The fact is that the DMA bandwidth can be a lot higher than the pin bandwidth too, so if you are doing a 4KB LSU NWrite tranfer or even multiple 256B transfers back to back, the DMA can fill the outbound buffers before a 1X SRIO port at 3.125Gbaud can drain it.  If there is any type of stall in the fabric, packets can temporarily back up and cause this situation.  If you have not already done so, I would recommend taking a look at DIOLib streaming example.  It can be found at:

https://gforge.ti.com/gf/project/sriodirectiolib/frs/

This example shows how to use the EDMA to drive the LSUs.  And it has a provision to handle the outbound credit issue and restart a transfer where it left off, if you are in the middle of a multi-packet transfer that only completed part way.

Regards,

Travis

Hi Matt and Travis

I tried the DIO library and it didn't complete the transaction when the error occured. I'll take another look at it to confirm. Below is my post.

http://e2e.ti.com/support/dsp/c6000_multi-core_dsps/f/439/p/99158/347004.aspx#347004

Cheers
Eddie

Hi Travis,

Thanks for replying. I haven't been working on the SRIO recently but I still want to understand what's happening.

I think you're saying the answers to the questions in my original post are:

MattB said:

Is it normal to get "outbound credit unavailable"?

Yes.

MattB said:

Is it normal to have to space out the LSU requests?

Yes.

But I'm still not sure I understand what the BSY field of LSU0_REG6 is telling me because it's obviously not telling me that the Tx buffers are empty.

Are there any registers that indicate how full the Tx buffers are?

If it is normal to have to space out the transactions then I think there are 2 options:

1. Queue transactions and remove transactions from the queue upon successful completion, repeat transactions that are unsuccessful.
2. Use a fix spacing to ensure the bandwidth doesn't get too high.

Option 1, which is what DIO attempts, is probably more robust and flexibly but the CSL and DIO contain considerable amounts of complicated code.

Option 2, which is what I've done, is probably a bit brittle and heavily dependant on finding a magic delay.

I think I was hoping for a third option along the lines of checking the available credit before requesting the transaction and delaying beforehand.

Any comments greatly appreciated,

Matt

Matt,

I don't know that I would use the word normal, because it depends on a lot of factors and may not happen in every case.  However, it can happen and yes software should be able to handle it.  It shouldn't be considered an error.  As a matter of fact, the reason this approach was taken in the design of the IP was to alleviate head of line blocking issues with the LSUs.  Basically, for a given transaction, multiple attempts are made to get outbound credit, after which the LSU sets the CC=7 to allow software to recognize the issue and potentially reuse the LSU at a higher priority.  I will say that this approach was changed in the C66x devices based on feedback, so those devices will never get outbound credit issues, but they will potentially HOL block forever.

The functionality of the BSY bit of the LSU is somewhat dependent on the type of transaction.  In all cases, it represents whether the LSU can be written to for the next transaction.  For NWRITE_R/NREAD/DOORBELL, the BSY bit is set for the duration of the transaction, including the reception of all the response packets.  For NWRITE/SWRITE, there are no responses, so as soon as all the required request packets (that the LSU needs to generate for a given transaction) are sent to the physical layer buffers, the LSU is done with that transaction and deasserts BSY.

Hope that helps,

Travis

tscheck said:

The functionality of the BSY bit of the LSU is somewhat dependent on the type of transaction.  In all cases, it represents whether the LSU can be written to for the next transaction.  For NWRITE_R/NREAD/DOORBELL, the BSY bit is set for the duration of the transaction, including the reception of all the response packets.  For NWRITE/SWRITE, there are no responses, so as soon as all the required request packets (that the LSU needs to generate for a given transaction) are sent to the physical layer buffers, the LSU is done with that transaction and deasserts BSY.

Hhmm, I think I'm getting there...

So I've started a transaction and the BSY flag has cleared because the data has moved to the physical layer. I can now write to the LSU registers, (LSU registers 0-4 are only configuration so won't cause much to happen) but writing to LSU_REG5 is more significant, I can do that but there is no guarantee that there are free physical/logical layer buffers for the data. And, unfortunately, it's not possible to check the whether or not there are physical/logical layer buffers free.

Changing the subject slightly I'm still not sure what to do with the packet priority and transmit credit threshold. My is pretty a pretty simple point-to-point system and I've got no reason to use multiple priorities. So if I use priority 0 for Direct I/O writes and don't change TX_PRI0_WM (its default is 3) I'm never going to use 3 of the 8 logical layer buffers? Should I change TX_PRI0_WM to 0?

tscheck said:

Hope that helps,

Yes, I think it is helping and I think it is important for me to try and understand a major component of our system!

But the real question is what the software should given that "unavailable outbound credit" can and will occur! Is my current solution really acceptable? Is there any advantage to doing something more complicated like not using a fixed delay and being prepared to retry?

Thanks,

Matt

Hi Travis,

tscheck said:

Correct, you've got it.  You can not check the physical layer buffer status via software, only the LSU itself can do this check.  So it is self throttling, but in some cases you will get the outbound credit CC back to software.  Also note, the request packet payload is never moved into the peripheral until credit is granted.

OK!

tscheck said:

DO NOT change the TX_PRIOx_WM values from their defaults

Right, OK, I won't!

tscheck said:

Sure using priority 2 on your requests will allow you to use more of the Tx physical layer buffers, but I can't tell you if that means you will get less out of credit CCs.

I'll stick with priority 0 then!

tscheck said:

I can't guarantee you'll never see CC=7 either.  If you can handle the CC=7 case, you know you are covered.

Yes, good point! You're right, I know what I've done is brittle, you've convinced me to do something about it!

tscheck said:

For example, if you programmed the LSU for a 4KB transfer, and you were able to send 2KB before receiving the CC=7, then the bytecount field of the LSU will reflect that.  I honestly can't remember if the address fields for destination/source are modified/updated too, (you could check that out on your setup), but essentially you can start transmission from where you left off.  No wasted bandwidth for retransmitting.  This is what the DIOLib EDMA streaming example does.

I hadn't understood this, now I do, thanks!

That was a very useful post, thank you very much,

Matt

I've haven't heard the phrase "head of line (HOL) blocking" before. I think it's self-explanatory but could you provide a few lines of description just to be sure I understand!

Thanks,

Matt

Hi  Travis

Way back in the thread you stated:

tscheck said:

Basically, for a given transaction, multiple attempts are made to get outbound credit, after which the LSU sets the CC=1

For the C6455 and C6472, CC = 1 in LSUn_REG6 COMPLETION CODE is "Transaction timeout occured on non-posted transaction",

How is this different from CC=0111b. "Transaction complete, packet not sent due to unavailable outbound credit at given priority"? I'm confused how CC=1 relates to the subject "Unavailable Outbound Credit". Should the recovery software deal with both CCs the same way?

Cheers

Eddie,

My bad, I mistakenly said CC = 1, when I should have said CC=7.  I changed all the above posts to correctly reflect CC=7 (outbound credit not available at given priority).  Sorry for the confusion and thanks for pointing that out.

To answer your question: 

CC = 1 is when a request packet is actually sent, but no response is received back from the targetted device.  For example, a NREAD or NWRITE_R response never happens before the response timeout occurs.

CC = 7 is explained pretty well above and means that the LSU request packet can't even be sent because of full TX buffers at a given priority.

Regards,

Travis

Thanks a bunch Travis