66AK2H12: SRIO Delineation Error for K2H

Are you using Processor SDK RTOS? Which version?

Best Regards,
Yordan

This is RTOS base 3P OS.

Customer try to debug non OS relate approach. So, we asked above questions.

Regards,

Kz777

Some guidance on the error and causes in an older post for an older device (applies to K2H also)

https://e2e.ti.com/support/processors/f/791/t/168502

Hi Mukul,

Thank you very much for quick response.

I refer this thread. Customer is looking for this root cause . And also, they would like to know how to recover system when this error happen.

So, we asked above question. Do you have any idea ? Currently, their system suspend when this issue happen. They would like to re-start to transfer data.

Regards,

Kz777

Hi Kaz-san

It is my understanding talking to someone more familiar with SRIO , that when this one error happens, the HW automatically recovers , so it may be observable by seeing “suspended” or delays in traffic, but it should recover without SW.  So can you explain what suspend means for the customer - and is it causing a hang in the system?

I was also told that it cannot be disabled as , it is a SRIO physical layer issue and the link needs to align. 

It should never happen, so if they are seeing it, they either have really bad bit errors, or the RX and TX configuration is not correct - it would be best to try to root cause the issue instead of trying to workaround by disabling it etc.

Regards

Mukul 

Hi Mukul,

Thanks for reply.

>So can you explain what suspend means for the customer - and is it causing a hang in the system?

It is only stopped SRIO and this SRIO data communication. another K2H system is working.

<Phenomenon from Tx and Rx devices> **********

TX device :  (Device A)

When device A send a data to K2H, it has sequence error. Rx device (K2H side) doesn't return "ACK".

RX device (K2H side)

K2H side stopped SRIO data communication. So, it doesn't work automatically recover.

We are trying to figure out root cause. So, we need additional information and confirmation for debug.

1:I think SRIO spec has automatically recovery system after delineation error happen. 

Usually, this automatically recover system return "Not accepted CS" from K2H side and wait Input-status CS on the SRIO. Then , it recover from error.

Is this correct ?

 2:Currently communication stopped as Delineation error statement.  Is this reason due to Tx side (device A)sending data not able to send error recovery information to K2H ?

3: Do you have any other reason why K2H didn't recover on the delineation error ?

Regards,

Kz777

Hi ,

Do you have any idea about my additional post ?

Could you please give us advice above 1), 2) query at first  for debug?

>1:I think SRIO spec has automatically recovery system after delineation error happen. 

>Usually, this automatically recover system return "Not accepted CS" from K2H side and wait Input-status CS on the SRIO. Then , it recover from error.

>Is this correct ?

 >2:Currently communication stopped as Delineation error statement.  Is this reason due to Tx side (device A)sending data not able to send error recovery >information to K2H ?

Omori-san

I have sent the post to our SRIO expert , but have not heard back from them yet.

Will ping them again.

Regards

Mukul 

Hi Mukul,

Thanks for support. I 'll wait  your response.

Regards,

Omori-san,

The Serial RapidIO link error detection and re-transmission protocol requires that the receiver acknowledge each transmitted packet until either the sender receives a packet-accepted control symbol acknowledgment for the packet or the sender determines that the packet has encountered an unrecoverable error. When a receiver detects a transmission error in the received data stream, the receiver causes its associated transmitter to send a Packet-Not- Accepted control symbol. When the link partner receives a Packet-Not-Accepted control symbol, it stops transmitting new packets and sends a Link Request/input-Status control symbol. After the link-request/input-status and link-response control symbol exchange, the sender begins re-transmission with either the packet that was corrupted during transmission or a higher-priority packet if one is awaiting transmission.

So

>1:I think SRIO spec has automatically recovery system after delineation error happen. 

>Usually, this automatically recover system return "Not accepted CS" from K2H side and wait Input-status CS on the SRIO. Then , it recover from error.

>Is this correct ?

Yes, this is correct.

 >2:Currently communication stopped as Delineation error statement.  Is this reason due to Tx side (device A)sending data not able to send error recovery >information to K2H ?

When the delineation error is shown in SPx_ERR_DET register, which error bit is set in SPn_ERR_STAT register? Input Error-stopped or Output Error-stopped?

Is the Keystone Software Assisted Error recovery described in the document implemented?1423.Keystone Software Assisted Error recovery_addendum.pdf

Regards,

Garrett

Hi Garrett, 

Sorry for late reply. I have got feedback from customer.

Regarding ERRSTAT,

Actually, this error occurred on the port2.

SP2_ERR_DET is

addr(0x0290c0c0) -> 0x00000004

SP2_ERR_STAT is
addr(0x0290b198) -> 0x00020302

Actually, this is input error. because, SRIO (DSP side) receive data only.

Based on this setting , Could you please check any strange point ?

Whenever Delineation error happen , does system recover automatically default DSP setting ,right ?

Do you have any register for disable automatically recovery ?

Regards,

Kz777

Hi,

So when the delineation error occurred, the SP2_ERR_STAT indicates that Port OK is still set and Port Error is 0.

bit 2: Port Error, Inbound or Outbound port has encountered an error from which the hardware was unable to recover (fatal error).

bit 9/8 are set:

bit 9: Input Error-encountered, Input port has encountered (and possibly recovered from) a transmission error. This bit
is set when bit 8 is set. Once set, remains set until written with a logic 1 to clear.

bit 8: Input Error-stopped, Input port is in the “input error-stopped” state (read-only).

See details about the error stopped state discussion here -

Again, the software recovery need to be implemented as the threads pointed out -

However, if the normal hardware recovery didn't work and the software recovery is needed, you may need to stop packet transmission in both directions until the states are cleared.  If there are packets being transmitted by one end of the link while the other is performing the recovery procedure, this out of sync behavior could cause what you are seeing

Regards,

Garrett

Hi Garret,

Thanks for reply.

1)

>However, if the normal hardware recovery didn't work and the software recovery is needed, you may need to stop packet transmission in both directions until >the states are cleared.  If there are packets being transmitted by one end of the link while the other is performing the recovery procedure, this out of sync >behavior could cause what you are seeing 

Usually, when system occur  "Delineation error ", it is normal hardware recovery as bellow file. However, our customer situation "input error stopped " situation.

http://e2e.ti.com/cfs-file.ashx/__key/CommunityServer.Discussions.Components.Files/38/5241.SRIO_5F00_Error_5F00_Recovery.pdf

This issue similar this "Software recovery sequence "  Device B have never clear "input error stopped". So, we have to implement software recovery to stop packet transmission to back to normal statement on Device B Is it correct ?

2) Actually, my customer statement is here. So, we have to check bit "1","8","9". You explained bit "2","8","9".  So, do you think any other issue because of bit "1" ?

>SP2_ERR_STAT is
>addr(0x0290b198) -> 0x00020302