PRU-ICSS-INDUSTRIAL-SW: HSR-PRP transmit not working for our custom board with AM3356 SOC

Hi Sreenivas,

Thanks for your query.

Can you please provide more info about your setup ?

1. Which TI-SDK you are working on ?

Sreenivas Ranganath said:

same hardware and software setup with respect to PRU is working for in Switch mode

2. Means you are able to Rx and Tx normal Ethernet packets (tagged / untagged) ?

3. Is the Node table updated on not (Tx) working node ?

4. How can I reproduce the issue with bare-minimal steps on my local setup ?

Best Regards

Ashwani

Hi Ashwini, 


1. We are not using TI SDK  directly we are using commercial RTOS and integrated TI SDK into that.
    But other than this  we have integrated below component from TI to our source. 

    "PRU-ICSS-HSR-PRP-DAN_01.00.05.0"    and "pdk_am335x_1_0_17"

      =>  We are using our own board/HW based on AM335x. 

      =>  We  have integrated  above two s/w modules to our project and is working in fine switch mode but for HSR we are facing issue.

2. We are able to receive super frame and other HSR tagged packet sent from other test node.

But our transmission is not working after sending few packet we are getting buffer FULL message as below  (in file:: hsrPrp_red_tx.c)

3. Node table in our device is getting updated with other node data but not sure about node table in other node with our HSR device as we are stuck in TX issue.

4. Instead of reproducing  at your end is it possible to setup call so that we will able to show what we did and if needed debug also.

Thanks and Regards,
Mital Pawar

Hi Mital,

Thanks for the information.

Please allow me some time to discuss this internally with SW team and get back to you.

Best Regards

Ashwani

Hi Ashwani,

Awaiting for your reply.

Thanks and Regards,
Mital 

HI Mital,

I apologize for late response.

Can you compare the SMEM, DMEM comparison between TI HSR PRP and your implementation and confirm if you are seeing the same issues with both the ports.

Thanks and Regards,

Rimika

Hi Rimika,

What do mean by "TI HSR PRP" and on which hardware we have to compare.

FYI 

We have tested HSR on IEC BOARD with TI SDK and it's  working.

Same code we have ported to our customized hardware and RTOS   We are able to Receive HSR packet from other nodes but facing issue in transmitting packets. Getting buffer overflow error after few packet.

Incomiing HSR => Working

Outgoing HSR => Not working  

Thanks and Regards,
Mital Pawar.

 Hi Mital,

1. From the description it looks like the firmware is not picking up the packet from the TX Queue.

Mital Pawar said:

What do mean by "TI HSR PRP" and on which hardware we have to compare.

2. it should be comparion of DMEM and SMEM logs between 2 setups:

Mital Pawar said:

Outgoing HSR => Not working

3. Are you seeing the same issue on boath Ports ?

Best Regards

Ashwani

3. 
    => This issue is with both ports.

   Is there any way log PRU state/status with ARM debugger.

Thanks and Regards,

Mital
    

Hi Mital,

Thanks for update on point-3.

Can you please provide update on point 1 and 2 as well?

Best Regards

Ashwani

Hi AshWani,

I am facing challenge in taking memory dump.
What would be the better way to do it.

Thanks and Regards,
Mital Pawar

Hi,

Want to know that  in which situation PRU will not consume data written transmit buffer,  so that parallelly we will look into that also.
Note that packet receive is working.


Thanks and regards,
Mital 

Hi,

Find attached Zip File for run time data captured while receive was working.

hsrPRPRuntimeDataCapturedRxWorking.zip

Thanks and Regards,
Mital Pawar

 

Hi TI Team,

I am working with Mital to resolve the issue.

We have tried to use the available example of "hsr_app_AM335x_arm" of PRU-ICSS to run the HSR, below are our observations with the example code:

Behavior on ICE board : ICE board is able to send the "HSR Supervision" and "Peer_Delay_Request message" frames.

Our board : No communication observed.

Below setup is used for example:

PDK version: pdk_am335x_1_0_17

PRU_ICSS version : PRU-ICSS-HSR-PRP-DAN_01.00.05.01

Kindly let us know how we can resolve this.

With regards,

Anuj Shah

Hello Anuj

Sorry to keep you waiting. I will remind the software developer to see if they can find anything from the memory dumps you have shared.

Please do note that our ability to support AM335x with Industrial Protocols is very limited We have also stopped supporting all SYSBIOS and TI RTOS based projects. For new designs we recommend evaluating AM64 or AM24x family 

If you are further down into this design (given you have already designed a board) - we may also recommend that you evaluate working with a 3P like CouthIT 

https://www.couthit.com/product-page/#industrial-communication-protocol

Again, I will ping HIMANSHU MITTAL to see if he has debug suggestions based on the info you have shared.

Regards

Mukul 

Hi Anuj, Mital,

Sorry for delayed response, I am checking the configuration and will see if this is correctly done. 

Regards,
Himanshu

Hi,

Issue: Same code we have ported to our customized hardware and RTOS. We are able to Receive HSR packet from other nodes but facing issue in transmitting packets. Getting buffer overflow error after few packet.

Please run the test for both ICE board and your customized board. After running the test (and on hitting the TX issue) record the memory as described earlier for both cases.
For AM335x the memory space is from 0x4A30_0000 to 0x4A37_FFFF. Store 0x80000 memory words when taking ICSS memory dump. (The currently shared memory dump seems to be from 0x402f0400 to 0x402f1400.) 

This will help me to compare and find if there is some configuration or some other issue on your setup.

Thanks,
Himanshu 

Hi Himanshu,

Kindly find attached memory dump of both ICE_Board ( with Tx working) and Custom Board (Tx not working).

Dump is downloaded after running Rx/Tx Test Mode for Transmission of GOOSE Test Frames(Executed commands on console: R->T->G).

Observation:

ICE Board : Able to see the GOOSE packets and HSR Supervision Frames in wireshark

Custom Board : Not able to see any packets in wireshark and Buffer overflow error on console after attempting to transmit 30-50 GOOSE packets.

PRU_ICSS_Data_Dump_29082023.zip

With Regards,

Anuj Shah

Hi Himanshu,

Is there any update on this?

With regards,

Anuj Shah

Hi Anuj,

We are working on this internally with dev team.

But, as dev team is busy in Industrial SDK Release related work. Please expect some delay in response. 

Best Regards

Ashwani

Hi Anuj,

We compared the memory dumps you provided, and we do see some issues with transmit side hardware stats.

--------------------------------------------------
Working             Non-Working
--------------------------------------------------
PRU_ICSS_INTC Registers:

SRSR0 Register (offset = 200h):
00105180            00305180
SRSR1 Register (offset = 200h):
0000000C            0008C08C

- Both event bits are set:
51 PRU-ICSS0 PORT1_TX_UNDERFLOW
39 PRU-ICSS0 PORT0_TX_UNDERFLOW

--------------------------------------------------
PRU_ICSS_MDIO Registers:

Alive Reg: 
0000000A            00000003
Link Reg:
00000002            00000003

- Link up is there for PHYs 0,1
- MDIOUSERPHYSEL0/1 registers are correctly set

--------------------------------------------------
PRU_ICSS_MII_RT Registers:

RXCFG0:
00000015            00000015
RXCFG1:
0000001D            0000001D
TXCFG0:
00400103            00400103
TXCFG1:
00400003            00400003
--------------------------------------------------

We are checking what can cause this issue.

Also, if you are sending the packets on 1 queue only, can you try by sending on different queues as well and check what is the result there. 

Thanks.

Hi Anuj,

HIMANSHU MITTAL said:

Also, if you are sending the packets on 1 queue only, can you try by sending on different queues as well and check what is the result there. 

Please let me know what the status is if you try to send the packets over a different queue when the transmit is stuck. This will help us know the state of the PRU. 

Also, are you able to receive packets on both the ports (after the issue)? 

Thanks,
Himanshu 

Hi Himanshu,

We are able to make PRU running with below set of configuration.

PDK: pdk_am335x_1_0_12

PRU_ICSS version : PRU-ICSS-HSR-PRP-DAN_01.00.04.02

If we use the latest one it is not working, and if we use the above mentioned packages we are able to see the HSR supervision frames coming.

Also, regarding the queue, we are using the default TI example code which is working on ice board, as it is on our board. The same code is working there and not working with our board.

In latest example code of HSR, the QUEUE0 is used for RX activity and QUEUE1 is used for TX activity. We are also using the same way.

int main()
{
...

    ICSSEMAC_InitConfig *switchEmacCfg;
    switchEmacCfg = (ICSSEMAC_InitConfig *)malloc(sizeof(ICSSEMAC_InitConfig));
    switchEmacCfg->phyAddr[0] = Board_getPhyAddress(PRUICSS_INSTANCE, 1);
    switchEmacCfg->phyAddr[1] = Board_getPhyAddress(PRUICSS_INSTANCE, 2);
    
    switchEmacCfg->portMask = ICSS_EMAC_MODE_SWITCH;
    switchEmacCfg->ethPrioQueue[ICSS_EMAC_QUEUE1] = 1 ;
    switchEmacCfg->ethPrioQueue[ICSS_EMAC_QUEUE2] = 0 ;
    switchEmacCfg->ethPrioQueue[ICSS_EMAC_QUEUE3] = 1 ;
    switchEmacCfg->ethPrioQueue[ICSS_EMAC_QUEUE4] = 0 ;
    switchEmacCfg->halfDuplexEnable = 0;

    switchEmacCfg->enableIntrPacing = ICSS_EMAC_ENABLE_PACING;
    switchEmacCfg->pacingThreshold = 100;
    
    ...

With regards,

Anuj Shah

Hi Anuj, 

We checked the shared ICSS dump for more details and we see that the speed configuration is being done as 10M. 
Line 2025, 4073 are currently set to 0xA (offset 0x1F9C in DMEM0 and DMEM1) but should be 0x64 for 100M configuration. 

From MII_RT space we see that in PRS0 register pr1_mii0_crs bit is set. 
Line 51216 is 0x2 (ICSS offset 0x3200 + 0x38).

It looks like link partner is not returning capability correctly from MDIO register dumps. This can force PHY to negotiate 10M HD as per IEEE spec. Can you please verify the PHY is configured properly.
- You can compare the PHY dump for working and non-working case. Please share PHY_STATUS and BMSR register values.
- Also, which PHY are using in your custom board?

Thanks,
Himanshu

Also: AM3 ICE board has TLK110 PHY. In line 51490 (MDIO space MDIOUSERACCESS0 register) the read value of PHY register 5 is 0 seems to be incorrect.  

Hi Himanshu,

In our custom board we have used "DP83822" PHY for Fiber Optic(FO) communication.

The PHY_STATUS and BMSR register values for working code is as below:

Phy0 Reg Add=0001 Data=784d
Phy0 Reg Add=0010 Data=0205
Phy1 Reg Add=0001 Data=784d
Phy1 Reg Add=0010 Data=0205

And the same register values in non working code is as below:

Phy0 Reg Add=0001 Data=7849
Phy0 Reg Add=0010 Data=0002
Phy1 Reg Add=0001 Data=784d
Phy1 Reg Add=0010 Data=0205

All of the above register values are from our custom board.

With regards,

Anuj Shah

Hi Anuj, 

As concluded previously, there seems to be issue with PHY register values of one of the PHY:

Register: 0x0001 Basic Mode Status Register (BMSR)
Value: 0x7849
=> Link Status not up

Register: 0x0010 PHY Status Register (PHYSTS)
Value: 0x0002
=> Speed Status: 10Mbps, Duplex Status: = Half-Duplex mode

Register: 0x0005 Auto-Negotiation Link Partner Ability Register (ANLPAR)
Value: 0x0
(From previously shared memory dump)

• Please verify that the PHYs are configured properly, and both the ports are connected.
• Also then, is this correct that: RX for 1 port is working and TX for that port is not working. (as the other PHY seems to have link down state)

Thanks,
Himanshu

Hi Himanshu,

I Agree with you on this, but the observation from our side as mentioned previously as well that:

If we use below combination of packages, same HW is working fine.

PDK: pdk_am335x_1_0_12

PRU_ICSS version : PRU-ICSS-HSR-PRP-DAN_01.00.04.02

And if we use the latest packages than on the same HW we are facing the issue, of which i have shared the memory dump.

Could you please look into this why older version is working where as latest version is not working?

Also, kindly share what needs to be modified in latest packages so that it will start working.

Since the working packages are not available on TI's portal, i assume it is of beta version, and we would like to use the latest release for our further development.

Let me know if more details are required.

 With regards,

Anuj Shah

Hi Anuj,
To reiterate current status:

ICE Board: Works with both 1.0.4 and 1.0.5 releases with OOB example.
Custom Board: Works with 1.0.4 but not with 1.0.5.
Open point: Are the required modifications for your custom board correctly done with 1.0.5 as well?

Differences between 1.0.4 and 1.0.5:
Application and driver side changes. No firmware changes (PRU). You can compare between the 2 packages to see if something might have impact in-case of your custom board. 

Suggestions for resolving/root-causing the issue:

1. Check PHY init sequence. Verify everything is being done properly. You may share the initialization code with us also for review.

2. Get complete PHY dump (all PHY registers value for both PHYs) for your custom board in both cases for working (1.0.4) and non-working case (1.0.5) and compare to debug.
You may also share that with us so that we can check and provide feedback. We already see from previous shared information that PHY config seems to be an issue - complete register map comparison may help in pin-pointing the issue.

3. "In our custom board we have used "DP83822" PHY for Fiber Optic(FO) communication."
Please follow up with PHY team as well for the PHY related issues being seen.

Thanks,
Himanshu

4. Also try by removing PHY reset call from main() (line 16 as shown below):

int main()
{
    TaskP_Params taskParams;

    PRUICSS_IntcInitData pruss_intc_initdata = PRUSS_INTC_INITDATA;

    HSR_PRP_socHwinit(PRUICSS_INSTANCE);

#ifndef SOC_AM65XX
    memset(&boardInfo, 0, sizeof(Board_IDInfo));
    Board_getIDInfo(&boardInfo);
    UART_printf("boardName: %s\n", boardInfo.boardName);
#endif

#if !defined(SOC_K2G) && !defined(iceAMIC11x) && !defined(SOC_AM65XX)
    Board_phyReset(2);
#endif /*SOC_K2G & iceAMIC11x*/

    prusshandle = PRUICSS_create((PRUICSS_Config *)pruss_config, PRUICSS_INSTANCE);

Hi Team,

We now tried to debug this with the PHY with the below mentioned results:

• PHY DP83822i Link Status is working fine, PHY Reset is not required to get the Link Status
• All the Register dump of PHY indicate the configuration and Boot Strapping is same in both working(01.00.04.02) and non working(01.00.05.01) code

Is there any update at Processor side ? 

Regards,

Alex

Hi, 

I do not have any new update here.
From previous discussions I see there was some inconsistency in PHY register values between the working and non-working case. For this I had asked to check for the 4 points mentioned in my previous 2 replies. If there is an update on those points, please let me know. 

Thanks,
Himanshu