• State Resolved
  • Locked Locked
  • Replies 13 replies
  • Subscribers 98 subscribers
  • Views 747 views
  • Users 0 members are here
Support feedback
Options
Options
Related

This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

PROCESSOR-SDK-AM437X: Understanding PRU logic on PRU-ICSS-HSR-PRP-DAN

Marko Kastelic
Expert 1770 points
Part Number: PROCESSOR-SDK-AM437X


Hi,

I observe issue related to PRU stop sending TX packets ("TX queue full" inside RedTxPacketEnqueue function) if CPU get bussy. Regardless of the new version (01.00.05.01), I also have a similar problem with version (01.00.02.00) on which my app currently running. If I check disassembly of PRU, it looks like it checking something and it looping in code, soo program doesn't halt or stuck. It also seems like it sending me an RX msg, but it doesn't TX. Can you please explain to me what conditions PRU check in case of TX? Queue is full .. why it doesn't send it out? 


This app runs on custom board, so we are afraid that we have HW issue with DP83822 or we have some memory corruption on L3 or PRU ram.
Any suggestions where and what to check?
The link seems to be up (also checked with MDIO read), but we see sometimes in statistic up to 7 link break's after program start.

Best Regards, Mare

  • 0
    TI__Mastermind 27904 points

    Hi
    We need some more details from you for further debug:

    1. What is the exact error code you see in "RedTxPacketEnqueue" function? Can you share the return value of the function?
    2. Do you see TX working for some packets initially?

    3. Marko Kastelic said:
      The link seems to be up (also checked with MDIO read), but we see sometimes in statistic up to 7 link break's after program start.
      This does not look correct. Do you have more information on why are you seeing link breaks?

    Regards

    Dhaval Khandla

  • 0 in reply to Dhaval Khandla
    Expert 1770 points

    Hi Dhaval,

    Currently, I can't answer to your points, because this case needs to be "hunted" and is not predictive.

    But I caught Rx issue (i think):

    S T A T I S T I C S

Lookup error A : 00000000
Lookup error B : 00000000

      Tx  WrongLan        Rx    Errors     Nodes     Proxy    Unique   Duplic.     Multi     OwnRx
--------  --------  --------  --------  --------  --------  --------  --------  --------  --------
00000758  00000000  00000000  00000000  00000001  00000000  00000000  00000000  00000000  00000000
00000000  00000000  00000000  00000000                      00000000  00000000  00000000  00000000
00000410  00000000  00000000  00000000                      00000000  00000000  00000000
 H_OFlow   F_OFlow  H_F_Acqu  H_F_Acqu    debug1     debu2    debug3   debug4
--------  --------  --------  --------  --------  --------  --------  --------
-2145860980  00000001  1073741824  -2139609600  -2145860980  -2146450668  00000001  00000001
-2146450668  00000001  -2147483632  -2146436644

SWITCH HANLER:
*****************************************
switch port:0 - errorCount:0 - rowCount:0
Queue[0] - errorCount:0 - rowCount:268
Queue[1] - errorCount:0 - rowCount:0
Queue[2] - errorCount:0 - rowCount:0
Queue[3] - errorCount:0 - rowCount:142
Queue[4] - errorCount:0 - rowCount:0
switch port:1 - errorCount:0 - rowCount:0
Queue[0] - errorCount:0 - rowCount:480
Queue[1] - errorCount:0 - rowCount:0
Queue[2] - errorCount:0 - rowCount:0
Queue[3] - errorCount:0 - rowCount:107
Queue[4] - errorCount:0 - rowCount:0
switch port:2 - errorCount:0 - rowCount:0
Queue[0] - errorCount:0 - rowCount:0
Queue[1] - errorCount:0 - rowCount:0
Queue[2] - errorCount:0 - rowCount:0
Queue[3] - errorCount:0 - rowCount:0
Queue[4] - errorCount:0 - rowCount:0
STATISTIC ICSS_EMAC
*******host***********
txUcast : 0
txBcast : 0
txMcast : 587
txOctets : 42958
rxUcast : 0
rxBcast : 0
rxMcast : 410
rxOctets : 35078
rxUnknownProtocol : 410
txDroppedPackets : 0
linkBreak : 2
*******PRU***********
linkStatus[i] : 1
MDIOlinkStatus : 1
txBcast : 0
txMcast : 689
txUcast : 69
txOctets : 51908
rxBcast : 0
rxMcast : 142
rxUcast : 268
rxOctets : 35078
tx64byte : 45
tx65_127byte : 713
tx128_255byte : 0
tx256_511byte : 0
tx512_1023byte : 0
tx1024byte : 0
rx64byte : 116
rx65_127byte : 275
rx128_255byte : 0
rx256_511byte : 19
rx512_1023byte : 0
rx1024byte : 0
lateColl : 0
singleColl : 0
multiColl : 0
excessColl : 0
rxMisAlignmentFrames : 0
stormPrevCounter : 0
macRxError : 0
SFDError : 0
defTx : 0
macTxError : 0
rxOverSizedFrames : 0
rxUnderSizedFrames : 0
rxCRCFrames : 0
droppedPackets : 0
txOverFlow : 0
txUnderFlow : 0
sqeTestError : 0
TXqueueLevel : 0
CSError : 0

    I don't understand why IP trafic don't work...
    How do you understad this statistic?

    Regards, Mare

  • 0 in reply to Marko Kastelic
    Expert 1770 points

    Hi

    Also I'm wondering what could be the reason for not wisible phy in MDIO:




    Here is one phy ALIVE and this is on CPSW. There should be two more!!.. PRP/switch port seems not alive in SW but LED on port is blinking...!?!
    This problem is not constant and is happening from time to time. It seems that all my mentioned problems have a common issue...just can't find it. Any idea what / where to look?



  • 0 in reply to Marko Kastelic
    TI__Mastermind 27904 points

    Hi

    This looks like a PHY initialization issue.

    1. Do you see the same value in "MDIO_ALIVE" register throughout the execution? Do you bits in "MDIO_ALIVE" set for switch ports at any point of time?
    2. How different is the PHY initialization sequence from example in TI? Are you running this on a custom board or a TI EVM?

    Regards
    Dhaval Khandla

  • 0 in reply to Dhaval Khandla
    Expert 1770 points

    Hi, Dhaval

    New case oserved:

    S T A T I S T I C S

Lookup error A : 00000000
Lookup error B : 00000000

      Tx  WrongLan        Rx    Errors     Nodes     Proxy    Unique   Duplic.     Multi     OwnRx
--------  --------  --------  --------  --------  --------  --------  --------  --------  --------
00000002  00000025  00000025  00000000  00000031  00000000  00000000  00000000  00000000  00000000
00000000  00000000  00000000  00000000                      00000000  00000000  00000000  00000000
00000806  00000000  00000016  00000000                      00000000  00000000  00000000
 H_OFlow   F_OFlow  H_F_Acqu  H_F_Acqu    debug1     debu2    debug3   debug4
--------  --------  --------  --------  --------  --------  --------  --------
00000000  -216733873  -190890157  -1482790163  682793718  -5953898  -1992583  2070148573
1104006501  1944975636  -824648868  -1348946083
switch port:0 - errorCount:0 - rowCount:0
Queue[0] - errorCount:0 - rowCount:77
Queue[1] - errorCount:0 - rowCount:0
Queue[2] - errorCount:0 - rowCount:0
Queue[3] - errorCount:0 - rowCount:729
Queue[4] - errorCount:0 - rowCount:0
switch port:1 - errorCount:0 - rowCount:0
Queue[0] - errorCount:54 - rowCount:32
Queue[1] - errorCount:0 - rowCount:0
Queue[2] - errorCount:0 - rowCount:0
Queue[3] - errorCount:112 - rowCount:34
Queue[4] - errorCount:0 - rowCount:0
switch port:2 - errorCount:0 - rowCount:0
Queue[0] - errorCount:0 - rowCount:0
Queue[1] - errorCount:0 - rowCount:0
Queue[2] - errorCount:0 - rowCount:0
Queue[3] - errorCount:0 - rowCount:0
Queue[4] - errorCount:0 - rowCount:0
*******host***********
txUcast : 16
txBcast : 0
txMcast : 50
txOctets : 4852
rxUcast : 25
rxBcast : 353
rxMcast : 428
rxOctets : 129407
rxUnknownProtocol : 139
txDroppedPackets : 54
linkBreak : 2
*******PRU***********
linkStatus[i] : 1
MDIOlinkStatus : 1
txBcast : 0
txMcast : 2
txUcast : 0
txOctets : 140
rxBcast : 353
rxMcast : 351
rxUcast : 152
rxOctets : 132907
tx64byte : 0
tx65_127byte : 2
tx128_255byte : 0
tx256_511byte : 0
tx512_1023byte : 0
tx1024byte : 0
rx64byte : 253
rx65_127byte : 380
rx128_255byte : 87
rx256_511byte : 75
rx512_1023byte : 61
rx1024byte : 0
lateColl : 0
singleColl : 0
multiColl : 0
excessColl : 0
rxMisAlignmentFrames : 856
stormPrevCounter : 0
macRxError : 0
SFDError : 0
defTx : 0
macTxError : 0
rxOverSizedFrames : 0
rxUnderSizedFrames : 0
rxCRCFrames : 0
droppedPackets : 0
txOverFlow : 0
txUnderFlow : 0
sqeTestError : 0
TXqueueLevel : 0
CSError : 0

    I implement also possibility to reset PHY in runtime (manualy over webpage - CPSW). Reset of PHY didn't resolve issue.
    MDIO:
    BMCR - 0: 0x3100

    What rxMisAlignmentFrames mean?

    To answer to jour post:
    1.) I can't tell you what is the status of MDIO_ALIVE during the normal run of the program. This status is observed whit debugger when I stop program. Should I periodically pull this register and monitor the value?

    2.) What we do is that we reset phy after boot and keep it in reset until ndk stack is initialized (we release reset in NetworkOpen()).

    Did I answer what you ask?

    Regards, Mare

  • 0 in reply to Marko Kastelic
    TI__Mastermind 27904 points

    Mare
    Thanks for the information.

    I have some follow-up questions:

    1. Can you provide the entire PHY register dump from offset 0x0 to 0x1f?
    2. Can you align the PHY reset sequence with TI example ? Any specific reason for following current sequence ?

    Regards
    Dhaval Khandla

  • 0 in reply to Dhaval Khandla
    Expert 1770 points 
    S T A T I S T I C S

Lookup error A : 00000000
Lookup error B : 00000008

      Tx  WrongLan        Rx    Errors     Nodes     Proxy    Unique   Duplic.     Multi     OwnRx
--------  --------  --------  --------  --------  --------  --------  --------  --------  --------
00000004  00000000  00000000  00000000  00000035  00000000  00000000  00000000  00000000  00000000
00000001  00029461  00000000  00000000                      00000000  00000000  00000000  00000000
00016980  00000000  00000017  00000000                      00000000  00000000  00000000
 H_OFlow   F_OFlow  H_F_Acqu  H_F_Acqu    debug1     debu2    debug3   debug4
--------  --------  --------  --------  --------  --------  --------  --------
-1094795586  -1094795586  -1094795586  -1094795586  -1094795586  -1094795586  -1094795586  -1094795586
-1094795586  -1094795586  -1094795586  -1094795586
switch port:0 - errorCount:0 - rowCount:0
Queue[0] - errorCount:0 - rowCount:4963
Queue[1] - errorCount:0 - rowCount:0
Queue[2] - errorCount:0 - rowCount:0
Queue[3] - errorCount:0 - rowCount:12018
Queue[4] - errorCount:0 - rowCount:0
switch port:1 - errorCount:0 - rowCount:0
Queue[0] - errorCount:400 - rowCount:32
Queue[1] - errorCount:0 - rowCount:0
Queue[2] - errorCount:0 - rowCount:0
Queue[3] - errorCount:1991 - rowCount:19
Queue[4] - errorCount:0 - rowCount:0
switch port:2 - errorCount:0 - rowCount:0
Queue[0] - errorCount:1643 - rowCount:32
Queue[1] - errorCount:0 - rowCount:0
Queue[2] - errorCount:0 - rowCount:0
Queue[3] - errorCount:1996 - rowCount:18
Queue[4] - errorCount:0 - rowCount:0
*******host***********
txUcast : 14
txBcast : 0
txMcast : 37
txOctets : 5428
rxUcast : 0
rxBcast : 0
rxMcast : 0
rxOctets : 0
rxUnknownProtocol : 0
txDroppedPackets : 400
linkBreak : 4
*******PRU***********
linkStatus[i] : 1
MDIOlinkStatus : 1
txBcast : 0
txMcast : 0
txUcast : 4
txOctets : 280
rxBcast : 0
rxMcast : 0
rxUcast : 0
rxOctets : 0
tx64byte : 0
tx65_127byte : 4
tx128_255byte : 0
tx256_511byte : 0
tx512_1023byte : 0
tx1024byte : 0
rx64byte : 0
rx65_127byte : 0
rx128_255byte : 0
rx256_511byte : 0
rx512_1023byte : 0
rx1024byte : 0
lateColl : 0
singleColl : 0
multiColl : 0
excessColl : 0
rxMisAlignmentFrames : 0
stormPrevCounter : 0
macRxError : 0
SFDError : 0
defTx : 0
macTxError : 0
rxOverSizedFrames : 0
rxUnderSizedFrames : 0
rxCRCFrames : 0
droppedPackets : 0
txOverFlow : 0
txUnderFlow : 0
sqeTestError : 0
TXqueueLevel : 0
CSError : 0
*******PRU***********
linkStatus[i] : 1
MDIOlinkStatus : 1
txBcast : 0
txMcast : 1
txUcast : 2
txOctets : 198
rxBcast : 6934
rxMcast : 5053
rxUcast : 22480
rxOctets : 3937825
tx64byte : 2
tx65_127byte : 1
tx128_255byte : 0
tx256_511byte : 0
tx512_1023byte : 0
tx1024byte : 0
rx64byte : 2480
rx65_127byte : 26394
rx128_255byte : 3027
rx256_511byte : 1442
rx512_1023byte : 1120
rx1024byte : 4
lateColl : 0
singleColl : 0
multiColl : 0
excessColl : 0
rxMisAlignmentFrames : 34467
stormPrevCounter : 0
macRxError : 0
SFDError : 0
defTx : 0
macTxError : 0
rxOverSizedFrames : 0
rxUnderSizedFrames : 0
rxCRCFrames : 0
droppedPackets : 0
txOverFlow : 0
txUnderFlow : 0
sqeTestError : 0
TXqueueLevel : 0
CSError : 0
*******PHY REG***********
PHY0 addr0 : 0x3100
PHY1 addr0 : 0x3100
PHY0 addr1 : 0x786D
PHY1 addr1 : 0x786D
PHY0 addr2 : 0x2000
PHY1 addr2 : 0x2000
PHY0 addr3 : 0xA240
PHY1 addr3 : 0xA240
PHY0 addr4 : 0x1E1
PHY1 addr4 : 0x1E1
PHY0 addr5 : 0x4101
PHY1 addr5 : 0x4101
PHY0 addr6 : 0x5
PHY1 addr6 : 0x5
PHY0 addr7 : 0x2001
PHY1 addr7 : 0x2001
PHY0 addr8 : 0x0
PHY1 addr8 : 0x0
PHY0 addr9 : 0x0
PHY1 addr9 : 0x0
PHY0 addr10 : 0x100
PHY1 addr10 : 0x100
PHY0 addr11 : 0x1000
PHY1 addr11 : 0x1000
PHY0 addr12 : 0x0
PHY1 addr12 : 0x0
PHY0 addr13 : 0x0
PHY1 addr13 : 0x0
PHY0 addr14 : 0x0
PHY1 addr14 : 0x0
PHY0 addr15 : 0x0
PHY1 addr15 : 0x0
PHY0 addr16 : 0x4615
PHY1 addr16 : 0x4615
PHY0 addr17 : 0x108
PHY1 addr17 : 0x108
PHY0 addr18 : 0x0
PHY1 addr18 : 0x0
PHY0 addr19 : 0x0
PHY1 addr19 : 0x0
PHY0 addr20 : 0x0
PHY1 addr20 : 0x0
PHY0 addr21 : 0x0
PHY1 addr21 : 0x0
PHY0 addr22 : 0x100
PHY1 addr22 : 0x100
PHY0 addr23 : 0x49
PHY1 addr23 : 0x49
PHY0 addr24 : 0x400
PHY1 addr24 : 0x400
PHY0 addr25 : 0x8C04
PHY1 addr25 : 0x8C06
PHY0 addr26 : 0x0
PHY1 addr26 : 0x0
PHY0 addr27 : 0x7D
PHY1 addr27 : 0x7D
PHY0 addr28 : 0x5EE
PHY1 addr28 : 0x5EE
PHY0 addr29 : 0x0
PHY1 addr29 : 0x0
PHY0 addr30 : 0x102
PHY1 addr30 : 0x102
PHY0 addr31 : 0x0
PHY1 addr31 : 0x0
*******END***********

    This is statistic of TX problem.
    PHY reg. for phy0 and phy1 on PRP is added.



    2.) We move "reset line release"  on NDK start because, before NDK start it need to read settings from MMC and lot of traffic make hardfault...soo we keep ETH quiet. Is rediculous fix but it was work... we think

    Regards, Mare 

  • 0 in reply to Marko Kastelic
    TI__Mastermind 27904 points

    Hi
    Were you able to identify the issue? Is everything working as expected now?

    Regards
    Dhaval Khandla

  • 0 in reply to Marko Kastelic
    Expert 1770 points

    Hi Dhaval,

    I press resolved by mistake... I can't undo mark :(

    You ask me for the location and return of function when TX buffer is full and TX not working:

    Function return RED_ERR.
    Do you have any idea what can be the reason for this case? Can you briefly explain what are the conditions for TX operations on PRU?
    What I also notice is that reloading debug session would not help in recovering this situation. It helps just if I manually unplug the device(power reset of processor). Because PRU is reloaded every time after boot of CPU (in taskPruss task) is this phenomenon so mysterious. 
    Can NDK initialization play any rule?

    Best regards, Mare

  • 0 in reply to Marko Kastelic
    Expert 1770 points

    Hi Dhaval !

    I put in RedTxPacketEnqueue a debug print of write & read pointer from the queue and new write pointer and length of the packet (multiplied with 18 as the driver does). I also print out addresses of data where length and write_ptr are written.
    When TX doesn't work:

    Version - HSR/PRP 1.0.2.0
Board name      :
Chip Revision   : PRUSS DONE!
SYS/BIOS PRP Sample application running
Mac Id          : 9c:1d:58:c5:49:a6
 q_wr_p: 0x14AC
 q_rd_p: 0x14AC
 wrk_p: 0x14B8

 wr_ptr: 0x14B8
 len: 0x1080000

 ptr add: 0x54403eb6
 len_adr: 0x544114ac



Device config : PRP

HSR/PRP Application Menu Options. Press key (Upper/Lower)
*******************************************
S : Show Statistics
C : Show HSR/PRP Configuration
N : Show Ring members/Node Table
I : Assign IP address
P : Show PTP/1588 status
R : Run Rx/Tx test
H : Help menu. Shows details on all the options
********************************************

 q_wr_p: 0x14B8
 q_rd_p: 0x14AC
 wrk_p: 0x14C4

 wr_ptr: 0x14C4
 len: 0x1080000

 ptr add: 0x54403eb6
 len_adr: 0x544114b8

 q_wr_p: 0x14C4
 q_rd_p: 0x14AC
 wrk_p: 0x14D0

 wr_ptr: 0x14D0
 len: 0x1080000

 ptr add: 0x54403eb6
 len_adr: 0x544114c4

 q_wr_p: 0x14D0
 q_rd_p: 0x14AC
 wrk_p: 0x14DC

 wr_ptr: 0x14DC
 len: 0x1080000

 ptr add: 0x54403eb6
 len_adr: 0x544114d0

    and soo on.... read pointer doesn't move.

    After hard reset:

    Version - HSR/PRP 1.0.2.0
Board name      :
Chip Revision   : PRUSS DONE!
SYS/BIOS PRP Sample application running
Mac Id          : 9c:1d:58:c5:49:a6
 q_wr_p: 0x14AC
 q_rd_p: 0x14AC
 wrk_p: 0x14B8

 wr_ptr: 0x14B8
 len: 0x1080000

 ptr add: 0x54403eb6
 len_adr: 0x544114ac



Device config : PRP

HSR/PRP Application Menu Options. Press key (Upper/Lower)
*******************************************
S : Show Statistics
C : Show HSR/PRP Configuration
N : Show Ring members/Node Table
I : Assign IP address
P : Show PTP/1588 status
R : Run Rx/Tx test
H : Help menu. Shows details on all the options
********************************************

 q_wr_p: 0x14B8
 q_rd_p: 0x14B8
 wrk_p: 0x14C4

 wr_ptr: 0x14C4
 len: 0x1080000

 ptr add: 0x54403eb6
 len_adr: 0x544114b8

 q_wr_p: 0x14C4
 q_rd_p: 0x14C4
 wrk_p: 0x14D0

 wr_ptr: 0x14D0
 len: 0x1080000

 ptr add: 0x54403eb6
 len_adr: 0x544114c4

 q_wr_p: 0x14D0
 q_rd_p: 0x14D0
 wrk_p: 0x14DC

 wr_ptr: 0x14DC
 len: 0x1080000

 ptr add: 0x54403eb6
 len_adr: 0x544114d0

    read pointer follow as it should! Did PRU handle an interrupt or anything else? Something that is not reset or it stays in an undefined state. Please just give me a hint of what to check!
    Could be related with the custom board? Because I don't have access to PRU code, I kindly ask for any direction or hints about what all is possible to check...

    Best regards, Mare

  • 0 in reply to Marko Kastelic
    Expert 1770 points

    Hi Dhaval,

    maybe is totally irelevant info, but "to try is not a sin" :)
    PRU_ICSS1_PRU0 make in dissemble code some long jumps and loops.
    PRU_ICSS1_PRU1 loop from addrss 0x000280 to 0x000284.... seems like "while()"... is that true? Is this observation goes or point anywhere?

    Regards, Mare

  • 0 in reply to Marko Kastelic
    Expert 1770 points

    Hi Dhaval,

    It seems that location belongs to IEP timer.

    This is loop:

    C26 point to IEP:

    I'm not familiar with that timer, but it seems disabled?
    Any idea?
    register R5 is some location from shared ram (probably PTP related value) which seems the same.

    Sir, I really need some feed back or idea... 
    Regards, Mare

  • 0 in reply to Marko Kastelic
    Expert 1770 points

    Hi Dhaval,

    do you still track this issue?

    Regards, Mare