• State TI Thinks Resolved
  • Locked Locked
  • Replies 71 replies
  • Answers 2 answers
  • Subscribers 102 subscribers
  • Views 19678 views
  • Users 0 members are here
Support feedback
Options
Options
Related

This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

AM5728: PRU EtherCAT driver stops receiving frames

Sylvain Lizotte88
Intellectual 370 points
Part Number: AM5728
Other Parts Discussed in Thread: SYSBIOS

We have an issue where our EtherCAT Master does not see any slaves because the PRU ICSS driver is stopping receiving frames after a ESD test until we reboot our target. 

Using WireShark we see that the EtherCAT Master is sending its messages successfully, it is getting the reply from the slaves but for some reason the EtherCAT Master application does not receive the reply and then returning 0 slave.

Prior to that the Master sees all slaves. Once we do the ESD test, it seems as the Master cannot receive any more ETherCAT frames although it sends frames successfully. This is not systematic but easily repeatable.

Rebooting the system is not acceptable for our application. After analysis, here the statistics that we have found:

it seems that after the reception of a bad CRC frame or a first misaligned frame, the PRU ICSS does not recover from it always receiving misaligned frames.

-- 120sec aSt:Idle nSl:7 TL: 0 3TL: 0 STL: 0--
txBcast: 239751 txMcast: 0 txUcast: 0 txOctets: 106407584 rxBcast: 239763 rxMcast: 0 rxUcast: 0 rxOctets: 106414344--
tx64byte: 82065 tx65_127byte: 1 tx128_255byte: 830 tx256_511byte: 78446 tx512_1023byte: 0 tx1024byte: 78449--
rx64byte: 82075 rx65_127byte: 1 rx128_255byte: 830 rx256_511byte: 78456 rx512_1023byte: 0 rx1024byte: 78459--
lateColl: 0 singleColl: 0 multiColl: 0 excessColl: 0 rxMisAlignmentFrames: 0 stormPrevCounter: 0 macRxError: 0 SFDError: 0 defTx: 0 macTxError: 0--
rxOverSizedFrames: 0 rxUnderSizedFrames: 0 rxCRCFrames: 0 droppedPackets: 0 txOverFlow: 0 txUnderFlow: 0 sqeTestError: 0 TXqueueLevel: 100 CSError: 4194305--

-- 125sec aSt:Idle nSl:7 TL: 0 3TL: 0 STL: 0--
txBcast: 254604 txMcast: 0 txUcast: 0 txOctets: 113101336 rxBcast: 254616 rxMcast: 0 rxUcast: 0 rxOctets: 113108096--
tx64byte: 87016 tx65_127byte: 1 tx128_255byte: 830 tx256_511byte: 83397 tx512_1023byte: 0 tx1024byte: 83400--
rx64byte: 87026 rx65_127byte: 1 rx128_255byte: 830 rx256_511byte: 83406 rx512_1023byte: 0 rx1024byte: 83410--
lateColl: 0 singleColl: 0 multiColl: 0 excessColl: 0 rxMisAlignmentFrames: 0 stormPrevCounter: 0 macRxError: 0 SFDError: 0 defTx: 0 macTxError: 0--
rxOverSizedFrames: 0 rxUnderSizedFrames: 0 rxCRCFrames: 0 droppedPackets: 0 txOverFlow: 0 txUnderFlow: 0 sqeTestError: 0 TXqueueLevel: 100 CSError: 4194305--

-- 130sec aSt:Idle nSl:7 TL: 0 3TL: 0 STL: 0--
txBcast: 269463 txMcast: 0 txUcast: 0 txOctets: 119797792 rxBcast: 269475 rxMcast: 0 rxUcast: 0 rxOctets: 119804552--
tx64byte: 91969 tx65_127byte: 1 tx128_255byte: 830 tx256_511byte: 88350 tx512_1023byte: 0 tx1024byte: 88353--
rx64byte: 91979 rx65_127byte: 1 rx128_255byte: 830 rx256_511byte: 88359 rx512_1023byte: 0 rx1024byte: 88363--
lateColl: 0 singleColl: 0 multiColl: 0 excessColl: 0 rxMisAlignmentFrames: 0 stormPrevCounter: 0 macRxError: 0 SFDError: 0 defTx: 0 macTxError: 0--
rxOverSizedFrames: 0 rxUnderSizedFrames: 0 rxCRCFrames: 0 droppedPackets: 0 txOverFlow: 0 txUnderFlow: 0 sqeTestError: 0 TXqueueLevel: 100 CSError: 4194305--

D: ECAT: master state change from 'Op' to 'Unknown' at 131655885
D: EthercatMaster: Unknown: Setting Ethercat Master State to Initialization at 131656358
D: ECATM: Req Master State From 'Unknown' to 'Init' at 131664020

-- 135sec aSt:Idle nSl:0 TL: 0 3TL: 0 STL: 0--
txBcast: 276385 txMcast: 0 txUcast: 0 txOctets: 122118766 rxBcast: 274323 rxMcast: 0 rxUcast: 0 rxOctets: 121985328--
tx64byte: 95640 tx65_127byte: 1 tx128_255byte: 833 tx256_511byte: 89957 tx512_1023byte: 0 tx1024byte: 89957--
rx64byte: 93580 rx65_127byte: 1 rx128_255byte: 830 rx256_511byte: 89956 rx512_1023byte: 0 rx1024byte: 89956--
lateColl: 0 singleColl: 0 multiColl: 0 excessColl: 0 rxMisAlignmentFrames: 2073 stormPrevCounter: 0 macRxError: 0 SFDError: 0 defTx: 0 macTxError: 0--
rxOverSizedFrames: 0 rxUnderSizedFrames: 0 rxCRCFrames: 1 droppedPackets: 0 txOverFlow: 0 txUnderFlow: 0 sqeTestError: 0 TXqueueLevel: 100 CSError: 4194305--
E: ECATM: Req Master State From 'Unknown' to 'Init' Failed at 135721777
D: ECATM: Req Master State From 'Unknown' to 'Init' at 136768452
E: ECATM: Req Master State From 'Unknown' to 'Init' Failed at 141063808

-- 140sec aSt:Fault SF: 0x00000004 nSl:0 TL: 0 3TL: 0 STL: 0--
txBcast: 278659 txMcast: 0 txUcast: 0 txOctets: 122264238 rxBcast: 274323 rxMcast: 0 rxUcast: 0 rxOctets: 121985328--
tx64byte: 97911 tx65_127byte: 1 tx128_255byte: 833 tx256_511byte: 89957 tx512_1023byte: 0 tx1024byte: 89957--
rx64byte: 93580 rx65_127byte: 1 rx128_255byte: 830 rx256_511byte: 89956 rx512_1023byte: 0 rx1024byte: 89956--
lateColl: 0 singleColl: 0 multiColl: 0 excessColl: 0 rxMisAlignmentFrames: 4336 stormPrevCounter: 0 macRxError: 0 SFDError: 0 defTx: 0 macTxError: 0--
rxOverSizedFrames: 0 rxUnderSizedFrames: 0 rxCRCFrames: 1 droppedPackets: 0 txOverFlow: 0 txUnderFlow: 0 sqeTestError: 0 TXqueueLevel: 100 CSError: 4194305--

-- 145sec aSt:Fault SF: 0x00000004 nSl:0 TL: 0 3TL: 0 STL: 0--
txBcast: 280616 txMcast: 0 txUcast: 0 txOctets: 122389486 rxBcast: 274323 rxMcast: 0 rxUcast: 0 rxOctets: 121985328--
tx64byte: 99868 tx65_127byte: 1 tx128_255byte: 833 tx256_511byte: 89957 tx512_1023byte: 0 tx1024byte: 89957--
rx64byte: 93580 rx65_127byte: 1 rx128_255byte: 830 rx256_511byte: 89956 rx512_1023byte: 0 rx1024byte: 89956--
lateColl: 0 singleColl: 0 multiColl: 0 excessColl: 0 rxMisAlignmentFrames: 6293 stormPrevCounter: 0 macRxError: 0 SFDError: 0 defTx: 0 macTxError: 0--
rxOverSizedFrames: 0 rxUnderSizedFrames: 0 rxCRCFrames: 1 droppedPackets: 0 txOverFlow: 0 txUnderFlow: 0 sqeTestError: 0 TXqueueLevel: 100 CSError: 4194305--

-- 150sec aSt:Fault SF: 0x00000004 nSl:0 TL: 0 3TL: 0 STL: 0--
txBcast: 282573 txMcast: 0 txUcast: 0 txOctets: 122514734 rxBcast: 274323 rxMcast: 0 rxUcast: 0 rxOctets: 121985328--
tx64byte: 101825 tx65_127byte: 1 tx128_255byte: 833 tx256_511byte: 89957 tx512_1023byte: 0 tx1024byte: 89957--
rx64byte: 93580 rx65_127byte: 1 rx128_255byte: 830 rx256_511byte: 89956 rx512_1023byte: 0 rx1024byte: 89956--
lateColl: 0 singleColl: 0 multiColl: 0 excessColl: 0 rxMisAlignmentFrames: 8250 stormPrevCounter: 0 macRxError: 0 SFDError: 0 defTx: 0 macTxError: 0--
rxOverSizedFrames: 0 rxUnderSizedFrames: 0 rxCRCFrames: 1 droppedPackets: 0 txOverFlow: 0 txUnderFlow: 0 sqeTestError: 0 TXqueueLevel: 100 CSError: 4194305--

So it seems that the problem is more TI related than EtherCAT related. 

What do you think? Is this a known issue? How can I address it?

I did test a solution that calls PRUICSS_pruDisable(), PRUICSS_pruReset(), PRUICSS_pruEnable() and then restart the master but it did not solve the issue (I tried only disable/enable and only reset and both).

Thanks,

     Sylvain

  • 0 in reply to Sylvain Lizotte88
    TI__Mastermind 39420 points
    That is the memory map create at the end.
  • 0 in reply to Paula Carrillo
    Intellectual 370 points
    I do understand this, I'm looking for the symbol APP_CACHED_DATA_BLK1_MEM?
  • 0 in reply to Sylvain Lizotte88
    TI__Mastermind 39420 points

    Sylvian, the memory map is an output file that is created along with app.out, so you can see where everythings is located. Please add it to your project, just use your appName dot map

    On the other hand, I found where is located APP_CACHED_DATA_BLK1, below snapshot:

    As mentioned before, please use loopback demo from PDK1.0.7 as your project example

    Thank you,

    Paula

  • 0 in reply to Paula Carrillo
    TI__Mastermind 39420 points

    About your second error

    Do you have "IntXbar" variable configured in you project *.cfg file? (as shown below)

    - var IntXbar     = xdc.useModule('ti.sysbios.family.shared.vayu.IntXbar');

    If so, could you comment this?

    Please see attached loopback test project configuration file for your reference:

    icss_emac_arm_wSoCFile.cfg

    thank you,

    Paula

  • 0 in reply to Paula Carrillo
    Intellectual 370 points

    Hi Paula,

    1. We are at the end of a 2-year development project, so we are using an old cfg file, adding the line "var IntXbar     = xdc.useModule('ti.sysbios.family.shared.vayu.IntXbar');" just solve the second issue "In function `MuxIntcP_create': AM572X/muxintcp/MuxIntcP_tirtos.c:60: undefined reference to `ti_sysbios_family_shared_vayu_IntXbar_connect__E'", thanks.

    2. About the build configuration file to get APP_CACHED_DATA_BLK1_MEM, what are the implication, we are not using one right now. If I include C:\ti\pdk_am57xx_1_0_7\packages\ti\build\am572x\config_am572x_a15.bld in my project what will happen with the memory mapping?

    Regards,

         Sylvain

  • 0 in reply to Sylvain Lizotte88
    TI__Mastermind 39420 points
    Hi Sylvain, understand your concern, please check/modify "config_am572x_a15.bld" to fit your project needs. Maybe you can try a quick (and dirty) include as-is for a quick test, just to see if it works

    thank you,
    Paula
  • 0 in reply to Paula Carrillo
    Intellectual 370 points
    Hi Paula,

    It seems as if I want to use "config_am572x_a15.bld" then I need to completely rebuild my project from scratch. Is there any other way to link the new PRU binary as before? This is just a linking problem to make an upgrade for a fix in the PRU. I would rather use the old header file mechanism, less risky and quicker to integrate. I feel like I getting into something bigger than it needs to be. I will comeback to this issue on Monday morning, hopefully you can provide me an easier upgrade path by then.

    Regards,

    Sylvain
  • 0 in reply to Sylvain Lizotte88
    Intellectual 370 points

    Hi Paula,

    This is my application configuration file: app.zip

    Regards,

    Sylvain

  • 0 in reply to Sylvain Lizotte88
    TI__Mastermind 39420 points
    Hi Sylvain, you are right, we can just change where to locate in memory "PRU0_REV2_XX" inside "lnk_a8_a9_a15_REV2.cmd". We can try to use where you current dataMemory is.

    Just say, you used "EXT_RAM" name for data memory section (please check on your linker.cmd) then, you just need to change in lnk_a8_a9_a15_REV2.cmd:
    > APP_CACHED_DATA_BLK1_MEM
    to
    > EXT_RAM

    let me know if this works.

    thank you,
    Paula
  • 0 in reply to Paula Carrillo
    Intellectual 370 points

    Hi Paula,

        I am trying this, but it seems as making a clean build fails right now having some undefined references that I was not experiencing on Friday. I'm investigating this.

    Meanwhile, it would be much more easier to get the equivalent header file for dual_emac:

    #include <ti/drv/icss_emac/firmware/am57x/v2_1/icss_emac_pru0_bin.h>
    #include <ti/drv/icss_emac/firmware/am57x/v2_1/icss_emac_pru1_bin.h>

    to link the binary as we are doing it before...

    Regards,

        Sylvain

  • 0 in reply to Sylvain Lizotte88
    Intellectual 370 points
    Hi Paula,

    I succeed in linking an image including the new PRU, I will keep you posted on the test results later today.

    Sylvain
  • 0 in reply to Sylvain Lizotte88
    TI__Mastermind 39420 points

    Hi Sylvain, great! thanks for let me know

    Paula

  • 0 in reply to Paula Carrillo
    Intellectual 370 points
    Hi Paula,

    So far, our tests are positive. We did not reproduced the problem in 30 trials, whereas it was reproduceable before with 60% chances. We are continuing testing and we are expecting to confirm our present results by the end of the day.

    Regards,

    Sylvain
  • 0 in reply to Sylvain Lizotte88
    TI__Mastermind 39420 points
    Hi Sylvain, thanks for the note
    Paula
  • 0 in reply to Paula Carrillo
    Intellectual 370 points
    Hi Paula,

    Our tests are positives, we cannot reproduce the problem anymore.

    Now, our problem is a test effort one because we did update all TI libraries that have been used (and tested) for the past year.

    From
    com.ti.sdo.edma3:2.12.1;com.ti.rtsc.NDK:2.25.0.09;com.ti.rtsc.SYSBIOS:6.45.1.29;com.ti.pdk.am57xx:1.0.5
    To
    com.ti.sdo.edma3:2.12.4;com.ti.rtsc.NDK:2.25.1.11;com.ti.rtsc.SYSBIOS:6.46.5.55;com.ti.pdk.am57xx:1.0.7;

    We need to retest everything. So If it is possible it would help us a lot to get a simple update of PDK 1.0.5 with this fix. This would reduce a lot our risk and our test effort.

    Regards,

    Sylvain
  • 0 in reply to Sylvain Lizotte88
    TI__Mastermind 39420 points
    Hi Sylvain, I understand, but as far as I know we don't do back fix porting, as it would be difficult for us to maintain. We always suggest our user to update to latest

    Paula
  • 0 in reply to Paula Carrillo
    Intellectual 370 points

    Hi Paula,

       I have done an overnight test without ESD testing and I have the following statistics:

    -- 55165sec aSt:Idle nSl:1 TL: 0 3TL: 0 STL: 0--

    txBcast: 164916074 txMcast: 0 txUcast: 0 txOctets: 411162832 rxBcast: 164916227 rxMcast: 0 rxUcast: 0 rxOctets: 411179152--
    tx64byte: 109944442 tx65_127byte: 1 tx128_255byte: 54971979 tx256_511byte: 0 tx512_1023byte: 0 tx1024byte: 0--
    rx64byte: 109944624 rx65_127byte: 1 rx128_255byte: 54972069 rx256_511byte: 0 rx512_1023byte: 0 rx1024byte: 0--
    lateColl: 0 singleColl: 0 multiColl: 0 excessColl: 0 rxMisAlignmentFrames: 0 stormPrevCounter: 0 macRxError: 0 SFDError: 0 defTx: 0 macTxError: 0--
    rxOverSizedFrames: 0 rxUnderSizedFrames: 0 rxCRCFrames: 0 droppedPackets: 0 txOverFlow: 0 txUnderFlow: 0 sqeTestError: 0 TXqueueLevel: 0 CSError: 4194305--

    Why the PRU ICSS is reporting that many carrier sense errors?

    It looks like an used and uninitialized statistic since it is always set to this value at startup.

    -- 5sec aSt:Idle nSl:1 TL: 0 3TL: 0 STL: 0--
    txBcast: 6106 txMcast: 0 txUcast: 0 txOctets: 629712 rxBcast: 6175 rxMcast: 0 rxUcast: 0 rxOctets: 637392--
    tx64byte: 4349 tx65_127byte: 1 tx128_255byte: 1920 tx256_511byte: 0 tx512_1023byte: 0 tx1024byte: 0--
    rx64byte: 4445 rx65_127byte: 1 rx128_255byte: 1969 rx256_511byte: 0 rx512_1023byte: 0 rx1024byte: 0--
    lateColl: 0 singleColl: 0 multiColl: 0 excessColl: 0 rxMisAlignmentFrames: 0 stormPrevCounter: 0 macRxError: 0 SFDError: 0 defTx: 0 macTxError: 0--
    rxOverSizedFrames: 0 rxUnderSizedFrames: 0 rxCRCFrames: 0 droppedPackets: 0 txOverFlow: 0 txUnderFlow: 0 sqeTestError: 0 TXqueueLevel: 0 CSError: 4194305--

    Thanks,

        Sylvain

  • 0 in reply to Sylvain Lizotte88
    TI__Mastermind 39420 points
    Hi Sylvain, I will check this and come back to you. Please for new questions help me to open new E2E threads, so it would be easier to follow-up and to assign to the right person
    thank you,
    Paula
  • 0 in reply to Sylvain Lizotte88
    TI__Mastermind 39420 points

    Hi Sylvain, I found in "icss_emacStatistics.h" below comment:

    //Debug variables, these are not part of standard MIB. Useful for debugging

    // Reserved for future Use

       volatile uint32_t txOverFlow;               /**Tx FIFO overflow count*/

       volatile uint32_t txUnderFlow;              /**Tx FIFO underflow count*/

       volatile uint32_t sqeTestError;             /**Number of MAC receive errors*/

       volatile uint32_t TXqueueLevel;             /**Current Tx queue level*/

       volatile uint32_t CSError;                  /**Number of carrier sense errors*/

    I

    I will confirm that this is the case and come back to you.

    thank you,

    Paula

  • 0 in reply to Paula Carrillo
    Intellectual 370 points

    Hi Paula,

       We have completed our tests and we could not reproduce the problem. So we can close this issue.

    Thanks,

          Sylvain

  • 0 in reply to Sylvain Lizotte88
    TI__Mastermind 39420 points

    Hi Sylvain, thanks for let us know =)

    Paula