• State
  • Locked Locked
  • Replies 28 replies
  • Subscribers 80 subscribers
  • Views 2711 views
  • Users 0 members are here
Support feedback
Options
Options
Similar topics

This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

TMDSCNCD28M36 R 1.1 Ethernet failure

Galen Hammerle
Intellectual 810 points
Other Parts Discussed in Thread: TLK110

I have had two TMDSCNCD28M36 Control Cards installed in an instrument (custom daughter board interface) in which the Ethernet connection just stopped working. Once I plug a cable into the port the lights simply turn off. has anyone else seen this issue? Is there any value in sending these back to TI for investigation? We use these control cards in a 10 custom applications and are becoming concerned. The serial numbers are 1210030 and 1210059 if that helps with any diagnostics. The Ethernet is connected to a host PC using a Belkin USB 2.0 to Ethernet adapter. Other questions are welcome.

  • 0
    TI__Guru 57255 points

    Hi,

    I think you're going to need to provide more information for any response to be helpful.

    Things like:
    - What revision controlCARD and F28M36x chip are you using?
    - Before the cCARD suddenly stopped working, had the controlCARD been working correctly? 
    - Is there anything special that happened around the time that the failure occurred?
    - Does there appear to be any damage/wear on or around the Ethernet connector (especially if you're doing a fair amount of connect/disconnect cycling)
    - Can you reprogram the F28M36x chip still?  Can you run an ENET example successfully on the non-working boards?


    Thank you,
    Brett

  • 0 in reply to Brett Larimore
    Intellectual 810 points
    Brett,

    The revision is 1.1.
    The control card (most recent to fail) has been installed for 5-6 months running M-F since.
    I have not seen any damage even using a microscope; on components or connectors.
    I can still program the chip. I will try the ENET example again, but do not think I will have much luck as the port lights turn off once a device is connected. Even the PC (in the instrument and at my desk) ENET lights turn off if I remember correctly. I will verify this information later today.

    Galen
  • 0 in reply to Galen Hammerle
    Intellectual 810 points

    So i have downloaded the enet_lwip_m3 sample program it dies on line 312 strange it is a UARTprintf call. The connector lights never come one anyway when a cable is plugged in so I am not sure what is next i really do not want to debug the sample code. All my working cards are installed so i can not see if a working card works with the sample code at this time.

    The version of the sample code is :

    // $TI Release: F28M36x Support Library v200 $
    // $Release Date: Wed Apr 17 14:46:03 CDT 2013 $


    I think this is after all the pinout files were updated but not sure.

    Galen

  • 0 in reply to Galen Hammerle
    Intellectual 810 points
    Any update? I have answered the initial questions sent out.
  • 0 in reply to Galen Hammerle
    TI__Mastermind 19810 points
    can you check the PHYRST status; also check all the GPIOs on control card are connected to the PHY, I beleive there are switches that needs to be flipped right?
    Also it might be a good idea to test the Ethernet on a controlcard before you install in the instrument, you might have already did this? and it worked?
    Best RegardsSantosh Athuru
    once you have checked the HW, getting the example to run is probably the right step to debug this. Can you debug and check why it is failing at the UART printf? you don;t need UART printf to be working for this, see if you can disable that DEBUG.
  • 0 in reply to Santosh Athuru
    TI__Guru 57255 points

    Hi Galen,

    One other minor question that may help us: What revision of F28M36x MCU are you using?  You can find this out by looking at the second line on the chip itself.

    "YF# -.....", where # is the revision number.  # = blank -> Rev0; # = A -> RevA; # = B -> RevB; etc


    Thank you,
    Brett

  • 0 in reply to Santosh Athuru
    Intellectual 810 points

    can you check the PHYRST status

    How do I do this?

     also check all the GPIOs on control card are connected to the PHY, I beleive there are switches that needs to be flipped right?

    Also it might be a good idea to test the Ethernet on a controlcard before you install in the instrument, you might have already did this? and it worked? 

    Yes I have firmware that we have been running for around a year on these control cards. Only two cards have failed after running for a number of months.

    Best RegardsSantosh Athuru

    once you have checked the HW, getting the example to run is probably the right step to debug this.

    Is the version of the sample code I listed above one that has had all the IO pin outs corrected? (I know this was a problem early one where I had to update the sample code to match the schematic.)

    Can you debug and check why it is failing at the UART printf?

    I could but the UART printf does work through on the bad Ethernet boards when I run my firmware. (Initialization send status to Debug port)

    you don;t need UART printf to be working for this, see if you can disable that DEBUG.

  • 0 in reply to Brett Larimore
    Intellectual 810 points
    Brett,

    The chip revision is YF28M36P63C2T 29A/GPW for both control card chips.

    Could an IO from the control chip be hanging the Ethernet chip? Is that the thought?
  • 0 in reply to Galen Hammerle
    TI__Guru 57255 points

    Hi Galen,

    Sorry for the delay in reply.

    A few of us have taken a look at your posts and we have a few thoughts:
    1) In your original post, you say that, " once I plug a cable into the port the lights simply turn off".  Which lights are you referring to?
    2) Can you try to put the cCARD that is no longer working into a Docking Station baseboard and see if you still have an issue?  If so, can you try placing a jumper wire between ground and the pin labelled 120 in the Docking Station (which ties to the MCU reset pin) to see if you still see the same behavior?
    3) If you baseboard has high-power electonics on it (ex. the F28M36 chip controls some power stage), have you utilized isolation between the hot-side ground and the cCARD's/ENET's ground?  If not, perhaps the hot side ground is creating transients and is potentially creating ground loops through the ethernet connection (and I could see this potentially damaging equipment - perhaps the TLK110).


    Thank you,
    Brett 

  • 0 in reply to Brett Larimore
    Intellectual 810 points
    The lights are the lights on the RJ45 connector. The light actually turns one on power up then stays off with out a cable plugged in I need to check normal behavior with a different card.

    I tried shorting pin 120 to ground. When I release the pin my program does not restart until I power it up. (Sounds like a problem in my code to me, since we do not use the reset pin.) after I cycle power my program runs but I can not connect with Ethernet.

    Nothing connected to the base board is high power (in my mind) we have some buffered digital IO two Dac's using the SPI ports. a few analog inputs and some serial ports. My grounds are not isolated from each other though. In the instrument we do have stepper motors and their amplifiers are tied to chassis ground near the amps. In the past (other systems) if we had ground loops our galvanometers would show an issues as the analog voltage would change creating movement we would detect in our imaging system. I have not seen any instances of this.

    We do have new cards on order to keep the unit running. I guess it comes down to is it worth TI engineering time to investigate the two failed cards?

    Galen
  • 0 in reply to Galen Hammerle
    TI__Guru 57255 points

    Hi Galen,

    The PHY itself runs and chooses how the LEDs on the ENET connector are to be lit.  My thought is that the issue you're running into had something to do with the PHY chip failing - my best guess was something to do with ground loops.  My purpose in having you tie the F28M36 in reset (by putting pin 120 to ground) was to see if the PHY would work as expected when the C2000 could not do anything to it. 

    Based on your new response I'm a bit less sure - potential power supply issues are now seeming feasible.

    ===

    With respect to your main question:  I don't think so.  I haven't heard of this issue outside of your situation - which makes me think it's something specific to your setup.  That being said, if you see another board fail, and the failure is consistent with the others, go ahead and update this thread.  Seeing that another failure occurred, separate in time from the other failures, may provoke more investigation from our side.


    Thank you,
    Brett

  • 0 in reply to Brett Larimore
    Intellectual 810 points
    So it has been six months or so and another Control Card has failed in the same manner. The serial number is 1210019 and the chip rev is the same as the others mentioned in this thread. All of these that have failed appear to have been produced close in time based on serial numbers. Hopefully the replacement cards will run a bit longer.
  • 0 in reply to Galen Hammerle
    TI__Guru 57255 points

    Hi Galen,

    I'm surprised I didn't ask this prior, but:

    Does anything on the board get hot after the failure?
    Do you have the ability/Is it possible for you to try replacing the TLK110 chip? 


    Thank you,
    Brett

  • 0 in reply to Brett Larimore
    Intellectual 810 points

    I have not noticed any physical damage when i place the boards under a microscope. Nothing is warm to the touch when the board is plugged in either.

    I have not changed out the TLK110 chip but was leaning toward sending the board out to have the chip replaced. (Our rework capability is a bit limited and do not want to add another variable to the mix.) I can order some parts today as I need to order other components for another project.

    Should this be a direct part swap and then start up?

    Galen

  • 0 in reply to Galen Hammerle
    TI__Guru 57255 points

    Hi Galen,

    Should this be a direct part swap and then start up?
    [BL] Yes, that is my understanding.

    Please let me know how it goes.


    Thank you,
    Brett

  • 0 in reply to Brett Larimore
    Intellectual 810 points
    So another board Serial 1210026 died on Friday (4/22/2016). These are all still from an earlier order. I did order the TLK110 chip I have tried to replace one so far it is not working I will inspect my workmanship to see if I have a solder bridge somewhere. The rest of the card is still working so at least I did not short the power along the way.
  • 0 in reply to Galen Hammerle
    Intellectual 810 points
    No obvious shorts from the rework.... not sure what to try next. Any tips on where to probe around with a scope perhaps?
  • 0 in reply to Galen Hammerle
    TI__Mastermind 19810 points
    do you see any clocks on TXCLK on the PHY?
  • 0 in reply to Santosh Athuru
    Intellectual 810 points
    Bad boards have TXCLK. The boards I attempted to swap the TLK110 does not, guess my rework skills are not up to par.
  • 0 in reply to Galen Hammerle
    Intellectual 810 points
    So another piece of information the bad (ignore the rework control card) boards have a TXCLK at 2.5Mhz and does not change when a cable is plugged in. A good board will be at 2.5Mhz and then go to 25Mhz when connected.
  • 0 in reply to Galen Hammerle
    TI__Mastermind 19810 points
    That depends on if the PHY is able to auto-negotiate to 100MBps on link. Check the Boot strap config you have for the PHY as per the PHY manual and see if auto-negotiation is enabled or not.

    Best Regards
    Santosh Athuru
  • 0 in reply to Santosh Athuru
    Intellectual 810 points

    Santosh,

    I think this is just another symptom of the failure. I am using control card as my controller for a daughter board. The firmware is identical on a good and bad control card. The TLK110 just seems (not proven) to fail after we run for about 6 months or so on the earlier batches.

    Galen

  • 0 in reply to Galen Hammerle
    TI__Mastermind 19810 points
    Galen,
    right, if you still have a board which was working for 6 months and stopped working suddenly - can we check for the TXCLK on that board. We might have to use the MII interface on Concerto MAC and read some PHY registers and see what is going on.

    At least we need to know if the PHY failed or not or do we know that already?

    Best Regards
    Santosh
  • 0 in reply to Santosh Athuru
    Intellectual 810 points
    The TXCLK is present at 2.5Mhz on a board that failed after 6 months of use. It would be interesting to read the PHY registers could you guide me thru this? All I have done was set up the ENET pins a few years ago following the example in Control Suite.
  • 0 in reply to Galen Hammerle
    TI__Mastermind 19810 points
    Galen,
    refer to section 21.6 in M36x TRM, it tells about the MII registers and the registers you can use to read the PHY registers. This along with the PHY registers from the PHY manual should help you identify what is going on with PHY.

    But it already seems like it is failing auto-negotiation? Try to set your HOST/PC to 10MBPS and see if the communication comes back again on the failed board?

    Best Regards
    Santosh Athuru
  • 0 in reply to Santosh Athuru
    Intellectual 810 points

    So I went back and tried the rework swap of TLK110 again and success; I once again have Ethernet communication. The TLK110 on my failed boards (the TI F28M36x Control Cards) appears to be the bad component. Question is why would this component fail? At least I have a >$10 part repair cost to salvage the boards that have failed. These are in house units but i would like to use this control card in another project I am working on but have concerns as this project will not stay local.

  • 0 in reply to Galen Hammerle
    TI__Guru 57255 points

    Hi Galen,

    My best guess comes from the discussion in an E2E thread that came up not too long ago.

    In this controlCARD, TLK110 pins 18, 23, and 37 are tied to VDD_3V3.  According to the TLK110 datasheet, these pins should not be tied to 3.3V.  Instead (and because the TLK110 in this system is supposed to be used in its single-supply configuration), PFBOUT is supposed to source the PFBINs and have capacitance on the rail for stability.  My guess is that the problem you are seeing with aging is some combination of (1) the chip being partially powered over specifications long term and/or (2) the TLK110's internal supply attempting to overpower the 3.3V rail long term.

    This would also explain why replacing only the TLK110 resolved the issue.

    I took a look at the controlCARD PCB design and, unfortunately, there doesn't appear to be an easy way to blue-wire around the problem. 

    Realistically, it will be quite some time before we have the ability to design and build revised controlCARDs where the problem described here is fixed.


    Thank you,
    Brett

  • 0 in reply to Brett Larimore
    Intellectual 810 points
    Thanks Brett.

    So to close this thread out; should we leave it with your above comment and just a "Be Aware" to anyone using the control card for long term Ethernet usage that the TLK110 may fail and need to be replaced.