The Hindeburg Buring in 1937In my previous blog “Where Did I Go Wrong? Funny Mistakes in Electrical Engineering” I discussed some simple, yet devastating things that led to circuit failures.  In this post, I’m going to touch on a couple items that caused several system wide failures and yes… one is energy related. So sit back, enjoy a cup of your favorite caffeinated beverage (or decaf if you have the jitters) and read on.

The Dreaded “Test” Pin

It was the early 90’s and my company was involved with designing a new type of fax machine for Eastern Europe.  The Berlin Wall had come down and Germany had been reunited… the only problem was my German brethren had inherited the ancient Soviet era electronics and telecommunications from the east.  The Deutsche Bundespost (the equivalent of our US postal service) controls all telecommunications, so they were tasked with “upgrading” all the systems to their current standards.  The problem was in the quality of the wiring and amplifiers used by the Soviets… a normal fax machine would not communicate reliably due to the high error rates.  So, our customer proposed using Forward Error Correction (FEC) in the protocol – unique to Germany – to fix the problem.  The only issue was that it didn’t exist yet.  They won the contract with the Bundespost and came to us for help. 

We had a fax machine reference design that was based on a processor and an ASIC which made up the majority of the electronics.  All that was needed were a few changes to the ASIC and some code to facilitate the FEC.  It took about 6 months to build the first working prototypes based on our reference design… and all was going well – until the phone call came in.  It appeared that the ASIC was draining the coin cell backup battery in about 2 days losing all the settings.  My boss at the time challenged me to find a solution, and if I could without re-spinning the ASIC, a case of Champagne would be mine!

As you can imagine, this was going to be a fox hunt.  The problem was hiding somewhere in the ASIC, but where and why? I flew to California to work with the ASIC designers and see if anyone had any idea why this was happening.  After poring over die plots, schematics and system block diagrams I finally asked the question if anyone had run the system under a microscope... the answer was “no”.  I was also told this was impossible due to the light of the microscope causing offsets in the logic due to the photoelectric effect.  You couldn’t probe the die while looking at it.  I suggested we place the probe and turn the lights off… all of them – lab included.  After everyone stopped laughing we were busy hooking it up in the lab.

About a day of probing found a signal that was not detected during the device functional testing running around the ASIC even when powered down (in stand-by).  It was noise from some source – but where? With a bit of snooping we found that a system test pin with extremely high input impedance was the source.  The documentation showed the pin as a “no-connect”, so the customer did just that… they left it floating.  We grounded the pin and the problem was solved.  The RTC now ran for years without draining the coin cell and the customer was extremely happy… Oh and that case of Champagne… I’m still waiting!  But the satisfaction of finding such an obscure thing was reward enough… plus I can share it with you.  The lesson here is to question things like “no-connect pins”… and exactly what they are for.  That pin had been connected on the tester and held high (test mode), but was missing an internal pull-down which would have solved the system wide problem of draining the stand-by battery.

The Invisible LCD Display

OK, here’s a system integration faux pas that goes beyond a single piece of equipment.  I couldn’t resist writing about this due to the nature of the mistake – it’s on the order of building a steam turbine power plant in the desert (hint: you need water to make steam).  My car was down for repair recently and the dealership provided a loaner.  It was an older model (2 to 3 years old), but was fine for getting around until my new car was repaired.  I jumped into the loaner, put on my Ray Ban – P polarized sunglasses, put the car in drive and left. As I was driving I looked down to see the time on the clock / radio and it had gone black - “Weird” I thought.  I hit the power button a few times – nothing.  I moved on to other thoughts – especially the large tractor trailer rig attempting to turn me into paste, but on arrival at my destination, I removed my glasses and looked back at the clock.  It was back on!  It then occurred to me to put my glasses back on while looking at the display.  Sure enough, the engineers that selected the LCD display for a “car” radio used a display with horizontal polarization.  Sunlight reflecting off non-metallic surfaces – such as water, asphalt, snow, etc. – converts some of the light’s polarization from random to horizontal.  This discovery led to polarized lenses that are “vertically” polarized and almost completely block the horizontally polarized reflected light and thus reduce glare.   Unfortunately, I think the designers of this particular car radio missed that day in physics class…

The moral of these stories are that systems add complexity beyond the combination of the components. Designers need to communicate and understand how circuits, sub-systems and even full systems will be integrated with other components.  I’m sure as the 10,000’s radio came off the line and the phone rang regarding the display someone must have gone – D'oh!  Till next time!