Sunday, May 15, 2016

Diagnostics: But Everything is the Same, Almost.

Here is how asking the right questions and looking for what changed can help you diagnosis a system issue.

I had recently joined a company that designs and builds industrial test systems and their largest client had just outfitted a new test lab with 20 of our automated life-test systems (testing DeWalt power tools) when they called with an issue. The client was getting erratic data speed data during tool testing.

The first step for us was to duplicate the problem. We pulled a development unit back into the test area and tried to match the customer set up but could not duplicate the issue. Back on the phone with the customer to ask if anything had changed since they signed off on the design prior to the production build. The customer did finally mention that that they found a better price for rotary encoders for the test benches and had just standardized on the new model. So we asked them to send us one of the new encoders next-day air.

We installed the new encoder and were able to duplicate the erratic speed data. Now we were getting somewhere! Next I put a scope probe (older analog Tek) on the signal. At first glance the encoder signals looked clean (looked at both A and B outputs but only monitoring one with the single counter/timer input) but a closer inspection showed an irregular narrow ghost of a 900-1200 mv glitch when the signal was in the Low state. I verified that the National Instruments data acquisition card's counter/timer channel had a TTL input and took a quick look at the TTL and CMOS voltage limits.
Examples of binary logic levels
TechnologyL voltageH voltageNotes
CMOS0 V to 1/3 VDD2/3 VDD to VDDVDD = supply voltage
TTL0 V to 0.8 V2 V to VCCVCC = 5 V ±10%

I looks like the signal bounce is close to the voltage difference between a TTL low and TTL high and could be causing the problem. From looking at the chart we can see that changing to a CMOS input would increase the noise margin from 1.2V for TTL up to 1.5V for CMOS. The noise margin improvement might be enough to improve the speed readings but I needed a way to test the theory.

Now how to get my hands on a CMOS buffer? A quick search of the parts bin turned up a box of CMOS Quad NAND chips. So I wired up two NAND gates as inverters and put them in series to give us a buffer that we can run from 5V that will duplicate the incoming TTL. I may have been able to use a single CMOS inverter but I was not sure if the system performed any duty cycle measurements so I wanted to keep the buffered output in-phase with the original encoder output. The signal coming from the NAND chip was clean and the rpm measurement glitch disappeared.

Being an engineer by nature, I quickly design a small circuit board that could attach to the back of the tach connector on the test system and use the existing wiring to provide the additional signal conditioning needed to clean up the signal from the cheap encoders. I got a quote for 40 boards (pcbs are cheap, always order extras) to update the systems and called our client to share the results.

In the end the customer elected to go back to their previous encoders in the new lab rather than implement a fix to make the cheaper encoders work.  A good decision since the the cheaper encoders would have problems if installed on other units in their installed base of test systems.

I hope this information can help you ask the right questions and find the clues you need during your next diagnostic adventure!