Some Hardware Problems You Cannot Fix in Software

Many years ago I designed a fast, high-power DC module for  National Instruments' Signal Conditioning Chassis that would allow our client to use a low level logic signal to pulse width modulate control a 24VD DC fan clutch being designed for an automotive cooling system, while also providing signal conditioning of a high-voltage signal from an inductive tachometer.  

The NI Signal Conditioning Chassis (SCC) system accepts modular signal conditioning modules that allow customers to mix and match input and output signal levels.  Our versatile DC I/O module design actually proved quite versatile, and was used in several other applications for power control, and in at least one production test system for marine controls for dual high power CAT diesel marine engines. 

I used this Feedthrough module to develop my prototype.
http://sine.ni.com/psp/app/doc/p/id/psp-473/lang/en

The client was very happy with the resulting modules, and specified them as standard equipment for all of their product development test stands in both North Carolina and Michigan.  

Speed Glitches?

During one follow-up call the client casually mention being frustrated by occasionally erratic readings with my module, and being an engineer who wants their designs to ALLWAYS work, caused me some anxiety. A few quick questions clarified that the same problem predated my module design. It turns out they had been trying to address the erratic tachometer readings with software filters for almost two years. 

With a little prodding, the client put scope on the module and verified that tachometer input into my module, and the conditioned signal output of my module were identical other than scaling and voltage shifting, so my circuit design was in the clear. The client still had a problem to solve but could provide no additional information. 

Our local field engineer visited the customer asking if they have any new test or automation projects. The engineer called me after the visit and told me the client was very pleased with the modules I designed, and the client demonstrated the tachometer speed reading glitch for him.  He said he could see the incoming signal trace moving up the oscilloscope screen on my input signal, while doing the same thing on the output signal from my module. 

Something sounded off in his description, so I asked him "Do you mean you saw the rising edge of the square-wave? His reply was "No, it was not a square-wave, we could watch the voltage rise smoothly as the voltage from the inductive pickup increase". I felt like we had cornered the dragon and found the detail that would let me solve the issue!

Slowly changing signals can wreak havoc on a digital input, with any noise causing multiple transitions for what should be a single edge, and the slower the signal, the worse the problem. But this is well-trodden ground with known solutions. Adding just a little positive feedback can add hysteresis to the system by creating separate thresholds for low-to-high and high-to-low transitions. Having two different thresholds can prevent multiple-transitions. 

I suspected that the quickest solution would be changing to an optocoupler with a Schmitt-trigger input. A quick search through DigiKey turned up a chip that was signal level compatible with my design but with a different pinout. I ordered a handful of chips for overnight delivery, and showed up early the next day to watch for the FedEx van to roll through the parking lot!

I quickly modified a module dead-bug style and verified the Schmitt-trigger input device worked on my bench. Next I called the client and told him I have an idea that might solve his signal glitch, and that I can hand-modify a handful of modules for verification testing, and I should be able to send them out by the end of the week.  

A week later the engineer called, thrilled to tell me the signal glitches were completely gone! Not better, gone! For the first time in two years there were no glitches in their speed data, and his test stand was the only one in the world that could do it properly! The client immediately ordered replacements for every IO module (24 units) to upgrade all of their test stands in North Carolina and Michigan. 

One stumbling block we had was that the client didn't want to return the hand-prototypes modules from his system because he now had the only properly working test bench in the world for that product line. Out compromise was that I would no-charge him for the last three replacement modules, but wouldn't ship them out until I received my returned hand-wired prototypes. 

My client was a company hero for fixing the long lingering problem with the tach sensor's slow-speed reading jitter. I love it when something clicks from one of those early EE lectures, or those hours of reading application notes, and hands you exactly the tool you need to slay a dragon!

Moccamaster KBTS - Single Cup Brewing Adapter

I created this adapter to let me use my Moccamaster KBTS coffee maker as a single cup brewer. This model comes with a thermal carafe that normally presses against a button on the base of the machine to initiate brewing. Using one of my favorite coffee cups to press the button positioned the stream of brewed coffee near the right lip of the cup. While it worked, it made me nervous about the risk of misalignment or using a smaller coffee mug. So I put on my designer hat and went to work.

https://makerworld.com/en/models/1038925#profileId-1023339

What I needed was something to center my cup under the brew basket that also presses the brewing. After sketching a few concepts, 3D printing some rough models, I settled on the design you see here that  positions my cup exactly where I want, while an integrated sliding bar presses the brew button.

You might be thinking, "Isn't the Moccamaster Cup-One designed to brew a single cup of coffee?" When I decided to try a Moccamaster brewer, choosing between the single-cup model and a larger model was a challenge. While I typically brew one cup at a time, I liked the flexibility of brewing into a thermal carafe when needed.

The deciding factor came down to the difference in water distribution systems. The Moccamaster Cup-One features a water outlet arm with a single spout. While researching how to get the best cup of coffee from the machine, I found several videos suggesting stirring the grounds for more uniform wetting. 

These aren't affiliate links, I just want to make it easier to learn more about the machines if you are interested. 

https://us.moccamaster.com/collections/single-cup-brewers/products/cup-one?variant=40377761726627

In contrast, the KBTS uses a 9-Hole Outlet Arm to ensure even wetting of the coffee grounds. The KBTS also features a selector on the brew basket that lets you choose between high flow for full-carafe brewing, low flow for four cups or less, or stopping the water flow to allow timed soaking of the grounds. Stopping the brew basket flow also prevents drips when the carafe or coffee cup is removed.

https://us.moccamaster.com/products/kbts?srsltid=AfmBOoplqVF91G7LbxcNGmwP8-JSGMigKM-Zb4dk_gBvDV48bIlCAopj

Additionally, the KBTS includes a thermal carafe with a Brew-Thru carafe lid, which is an impressive piece of engineering. The Brew-Thru lid minimizes coffee exposure to air and introduces coffee into the carafe from the bottom up. This design helps maintain the coffee's strength and temperature uniformly within the carafe. It also reduces oxidation, minimizes evaporative cooling, and preserves flavorful volatile compounds, keeping the coffee as warm and flavorful as possible without additional heating. Very nice engineering!

If you own a KBTS, I suggest downloading my adapter model and giving single-cup brewing a try.

Thank you!
Shane

Minimal Carry Handle for FRC Robot Batteries

This 3D printed minimal battery handle requires less than 15 grams of filament (I recommend PETG in a bright color) and quickly clips on and off of the battery. 

https://makerworld.com/en/models/1015389#profileId-995541

One of the most common electrical issues with FRC robots is loose batter terminal connections caused by lifting the battery by the cables. Loose terminals tend to reduce the quality of the contact surface with the battery terminal, and can lead to random reboots or loss of radio connection with the FIRST Field Management System.

Rechecking (because you checked them back in lab, right?) the terminal connections on all of your batteries the morning your competition is always a good idea. If you can wiggle a battery cable terminal using only three fingers, then it is too loose for match play. But it only takes a few minutes to loosen, clean, and retighten the ring terminals. 

I hope this can help provide a starting point for improving battery handling habits.

Good luck on the field!

Shane

Battery Nerd

Mentor, FRC #4561 TerrorBytes

Building a Physical Model for Testing Your Code

The Value of a Physical Model

With most development projects there is usually a little tension between the fabrication team wanting working software to test the the hardware during assembly, and the development team wanting known good hardware for use during development. There are many ways to address this tension and strike a balance. 

My approach of choice is the use of a Physical Model that has functional inputs and outputs similar to those of the unit under development. The goal is create a test board that will allow you to work with an analog of your system's target hardware while you are writing the system's software. 

For a PC based test system, the Physical Model may be a as simple as a bank of switches and LEDs combined with a few voltage displays and a few channels of adjustable power outputs that all connect to the PC's data acquisition system. Systems involving motion control get a little more complicated.

Here is an overview of a Physical Model I built to allow me to validate the electrical design and test the firmware for a proof-of-concept device for a patented process of producing subcutaneous, or plastic surgeons' sutures. The physical model provided a hardware platform for testing the firmware to ensure proper operation before the hardware for the demonstration model was available. Waiting to test on the actual hardware would have made it difficult, if not impossible to meet the client's timeline. 

I used a laser cutter me to quickly layout and cut a panel that would house all of the needed elements and even provide engraved labels. The system below included three DC servo motors with drivers controlled via an RS-485 network. 

The client is a small startup that has multiple patents for automated subcutaneous sutures and needed a hand-held model that could demonstrate the process to potential customers and investors. I worked with them in their earliest efforts which gave me a head start on designing a the proof of concept device to help them move forward.

My assignment was to help with the electrical design and firmware to demonstrate a proof of concept of the clients patented process for automating the application of subcutaneous sutures, or plastic surgeon's sutures. The goal is a works like device to be built into a looks like hand held model. My contribution was electrical design and firmware development while the client was responsible for the physical construction and final assembly of the device.

The client had an aggressive schedule based on demonstrations that they were working on scheduling and an upcoming trade show. As is most often the case, the key to success was using as many off-the-shelf parts as possible.

I knew that the hardware assembly timeline was likely to slip and that I would have little if any access to the final assembly prior to their deadline. So I decided to build my own Physical Model to allow me to test the firmware on a target platform that would simulate all of the electron mechanical parts of the final system.

Why Product Design is Challenging

One of the earliest lessons I learned about designing automated test systems is that the hardware people want working software to test the hardware during the build and the software people want working hardware to test the software during development. There are two components needed to address this chicken and the egg problem.

First, you need a way to exercise all of the inputs and outputs of the data acquisition and control system so the hardware team can test their wiring during assembly. If you are using National Instruments hardware, then you can use their set up panels that allow you set all of the outputs and read all of the inputs of the data acquisition hardware. Still we would often have the programmer start with a minimal framework that exercises all of the system inputs and outputs and displays the status on screen. Writing this software was a great way for the programmer to get comfortable with configuring the hardware without needing to address the functional and user interface aspects of the system under development.

The second component in this solution was building a Physical Model that simulated all of the system IO to allow the programmer to test the functionality of their software against the model during development. Typically this was a panel with LEDs and switches for the digital IO and potentiometers voltage displays for the analog components.

Assembling All of the Pieces

Being an early stage, proof of concept design, I wanted the client to be able to modify and update their control sequence without needing to come back to me with each small change. Using off the shelf hardware at every possible step is the key to successful prototyping. 

I selected SparkFun's Arduino Pro Mini as the control device based on the ease of use of the Arduino IDE (to allow the client to make their own adjustments to the motion sequence target positions) and the small form factor.

I implemented the command sequence as a two dimensional array of data points that would allow the client to modify target position and motor speed settings by simply modifying the array and downloading the new firmware via the Arduino IDE. They could even add or remove steps of the motion sequence by adding or deleting rows from the array.

The system also included several machine safety interlocks that would prevent a data error from physically crashing the servo motor actuators. 

Another useful feature was a serial output that logged the system status enabling the client to see exactly what was going on inside the device. The data reporting was exceedingly useful for determining if a motion sequent stopped because a motion safety was triggered, or if the actuator was unable to reach the target position or if the system stopped due to an over current or under voltage condition. 

The design included three Maxon brushed DC servo motors with high-resolution encoders, controlled by AllMotion EZSV10  DC Servo Motion Controllers. The EZSV10 controllers communicate using RS-485 to configure the motor driver settings, receive position command and to read encoder outputs. The EZSV10 Servo Controllers pack an amazing amount of performance into a tiny package. I highly recommend their quick start kit if you want to jump into brushed DC motor servo control.

http://www.allmotion.com/Servo_Pages/EZSV10Kitdescription.htm

The initial design was powered from a single lithium ion 18650 battery with the servo motor rail voltage being produced by a Pololu U3V50F12 DC-DC Step-Up Regulator.

https://www.pololu.com/product/2568

The next iteration of the design transitioned to an external 12V source. The single 18650 battery had plenty of capacity for the application but could not power the system continuously for extended test session. Catching voltage droop during high-current moves was essential to ensure adequate power to the motors.

Custom Maxon brushed DC servo motors were selected for the final application. But the price and lead time on the Maxon motors motivated me to find alternative DC brushed servo motors for my Physical Model. I found the Faulhaber gearhead motors below on eBay with quick shipping and a reasonable price.

This configuration allowed me to test the actuator and limit switches for all three servo motors. I did need to address the differences in the encoder resolution between the Faulhaber gear-head motors and the Maxon motors. I added a jumper to my test board that the firmware checked at start up. If the jumper is found when the firmware boots up, then the software scales the motion target values to match the low resolution encoders on the Faulhaber's. If the jumper is not found, the firmware uses the high resolution target values for the Maxon motors.

The flexibility of laser cutting really stood out when it came to needing a way to test the homing switches and verify the motors were reaching the desired target positions during the sequence. Below is one of the actuators that I drew up and cut on the laser. The bump actuates the limit switch used for homing the motor and the scale is customized for each axis. Note the mounting hole was also laser cut to match the motor output gear tooth profile with a friction fit. 

For example, in the image above the highest tick mark is much greater than 360 degrees. This is to allow me to simulate an actuator that requires multiple  motor rotations while using less than one rotation on my physical model. Otherwise I need a way to keep track of which actuation is the correct actuation of the home switch. The hand-held demonstration model actually includes two limit switches, one triggered every rotation at the same point and a second that closes opens after the first revolution (the two switches were wired in an AND configuration to create a single home position indicator).

The Physical Model was indespeinvible for functional validation of the firmware during development, allowing me to meet the aggressive development schedule. Additionally, when the client requested operational changes, the Physical Model provided a way for me to validate firmware modifications at my workbench before sending updates to the client. 

I hope this has been informative and will help you develop a physical model to support your firmware project. 

LEGO EV3 Minimal Gyro Proportional Drive Straight for FLL Robots

How to Help Your Robot Drive Straight Using the Gyro Sensor

Wait, It Drifts Left AND Right? 
During my second season coaching FLL, the team had a robot that would sometimes drift to the left, but other times drift to the right, sometimes during the same run! This drove them crazy most of the season. Worst of all, when they put a mat on a hard floor to simulate a perfectly flat board, the drifting all but disappeared! 

We finally figured out that the northside of our regular practice board was slightly higher than the south side.  The robot had much of its weight on the back which made the back of the robot want to slide downhill while the robot was driving. This made the robot tend to drive uphill! So driving west to east the robot would drift left, but driving east to west the robot would drift right (always uphill). 

Driving straight can be one of the most challenging tasks for any robot, including those in FIRST LEGO League. The issue is that robots do not naturally drive straight. Actually very few things naturally move in a straight line (even people, see MythBusters Season 9 Episode 15: Walk a Straight Line). The article below discusses some of the reasons driving straight is such a challenge for robots.

http://www.legoengineering.com/why-doesnt-my-robot-drive-straight/

When properly applied, a Gyro Sensor can help your drive robot straight. I am going to walk you through the simplest way I know to use a Gyro sensor, on an EV3 in this example. We are going to start simple, and explain in more detail as we progress. 

LEGO EV3 Gyro

The program below uses the Gyro Sensor to help the robot driving straight. We want the robot to drive straight in the direction, or heading, of 0 degrees at 50 power for a total of 4 rotations. We use 0 degrees because that is the initial reading from the Gyro Sensor anytime you run a new program.

One of the first things to notice is that we are using the Gyro Sensor in Angle mode. When the program starts the Gyro Sensor resets to 0 so we are using 0 degree as our heading ("a" in the red Math Block).

If the robot drifts left, the Gyro Sensor will report how far the robot has turned left in negative degrees. For example, if the robot is pointing 5 degrees left of where it was pointing when the program started, the Gyro Sensor will read -5. And if the robot drifts to the right the Gyro Sensor will report how far it has turned to the right in positive degrees. So if the robot is pointing 10 degrees to the right of where it was pointing when the program started, the Gyro Sensor will read  10. 

The program works by using a LOOP to read Gyro Sensor on port 2, then calculate the heading error by subtracting the gyro reading from our desired heading, 0 degrees, and using the heading error as the Steering Angle for the Move Steering block (the Motor Power is fixed at 60). This happens as fast as the Loop can run (many times per second). This continues until Motor B rotations are greater than or equal to 4, or for four rotations.

This is a Proportional Control or Proportional Feedback system. It is called this because you are changing the output, the Steering Angle, by an amount that is proportional to the error, or the difference between the desired heading (0 degrees) and the measured direction.

A control system work to keep the system error as close to zero a possible. In this case it just means that the EV3 blocks above will keep adjusting the move block's steering angle to keep the robot's gyro sensor, and the robot, pointing at a heading of zero degrees. 

Understanding Proportional Feedback to Help Your Robot Drive Straight

The Good Stuff - Proportional Feedback and Proportional Control
We are going to walk through how to use the Gyro Sensor and Proportional Feedback to make your EV3 based LEGO robot drive straight. The robot will drive straight even if the wheels slip or if the wheels are even different sizes. I am still impressed every time I push the back of the robot to make it turn off course and it turns back to follow the desired heading.

Gyro Sensor
We are going to assume that your robot's Gyro Sensor is mounted with the arrows facing up (ensuring that rotating the robot to the left will produce negative readings from the sensor) and somewhere near the centerline of the robot (ensuring that turns to the left and right each require similar adjustments to the robot's drive system.

The Gyro Sensor does have a few unique traits. First, it calibrates while the EV3 brick is booting up. This means your robot needs to be very still while booting. I recommend putting the robot on a clean piece of paper on the floor for booting up.

You can force the Gyro Sensor to recalibrate by plugging and unplugging the sensor's cable or using a specific combination of programming blocks. I will not cover software methods for resting the Gyro Sensor because it very often causes more problems than it solves.

Move Steering
The Move Steering block makes our task simpler by allowing us to change the robot's direction by adjusting the Steering Angle input.

Proportional Control simply means that you are making a change, this case steering angle, based on the size of the measured error. Here our error is the difference between the direction we want to drive and the angle the gyro says we are driving. output to a value that is proportional to the system error. In the case of the Gyro Sensor, any errors to the right, or clockwise, are read as positive angles and error to the left are negative angles. And we are going to use a Move Steering block and adjust the Steering Angle based on the Gyro Sensor reading to make the robot drive straight. So if the robot is going a little left of where we want, say -5 degrees, then we want to set the Steering Angle to steer to the right, perhaps 5 degrees.

Below is a Minimal Gyro Drive Straight program. I think it is very cool that you can build it with only five blocks, including the Start block!

Going Straight
Going in a straight line is not a natural thing for a robot, or for humans for that matter (see MythBusters episode 173: Walk a Straight Line). It turns out that humans and robots both need some kind of feedback to go in a straight line. For humans, the feedback is usually visual but you can also use a compass as a reference to keep you pointed in the same direction. Robots are similar in that you can use a Color Sensor to follow a line or use the Gyro Sensor to follow a heading.

Why Straight is Difficult for Robots
Driving straight using only mechanical design is challenging. The robot needs to have wheels of the same diameter, precisely aligned to each other and in line with the robot chassis, the motors must rotate the wheels at the same speed (be synchronized), the weight distribution must be balanced between the wheel to ensure the robot has a center of gravity centered between the wheels and the wheels cannot slip. 

Rarely if ever are all of these conditions true, so most robots, even good designs, do not actually drive straight. For example, a ball bearing castor will often provide changing forces during a straight drive and cause small changes in direction. Some of the driving straight issues can be difficult to diagnose. This is one reason that robust FLL robot strategies always include aligning on mission models, or lines or walls at multiple points during a run. Alignments and referencing are a top of their own that I hope to touch on in the future.

Potential Gyro Pitfalls and How to Avoid Them

Gyro sensors are very useful sensor but can be a little confusing, and has some pitfalls that have tripped up even advanced teams.

First, Gyro sensors usually calibrate automatically at power on, like when the EV3 boots up, which means any movement of the robot, or table, while the EV3 brick powers up can produce drifting in the gyro. I recommend setting the robot on the floor on a sheet of paper (to keep dirt off of the wheels) and let it boot up without being disturbed. 

Second, poor wire management can lead to connection glitches that cause the Gyro Sensor to restart and recalibrate. The robot may drive in sharp arcs or even spin in place follow a gyro reset due to a poor connection while the robot was being handled by the team. This can be a very tough problem to identify because the problem happens when the wires are bumped but the robot driving failure doesn't show up until you go to use the gyro later. So the robot can be great on a practice table but suffer a gyro reset while walking to the competition tables or even during a match.

I recommend NOT using a program to force the gyro to recalibrate. I have worked with many teams who had a great plan for how and when they would calibrate the gyro. But most, if not all, eventually had the gyro fail them during a competition run. 

I do recommend that early in the season adding some programming that will alert you if the gyro reading is changing (my team liked programming the robot to beep on every degree change from the change when stopped in the Home area) when the robot should not be moving. This will let you find out if and when your robot's gyro starts drifting. This can warn you when someone didn't follow the EV3 startup procedure, or alert you when cable movement caused a gyro reset (another reason to key your wiring neat).

I hope this has been helpful and encourages you to look into trying out a gyro sensor on your robot! 

The Wharf Rats and the Hail Mary Mods at the Houston 2019 FIRST LEGO League Wold Championship

The Wharf Rats qualified for the FIRST LEGO League World Championship in Houston when they received the Grand Champion award at the North Carolina's FLL  State Championship - Track B for the FLL Into Orbit Season.

I want to share a little of the real world engineering experience the team had in Houston. During practice rounds they saw new features (missions) break in new ways under new test conditions and had to respond to the new data. Fix a bug or delete the feature so you can ship the product? Their "ship date" was inflexible. So they learned that sometimes you have to delete features to move a project forward.

Their robot had a rough day during Wednesday's Practice Rounds (scored runs that do not count toward competition scores). Some fixes appeared to work but then they saw new failure modes in missions later in the run. The more they tried to fix it, the worse it seemed to get. The robot was score was just over 100 where it scored 192 at the state championship, and closer to 300 in practice sessions prior to Worlds. They were discouraged and stopped working on the robot before the pits closed at 4:30 pm. Instead they went to the party in the park, listened to the music, played games, danced and just had a good time.

When the party was winding down, I shared that it was a common programming practice to eliminated unstable features from a product to allow you to hit your ship date. So we talked about looking for a key failure point that might have the most impact on their total score. They picked Run 4's Strength Exercise mission whose failure impacted the most missions down the line. They agreed it would be better to eliminate that mission, and it's points, to improve the robot's reliability for missions that follow. But that would require modifying the robot, changing mission fixtures and updating the firmware (thing they worked on for months). Their robot and fixtures are locked up in the pit until 7 am the next morning and their first competition run is at 8:40 am. Time for a Hail Mary! No fear! 

This is engineering! A bug is found in late testing. Your project ships tomorrow at 8:40 am. You can start working on it at 7 am. Now fix it!

Here is the team at 7:33 am Thursday morning. Everyone is tired but ready to go. They are working on their robot with less than an hour before they need to be in the queue for their 8:40 am run. You can see the venue is almost empty (the practice boards do not open until 8 am).

They modified the robot's attachment to remove the mission being deleted, update software and signed up for an 8 am  practice table. They had time for one 10 minute slot to test the mechanical changes and dial-in the programming changes. After multiple corrections, the robot turned the wrong direction then started driving the wrong way, they finally got Run 4 to work ONCE before they had to run to the pit to drop-off the laptop and get in the queue by 8:30 am for their first run! Their 8:40 am run scored 250 which held on to give them 13th place in robot performance! Video linked below.

Wharf Rats Run 1 at Houston World Championship

My favorite part of the run is the 180 degree turn at 1:51. The turn was needed because the robot had been aligning on the wall using the Strength Training fixture on the back of the robot that was deleted. The fixture that was removed left the back of the robot uneven, so the robot had to spin and do the wall alignment using the front of the robot. That also meant that the first mission after the wall alignment, 3D Printer, had to be updated to turn the opposite direction since the robot was now facing the wall rather than backing into the wall for the alignment. (I also like how at 0:39 seconds that the robot does a 180 degree turn when a fixture is removed to point in the correct direction for the next fixture)

And for a while, they were at the top of the robot game leader board.

They also had a great research project (a system to reduce the deterioration of the human balance system during extended exposure to micro-gravity) that we hope to continue to advance. I could not be more proud of what they accomplished this season!

If you are curious, below is their YouTube video showing their planned robot game strategy a week before worlds. They did get that first cart to roll down the ramp. You get a better look at the robot rotating in base at 0:44 in this video.

Wharf Rats - Into Orbit - One Week to Worlds

You can support FIRST by volunteering as a coach, mentor or assistant! Teams can always use extra help with something. Find a team (or teams!) in your area and share your skills! Email me if you want help finding a local team. And contact me if you are interested in establishing an FLL program at your school. 

https://www.firstinspires.org 

Thanks!

Shane

Coach - Wharf Rats - FLL #5027

Instructor - Summer Robotics Camp