Adequate testing means more than just checking against the specification
Scott Rosenthal
July, 1995
Testing an embedded system involves many more issues than the testing of a PC program.
Not only does the embedded system contain a program analogous to the PC, it must also
contend with startup code, hardware-control issues and asynchronous power interruptions.
Hence, embedded-system testing requires not only testing to a requirements document, but
it also demands of the tester an innate system knowledge, skepticism-and a healthy dose of
luck. No matter how detailed the document, my experience is that this process finds few
system problems. If you think about it, the reason for this truth should be obvious.
Consider that any designer worth his salt develops systems to address the design's
requirements. Who ever set out purposely to design something that didn't meet the system's
requirements?
Therefore, the most likely places defects show up are in a system's "undocumented
requirements." For example, I once designed an analyzer for use in a grain elevator.
One of these instruments came back for repair after having been in the field for a while.
Inside the instrument was a soft rubber wheel for moving grain. Apparently, this soft
rubber was also ideal for mice to use in whatever it is mice do with soft rubber. The
wheel was chewed apart, and we had to switch to a hardier wheel. The instrument met the
requirements testing, which apparently didn't take into account all environmental
conditions.
Xilinx revisited
Proving that these problems don't ever go away, in a previous column (Ref 1) I detailed a problem with a Xilinx chip powering up
into an illegal state from a brownout condition. Although none of us were smart enough to
anticipate this type of problem, any good tester will tell you, once you've identified a
weakness in the system, there's a good chance that additional problems are hiding down the
same path. With that lead in, let me describe a really neat tool we used with Xilinx
debugging, fixing and testing.
First, a quick review of the problem: In a brownout condition (caused by quickly
switching the instrument Off then On), the Xilinx chip wouldn't reset properly unless one
of two conditions was met: either Vcc has to drop below ~0.1V or a specific timing
sequence had to occur. The tough problem was how to devise a test that would guarantee
that the Xilinx problem was really fixed. That's how I came across a simple tool to verify
my fix.
A recent article (Ref 2) describes this tool, called Poc-it from MicroTools Inc
(Simsbury, CT (800) 651-6170). It switches AC power On and Off either directly or through
a relay. In addition, to close the loop, the instrument can also count the occurrences of
two TTL-level inputs representing good and bad results. What you end up with is a simple
tool (aren't the best ones simple?) that inexpensively allows you to design a true test of
the Xilinx problem.
I connected four UUTs to the instrument's switched AC outlet and set it to quickly
cycle the power- 10 msec On and 25 msec Off-and started it on its torture testing. Within
100 cycles, all four instruments had their Xilinx chips fail to reset properly! Great, I
now had a definitive test of the problem. I then installed a patch board with a PIC
processor for controlling the Xilinx reset and reran the procedure. I stopped it after
20,000 cycles and had no failures! The Poc-it allowed me to run a controlled test, which
not only proved that it could detect failures, but it proved the fix.
What's next?
After seeing the Poc-it's success, I thought of other ways to automate our testing. I
put together an extensible automated tester that allowed me to design tests around system
requirements while adding tests to challenge the system in unexpected ways. The tester
also gives me traceability, which is important in today's regulatory environment.
The automated tester, like Poc-it, is simple in concept and implementation. When
designing a test system for an embedded device, obviously the first thing to do is figure
out how it can stimulate the UUT. One of the more-obvious places is through its keypad.
For generating simulated user inputs, three basic options are available: implement
solenoids to physically press the keys, use a serial port to simulate keypresses or wire
relays in parallel with the keys on the keypad. I went with the relays because this option
requires no changes to the UUT software, and wiring the relays is relatively easy on most
embedded systems. With the relay in parallel with a switch, closing the relay stimulates
the hardware in the same fashion as pressing a key.
The UUT for which I originally designed the test system sports a 16-position keypad. It
also has four other digital inputs, including a Door Open switch and a Lamp On signal. To
stimulate this device, I purchased a CIO-ERB24 relay card from ComputerBoards Inc
(Mansfield, MA (508) 261-1123), which provides 24 Form C relays. I controlled the board
with an old laptop computer (no hard drive, single floppy) using CBI's PPIO-DIO24H box
connected to the parallel printer port. I then wired the relays in a matrix arrangement
that matched the UUT's keypad and attached a passthrough connector on the end of the
cable. With it I can remove the keypad cable in the UUT, plug the relay cable into the UUT
and reattach the keypad cable to the relay cable. This arrangement gives me the option to
press the keys with my fingers or simulate pressing the keys with a relay. I then
connected the remaining relays to the other digital inputs to simulate their functions.
Now, with just a bit of software, I had the beginnings of a powerful test system.
When putting together a home-brew system, basically you have two choices: First, assume
you'll never need this setup again and not design an extensible system. We've all been
guilty of this crime-justifying it with complaints that there's no time or budget to do
anything else. Whatever the excuses, the result is a monolithic program that's fragile and
difficult to change.
The alternative is to take a few hours and craft an extensible program that you can use
for a long time. For my test application, I started with one of my monitor programs (see
Refs 3 and 4) and added a new
command to activate a script processor. To control the relays and specify test sequences,
I designed a simple script language. It's not fancy, but it does allow an easy path to
develop new test scripts. Also, because I wrote the code, I can add new elements to the
"language" any time. The program design is small and written in C for easy
future extensibility.
The language concept is simple. I created an object type (see Listing 1) with various
properties, including the instance name, the script string code and the relay to use.
Listing 1-Testing object
typedef struct {
char *Name; /* name of this data */
char *Code; /* character code for test script */
int Default; /* the default value on program start */
int RelayPort; /* the relay port to use */
unsigned int Mask; /* the mask to use for the I/O port */
int (*Function)(void *ObjectPtr); /* the function to execute */
char *StatusMsg[2]; /* status message for 0 and 1 */
/* the program fills the following: */
int CurrentValue; /* the current value of this data */
long Value; /* the value of the object */
} TObject;
/* functions for relay and time delay control, and keypresses */
int RelayFunction(TObject *ObjectPtr);
int TimeFunction(TObject *ObjectPtr);
int KeyFunction(TObject *ObjectPtr);
/* always present, the delay object */
TObject Wait = {"Time Delay", "T", NO, EOF, 0x08, (void*)TimeFunction, "", ""};
/* F1 key */
TObject Key1 = {"Key 1", "k1", NO, PORTC, 0x20, (void*)KeyFunction, "", ""};
I then made instances of the object for each function I wanted to control, for example
Door, Key1 and Wait. The script language itself is also simple. It consists of a command
and a parameter. The command is just the code for an object instance. Again, it could be
Door, Key1 or Wait. The parameter is some condition for the command. For example, with
Door, 0 could mean open the door and 1 could mean close the door. For Wait, the parameter
is the delay time in milliseconds. For Key1, the parameter indicates how long to pause
after pressing the key. Another idea might be to indicate how long to hold down the key.
Now, it's easy to string these object instances together to create a test script (see
Listing 2).
Listing 2-Typical test script
; all test after a semi-colon are
; comments
k1,0 ; press key 1
k2,1000 ; press key 2 and wait 1
; second
lamp,1 ; turn on the lamp
t,10000 ; wait 10 seconds
k3,0 ; press key 3
; this sequence will now repeat
; indefinitely
The script processor works by reading the script file one line at a time and processing
it. At the end of the list, it starts over. In the future, I plan to add features
including digital inputs, feedback and script linking. Having written the code myself and
keeping it object-oriented, I can easily add new objects as needed. Hence, you can use the
same program fundamentals for testing different systems.
Other benefits
In addition to the obvious advantages of automated testing, this effort also offers
other benefits. For example, it uncovered a number of subtle defects in my compiler, but
the one that might strike many developers is a caution about using a heap with malloc().
The function malloc() allows a program to grab space on the heap. Conversely, the function
free() returns this space to the heap. The problem I had was that after calling free(),
the next malloc() started looking for available heap space at the last location freed with
free() instead of at the beginning of the heap as I would've thought. Hence, depending
upon the sequence of key presses, the program would at times severely fragment the heap,
which in combination with the free() idiosyncrasy gradually stepped the pointer to the top
of the heap higher in memory until it ran out of room. The result was that the UUT ran out
of memory even though there was plenty of unused memory available. Once I figured out what
was happening it became clear that the only way to fix the problem was to rework the
operation of the malloc() function, a process that would be difficult at best-even if I
could sweet talk my compiler vendor out of the source code for the built-in functions.
Finally I decided that since the only way I found this problem was through a torture
test, the error was an acceptable situation. However, regardless of the outcome, I'm still
ahead because I now know about a potentially dangerous "quirk" in my compiler,
and I have a system to automate the testing of my products. PE&IN
References
1. Rosenthal, S, "It sometimes takes a
microcontroller to boot up an FPGA," PE&IN, Nov 1994, pgs 75-78.
2. Japenga, B, "If it's not tested it doesn't work!" PE&IN, Apr
1995, pgs 53-56.
3. Rosenthal, S, "For digging out
hardware bugs, monitor programs fill the bill," PE&IN, July 1994, pgs
56-58.
4. Rosenthal, S, "Monitors are valuable,
easy to use and create," PE&IN, Sept 1994, pgs 73-78.
Adapted from an article that appeared in Personal Engineering & Instrumentation
News.
Return to the article index.
|
