Testing is a significant part of any product development and that is especially true for products that have the potential of harming the user at the worst. At the least traumatic, we have quality problems that cost our company in both dollars and in customer perception of our company. A profitable vehicle model can easily fail the strategic objectives of the organization if warranty costs impact revenue stream due to rework, and customer orders drop due to customer trepidation. Vehicles are complex systems, consisting of hardware and software elements that must reliably and repeatably work within a range of environments. Vehicles must work in very cold, and very hot weather, moist and dry, bouncing on roads, or on well paved roads. The manner we set about testing will have a big impact on the success. Below is an extract from our book, Testing Complex and Embedded Systems, specifically how to do bad testing. Vehicles do different things, passenger car, mining equipment, construction vehicles and over the road tractors that all will require testing, and though the details of the tests may vary significantly, the philosophy of testing will be fundamentally common. We want to know the product, before we send it out to the customer. We are aggressively testing the hypothesis that the product is good and of sound quality.
Figure 1Excerpt from Testing Complex and Embedded Systems, Chapter 8
How to perform bad tests
Bad testing can have a big impact upon the customer’s experience of the product. This adverse customer impact, has an impact upon the profit margins on the product and volumes of the product sold, not too mention the impact of possible legal action. Poor testing can be expensive. Spending many hours and much money on testing and missing product non-conformances or other problem areas is worse yet. Not only is this a waste of resources, it also provides a false sense of security for the organization.
Figure 8.1: Bad Testing
8.1 Do not
8.1.1 Add stress
Testing the product to a very low level of stimulus or minimum rigor does not help learn something from the product.
8.1.2 Go beyond the specified “limits”
Specifications are a good starting point, however, frequently these do not capture the total exposure of stimuli to which a product will be subjected. We have seen many cases, where the product ”passes” testing to specification, and the summarily fails in the field. Variation of key product design areas as well as the variation the external environment can all add up to produce a failure in the field.
It can take a great amount of time to understand the real demands upon the product when it is deployed by the customer. Gather the information requires instrumentation of the example uses of the product with sufficient sample size and variation to provide some statistical significant understanding, and takes time. Experience suggests development personnel rely heavily upon standards. These are helpful, but there is always a question as to the percentage of the customer population represented by the standard. One of us has experienced the condition where the product passes the electrical transient portion of a specification only to go on to produce failures in the field. This anomaly is not an occasional occurrence, but all too frequent—pointing to testing to specifications as a short-sighted methodology.
8.1.3 Use unusual combinations of events
The real world is more complicated and chaotic than we believe. Testing only to typical combinations is very shortsighted and unrealistic. We can use Cynefin model to show the sequence to show the sequence of simple to chaotic systems: