NL-V71-03

By Dr. David Tozer, Ph.D., ASQ CQE and SSBB, Education & Audit Chair.

Over the years I have seen many presentations where people collect data to perform evaluations and then try to draw conclusions from the data.  In many cases it is difficult to draw conclusions from the data collected.  A common reason for this difficulty is the data were collected by an experiment or evaluation that did not use designed experiment methodology to guide how to collect data.

For almost 100 years, we have been teaching Design of Experiments (DOE) to students.  These methods are more efficient and effective, from an economic perspective, than other methods.  In some industries, designed experiments are used regularly.  Examples include agriculture, chemical and pharmaceutical safety and efficacy testing (pre-clinical and clinical trials).  In other parts of industry, designed experiments are uncommon.  Many of us are involved in doing experiments or evaluations.  I think it would be useful to use a scientific method to perform experiments or evaluations.

Scientific work is based on having standards.  I am not referring to ISO standards, but the physical standards that are the basis for commerce, physics, chemistry, biology, engineering and medicine.  Standards reduce bias and allow us to use a common language to describe the world.  These physical standards include the degree (temperature), ampere, kilogram, metre, second and mole.  In the case of biological and social systems, we often do not have well-defined physical standards when performing experiments or evaluations.  In these circumstances we designate a sample from the population as a control to serve as the standard.  Controls receive a reference treatment and can include placebos (sugar pill), untreated subjects or a current treatment.

Designed experiments can be used in almost all industries and get useful results. The simplest experiment that could be used by many organizations is described in the following example. 

An activity that is done in almost all industries is training.  Another thing that is common to many industries: the money spent on training does not seem to yield the expected results in increased productivity or effectiveness. 

So, to assess training effectiveness the following process could be used:

1) Before training begins:

• Training methods are developed and documented;
• Important performance metrics are identified;
• People being sent on training are evaluated to access current performance, the control, and data collected on the current performance. 

2) The people are then trained in the required skill using the developed methods. 

3) After training is completed, the trained people are evaluated for their performance of the required skill, the treatment effect. 

4) The difference in performance between the control and the treatment is assessed to see if there is a training effect.

In more technical terms, this set up is a single factor (training) repeated measures (repeated on the same people) two level experiment (control and treatment). 

The analysis of the results requires the use of the first statistical test discovered in the early 1900s.  It too is the simplest possible statistical test.

We also need to make sure the environment and selection of trainees is done in as uniform a manner as possible.  It is important to ensure the environment, in which any experiment or evaluation is done, is understood and documented.  All results are conditional on the environment the data were collected in.  In the case of the training example the results are conditional on the training methods.

An important take away from this short discussion is the idea of a control as a reference standard.  It is not the same as a physical standard, but it is a standard nonetheless.  Standards in the form of controls should form the basis for evaluations and experiments in many business situations.

As mentioned earlier, the example demonstrates the simplest designed experiment possible.  The real world is a lot more complicated.  For more complicated systems, more complicated designs need to be used.  Many economical methods have been developed over the years to handle complicated situations.  The methods can be used for evaluation, screening and optimization.  Some designs look for relative changes and may not, at first glance, appear to have a standard or control.  It is always a useful exercise to determined what the actual control is when doing an experiment or evaluation.

By performing designed experiments, data collected during the evaluation are turned into information about the effectiveness of an intervention.  By using information, we can make better informed decisions.