4.5 Check for Internal Validity

[Mike Dillinger, PhD]

Reality Check #1: Review your experimental design. Use the information below to answer these questions.

• a. How confident are you that only your factors will cause the differences in your sub-process that you observe from the participants’ reponses?
• b. What other factors might possibly cause differences in your sub-process?
• c. How will you control the effects of these other factors?

There are several goals that researchers keep in mind when they design an experiment and specify its methods. One of the most important of these goals is the following.

Maximize Internal Validity. Be sure that your factors are causing the differences that you hope to observe.

Is your contrast (or difference between the levels of the factor [~ independent variable]), rather than some other extraneous factor, causing the changes that you observe? (See Mitchell & Jolley, 2005, Ch. 8) The more evidence there is that the difference in your factors is what is actually causing any differences that show up, the greater is the internal validity of your experiment.

The world is very complex so a very wide range of factors can affect the outcomes that you observe in any experimental setting. Researchers have to have a very clear idea of as many potentially important factors as possible. That allows them to isolate or control the factors that they are not interested in and focus on the ones that they want to study in a given experiment. One of the most frequent questions that researchers get is “Why didn’t you study factor x?”, so thinking about the possibilities ahead of time allows you to give a clearer answer and avoid being caught by surprise.

We can think of factors in two groups: irrelevant factors and relevant factors.

Irrelevant factors. Irrelevant factors in fact do not affect the behaviors that you are studying. Some of these we know to be irrelevant from other studies. For example, Eye Color is irrelevant for reading performance. Other factors we don’t pay attention to out of ignorance, and in some cases we’re right without knowing it: i.e., there are some factors we don’t know about that really are irrelevant. Either way, it’s safe to ignore these factors. Even if they are irrelevant, experimenters have to be aware of them and sometimes need to provide evidence that they are in fact irrelevant.

Relevant factors. These are the ones researchers need to worry about, and you can think about four kinds of them: the factors that are indirectly relevant, the ones we don’t even know about, the ones we know are relevant, and the ones we suspect are relevant.

• Indirectly relevant factors are those things that affect the factors that we know are relevant. It’s risky to ignore them, but most people do. Expert researchers do their best to measure them and keep them under control. These factors show up when you do an extra careful literature review and during data analysis. One corollary of Murphy’s Law states that these indirectly relevant factors will make a mess of your results when you least expect it.
• Unknown relevant factors are those things that in fact affect the processes that we are studying, but no one knows about them yet. When we design an experiment, we have no choice but to ignore them. But when we analyze our data, we’re always on the look out for any relevant factors, especially those that no one knew about before.
• Known-to-be-relevant factors are those that in fact affect the processes that we are studying, based on evidence from other studies. These factors are very, very important: we either focus on studying them or control their effects. We ignore them at the risk of invalidating the experiment.

Your factors are the factors that you suspect can affect the processes that you are studying, even if you don’t have enough evidence to say with any certainty – which is why you are studying them.

Your factors will get built into your experiment as:

• Contrasting groups of participants (e.g., males vs. females)
• Contrasting materials (e.g., different instructions, different texts, etc.)
• Contrasting tasks (e.g., count the letter “e” vs. read for meaning)
• Contrasting settings (e.g., with or without distracting stimuli)
• Or combinations of them, with one contrast for each factor.

Maximizing internal validity means controlling the other relevant factors so that the effects of your studied factors can stand out.

Experimental control is what makes experimental research different from (and more reliable than) other kinds of research.

Experimental control. Experimental control is the key to maximizing internal validity. The main idea is to control the effects of everything that is or might be relevant for your process so that only the factors that you are studying affect the process. Experimental control is also what makes experimental research different from (and more reliable than) other kinds of research.

There are basically two methods for controlling the effects of a relevant factor that you don’t want to study:

• Make sure that there is only one value of that factor -- i.e., freeze the value;
  For example, we know that the way instructions are phrased makes a big difference in how people respond. Because we need to control for the possible effects of instructions, we need to give all of the participants the same single set of instructions. This way, it’s safe to say that the instructions did not cause any of the differences that we observe later. In other words, the factor Instructions only has one level or one value: the single set of instructions that was used.
• Make sure that the values of the factor appear at random – i.e., random selection;
  We also know that people respond differently to the same situations, simply because people are different. We can’t use only one value for the factor Participant – i.e., only one participant – because in that case we can’t say anything about how other participants would act. The main option for dealing with the fact that people are so different is to use lots of participants. We can then use special methods to sample the participants at random. Some people will have more, others will have less of each characteristic, but the randomness will tend to cancel out the effects of these differences when you look at the whole group. Random sampling is one of the most powerful techniques in experimental research.

Review this task: Design your experiment