How to do your own experiments in psychology and education
4.3 Identify your experimental conditions
Each combination of the levels of your factors is called an experimental condition. Each experimental condition describes a situation in which you’ll collect data.
Most researchers like to make a table like the one below to think about their experimental conditions. The four grey cells in the table correspond to the four experimental conditions in the experiment. In this case, the data collection situations are described in terms of the characteristics of the participants. But in other experiments, the experimental conditions can vary in terms of the materials, the setting, the materials, the tasks, or other aspects of the factors.
Two factors: Age, two levels; Gender, two levels = 4 experimental conditions
In this example, the factors are Age and Gender and the levels are Male, Female and Young, Old.
To calculate the total number of conditions in your experiment, you just have to multiply the number of levels for each factor. For example: for Gender (male, female) and Music (presence, absence), you have 2 (two) levels of Gender and 2 (two) levels of Music. 2 x 2 = 4 (four) experimental conditions. Remember that experiments with more conditions are more difficult and time consuming to carry out, so for this course, stick to factors with only 2 or 3 levels.
Here are some examples of similar tables for other possible experiments, with different numbers of factors or levels. Again, the grey cells in the table correspond to the conditions in each experiment.
One factor: IQ, three levels 1 x 3 = 3 experimental conditions
Two factors: IQ, two levels; Text, two levels 2 x 2 = 4 experimental conditions
Three factors: IQ, two levels; Gender, two levels; Order of presentation, two levels 2 x 2 x 2 = 8 experimental conditions
In this last, more complex example, think through what is going on in each of the numbered conditions. In condition #1, for example, there will be low-IQ males who will be exposed to the stimuli in Order 1.
In these examples, ALL of the possible conditions (multiplying out the levels of all of the factors) are in fact used in the experiment. These are called fully factorial designs: all of the possible combinations of levels are included in the experiment.
Background: Other design options
Incomplete designs. In some situations, it may be the case that not all of the conditions are relevant or interesting. For example, consider an experiment with two, two-level factors: Task (listen, translate) and Expertise (expert, novice). A fully factorial design would yield four conditions:
But it makes little sense to compare “expert” and “novice” listeners: native speakers are all pretty much experts at listening. You’re interested in how listeners compare (as a baseline) to expert and novice translators. So, you’re only interested in conditions a, c, and d. To run participants in condition b would be a waste of effort. Collecting data for only some conditions of an experiment is referred to as using an incomplete design, if it’s planned that way. If it’s unplanned, it’s called “a big mistake”.
There usually has to be a strong theoretical motivation for this option because the statistical analyses are often more complex than for a complete design and lots of information is lost. This also why you need to plan data collection carefully: not collecting data for a given condition, for example, can be a very costly mistake.
Single-n designs. In other situations, the participants are so unique or the task so complex that it becomes impractical to study groups of randomly selected participants and the researcher has to focus on individuals. Of course, generalizability is limited, but these designs are often the best option available. To ensure internal validity, a common strategy is to use an A-B design in which the same subject is tested once (the A condition) without any treatment to establish baseline performance and then again (the B condition) after the experimental treatment. Treatment is seen as a within-subjects factor and the participant serves as his/her own “control group”. See Ch. 13 in Mitchell & Jolley, 2005 for more information.
Longitudinal designs. Many research problems investigate how a particular skill or process – for example, reading ability – develops over time. One strategy is to sample different participants at different stages of the development period, for example, 4th, 6th, 8th, and 10th graders. This strategy has well-known disadvantages because of the variability from group to group and the need to infer that differences between the groups are representative of individual development. Another strategy is a design in which an additional within-subjects factor (Time) is added: the same experiment is repeated with the same participants at (usually) regular intervals to chart the evolution in participants’ performance. This is called a longitudinal design. Internal validity is an issue with these designs because as time goes on, there are more possibilities for important non-experimental influences to happen.
There are many other options; these are some of the most commonly used designs.
Read this topic next: Decide how to test your factors
Most researchers like to make a table like the one below to think about their experimental conditions. The four grey cells in the table correspond to the four experimental conditions in the experiment. In this case, the data collection situations are described in terms of the characteristics of the participants. But in other experiments, the experimental conditions can vary in terms of the materials, the setting, the materials, the tasks, or other aspects of the factors.
| Age | |||||
| Young | Old | ||||
| Gender | Male | Young males | Old males | ||
| Female | Young females | Old females | |||
In this example, the factors are Age and Gender and the levels are Male, Female and Young, Old.
To calculate the total number of conditions in your experiment, you just have to multiply the number of levels for each factor. For example: for Gender (male, female) and Music (presence, absence), you have 2 (two) levels of Gender and 2 (two) levels of Music. 2 x 2 = 4 (four) experimental conditions. Remember that experiments with more conditions are more difficult and time consuming to carry out, so for this course, stick to factors with only 2 or 3 levels.
Here are some examples of similar tables for other possible experiments, with different numbers of factors or levels. Again, the grey cells in the table correspond to the conditions in each experiment.
One factor: IQ, three levels 1 x 3 = 3 experimental conditions
| IQ | Low | Medium | High |
| Low IQ | Medium IQ | High IQ |
Two factors: IQ, two levels; Text, two levels 2 x 2 = 4 experimental conditions
| IQ | |||||
| Low | High | ||||
| Text Type | Story | ||||
| Manual | |||||
Three factors: IQ, two levels; Gender, two levels; Order of presentation, two levels 2 x 2 x 2 = 8 experimental conditions
| IQ | |||||
| Order 1 | Low | High | |||
| Gender | Male | 1 | 2 | ||
| Female | 3 | 4 | |||
| IQ | |||||
| Order 2 | Low | High | |||
| Gender | Male | 5 | 6 | ||
| Female | 7 | 8 | |||
In this last, more complex example, think through what is going on in each of the numbered conditions. In condition #1, for example, there will be low-IQ males who will be exposed to the stimuli in Order 1.
In these examples, ALL of the possible conditions (multiplying out the levels of all of the factors) are in fact used in the experiment. These are called fully factorial designs: all of the possible combinations of levels are included in the experiment.
Background: Other design options
Incomplete designs. In some situations, it may be the case that not all of the conditions are relevant or interesting. For example, consider an experiment with two, two-level factors: Task (listen, translate) and Expertise (expert, novice). A fully factorial design would yield four conditions:
- a. Expert, listen
- b. Novice, listen
- c. Expert, translate
- d. Novice, translate
But it makes little sense to compare “expert” and “novice” listeners: native speakers are all pretty much experts at listening. You’re interested in how listeners compare (as a baseline) to expert and novice translators. So, you’re only interested in conditions a, c, and d. To run participants in condition b would be a waste of effort. Collecting data for only some conditions of an experiment is referred to as using an incomplete design, if it’s planned that way. If it’s unplanned, it’s called “a big mistake”.
There usually has to be a strong theoretical motivation for this option because the statistical analyses are often more complex than for a complete design and lots of information is lost. This also why you need to plan data collection carefully: not collecting data for a given condition, for example, can be a very costly mistake.
Single-n designs. In other situations, the participants are so unique or the task so complex that it becomes impractical to study groups of randomly selected participants and the researcher has to focus on individuals. Of course, generalizability is limited, but these designs are often the best option available. To ensure internal validity, a common strategy is to use an A-B design in which the same subject is tested once (the A condition) without any treatment to establish baseline performance and then again (the B condition) after the experimental treatment. Treatment is seen as a within-subjects factor and the participant serves as his/her own “control group”. See Ch. 13 in Mitchell & Jolley, 2005 for more information.
Longitudinal designs. Many research problems investigate how a particular skill or process – for example, reading ability – develops over time. One strategy is to sample different participants at different stages of the development period, for example, 4th, 6th, 8th, and 10th graders. This strategy has well-known disadvantages because of the variability from group to group and the need to infer that differences between the groups are representative of individual development. Another strategy is a design in which an additional within-subjects factor (Time) is added: the same experiment is repeated with the same participants at (usually) regular intervals to chart the evolution in participants’ performance. This is called a longitudinal design. Internal validity is an issue with these designs because as time goes on, there are more possibilities for important non-experimental influences to happen.
There are many other options; these are some of the most commonly used designs.
Read this topic next: Decide how to test your factors
About
© 2009 Created by Mike Dillinger, PhD on Ning. Create a Ning Network!
