How to do your own experiments in psychology and education
4.4 Decide how to test your factors
It’s very important to build your experimental design with a particular statistical analysis technique in mind. The experimental design is a plan for how you will do your statistical analyses after the data has been collected. When you think this way, you plan ahead so that you collect the right data to fit the statistical analyses that you will do.
Given that you’re going to use ANOVA, you have to know which factors are in the design and how many levels of each. You already specified this information in the exercise above. The next step is to decide how to test your factors. This section describes two ways of testing your factors: with participants in only one experimental condition or with participants in more than one condition.
Decide, for each factor, whether you will test it between subjects or within subjects. The information below can help you choose.
Between-subjects testing
Testing a factor “between subjects” means that you will use different participants for each level of the factor. In other words, participants will only be in one experimental condition.
Some things simply require researchers to test them between subjects, like Gender and IQ. People don’t have two different values for them at the time of a given experiment so they can’t be tested twice for that factor. Of course, someone might have a sex-change operation or get very much smarter, but then you’re talking about the same participants at very different points in time and this is another kind of (“longitudinal”) study that won’t be considered here.
One of the advantages of between-subjects testing is the fact that the statistical models are simpler and some people find them conceptually easier to understand.
One of the disadvantages of between-subjects testing is the fact that studying something as a between-subjects factor requires more participants and introduces more random variation (or error variance) in the data and so reduces the statistical power of the analysis.
Researchers classify a factor as a between-subjects factor if they test it in a way where individual participants only provide data for one of the levels of that factor. That is, the participants only appear in one of the experimental conditions for that factor. For example, one group of individuals appears in the young group for the factor Age, and a different group of individuals appears in the old group.
Example. In the example below, the researcher had to use different people in each condition. None of the participants can be tested as a male and then as a female or first as a low-IQ participant, and then as a low-IQ participant. So, the participants do not appear in more than one condition. Researchers call IQ and Gender Between-subjects Factors because the different levels of a factor are split between different participants: some participants are in one level, some in another.
Each individual participates in only one experimenal condition in this design. There are a totoal of 60 participants for four conditions in this example. There are four groups of participants in this experiment: different people in each experimental condition.
Within-subjects testing
Testing a factor within subjects means that you will use the same participants for each level of the factor. If participants can experience multiple values for a factor in the course of a single experiment, then this factor is a candidate for within-subjects testing. So, for example the same participant might be trained or tested in two different ways, or may use two different materials.
Among the advantages of within-subjects testing is that the participants serve as their own control group – instead of having similar, comparable participants in two experimental conditions, you have the same individuals in both conditions. This helps control for possible effects of individual differences and gives the analysis more statistical power. Also, the number of participants needed for a within-subjects test is quite a bit smaller than the number needed for a between-subjects test. In the end, this means that you can detect much smaller differences or more subtle effects when you study something within-subjects. It’s important to emphasize that within-subjects testing is much more effective for most experimental situations, when it’s possible to use it.
Among the disadvantages of within-subjects testing are the fact that it requires more complex statistical models (called repeated-measures analysis) and the fact that some within-subjects testing will require the researcher to introduce an additional factor into the design, for additional experimental control. This additional factor is called a nuisance variable because it’s not the researcher’s main focus of study (so it’s a real nuisance!).
For example, if Text Type is studied as a within-subjects factor (or is “tested within subjects”), then each participant will read two texts, say a narrative and a dialogue. The nuisance variable would be Order of Presentation: participants have to read them in one order or the other and that may affect the outcomes. So, you add the nuisance variable to control whether reading one type of text facilitated reading the other, even though you may not be particularly interested in this question.
Researchers classify a factor as a within-subjects factor if they test it in a way where individual participants provide data for all of the levels of that factor. For example, all of the individuals in the narrative group (for the factor Text Type) also appear in the dialogue group, i.e., the same participants appear for all the levels of the factor. The same participants will provide data in two or more different experimental conditions.
Example. In the next design, each participant will read both a narrative and a procedural text. Some participants will read the narrative first (the N-P order), the others will read the procedure first (the P-N order). So, the participants are tested in two conditions each and there are two sets of measures for each participant. Order (of presentation) is another between-subjects factor: the levels are split between different participants. Text Type is a different kind of factor because you will observe both of the levels within the same participant: it’s called a Within-subjects Factor.
Notice the change in the table: the same 15 participants in N-P order read a narrative then a procedure. A different group of 15 participants in N-P order read a procedure then a narrative. Each individual participates in two different experimental conditions in this design. There are a total of 30 participants in four experimental conditions in this example. The participants are in two groups: the group for N-P order and the group for P-N order.
Read this topic next: Check your design for internal validity
Given that you’re going to use ANOVA, you have to know which factors are in the design and how many levels of each. You already specified this information in the exercise above. The next step is to decide how to test your factors. This section describes two ways of testing your factors: with participants in only one experimental condition or with participants in more than one condition.
Decide, for each factor, whether you will test it between subjects or within subjects. The information below can help you choose.
Between-subjects testing
Testing a factor “between subjects” means that you will use different participants for each level of the factor. In other words, participants will only be in one experimental condition.
Some things simply require researchers to test them between subjects, like Gender and IQ. People don’t have two different values for them at the time of a given experiment so they can’t be tested twice for that factor. Of course, someone might have a sex-change operation or get very much smarter, but then you’re talking about the same participants at very different points in time and this is another kind of (“longitudinal”) study that won’t be considered here.
One of the advantages of between-subjects testing is the fact that the statistical models are simpler and some people find them conceptually easier to understand.
One of the disadvantages of between-subjects testing is the fact that studying something as a between-subjects factor requires more participants and introduces more random variation (or error variance) in the data and so reduces the statistical power of the analysis.
Researchers classify a factor as a between-subjects factor if they test it in a way where individual participants only provide data for one of the levels of that factor. That is, the participants only appear in one of the experimental conditions for that factor. For example, one group of individuals appears in the young group for the factor Age, and a different group of individuals appears in the old group.
Example. In the example below, the researcher had to use different people in each condition. None of the participants can be tested as a male and then as a female or first as a low-IQ participant, and then as a low-IQ participant. So, the participants do not appear in more than one condition. Researchers call IQ and Gender Between-subjects Factors because the different levels of a factor are split between different participants: some participants are in one level, some in another.
| IQ | |||||
| N = 60 | Low | High | |||
| Gender | Male | n = 15 | n = 15 | ||
| Female | n = 15 | n = 15 | |||
Each individual participates in only one experimenal condition in this design. There are a totoal of 60 participants for four conditions in this example. There are four groups of participants in this experiment: different people in each experimental condition.
Within-subjects testing
Testing a factor within subjects means that you will use the same participants for each level of the factor. If participants can experience multiple values for a factor in the course of a single experiment, then this factor is a candidate for within-subjects testing. So, for example the same participant might be trained or tested in two different ways, or may use two different materials.
Among the advantages of within-subjects testing is that the participants serve as their own control group – instead of having similar, comparable participants in two experimental conditions, you have the same individuals in both conditions. This helps control for possible effects of individual differences and gives the analysis more statistical power. Also, the number of participants needed for a within-subjects test is quite a bit smaller than the number needed for a between-subjects test. In the end, this means that you can detect much smaller differences or more subtle effects when you study something within-subjects. It’s important to emphasize that within-subjects testing is much more effective for most experimental situations, when it’s possible to use it.
Among the disadvantages of within-subjects testing are the fact that it requires more complex statistical models (called repeated-measures analysis) and the fact that some within-subjects testing will require the researcher to introduce an additional factor into the design, for additional experimental control. This additional factor is called a nuisance variable because it’s not the researcher’s main focus of study (so it’s a real nuisance!).
For example, if Text Type is studied as a within-subjects factor (or is “tested within subjects”), then each participant will read two texts, say a narrative and a dialogue. The nuisance variable would be Order of Presentation: participants have to read them in one order or the other and that may affect the outcomes. So, you add the nuisance variable to control whether reading one type of text facilitated reading the other, even though you may not be particularly interested in this question.
Researchers classify a factor as a within-subjects factor if they test it in a way where individual participants provide data for all of the levels of that factor. For example, all of the individuals in the narrative group (for the factor Text Type) also appear in the dialogue group, i.e., the same participants appear for all the levels of the factor. The same participants will provide data in two or more different experimental conditions.
Example. In the next design, each participant will read both a narrative and a procedural text. Some participants will read the narrative first (the N-P order), the others will read the procedure first (the P-N order). So, the participants are tested in two conditions each and there are two sets of measures for each participant. Order (of presentation) is another between-subjects factor: the levels are split between different participants. Text Type is a different kind of factor because you will observe both of the levels within the same participant: it’s called a Within-subjects Factor.
| Text Type | |||||
| N = 30 | Story | Manual | |||
| Order | Story-Manual | n = 15 | |||
| Manual-Story | n = 15 | ||||
Notice the change in the table: the same 15 participants in N-P order read a narrative then a procedure. A different group of 15 participants in N-P order read a procedure then a narrative. Each individual participates in two different experimental conditions in this design. There are a total of 30 participants in four experimental conditions in this example. The participants are in two groups: the group for N-P order and the group for P-N order.
Read this topic next: Check your design for internal validity
About
© 2009 Created by Mike Dillinger, PhD on Ning. Create a Ning Network!
