How to do your own experiments in psychology and education
5.5 Review your Methods
The experimental design specifies more details about how you will investigate your factors and try to maximize internal validity. Your dependent variables are a way to measure what’s happening (or not happening) with your process and specifying them carefully helps you maximize construct validity.
Reality Check #2: Review how you will measure what happens to your sub-process. Use the information below to determine how much you risk making a Type II statistical error. Make any changes to your methods that you can to reduce your risk.
Don’t miss genuine effects
Another one of the most important goals that researchers keep in mind is this.
One goal of having systematic methods is to avoid missing any genuine effects that may result from your experimental manipulation of the “contrast” – this is called “avoiding Type II statistical error”. This kind of error doesn’t mean that there was a mis-application of statistical techniques; it means that the researchers didn’t design and/or execute the experiment with enough safeguards like those described below.
The three most common ways of making a Type II statistical error are:
Working backwards from the statistical techniques that we will use for analysis, we know that the following steps help to avoid these problems with Type II error and help to optimize experimental results:
Goal 1. Minimize the influence of variation between individual participants. The variability in the responses between participants is error variance and higher error variance makes it harder to detect the effects of your factors. There are several ways to minimize individual variation and error variance:
• Maximize the homogeneity of your participants (within each group of participants);
• Maximize the randomness of sampling (of stimuli and of participants);
• Maximize the number of participants;
Goal 2. Maximize differences in the levels of each Factor, i.e., maximize the size of the contrast. Small or subtle differences between levels of a factor are very hard to detect reliably. People vary widely, so systematic differences between 18-year-olds and 19-year-olds will be hard to detect. So, if 18-year-olds and 19-year-olds are two levels of the factor Age, then the researchers will probably not detect any systematic differences. A larger contrast, such as 15-year-olds vs. 20-year-olds will work better.
Goal 3. Maximize the sensitivity and reliability of your dependent measures.
If two or more people are analyzing parts of the same data, then they will analyze it slightly differently so the dependent measures will be less reliable, especially if there is some subjectivity in the analysis. This introduces more error variance into the measurement process and makes it harder to detect systematic effects of the factors.
Similarly, if your participants can answer only yes or no to some of your questions, then your measures won’t be sensitive to the needs of participants who want to answer maybe. This will add some error variance because the maybes may answer yes to some questions and no to others.
There are several ways to optimize your measurement process.
• Maximize the simplicity and the clarity of instructions to the participants;
• Maximize the standardization of your collection and analysis procedures;
• Maximize reliability of data coding procedures;
Reality Check #2: Review how you will measure what happens to your sub-process. Use the information below to determine how much you risk making a Type II statistical error. Make any changes to your methods that you can to reduce your risk.
Don’t miss genuine effects
Another one of the most important goals that researchers keep in mind is this.
| Minimize Type II statistical error. Make sure that you don’t miss any real differences in the sub-process that you’re studying. |
One goal of having systematic methods is to avoid missing any genuine effects that may result from your experimental manipulation of the “contrast” – this is called “avoiding Type II statistical error”. This kind of error doesn’t mean that there was a mis-application of statistical techniques; it means that the researchers didn’t design and/or execute the experiment with enough safeguards like those described below.
The three most common ways of making a Type II statistical error are:
- there were not enough participants;
- the contrast was too subtle;
- the dependent measures were not precise or reliable enough;
Working backwards from the statistical techniques that we will use for analysis, we know that the following steps help to avoid these problems with Type II error and help to optimize experimental results:
Goal 1. Minimize the influence of variation between individual participants. The variability in the responses between participants is error variance and higher error variance makes it harder to detect the effects of your factors. There are several ways to minimize individual variation and error variance:
• Maximize the homogeneity of your participants (within each group of participants);
• Maximize the randomness of sampling (of stimuli and of participants);
• Maximize the number of participants;
Goal 2. Maximize differences in the levels of each Factor, i.e., maximize the size of the contrast. Small or subtle differences between levels of a factor are very hard to detect reliably. People vary widely, so systematic differences between 18-year-olds and 19-year-olds will be hard to detect. So, if 18-year-olds and 19-year-olds are two levels of the factor Age, then the researchers will probably not detect any systematic differences. A larger contrast, such as 15-year-olds vs. 20-year-olds will work better.
Goal 3. Maximize the sensitivity and reliability of your dependent measures.
If two or more people are analyzing parts of the same data, then they will analyze it slightly differently so the dependent measures will be less reliable, especially if there is some subjectivity in the analysis. This introduces more error variance into the measurement process and makes it harder to detect systematic effects of the factors.
Similarly, if your participants can answer only yes or no to some of your questions, then your measures won’t be sensitive to the needs of participants who want to answer maybe. This will add some error variance because the maybes may answer yes to some questions and no to others.
There are several ways to optimize your measurement process.
• Maximize the simplicity and the clarity of instructions to the participants;
• Maximize the standardization of your collection and analysis procedures;
• Maximize reliability of data coding procedures;
| Read this topic next: | Write up your Methods |
About
© 2009 Created by Mike Dillinger, PhD on Ning. Create a Ning Network!
