How to do your own experiments in psychology and education
5.4 Plan your procedures
The Procedure section of a research report describes the sequence of steps that you will follow to collect and analyze your data. The Procedure specifies the tasks that the participants will carry out, the materials that they will use, the setting in which the experiment will take place, and the steps that the experimenter will take to make measurements and analyze the data.
It’s very important to plan the procedure out and think it through in great detail so that all of the participants go through the same exact steps. This is one part of experimental control: ensuring that variations in the procedure (which are not being studied) do not lead to different outcomes.
It’s worth repeating that one of the main reasons for planning out your procedure very carefully is to be sure that you don’t miss any real differences. One goal of having systematic methods is to avoid missing any genuine effects that may result from your experimental manipulation of the “contrast” – this is called “avoiding Type II statistical error”. Different ways of minimizing Type II statistical error are described in the following section. Careful planning of the procedure helps you maximize the standardization of your collection and analysis procedures, so you reduce error variance and make it easier to detect significant effects of your factors.
Sessions
You probably won’t collect all of your data at once. It’s more likely that you’ll run through the procedure until you get all of the conditions done or accumulate enough participants.
Each time you run through your procedure, you’ll have completed a data collection session. It’s important to know how long a session takes and how many sessions you’ll need so that you can double check how long you’ll need to collect all of your data. Pilot testing will help you estimate this better.
Don’t forget to include an estimate of how long it will take to distribute and collect all of the materials for each session. When you have 30 participants, this can take quite a while. You really should bring a research assistant to help with this.
Sample Script
“Hello everyone, please be seated”
“I am going to pass out a consent form for this experiment. Please read the form and sign the bottom if you agree to further participate in this experiment.”
“I am now going to pass out a questionnaire, please answer the questions as accurately as possible. When you are finished turn it over and leave it on you desk.”
“I am now going to pass out two papers to you, one containing a story and the other blank. Please keep the paper with writing face down until instructed to turn it over.”
“You will have 1 min 45 s to read the text. After the time is up, turn the paper back over and I will come and collect them. When I collect you paper I will also give you a pen you will be required to write with. You will then write on the blank sheet what you remember from the story. You may now turn over your paper and begin reading.”
“Stop. Please turn you paper face down and I will come collect them.”
“With the pen that was given to you please write what you remember from the story. When you are finished turn you paper face down and raise you hand. At this time I will come collect your answer sheets and pens.
“Thank you for participating in this study. The goal of this study was to examine how personality types and musical distractions affect reading comprehension. Does anyone have any questions?”
“Thanks for coming. You are free to leave at this time.”
Pilot Testing
Once you have decided on your tasks, instructions, materials and testing procedure, you should find a few volunteers to try them out on. This is called pilot testing: You rehearse the experimental procedure to make sure that the participants can actually understand your instructions (a frequent problem) and have an appropriate amount of time for each part of the procedure (neither too little nor too much time). Then you can check the responses from the pilot test to see whether you’re getting the data that you need.
Previous students in Research Methods often showed surprise at how important it is to rehearse the procedure before collecting data and to do pilot testing to double check instructions, materials and timing. Several of them had to discard data and start again, which was very stressful!
You can’t just improvise your data collection procedure! Too many things can go wrong.
Data analysis procedures
Design and Analyses
An extremely important part of planning an experiment is deciding which techniques will be used to analyze the data and exactly what kinds of data those techniques require. Many too many people happily go out and collect data and then panic when they don’t know how to analyze it. Also, if you don’t plan the analyses ahead of time, you run a serious risk of not collecting the right data. Then you have to go back and collect more, which is very time consuming and stressful. That’s why we talked about the experimental design first.
Think ahead for a minute. You’ll end up with (for example) 15 participants in each of 4 experimental conditions. For each participant you’ll have both the background questionnaire and the measures that you generated from their responses to the experiment. This is your data set. How will you analyze it statistically?
One question that comes up is “Do I have to do statistics?” The short answer is “Yes.” At the very least, to see how they’re used and to learn to do them yourself.
Experimental Design. When you did your experimental design it was also a plan for doing statistical analyses. In our case, the plan is to use Analysis of Variance (ANOVA) – that’s what we planned the data collection for. Most of you will be using a simple two- or three- factor design. Here you need to specify what factors correspond to your independent variables and how many levels (or groups) there are for each factor.
So, in our simple case, you’ll have (for example) Gender and Music as between-subjects factors, where each has two levels. This is just re-stating your design as a plan for analysis.
Statistical techniques. Most of you will be doing Analysis of Variance (ANOVA). Why use ANOVA? For one thing, it’s the most widely used technique in experimental psychology. For another, it’s the technique of choice for comparing groups like “Male” and “Female” or “Narrative text” and “Procedural text”. Another (strong) reason is that we planned our data collection specifically for this technique.
If we had factors that were measured on a continuous scale, like Age and IQ, then we would have two choices. We could sacrifice some information by forcing the factors into groups (like “Older” and “Younger”) so that we could use ANOVA. Or we could keep the continuous-scale factors and move to another statistical technique like regression.
There are very many options for statistical analysis. The main point here is that you need to choose your statistical approach first and then build your data collection method to fit that approach.
Interpretation. If there is a significant difference due to your factor(s) – a main effect – what does that mean? The direction of the effect will be important – think through each scenario: group A performs significantly more than group B, group B performs significantly more than group A, or no difference. What will each kind of result mean? If there are two factors or independent variables, then you have to consider what a significant interaction or joint effect means – when they create significant differences jointly. (See Mitchell & Jolley, 2005, Ch. 9 for more discussion)
It’s very important to plan the procedure out and think it through in great detail so that all of the participants go through the same exact steps. This is one part of experimental control: ensuring that variations in the procedure (which are not being studied) do not lead to different outcomes.
It’s worth repeating that one of the main reasons for planning out your procedure very carefully is to be sure that you don’t miss any real differences. One goal of having systematic methods is to avoid missing any genuine effects that may result from your experimental manipulation of the “contrast” – this is called “avoiding Type II statistical error”. Different ways of minimizing Type II statistical error are described in the following section. Careful planning of the procedure helps you maximize the standardization of your collection and analysis procedures, so you reduce error variance and make it easier to detect significant effects of your factors.
| Tasks | Describe the specific tasks that the participants will carry out during the experiment. For example: “The participants will each read the experimental text and answered a 20-item reading comprehension test.” If differences in the tasks are the focus of the experiment, then the different tasks have to be described in detail. Why were these tasks chosen? |
||
| Setting | Describe the setting where the participants will carry out the experiment. If differences in the setting (such as ambient noise or music) are not being studied, this description can be as simple as “a quiet room” or “their usual classroom”. If differences in the setting are the focus of the experiment, then the different conditions have to be described in detail. Why were these settings chosen? |
||
| Measurements | Describe as precisely as possible what you will be quantifying to try to measure your process. What will you count or measure and why? How many measurements will you have for each participant? In many cases, you’ll simply be able to count up the responses based on a list of correct or expected responses. In other cases, you’ll need to describe how you’ll code or classify the responses. Coding the responses. You also need to describe how you will classify or “code” the participants’ responses before you count them up. For example, if you are giving the participants a comprehension test for which you have the “correct” answers, then that’s what you will use to code the answers as correct or not. Answering questions. If the participants are writing out free responses to questions, then things get more complicated. You have to explain what you will consider to be a “positive” response or a “negative” response, or a “correct” or “incorrect” response, or however you want to categorize them. You can try things like “A positive response will contain words from List A and no words from List B” or “A correct response will include the information that the brain has two hemispheres, even if the response uses synonyms or paraphrases of that information” (making a list of acceptable synonyms will be helpful). Recall protocols. In reading research, another technique is to have participants write down everything that they can remember of what they’ve read (this is called a “recall protocol”). Then, for each sentence of the original text, see whether (or how much of) the same information (perhaps with synonyms or a paraphrase) for that sentence appears in the participant’s recall protocol. The information might appear as the same as the original (one measure), or might be altered or re-organized (another measure). This can show which parts of the original text were more important to the participants and what kinds of processes they used for each (with or without changes). The important thing in this section is to be explicit and clear enough so that someone else could actually do the coding for you. This degree of clarity helps you avoid situations where you’re not sure how to code the data, even when you’re doing it yourself. |
||
| Instructions | You always need to provide clear, direct instructions so that the participants know what to do. There are usually different instructions for each part of the experiment: instructions on how to start the experiment, on how to finish, on what to do next, etc. These instructions may be different in different experimental conditions of the same experiment, as well. Every participant in a given experimental condition must get the same exact instructions. There should be minimal differences in instructions from one condition to another. There is a sizeable experimental literature showing that even small variations in phrasing instructions can lead to very different results. Use very short, simple sentences and avoid any technical terms. Decide: What instructions will you provide at each step?
A better strategy is to provide only the immediately relevant information at each step of the experiment. Just answer “what should I do right now, for this step?” Example: First: “Study the text carefully so that you can answer some questions about it. You will have 3 minutes to study it.” Later: “Please stop reading and pass the text to the front of the class. Now please answer the questions on the next sheet that you receive according to what you read. You will have 7 minutes.” Later: “Please put your pen down now. Pass the sheets to the front of the class.” Later: “Thank you very much for your participation. Do you have any questions about the study?” To implement this strategy, as part of this task, you will develop a Script, just like for a movie or a play: a list of who says and does what at each step of the experiment. The script is not included in the final research report or in the published version. It’s used here to help plan the experiment in more detail. Submit it as a separate document. |
Sessions
You probably won’t collect all of your data at once. It’s more likely that you’ll run through the procedure until you get all of the conditions done or accumulate enough participants.
Each time you run through your procedure, you’ll have completed a data collection session. It’s important to know how long a session takes and how many sessions you’ll need so that you can double check how long you’ll need to collect all of your data. Pilot testing will help you estimate this better.
Don’t forget to include an estimate of how long it will take to distribute and collect all of the materials for each session. When you have 30 participants, this can take quite a while. You really should bring a research assistant to help with this.
Sample Script
“Hello everyone, please be seated”
“I am going to pass out a consent form for this experiment. Please read the form and sign the bottom if you agree to further participate in this experiment.”
“I am now going to pass out a questionnaire, please answer the questions as accurately as possible. When you are finished turn it over and leave it on you desk.”
“I am now going to pass out two papers to you, one containing a story and the other blank. Please keep the paper with writing face down until instructed to turn it over.”
“You will have 1 min 45 s to read the text. After the time is up, turn the paper back over and I will come and collect them. When I collect you paper I will also give you a pen you will be required to write with. You will then write on the blank sheet what you remember from the story. You may now turn over your paper and begin reading.”
“Stop. Please turn you paper face down and I will come collect them.”
“With the pen that was given to you please write what you remember from the story. When you are finished turn you paper face down and raise you hand. At this time I will come collect your answer sheets and pens.
“Thank you for participating in this study. The goal of this study was to examine how personality types and musical distractions affect reading comprehension. Does anyone have any questions?”
“Thanks for coming. You are free to leave at this time.”
Pilot Testing
Once you have decided on your tasks, instructions, materials and testing procedure, you should find a few volunteers to try them out on. This is called pilot testing: You rehearse the experimental procedure to make sure that the participants can actually understand your instructions (a frequent problem) and have an appropriate amount of time for each part of the procedure (neither too little nor too much time). Then you can check the responses from the pilot test to see whether you’re getting the data that you need.
Previous students in Research Methods often showed surprise at how important it is to rehearse the procedure before collecting data and to do pilot testing to double check instructions, materials and timing. Several of them had to discard data and start again, which was very stressful!
You can’t just improvise your data collection procedure! Too many things can go wrong.
Data analysis procedures
Design and Analyses
An extremely important part of planning an experiment is deciding which techniques will be used to analyze the data and exactly what kinds of data those techniques require. Many too many people happily go out and collect data and then panic when they don’t know how to analyze it. Also, if you don’t plan the analyses ahead of time, you run a serious risk of not collecting the right data. Then you have to go back and collect more, which is very time consuming and stressful. That’s why we talked about the experimental design first.
Think ahead for a minute. You’ll end up with (for example) 15 participants in each of 4 experimental conditions. For each participant you’ll have both the background questionnaire and the measures that you generated from their responses to the experiment. This is your data set. How will you analyze it statistically?
One question that comes up is “Do I have to do statistics?” The short answer is “Yes.” At the very least, to see how they’re used and to learn to do them yourself.
Experimental Design. When you did your experimental design it was also a plan for doing statistical analyses. In our case, the plan is to use Analysis of Variance (ANOVA) – that’s what we planned the data collection for. Most of you will be using a simple two- or three- factor design. Here you need to specify what factors correspond to your independent variables and how many levels (or groups) there are for each factor.
So, in our simple case, you’ll have (for example) Gender and Music as between-subjects factors, where each has two levels. This is just re-stating your design as a plan for analysis.
Statistical techniques. Most of you will be doing Analysis of Variance (ANOVA). Why use ANOVA? For one thing, it’s the most widely used technique in experimental psychology. For another, it’s the technique of choice for comparing groups like “Male” and “Female” or “Narrative text” and “Procedural text”. Another (strong) reason is that we planned our data collection specifically for this technique.
If we had factors that were measured on a continuous scale, like Age and IQ, then we would have two choices. We could sacrifice some information by forcing the factors into groups (like “Older” and “Younger”) so that we could use ANOVA. Or we could keep the continuous-scale factors and move to another statistical technique like regression.
There are very many options for statistical analysis. The main point here is that you need to choose your statistical approach first and then build your data collection method to fit that approach.
Interpretation. If there is a significant difference due to your factor(s) – a main effect – what does that mean? The direction of the effect will be important – think through each scenario: group A performs significantly more than group B, group B performs significantly more than group A, or no difference. What will each kind of result mean? If there are two factors or independent variables, then you have to consider what a significant interaction or joint effect means – when they create significant differences jointly. (See Mitchell & Jolley, 2005, Ch. 9 for more discussion)
| Read this topic next: | Review your Methods |
About
© 2009 Created by Mike Dillinger, PhD on Ning. Create a Ning Network!
