How to do your own experiments in psychology and education
5.3 Develop your materials
The second part of the Methods section describes what materials the participants worked with and any apparatus that was used. Your choice of materials is very important to focus the experiment on a particular process or sub-process and to make the participants provide the type of responses that you are looking for. Don’t underestimate just how important the materials are.
There are three steps for developing your materials:
Your choice of materials is very important to focus the experiment on a particular process or sub-process and to provide the type of responses that you are looking for. Don’t underestimate just how important the materials are.
Choose stimuli and distractors
When we do an experiment, because of practical limitations we work with a sample of the all the materials of possible interest. For example, we study students reading one text (the sample) because we’re interesting in how adults read any text (the population). Then, we would like to make a generalization and say that what happened with the one text would happen with any text. More general (and true) statements are more valuable, so we want to maximize our generalizations.
To make the generalization plausible, we have to either show that the sample of texts is representative of the population of all reading materials or restrict our generalization to a particular kind of reading materials. It’s best to plan for the broader generalization.
Representativity here means that the sample has the same characteristics as the population, distributed in similar ways and amounts. For example, a similar distribution of words and sentences, high- and low-frequency words, long and short sentences, etc. – whichever characteristics are considered relevant.
External validity increases as the researcher can show that the sample materials are representative of the respective population of materials. Ideally, some sort of random sampling would be used. In practice, however, it is more common to give reasons to think that the sample materials are representative.
Will you provide the participants with stimuli to react to? A picture to view? A text to read? Sometimes there are no stimuli. For example, in many experiments about writing, there are only instructions to write about this or that topic, with no other stimuli. Sometimes there are multiple stimuli, for example when there are distractors, as in the example above.
If you are using stimuli, then describe them in detail, including the source where you got them. If you have more than one set of stimuli, then describe each set.
Why did you choose these particular stimuli? It’s important to show that they’re representative of some population of relevant stimuli and how they’re related to the goals of the experiment. Convince the reader that they will help reach the goals of the experiment.
Think through the same questions for any distractors that you may want to use. Why do you think that they will be distracting? Will they interfere with the specific sub-process that you want to study or with processing in general?
Response sheets
How will your participants respond to the stimuli? Common participant responses that psychologists have used are: spontaneous behavior, pointing, pressing a button, choosing from a list of options, and verbal responses. Verbal responses can be as simple as yes or no, as common as short answers to questions, or as complex as long written texts.
To improve the reliability of your measurements, participant responses should be easy to record (in writing, on tape, on video, by computer, etc.) so that you can review and analyze them more than once. That will allow you and other people to review your analyses, which will make the results more reliable. Taking notes while participants respond is both unreliable and extremely stressful for the researcher.
One option is to give participants a response sheet where they can record their responses, either by marking an option or by writing in an answer.
It usually works best to have the question or prompt right next to the place where they should put their answers. If that’s not practical, then be sure to number both the stimuli and the places for responses very clearly.
The response sheet should have several things:
A useful technique for using Likert-type questions is to ask participants to respond by making a mark that crosses the horizontal line (rather than circle one of the options) at the place that best reflects their answer, as in the example below. It’s best if the scales are all the same exact length.
For your dependent variable, you can measure the distance from one end of the scale to the mark. This will be much more precise than circling an option and will help you manage error variance better.
Background Questionnaires
The background questionnaire is one kind of “experimenter’s insurance”. You can use it to double check the representativity of your sample and check for any characteristics that coincidentally were present more frequently in one condition than in another.
You can also use the information from the background questionnaire to do follow-up analyses. Say, for example, that you detected no significant differences when you test your factors. This is where the “insurance” kicks in: you can reanalyze the data with new factors derived from the information on the background questionnaire. For example, maybe there were age or sex differences in your sample that weren’t your main focus. Reanalyzing the data might give you unintended results that you can report, even if your planned factors did not seem to affect the process you are studying. That’s much more practical than going out to collect more data.
Anonymity. When you take a look at the consent form in the next section, you’ll see that participants are guaranteed anonymity.
What most experimenters do is assign each participant an arbitrary number or code as soon as they show up to participate. Avoid having participants put their names on whatever data they provide for you, including the background questionnaire.
If there’s a mistake or a missing code, then all of that participant’s data has to go to the garbage and you will probably have to collect replacement data.
Questions. What should you ask on the background questionnaire? Questions about things that you think might affect the process that you are studying. For example, age, major, and gender are often relevant. If you’re studying reading or problem solving, you might want to add some questions to get an idea of how much the participants know about the topic that’s involved.
Name, address, telephone number, student ID, and shoe size do not affect any known psychological processes, so don’t ask about them.
How many questions? There are no fixed limits either way. If you ask only a few questions, then you’ll have fewer opportunities for reanalyzing the data, i.e., less “insurance”. But asking too many questions will take up too much time. As a rule of thumb, 20 questions is fine, but not much more than that, unless you have something special in mind.
What kinds of questions? Multiple choice, yes/no, fill in the blanks, and open-ended questions can all be valuable. Likert-scale questions are also widely used: the participant has to put a mark along a scale of 5 or 7 items, e.g., from “strongly disagree” to “strongly agree”.
Wording. There’s a fine art of writing questions for questionnaires. There are professionals who are experts at doing only that. This is because people have a strong tendency to misunderstand your questions and provide you with irrelevant responses. One solution is to word the questions very, very simply, very directly, and in a way that is clear to the readers (not only to you!). Another solution is to test the questions with some extra participants to make sure that they understand your questions. This is called pilot testing.
There are three steps for developing your materials:
| a Make a background questionnaire |
b Choose your stimuli |
c Make a response sheet |
| You will need to double check and to document what your participants are really like. | Decide which stimuli your participants will respond to. Check that your stimuli will really activate the sub-process that you want to study. | How will your participants respond to the stimuli? Give them a way to provide organized responses. |
Choose stimuli and distractors
| Maximize External Validity. Design the experiment for more reliable generalizations. |
When we do an experiment, because of practical limitations we work with a sample of the all the materials of possible interest. For example, we study students reading one text (the sample) because we’re interesting in how adults read any text (the population). Then, we would like to make a generalization and say that what happened with the one text would happen with any text. More general (and true) statements are more valuable, so we want to maximize our generalizations.
To make the generalization plausible, we have to either show that the sample of texts is representative of the population of all reading materials or restrict our generalization to a particular kind of reading materials. It’s best to plan for the broader generalization.
Representativity here means that the sample has the same characteristics as the population, distributed in similar ways and amounts. For example, a similar distribution of words and sentences, high- and low-frequency words, long and short sentences, etc. – whichever characteristics are considered relevant.
External validity increases as the researcher can show that the sample materials are representative of the respective population of materials. Ideally, some sort of random sampling would be used. In practice, however, it is more common to give reasons to think that the sample materials are representative.
Will you provide the participants with stimuli to react to? A picture to view? A text to read? Sometimes there are no stimuli. For example, in many experiments about writing, there are only instructions to write about this or that topic, with no other stimuli. Sometimes there are multiple stimuli, for example when there are distractors, as in the example above.
If you are using stimuli, then describe them in detail, including the source where you got them. If you have more than one set of stimuli, then describe each set.
Why did you choose these particular stimuli? It’s important to show that they’re representative of some population of relevant stimuli and how they’re related to the goals of the experiment. Convince the reader that they will help reach the goals of the experiment.
Think through the same questions for any distractors that you may want to use. Why do you think that they will be distracting? Will they interfere with the specific sub-process that you want to study or with processing in general?
Response sheets
How will your participants respond to the stimuli? Common participant responses that psychologists have used are: spontaneous behavior, pointing, pressing a button, choosing from a list of options, and verbal responses. Verbal responses can be as simple as yes or no, as common as short answers to questions, or as complex as long written texts.
To improve the reliability of your measurements, participant responses should be easy to record (in writing, on tape, on video, by computer, etc.) so that you can review and analyze them more than once. That will allow you and other people to review your analyses, which will make the results more reliable. Taking notes while participants respond is both unreliable and extremely stressful for the researcher.
One option is to give participants a response sheet where they can record their responses, either by marking an option or by writing in an answer.
| A clear, extremely easy-to-understand response sheet is essential – if participants don’t know how to respond, your data will be inaccurate or unreliable. |
It usually works best to have the question or prompt right next to the place where they should put their answers. If that’s not practical, then be sure to number both the stimuli and the places for responses very clearly.
The response sheet should have several things:
- a code that will tell you (but not the participant) which condition the participant was tested in;
- the participant’s number. Make sure to include the participant’s number on the response sheet yourself. Asking participants to do it is very unreliable.
- instructions on how to respond and what to pay attention to. That way participants can refer back to the instructions if they have any questions or doubts.
- space for each response, probably numbered so that participants have no doubt which stimulus or question the response is for; Some common items for your response sheets are:
- Multiple-choice questions;
- Open-ended short-answer or “essay” questions;
- Likert-type questions;
A useful technique for using Likert-type questions is to ask participants to respond by making a mark that crosses the horizontal line (rather than circle one of the options) at the place that best reflects their answer, as in the example below. It’s best if the scales are all the same exact length.
For your dependent variable, you can measure the distance from one end of the scale to the mark. This will be much more precise than circling an option and will help you manage error variance better.
Background Questionnaires
The background questionnaire is one kind of “experimenter’s insurance”. You can use it to double check the representativity of your sample and check for any characteristics that coincidentally were present more frequently in one condition than in another.
You can also use the information from the background questionnaire to do follow-up analyses. Say, for example, that you detected no significant differences when you test your factors. This is where the “insurance” kicks in: you can reanalyze the data with new factors derived from the information on the background questionnaire. For example, maybe there were age or sex differences in your sample that weren’t your main focus. Reanalyzing the data might give you unintended results that you can report, even if your planned factors did not seem to affect the process you are studying. That’s much more practical than going out to collect more data.
Anonymity. When you take a look at the consent form in the next section, you’ll see that participants are guaranteed anonymity.
What most experimenters do is assign each participant an arbitrary number or code as soon as they show up to participate. Avoid having participants put their names on whatever data they provide for you, including the background questionnaire.
| Use maximally ethical procedures. You have to be extra, super sure that the same code goes on the same participant’s background questionnaire and all response sheets. |
If there’s a mistake or a missing code, then all of that participant’s data has to go to the garbage and you will probably have to collect replacement data.
Questions. What should you ask on the background questionnaire? Questions about things that you think might affect the process that you are studying. For example, age, major, and gender are often relevant. If you’re studying reading or problem solving, you might want to add some questions to get an idea of how much the participants know about the topic that’s involved.
Name, address, telephone number, student ID, and shoe size do not affect any known psychological processes, so don’t ask about them.
How many questions? There are no fixed limits either way. If you ask only a few questions, then you’ll have fewer opportunities for reanalyzing the data, i.e., less “insurance”. But asking too many questions will take up too much time. As a rule of thumb, 20 questions is fine, but not much more than that, unless you have something special in mind.
What kinds of questions? Multiple choice, yes/no, fill in the blanks, and open-ended questions can all be valuable. Likert-scale questions are also widely used: the participant has to put a mark along a scale of 5 or 7 items, e.g., from “strongly disagree” to “strongly agree”.
Wording. There’s a fine art of writing questions for questionnaires. There are professionals who are experts at doing only that. This is because people have a strong tendency to misunderstand your questions and provide you with irrelevant responses. One solution is to word the questions very, very simply, very directly, and in a way that is clear to the readers (not only to you!). Another solution is to test the questions with some extra participants to make sure that they understand your questions. This is called pilot testing.
| Read this topic next: | Plan your procedures |
About
© 2009 Created by Mike Dillinger, PhD on Ning. Create a Ning Network!
