How to do your own experiments in psychology and education
5.1 Decide how to measure your sub-process
Your experimental design sketches out how your factors (or independent variables) determine your data collection procedure. You chose these factors because you suspect that they will affect the process or sub-process that you are studying.
The next step, then, is to decide how you will measure the changes that do (or do not) happen in your sub-process, in other words, you need to decide on your measures or dependent variables. This is an extremely important decision!
If your measures aren’t detailed enough or relevant enough then the experiment fails because you’re unable to detect differences that do in fact happen in your sub-process.
For psychologists, the big issue is that it’s simply not possible to observe comprehension or knowledge integration directly: no one knows how to open up someone’s brain and film what’s happening, and even if someone did know, he wouldn’t be allowed to do it. So researchers have to measure mental processes indirectly through the participants’ responses: their answers to multiple-choice questions, their summaries, their actions, etc.
In sum, participants provide some kind of observable response that you can link to the sub-process that you are studying. The dependent variable is what you count up in those responses.
Example
Say you want to study how people comprehend the user manual for a coffee maker. In particular, you’re interested in how readers use prior knowledge (about coffe makers and other things) to try to understand the text. So, in this case, your process is comprehension, and your sub-process is knowledge integration. You may think that terminology, sentence length, gender, or musical distractions will affect comprehension, but the factors aren’t important for the moment.
One common approach is to ask people to read the manual and then have them answer questions about what they read. The number of correct responses, in this case, measures comprehension: more correct responses suggest greater comprehension. So far, so good.
But, what will the number correct tell you about knowledge integration (your sub-process)? Not very much at all, unless you used a task and/or questions that were specifically designed to require the readers to do knowledge integration. The type of question that you ask is very important. For example, a question like What is a cappuccino? will provide information about word knowledge that the reader has, but not about how the reader combines this prior knowledge with the information from the manual. It might be a useful question to investigate the sub-process of word recognition.
On the other hand, “trick questions” with plausible information might serve as better indicators of knowledge integration – to see how the reader mixes prior knowledge with text information. For example, Where should you put your coffee mug to receive the coffee? might lead to incorrect answers like under to filter holder or under the spout even if the instructions say explicitly not to use a coffee mug because it won’t fit under the filter holder. It’s common to put a coffee mug under the filter holder in other coffee machines, so it seems that the readers who give these answers are using prior knowledge rather than text information. An answer like this machine is different from my other coffee maker -- you can’t use a mug shows that the reader is comparing his prior knowledge with the text information. A question like How is this coffee maker different from other coffee makers that you have used? will push participants to do more knowledge integration.
It’s more common for researchers to ask for a recall or summary of the text to study knowledge integration. Good indicators of knowledge integration are information that was not in the original text (added information), changes to the text information (changed information) in the participants’ responses, and comparisons of new information with prior knowledge (comparative information).
To focus on a sub-process, the measurements have to be much more precise, so you have to pay more attention to exactly how you are measuring.
You may use specific kinds of questions, classify responses in a specific way, or both. It’s also very common to try to measure the same thing in several different ways, for example, asking for answers to questions and for a summary, as well.
Because measurement of psychological processes and states is so indirect, there is plenty of methodological discussion about the different options that are available. Some researchers argue that answering multiple-choice questions after reading a whole text is a measure of (retrieval from) memory but not a good measure of the process of getting the textual information into memory (i.e., comprehension). They say that there are too many other things happening between comprehension and the questions for this measure to say anything interesting about comprehension and, of course, propose other methods (based on reaction times or eye movements during reading). They raise the question: “does your ‘comprehension test’ really measure comprehension or something else?” You need to be ready to answer this question about your measures, as well!
There usually is no definitive answer for this kind of discussion, just a lot of argument back and forth about the strengths and weaknesses of different measures. This is one reason to mention other studies that have used the same measures [dependent variables] that you use. The implicit reasoning is that if other researchers have successfully used the same measures, then at the very least your results will be comparable with theirs.
Measures, then, have three parts:
Many experiments have test materials that are separate from the stimuli. In the example above, the participants read a text and the experimenter tested their comprehension with a set of questions. The target text and the set of questions is one example of test materials. Another experimenter might use the same text and ask for a summary: the target text and the instructions for the summary are a second example of test materials.
In some simpler experiments, like a study of writing, participants may just respond to instructions about what to write about. Note that in this kind experiment there aren’t any stimuli. Participants just get instructions and write in response to the instructions. So, some test materials may also be optional. However, there has to be some response from the participants that will allow you to measure the process that you are studying.
To take another example, in reaction time experiments, participants react to stimuli and there is no separate response sheet as there would be for questions or summaries.
Participants’ responses
Common participant responses that psychologists have used are: spontaneous behavior, pointing, pressing a button, choosing from a list of options, and verbal responses. Verbal responses can be as simple as yes or no, as common as short answers to questions, or as complex as long written texts.
To improve the reliability of your measurements, participant responses should be easy to record (in writing, on tape, on video, by computer, etc.) so that you can review and analyze them more than once. That will allow you and other people to review your analyses, which will make the results more reliable.
Your dependent measures are numbers, so you will need to count or measure something. Often, you will measure the same thing in different ways or measure more than one thing. So, it’s common to have several dependent variables.
Construct validity
Why did you choose these test materials? Why did you choose these specific kinds of responses?
You have to explain how your materials and responses relate to the specific process under study – this is called “construct validity”. Convince the reader that they are a good way of measuring your process. This is another important goal that researchers try to reach.
Maximize Construct Validity. Make sure that you’re measuring your process or sub-process and not something else.
Construct validity is the technical term for how relevant and reliable your measures are for measuring some underlying or unobservable process or theoretical construct. For example, the Wechsler Intelligence Scale for Children (WISC) score (the measure) has high construct validity for measuring intelligence (the construct), but a low construct validity for measuring chess ability (a different construct). We know because there have been many studies to check how reliable and relevant it is – it’s a standardized test. If you are using standardized testing materials or materials from some other experiment, be sure to cite the source; that will help convince the reader that your measures are reliable.
Reading comprehension tests, for example the standardized tests used for the SAT and GRE, are an interesting case. They provide relevant and reliable information about reading comprehension as a whole, but different questions on the tests provide information about different parts of the reading process. So, a standard reading comprehension test will not be useful for measuring only one sub-process of reading. To increase construct validity, you would have to choose the specific kinds of comprehension question that are related to the particular sub-process that you want to focus on.
Don’t take your measures for granted. Prepare your reasons for convincing the reader that you are really measuring your process.
The next step, then, is to decide how you will measure the changes that do (or do not) happen in your sub-process, in other words, you need to decide on your measures or dependent variables. This is an extremely important decision!
If your measures aren’t detailed enough or relevant enough then the experiment fails because you’re unable to detect differences that do in fact happen in your sub-process.
For psychologists, the big issue is that it’s simply not possible to observe comprehension or knowledge integration directly: no one knows how to open up someone’s brain and film what’s happening, and even if someone did know, he wouldn’t be allowed to do it. So researchers have to measure mental processes indirectly through the participants’ responses: their answers to multiple-choice questions, their summaries, their actions, etc.
In sum, participants provide some kind of observable response that you can link to the sub-process that you are studying. The dependent variable is what you count up in those responses.
Example
Say you want to study how people comprehend the user manual for a coffee maker. In particular, you’re interested in how readers use prior knowledge (about coffe makers and other things) to try to understand the text. So, in this case, your process is comprehension, and your sub-process is knowledge integration. You may think that terminology, sentence length, gender, or musical distractions will affect comprehension, but the factors aren’t important for the moment.
One common approach is to ask people to read the manual and then have them answer questions about what they read. The number of correct responses, in this case, measures comprehension: more correct responses suggest greater comprehension. So far, so good.
But, what will the number correct tell you about knowledge integration (your sub-process)? Not very much at all, unless you used a task and/or questions that were specifically designed to require the readers to do knowledge integration. The type of question that you ask is very important. For example, a question like What is a cappuccino? will provide information about word knowledge that the reader has, but not about how the reader combines this prior knowledge with the information from the manual. It might be a useful question to investigate the sub-process of word recognition.
On the other hand, “trick questions” with plausible information might serve as better indicators of knowledge integration – to see how the reader mixes prior knowledge with text information. For example, Where should you put your coffee mug to receive the coffee? might lead to incorrect answers like under to filter holder or under the spout even if the instructions say explicitly not to use a coffee mug because it won’t fit under the filter holder. It’s common to put a coffee mug under the filter holder in other coffee machines, so it seems that the readers who give these answers are using prior knowledge rather than text information. An answer like this machine is different from my other coffee maker -- you can’t use a mug shows that the reader is comparing his prior knowledge with the text information. A question like How is this coffee maker different from other coffee makers that you have used? will push participants to do more knowledge integration.
It’s more common for researchers to ask for a recall or summary of the text to study knowledge integration. Good indicators of knowledge integration are information that was not in the original text (added information), changes to the text information (changed information) in the participants’ responses, and comparisons of new information with prior knowledge (comparative information).
To focus on a sub-process, the measurements have to be much more precise, so you have to pay more attention to exactly how you are measuring.
You may use specific kinds of questions, classify responses in a specific way, or both. It’s also very common to try to measure the same thing in several different ways, for example, asking for answers to questions and for a summary, as well.
Because measurement of psychological processes and states is so indirect, there is plenty of methodological discussion about the different options that are available. Some researchers argue that answering multiple-choice questions after reading a whole text is a measure of (retrieval from) memory but not a good measure of the process of getting the textual information into memory (i.e., comprehension). They say that there are too many other things happening between comprehension and the questions for this measure to say anything interesting about comprehension and, of course, propose other methods (based on reaction times or eye movements during reading). They raise the question: “does your ‘comprehension test’ really measure comprehension or something else?” You need to be ready to answer this question about your measures, as well!
There usually is no definitive answer for this kind of discussion, just a lot of argument back and forth about the strengths and weaknesses of different measures. This is one reason to mention other studies that have used the same measures [dependent variables] that you use. The implicit reasoning is that if other researchers have successfully used the same measures, then at the very least your results will be comparable with theirs.
Measures, then, have three parts:
- the test or “instrument” that participants are responding to [ex: questions],
- the participant’s response [ex: answers or answer types], and
- an explanation of how each type of response relates to the specific process under study.
Many experiments have test materials that are separate from the stimuli. In the example above, the participants read a text and the experimenter tested their comprehension with a set of questions. The target text and the set of questions is one example of test materials. Another experimenter might use the same text and ask for a summary: the target text and the instructions for the summary are a second example of test materials.
In some simpler experiments, like a study of writing, participants may just respond to instructions about what to write about. Note that in this kind experiment there aren’t any stimuli. Participants just get instructions and write in response to the instructions. So, some test materials may also be optional. However, there has to be some response from the participants that will allow you to measure the process that you are studying.
To take another example, in reaction time experiments, participants react to stimuli and there is no separate response sheet as there would be for questions or summaries.
Participants’ responses
Common participant responses that psychologists have used are: spontaneous behavior, pointing, pressing a button, choosing from a list of options, and verbal responses. Verbal responses can be as simple as yes or no, as common as short answers to questions, or as complex as long written texts.
To improve the reliability of your measurements, participant responses should be easy to record (in writing, on tape, on video, by computer, etc.) so that you can review and analyze them more than once. That will allow you and other people to review your analyses, which will make the results more reliable.
Your dependent measures are numbers, so you will need to count or measure something. Often, you will measure the same thing in different ways or measure more than one thing. So, it’s common to have several dependent variables.
Construct validity
Why did you choose these test materials? Why did you choose these specific kinds of responses?
You have to explain how your materials and responses relate to the specific process under study – this is called “construct validity”. Convince the reader that they are a good way of measuring your process. This is another important goal that researchers try to reach.
Maximize Construct Validity. Make sure that you’re measuring your process or sub-process and not something else.
Construct validity is the technical term for how relevant and reliable your measures are for measuring some underlying or unobservable process or theoretical construct. For example, the Wechsler Intelligence Scale for Children (WISC) score (the measure) has high construct validity for measuring intelligence (the construct), but a low construct validity for measuring chess ability (a different construct). We know because there have been many studies to check how reliable and relevant it is – it’s a standardized test. If you are using standardized testing materials or materials from some other experiment, be sure to cite the source; that will help convince the reader that your measures are reliable.
Reading comprehension tests, for example the standardized tests used for the SAT and GRE, are an interesting case. They provide relevant and reliable information about reading comprehension as a whole, but different questions on the tests provide information about different parts of the reading process. So, a standard reading comprehension test will not be useful for measuring only one sub-process of reading. To increase construct validity, you would have to choose the specific kinds of comprehension question that are related to the particular sub-process that you want to focus on.
Don’t take your measures for granted. Prepare your reasons for convincing the reader that you are really measuring your process.
| Read this topic next: | Define and find your participants |
About
© 2009 Created by Mike Dillinger, PhD on Ning. Create a Ning Network!
