How to do your own experiments in psychology and education
5.2b Choose a sample of the possible participants
You can’t study everyone. How will you choose your participants?
When we do an experiment, because of practical limitations we work with a sample of the population of interest. For example, we study 40 college students (the sample) because we’re interesting in how adults in general (the population) perform some task. Then, we would like to make a generalization and say that what happened with the college students would happen with any adult. More general (and true) statements are more valuable, so we want to maximize our generalizations.
To make the generalization plausible, we have to either show that the sample of students is representative of the population of adults or restrict our generalization to a smaller population. It’s best to plan for the broader generalization.
Representativity here means that the sample has the same characteristics as the population, distributed in similar ways and amounts. For example, a similar distribution of males and females, high- and low-IQ individuals, republicans and democrats, etc. – whichever characteristics are considered relevant.
How representative are these participants of the population that you want to understand? If your sample of participants is representative, and you can show it, then you can generalize your results. This means that although you only studied 20 or 40 or 100 people, you can convince your readers to conclude that the same results would be found with anyone in the population.
It’s generally accepted that the pool of participants taken from the students in Introduction to Psychology are a fairly random sample of the university population. Although it’s questionable, most researchers also think that they’re fairly representative of adults in general, for most research problems.
To establish representativity, you need to show that the participants have (almost) the same values as the population for the same characteristics. For example, similar mean IQ (and similar standard deviation), similar reading level, similar proportion of males and females, etc. according to what’s relevant for the problem that you’re studying.
Some people end the discussion here: with the sampling of the participants. However, all of the same principles hold for the representativity of the materials that are used, the tasks, the instructions, and the testing instruments. External validity increases as the researcher can show that the parts of the experiment are representative of the respective population (of materials, tasks, etc.).
How do we work toward maximizing external validity? By using systematic sampling techniques. There are several things that the experimenter has to sample: which participants, which materials, which tasks, etc.
For example, if the population of interest is all American adults, then the experimenter might choose a random sample by race (independent samples for each race) and ensure that the distribution of races in the sample follows the distribution of races in the population (e.g., 10% Black, 4% Asian-American, 1% Native American, etc. – this is called proportional stratified sampling).
Often, even apparently small differences among the participants can lead to big differences in the experimental results. One group may be more intelligent or more experienced than another, for example. This is why being careful about selecting subjects randomly is very important, so that these differences tend to cancel each other out. The background questionnaire is a kind of “insurance” so that the experimenter can double-check this assumption after collecting the data.
When we do an experiment, because of practical limitations we work with a sample of the population of interest. For example, we study 40 college students (the sample) because we’re interesting in how adults in general (the population) perform some task. Then, we would like to make a generalization and say that what happened with the college students would happen with any adult. More general (and true) statements are more valuable, so we want to maximize our generalizations.
To make the generalization plausible, we have to either show that the sample of students is representative of the population of adults or restrict our generalization to a smaller population. It’s best to plan for the broader generalization.
Representativity here means that the sample has the same characteristics as the population, distributed in similar ways and amounts. For example, a similar distribution of males and females, high- and low-IQ individuals, republicans and democrats, etc. – whichever characteristics are considered relevant.
How representative are these participants of the population that you want to understand? If your sample of participants is representative, and you can show it, then you can generalize your results. This means that although you only studied 20 or 40 or 100 people, you can convince your readers to conclude that the same results would be found with anyone in the population.
It’s generally accepted that the pool of participants taken from the students in Introduction to Psychology are a fairly random sample of the university population. Although it’s questionable, most researchers also think that they’re fairly representative of adults in general, for most research problems.
To establish representativity, you need to show that the participants have (almost) the same values as the population for the same characteristics. For example, similar mean IQ (and similar standard deviation), similar reading level, similar proportion of males and females, etc. according to what’s relevant for the problem that you’re studying.
Some people end the discussion here: with the sampling of the participants. However, all of the same principles hold for the representativity of the materials that are used, the tasks, the instructions, and the testing instruments. External validity increases as the researcher can show that the parts of the experiment are representative of the respective population (of materials, tasks, etc.).
| Maximize External Validity. Design the experiment for more reliable generalizations. External validity increases as the researcher can show that the samples for each part of the experiment are representative of the respective population (of materials, tasks, etc.). |
How do we work toward maximizing external validity? By using systematic sampling techniques. There are several things that the experimenter has to sample: which participants, which materials, which tasks, etc.
For example, if the population of interest is all American adults, then the experimenter might choose a random sample by race (independent samples for each race) and ensure that the distribution of races in the sample follows the distribution of races in the population (e.g., 10% Black, 4% Asian-American, 1% Native American, etc. – this is called proportional stratified sampling).
| Random sampling. | In random sampling, the researcher takes steps to avoid any systematic bias in deciding which individuals from the population of interest actually participate in the experiment: each member of the population has exactly the same chance of participating in the experiment. Bias here means that (usually inadvertently) more of one subtype of participant gets assigned to one experimental condition than to another condition. This kind of bias can lead the research to (falsely) conclude that the factor investigated caused differences in the observed outcomes, when in fact the different subtypes of participants were the cause. Truly random sampling from a population of participants is an ideal that is difficult to attain and is usually an issue only with large-scale studies that have hundreds or thousands of participants. |
| Stratified sampling. | Stratified (or probability) sampling is the procedure in which approximately random sampling is applied to parts of the population. |
| Convenience sampling. | The reality of student research is that you will be happy to accept for testing anyone you can convince to volunteer. This is simply the sample that is most convenient for the experimenter, with absolutely no guarantees (and little likelihood) that the sample will be at all representative. This means that it will be difficult to accept any generalization of the results to a broader population. |
| For this course, you will use convenience sampling and run the risk that your results will be difficult to generalize. |
Often, even apparently small differences among the participants can lead to big differences in the experimental results. One group may be more intelligent or more experienced than another, for example. This is why being careful about selecting subjects randomly is very important, so that these differences tend to cancel each other out. The background questionnaire is a kind of “insurance” so that the experimenter can double-check this assumption after collecting the data.
| Read this topic next: | Assign the participants to your experimental conditions |
About
© 2009 Created by Mike Dillinger, PhD on Ning. Create a Ning Network!
