How to do your own experiments in psychology and education
4.1 Different kinds of Research
You suspect that each of your factors will have an effect on the process or sub-process that you’re studying. During your review of the literature you will have found other researchers who have studied the same factors and who found evidence that they do in fact (or do not) have an effect on your process. You also want to discover whether your factors interfere with each other or show some sort of synergy or additive effect. You probably chose factors like Gender, Music, Major, Amount of Sleep, etc. [Note that it’s a standard convention to capitalize the name of each factor.]
The methodological choices that you make now will determine what kind of research you will carry out. This Research Methods course focuses on experimental research but there are other options to consider.
The first thing to observe is that experiments must involve data, but
Not every study that uses data is an experiment.
Why not? Because experiments have special characteristics that other kinds of research do not have. To put this in context, consider some types of non-experimental research, described below.
A first important point to make is that non-experimental research is perfectly valid and is often the only kind of research possible in a particular field or for a particular problem. Non-experimental research, however, generally studies different questions and provides weaker evidence than experimental research.
It is extremely important not to think that non-experimental means non-scientific or invalid.
Non-experimental research
Here are a few kinds of non-experimental research.
Observational studies. In naturalistic or observational studies, the researcher observes directly or makes recordings of what people or animals do in a particular situation. They’re called naturalistic exactly because the researcher decides not to control any factors that might influence the outcomes. Another difference is that these studies try to describe observable behavior, where experiments usually try to identify the unobservable mechanisms that theories postulate. Ethnographic research is one well-known and systematic approach to observational research.
A specific kind of observational study is one where the researcher asks people questions – this is called survey-based research. These studies are very common before elections: hordes of researchers ask people in person and by phone a list of questions (a questionnaire) about the candidates and their positions.
Small-N studies like case studies are another kind of observational study. The researcher may only have access to one or two patients with a certain kind of brain damage or rare disease. This is a real-life limitation that makes experimental research impossible. Rather than not study the problem at all, researchers try to accumulate weaker observational data, even though they know that they can’t reliably generalize the conclusions to other people.
Of course, people are very variable, especially when they know they’re being observed. Sometimes they don’t feel well or are distracted; sometimes they answer questions according to what they think the researcher wants to hear; sometimes they intentionally try to fool researchers. Just observing them, without measuring and taking into consideration these other factors, leads to less reliable results than a carfully controlled experiment would produce.
Correlational studies. In correlational studies, the researcher measures two (or more) factors of interest and evaluates statistically what their strength of association is, i.e., when one factor changes, does the other change, too? Do they change in the same direction (a positive correlation)? Do they always or very frequently change together (a strong correlation) or not?
Correlation is not the same as causation. It might be true that in a particular sample, the people with long noses have bigger shoe sizes or salaries. This does not mean that nose length makes their shoes or salaries bigger! It just means that these things appear together. The tobacco companies used this argument for very many years to say that the research did not show that smoking causes cancer. They were quite correct because most of the research was correlational, so by definition it did not show cause. Later (experimental) research, of course, showed conclusively that the tobacco companies are criminals because smoking in fact creates chemical dependencies and causes cancer.
The technique called meta-analysis uses other researchers’ statistical results as raw data, most often to find correlations among those results. One limitation of meta-analysis is that this kind of research uses only the data that is already available to test a given hypothesis. The available data may be more or less directly related to the specific question that a researcher wants to answer. Experimental research, on the other hand, generates special data that is custom designed to be as relevant as possible to the hypothesis under study.
Correlational studies are particularly useful in establishing strong candidates for important determining factors. They ask the question (rather than answer it): are these factors strong candidates for the things that are really causing what’s going on?
Advantages of non-experimental research. The strength of non-experimental research is that very often it looks at broader, less controlled situations; so, many phenomena appear that experimental research has not yet dealt with. Non-experimental research does an excellent job of separating illusions and coincidences from systematic phenomena: it shows that x does (or does not) in fact occur systematically. The next question, of course, is why or how does x occur, and that’s an issue for experimental research. Correlations provide excellent hints for study: they’re good candidates for determining factors, if they fit in with existing theories and data.
Non-experimental research is also an excellent option when experimental research is not possible, practical, or ethical.
Qualitative “vs.” Quantitative research. It’s really unfortunate that there’s a widespread and very misleading belief that research comes in two basic (stereo)types: numbers-only research that misses the “essence” of what it studies (also called “hard” science) and more detail-oriented, intuitive research that focuses on the qualities of what is studied without using much math (also called “soft” science). This extremely ill-informed opinion arose from discussions about what science “really” is with the appearance of the social sciences in the late 1800s. Social scientists who neither had the training nor the interest in mathematical methods worked with problems that were so new that it was not at all clear how to mathematize the things that they wanted to study.
On the other hand, the social scientists also wanted the prestige of doing “real research”, so they “adjusted” the notion of research to fit their needs. However, they did a really poor job of it. “Quantitative” research uses numbers to measure amounts of qualities, so it, too, is qualitative. Otherwise the numbers would be meaningless. Only mathematics studies numbers without paying attention to some form of measurement. On the other hand, qualitative research also routinely uses mathematical methods: quantities, frequencies, relative amounts, etc. even if they are simpler. More recently, the social sciences have seen an amazing increase in the use of advanced statistical modeling to the point where the distinction between quantitative and qualitative research becomes totally meaningless. Forget you ever heard about it.
Experimental research
With these kinds of non-experimental research in mind, it’s easier to characterize experimental research. In this Research Methods course, the emphasis will be on experimental research so that you have a clearer idea of what you lose when you do non-experimental research.
Experimental research focuses on different, deeper questions than non-experimental research. Non-experimental research tends to focus on what’s happening, and experimental research focuses on the more theoretical or profound question of why or how it’s happening. So, when experimental research is a good option, researchers avoid non-experimental research.
Data collection in experimental research is more controlled. The key thing that makes experimental research different from other kinds of research is this fact:
The experimenter systematically controls a range of factors that might influence the observed outcomes.
Experimental control is essential to generate tightly focused or “uncontaminated” data that is immediately and directly relevant to the hypothesis under investigation.
The Method section has to make clear not only what the experimenter is studying, but also what is not being studied and how the experimenter is controlling the effects of these other factors.
Read this topic next: Choose the levels for your factors
The methodological choices that you make now will determine what kind of research you will carry out. This Research Methods course focuses on experimental research but there are other options to consider.
The first thing to observe is that experiments must involve data, but
Not every study that uses data is an experiment.
Why not? Because experiments have special characteristics that other kinds of research do not have. To put this in context, consider some types of non-experimental research, described below.
A first important point to make is that non-experimental research is perfectly valid and is often the only kind of research possible in a particular field or for a particular problem. Non-experimental research, however, generally studies different questions and provides weaker evidence than experimental research.
It is extremely important not to think that non-experimental means non-scientific or invalid.
Non-experimental research
Here are a few kinds of non-experimental research.
Observational studies. In naturalistic or observational studies, the researcher observes directly or makes recordings of what people or animals do in a particular situation. They’re called naturalistic exactly because the researcher decides not to control any factors that might influence the outcomes. Another difference is that these studies try to describe observable behavior, where experiments usually try to identify the unobservable mechanisms that theories postulate. Ethnographic research is one well-known and systematic approach to observational research.
A specific kind of observational study is one where the researcher asks people questions – this is called survey-based research. These studies are very common before elections: hordes of researchers ask people in person and by phone a list of questions (a questionnaire) about the candidates and their positions.
Small-N studies like case studies are another kind of observational study. The researcher may only have access to one or two patients with a certain kind of brain damage or rare disease. This is a real-life limitation that makes experimental research impossible. Rather than not study the problem at all, researchers try to accumulate weaker observational data, even though they know that they can’t reliably generalize the conclusions to other people.
Of course, people are very variable, especially when they know they’re being observed. Sometimes they don’t feel well or are distracted; sometimes they answer questions according to what they think the researcher wants to hear; sometimes they intentionally try to fool researchers. Just observing them, without measuring and taking into consideration these other factors, leads to less reliable results than a carfully controlled experiment would produce.
Correlational studies. In correlational studies, the researcher measures two (or more) factors of interest and evaluates statistically what their strength of association is, i.e., when one factor changes, does the other change, too? Do they change in the same direction (a positive correlation)? Do they always or very frequently change together (a strong correlation) or not?
Correlation is not the same as causation. It might be true that in a particular sample, the people with long noses have bigger shoe sizes or salaries. This does not mean that nose length makes their shoes or salaries bigger! It just means that these things appear together. The tobacco companies used this argument for very many years to say that the research did not show that smoking causes cancer. They were quite correct because most of the research was correlational, so by definition it did not show cause. Later (experimental) research, of course, showed conclusively that the tobacco companies are criminals because smoking in fact creates chemical dependencies and causes cancer.
The technique called meta-analysis uses other researchers’ statistical results as raw data, most often to find correlations among those results. One limitation of meta-analysis is that this kind of research uses only the data that is already available to test a given hypothesis. The available data may be more or less directly related to the specific question that a researcher wants to answer. Experimental research, on the other hand, generates special data that is custom designed to be as relevant as possible to the hypothesis under study.
Correlational studies are particularly useful in establishing strong candidates for important determining factors. They ask the question (rather than answer it): are these factors strong candidates for the things that are really causing what’s going on?
Advantages of non-experimental research. The strength of non-experimental research is that very often it looks at broader, less controlled situations; so, many phenomena appear that experimental research has not yet dealt with. Non-experimental research does an excellent job of separating illusions and coincidences from systematic phenomena: it shows that x does (or does not) in fact occur systematically. The next question, of course, is why or how does x occur, and that’s an issue for experimental research. Correlations provide excellent hints for study: they’re good candidates for determining factors, if they fit in with existing theories and data.
Non-experimental research is also an excellent option when experimental research is not possible, practical, or ethical.
Qualitative “vs.” Quantitative research. It’s really unfortunate that there’s a widespread and very misleading belief that research comes in two basic (stereo)types: numbers-only research that misses the “essence” of what it studies (also called “hard” science) and more detail-oriented, intuitive research that focuses on the qualities of what is studied without using much math (also called “soft” science). This extremely ill-informed opinion arose from discussions about what science “really” is with the appearance of the social sciences in the late 1800s. Social scientists who neither had the training nor the interest in mathematical methods worked with problems that were so new that it was not at all clear how to mathematize the things that they wanted to study.
On the other hand, the social scientists also wanted the prestige of doing “real research”, so they “adjusted” the notion of research to fit their needs. However, they did a really poor job of it. “Quantitative” research uses numbers to measure amounts of qualities, so it, too, is qualitative. Otherwise the numbers would be meaningless. Only mathematics studies numbers without paying attention to some form of measurement. On the other hand, qualitative research also routinely uses mathematical methods: quantities, frequencies, relative amounts, etc. even if they are simpler. More recently, the social sciences have seen an amazing increase in the use of advanced statistical modeling to the point where the distinction between quantitative and qualitative research becomes totally meaningless. Forget you ever heard about it.
Experimental research
With these kinds of non-experimental research in mind, it’s easier to characterize experimental research. In this Research Methods course, the emphasis will be on experimental research so that you have a clearer idea of what you lose when you do non-experimental research.
Experimental research focuses on different, deeper questions than non-experimental research. Non-experimental research tends to focus on what’s happening, and experimental research focuses on the more theoretical or profound question of why or how it’s happening. So, when experimental research is a good option, researchers avoid non-experimental research.
Data collection in experimental research is more controlled. The key thing that makes experimental research different from other kinds of research is this fact:
The experimenter systematically controls a range of factors that might influence the observed outcomes.
Experimental control is essential to generate tightly focused or “uncontaminated” data that is immediately and directly relevant to the hypothesis under investigation.
The Method section has to make clear not only what the experimenter is studying, but also what is not being studied and how the experimenter is controlling the effects of these other factors.
Read this topic next: Choose the levels for your factors
About
© 2009 Created by Mike Dillinger, PhD on Ning. Create a Ning Network!
