How do education researchers design experiments?
February 6, 2011 10:13 PM Subscribe
How do education researchers design experiments?
I'm looking for an overview or introduction to how educational research experiments are designed and executed. Any guidelines, standards, book recommendations, how-to's, PDFs, or papers on education research would be greatly appreciated. I (think) I understand how physical science research gets done, please help me understand the basics of what goes into educational research.
I have access to most journals online, so feel free to link to any.
I'm looking for an overview or introduction to how educational research experiments are designed and executed. Any guidelines, standards, book recommendations, how-to's, PDFs, or papers on education research would be greatly appreciated. I (think) I understand how physical science research gets done, please help me understand the basics of what goes into educational research.
I have access to most journals online, so feel free to link to any.
There are a board number of methodologies. The biggest factor is what level of detail you're studying at. Studying how a particular idea in science or mathematics develops, for example, usually requires intensive interviews of a very small number of people, while a study on how socio-economic factors affect school achievement is usually done on a large scale of many schools, and is primarily statistical.
For a well written and detailed set of instructions of one of the many case study methodologies in education, I highly recommend Steffe and Thompson 2000. In fact, the entire book that it comes from is quite excellent.
As a more general comment, education research is frequently aimed toward developing a working product. As such, it more closely resembles engineering research methodologies such as design research than it does research in the physical sciences.
posted by yeolcoatl at 11:19 PM on February 6, 2011
For a well written and detailed set of instructions of one of the many case study methodologies in education, I highly recommend Steffe and Thompson 2000. In fact, the entire book that it comes from is quite excellent.
As a more general comment, education research is frequently aimed toward developing a working product. As such, it more closely resembles engineering research methodologies such as design research than it does research in the physical sciences.
posted by yeolcoatl at 11:19 PM on February 6, 2011
More thoughts.
Another commonly used methodology for qualitative research is Strauss and Corbin's grounded theory. I'm focusing on qualitative methods and avoiding giving you specifics on statistical methods both because I think they're boring, and because statistical methods don't change much from field to field.
Treatment/Control is another popular methodology and one that brings in a lot of money, but I personally don't support its use. Feel free to ask me more privately if you care about why.
posted by yeolcoatl at 11:26 PM on February 6, 2011
Another commonly used methodology for qualitative research is Strauss and Corbin's grounded theory. I'm focusing on qualitative methods and avoiding giving you specifics on statistical methods both because I think they're boring, and because statistical methods don't change much from field to field.
Treatment/Control is another popular methodology and one that brings in a lot of money, but I personally don't support its use. Feel free to ask me more privately if you care about why.
posted by yeolcoatl at 11:26 PM on February 6, 2011
Response by poster: Felex, thanks for the response; I had the feeling the wording of my question gave away my ignorance. Definitely too broad of a question.
To be more specific, I was interested in reading about how to conduct trails comparing the effectiveness of slightly different learning situations. Something like: preteset, Class A is assigned one type of math problem, Class B is assigned another type, test again. Or class A begins each day with warm-up exercise, B does not, ect.
"Academic papers on research describe the process in detail most of the time."
I've read a few, and the procedures seem straightforward for these types of experiments, but I don't know if I'm missing something.
I had read the Wikipedia article on educational research, but forgot to check out the citations- I'll do that now.
posted by SouthCNorthNY at 11:54 PM on February 6, 2011
To be more specific, I was interested in reading about how to conduct trails comparing the effectiveness of slightly different learning situations. Something like: preteset, Class A is assigned one type of math problem, Class B is assigned another type, test again. Or class A begins each day with warm-up exercise, B does not, ect.
"Academic papers on research describe the process in detail most of the time."
I've read a few, and the procedures seem straightforward for these types of experiments, but I don't know if I'm missing something.
I had read the Wikipedia article on educational research, but forgot to check out the citations- I'll do that now.
posted by SouthCNorthNY at 11:54 PM on February 6, 2011
One place for standards would be the U.S. Department of Education's What Works Clearinghouse. They have a procedures and standards handbook that details the different standards. More recently they added standards for single case design and regression discontinuity design.
posted by statsgirl at 3:48 AM on February 7, 2011
posted by statsgirl at 3:48 AM on February 7, 2011
Education is not in and of itself an academic discipline. Education researchers have deep grounding in some other discipline and use the research methods common to that discipline to study education. What you're describing would probably fit best in educational psychology so you should look for research methods in educational psychology.
posted by mareli at 5:16 AM on February 7, 2011
posted by mareli at 5:16 AM on February 7, 2011
Yep you're looking into treatment control methodology and there really isn't much more too it. I would recommend something like a teaching experiment instead. It can fill the same role, but gives you much more insight into the mechanisms that cause the differences. For example, Treatment control can tell you that Type A math problem is better, while a teaching experiment can tell you *why* type A math problems are better, how students approach type A problems, how to improve on type A problems, what exactly the boundary cases of a Type A problem are, and much more.
posted by yeolcoatl at 5:46 AM on February 7, 2011
posted by yeolcoatl at 5:46 AM on February 7, 2011
Another excellent paper for you would be Flyvbjerg's Five Misunderstandings About Case-Study Research
posted by yeolcoatl at 5:51 AM on February 7, 2011 [1 favorite]
posted by yeolcoatl at 5:51 AM on February 7, 2011 [1 favorite]
In other words, you're not missing anything about treatment/control studies. Those really are that straight forward. However you are missing out on a whole hosts of alternatives that are a significant part of the education research, and some of which may better answer questions that you're interested in.
posted by yeolcoatl at 5:53 AM on February 7, 2011
posted by yeolcoatl at 5:53 AM on February 7, 2011
I was under the impression that quasi-experimental research (research with a limited capacity to isolate cause and effect) was common in education contexts. Here's a brief article that may serve as a primer.
posted by Nomyte at 6:28 AM on February 7, 2011
posted by Nomyte at 6:28 AM on February 7, 2011
I second that a good chapter on quasi-experimental research might be worthwhile. Add to that some discussion of threats to internal validity that have to be considered with that type of research.
posted by bizzyb at 6:40 AM on February 7, 2011 [1 favorite]
posted by bizzyb at 6:40 AM on February 7, 2011 [1 favorite]
Two more things
Data Mining is also gaining popularity. at first glance it may not seem like an experimental methodology, but when applied to an experimental class or experimental curriculum it becomes one. Sorry I don't have a good reference for that, I'm just beginning to learn data mining myself. It is the blackest of black magics. In that it seems really questionable,but also produces valuable results.
Two:
It may also be helpful to make a distinction between theory-building experimental methodologies and theory-testing methodologies. You said you're familiar with the physical sciences so I'll use examples from there.
Theory-building experiments are experiments that are designed to generate good hypotheses on the basis of a theoretical framework. Theory-building experiments are frequently case studies. Examples include Einstein's gedankenexeriments and early experiments in gravity such as the inclined plane experiment. Other 'black magic' techniques like data mining also produce hypotheses. These methodologies are free to be less rigorous because they're aimed at creating things for testing, not determining the truth of anything, but this does not make them less valuable. You need a good hypothesis to test before you can start testing hypotheses, and good hypotheses don't come from nowhere.
Theory-testing experiments are experiments that test hypotheses generated from theories in order to test the validity of the theory. These are frequently treatment/control experiments such as medical trials, but can also be case studies (the search for the higgs boson), or simulations.
There are also methodologies that do both theory-building and theory-testing, like the design research cycle, which aims to improve a theory by finding flaws in prototype designs based on that theory.
All of these methodologies have a place in the physical sciences, and similarly, all of these methodologies have a place in education research. There's no reason to believe that one way of doing research is sufficient in any field. Studying experimental research methodology is not a matter of 'do it like this' but more a matter of finding the right fit for the specific problem that you have.
posted by yeolcoatl at 7:57 AM on February 7, 2011
Data Mining is also gaining popularity. at first glance it may not seem like an experimental methodology, but when applied to an experimental class or experimental curriculum it becomes one. Sorry I don't have a good reference for that, I'm just beginning to learn data mining myself. It is the blackest of black magics. In that it seems really questionable,but also produces valuable results.
Two:
It may also be helpful to make a distinction between theory-building experimental methodologies and theory-testing methodologies. You said you're familiar with the physical sciences so I'll use examples from there.
Theory-building experiments are experiments that are designed to generate good hypotheses on the basis of a theoretical framework. Theory-building experiments are frequently case studies. Examples include Einstein's gedankenexeriments and early experiments in gravity such as the inclined plane experiment. Other 'black magic' techniques like data mining also produce hypotheses. These methodologies are free to be less rigorous because they're aimed at creating things for testing, not determining the truth of anything, but this does not make them less valuable. You need a good hypothesis to test before you can start testing hypotheses, and good hypotheses don't come from nowhere.
Theory-testing experiments are experiments that test hypotheses generated from theories in order to test the validity of the theory. These are frequently treatment/control experiments such as medical trials, but can also be case studies (the search for the higgs boson), or simulations.
There are also methodologies that do both theory-building and theory-testing, like the design research cycle, which aims to improve a theory by finding flaws in prototype designs based on that theory.
All of these methodologies have a place in the physical sciences, and similarly, all of these methodologies have a place in education research. There's no reason to believe that one way of doing research is sufficient in any field. Studying experimental research methodology is not a matter of 'do it like this' but more a matter of finding the right fit for the specific problem that you have.
posted by yeolcoatl at 7:57 AM on February 7, 2011
There are books on doing ed research in various specific fields; e. g. Second Language Research: Methodology and Design is on my Amazon wishlist (for ESL-teaching research), and there are a dozen more related books. A university library should have these books, if you have access to one. You can also google "action research" + (subject name) to read about one style of quasi-experimental research.
You might be able to find some overviews of research methods meant for new grad students if you search scholar.google.com for (subject name) and (education "research methods"). I found several promising results when I tried that just now for math.
posted by wintersweet at 10:17 AM on February 7, 2011
You might be able to find some overviews of research methods meant for new grad students if you search scholar.google.com for (subject name) and (education "research methods"). I found several promising results when I tried that just now for math.
posted by wintersweet at 10:17 AM on February 7, 2011
Educators usually engage in action research. It can be applied in many school contexts.
posted by tamitang at 6:47 PM on February 7, 2011
posted by tamitang at 6:47 PM on February 7, 2011
Response by poster: Huge thanks to everybody who responded. I have a lot of reading to do!
posted by SouthCNorthNY at 6:37 AM on February 8, 2011
posted by SouthCNorthNY at 6:37 AM on February 8, 2011
This thread is closed to new comments.
In order for any answer to be meaningful you should specify what kind of research you are looking for, conducting empirical research looks really different from normative theoretical work. I would suggest looking through the wiki on educational research and deciding what direction you want to look into. From there I would look for top papers in the field you are looking at.
As you must be a student (access to journals) look through the database of education journals and see what fits the bill. Academic papers on research describe the process in detail most of the time.
I know many people who have a PHD in a very small portion of that question. They could not answer the question as you have posted it here.
posted by Felex at 10:55 PM on February 6, 2011