Prove the scientific method works

March 18, 2010 7:29 PM

How do I prove that the scientific method works?

I've been engaged in an argument with someone who appears to put little confidence in science. In the latest instalment, I was asked for proof that peer-reviewed scientific studies are more reliable sources of information than anecdotally-based assertions. This is something that has always just seemed obvious to me, so I've taken it for granted. It made me realize that I had never really thought about providing evidence that the scientific method is a reliable tool for testing hypotheses. So, I'm wondering if there are any resources out there that provide some sort of "proof" or explanation about just why the scientific method works?

Also, I realize the limits to peer-reviewed studies. I know that there are times when biases, conflicts of interest, or even fraud can call the research into question. I'm more interested in any information about anyone trying to prove that the scientific method works, or proving that it is more reliable than anecdotal evidence.

posted by lexicakes to science & nature (32 answers total)

Rather than "prove the scientific method," you need to explore the concepts of falsifiability and the idea of a "crucial experiment," which are the only things the scientific method can really do.
posted by Cool Papa Bell at 7:34 PM on March 18, 2010

The problem of induction should dissuade you from looking for a proof of the scientific method. Its astonishing practical success should inspire a great deal of faith in it.
posted by drdanger at 7:49 PM on March 18, 2010 [7 favorites]

First, does your friend mean that peer-reviewed scientific studies aren't reliable in the same way anecdotes are unreliable? Or does your friend believe anecdotes are actually reliable?
posted by Green With You at 7:49 PM on March 18, 2010

The only reason we use the scientific method is because it has worked for us in the past. There's no fundamental metaphysical or ontological reason why it should work. Our reliance on it is purely utilitarian.
posted by mr_roboto at 7:52 PM on March 18, 2010 [3 favorites]

How do I prove that the scientific method works?

Generally, a picture of an atomic explosion is all you need. It doesn't work? Really?

Get him to admit it works some of the time.
posted by Ironmouth at 7:53 PM on March 18, 2010

Your friend needs to understand that peer review isn't a part of the scientific method, nor is it a scientific process: it's an editorial process, designed to weed out things that are obviously wrong or poorly done. It doesn't add to knowledge.
posted by Fiasco da Gama at 7:53 PM on March 18, 2010 [2 favorites]

anecdotal-based assertions tend to try and apply individual specifics to to broad categories.

This is an ok place to start.

The truly powerful part of the "scientific method" is that new data can replace old data if new data is shown to explain things more accurately, it is a continual process of refinement and exploration and has at it's basis everlasting curiosity. Anecdotal assertions, while sometimes useful, have little weight behind them and often times seem absolute, authoritative and incurious.

I suspect your friend may have a flawed view of the basic nature of the scientific process. If he is open to it, Bill Bryson's A Short History of Nearly Everything may be a good read for him. It is un-stuffy and very approachable.
posted by edgeways at 7:56 PM on March 18, 2010 [2 favorites]

Peer-reviewed studies are not the same thing as the scientific method. The scientific method, in its simplest version is: predict, test, refine. After you've done this a lot, you submit your paper describing your success (or failure) at this process for peer review.

There are valid critiques of the scientific paper publishing process as it stands today, but this is completely removed from the idea of the scientific method. The best way to believe in the method is to try it. Find something counter intuitive and then prove that it is so through experimentation.
posted by jeffamaphone at 7:58 PM on March 18, 2010 [3 favorites]

First, does your friend mean that peer-reviewed scientific studies aren't reliable in the same way anecdotes are unreliable? Or does your friend believe anecdotes are actually reliable?

This person does not think that scientific studies are reliable, and presents counter arguments based on anecdotes or information from "trusted" sources, but which isn't based on any scientific evidence.
posted by lexicakes at 8:06 PM on March 18, 2010

Tell this person you're going to make them write a paper asserting that anecdotes are reliable. Then they have to post it to a prominent online forum (Slashdot, Metafilter, Fark, whatever they're most familiar with) and ask that people respond with their feedback regarding the paper's methods, data, assumptions, etc.

If everyone on that forum is in agreement that the paper passes muster, then the paper stands. If not, then it's back to the drawing board!
posted by ErikaB at 8:07 PM on March 18, 2010

In light of your response, I also want to recommend this wikipedia article on the Availability heuristic which someone posted here recently. It's fascinating, and also relevant to your situation!
posted by ErikaB at 8:08 PM on March 18, 2010

The problem of induction should dissuade you from looking for a proof of the scientific method. Its astonishing practical success should inspire a great deal of faith in it.

Yeah, this (and similar comments). Science is based on empiricism and being able to reproduce with a certain amount of confidence certain circumstances and events. Science is pragmatic.

I could understand if your friend was very religious, and had a problem with science not being able to offer 'deep truths' about existence. But the whole 'anecdotes are just as good' is pretty ridiculous. The reason science has been so successful (it works! bitches!) is because it isolates factors - which is the very thing that gives us the empirical insights that have allowed us to, I dunno, eradicate small pox, land on the moon...

In fact, it is this delimiting - this isolating - this artifice, the exact opposite of an anecdote, that gives science it's balls. It makes things repeatable and observable. If anecdotes were just as good, we'd still think that bushes can suddenly burst into flames and staffs can turn into serpents at random.
posted by Lutoslawski at 8:09 PM on March 18, 2010 [4 favorites]

You don't have to convince anyone of anything. Hopefully, the descendants of your friends and family will outlive the descendants of anyone who fervently disbelieves in the scientific method.
posted by ovvl at 8:10 PM on March 18, 2010 [3 favorites]

When an experiment is performed, it usually confirms (or disconfirms) a hypothesis by producing a specific sample of empirical data. Conclusions can be drawn about the populations represented by the sample within some confidence level. For instance, a study of 1000 men may show that men who do X will experience Y with Z% greater frequency. This can be construed to assert that X is correlated with Y for men in general.

Peer review, as well as independent reporductions experiments, can help ensure that the assumptions, sampling, control procedure, etc. were done properly, thus helping us gain confidence in the assertions drawn from the experiment. You can tell a real scientist by how (s)he hedges and qualifies the results, being very careful that the proper conclusions are being made. Scientists are often frustrated with breathless science reporting that runs with the conclusions and drops the qualifications. That and the real prospect of bias may be the source of your friend's skepticism.

That said, a simple counterexample like "Well, I know Fred and he did X all his life and never experienced Y." in no way disproves the experiment. One or a few counterexamples can only refute a universal assertion that X causes Y for ALL men, which is never found in science.
posted by cross_impact at 8:31 PM on March 18, 2010

For understanding why the scientific process (meaning the way the method is practiced and institutionalize works, as opposed to method) works, you might be interested in reading a little about Merton's norms of science. This whole thing we've built around science works because we socialize scientists in these norms.
posted by If only I had a penguin... at 8:31 PM on March 18, 2010 [2 favorites]

Good on you for actually having this conversation with your skeptical friend. I wouldn't even know where to begin with someone like that.
posted by Alt F4 at 8:37 PM on March 18, 2010 [1 favorite]

This person does not think that scientific studies are reliable, and presents counter arguments based on anecdotes or information from "trusted" sources, but which isn't based on any scientific evidence.

Um, not all conclusions drawn from scientific studies are reliable, or created equal. The fact that an article uses a specific methodology or is peer reviewed is no guarantee that the explanation that it posits for an event is worth anything. What kind of "science" are you talking about? Are you looking at measures of variance explained, or just whether the scientific method was followed? Because in some areas of study, the explanation provided by the analysis is about as good as flipping a coin to decide whether you think that something works, or is true. And some scientific conventions about hypothesis testing are no more than meaningless traditions that we keep adhering to for no quantifiable reason.

Not to rain on the "hurray science" parade, but new ideas and explanations often seemed to be based on an intuitive leap of imagination rather than a particular set of methodology. It seems like Einstein did that, Newton did it, and other people who were less famous do this all the time. Fortunately for their ability to be taken seriously, it was possible to test their notions about how the world worked through observation and it turned out that their ideas had merit. Of course, now we understand that their theories were flawed--no matter that they used to be considered the absolute truth.

In the physical sciences, the scientific method is pretty awesome tool for explanations/ predictions. Once you get into studying things that are more complex than blind matter, it gets a lot more complicated (and results get less trustworthy). Hell, qualitative research was established with the notion that you can't get to some truths using traditional quantitative methods. Of course, if what you're talking about is actually something quantitatively observable, then nthing the suggestion to look at falsifiability. I love Popper.

Not saying that I think your friend is right, but healthy skepticism is a good thing. Speaking personally, I'm deeply troubled by what I feel is an unexamined obedience to authority that carries the name "science" by many of my peers.

Does this person believe in skepticism, and are they interested in skeptically examining their ideas? Because fair is fair--if science has skeptics built in, then should "truths" that come from anecdotes be opened to question? Are you seriously interested in questioning your own notions? Because if not, it seems like you are both wasting your breath.
posted by _cave at 9:20 PM on March 18, 2010 [4 favorites]

Does this person believe in skepticism, and are they interested in skeptically examining their ideas? Because fair is fair--if science has skeptics built in, then should "truths" that come from anecdotes be opened to question? Are you seriously interested in questioning your own notions? Because if not, it seems like you are both wasting your breath.

Thanks for raising this point. I was sort of avoiding giving specific details about the argument, because it's something that's come up on the blue and there has been plenty of debate about this topic. It isn't that I don't want my views challenged, but rather that I don't want this question to turn into a debate about the topic of the argument, rather than the question at hand.

Here's what I can say. The argument is about a matter of physical science. By that, I mean that it isn't some impossible to falsify question like whether the tooth fairy exists or there is a dragon in my garage. The other person has presented points that I know to be completely false, or that I can point to as being at least scientifically implausible, if not outright false. I actually consider myself to be the skeptic in the debate. I am aware that there are reasonable arguments that are supported by evidence on both sides. The thing that baffles me is that she refuses to use any scientific studies to back up her argument, and instead relies on "information" from sources that she trusts, but happen to be completely wrong. I would rather have a debate in which both sides present and examine the evidence, but this person has a fundamental distrust of the way that scientific studies are conducted.

I guess this comment is longer than I meant to make it. I am not really interested in winning this particular argument, because I don't think that it can really continue with someone who refuses to examine evidence. To answer your question, I am interested in questioning my notions (although the other person is clearly not). It has just occurred to me that I haven't questioned my notions about science. I am trying to avoid falling into thinking that science is somehow infallible, which is why I'm wondering about what sort of proof there is that the scientific method does what it's supposed to do.
posted by lexicakes at 10:14 PM on March 18, 2010

this person has a fundamental distrust of the way that scientific studies are conducted

I think this whole thing hinges on what those objections are.

Here's the thing. People rarely reject "science" as a whole. Instead, they reject particular conclusions that don't seem right to them for whatever reason. And when they reject it, they usually conclude that science has broken down.

Take, for instance, global warming. From what I can tell, people who reject global warming -- at least the majority of them -- aren't saying "Science doesn't work! You can't draw conclusions by forming hypotheses and testing them rigorously!" They're saying "Science, in this particular case, has broken down. The conclusions have been tainted by politics and groupthink, and the truth is being suppressed for unscientific reasons" or something like that. Even creationists convince themselves that they're more scientific than actual scientists.

So proving that Science Works isn't going to help. It's like proving that aeronautics works to someone who's convinced there's a gremlin on the wing. They're not talking about air travel as a whole, they're talking about this particular airplane.
posted by lore at 11:20 PM on March 18, 2010 [4 favorites]

Unfortunately the scientific method cannot itself be proven. However, people using the scientific method often get better results then if they don't.
posted by delmoi at 11:47 PM on March 18, 2010

I think the role of Falsification is also an important aspect of the scientific method. Science wants to prove things false and thus get rid of bad data and methods and make the evidence for an hypothesis better as a result.
posted by occidental at 11:53 PM on March 18, 2010

Actually, there is one way you could show the scientific method is reliable: Create a virtual, rules-based system on a computer (like, for example, a cellular automata world) and then use the scientific method to determine facts about that imaginary world. The game of life, which people do research in as a hobby would be an example. You can't just "do the math" to figure out "everything" about the world, because it would take too much time computationally.

But, if they were smart enough to understand that argument, they would probably already accept the scientific method.
posted by delmoi at 11:59 PM on March 18, 2010

To be frank, I would have likely stopped talking to this person long ago, because people like this either do not understand science, do not want to understand science, or do not want to stop arguing. Either way they will never be convinced. They will keep offering new ways out of the argument, or changing its parameters slightly, in order to prolong the affair. "But what if X?" or "But couldn't Y?".

If this is getting on your nerves, then stop it. Require unmitigated evidence from him. Make him back up his extraordinary claims with extraordinary evidence. The beautiful thing about the scientific method is that you can use it to find a superior approach to the scientific method. Does he have a better method? We would all like to hear it.

I've been engaged in an argument with someone who appears to put little confidence in science...I was asked for proof that peer-reviewed scientific studies are more reliable sources of information than anecdotally-based assertions.

No problem. First, you must wait until he gets sick. Something gruesome, preferably, not a common cold. When this happens tell him that, anecdotally, you've heard M&Ms will cure that condition right up. Tell him your aunt and this guy you met on a bus also think the same thing. Tell him that he'll get better right away if he munches a big bag of M&Ms. Skittles will also work. Use terms like "I swear!". Also have to tell him that if he's taking any scientifically-developed medication, he has to stop that right away. It'll clog up the M&Ms and they won't work.

If he takes the M&Ms, you will know the argument is over and you have lost because he truly is an idiot.

If he takes the medication, you will know the argument is over and you have lost because you spent so much time trying to convince someone who lacks the courage of their convictions.
posted by Sutekh at 12:19 AM on March 19, 2010 [4 favorites]

Ah, you want to look at the Cosmic Background Explorer mission. It looked at the background microwave glow of the universe and found that it fit perfectly with the idea that the universe used to be really hot everywhere. This strongly reinforced the Big Bang theory and was one of the most dramatic examples of an experiment agreeing with a theory in history -- the data points fit perfectly, with error bars too small to draw on the graph. It's one of the most triumphant scientific results in history.

There's even a t-shirt.
posted by Mwongozi at 1:12 AM on March 19, 2010

This person does not think that scientific studies are reliable, and presents counter arguments based on anecdotes or information from "trusted" sources, but which isn't based on any scientific evidence.

A key weakness of relying on anecdotes is when they contradict one another. For example, if someone says "My grandfather smoked, and he died at age 50, therefore smoking kills" I could reply with "My grandfather smoked, and he lived to 120, therefore smoking prolongs life". I could then point out that, if anecdotes were sufficient to prove something true, we would have proved two contradictory things; hence, anecdotes are not sufficient to prove something true.

I once heard this called "duelling anecdotes" but I'm not sure if that's a common term.

The thing that baffles me is that she refuses to use any scientific studies to back up her argument, [...] this person has a fundamental distrust of the way that scientific studies are conducted.

For me, a key feature of scientific papers is that they state the details that more popular sources might gloss over, giving readers the ability to determine the meaningfulness of their findings.

For example, a paper about a study of smoking would tell you the sample size, how the sample population was selected, what confounding factors were controlled for and how, the results and their statistical properties, and how the results compared to a control group.

In other words, given that I think it's easy to make mistakes, I am most inclined to trust people who state they know about and have accounted for all the mistakes I can think of.
posted by Mike1024 at 1:52 AM on March 19, 2010 [3 favorites]

You want a proof of proofs? Not going to happen. Sorry.
posted by koeselitz at 4:16 AM on March 19, 2010

To give a slightly more lengthy answer: in what might be the greatest work ever written by one of the most logical and rational human beings who ever existed - Plato - the Athenian Stranger makes an interesting remark in passing on the subject of witches and conjurers (whom he says should be expelled from the city and disallowed from taking people in with their shams). He of course dismisses them and says that they have to be sent away to keep the people from falling for their gimmicks, but then he notes: isn't it strange how very hard it is to prove that magic doesn't exist?

I think this runs to the very heart of Plato's doctrine, and of human rationalism. Those of us who seek the truth about the world through reason do so because we believe that it can be so found, and those who seek it elsewhere believe that it cannot. This is, I think, why Socrates talked about philosophy rather than sophy – about being a friend of wisdom rather than wise – because, to even this, there has not yet been an answer. And would that there could be, because it would clear up a good number of difficult points. The fact that Socrates couldn't do it doesn't mean it can't be done; but I've never read a proof that made sense that demonstrated to me that human rationality is equal to the task of comprehending the universe - which, of course, you would have to prove in order to prove that the scientific method works. (For what it's worth, the most convincing argument for this that I've read is in Aristotle's short text On The Soul.)

The difficulty remains. The possibility is always that there is no reason for things that happen, no explanation. 250 years ago, David Hume pointed out the unfortunate fact that we can have no certainty of the connection of cause and effect – that all we've ever had is the experience that some things happen after other things happen, and a blind, evidence-less assumption lies at the heart of our belief that events in the world have causes. But (though I'm almost certain he didn't know it) this had already been pointed out 700 years before him, by a Sufi polemicist and thinker of great stature called Abu Hamid Muhammad Al-Ghazali in his great work The Incoherence of the Philosophers, a diatribe against the schools of philosophy then springing up within the Islamic world.

Anyway: Al-Ghazali pointed out there is in fact no answer whatsoever to have faith in any object even within the sphere of our experience. Why, he asked, does this book not become a donkey? Why does the house not turn into a sandal? There is no thread connecting the cause with the effect, there is only our thinking about cause and effect – nothing real that we can point to or know to be the link between an event and its cause. So why, asks Al-Ghazali, is the world so? Why do things seem so to make sense, such that I am not surprised when the book does not turn into a donkey? Al-Ghazali's answer is: this can be nothing else but a gift from God which must be utilized at every opportunity, the gift of making sense of the world. To Al-Ghazali, causality is the end of human reason, and makes sense only as a kind of divine law.

Of course, The Incoherence of the Philosophers starts with a preface that points out that the arguments are intended to refute and not to state the truth, so Al-Ghazali may be arguing rhetorically there. But I prefer to see the world this way: causality is a gift from beyond, the provenance of which I cannot ken, but which makes life as a human being bearable and even divine itself.
posted by koeselitz at 4:44 AM on March 19, 2010 [9 favorites]

lexicakes: “The thing that baffles me is that she refuses to use any scientific studies to back up her argument, and instead relies on "information" from sources that she trusts, but happen to be completely wrong. I would rather have a debate in which both sides present and examine the evidence, but this person has a fundamental distrust of the way that scientific studies are conducted.”

Ah. (Sorry about that treatise there, then; it's clearly not what you're looking for at all.)

So, as far as I can tell, you've misunderstood her position completely. She doesn't have a fundamental distrust of science; she has a fundamental distrust of some particular scientists. It is hardly likely that this is an existential doubt she has about the validity of science as a whole; she merely misunderstands the science in this circumstance. And, to be fair, you have to accept the possibility that she has a point; it could be that the scientific studies done on the subject have been done unscientifically. This has happened before in the past, and could happen again in the future.

It sounds like you've both been beating your heads against a wall. She insists on one point of view, which she backs up with anecdotal evidence from sources she trusts; you insist on another point of view, which you back up with scientific studies which you trust. You say you know that her sources are incorrect, but it appears that the reason you know this is because the scientific evidence on your side is so overwhelming. She will not accept your scientific evidence. In fact, she will probably never accept your scientific evidence if you simply continue to present it positively; there will always be room for doubt there. The goal is to convince her that there is more doubt in her sources than in yours.

The best plan in this situation, then, is to quit trying to point out that your scientific studies are valid. Forget about them altogether; simply remember what they taught you about proper method. What you should do is interrogate her sources, the places she got her information, the way it was obtained by the people offering anecdotal evidence. If she is incorrect, then at some point someone made a mistake. Mistakes aren't evil, and you're not attacking anyone, but you have to find where the mistake was made in order to show her why her position isn't valid.

Focus on her position; that way, you can bring her position down to your level in her eyes.
posted by koeselitz at 4:56 AM on March 19, 2010 [2 favorites]

I think koeselitz has it - you can't convince her that your sources are correct as they contradict her sources that she has 100% trust in. You must instead convince her that her sources are not 100% trustworthy (find out how they came to the conclusions they did, whether their evidence/methods hold up to scrutiny, etc), and then later when that is established, demonstrate why your sources do not suffer from the inadequacies that hers do (i.e. the evidence is the result of repeatable experiments, has been verified by independent sources, etc).
posted by EndsOfInvention at 5:56 AM on March 19, 2010

I read an article recently (maybe linked on the blue) about climate change deniers, where Monbiot asks the question "What would it take to persuade you ?". It's a good place to start, as it makes the person you are discussing/arguing with either state that they will not accept any evidence to persuade them, in which case you are wasting your time (and can probably state that), or you can start building towards their acceptable refutation. It may also help if you state you amenability to change, and what would challenge your assumptions. This of course, mirrors the scientific method itself, by proposing a 'falsifiability' test.
posted by doozer_ex_machina at 7:14 AM on March 19, 2010

I am trying to avoid falling into thinking that science is somehow infallible, which is why I'm wondering about what sort of proof there is that the scientific method does what it's supposed to do.

I see. Go with falsifiability, then. The strength of the scientific method is that it:
1. Ties the claim maker to a specific statement about the nature of reality.
2. Proposes a specific test to probe the nature of reality.
3. *Provides a way to reject demonstrably false statements*

Most scientists more or less adhere to the philosophy of post-positivism, which, coincidentally, is another notion of Popper's. What is exciting about this idea is that it limits the claims that science can make about the nature of reality--that instead of claiming to establish facts, science can instead be viewed as probing reality for durable truth that is a good enough explanation for how the world works until the next test allows us to understand reality somewhat better.

To put it another way, scientists aren't all that much different from the people who produce the anecdotes that your friend places trust in. Mostly, they agree that they begin with a set of assumptions about the world based on their culture, beliefs, and history and norms of their discipline. The difference is that they go to great lengths to test their assumptions--that is, the scientific method. This method isn't important because it produces facts, but rather, because it creates an environment where ideas can be tested while limiting the effects of human assumptions about reality on the results of the test.
Hopefully.

The other posts that talk about the material benefits that have been produced by science give evidence of the value of notions developed by specific scientific disciplines, but science doesn't really prove anything, as drdanger already noted.
posted by _cave at 10:53 AM on March 19, 2010

Thanks for all the comments. This was exactly what I was looking for.
posted by lexicakes at 8:07 PM on March 19, 2010