I recently conducted a sociology experiment that I called the "Wallet Test"... I deliberately dropped 100 wallets in front of hidden cameras to test honesty. You can read the details on my website here: http://www.wallettest.com/

I've received many complaints from people saying that my experiment "...was not at all scientific." So could someone please tell me what qualifies as "Scientific" and how I could have made my test more so?

Thank you in advance for your help.
- Paul Kinsella

I'm not a scientist, and I didn't read the nitty gritty details of your scenarios. (it's been a long day and I'm a little tired.)

However I believe to be scientific study, you have to have two groups a controlled group and an uncontrolled group. The control group you dictate and control just about every aspect of the experiment. The second you just let nature takes it's course and record the consequences. Beyond that I'm not sure. Interesting proposal though.

suncrafter,

I'm not sure what you are fishing for, (validation or debate?), but here are my opinions on the scientific gravitas of the wallet test:

* You conducted a Naturalistic Observation experiment. This is valid science.

* You proposed no hypothesis, in order that the data you later collected might support (or not support) said hypothesis. This is bad science.

* Your sample size of 100 dropped wallet events was reasonable (if not a bit costly to the experimenter and/or his compatriots), for an experiment of this sort. Considering that the population of Belleville, IL (at the time of the last census in July of 2005) was 41,143 warm bodies, this was a credible sample size. Valid science.

* You videotaped each dropped wallet event, rather than simply collecting the data for each event on a standardized data-collection sheet. The use of this method would indicate that the experiment was predestined to be fodder for a website or web-forum, not a truly objective, scientific experiment. Questionable science.

* You neglected to subject your data to the usual (for a naturalist observation experiment of this sort) statistical Chi Square Analysis. Bad science. (I would crunch these numbers for you, using the Chi Square formula, were it not for the lack of a basic hypothesis. To do so would be analogous to 'the tail wagging the dog'.)

* Your assessment of 'honest' versus 'dishonest' is highly subjective. Cloaking the observed relative politeness behaviors in terms such as conscientiousness or considerateness, would better serve your purpose here. Bad science.

In summary: The dropped wallet experiment was an interesting and well-executed exercise, but it was hardly good science.

It's sociology. And similar to psychology, not a science anyways. So don't bother.

True.

But the original question was why people complained that the exercise was 'not at all scientific'.

Whether your experiment is in the arena of hard sciences (physics, chemistry, etc) or soft sciences (psychology, sociology, politics), certain scientific principles or methodologies, if applied, will render a result or conclusion that will hold water with an educated audience.

suncrafter and crew conducted an awesome experiment, but applied very few sound scientific protocols to the exercise, and as a result their conclusions are considered 'suspect'.

I myself thought it was pretty cool.
I myself have conducted the dropped wallet experiment, but only for my own amusement; On a few occasions over the years, I have found myself trapped in an airport, waiting for the next flight, and bored out of my skull.
They have these leather concessions in the concourses, and for just about $12 you can obtain a new wallet for yourself, and your old wallet now becomes the object of some serious sport.

I would leave the old wallet by a bank of pay phones, with just $2 in cash sticking part of the way out of it, along with a business card in the ID section. Then I would sit in the comfy chairs at my boarding gate, a discrete distance away, and watch to see how people reacted to the 'lost' wallet.
It's pretty funny, actually.

Some people stroll by, give it a furtive glance, but then continue on their way.
Some will make repeated passes by the wallet, and then nervously pick it up and examine the contents. Most will put it back, in hopes that the person who lost the wallet will come back to retrieve it.
One time, this airport security guy actually swooped on it, and quickly pocketed it. He kept his stride, and hauled ass right on out of the area. He clearly wanted the contents of the wallet, but didn't want to be seen rifling through it. Unbelievable.

Anyway, back to the original point; my 'airport wallet experiment' was not at all scientific, it was just a source of cheap entertainment for a bored traveller (me).

An interesting control group would have been to place wallets with no cash or other items of value, but still containing identification, to see if the reaction was different. Would the rate of wallets returned have changed?

Good point, sir.
There are dozens of ways in which you might spice up this experiment, in order to test any number of hypothesis.

suncrafter,
Here is a brief primer on the basics of the 'scientific method'.
This is not the be-all and end-all on the topic, but it's a good start:
http://teacher.pas.rochester.edu/phy_labs/AppendixE/AppendixE.html

I haven't read your study but I can think of a few basic things you need to validate a study:

A) You need to make your sampling of people as random as possible. For instance, a sociological study in areas of Illonois will probably have very different results from the same study conducted in New York, NY, so your study would not be representative of anywhere but the area where you sampled behavior.

B) You need to do a statistical analysis of your study....that means you'll need an idea of what the "statistical significance" of your study is before you can extrapolate the behavior of 100 people to represent the behavior of the entire nation or something of that nature. Generally, an acceptable level of statistical significance is 95%...that is you've calculated that your study is 95% likely to be representative of the sampled population.

C) You need to follow the scientific method (e.g. Form hypothesis, collect data, make conclusion, etc.) . It's rigid but effective.

suncrafter,

I'm not sure what you are fishing for, (validation or debate?), but here are my opinions on the scientific gravitas of the wallet test:

* You conducted a Naturalistic Observation experiment. This is valid science.

* You proposed no hypothesis, in order that the data you later collected might support (or not support) said hypothesis. This is bad science.

* Your sample size of 100 dropped wallet events was reasonable (if not a bit costly to the experimenter and/or his compatriots), for an experiment of this sort. Considering that the population of Belleville, IL (at the time of the last census in July of 2005) was 41,143 warm bodies, this was a credible sample size. Valid science.

* You videotaped each dropped wallet event, rather than simply collecting the data for each event on a standardized data-collection sheet. The use of this method would indicate that the experiment was predestined to be fodder for a website or web-forum, not a truly objective, scientific experiment. Questionable science.

* You neglected to subject your data to the usual (for a naturalist observation experiment of this sort) statistical Chi Square Analysis. Bad science. (I would crunch these numbers for you, using the Chi Square formula, were it not for the lack of a basic hypothesis. To do so would be analogous to 'the tail wagging the dog'.)

* Your assessment of 'honest' versus 'dishonest' is highly subjective. Cloaking the observed relative politeness behaviors in terms such as conscientiousness or considerateness, would better serve your purpose here. Bad science.

In summary: The dropped wallet experiment was an interesting and well-executed exercise, but it was hardly good science.

Not what I wanted to hear - but you speak the truth. :(
Thank you.

Your study would be determined bad science because you collected data and then subjected to analyses without any consideration of the hypotheses you may have been attempting to prove, or disprove.

You should first establish the hypothesis you wish to prove, or disprove. Remember, unproven is a possible result. You cannot assume that not proving the hypothesis, is the same as disproving it.
The sample size you chose must be capable of yielding statistically significant results. Further the sampling must be from a homogeneous population. That is the samples should be identical in all respects, except for the characteristics you are trying to determine. The number of variables involved will dramatically increase the sample size required. In your study, you used a sample of 100. Unfortunately, you then break into gender, race, age, etc. The sample sizes for these comparisons varied greatly and some were too small for statistically valid results.

Percentages are not useful in determining statistical significant results. e.g Take the old commercial where 7 out of 10 dentists recommend a specific toothpaste. This sounds like an endorsement. However, statistically it is meaningless and proves nothing.

Cheer up though, you are not alone. Many medical studies make exactly the same errors that you have. Some scientists also claim that some proponents of global warming have also made the same mistakes.

"You should first establish the hypothesis you wish to prove, or disprove." --Oltimer

Huh? Lots of experiments are done without any idea at all what is up. This is especially true in astronomy, where phenomena are observed and no one has any idea at all how to explain it, so all they can do is keep looking (the astronomical equivalent of an experiment).

I am also persuaded that a lot of published studies, where a hypothesis is framed and then demonstrated, is not really what happened. Instead they found that the study results demonstrate a certain hypothesis, so they tell the story as though that were what they were testing for all along.

I have a healthy disrespect for textbook writers who try to teach the "scientific method" as though it were a wonderful magical procedure.

Huh? Lots of experiments are done without any idea at all what is up. This is especially true in astronomy, where phenomena are observed and no one has any idea at all how to explain it, so all they can do is keep looking (the astronomical equivalent of an experiment).

Of course you can collect data or conduct experiments without establishing any hypothesis. However, if you want to prove something then you had better design the experiment correctly so that you collect valid data.

I am also persuaded that a lot of published studies, where a hypothesis is framed and then demonstrated, is not really what happened. Instead they found that the study results demonstrate a certain hypothesis, so they tell the story as though that were what they were testing for all along.

That's the real world. Studies cost time and money. If you do them, you publish, at least the ones you read are published. The same set of data can be used to prove one hypothesis or dis-prove another.

I have a healthy disrespect for textbook writers who try to teach the "scientific method" as though it were a wonderful magical procedure.

Scientific methods and sound math have a greater value than other methods.

"Scientific methods and sound math have a greater value than other methods." --Oldtimer

This strikes a raw nerve with me. The general public often thinks of "science" as something not far afield from magical practice--such as people accusing others of "worshipping science."

Part of this confusion comes, I think, from the scientist himself--especially from the philosopher of science--who parades about this special "method"--"the scientific method" about which every science textbook in high school has a chapter.

The reality is that science is not anything special. It is just organized (institutes and universities and peer reviewed periodicals and so on).

When you look out your window and see dark clouds and decide to take an umbrella, thinking "It's going to rain," you have done "science."

My point is that it is not so special--it would fall under the category "common sense" except it has a limited purview, and its pratitioners self-limit themselves further (specialize) out of practical necessity.

That makes them experts, and science just the consensus of the experts.

Unfortunately, common sense is not so common.
However, consider the following. You toss a coin 10 times. 7 times it turns up heads, 3 times it turns up tails. That is it turns up heads more than twice the number of times it turns up tails. What conclusions can you draw?

Well, of course, the obvious response is that ten tries at a two-outcome result is not enough to make much of a conclusion--and here is where your math comes in to help you understand and express how strong a conclusion can be drawn from a certain number of trials.

If you are limited to just ten tries, the conclusion you would draw would be that both outcomes are possible and there appears to be a bias in the coin (since the outcome you postulate is unlikely.

This brings me to theoretical considerations; often one has theoretical reason for expecting a certain result--in the case of coins we expect 50-50 results--but even a billion tosses does not prove any such thing.

Theoretical prediction can be supported empirically, but not proved. (It cannot be disproved either, although it can be strongly weakened--enough to send you out looking for a new theory).

How right you are that a billion observations may not prove anything. You're almost getting me going on a rant that math, as taught in schools, is wrong. It's just useful for everyday tasks. Suffice to say that we taught, and most believe, that life, math included, is deterministic. In fact, it isn't, it's probabilistic. The secret is knowing which probabilities are close to unity.
Regarding my question on the coin. You're correct, except that you cannot draw any conclusions from the test. If the coin were unbiased, you would theoretically expect 5 heads 5 tails. You therefore have to compare 7 with 5, if you are concerned with heads, or 3 with 5 for tails.
If you do the math, there is no difference of any significance between the observed result and the theoretical result. The coin may be biased, or it may not.

Of course you can collect data or conduct experiments without establishing any hypothesis. However, if you want to prove something then you had better design the experiment correctly so that you collect valid data.

But that was it. I was not out to "prove" anything. I was just after the truth - what ever that might have been.

Actually it's quite likely, as you say do the math.
Probability, statistics and the associated math are so mis-understood and mis-used that it is almost comical.

Actually it's quite likely, as you say do the math.
Probability, statistics and the associated math are so mis-understood and mis-used that it is almost comical.

That's for sure.