Who Knows and Who's Guessing?

Suppose we take 128 people from all over the world and we give them a quiz with 7 true-false questions.

If all 128 people answer all 7 questions correctly, we would find it hard to imagine that all, or even most, of them guessed their way to the correct answers. It would be more likely that all of them found little difficulty in answering the questions correctly.

Similarly, if all 128 people answer all 7 questions incorrectly, we must suppose that none of them knew any of the answers.

Things get trickier if the answers come out in a bell curve, that is: 2 people answered either all 7 questions correctly or none correctly, 14 answered either 6 or 1 questions correctly, 42 answered 5 or 2 correctly, and 70 answered either 4 or 3 correctly. We could draw 1 of 2 conclusions from this: either the sample of people we chose represents a perfect sampling of knowledge vs ignorance, such that they were equally likely as not to know the answers, or that all of them chose answers at random and the result was a matter of pure luck.

There are some odd things to note about this. The first is how the results of some participants affect the evaluation of how well other, seemingly unrelated participants performed. If you got 6 answers correct under the bell curve scenario, it is impossible to know whether you know anything. You might have guessed on all of them. However, if it turns out that that guy from Nepal, who we originally thought answered only 1 question correctly actually answered all 7 correctly, the likelihood that you answered correctly based on actual knowledge rather than by random guessing goes up.

Then there is the problem of just which questions were the ones that were not answered correctly. Take question 1. If everyone got it right, the odds are that you knew the answer. If half got it right and half got it wrong, the odds are that you and everyone guessed. If you actually knew the answer, the distribution would, mathematically, have been slightly skewed more toward people getting the answer correct (by half an answer). The more people who answer the question correctly, the more likely it is that those people who got it correctly didn't do so by accident. However, if you're the only one who got it wrong, we have no idea if you didn't know the answer or if you guessed.

One of the most important things to note is that a bell distribution or its equivalent implies that some people will answer the answers correctly by accident. This is why there is no proof that a financial adviser, economist, and so on, who guesses right several times in a row about the economy or what have you has any idea about what they're doing ... if the other people who guessed had some right and some wrong. Or that a study or series of studies showing the benefit or ill-effect of something means anything, if you don't know out of how many unreported tests the results were taken (see here).