Levels of significance and significance testing
Statisticians use significance testing is used to tell whether a result is significant. They choose a level and state that the 'result is significant' if the probability of obtaining that result by chance is LESS than their chosen level. In the example given in the probability page, the probability of getting the first two cards correct is 4% (i.e. you would expect this by chance once every 25 times you started the game). As 4% is less than 5%, at the 5% level this is a significant result, but it is not significant at the 1% level.
Unfortunately, working out the probability of obtaining the score for significance testing is not as straightforward as working out the probability of getting a certain number of cards correct over a given number of rounds. The reason for this is because you should also work out all the probabilities of getting a certain number of cards or more (up to the number of rounds played) correct. These probabilities are then added together. The question that is asked is: "is the probability of getting this number of cards or more correct LESS than the level that was set?" If the answer to this question is yes, then the result is considered significant.
Example
For
example, suppose that we set the level of significance at 5% and that after
15 rounds you had correctly guessed six cards. We ask the question: Is the
probability of getting six cards or more correct likely to be more than 5%?
If the likelihood of matching six or more cards on an entirely by-chance basis
is more than 5%, then we deem the result to be 'not significant'. However,
if this result turns out to be expected to occur less than 5% of the time
(i.e. probability < 0.05) then we deem the result to be significant.
The probability of getting six cards correct is given by inserting the following values for n, r, p and q in the general binomial equation: 15, 5, 0.2 and 0.8.
As a percentage this is 4.3%. At first glance we may consider that it is significant, but this is only the probability of getting 6 out of 15. We must remember that as the number of rounds increases the probability of getting any single result will decrease, simply because there are more options to choose from. For the result to be significant we have to consider the probability of getting 7, 8, 9,…, 15 cards correct.
These we calculate in the same way, but replace r with the appropriate value. This can be rather longwinded, especially if we get fewer correct than more correct. Another way to look at the problem is to find the probabilities of getting 0, …, 5 correct and then subtracting these probabilities form 1.
Back to front?
The
probability of getting a result (any result) is 1.
The
probability of matching 0, 1, 2, 3, 4 or 5 cards is the sum of
P(x = 0) + P(x = 1) + P(x = 2) + P(x = 3) + P(x = 4) + P(x = 5) = P(x < 6).
(In P(x = 0) the capital P is shorthand for probability that the statement in the brackets occurs. The x = 0 is just a statement for no matching cards. The whole statement is simply the probability that no cards are matched. And likewise for the other values of x.)
The probability for any result is
P(x < 6) + P(x ≥ 6) = 1.
This just means that we either get less than six [P(x < 6)] or six or more [P(x ≥ 6)] - the only (and all) possible options.
Rearranging this expression we get P(x ≥ 6) = 1 – P(x < 6). We can more easily work out the six results for P(x < 6) than the nine results for P(x ≥ 6), as there are three less results to calculate:
| Card matched |
Probability |
Probability (%) |
| P(x = 0) |
0.0352 |
3.5 |
| P(x = 1) |
0.1319 |
13.2 |
| P(x = 2) |
0.2309 |
23.1 |
| P(x = 3) |
0.2501 |
25.0 |
| P(x = 4) |
0.1876 |
18.8 |
| P(x = 5) |
0.1032 |
10.3 |
| P(x < 6) |
0.939 |
93.9 |
Putting the value for P(x < 6) into P(x ≥ 6) = 1 – P(x < 6) we get P(x ≥ 6) = 1 – 0.939 = 0.0611. As a percentage this is 6.1%, and so the result is NOT significant.