Surprising scores
Henry Morse, M.A.
Instructor IV
President, Fire Service Testing
Company, Inc.
Member, Federal Point (NC) Volunteer
Fire Department
Most departments consider 70 percent to be a passing score on a written test. That has worked pretty well, and there is perhaps no need to change it now. It can be interesting, however, to consider some things that score tells us and some things it doesn`t.
If a candidate knew absolutely nothing on a four-choice or multiple-choice test and answered every question on a properly constructed test purely by guessing, that candidate should score 25 percent if he guesses using the best guessing system. Imagine that. Yet the odds of such a candidate scoring more than that go up tremendously. The odds of that candidate`s scoring 100 percent by guessing alone are one in a pentadecillion–that is, one in 1,000,000,000,000,000,-000,000,000,000,000,000,000,000,000,000. That`s using a four-choice multiple choice test and assuming each decoy is equally plausible.
It would seem that such a test is a pretty sure measure of one`s knowledge. However, consider this. Just as our imaginary candidate who knew nothing could expect 25 percent by guessing alone (using the best guessing system), then so, too, can any student expect to get credit for 25 percent of the questions for which he does not know the answers–added on to those answers he did know. For example, if a candidate knew 60 of the 100 questions and got them right because he knew the material, that would leave 40 questions that would have to be guessed. But, guessing intelligently, he could expect to get 25 percent of those 40 right. That means he could expect to get 10 of the 40 right. Adding those to the 60 he really knew, he would achieve a passing score of 70 percent.
Yet it can be seen that this 70 percent score most likely results from someone who knew 60 percent of the test and guessed on the other 40, getting a quarter of those right by chance. A 70 percent score, then, really represents a 60 percent command of the material.
What score would a candidate who really did know 70 percent of the material present? It would be 70 percent plus one fourth of 30, which is 7.5, rounded to 8. A candidate who firmly knew 70 percent and guessed intelligently on the rest would show a score of 78 percent.
Should we make 78 percent the passing cutoff? It`s hard to say. But we should be aware of the surprising, perhaps shocking, fact that the math is inescapable and a score of 70 percent on a four-choice, multiple-choice test really represents as little as a 60 percent actual command of the material. And that assumes that each decoy is plausible, not the situation in which two of the decoys are easily dismissed, leaving really only a two-choice multiple choice, which many amateur test-writers mistakenly think is desirable.
In that case, it would require only a 40 percent actual command of the material to show a 70 percent score. Can you imagine? Yet, it`s true. If one knew 40 percent, that would leave 60 percent to guess. But on a two-choice, multiple-choice test, you could expect to get half of them right by guessing alone–which is 30 more questions, 40 + 30 = 70.
This applies also to true/false tests, which are in fact two-choice, multiple-choice tests.
Going to five-choice, multiple-choice tests is arduous on the test-writer and only raises the necessary command of the material from 60 percent to 62 percent to show a 70 percent score.
All in all, to ensure a 70 percent actual command of the material, the numbers seem to clearly show that the most suitable instrument is a four-choice, multiple-choice test with a 78 percent passing cutoff score.
It may not be wise to jump to 78 percent as a new cutoff score, but it would be wise to realize what today`s 70 percent really means, if only for our own understanding and expectation of these candidates as they turn in these scores.