Exams that Harm

Inspector exams with too many difficult-to-answer questions are harmful to consumers.

The International Association of Certified Home Inspectors (InterNACHI) has administered more inspection exams than all other sources combined. As a U.S. Department of Education-accredited school, it is the leader in gathering home inspector-competence evaluation data, right down to the pass/fail rates of every question ever asked.

Often, we will hear someone in the industry complain that "the questions should have been more difficult to answer" on this or that exam. On the surface, it may appear that an exam is improved to the benefit of the consumer if the questions are made to be more difficult. Let us dispel this myth now.

For the purpose and ease of discussion, let's assume that we have a 100-point true/false exam. Because a multiple-choice question often has at least one clearly wrong answer choice, a multiple-choice question is not much different (and only a bit more difficult) mathematically than a true/false question.  Let's also assume that we have two exam-takers. One -- we'll call him Mr. Veteran -- knows 10 times what the other -- we'll call him Mr. Newbie -- knows about home inspections.

 

Now, we all know that if we make the exam too easy, Mr. Veteran and Mr. Newbie will score similarly, just as if we asked a Pulitzer Prize winner and a 6-year-old how to spell the word "cat."  This is known as "proof by extremes."  In the inspection industry, an exam that is too easy is harmful to consumers because it makes a weak distinction between Mr. Veteran and Mr. Newbie... and they both pass.

But what happens if the questions in the exam are too difficult to answer?  Let's find out using another proof by extremes.   Let's say we create a 100-point true/false home inspection exam that has so many difficult-to-answer questions that Mr. Veteran only knows the answer to 20 of the 100 questions. That's a pretty hard exam! Mr. Veteran will have to guess the answer to 80 of the questions. On average, he will score 60: 20 for the ones he knows the answer to, plus half of the 80 that he guesses at. Mr. Newbie knows a tenth of Mr. Veteran. So, he knows the answer to only two questions. On average, he will score 51: two for the ones he knows the answer to, plus half of the 98 he guesses at.

Now, if you flip a coin 10 times, you should get heads five times, on average. But, often, if you flip a coin 10 times, you will get more than five heads or fewer than five heads. The same is true for the questions our exam-takers are guessing at. Sometimes, Mr. Veteran will score worse than 60, and sometimes Mr. Newbie will score better than 51.  Using an online binomial calculator, one finds that one in 10 Mr. Newbies will score as well as the average Mr. Veteran, and that one in 10 Mr. Veterans will score as low as the average Mr. Newbie, all based solely on chance! Often, Mr. Newbie will score better than Mr. Veteran simply because the score on an exam that contains a large percentage of difficult-to-answer questions, that neither exam-taker knows the answers to, is determined solely by luck. That's not too good for consumers.

But, wait --  it gets worse for consumers! As we make the questions more difficult to answer, the percentage of questions that our exam-takers have to guess at goes up, which increases the reliance of the score on chance, which, in turn, increases the odds that Mr. Newbie will score as well or even better than Mr. Veteran, which weakens the exam's ability to determine who is competent and who isn't, which is worse for consumers. In fact, on an exam full of so many difficult-to-answer questions that Mr. Veteran only knows the answer to 10 of the 100, Mr. Veteran will (on average) score only 4.5 points better than Mr. Newbie.

But, wait -- it gets worse for consumers!  By making the questions more difficult-to-answer and increasing the chances that Mr. Newbie will be able to pass by being lucky, we also increase the chance that Mr. Veteran will fail due to bad luck. If the exam is used for certification (which gives the exam-passer a market advantage, and denies the exam-failer the market advantage), or, worse... if the exam is used for licensing (which puts the exam-passer into the market, and prohibits the exam-failer from entering the market), an exam with more difficult-to-answer questions increases the ratio of Mr. Newbies-to-Mr. Veterans in the marketplace by increasing the number of Mr. Newbies and decreasing the number of Mr. Veterans, which is horrible for consumers.

But, wait -- it gets worse for consumers!  If the exam is used for licensing, Mr. Newbie need not score anywhere near as high as Mr. Veteran to earn the right to wave the exact same government-issued credential (license) as Mr. Veteran has. This is especially true when the licensing exam uses a low passing cut-off score (like the NHIE does). The government displays both Mr. Veteran and Mr. Newbie as equally licensed in the eyes of the consumer, even though, in reality, their levels of competency differ greatly.
 

But, wait -- here's where it gets better for consumers. InterNACHI's exams don't rely much on chance.  InterNACHI's exams contain what some would describe as easy-to-answer questions that every Mr. Veteran should know the answer to. InterNACHI's exams are sometimes criticized for containing such questions. What these critics don't understand is our superior scoring system. If the exam-taker answers these questions correctly, he gets no credit for them, because we can't tell if he answered the questions correctly because he is a Mr. Veteran, or if he is a just a lucky Mr. Newbie. However, if the exam-taker answers them incorrectly, we can assume that he is very likely a Mr. Newbie, and the exam-taker is severely penalized (in terms of score) for failing to answer them correctly. For, you see, it is much easier to determine incompetence than competence.  This system, combined with InterNACHI's high passing cut-off scores, result in InterNACHI's exams being superior to other exams at distinguishing between competent and incompetent inspectors, who can't rely on chance and luck.