In a previous blog post, we noted that the NAPLEX blueprint is changing in 2021 and a standard setting panel was conducted this past summer.
The purpose of standard setting is to determine a point along a score scale that classifies individuals into specific proficiency categories. For example, a fourth-grade reading test may be designed to identify students as “Basic,” “Proficient,” or “Advanced” – or whatever other classification schema has been determined. For the NAPLEX, standard setting is designed to examine the knowledge, skills, and abilities required for professional pharmacy practice and then choose a specific point along the NAPLEX score continuum (i.e., 0-150) that separates those who are qualified for independent, unsupervised practice from those who are not.
The Angoff Method
There are numerous different methods that may be used to identify a passing standard, but the Angoff (1971) method is the most common. This method asks subject matter experts to examine each test item and determine the probability that a “minimally qualified examinee” would get each question correct. The Angoff passing standard for each panelist is the sum of their item ratings across all items.
Take an example of a five-item test with five judges. Each judge is asked to examine each of the five items and determine the probability that a “minimally qualified candidate” would answer each item correctly.
Because the ratings are the probability of a correct response, a higher number represents an easier item. An item rating of .90 means that the judge thinks 90% of minimally qualified candidates would answer the question correctly and a rating of .10 means that only 10% would answer the question correctly. For our sample test,
- Judge A tends to think the items are relatively easy compared to the other judges as shown by the higher probability ratings;
- Judge B tends to think the items are harder; and
- Judge D’s item ratings look inconsistent with the other judges.
The “Total” column is the sum of each judge’s ratings. For our test,
- Judge A thinks that an examinee should need to answer 3.6 items correctly to pass;
- Judge B thinks 2.5 correct answers is an appropriate passing standard; and
- Judges C and D both think 3.3 is an appropriate passing standard even though Judge D has some irregular responses.
To identify a passing standard for the entire group, we find the mean score by summing the total for each judge and then dividing by the number of judges.
For our sample test, the sum of the total ratings is 15.9 and gives us an average of 3.18. Obviously, someone cannot answer 3.18 questions correctly, so a decision needs to be made whether the passing standard should be 3 or 4 correct responses.
Using Data to Set Policy
The responsibility to set a passing standard ultimately lies with the governing bodies. Psychometricians can utilize any number of different standard setting models and conduct different types of analyses, but the governing bodies must balance this data with other information about the profession to determine a passing standard that is fair and defensible.
The passing standard for the NAPLEX is re-evaluated every five years in accordance with commonly held practices for licensure testing. The results from the NAPLEX standard setting panel will be reviewed by NABP’s Executive Committee in December. If the Executive Committee decides that a change in the passing standard is warranted, NABP will communicate this change on the NAPLEX web page.