This is the first in a series of posts that explores the rigorous process of building NABP’s examinations. First up: Laying the foundation for an exam’s validity through the interpretation and use argument.
The validity of an examination, or assessment, is the most fundamental consideration in the development and evaluation of assessments. Validity is the extent to which evidence and theory support the interpretations of assessment results for the proposed uses of an assessment.1
The interpretation and use argument 2–4 (IUA) of an assessment considers all of the claims based on assessment results and scores. The validity of a proposed interpretation and use is the plausibility and appropriateness of the proposed interpretation and use at the time of the claim. The evidence required to support the IUA depends on the claims being made and must be continually collected, as the IUA may change over time. It is important to note that validity is not a property of the assessment, but rather a property of the IUA. Assessments with strong evidence to support the IUA are considered to have “high validity,” while those whose IUA is not adequately supported by evidence are considered to have “low validity.”
As the [interpretation and use argument] for NAPLEX relates exclusively to the classification of individuals as either “competent” or “not competent” for entry-level practice, any use of NAPLEX scores or results outside of this claim is not supported by evidence and therefore not valid.
For example, NABP’s North American Pharmacist Licensure Examination (NAPLEX) measures a candidate’s application of the knowledge and skills necessary for competent entry-level pharmacy practice and is required by all United States boards of pharmacy as part of their assessment of competence to practice and the granting of pharmacist licensure. As the IUA for NAPLEX relates exclusively to the classification of individuals as either “competent” or “not competent” for entry-level practice, any use of NAPLEX scores or results outside of this claim is not supported by evidence and therefore not valid.
Anecdotally, claims have been made that NAPLEX scores assist with individual student assessment and curricular improvement. As previously noted, these claims are outside the scope of the NAPLEX IUA and therefore constitute an invalid use of the test. However, NABP does provide another examination related to student assessment and curriculum improvement. The Pharmacy Curriculum Outcomes Assessment (PCOA) is a comprehensive examination to align with the PharmD program curricula. The purpose of the PCOA is to provide an objective and external measure of student performance for examining program quality.
Thus, as the NAPLEX is designed for licensure decisions and the PCOA is designed for curriculum improvement, the IUA for each does not support the IUA of the other even though there is some correlation between the scores.
There are multiple types of validity evidence that may be collected to support the claims made by the IUA. A description of the test development process, such as job/practice analyses, exam item writing procedures, and the development of item maps and task models, often serve as the first step. Technical details on the scoring, scaling, and equating processes are also common sources of validity evidence. For norm-references tests, or exams that rank a candidate within a group or compare a candidate to other candidates, a description of the overall comparison group is a common source of validity evidence. For tests that group candidates into achievement levels, such as pass/fail or novice/proficient/master, a description of achievement level categorization is a common source of validity evidence. Depending on the type of testing program, varying sources of validity evidence may be publicly available or available to specific stakeholders.
NABP has recently embarked on a project to make the validity evidence collected in support of the IUA for our programs more accessible to both the public and stakeholders. Watch this space for updates.
REFERENCES
- AERA, APA, NCME. Standards for Educational and Psychological Testing. American Educational Research Association; 2014.
- Kane M. Validating the Interpretations and Uses of Test Scores. J Educ Meas. 2013;50(1):1-73.
- Kane M. Validation. In: Brennan RL, ed. Educational Measurement. 4th ed. American Council on Education and Praeger; 2006:17-64.
- Cronbach LJ, Meehl PE. Construct Validity in Psychological Tests. Psychol Bull. 1955;52(4):281-302.