What Really Matters When Using Psychosocial Measurement Tools

Fred B. Bryant, Ph.D., Emeritus Professor of Psychology, Loyola University Chicago
 
When researchers or practitioners want to measure a particular variable or outcome in their work, they look for a measurement instrument that has both reliability and validity. But what exactly does it mean for a measurement instrument to be “reliable” or “valid,” why do reliability and validity matter, and how do you go about assessing the reliability and validity of a measurement tool? Below I address these critical questions.
 
The Meaning of Reliability in the Context of Measurement
 
Imagine a medical researcher wants to evaluate the effectiveness of a 6-month weight-loss program in helping people lose weight. The researcher first weighs clients at the start of the 6-month program and then weighs them again at the end of the six-month program. But imagine that the weighing scale the researcher uses to assess each client’s weight contains a damaged spring that leads it to produce unreliable readings that fluctuate wildly over time, even when the weight of the individual being measured is not actually changing at all. Because this weighing scale is not reliable, you cannot trust it to provide accurate measurements. And for this reason, the weighing scale would be a poor measure to use in trying to assess the effectiveness of the weight-loss clinic in helping people lose weight.

Why Reliability Matters

The more unreliable an assessment is, the less useful it will be in research or practice. A basic tenet of classical test theory in the field of psychometrics is that an observed score obtained from a measurement of a particular variable for a given individual is a function of two influences: (1) the actual true score of the individual on this variable, and (2) error. In others words, Observed Score = True Score + Error. The less error that exists in a measurement, the closer observed scores will be to true scores; when there is no error in measurement, observed scores will be perfectly reliable and will consistently reflect true scores. Because unreliable measurements are filled with error, they are likely to provide false, inconsistent assessments of what one wants to measure.

Psychosocial Measurement

How to Assess the Reliability of Measurement

There are at least three primary methods that researchers use to evaluate how reliable a measurement instrument is.

1. Internal Consistency Reliability. If an instrument consists of multiple self-report items or questions, each of which is intended to measure the same underlying concept, then researchers can assess the degree to which respondents provide consistent responses across the full set of items. In this case, researchers typically compute a coefficient of internal consistency (such as coefficient alpha, coefficient omega, a split-half correlation, or an item-total correlation) that indicates the degree to which respondents provide comparable responses across all of the items in the instrument.

2. Inter-Rater (or Inter-Observer) Reliability. If an instrument is designed to be used by observers or trained experts who provide ratings of an underlying concept in a sample of individuals, then researchers can assess the degree to which different raters give consistent ratings of the underlying concept for the same individual. In this case, researchers typically compute a coefficient of inter-rater agreement (such as coefficient kappa, Kendall’s tau, the Spearman rank-order correlation, or an intra-class correlation) that indicates the degree to which different raters tend to agree in their assessments of the same individual.

3. Test-Retest Reliability. If an instrument can be administered to the same sample of individuals at two separate points in time, then researchers can assess the degree to which respondents provide stable, consistent responses to the same instrument over time. In this case, researchers typically compute a coefficient of test-retest reliability (such as the Pearson correlation coefficient) that indicates the degree to which individual responses are stable over time. If one can assume that the underlying concept being measured is a relatively stable trait that should not change across the two time points, then this correlation provides an assessment of the degree to which scores on the measurement instrument are influenced by random error.

The Meaning of Validity in the Context of Measurement

Whereas reliability concerns how consistently an instrument measures a target concept, validity refers to whether a particular instrument actually measures the target concept it is intended to measure. Because inconsistency in measurement would make an instrument inaccurate, an instrument must be reliable to be valid (Nunnally & Bernstein, 2000). However, a reliable instrument is not necessarily valid.

An instrument might be reliable in providing consistent responses across multiple items (internal consistency reliability), multiple raters (inter-rater reliability), or multiple time-points (test-retest reliability), but it might nevertheless measure a different underlying concept than it is assumed to assess.

For example, consider a new multi-item self-report instrument that has been specifically designed to measure individual differences in empathy (conceptually defined as the ability to understand and vicariously experience other people’s feelings). Although research might demonstrate that scores on this instrument show high levels of internal consistency and test-retest reliability, it might also be the case that scores on the test are only weakly correlated with other existing measures of empathy and are actually more strongly correlated with measures of compassion (conceptually defined as motivation to help others who are suffering). In this case, the new empathy measure would be said to lack validity as a measure of empathy. To understand more clearly how reliability and validity operate together and relate to one another, imagine an archer who is shooting arrows at a target that has a bullseye in the middle. Shooting an arrow into the bullseye of the target, on the one hand, is analogous to “hitting the mark” with respect to measurement accuracy or validity. Hitting the same mark consistently, on the other hand, is analogous to measurement consistency or reliability.

Regarding this situation, classical test theory allows for only three possible outcomes regarding the consistency (reliability) and accuracy (validity) of measurement. Specifically, a measure can be:

• Neither Reliable nor Valid: The archer’s arrows scatter randomly all over the target. These results are neither consistent nor accurate—that is, they are unreliable and invalid.

• Reliable but not Valid: The archer consistently hits the exact same spot on the target, but it is in the bottom left corner, for example, rather than in the bullseye. These results are highly consistent, but inaccurate—that is, they are reliable, but not valid.

• Reliable and Valid: The archer consistently hits the bullseye. The results are both highly consistent and highly accurate—that is, they are reliable and valid. This is what researchers strive for in measurement.

• Valid but not Reliable: According to classical test theory (Nunnally & Bernstein, 1994), this outcome is conceptually impossible. Scientific consensus dictates that reliability is a prerequisite for validity—in other words, an instrument must be consistent (i.e., reliable) in order to truly measure what it is supposed to measure (i.e., in order to be valid). Otherwise, the instrument would be incapable of providing an accurate assessment of the true level of the intended target concept because of measurement error.

It is important to note that validity is not an intrinsic characteristic of an instrument, but rather is a context-specific property of the use of an instrument for a particular purpose in a particular setting with a particular population. For example, a given instrument might be valid for one particular use (e.g., as a measure of compassion for adults), but not for another (e.g., as a measure of compassion for children).

Why Validity Matters

If an assessment tool does not actually measure what it was designed to measure, then it provides an inaccurate result which has little or nothing to do with what one wants to measure. Invalid instruments provide misleading information that is worthless or harmful in drawing meaningful conclusions.

How to Assess the Validity of Measurement

Although researchers can adopt many different strategies to validate the use of an instrument for a particular purpose, the most commonly used approaches to assessing measurement validity focus on how thoroughly (content validity) or accurately (construct validity) an instrument measures its intended target concept, or how useful it is in predicting important outcomes that should be related to what it supposedly measures (criterion validity).

1. Content Validity. Content validity refers to the degree to which an instrument covers all relevant aspects of the conceptual or behavioral domain it is intended to measure. Although the content validity of an instrument is sometimes assessed crudely in terms of a researcher’s subjective impression of how thoroughly its items cover the topics they are supposed to cover, researchers often use multivariate statistical methods (e.g., principal components analysis, or exploratory or confirmatory factor analysis) to obtain a more precise estimate of the degree to which an instrument’s items accurately represent a full breadth of conceptual coverage. These more rigorous quantitative assessments of content validity can be used to test whether people’s responses to different subsets of the instrument’s items actually reflect the various conceptual facets of the target concept that an instrument should include.

2. Construct Validity. Construct validity refers to the degree to which an instrument actually measures what it is supposed to measure. The two most commonly used strategies for assessing the construct validity of an instrument involve correlating scores on the particular instrument with scores on other established instruments that have been shown to measure either (1) the same underlying concept that the instrument being validated is intended to measure (convergent validity) or (2) underlying concepts that are different from the concept that the instrument being validated is intended to measure (discriminant validity).

Convergent and discriminant validity are often evaluated in relation to one another, by including within the same study multiple measures, some of which are hypothesized to demonstrate stronger relationships than others to the instrument being validated. The most typical approach to construct validation has traditionally been to compute Pearson correlation coefficients among scores on a battery of instruments and then inspect the resulting patterns of association for evidence that scores on the instrument being validated correlate more strongly with other measures of the same concept (convergent validity) than with measures of different concepts (discriminant validity).

3. Criterion Validity. Criterion validity refers to the degree to which an instrument can be used to predict an important external outcome (or criterion) associated with the concept it is supposed to measure. The validational criterion of interest may be an outcome, event, or behavior measured either in the past (retrospective validity), present (concurrent validity), or future (prospective validity). The assessment of an instrument’s criterion validity typically involves statistical analyses that quantify the degree to which individuals’ scores on the instrument can be used to predict or classify accurately their scores or values on the criterion measure. If the criterion is measured using a continuous, equal interval scale, then the most commonly used statistical method for assessing criterion validity is linear regression analysis; if the criterion is measured using a dichotomous (or multiple-category) nominal scale, then the most commonly used statistical method is binary (or multinomial) logistic regression.

Psychosocial Measurement Tools

How to Use the HaPI Database to Select Appropriate Measurement Instruments for Research

The Health and Psychosocial Instruments (HaPI) database provides primary bibliographic records and summary information for more than 50,000 unique measurement tools (contained in records called “primary sources”) that have been reported in the health and psychosocial research literature. Users can search the electronic database specifying combinations of key terms to identify relevant measures to administer in their research; and they can also evaluate the reliability and validity of these relevant tools across later studies (called “secondary sources” in HaPI) that have used these same measures. Users can examine reliability and validity across the primary and secondary sources for an instrument, to make informed, evidence-based decisions about which instruments to use in their own research. Through this process, the HaPI database can help users find the most appropriate instrument, whose content most closely matches their conceptual needs and for which sufficient evidence exists regarding reliability and validity.

In addition to helping researchers identify the most appropriate tools for assessing key variables in experiments and surveys, the HaPI database is also invaluable in helping users develop and validate new measurement instruments. In particular, developers of new instruments can systematically use the database to identify relevant measures of: (a) similar, related concepts with which a new instrument should correlate in evaluating convergent validity; (b) distinct, unrelated concepts with which the new instrument should be uncorrelated in evaluating discriminant validity; and (c) important external outcomes that the new instrument should predict in evaluating criterion validity. Thus, the HaPI database is not only a powerful and versatile tool for identifying reliable and valid health measurement tools in health and psychosocial research, but also a rich and unparalleled resource in the development and validation of new measurement instruments.

Additional Resources for Interested Readers:

Bryant, F. B. (2000). Assessing the validity of measurement. In L. G. Grimm & P. R. Yarnold (Eds.), Reading and understanding more multivariate statistics (pp. 99–146). American Psychological Association.

Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory (3rd ed.). McGraw-Hill.

Streiner, D. L., Norman, G. R., & Cairney, J. (2015). Health measurement scales: A practical guide to their development and use (5th ed.). Oxford University Press.