Vol. 6, No. 1 Winter 1999
Introduction to This Issue
In this issue of The Behavioral Measurement Letter, Howard Shaffer examines problems in defining and measuring problem gambling in “Addiction and Gambling Disorders: On Matters of Measurement and Validity.” His piece illustrates well a fundamental problem in the health, behavioral, and social sciences, construct definition. In the health sciences, construct definition has obvious implications for treatment as well as measurement, for application as well as theory. Does the construct represent a pathologic condition itself or a symptom of pathology? If it is a symptom, is it part of a syndrome (a configuration of symptoms that characterizes a pathologic condition)? If a condition, does it coexist with other pathologic conditions? In terms of measurement in general, it is obvious that if one cannot define what one is measuring, it is meaningless to speak of validity and reliability. As Shaffer points out in the case of addictive disorders, both theory and the measurements that underlie theory must differentiate between first-order and second- order constructs (i.e., between unitary constructs and those comprised of other, subordinate constructs), and for both theory and its application, one must understand how constructs interrelate.
The use of measurement modeling as a tool for differentiating between first- and second-order constructs and for understanding how constructs interrelate are topics covered by Fred Bryant in his column, “Measurement Modeling: Identifying the Constructs Underlying the Center for Epidemiologic Studies Depression Scale (CES-D).” This article is the third in his series on measurement modeling as an important, increasingly used psychometric tool. In this installment, Bryant illustrates the application and value of measurement modeling in construct definition, in this case, defining the construct or constructs measured by the CES-D. He reports on a study in which “distress” as measured using the CES-D is modeled as a unidimensional construct, a three-factor construct, a four-factor construct, and a second-order construct consisting of four first-order constructs. Bryant’s piece further illustrates the centrality of construct definition in the social and behavioral sciences, underscoring points made in Shaffer’s discussion of construct definition and measurement in research on pathologic gambling.
As an old song says, “Little things mean a lot.” In research, as in day-to-day living, simple things, “obvious” things, are often overlooked or taken for granted. Time and again we ignore the simple or obvious, much to our peril, and often with ill consequences. Similarly, negative consequences can result from overlooking the nonobvious, the subtleties. Ora Strickland’s column, “The Practical Side of Measurement,” reminds us that in research as in life, one must attend to the obvious and the nonobvious details, that good measurement involves more than just selecting instruments based primarily or solely on psychometric considerations and then administering them with little or no regard to subjects’ capabilities and condition. One size does not fit all, that is, as Strickland points out, “instruments that have been reliable and valid in one situation may not function well under other circumstances,” instruments that are valid for one age-group of children may not be valid for others, instruments that can be administered easily to able, middle-aged adults may be difficult or impossible to administer to severely debilitated or elderly persons. Strickland’s piece demonstrates that good measurement involves thoughtful attention to detail as well as technical matters.
HaPI reading.
Al K. DeRoy, Editor
Addiction and Gambling Disorders: On Matters of Measurement and Validity
Howard J. Shaffer
The purpose of this article is to examine some of the validity and measurement issues associated with a contemporary perspective on addictive behaviors in general and disordered gambling in particular. Scientists have tried to understand addictive behaviors (e.g., alcoholism, drug abuse and dependence, pathological gambling) from a variety of viewpoints. For example, neurobiologists have tended to see addiction as a problem of neurotransmitters, receptor sites and transporter mechanisms. Psychologists have gravitated toward psychodynamics, learning, trauma, and other intrapersonal and environmental influences as vital to an understanding of addiction. Sociologists often see addiction as a result of family, community, social network, cultural and other social setting factors. Public policy makers often consider addiction to result from the laws that permit or discourage certain patterns of behavior. With some notable exceptions, there has been a tendency to view addictive behavior from the limited ideological perspectives associated with a particular discipline.
When clinicians examine people who are struggling with addiction, they often observe a range of concurrent or sequential problems. These apparently coexisting or overlapping difficulties suggest there may be an underlying addiction “factor.” This matter complicates the task of psychometricians. For example, consider a measure of gambling that correctly identifies cases of alcoholism, drug abuse, and depression even though these behaviors were not the target of the gambling screen. What can we say about the screen? In their analysis of a very similar problem (i.e., the validity of a measure of psychological “adjustment” and its positive correlation with a measure of “intelligence”), Rosenthal and Rosnow (1975) conclude: “That question is difficult to answer, but we could not claim on the basis of these results to understand our [gambling] test very well. It was not intended, after all, to be a measure of [alcoholism or depression]. In short, our test has good concurrent validity but poor differential validity. It does not correlate differentially with criteria for different types of observation” (p.70). Rosenthal and Rosnow remind us that we must be sensitive to the relationship between appropriate and inappropriate criteria during the process of establishing construct validity.
What can we say about the gambling problems that this screen identifies? Gambling problems may be the cause of excessive alcohol or drug use. Gambling problems also can stimulate depression. Depression also may result indirectly from the confluence of gambling with co-existing alcoholism or drug abuse. Alternatively, alcohol or drug use may stimulate depression that, in turn, encourages intemperate gambling-as an anodyne for emotional discomfort. To determine whether a pattern of excessive gambling, drinking, or drug using represents a distinct disorder, scientists must establish construct validity. While these and other possible permutations of these problems can explain disordered gambling behavior, it also is possible that all of these behaviors reflect the expression of an underlying addiction syndrome.
A syndrome is a group of symptoms that collectively indicate or characterize an undesirable condition or quality, a disease, a psychological disorder, or another abnormal condition. Not all elements of the syndrome need be present all of the time. It is the pattern of elements rather than the intensity of any one component that is critical to identifying a syndrome. Establishing criteria and instruments to identify a syndrome represent a daunting measurement task. This challenge has led to the variety of problems associated with psychometric attempts to distinguish non-pathologic drug use, excessive drug use, and drug abuse from drug dependence and addiction (e.g., Shaffer, 1996; 1997) or “heavy” and problem gambling from pathological gambling.
Given that the field of psychiatry in general, and disordered gambling in particular, has no gold standards (Faraone & Tsuang, 1994), there is little rationale for placing one instrument, or even clinical diagnosis, as the gold standard against which another instrument is measured. As Faraone and Tsuang note, “many studies of psychiatric diagnosis compute accuracy statistics. However, these assess the accuracy of one diagnostic approach (e.g., DSM-IV) with respect to another (e.g., expert clinical diagnoses). They do not assess the accuracy of diagnostic procedures with reference to the ‘true’ but unobservable latent state of illness” (p. 651). Determining the validity of a latent state is complicated further when we consider this circumstance to reflect a syndrome rather than a more coherent unidimensional disorder.
The problems associated with determining the validity of gambling or addictive behaviors are not limited to the matters of syndromal behaviors, but actually begin with the very definition of validity. Validity is the capacity of an instrument to measure what it purports to measure. Validity is neither a static nor an inherent characteristic of a screening instrument. As Goldstein and Simpson (1995) suggest, “validity refers to the questions ‘for what purpose is the indicator being used?’ … and ‘how accurate is it for that purpose?”” (pp. 229-230). Determining instrument validity is an unending and dynamic investigative process. We cannot simply conclude that an instrument has been shown to be valid for all purposes and all settings. “An indicator (e.g., an instrument, such as a test, a rating, or an interview) can be valid for one purpose, but not for another” (Goldstein & Simpson, 1995, p. 230).
Directed by theoretical and ultimately practical purposes, validity is the dynamic consequence of applying an instrument to a measurement task. In the field of gambling studies, however, theory is conspicuously absent from most prevalence research (Shaffer, Hall, & Vander Bilt, 1997). When conventional wisdom and scientific theory shift or change, the validity of an instrument can be nullified abruptly. The history of the South Oaks Gambling Screen (Lesieur & Blume, 1987) provides an example of the relative nature of validity. Although for some time researchers considered that the SOGS lifetime measure had “been found valid and reliable” (Volberg, 1994, p. 238), p. 238), some investigators now suggest that the SOGS lifetime measures over-state the actual prevalence of pathological gambling” (Volberg, 1997, p. 41). Assessments of validity-even by the same scientist-can wax and wane depending upon the objectives established for an instrument’s use and the shifting cultural values that energize those purposes. For example, in her study of Washington state adolescents, Volberg states that “Our approach, while conservative, is intended to focus as clearly as possible on those adolescents who show incontrovertible signs of problematic involvement in gambling” (Volberg, 1993, p. 17). Volberg uses this conservative approach, in part, because she is uncomfortable with a method [i.e., the
SOGS] that classifies 8% of adolescents as problem or probable pathological gamblers” (1993a, p. 34). In other words, the criteria for identifying pathological gambling are not fixed, but vary depending on whether a given set of criteria yields an “acceptable” prevalence rate among a particular group.
Absent a gold standard for determining pathological gambling, we do not know whether the SOGS over-estimates the prevalence rate of gambling disorders or whether clinical assessment and DSM-based instruments under- estimate the prevalence rate. This problem of anchoring reveals itself often as scientists attempt to determine how best to frame prevalence estimates.
Ultimately, the field of gambling studies is in need of research that can provide additional evidence about the construct validity of the notions that represent disordered gambling (e.g., problem, pathological, probable pathological, etc.). While new evidence (Shaffer, Hall, & Vander Bilt, 1997) provides considerable support for a recognizable and identifiable pattern of behaviors that can be considered disordered gambling, there are important conceptual questions that still remain. A primary construct validity question requires scientists to focus on whether disordered gambling is a primary or a secondary disorder. For example, according to the DSM-IV, a person meeting all of the criteria for pathological gambling is not considered a pathological gambler if he or she also concurrently meets the criteria for a Manic Episode, and the Manic Episode is responsible for the excessive gambling (APA, 1994). In this case, pathological gambling is not considered a unique disorder, but rather a cluster of symptoms associated with another disorder. If pathological gambling represents a primary disorder, then it can emerge in the absence of other comorbidity and cause sequelae independent of any other condition. However, if it is a secondary disorder, subordinate to other dysfunctional behavior, then pathological gambling will only exist as a consequent of another condition (e.g., manic episode, anti- social personality, alcohol abuse, obsessive- compulsive disorder, or adolescence; Jessor & Jessor, 1977). Although Shaffer et al. (1997) suggest that researchers of disordered gambling have measured a relatively stable and robust phenomenon, the field of gambling studies has not yet established with ample certainty that this phenomenon represents a unique construct.
Since gambling researchers have paid very little attention to this important conceptual issue of discrete and comorbid phenomena2 – and the associated matter of differential validity – the possibility remains that pathological gambling is not a discrete primary disorder. Alternatively, it is possible that pathological gambling is a discrete and primary disorder (i.e., it will exist independent of any other disorder). Current research is beginning to suggest that, in many instances, other primary psychiatric disorders provide better explanations of excessive gambling (i.e., (i.e., disordered gambling is considered to be subordinate to other primary disorders). According to the current version of the DSM-IV (American Psychiatric Association, 1994), pathological gambling can have either primary or secondary status. In some cases it is considered to be a primary disorder (i.e., independent of other diagnoses), and in other cases it is considered to be the sequelae of another disorder. The implications of this issue are potentially significant for the development of both treatment prescriptions and social policy initiatives designed to ameliorate or regulate gambling-related problems.
Whether we view disordered gambling as primary or secondary, intemperate gambling inflicts human suffering. Whether a primary or secondary problem, pathological gambling often warrants professional assistance. If it is a disorder secondary to another problem, clinicians should focus on providing attention to the gambling issues in addition to the problems related to the primary disorder. Future research will help clarify these theoretical, research, and clinical issues. The scientific measurement of these problems will demand careful conceptual guidance. It is difficult indeed to distinguish the “cart” from the “horse” without some systematic experience with carts and horses.
The author thanks Matthew Hall and Joni Vander Bilt for their important contributions and thoughtful comments on earlier versions of this article. This article was supported in part by grants from the National Center for Responsible Gaming (Kansas City, MO) and the Center for Substance Abuse Treatment (#1U98T100846).
Correspondence regarding this article should be sent to Dr. Shaffer at Harvard Medical School, The Division on Addictions, 220 Longwood Avenue, Boston, MA 02115.
References
American Psychiatric Association. (1994). DSM-IV: Diagnostic and statistical manual of mental disorders. (Fourth ed.). Washington, DC: American Psychiatric Association.
Briggs, J.R., Goodin, B.J., & Nelson, T. (1996). Pathological gamblers and alcoholics: Do they share the same addiction? Addictive Behaviors, 21, 515-519.
Faraone, S.V., & Tsuang, M.T. (1994). Measuring diagnostic accuracy in the absence of a “gold standard”. American Journal of Psychiatry, 151, 650-657.
Goldstein, J.M., & Simpson, J.C. (1995). Validity: Definitions and applications to psychiatric research. In M. T. Tsuang, M. Tohen, & G. E. Zahner (Eds.), Textbook in psychiatric epidemiology (pp. 229-242). New York: Wiley-Liss.
Lesieur, H.R., & Blume, S.B. (1987). The South Oaks Gambling Screen (SOGS): A new instrument for the identification of pathological gamblers. American Journal of Psychiatry, 144, 1184-1188.
Rosenthal, R., & Rosnow, R.L. (1975). Primer of methods for the behavioral sciences. New York: Wiley & Sons.
Shaffer, H.J. (1996). Understanding the means and objects of addiction: Technology, the Internet, and gambling. Journal of Gambling Studies, 12, 461-469.
Shaffer, H.J. (1997). The most important unresolved issue in the addictions: Conceptual chaos. Substance Use & Misuse, 32, 1573-1580.
Shaffer, H.J., Hall, M.N., & Vander Bilt, J. (1997). Estimating the prevalence of disordered gambling behavior in the United States and Canada: A meta- analysis. Boston: Presidents and Fellows of Harvard College.
Volberg, R. (1993). Gambling and problem gambling among adolescents in Washington state. Albany, NY: Gemini Research.
Volberg, R.A. (1994). The prevalence and demographics of pathological gamblers: Implications for public health. American Journal of Public Health, 84, 237-241.
Volberg, R.A. (1997). Gambling and problem gambling in Mississippi: A report to the Mississippi Council on Compulsive Gambling (Social Research Report Series 97- 1). Mississippi State: Mississippi State University, Social Science Research Center.
Howard J. Shaffer, PhD, CAS, is Associate Professor, Director of the Division on Addictions, and founding Director of the Norman E. Zinberg Center for Addiction Studies at Harvard Medical School. He is also founder and currently President of the Board of Trustees of the American Academy of Health Care Providers in the Addictive Disorders, the first international credentialing body for clinicians working that field, Editor of The Journal of Gambling Studies, Associate Editor of The Journal of Substance Abuse Treatment, on the editorial boards of five other journals, a reviewer for The New England Journal of Medicine, and author of six books. In addition to measurement methods in gambling and other addictions, his interests include the natural history of addiction, impulse control regulation, and social perceptions of addiction.
Read additional articles from this newsletter:
The Practical Side of Measurement
6-2-fall-1999
