There is a movement in psychology to require the usage of psychological measures to aid in the diagnosis of some mental illnesses. While developing the DSM-V, there was some discussion of including results of specific measures as part of diagnostic criteria. Considering the two articles you read, Limitations of Diagnostic Precision and Predictive Utility in the Individual Case: A Challenge for Forensic Practice\u00c2 and Diagnostic Utility of the NAB List Learning Test in Alzheimer\u00e2\u20ac\u2122s Disease and Amnestic Mild Cognitive Impairment,write a response paper discussing the utility of using psychological measures as suggested above in the diagnosis of mental illness.<\/p>\n
ORIGINAL ARTICLE
\nLimitations of Diagnostic Precision and Predictive Utility
\nin the Individual Case: A Challenge for Forensic Practice
\nDavid J. Cooke \u00c6 Christine Michie
\nReceived: 24 August 2007 \/ Accepted: 2 February 2009 \/ Published online: 11 March 2009
\nAmerican Psychology-Law Society\/Division 41 of the American Psychological Association 2009
\nAbstract Knowledge of group tendencies may not Help
\naccurate predictions in the individual case. This has
\nimportance for forensic decision making and for the
\nassessment tools routinely applied in forensic Assessments.
\nIn this article, we applied Monte Carlo methods to examine
\ndiagnostic agreement with different levels of inter-rater
\nagreement given the distributional characteristics of PCL-R
\nscores. Diagnostic agreement and score agreement were
\nsubstantially less than expected. In addition, we examined
\nthe confidence intervals associated with individual predictions
\nof violent recidivism. On the basis of empirical
\nfindings, statistical theory, and logic, we conclude that
\npredictions of future offending cannot be achieved in the
\nindividual case with any degree of confidence. We discuss
\nthe problems identified in relation to the PCL-R in terms of
\nthe broader relevance to all instruments used in forensic
\ndecision making.
\nThere is an important disjunction between the perspective
\nof science and the perspective of the law; while science
\nseeks universal principles that apply across cases, the law
\nseeks to apply universal principles to the individual case.
\nBridging these perspectives is a major challenge for psychology
\n(Faigman, 2007). It is recognized by statisticians
\nthat knowledge of group tendencies\u2014even when precise\u2014
\nmay not Help accurate Assessment of the individual case
\n(e.g., Colditz, 2001; Henderson & Keiding, 2005; Rockhill,
\n2001; Tam & Lopman, 2003). It is a statistical truism that
\nthe mean of a distribution tells us about everyone, yet no
\none. This has serious implications for the use of psychological
\ntests in forensic decision making. To illustrate these
\nlimitations, we focus on one of the most widely used, and
\nperhaps the most extensively validated, test in the forensic
\narena\u2014the Psychopathy Checklist Revised (PCL-R1; Hare,
\n2003). We emphasize, however, that all psychological tests
\nused in the same way in the forensic arena will suffer from
\nsimilar limitations (e.g., VRAG, Quinsey, Harris, Rice, &
\nCormier, 1998; Static-99, Hanson & Thornton, 1999;
\nCOVR, Monahan et al., 2005).
\nMental health professionals are frequently asked to
\nopine whether an individual might be violent in the future;
\npsychopathic personality disorder is an important risk
\nfactor to consider (Hart, 1998). The PCL-R is the most
\nfrequently used measure of psychopathic personality disorder;
\nit has been described as the \u2018\u2018gold standard\u2019\u2019 for that
\npurpose (Edens, Skeem, Cruise, & Cauffman, 2001; as
\ncited in Hare, 2003). There can be little doubt that the PCLR
\nhas made a major contribution to our understanding of
\nviolence (Hart, 1998); nonetheless, it is important for the
\nfield to consider both its strengths and its limitations.
\nFindings for this instrument will have implications for less
\nwell-validated tools. In this introduction, we consider two
\nissues; first, the use of PCL-R scores in forensic practice
\nand second, the general problem of the precision of predictions
\nabout an individual case.
\nD. J. Cooke (&) C. Michie
\nDepartment of Psychology, Glasgow Caledonian University,
\nGlasgow G4 0BA, UK
\ne-mail: djcooke@rgardens.vianw.co.uk
\n1 The PCL-R is a 20-item rating scale of traits and behaviors intended
\nfor use in a range of forensic settings. Definitions of each item are
\nprovided and evaluators rate the lifetime presence of each item on a
\n3-point scale (0 = absent, 1 = possibly or partially present, and
\n2 = definitely present) on the basis of an interview with the
\nparticipant and a review of case history information.
\n123
\nLaw Hum Behav (2010) 34:259\u2013274
\nDOI 10.1007\/s10979-009-9176-x
\nPCL-R SCORES AND FORENSIC PRACTICE
\nMuch of the interest in the construct psychopathy comes
\nfrom the relationship between the PCL-R and future
\ncriminal behavior (Lyon & Ogloff, 2000). Previous
\nresearch suggests that psychopathy\u2014as assessed using the
\nPsychopathy Checklist-Revised (PCL-R; Hare, 1991)\u2014is
\nan important risk marker for criminal and violent behavior
\n(Douglas, Vincent, & Edens, 2006; Hart, 1998; Hart &
\nHare, 1997; Hemphill, Hare, & Wong, 1998; Leistico,
\nSalekin, DeCoster, & Rogers, 2008; Salekin, Rogers, &
\nSewell, 1996). In fact, the PCL-R has been lauded as an
\n\u2018\u2018unparalleled\u2019\u2019 single predictor of violence (Salekin et al.,
\n1996). Hart (1998) argued that failure to consider psychopathy
\nin a violence risk assessment may constitute
\nprofessional negligence. This empirical base has resulted in
\nthe PCL-R being used, not merely to measure the trait
\nstrength of psychopathy in an individual, but also to make
\npredictions about what he or she will do in the future (Hare,
\n1993). As we demonstrate formally below, this additional
\nstep of prediction means that the potential for imprecision
\nin forensic evidence is greatly increased: It expands the
\ngulf between inferences about groups and inferences about
\nindividuals.
\nThe PCL-R has been incorporated into statutory or legal
\ndecision making (Hare, 2003). Within England and Wales,
\na PCL-R score above a cut-off of 25 or 30 can lead to
\ndetention in either a Special Hospital or a prison (Maden &
\nTyrer, 2003); in certain Canadian provinces parole boards
\nexplicitly consider PCL-R scores (Hare, 2003), and in
\nTexas psychopathy assessments are mandated by statute for
\nsexual predator Assessment (Edens & Petrila, 2006).2 The
\nPCL-R plays a role in criminal sentencing, including
\ndecisions regarding indefinite commitment and capital
\npunishment, institutional placement and treatment, conditional
\nrelease, juvenile transfer, child custody, witness
\ncredibility, civil torts, and indeterminate civil commitment
\n(DeMatteo & Edens, 2006; Fitch & Ortega, 2000; Hart,
\n2001; Hemphill & Hart, 2002; Lyon & Ogloff, 2000;
\nWalsh & Walsh, 2006; Zinger & Forth, 1998). The PCL-R
\nis regarded by many as the best method for operationalizing
\nthe construct of psychopathy. For example, Lyon and
\nOgloff (2000) argued that \u2018\u2018\u2026it is critical that the assessment
\nis made using the PCL-R\u2019\u2019 (p. 166) when evidence
\nabout violence risk, based on psychopathy, is provided.
\nBecause of its central role in forensic decision making it is
\nvital to assess its strengths and limitations and, by comparison,
\nthe limitations of less well-validated procedures.
\nPREDICTIONS FOR INDIVIDUALS VERSUS
\nPREDICTIONS FOR GROUPS
\nPrediction is the raison d\u2019e\u02c6tre of many forensic instruments
\n(e.g., VRAG, Quinsey et al., 1998; Static-99, Hanson &
\nThornton, 1999; COVR, Monahan et al., 2005). While this is
\nnot true of the PCL-R its frequent use in forensic practice is
\nunderpinned by the assumption\u2014implicit or explicit\u2014that it
\ncan predict future offending (Walsh & Walsh, 2006). How
\nprecise can such predictions be The precision of any estimate
\nof a parameter (e.g., mean rate of recidivism of a group)
\ncan be measured by the width of a confidence interval (CI); a
\nCI gives an estimated range of values, which is likely to
\ninclude an unknown population parameter. If independent
\nsamples are taken repeatedly from the same population, and a
\nCI calculated for each sample, then a certain percentage
\n(confidence level) of the intervals will include the unknown
\npopulation parameter. Typically, 95% of these intervals
\nshould include the unknown population parameter; other
\nintervals may be used (e.g., 68% and 99%). The width of this
\ninterval provides a measure of the precision\u2014or certainty\u2014
\nthat we can have in the estimate of the population parameter.
\nThe width of a CI of a population parameter is linked, in part,
\nto the sample size used to estimate the population parameter
\n(see below for a more technical explanation).
\nThe prevailing prediction paradigm has two stages.
\nFirst, the parameters (mean, slope, and variance) of a
\nregression model linking an independent variable (e.g.,
\nPCL-R score) to a dependent variable (e.g., likelihood of
\nreconviction) are estimated. Each of these parameters has
\nuncertainty associated with them, which can be expressed
\nby confidence bands about the regression line. Second, a
\nnew case is selected and the PCL-R score is assessed, the
\nmodel is applied and the likelihood of reconviction is
\nestimated. The best estimate of the likelihood of reconviction
\nfor a new case will be identical to the point on the
\nregression line for that PCL-R score. This new estimate has
\na CI\u2014also known as a prediction interval\u2014that expresses
\nthe precision, or certainty, that should be associated with
\nthe prediction made about the new case. Often the two
\nsteps are conflated, with the unrecognized assumption
\nbeing made that the prediction interval for the new case is
\ncomparable to the CIs for the model. It is not (see below).
\nThe problem of making predictions for individuals from
\nstatistical models is now recognized in other disciplines. In
\nrelation to medical risks, Rose (1992) expressed the position
\nclearly: \u2018\u2018Unfortunately the ability to estimate the
\naverage risk for a group, which may be good, is not matched
\nby any corresponding ability to predict which
\nindividuals are going to fall ill soon\u2019\u2019 (p. 48). In relation to
\nreoffending, Copas and Marshall (1998) made a related
\npoint \u2018\u2018\u2026the score is not a prediction about an individual
\n[italics added], but an estimate of what rate of conviction
\n2 The PCL-R is the most commonly used instrument for assessing
\npsychopathy in this setting (Mary Alice Conroy, Personal communication,
\n10 April 2007).
\n260 Law Hum Behav (2010) 34:259\u2013274
\n123
\nmight be expected of a group [italics added] of offenders
\nwho match that individual on the set of covariates used by
\nthe score\u2019\u2019 (p. 170) (see also Altman & Royston, 2000;
\nBradfield, Huntzickler, & Fruehan, 1970; Colditz, 2001;
\nElmore & Fletcher, 2006; Henderson, Jones, & Stare, 2001;
\nHenderson & Keiding, 2005; Rockhill, 2001; Rockhill,
\nKawachi, & Colditz, 2000; Tam & Lopman, 2003; Wald,
\nHackshaw, & Frost, 1999).
\nIt is not generally recognized that a risk factor must have
\na very strong relative risk (i.e.,[50) if it is to have utility as
\na screening instrument at the individual level (Rockhill
\net al., 2000; see also Kennaway, 1998). However, others set
\nthe bar higher:
\nA risk factor has to be extremely strongly associated
\nwith a disease within a population before it can be
\nconsidered to be a potentially useful screening test.
\nEven a relative odds of 200 between the highest and
\nlowest fifths will yield a detection rate of no more
\nthan about 56% for a 5% false positive rate\u2026 (Wald
\net al., 1999, p. 1564).
\nTo put this in perspective, the relative risk for the
\nassociation between lung cancer and smoking is between
\n10 and 15 (Rockhill et al., 2000), depending on the definition
\nof exposure. The relative risk for the PCL-R and
\nrecidivism is something of the order of 3 for general
\nrecidivism and 4 for violent recidivism (Hare, 2003).
\nDoes the application of current forensic tools provide an
\nadequate basis for testimony concerning the individual case
\nIn this article, we attempt to answer this question by considering
\nthree issues pertaining to PCL-R data. How
\nconfident can clinicians and legal decision makers be, first, in
\nthe use of critical diagnostic cut-offs; second, in the
\nnumerical value of PCL-R scores; and third, in individual
\npredictions of violent recidivism We describe two studies.
\nThe first study addresses the accuracy of diagnostic decisions
\nand the potential range of discrepancies between two raters.
\nThe second study addresses the accuracy of prediction of
\nfuture violence in the individual case. The results have relevance
\nbeyond the PCL-R to the use of other psychometric
\ninstruments in forensic practice: The same limitations may
\napply to many forensic assessment instruments.
\nSTUDY ONE
\nIn the first study, we examined diagnostic accuracy, specifically
\nthe allocation of individuals around two critical
\ncut-offs, i.e., around 30 and around 25; the first is the
\nstandard PCL-R cut-off for the diagnosis of psychopathy
\nand the second, often adopted in the UK, has proven useful
\nin that context including in decisions regarding treatment
\nallocation (Hare, 2003).
\nThe inter-rater reliability figures presented in the PCL-R
\nmanual can be regarded as good (Nunnally & Bernstein,
\n1994); intraclass correlation coefficient for single ratings
\n(ICC1) are estimated in some research studies as being
\nabove .80 (Male offenders = .86; Male forensic psychiatric
\npatients = .88; Hare, 2003, Table 5.4).3 Edens and
\nPetrila (2006) indicated that these are probably \u2018\u2018best case\u2019\u2019
\nestimates and \u2018\u2018real world\u2019\u2019 reliabilities may be substantially
\npoorer.4 Murrie, Boccaccini, Johnson, and Janke
\n(2008), in one \u2018\u2018real world\u2019\u2019 study, demonstrated poor
\nagreement (ICC1 = .39). These views and findings echo
\nconcerns expressed by Hare (1998), that while researchers
\ntake great pains to ensure reliability in their studies, the
\nlevel of reliability achieved by individual clinicians
\nremains unknown\u2014and by implication\u2014is likely to be
\npoorer than published studies. Inter-rater reliability is not
\nthe only relevant consideration: Diagnostic precision is
\nalso influenced by the underlying distribution of test scores.
\nDiagnostic precision is influenced by the location of the
\ncut-off and the shape of the distribution of scores\u2014both
\nskewness and kurtosis. Estimates of the precision of a test
\nscore (e.g., standard errors of measurement, SEM) are
\nweighted toward the mean of the distribution whereas cutoffs
\nare generally located substantially above the mean.
\nItem Response Theory (IRT) studies demonstrate that the
\nmeasurement precision of the PCL-R\u2014in terms of measurement
\ninformation\u2014falls toward the diagnostic cut-off
\n(Cooke, Michie, & Hart, 2006); thus, the SEM estimated
\non the mean will provide an optimistic estimate of diagnostic
\nprecision.
\nThe SEM cannot be directly translated into estimates of
\nprecision of diagnosis because of the impact of the score
\ndistributions. Equally, it is not possible to estimate misclassification
\nrates directly using ICC1 values; therefore,
\nsimulation approaches are required. Study one describes a
\nsimulation that examines the impact of unreliability on
\ndiagnostic accuracy.
\nMethod
\nMonte Carlo studies allow the investigation of the properties
\nof distributions and estimates of parameters where
\nresults cannot be derived theoretically (Mooney, 1997;
\nRobert, 2004). Large numbers of simulated datasets can be
\n3 The estimates of reliability are frequently obtained by re-rating the
\nsame interview or with an observer simultaneously rating within an
\ninterview. This will tend to inflate reliability, but not validity, as the
\nsame information source is being used.
\n4 The case of THE PEOPLE, Plaintiff and Respondent, v. KURT
\nADRIAN PARKER, a Sexual Violent Predator ACT case highlights
\nthe variability that can emerge in some cases; five accredited experts
\nfurnished five PCL-R scores that ranged from 10 to 25. (Edens, John,
\nPersonal Communication, 22 May 2006).
\nLaw Hum Behav (2010) 34:259\u2013274 261
\n123
\ncreated based on an explicit and replicable data-generation
\nprocess. The effect of known features designed into the
\ndata, such as levels of inter-rater reliability, on outcomes,
\nsuch as diagnostic precision, can be assessed. Multiple
\ntrials of procedures are carried out to allow precise estimation
\nof outcomes. Mooney (1997) argued that Monte
\nCarlo simulations could allow social scientists to test
\nclassical parametric inference methods and provide more
\naccurate statistical models. In our view, this mainstream
\nstatistical technique is underused in forensic research.
\nMaterials
\nWe used Monte Carlo techniques based on distribution
\ninformation from two datasets of PCL-R total scores: (1)
\ndata for North American Male Offenders (Table 9.1, Hare,
\n2003) and (2) data from UK prisoners (Cooke, Michie,
\nHart, & Clark, 2005).5
\nThe first distribution, being the largest, probably provides
\nthe best estimate of the true distribution of scores
\nunderlying the PCL-R and is described as \u2018\u2018approximately
\nnormal\u2019\u2019 (Hare, 2003, p. 55). Given the potential impact of
\na departure from normality we tested whether this distribution
\nwas in fact normal. The departure from normality
\nwas highly significant (Kolmogorov\u2013Smirnov = .068,
\ndf = 5408, p.0001, Skewness = -.33, Kurtosis =
\n-.570). Examination of Fig. 1 demonstrates that around the
\nstandard cut-off of 30 cases are over-represented while in
\nthe right tail of the distribution they are under-represented.
\nIn the simulation study,we generated two randomvariables
\nper case using MATHCAD.13 (2005). These random variableswere
\nscaled according to one of the two datasets referred
\nto above with mean l and standard deviation r.6 This gives
\ntwo uncorrelated ratings (x1 and x2) from the distribution of
\nscores: x1 is our first rating on the subject, PCL1. We then
\ncalculated a linear combination of the two ratings to provide a
\nsecond rating on the same subject, which has a correlation of
\nq with the first rating. The linear combination is PCL2 \u00bc
\nroundf l \u00fe \u00f0x1 l\u00deq \u00fe \u00f0x2 l\u00de
\nffiffiffiffiffiffiffiffiffiffiffiffiffi
\n1 q2
\np
\ng ; using rounding
\nto ensure an integer score. This process gives two random,
\ncorrelated scores from the distribution. There is a very small
\nprobability of obtaining second ratings less than 0 or greater
\nthan 40: These scores have been taken as 0 or 40, respectively.
\nAssuming that the ICC1 represents the best estimate of
\nthe correlation between the two scores, we estimated the
\ndistributions for four values of reliability, i.e., ICC1 values
\nof .75, .80, .85, and .90. The .80 value is a lower bound
\nestimate for reasonable practice. Hare (1998) indicated that
\nat least this level should be achievable with \u2018\u2018\u2026properly
\nconducted assessments\u2019\u2019 (p. 107). The .85 level may be
\nachievable by one rater with good training; the .90 level,
\nperhaps the best case scenario, is the level achievable where
\ntwo independent sets of ratings are averaged. Values above
\n.90 are rarely if ever achievable\u2014Hare (1998) describes .95
\nand higher as \u2018\u2018unbelievably high\u2019\u2019 (p. 107). The .75 provides
\na lower-bound estimate of what may be obtained in
\nclinical practice. These values probably represent optimistic
\nestimates for actual clinical practice; we did not assess the
\n\u2018\u2018worst case\u2019\u2019 scenarios implied by Edens and Petrila (2006)
\nand by Murrie et al. (2008). The estimation procedure was
\nrepeated 10,000,000 times for each of the four levels of
\nICC1 to provide stable estimates of the distribution of the
\ncorrelated ratings and to ensure at least 10,000 cases within
\neach of the extreme score bands. We examine discrepancies
\nin two ways: First, in terms of diagnostic disagreement and
\nsecond, in terms of disagreements about total scores.
\nWhat is the level of diagnostic agreement Kappa (j)
\ncoefficients measure the proportion of diagnostic agreements
\ncorrected for observed base rates (Fleiss, 1981).
\nConventionally, j.75 represents excellent agreement,
\n.40j.75 represents fair to good agreement and
\nj.40 represents poor agreement (Gail & Benichou,
\n2000). Kappa values for three distributions and the four
\nICC1 values are given in Table 1. We calculated Kappa
\ncoefficients for agreement in diagnosis between the two
\nratings using both common cut-offs, i.e., 30 and 25. The
\nvast majority of Kappa values are only in the fair to good
\nrange; few values approach the poor range.
\nKappa is an omnibus statistic, which is useful for
\nsummarizing group results; however, it tells us little about
\nagreement in the individual case. The potential for misclassification
\nis clearer when distributions of disagreements
\n0 10 20 30 40
\nPCL-R
\n0
\n100
\n200
\n300
\nFrequency
\nFig. 1 Distribution of North American male prisoners and normal
\ncurve
\n5 We carried out a similar analysis of data for Male Forensic
\nPsychiatric Patients (Table 9.2, Hare, 2003); the results, which
\ndemonstrate the same pattern, can be obtained from the first author.
\n6 A full description of the simulation study including the Mathcad
\ncode can be obtained from the first author.
\n262 Law Hum Behav (2010) 34:259\u2013274
\n123
\nare considered. The distributions based on the North
\nAmerican Male Offenders are in Table 2. For ease of
\ninterpretation, we tabulated the distributions in 5-point
\nranges. Examination of the sub-table for ICC1 = .80
\nindicates that if one rater gives a score between 30 and 34,
\ni.e., just above the diagnostic cut-off then only in 46% of
\noccasions\u2014approximately half the time\u2014will the other
\nrater obtain a score within the same range. In 44% of the
\noccasions, the second rater would place the individual
\nbelow the critical cut-off. Even in the best case scenario,
\ni.e., ICC1 = .90, if one rater gives a score between 30 and
\n34 then only in 60% of occasions will the other rater obtain
\na score within the same range. On 29% of occasions, the
\nsecond rater would place the participant below the critical
\ncut-off.
\nThe distributions based on the UK prisoners are in
\nTable 3. Examination of the table for ICC1 = .80 indicates
\nthat if one rater gives a score between 30 and 34, i.e., just
\nabove the diagnostic cut-off then only in 39% of occasions
\nwill the second rater obtain a score within the same range. In
\n54% of the cases, the second rater would place the individual
\nbelow the critical cut-off.As previously, even in the best case
\nscenario, i.e., ICC1 = .90, if one rater gives a score between
\n30 and 34 then only in 53% of cases will the other rater obtain
\na score within the same range. In 39% of cases, the second
\nrater would place the participant below the critical cut-off.
\nIn the UK, the cut-off of 25, as well as 30, is often applied
\n(DSPD Programme, 2005; Hare, 2003). Examination of the
\ntable for ICC1 = .80 indicates that if one rater gives a score
\nbetween 25 and 29, i.e., just above theUKdiagnostic cut-off,
\nthen only in 29% of occasions will the other rater obtain a
\nscore within the same range. On 49% of occasions, the second
\nrater would place the individual below the critical cutoff.
\nEven in the best case scenario, i.e., ICC1 = .90, if one
\nrater gives a score between 25 and 29 then only in 37% of
\ncases will the other rater obtain a score within the same
\nrange. On 37% of occasions, the second rater would place the
\nparticipant below the critical cut-off.
\nTherefore, in broad terms, all of the findings reported
\nabove demonstrate that the allocation of an individual
\nabove or below diagnostic cut-offs is much less precise
\nthan previously thought.
\nAnother way of considering the precision of PCL-R
\nscores is to examine expected discrepancies in scores based
\non variations in ICC1 while taking into account the distributional
\ncharacteristics of the PCL-R scores. The PCL-R
\nmanual suggests that in 68% of cases the discrepancies
\nbetween two raters should be up to 3 points, and in 95% of
\ncases it should be up to 6 points (Hare, 2003). This assumes
\nnormality of the PCL-R score distribution, an assumption
\nthat is not met (see above). The cumulative distribution of
\nscore discrepancies estimated from the Monte Carlo studies
\nare tabulated in Table 4. With the North American prisoner
\nsample and an ICC1 of .80, a discrepancy of between 8 and
\n9 points would be expected in 9% of cases, around 10
\npoints in 5% of cases, and between 12 and 13 points in 1%
\nof cases. With the UK prisoner sample, and an ICC1 of .80,
\na discrepancy of between 8 and 9 points would be expected
\nin 23% of cases, around 10 points in 5% of cases, and
\naround 12 points in 1% of cases.
\nAn alternative approach to summarize the range of
\npossible discrepancies is to estimate the distribution of a
\n2nd PCL-R rating given the 1st PCL-R rating. This conditional
\ndistribution can be summarized by a CI that
\ncontains 95% of the 2nd ratings. This interval is thus
\ndefined by the lower and upper limits LL and UL given by
\nProb(LL2nd ratingULj1st rating) \u00bc 0:95:
\nResults for both 68% and 95% CIs for ICC1 = .80, and
\nfor both samples, are presented in Table 5. For example, in
\nthe North American prisoner sample, if rater one obtains a
\ntotal score of 30, then the 95% CI for rater two\u2019s total score
\nwill be between 19 and 36 (i.e., between the 35th and 99th
\npercentile).
\nAll the estimates in this study are conservative; that is,
\nthey assume that the SEM that applies at the mean applies
\nTable 1 Kappa coefficients and levels of agreement for four levels of correlation (q) for two distributions
\nq Both30 Both C 30 Different j Both25 Both C 25 Different j
\nNorth American male offenders
\n0.75 72.9 10.6 16.4 .46 48.0 29.6 22.4 .54
\n0.80 73.3 11.5 15.1 .51 48.9 31.0 20.0 .59
\n0.85 74.4 12.2 13.5 .56 50.5 32.3 17.2 .64
\n0.90 75.5 13.4 11.1 .64 51.7 34.2 14.1 .71
\nUnited Kingdom prisoners
\n0.75 91.8 2.1 6.1 .38 79.8 7.6 12.6 .47
\n0.80 92.0 2.4 5.6 .43 83.1 6.8 10.2 .52
\n0.85 92.4 2.7 5.0 .49 81.2 9.1 9.7 .60
\n0.90 92.7 3.1 4.2 .57 82.0 10.1 7.9 .67
\nLaw Hum Behav (2010) 34:259\u2013274 263
\n123
\naround the cut-off. However, this is an unwarranted
\nassumption. The overall variance of errors of measurement is
\na weighted average of the errors that pertain across the range
\nof true score values. Precision of measurement of the PCL-R
\ndrops as scores approach the diagnostic cut-off (e.g., Cooke
\n& Michie, 1997; Cooke et al., 2006). Thus, the degree of
\ndiagnostic misclassification and score discrepancy is likely
\nto be greater in practice than demonstrated in the simulation
\nabove. The conditional SEM (CSEM)7 is the square root of
\nthe variance of errors at a particular level of true scores. To
\nTable 2 Distribution of diagnostic disagreements by four levels of
\ncorrelation between raters based on distribution of North American
\nmale offenders
\nPCL-R score
\n0\u20134 5\u20139 10\u201314 15\u201319 20\u201324 25\u201329 30\u201334 35\u201340
\nq = 0.75
\n0\u20134 .209 .086 .033 .006 0 0 0 0
\n5\u20139 .505 .305 .200 .079 .008 0 0 0
\n10\u201314 .245 .318 .250 .188 .072 .005 0 0
\n15\u201319 .041 .235 .266 .270 .239 .078 .002 0
\n20\u201324 0 .053 .192 .256 .310 .281 .097 0
\n25\u201329 0 .002 .056 .155 .236 .369 .371 .130
\n30\u201334 0 0 .003 .044 .121 .230 .445 .551
\n35\u201340 0 0 0 .001 .014 .037 .085 .319
\nq = 0.8
\n0\u20134 .245 .096 .032 .002 0 0 0 0
\n5\u20139 .523 .372 .211 .065 .003 0 0 0
\n10\u201314 .217 .315 .288 .198 .055 .001 0 0
\n15\u201319 .014 .198 .280 .306 .240 .055 0 0
\n20\u201324 0 .019 .162 .269 .340 .283 .063 0
\n25\u201329 0 0 .026 .137 .247 .391 .379 .076
\n30\u201334 0 0 0 .023 .106 .237 .465 .585
\n35\u201340 0 0 0 0 .010 .031 .093 .339
\nq = 0.85
\n0\u20134 .285 .105 .021 0 0 0 0 0
\n5\u20139 .552 .423 .217 .038 0 0 0 0
\n10\u201314 .158 .328 .333 .202 .028 0 0 0
\n15\u201319 .005 .141 .303 .354 .231 .024 0 0
\n20\u201324 0 .003 .121 .287 .386 .265 .029 0
\n25\u201329 0 0 .005 .111 .266 .430 .351 .038
\n30\u201334 0 0 0 .008 .085 .252 .519 .569
\n35\u201340 0 0 0 0 .003 .029 .101 .393
\nq = 0.9
\n0\u20134 .361 .103 .009 0 0 0 0 0
\n5\u20139 .578 .486 .215 .012 0 0 0 0
\n10\u201314 .061 .348 .390 .190 .010 0 0 0
\n15\u201319 0 .063 .319 .422 .216 .005 0 0
\n20\u201324 0 0 .067 .304 .449 .238 .004 0
\n25\u201329 0 0 0 .071 .267 .500 .289 .006
\n30\u201334 0 0 0 0 .056 .239 .609 .494
\n35\u201340 0 0 0 0 0 .018 .098 .501
\nThe tables show column percentages, which sum to 1 within rounding
\nerror. The rows therefore do not sum to 1
\nTable 3 Distribution of diagnostic disagreements by four levels of correlation
\nbetween raters based on distribution of UK prisoners
\nPCL-R score
\n0\u20134 5\u20139 10\u201314 15\u201319 20\u201324 25\u201329 30\u201334 35\u201340
\nq = 0.75
\n0\u20134 .395 .158 .064 .020 .002 0 0 0
\n5\u20139 .477 .352 .228 .107 .035 .002 0 0
\n10\u201314 .128 .332 .309 .219 .131 .032 0 0
\n15\u201319 0 .153 .271 .336 .265 .198 .041 0
\n20\u201324 0 .005 .119 .215 .316 .297 .266 .031
\n25\u201329 0 0 .009 .086 .162 .261 .291 .254
\n30\u201334 0 0 0 .017 .083 .191 .328 .529
\n35\u201340 0 0 0 0 .006 .020 .075 .186
\nq = 0.8
\n0\u20134 .438 .163 .059 .011 0 0 0 0
\n5\u20139 .485 .387 .232 .097 .014 0 0 0
\n10\u201314 .077 .354 .328 .229 .111 .016 0 0
\n15\u201319 0 .095 .296 .354 .289 .154 .023 0
\n20\u201324 0 0 .083 .237 .331 .321 .214 .007
\n25\u201329 0 0 .002 .067 .176 .287 .303 .224
\n30\u201334 0 0 0 .005 .077 .200 .387 .545
\n35\u201340 0 0 0 0 .002 .022 .073 .224
\nq = 0.85
\n0\u20134 .470 .168 .042 .003 0 0 0 0
\n5\u20139 .474 .437 .224 .069 .003 0 0 0
\n10\u201314 .056 .342 .375 .226 .075 .003 0 0
\n15\u201319 0 .053 .308 .404 .287 .106 .001 0
\n20\u201324 0 0 .050 .251 .383 .325 .140 0
\n25\u201329 0 0 0 .047 .194 .321 .328 .145
\n30\u201334 0 0 0 0 .058 .229 .447 .578
\n35\u201340 0 0 0 0 0 .017 .084 .277
\nq = 0.9
\n0\u20134 .530 .169 .020 0 0 0 0 0
\n5\u20139 .450 .501 .219 .033 0 0 0 0
\n10\u201314 .019 .312 .457 .207 .037 0 0 0
\n15\u201319 0 .018 .286 .489 .275 .050 0 0
\n20\u201324 0 0 .017 .252 .450 .318 .062 0
\n25\u201329 0 0 0 .018 .214 .367 .325 .046
\n30\u201334 0 0 0 0 .024 .257 .528 .587
\n35\u201340 0 0 0 0 0 .008 .085 .367
\n7 Professional standards indicate that the CSEM is an important piece
\nof information that should be provided in a test manual. For example,
\nStandard 2.14 \u2018\u2018Conditional standard error of measurements should be
\nreported at several score levels if constancy cannot be assumed.
\nWhere cut scores are specified for selection or classification, the
\nstandard errors of measurement should be reported in the vicinity of
\neach cut score.\u2019\u2019 (American Educational Research Association\/
\nAmerican Psychological Association, 1999; p. 35 emphasis added).
\n264 Law Hum Behav (2010) 34:259\u2013274
\n123
\nevaluate the true level of agreement of diagnosis likely to
\napply around a cut-off it is necessary to take the CSEM into
\naccount.
\nItem Response Theory indicates that the error of measurement
\nvaries with location on the trait (h).
\nIRT gives
\nSE\u00f0h\u00de \u00bc
\n1
\nffiffiffiffiffiffiffiffi
\nI\u00f0h\u00de
\np
\nwhere I(h) is the information at h.
\nCTT gives
\nSEM \u00bc SD
\nffiffiffiffiffiffiffiffiffiffiffi
\n1 q
\np
\nLet q1 be the correlation at location 1 (h1), q2 be the
\ncorrelation at location 2 (h2).
\nThen
\nq2 \u00bc 1 \u00f01 q1\u00de
\nI\u00f0h1\u00de
\nI\u00f0h2\u00de
\nLocation 1 is h = 0.0 (PCL-R = 20) and Location 2 is
\nh = 1.0 (PCL-R = 30) (Approximate locations from Hare,
\n2003; Fig. 6.6; see also Cooke & Michie, 1997). Overall,
\nthe impact of the location of the estimated ICC1 is limited,
\ndropping\u2014at a maximum\u2014from .75 to .69. However, as
\nnoted above, even small drops in ICC1 (e.g., from .85 to
\n.80) can substantially affect the misclassification rate and
\nthe range of likely score discrepancies (see Table 6). It is
\nnoteworthy that the magnitude of the drop appears to be
\nproportionately larger the poorer the mean estimated level
\nof inter-rater reliability. This suggests that the effect of the
\nCSEM is larger in cases that start with a relatively poor
\nlevel of inter-rater reliability. Equally, this would suggest
\nthat proportionately greater discrepancies would, in
\ngeneral, be obtained when factor or facet scores are
\nconsidered because they have lower levels of reliability
\nthan the total scores (Hare, 2003).
\nSTUDY TWO
\nThe use of the PCL-R in court is frequently justified based
\non its predictive utility, the support being garnered from
\nbetween-subject designs (Edens & Petrila, 2006; Hare,
\n2003; Walsh & Walsh, 2006). In this study, we are concerned
\nwith the individual. We examine the confidence that
\ncan be placed in a prediction that an individual with a
\nparticular PCL-R score will be reconvicted for a violent
\noffence.
\nAll measurements and estimates entail error. As noted
\nabove, the degree of error is expressed by CIs. For
\nTable 4 Cumulative distribution of expected discrepancies between two raters for different levels of correlation based on two sample
\ndistributions
\nPoint discrepancy SEMa North American male offenders United Kingdom prisoners
\nCorrelation Correlation
\n0.75 0.80 0.85 0.90 0.75 0.80 0.85 0.90
\n0 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
\n1 .741 .934 .927 .919 .901 .936 .928 .918 .900
\n2 .503 .804 .785 .758 .704 .804 .785 .758 .703
\n3 .317 .679 .647 .595 .518 .681 .647 .595 .518
\n4 .303 .560 .516 .453 .352 .561 .518 .454 .357
\n5 .095 .449 .397 .327 .217 .452 .400 .332 .222
\n6 .046 .349 .292 .215 .121 .353 .298 .221 .125
\n7 .020 .262 .205 .136 .058 .269 .212 .141 .063
\n8 .007 .190 .137 .079 .023 .196 .142 .083 .026
\n9 .002 .132 .086 .041 .007 .137 .090 .044 .009
\n10 .001 .088 .050 .018 .002 .091 .054 .021 .003
\n11 .055 .026 .007 .058 .030 .009 .001
\n12 .032 .012 .002 .035 .015 .004
\n13 .017 .005 .019 .007 .001
\n14 .008 .002 .010 .003
\n15 .003 .005 .001
\n16 .001 .002
\n17 .001
\na This column shows the cumulative distribution of discrepancies which was calculated assuming that discrepancies between two raters are
\nnormally distributed and that the SEM is 3 (Hare, 2003, pp. 66\u201367)
\nLaw Hum Behav (2010) 34:259\u2013274 265
\n123
\nexample, while the mean rate of reoffending for a \u2018\u2018High
\nRisk\u2019\u2019 group may be estimated as being 55%; the 95% CI
\nindicates that the true value of the mean rate or reoffending
\nfor this group will lie between 44% and 66%, 95% of the
\ntime, i.e., 19 times out of 20 (Hart, Michie, & Cooke,
\n2007). However, the clinician and the decision maker are
\ninterested in the individual case not the group. Therefore,
\nhow much confidence can the clinician and decision maker
\nhave in predictions of reoffending in the individual case
\nbased on PCL-R scores We examine CIs for group and
\nindividual predictions.
\nParticipants
\nTwo hundred fifty-five male prisoners between 18 and
\n40 years of age (M = 26.8, SD = 5.9) were interviewed in
\nScotland\u2019s largest prison for a study of psychological
\ncharacteristics and violence (Cooke, Michie, & Ryan,
\n2001; Michie & Cooke, 2006). Prisoners were selected by
\nsystematic random sampling of the prison. The average
\nsentence length was 39 months (SD = 23 months; range
\n= 3 months to 10 years and life).
\nPCL-R Ratings
\nPCL-R ratings were made according to instructions in the
\ntest manual (Hare, 1991). All PCL-R Assessments were
\nconducted by trained raters using both interview and file
\nreview (ICC1 = .86).
\nAssessment of Recidivism
\nReconviction data were obtained from two sources: The
\nScottish Criminal Records Office (SCRO) and the Police
\nNational Computer (PNC). The average follow-up period
\nwas 29 months. The point-biserial correlation between
\nPCL-R scores and recidivism (r = .31) was above average
\nfor the field (Walters, 2003). For the purposes of illustration,
\nwe consider reconviction for violence that resulted in
\na prison sentence (i.e., generally a more serious violent
\noffence). Follow-up data were available for 190 cases and
\nPCL-R data for 184 of these.
\nTable 5 The 68% and 95% confidence intervals for 2nd PCL-R total
\nscore given 1st PCL-R score and ICC = 0.8
\n1st PCLR
\nPrisoners UK
\nLL
\n.95
\nLL
\n.68
\nUL
\n.68
\nLL
\n.95
\nLL
\n.95
\nLL
\n.68
\nUL
\n.68
\nLL
\n.95
\n0 0 0 9 12 0 0 8 13
\n1 0 0 10 13 0 0 9 14
\n2 0 1 11 14 0 0 10 15
\n3 0 2 12 15 0 1 11 16
\n4 0 3 12 15 0 1 12 16
\n5 0 4 13 16 0 2 12 17
\n6 0 4 14 17 0 3 13 18
\n7 0 5 15 18 1 4 14 19
\n8 1 6 16 19 2 5 15 20
\n9 2 7 16 19 2 5 16 20
\n10 3 8 17 20 3 6 16 21
\n11 4 8 18 21 4 7 17 22
\n12 4 9 19 22 5 8 18 23
\n13 5 10 20 23 6 9 19 24
\n14 6 11 20 23 6 9 20 24
\n15 7 12 21 24 7 10 21 25
\n16 8 12 22 25 8 11 21 26
\n17 8 13 23 26 9 12 22 27
\n18 9 14 24 27 10 13 23 28
\n19 10 15 24 27 10 13 24 28
\n20 11 16 25 28 11 14 24 29
\n21 12 16 26 29 12 15 25 30
\n22 12 17 27 30 13 16 26 31
\n23 13 18 28 31 14 17 27 32
\n24 14 19 28 31 14 17 28 32
\n25 15 20 29 32 15 18 28 33
\n26 16 20 30 33 16 19 29 34
\n27 16 21 31 34 17 20 30 35
\n28 17 22 32 35 18 21 31 36
\n29 18 23 32 35 18 21 32 36
\n30 19 24 33 36 19 22 32 37
\n31 20 24 34 37 20 23 33 38
\n32 20 25 35 38 21 24 34 39
\n33 21 26 36 38 22 25 35 40
\n34 22 27 36 39 22 25 36 40
\n35 23 28 37 40
\n36 24 28 38 40 24 27 37 40
\n37 24 29 39 40 25 28 38 40
\n38 25 30 40 40 26 29 39 40
\n39 26 31 40 40
\n40 27 32 40 40
\nTable 6 Values of conditional standard error of measurement at
\ndiagnostic cut-off of 30 for different values of SEM and distributions
\nof the three samples
\nSEM q1 CSEM q2
\nNorth American male prisoners 0.75 0.70
\n0.80 0.76
\n0.85 0.82
\n0.90 0.88
\nUnited Kingdom prisoners 0.75 0.67
\n0.80 0.74
\n0.85 0.80
\n0.90 0.87
\n266 Law Hum Behav (2010) 34:259\u2013274
\n123
\nAnalysis
\nThere are standard methods for estimating CIs for groups;
\nhowever, methods for estimating CIs for the individual
\ncase are not generally covered in the standard statistical
\ntexts used in psychology and they may, we suspect, be
\nunfamiliar to the majority of psychologists. We explicate
\nthe method here. First, we consider the general case of CI
\nestimation before considering the specific approach based
\non linear logistic regression used for our analysis.
\nAny CI has the general form:
\nEstimate t (Estimate of Standard error)
\nwhere t is the Student\u2019s t-statistic with the appropriate
\ndegrees of freedom.
\nSuppose we are interested in a single variable, e.g.,
\nx = IQ, and have taken a sample of size n (x1; x2; . . .; xn) to
\nestimate the mean and variance of IQ in the population of
\ninterest, then the sample mean \u00f0x\u00de is the estimate of the
\npopulation mean. The accuracy of this estimate is given by
\na CI
\nx tn
\nffiffiffiffi
\ns2
\nn
\nr
\nwhere s is an estimate of the standard deviation of x.
\nSuppose we are now interested in predicting the next
\nobservation in the population, xn1. Then a CI for the
\nprediction (i.e., the prediction interval) is given by
\nx tn
\nffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
\ns2 1 \u00fe
\n1
\nn<\/p>\n
s
\n:
\nNote that the estimate of both the mean and the
\nprediction is x but that the prediction interval is (much)
\nwider than the CI for the mean. Note also that the size of
\nthe sample from which the model was derived has little
\ninfluence on the width of the prediction interval.
\nIn the linear regression situation, we have a sample of n
\npairs of observations (\u00f0x1; y1\u00de; \u00f0x2; y2\u00de; . . .; \u00f0xn; yn\u00de) from
\nwhich we estimate the intercept and slope of the line by B0
\nand B1 in the usual way. The accuracy of estimation of the
\nline would be given by the CI for the mean y for a given x.
\nThis is calculated in the standard manner (Steel, Torrie, &
\nDickey, 1997).
\nyL; yU \u00bc B0 \u00fe B1x tn
\nffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
\n^r2
\n1
\nn
\n\u00fe
\n\u00f0x x2\u00de
\nSS\u00f0X\u00de<\/p>\n
s
\nIf we have a new case for which we know the x-value,
\nxn1 and wish to predict the y-value, this is given by
\n^yn\u00fe1 \u00bc B0 \u00fe B1xn\u00fe1
\nwhich is the mean value of y for the given x. The CI for this
\nprediction (i.e., the prediction interval) is given by
\nyL; yU \u00bc B0 \u00fe B1xn\u00fe1 tn
\nffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
\n^r2 1 \u00fe
\n1
\nn
\n\u00fe
\n\u00f0xn\u00fe1 x2\u00de
\nSS\u00f0X\u00de<\/p>\n
s
\nAgain this prediction interval is much wider than the
\ninterval for the line and is not influenced to any significant
\ndegree by the size of the sample from which the model was
\ndeveloped. The square root term is the standard error of the
\npredicted value. Here, the expression in brackets takes into
\naccount three sources of error. The first is the variability in
\nparticipants, the second is the error in the estimate of
\nvariance \u00f0^r2\u00de; and the third allows for the fact that the error
\nin prediction varies with distance from the mean PCL-R
\nscore.
\nLinear logistic regression is the appropriate method for
\nmodeling the prediction of a binary outcome (e.g.,
\nreconviction). In linear logistic regression the model is
\ngiven by
\nPr\u00f0event\u00de \u00bc
\n1
\n1 \u00fe eZ
\nwhere
\nZ \u00bc B0 \u00fe B1x
\nWe have a linear regression of Z on x so the equation for
\nthe CI for Z is the same as the linear regression case. A
\nprediction interval for Z for a new individual from the same
\npopulation with score x0 can be constructed by ZL and ZU
\n(lower and upper values, respectively) from the equation
\nZL; ZU \u00bc B0 \u00fe B1x0 tn
\nffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
\n^r2 1 \u00fe
\n1
\nn
\n\u00fe
\n\u00f0x0 x\u00de2
\nSS\u00f0X\u00de
\n!
\nv
\nu
\nu
\nt
\nand the interval is then transformed in the following
\nmanner. Since the probability is a monotonic function, the
\nprediction interval for the probability is given by
\n1
\n1 eZL
\n;
\n1
\n1 \u00fe eZU<\/p>\n
(note, once again, that the size of sample on which the
\nmodel was developed has little influence on the width of
\nthe prediction interval).
\nInitially, we estimated the linear regression of Z on the
\nPCL-R total score (Fig. 2), and then estimated the mean
\nprobability of reconviction by PCL-R score (Fig. 3). This is
\na monotonic function with the probability of reconviction
\naccelerating with increasing PCL-R score. Those with an
\naverage PCL-R score (i.e., 12.5) had a 14% probability of
\nbeing reconvicted for a violent crime and being sentenced
\nto prison. Examination of the 95% CIs for the estimate of
\nthe mean rate of reconviction indicated that for an average
\nPCL-R score of 12.5 the true probability of reconviction
\nwas between 10% and 20% (19 times out of 20) and for a
\nPCL-R score of 25 the 95% CI was 18\u201354%. For a score of
\nLaw Hum Behav (2010) 34:259\u2013274 267
\n123
\n30, the 95% CI was 21\u201370% demonstrating that 95% CIs
\ngenerally widen the further scores are from the mean.
\nFundamentally, however, within the clinical or judicial
\ncontext, the individual\u2014not the group\u2014is the focus of
\ndecision making. Therefore, we estimated the CIs for the
\nlikelihood that an individual would be reconvicted using
\nthe method outlined above. For an individual with a mean
\nPCL-R score of 12.5, the best estimate was that he will
\nreoffend 14% of the time; but this estimate was very
\nimprecise because the 95% CI was 0\u201398%, i.e., the true
\nvalue of the prediction would lie in this range 95% of the
\ntime. For an individual with a score of 25, the 95% CI was
\n0\u201399% and for an individual with a score of 30 the 95% CI
\nwas 0\u201399.5%.
\nTo illustrate the extent of the uncertainty associated with
\nan individual prediction, we calculated the probability
\ndensity function associated with a prediction that an individual
\nwith a PCL-R score of 25 would return to prison
\nwithin 2 years for a violent crime (point estimate .33). The
\nprobability function is a means of describing degree of
\nuncertainty. It can be viewed as a smoothed version of a
\nhistogram depicting relative frequencies of the range of
\nprobabilities of reoffending consequent on the variability in
\nthe original sample. Figure 4 displays a relatively flat
\nprobability density function around the point estimate of
\n.33, with values ranging from 0 to 1.0, indicating that a
\nbroad range of values is likely in any individual case.
\nOne anonymous reviewer made the compelling case that
\na more liberal definition of harmful behavior that included
\nmore forms of offending should be considered. We carried
\nout the same analyses with three other outcome variables,
\ni.e., convictions for any crime or offence within 2 years of
\nrelease; any convictions leading to incarceration and any
\nconviction for a violent crime over the same time period.
\nFigure 5 displays results for any convictions within 2 years
\nof release and reveals the same pattern as for violent crime:
\nThe CI associated with the regression line being much
\nnarrower than the prediction interval.8
\nAnother anonymous reviewer suggested that our results
\nmight be due to sample size (this is highly unlikely given
\nthe mathematical basis for the analysis, see above) or
\nbecause of where the sample was drawn. We carried out
\nfurther analyses to clarify these points using data from the
\nMacArthur study of Mental Disorder and Violence (Monahan
\net al., 2001). Psychopathy as measured using the
\nPsychopathy Checklist: Screening Version (PCL:SV; Hart,
\nCox, & Hare, 1995) was the strongest risk factor for future
\nviolence in that study, i.e., violence in the 20 weeks following
\ndischarge; the sample size with PCL:SV ratings was
\nover four times the Scottish sample (n = 860). The pointbiserial
\ncorrelation between PCL:SV scores and recidivism
\nwas similar to the equivalent correlation in the Scottish
\nsample (r = 0.34, cf. r = 0.31). Figure 6 displays essentially
\nthe same pattern as the Scottish data: The slope is
\nsimilar in shape to the Scottish curve, a slowly accelerating
\ncurve with risk of violence increasing with PCL:SV score.
\nAs would be expected from consideration of the equations
\nabove increasing the sample size (among other things) has
\nresulted in a narrower CI around the regression line. But
\ncritically\u2014as would be expected from the mathematics\u2014
\nincreasing the sample size does not result in a narrower
\nprediction interval.
\nPerhaps a nonpsychological example may facilitate
\nexplanation. Given someone\u2019s height, how well can we
\npredict his or her weight This example has several
\nadvantages for the purpose of illustrating the pervasive
\nnature of the problem of predicting in the individual case.
\nFirst, the reliability of the measurement of height and
\n0 8 16 24 32 40
\nPCL Total Score
\n0
\n3
\n6
\n9
\n-3
\n-6
\n-9
\nZ
\nZ
\nMean
\nPrediction
\nFig. 2 Group and individual CIs for linear regression of Z on PCL-R
\nTotal Score
\n0 8 16 24 32 40
\nPCL-R total score
\n0
\n0.2
\n0.4
\n0.6
\n0.8
\n1
\nProbability of recidivism
\nProbability
\nMean
\nPrediction
\nFig. 3 Group and individual CIs around of prediction of violent
\nreoffending resulting in return to prison based on PCL-R score
\n8 Detailed descriptions of the results for the additional three outcome
\nvariables can be obtained from the first author.
\n268 Law Hum Behav (2010) 34:259\u2013274
\n123
\nweight should be substantially higher than the measurement
\nof either psychopathy or violent behavior. Second,
\nthe prediction is immediate and not degraded by the passage
\nof time. Third, the relationship between height and
\nweight is stronger than that between psychopathy and
\nviolent behavior. How well can we predict in the individual
\ncase under these more benign conditions We carried out a
\nMonte Carle simulation9 based on two sets of findings: The
\nheight of males in the UK is normally distributed (Guilford,
\nRona, & Chinn, 1992); and the relationship between height
\nand weight can be assumed to be linear (Hawthorne,
\nMurdoch, & Womersley, 1979). Figure 7 presents the linear
\nregression of weight on height with a sample of 2000.
\nThe CI of the regression line is very narrow; however,
\nwhen the prediction interval is calculated for an individual
\nit is very wide. For example, for an individual of average
\nheight (i.e., 1.75 m) his predicted weight would be 81.5 kg
\nbut the prediction interval is between 61.3 and 101.8 kg\u2014a
\nrange of around 40 kg.
\nIn conclusion, the results demonstrate that PCL-R (and
\nPCL:SV) scores provide little reliable information about the likelihood that an individual will reoffend violently.10
\nThis is not a problem peculiar to the PCL-R but will reflect
\nindividual variability on any scale (e.g., VRAG, Quinsey
\net al., 1998; Static-99, Hanson & Thornton, 1999; COVR,
\nMonahan et al., 2005).
\nDISCUSSION
\nOne broad conclusion can be drawn from these two studies:
\nClinicians must be extremely cautious in what they claim
\n0 8 16 24 32 40
\nPCL-R total score
\n0
\n0.2
\n0.4
\n0.6
\n0.8
\n1
\nProbability of reconviction
\nMean
\nPrediction
\nProb
\nFig. 5 Group and individual CIs around of prediction of any
\nreoffending based on PCL-R score
\n0 5 10 15 20
\nPCL:SV
\n0.0
\n0.2
\n0.4
\n0.6
\n0.8
\n1.0
\nProbability
\nProbability of violence
\nConfidence interval for line
\nConfidence interval for prediction
\nFig. 6 Group and individual CIs around the prediction of violence in
\nthe 20 weeks following discharge: Data from MacArthur Study of
\nMental Disorder and Violence
\n0 0.2 0.4 0.6 0.8 1
\nProbability of recidivism
\n0
\n0.03
\n0.06
\n0.09
\n0.12
\n0.15
\nProbability density
\nFig. 4 Probability density function of the probability of a return to
\nprison on conviction of a violent crime for a PCL-R score of 25
\n9 Following Guilford et al. (1992), the height of adult males in the
\nUK between 1973 and 1988 was shown to be normally distributed
\nwith a mean of approximately 1.75 m (SD = 0.07). The relationship
\nbetween weight and height for men aged 40\u201359 can be shown to be
\nlinear with the relationships for nonsmokers being Weight = 82.7
\nHeight = 63.4 with r = 0.50 (Hawthorne et al., 1979). A sample of
\n2,000 pairs of height and weight were generated using Mathcad. For
\neach subject, a height H was generated from a N(1.75, 0.49)
\ndistribution. A predicted weight was then calculated; a weight W was
\ngenerated by adding an error from a N(0,98.4) distribution. The
\ncorrelation between height and weight for this sample of 2,000 cases
\ndid not differ significantly from that reported in Hawthorne et al.
\n(1979) (0.50 vs. 0.499). The linear regression of weight on height
\ntogether with the CI for the regression line and the prediction interval
\nfor an individual whose height was known were calculated and
\npresented in Fig. 7. Linear regression rather than logistic regression
\nwas used because both variables are continuous. The basic distinction
\nbetween confidence intervals and prediction intervals remains the
\nsame.
\n10 We do not consider additional issues that would add \u2018\u2018noise\u2019\u2019 into
\nthe system including recalibration of the PCL-R in a new jurisdiction
\nin terms of the metric equivalence\u2014or otherwise\u2014of the scores
\n(Cooke et al., 2005), the differences of reliability in clinical practice
\nagainst research settings, and variations in the predictive validity of
\nthe PCL-R in a setting where detection and conviction rates may be
\ndifferent, etc.
\nLaw Hum Behav (2010) 34:259\u2013274 269
\n123
\nregarding diagnoses, numerical scores, and risk potential of
\nindividual clients based merely on a PCL-R score. First,
\nallocation above and below key diagnostic cut-offs (i.e., 30
\nor 25 on the PCL-R) is subject to far greater variability than
\npreviously demonstrated. Second, the precision of numerical
\nscores is less than previously considered. Third, the
\nclinician can have little confidence in statistical predictions
\nregarding an individual\u2019s likelihood of future offending
\nbased on a PCL-R score or the scores of violence risk
\nassessment instruments. Fourth, the concatenation of these
\ntwo sources of imprecision\u2014score and predictive\u2014is
\nlikely to further intensify uncertainty about what any one
\nindividual will do in the future.
\nWe emphasize again that these problems are not unique
\nto the PCL-R: The shape of underlying score distributions
\nwill influence the precision of any scores estimated or any
\ndiagnoses derived. Statistical predictions about individuals
\nwill always be poor (Hart et al., 2007). As noted above, all
\npsychological tests used in the same way in the forensic
\narena may suffer from similar limitations (e.g., VRAG,
\nQuinsey et al., 1998; Static-99, Hanson & Thornton, 1999;
\nCOVR, Monahan et al., 2005).
\nNeither are these problems unique to psychology. They
\nbedevil\u2014as our height and weight example demonstrates\u2014
\nany attempts to use group data to predict individual outcomes
\naccurately, whether the outcome is, for example,
\nheart attacks, cancer, juvenile delinquency, or recidivism
\n(Copas & Marshall, 1998; Elmore & Fletcher, 2006; Rose,
\n1992; Scott, 2003). The problem reflects inherent human
\nvariation.
\nThere are perhaps two broad findings to note when it
\ncomes to considering the precision of our estimates of trait
\nstrength (or indeed, diagnosis). First, the use of aggregate
\nstatistics (e.g., Kappa or ICC1) to measure agreement, or to
\ninfer precision of our measurement processes, can obscure
\nclinically important imprecision at the level of the individual.
\nSecond, untested assumptions (e.g., that scores are
\nnormally distributed) can be misleading when it comes to
\nestimating the precision of our estimates. The findings from
\nthe Monte Carlo study (Study 1, described above) whether
\nexpressed in terms of diagnostic agreement, score disagreement,
\nor range of score discrepancies may provide
\nsome explanation for the growing evidence of clinically
\nsignificant discrepancies in PCL-R ratings in forensic settings
\n(Boccaccini, Turner, & Murrie, 2008; Edens &
\nPetrila, 2006; Murrie et al., 2008; Murrie, Boccaccini,
\nTurner, et al., in press).
\nEthical forensic practice requires practitioners to maximize
\ntheir reliability. There are no panaceas but four steps
\nmay Help. The first step is ongoing education and training,
\nnot only regarding the research base of tests and measures
\nused in forensic practice, but also regarding advanced
\nclinical skills. Advanced clinical skills would include
\ntechniques for interviewing these challenging individuals to
\nensure the collection of relevant information; these
\nskills would also include techniques for generating case
\nformulations to ensure the appropriate application of the
\ninformation collected (Cooke, 2008, 2009a; Logan &
\nJohnstone, 2008). The second step is ensuring the availability
\nof comprehensive file information. The quality of file
\ninformation influences both the magnitude and reliability of
\nscores (Alterman, Cacciola, & Rutherford, 1993). The third
\nstep is the use of multiple raters in high stakes cases; average
\nratings should be eschewed, consensus ratings should be
\nsought. The fourth step is the implementation of audit systems\u2014
\nincluding peer review\u2014for the detection of rater
\ndrift (Cooke, 2009b).
\nDeriving Inferences About Individuals
\nfrom Inferences About Groups
\nWe recognize that some of our conclusions may be surprising\u2014
\nperhaps even controversial\u2014as there is a
\nwidespread acceptance of the prediction paradigm. However,
\nshould we be surprised that we find it difficult to
\npredict what any individual will do in the future Consider
\njust some of the factors that affect predictive accuracy: The
\nlack of reliability in the predictor and outcome variables;
\nthe relative weakness of the association between these
\nvariables; the inherent variability across individuals\u2014and
\nwithin individuals and their circumstances across time\u2014
\nand the multitudinous causes that result in violent crime.
\nPerhaps we have become over-confident. Studies of judgment
\nunder uncertainty have indicated human tendencies
\nboth to be overconfident in predictions (Kahneman &
\nTversky, 1973) and overly narrow in CI estimates (Alpert
\n& Raiffa, 1982). Professionals are not immune from these
\nbiases.
\n1.6 1.7 1.8 1.9
\nHeight (m)
\n40
\n60
\n80
\n100
\n120
\nWeight (kg)
\nPredicted Weight
\nMean
\nPrediction
\nFig. 7 Group and individual CIs around the prediction of weight
\nfrom knowledge of an individual\u2019s height: Simulation with n = 2,000
\n270 Law Hum Behav (2010) 34:259\u2013274
\n123
\nThe findings we present about predictions in individual
\ncases reflect a problem of inference that is long recognized
\nin psychology and other disciplines more generally (e.g.,
\nAltman & Royston, 2000; Henderson & Keiding, 2005).
\nDiscussing child development, Lewin (1931; as cited in
\nRichters, 1997) noted \u2018\u2018An inference from the average to
\nthe particular case is \u2026 impossible.\u2019\u2019 (Richters, 1997, p.
\n199). Discussing the medical application of prognostic
\nmodels, Altman and Royston (2000) noted \u2018\u2018\u2026the distinction
\nbetween what is achievable at the group and
\nindividual levels is not well understood\u2019\u2019 (p. 454). The
\nproblem pertains even under ideal conditions: Henderson
\nand Keiding (2005), discussing survival time prediction in
\nrelation to virulent non-small-cell lung cancer, indicated
\n\u2018\u2018\u2026the intrinsic statistical variations in life times are so
\nlarge that predictions based on statistical models and
\nindices are of little use for individual patients. This applies
\neven when the prognostic model is known to be true and
\nthere is no statistical uncertainty in parameter estimation\u2019\u2019
\n(p. 703). Why is this so
\nConfidence Intervals and Prediction Intervals
\nAs we indicated in our exegesis of the statistical principles
\nunderlying this problem, the CIs for model parameters are
\ndifferent from the CIs around the prediction for a new case.
\nThe latter always being substantially wider than the former.
\nAlso, prediction intervals are little influenced by the size of
\nthe sample used to develop the statistical model (Steel
\net al., 1997). Collecting bigger samples is not a solution.
\nWe demonstrated this empirically by contrasting the
\nScottish sample with the MacArthur sample.
\nThe distinction between CIs and prediction intervals is
\nmade in other areas of assessment, e.g., intelligence testing.
\nAn example may demonstrate the pervasiveness of the
\nprediction problem when applied to the individual. The
\nWechsler Abbreviated Scale of Intelligence (WASI; Psychological
\nCorporation, 1999) is a brief test of intellectual
\nfunctioning, which can be used to predict an individual\u2019s
\nperformance on the \u2018\u2018gold standard\u2019\u2019 Wechsler Intelligence
\nScale for Children\u2014Third Edition (WISC-III; Wechsler,
\n1991). Note that these are very reliable tests, note also, they
\nmeasure within the same conceptual domain (using very
\nsimilar procedures), and that the correlation between the two
\ntests is very high (Full-scale IQ r = .87). An individual
\nassessed on the WASI with a Full Scale IQ of 70 (90% CI
\n66\u201376) will have a predicted WISC-III Full Scale IQ of 70
\n(90% Prediction Interval 62\u201387; Psychological Corporation,
\n1999). Thus, even in these ideal conditions\u2014the same
\nconceptual domain, highly reliable tests that are highly
\ncorrelated\u2014the prediction interval is 2.5 times greater than
\nthe equivalent CI. It is not surprising that the difference
\nbetween the CI and the prediction interval is even greater
\nwhen the link between PCL-R scores and future violence is
\nconsidered. In the Scottish sample, for a mean PCL-R score
\nthe best estimate of the probability of reoffending violently
\nis 14%, the CI is between 10% and 20%, whereas the prediction
\ninterval is between 0% and 98%. In this case, the
\nprediction interval is almost ten times the CI.
\nThis problem of moving from the general to the specific
\nis not merely a matter of statistics; it is also a matter of
\nlogic (Haje\u00b4k & Hall, 2002; Hart et al., 2007). The application
\nof between-subject information to guide withinsubject
\ncausal inference is subject to the logical fallacy of
\ndivision (Rorer, 1990). One form of this fallacy rests on
\ndrawing an invalid conclusion about an individual member
\nof a group based on the collective properties of the group.
\nFor example, it is obviously fallacious to argue that if, in
\ngeneral, intelligent people earn more than less intelligent
\npeople then Jules, with an IQ of 120, will earn more than
\nJim with an IQ of 100. Equally, it is fallacious to argue that
\nalthough, in general, people who score highly on the PCLR
\nre-offend more than people who do not score highly, Bill
\nwith an PCL-R score of 30 will re-offend more often than
\nBrian with a PCL-R score of 10. A common defense of the
\nactuarial approach is founded upon this fallacy. \u2018\u2018If it is
\nalright for life insurance companies, it should be alright for
\npsychology.\u2019\u2019 The analogue is false. The actuary makes a
\nprofit by predicting the proportion of insured lives that will
\nend in a particular time period: The actuary has no interest
\nin predicting the deaths of particular individuals.
\nThere is a growing awareness in psychology that
\nbetween-subject models cannot test or support causal
\naccounts (e.g., pertaining to earning potential or violence)
\nthat are valid at the individual level (Borsboom, Mellenbergh,
\n& van Heeran, 2003; Richters, 1997). With a
\nbetween-subjects design it is possible to argue legitimately
\nthat within population differences in psychopathy can cause
\ndifferences in population differences in violent reoffending.
\nHowever, this position cannot be defended at the level of the
\nindividual; this is because there is an unspoken assumption
\nthat the mechanisms that operate at the level of the individual
\nalso explain variations between individuals. Richters
\n(1997) clarified the basis of the problem:
\nThe extraordinary human capacity for equifinal and
\nmultifinal functioning, however, render the structural
\nhomogeneity assumption untenable. Very similar
\npatterns of overt functioning may be caused by
\nqualitatively differing underlying structures both
\nwithin the same individual at different points in time,
\nand across different individuals at the same time
\n(equifinality) (pp. 206\u2013207).
\nIndividuals are violent for different reasons: Any one
\nindividual may be violent for different reasons on different
\noccasions.
\nLaw Hum Behav (2010) 34:259\u2013274 271
\n123
\nIn summary, on the basis of empirical findings, statistical
\ntheory, and logic it is clear that predictions of future
\noffending cannot be achieved, with any degree of confidence,
\nin the individual case.
\nCONCLUSION
\nWe emphasize again that the problems identified in this
\narticle are not unique to the PCL-R. In some sense our
\nability to demonstrate these problems with the PCL-R is a
\nreflection of the success of this test: It is used extensively
\nand thus large datasets are available; It has been subject to
\nconsiderable psychometric Assessment. Other tools used in
\nforensic settings will be subject to similar limitations. For
\nexample, the precision with which individuals can be
\nallocated to risk \u2018\u2018bins\u2019\u2019 by actuarial risk tools is influenced
\nby the reliability of scoring and the underlying distribution
\nof scores. Faigman (2007) argued that psychology has
\nignored the problem of translating scientific research into
\nfindings that help triers of fact; he indicates that psychology
\nhas to take on the \u2018\u2018monumental intellectual challenge\u2019\u2019
\n(p. 313) of making the inferential leap between populationlevel
\nfindings and individual-level findings relevant to
\ncourts. We ignore this challenge at our peril. Tentative
\nsteps toward meeting this challenge are discussed elsewhere
\n(Cooke, 2009b).
\nThis article is not without limitations. First, it is based on
\nmales. We know little about the reliability of the diagnosis
\nand predictive utility in females or, indeed, whether the
\ninstrument functions adequately in females or other populations
\n(Forouzan & Cooke, 2005; Verona & Vitale, 2006).
\nSecond, the study is focused on adults. The potential for lifechanging
\ndecisions may be even greater when related procedures
\nare applied to adolescents; less information is
\ngenerally available to make a diagnosis in adolescents
\n(Edens & Petrila, 2006). The methods we used are explicated
\nin detail in this article so that others can apply them to
\ntheir own\u2014hopefully diverse\u2014datasets.
\nREFERENCES
\nAlpert, M., & Raiffa, H. (1982). A progress report on the training of
\nprobability assessors. In D. Kahneman, P. Slovic, & A. Tversky
\n(Eds.), Judgment under uncertainty: Heuristics and biases (pp.
\n294\u2013305). New York: Cambridge University Press.
\nAlterman, A. I., Cacciola, J. S., & Rutherford, M. J. (1993). Reliability
\nof the Revised Psychopathy Checklist in substance abuse patients.
\nPsychological Assessment, 5, 442\u2013448. doi:10.1037\/1040-3590.
\n5.4.442.
\nAltman, D. G., & Royston, P. (2000). What do we mean by validating
\na prognostic model Statistics in Medicine, 19, 453\u2013473. doi:
\n10.1002\/(SICI)1097-0258(20000229)19:4453::AID-SIM350[
\n3.0.CO;2-5.
\nAmerican Educational Research Association\/American Psychological
\nAssociation. (1999). Standards for educational and psychological
\ntesting. Washington, DC: American Educational Research
\nAssociation.
\nBoccaccini, M. T., Turner, D. B., & Murrie, D. C. (2008). Do some
\nevaluators report consistently higher or lower PCL-R scores than
\nothers Findings from a statewide sample of sexually violent
\npredator Assessments. Psychology, Public Policy, and Law, 14,
\n262\u2013283. doi:10.1037\/a0014523.
\nBorsboom, D., Mellenbergh, G. J., & van Heeran, J. (2003). The
\ntheoretical status of latent variables. Psychological Review, 110,
\n203\u2013219. doi:10.1037\/0033-295X.110.2.203.
\nBradfield, R. B., Huntzickler, P. B., & Fruehan, G. J. (1970). Errors of
\ngroup regression for prediction of individual energy expenditure.
\nThe American Journal of Clinical Nutrition, 23, 1015\u20131016.
\nColditz, G. A. (2001). Cancer culture; epidemics, human behavior,
\nand the dubious search fro new risk factors. American Journal of
\nPublic Health, 91, 357\u2013359. doi:10.2105\/AJPH.91.3.357.
\nCooke, D. J. (2008). Psychopathy as an important forensic construct:
\nPast, present and future. In D. Canter & R. Zukauskiene (Eds.),
\nPsychology, crime & law. New horizons\u2014International perspectives.
\nAldershot: Ashgate.
\nCooke, D. J. (2009a). Psychopathy. In E. A. Campbell & J. Brown
\n(Eds.), Cambridge handbook of forensic psychology. Cambridge:
\nCambridge University Press.
\nCooke, D. J. (2009b). Strengths and limitations of the Psychopathy
\nChecklist Revised (PCL-R) in courts and other tribunals (Paper
\nunder preparation).
\nCooke, D. J., & Michie, C. (1997). An Item Response Theory
\nAssessment of Hare\u2019s Psychopathy Checklist. Psychological
\nAssessment, 9, 2\u201313. doi:10.1037\/1040-3590.9.1.3.
\nCooke, D. J., Michie, C., & Hart, S. D. (2006). Facets of clinical
\npsychopathy: Towards clearer measurement. In C. J. Patrick
\n(Ed.), Handbook of psychopathy (pp. 91\u2013106). New York: The
\nGuilford Press.
\nCooke, D. J., Michie, C., Hart, S. D., & Clark, D. (2005). Assessing
\npsychopathy in the United Kingdom: Concerns about crosscultural
\ngeneralisability. The British Journal of Psychiatry, 186,
\n339\u2013345. doi:10.1192\/bjp.186.4.335.
\nCooke, D. J., Michie, C., & Ryan, J. (2001). Evaluating risk for
\nviolence: A preliminary study of the HCR-20, PCL-R and VRAG
\nin a Scottish prison sample. Edinburgh: Scotland Office.
\nCopas, J., & Marshall, P. (1998). The offender group reconviction
\nscale: A statistical reconviction score for use by probation
\nofficers. Applied Statistics, 47, 159\u2013171. doi:10.1111\/1467-9876.
\n00104.
\nDeMatteo, D., & Edens, J. F. (2006). The role and relevance of the
\nPsychopathy Checklist-Revised in court. A case law survey of
\nU.S courts (1991\u20132004). Psychology, Public Policy, and Law,
\n12, 214\u2013241. doi:10.1037\/1076-8971.12.2.214.
\nDouglas, K. S., Vincent, G. M., & Edens, J. F. (2006). Risk for
\ncriminal recidivism: The role of psychopathy. In C. J. Patrick
\n(Ed.), Handbook of psychopathy (pp. 533\u2013554). New York: The
\nGuilford Press.
\nDSPD Programme. (2005). Dangerous and Severe Personality
\nDisorder (DSPD) High Secure Services for Men. London: DSPD
\nProgramme, Department of Health, Home Office, HM Prison
\nService.
\nEdens, J. F., & Petrila, J. (2006). Legal and ethical issues in the
\nassessment and treatment of psychopathy. In C. J. Patrick (Ed.),
\nHandbook of psychopathy (pp. 573\u2013588).New York: The Guilford
\nPress.
\nElmore, J. G., & Fletcher, S. W. (2006). The risk of cancer risk
\nprediction: \u2018\u2018What is my risk of getting breast cancer Journal of
\nthe National Cancer Institute, 98, 1673\u20131675.
\n272 Law Hum Behav (2010) 34:259\u2013274
\n123
\nFaigman, D. L. (2007). The limits of science in the courtroom. In E.
\nBorgida & S. T. Fiske (Eds.), Beyond common sense: Psychological
\nscience in the courtroom (pp. 303\u2013313). Oxford:
\nBlackwell.
\nFitch, W. L., & Ortega, R. J. (2000). Law and the confinement of
\npsychopaths. Behavioral Sciences & the Law, 18, 663\u2013678. doi:
\n10.1002\/1099-0798(200010)18:5663::AID-BSL408[3.0.CO;2-V.
\nFleiss, J. L. (1981). Statistical methods for rates and proportions.
\nNew York: Wiley.
\nForouzan, E., & Cooke, D. J. (2005). Figuring out la femme fatale:
\nConceptual and assessment issues concerning psychopathy in
\nfemales. Behavioral Sciences and the Law, 23, 765\u2013778.
\nGail, M. H., & Benichou, J. (2000). Encyclopedia of epidemiological
\nmethods. Chichester: Wiley.
\nGuilford, M. C., Rona, R. J., & Chinn, S. (1992). Trends in body mass
\nindex in young adults in England and Scotland from 1973 to 1988.
\nJournal of Epidemiology and Community Health, 46, 187\u2013190.
\ndoi:10.1136\/jech.46.3.187.
\nHaje\u00b4k, A.,&Hall, N. (2002). Induction and probability. In P. Machamer
\n& M. Silberstein (Eds.), Blackwell guide to the philosophy of
\nscience (pp. 149\u2013172). Oxford: Blackwell.
\nHanson, R. K., & Thornton, D. M. (1999). Static 99: Improving
\nactuarial risk assessments for sex offenders. Ottawa: Public Works
\nand Government Services Canada.
\nHare, R. D. (1991). The Hare Psychopathy Checklist\u2014Revised (1st
\ned.). Toronto: Multi-Health Systems.
\nHare, R. D. (1993). Without conscience: The disturbing world of the
\npsychopaths among us (1st ed.). New York: Pocket Books.
\nHare, R. D. (1998). The Hare PCL-R: Some issues concerning its use
\nand misuse. Legal and Criminological Psychology, 3, 101\u2013119.
\nHare, R.D. (2003). TheHare Psychopathy Checklist\u2014Revised (2nd ed.).
\nToronto: Multi-Health Systems.
\nHart, S. D. (1998). The role of psychopathy in assessing risk for
\nviolence: Conceptual and methodological issues. Legal and
\nCriminological Psychology, 3, 121\u2013137.
\nHart, S. D. (2001). Forensic issues. In W. J. Livesley (Ed.), Handbook
\nof personality disorders: Theory, research, and treatment (pp.
\n555\u2013569). New York: The Guilford Press.
\nHart, S. D., Cox, D. N., & Hare, R. D. (1995). The Hare Psychopathy
\nChecklist: Screening version (1st ed.). Toronto: Multi-Health
\nSystems.
\nHart, S. D., & Hare, R. D. (1997). Psychopathy: Assessment and
\nassociation with criminal conduct. In D. M. Stoff, J. Breiling, &
\nJ. D. Maser (Eds.), Handbook of antisocial behavior (pp. 22\u201335).
\nNew York: Wiley.
\nHart, S. D., Michie, C.,&Cooke, D. J. (2007). The precision of actuarial
\nrisk assessment instruments: Evaluating the \u2018\u2018Margins of Error\u2019\u2019 of
\ngroup versus individual predictions of violence. The British
\nJournal of Psychiatry, 170(Suppl 49), 60\u201365. doi:10.1192\/bjp.
\n190.5.s60.
\nHawthorne, V. M., Murdoch, R. M., & Womersley, J. (1979). Body
\nweight of men and women aged 40\u201364 years from an urban area
\nin the West of Scotland. Community Medicine, 1, 229\u2013235.
\nHemphill, J. F., Hare, R. D., & Wong, S. (1998). Psychopathy and
\nrecidivism: A review. Legal and Criminological Psychology, 3,
\n139\u2013170.
\nHemphill, J. F., & Hart, S. D. (2002). Motivating the unmotivated:
\nPsychopathy, treatment, and change. In M. McMurran (Ed.),
\nMotivating offenders to change: A guide to enhancing engagement
\nin therapy (pp. 193\u2013220). Chichester: Wiley.
\nHenderson, R., Jones, M., & Stare, J. (2001). Accuracy of point
\npredictions in survival analysis. Statistics in Medicine, 20, 3083\u2013
\n3096. doi:10.1002\/sim.913.
\nHenderson, R., & Keiding, N. (2005). Individual survival time
\nprediction using statistical models. Journal of Medical Ethics,
\n31, 703\u2013706. doi:10.1136\/jme.2005.012427.
\nKahneman, D.,&Tversky, A. (1973). On the psychology of prediction.
\nPsychological Review, 80, 237\u2013251. doi:10.1037\/h0034747.
\nKennaway, R. (1998). Population statistics cannot be used for
\nreliable individual prediction. Retrieved October 12, 2006, from
\nhttp:\/\/citeseer.ist.psu.edu\/328224.html.
\nLeistico, A. R., Salekin, R. T., DeCoster, J., & Rogers, R. (2008). A
\nlarge-scale meta-analysis relating the Hare measures of psychopathy
\nto antisocial conduct. Law and Human Behavior, 32,
\n28\u201345. doi:10.1007\/s10979-007-9096-6.
\nLogan, C., & Johnstone, L. (2008). Personality disorders: Clinical and
\nrisk formulations (Paper under review).
\nLyon, D., & Ogloff, J. R. P. (2000). Legal and ethical issues in
\npsychopathy assessment. In C. B. Gacono (Ed.), The clinical and
\nforensic assessment of psychopathy (pp. 139\u2013173). Mahwah, NJ:
\nLawrence Erlbaum Associates.
\nMaden, A., & Tyrer, P. (2003). Dangerous and severe personality
\ndisorders: A new personality concept from the United Kingdom.
\nJournal of Personality Disorders, 17, 489\u2013496. doi:10.1521\/pedi.
\n17.6.489.25356.
\nMATHCAD.13. (2005). Mathcad 13 user\u2019s guide. Cambridge, MA:
\nMathsoft Engineering and Education, Inc.
\nMichie, C., & Cooke, D. J. (2006). The structure of violent behavior: A
\nhierarchical model. Criminal Justice and Behavior, 33, 706\u2013737.
\ndoi:10.1177\/0093854806288941.
\nMonahan, J., Steadman, H., Robbins, P. C., Appelbaum, P., Banks, S.,
\nGrisso, T., et al. (2005).An actuarialmodel of violence. Psychiatric
\nServices, 56, 810\u2013815. doi:10.1176\/appi.ps.56.7.810.
\nMonahan, J., Steadman, H., Silver, E., Appelbaum, P., Robbins, P. C.,
\nMulvey, E. P., et al. (2001). Rethinking risk assessment: The
\nMacArthur study of mental disorder and violence (1st ed.). New
\nYork: Oxford University Press.
\nMooney, C. Z. (1997). Monte Carlo simulation. Thousand Oaks, CA:
\nSage.
\nMurrie, D. C., Boccaccini, M. T., Johnson, J. T., & Janke, C. (2008).
\nDoes interrater (dis)agreement on Psychopathy Checklist scores in
\nSexually Violent Predator trials suggest partisan allegiance in
\nforensic Assessments Law and Human Behavior, 32(4), 352\u2013362.
\ndoi:10.1007\/s10979-007-9097-5.
\nMurrie, D. C., Boccaccini, M. T., Turner, D., Meeks, M., Woods, C.,
\n& Tussey, C. Rater (dis)agreement on risk assessment measures
\nin sexually violent predator proceedings: Evidence of adversarial
\nallegiance in forensic Assessment. Psychology, Public Policy, and
\nLaw, in press.
\nNunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory (3rd
\ned.). New York: McGraw-Hill.
\nPsychological Corporation. (1999). Wechsler Abbreviated Scale of
\nIntelligence (WASI) manual. San Antonio, TX: Psychological
\nCorporation.
\nQuinsey, V. L., Harris, G. T., Rice, M. E., & Cormier, C. A. (1998).
\nViolent offenders: Appraising and managing risk (1st ed.).
\nWashington, DC: American Psychological Association.
\nRichters, J. E. (1997). The Hubble hypothesis and the developmentalist\u2019s
\ndilemma. Development and Psychopathology, 9, 193\u2013229.
\ndoi:10.1017\/S0954579497002022.
\nRobert, C. P. (2004). Monte Carlo statistical methods. New York:
\nSpringer-Verlag.
\nRockhill, B. (2001). The privatization of risk. American Journal of
\nPublic Health, 91, 365\u2013368. doi:10.2105\/AJPH.91.3.365.
\nRockhill, B., Kawachi, I., & Colditz, G. A. (2000). Individual risk
\nprediction and population-wide disease prevention. Epidemiologic
\nReviews, 22, 176\u2013180.
\nRorer, L. (1990). Personality assessment: A conceptual survey. In L.
\nA. Pervin (Ed.), Handbook of personality: Theory and research
\n(pp. 693\u2013720). New York: The Guilford Press.
\nRose, G. (1992). The strategy of preventative medicine. Oxford:
\nOxford Medical Publications.
\nLaw Hum Behav (2010) 34:259\u2013274 273
\n123
\nSalekin, R. T., Rogers, R., & Sewell, K. W. (1996). A review and
\nmeta-analysis of the Psychopathy Checklist and Psychopathy
\nChecklist-Revised: Predictive validity of dangerousness. Clinical
\nPsychology: Science and Practice, 3, 203\u2013215.
\nScott, K. G. (2003). Commentary: Individual risk prediction, individual
\nrisk, and population risk. Journal of Clinical Child and Adolescent
\nPsychology, 32, 243\u2013245. doi:10.1207\/S15374424JCCP3202_9.
\nSteel, R. G. D., Torrie, J. H., & Dickey, D. A. (1997). Principles and
\nprocedures of statistics: A biometrical approach. New York:
\nMcGraw Hill.
\nTam, C. C., & Lopman, B. A. (2003). Determinism versus stochasticism:
\nIn support of long coffee breaks. Journal of Epidemiology and
\nCommunity Health, 57, 478. doi:10.1136\/jech.57.7.477.
\nVerona, E., & Vitale, J. (2006). Psychopathy in women: Assessment,
\nmanifestations and etiology. In C. J. Patrick (Ed.), Handbook of
\npsychopathy (pp. 415\u2013436). New York: The Guilford Press.
\nWald, N. J., Hackshaw, A. K., & Frost, C. D. (1999). When can a risk
\nfactor be used as a worthwhile screening test. British Medical
\nJournal, 319, 1562\u20131565.
\nWalsh, T., & Walsh, Z. (2006). The evidentiary introduction of the
\nPsychopathy Checklist-Revised assessed psychopathy in U.S.
\ncourts: Extent and appropriateness. Law and Human Behavior,
\n30, 493\u2013507. doi:10.1007\/s10979-006-9042-z.
\nWalters, G. D. (2003). Predicting criminal justice outcomes with the
\nPsychopathy Checklist and Lifestyle Criminality Screening
\nForm: A meta-analytic comparison. Behavioral Sciences & the
\nLaw, 21, 89\u2013102. doi:10.1002\/bsl.519.
\nWechsler, D. (1991). Wechsler Intelligence Scale for Children (3rd
\ned.). San Antonio, TX: The Psychological Corporation.
\nZinger, I., & Forth, A. E. (1998). Psychopathy and Canadian criminal
\nproceedings: The potential for human rights abuses. Canadian
\nJournal of Criminology, 40, 237\u2013276.
\n274 Law Hum Behav (2010) 34:259\u2013274
\n123
\nReproduced with permission of the copyright owner. Further reproduction prohibited without permission.