Over-diagnosis and “investigation momentum”
One of my earlier posts, “Making the Wrong Decisions for the Right Reasons”, focused on conditions under which it is futile to pursue greater certainty in the name of better decisions. In this post, I’ll investigate settings in which a highly uncertain outcome should motivate us more strongly than no outcome at all to seek greater certainty. The primary stimulus for this post is a recent letter to JAMA Internal Medicine (Sah, Elias, & Ariely, 2013), entitled “Investigation momentum: the relentless pursuit to resolve uncertainty.” The authors present what they refer to as “another potential downside” to unreliable tests such as prostate-specific antigen (PSA) screening, namely the effect of receiving a “don’t know” result instead of a “positive” or “negative” diagnosis. Their chief claim is that the inconclusive result increases psychological uncertainty which motivates people to seek additional diagnostic testing, whereas untested people would not be motivated to get diagnostic tests. They term this motivation “investigation momentum”, and the obvious metaphor here is that once the juggernaut of testing gets going, it obeys a kind of psychological Newton’s First Law.
The authors’ evidence is an online survey of 727 men aged between 40 and 75 years, with a focus on prostate cancer and PSA screening (e.g., participants were asked to rate their likelihood of developing prostate cancer). The participants were randomly assigned to one of four conditions. In the “no PSA,” condition, participants were given risk information about prostate biopsies. In the other conditions, participants were given information about PSA tests and prostate biopsies. They then were asked to imagine they had just received their PSA test result, which was either “normal”, “elevated”, or “inconclusive”. In the “inconclusive” condition participants were told “This result provides no information about whether or not you have cancer.” After receiving information and (in three conditions) the scenario, participants were asked to indicate whether, considering what they had just been given, they would undergo a biopsy and their level of certainty in that decision.
The study results revealed that, as would be expected, the men whose test result was “elevated” were more likely to say they would get a biopsy than the men in any of the other conditions (61.5% versus 12.7% for those whose result was “normal”, 39.5% for those whose result was “inconclusive”, and 24.5% for those with no PSA). Likewise, the men whose hypothetical result was “normal” were least likely to opt for a biopsy. However, a significantly greater percentage opted for a biopsy if their test was “inconclusive” than those who had no test at all. This latter finding concerned the authors because, as they said, when “tests give no diagnostic information, rationally, from an information perspective, it should be equivalent to never having had the test for the purpose of future decision making.”
This claim amounts to stating that the patient’s subjective probability of having cancer should remain unchanged after the inconclusive test result. Whether that (rationally) should be the case depends on three things: The patient’s prior subjective probability of having cancer, the patient’s attribution of a probability of cancer to the ambiguous test result, and the relative weights assigned by the patient to the prior and the test result. There are two conditions under which the patient’s prior subjective probability should remain unchanged: (a) The prior subjective probability is identical to the probability attributed to the test, or (b) The test is given a weight of 0. Option (b) seems implausible, so option (a) is the limiting case. Now, it should be clear that if P(cancer|ambiguous test) > P(cancer|prior belief) then P(cancer|prior belief and ambiguous test) > P(cancer|prior belief). Therefore, it could be rational for people to be more inclined to get further tests after an initial test returns an ambiguous result than if they have not yet had any tests.
Let us take one more step. It is plausible that for many people an ambiguous test result would cause them to impute P(cancer|ambiguous test) to be somewhere in the neighbourhood of 1/2. So, for the sake of argument, let’s set P(cancer|ambiguous test) = 1/2. It also is plausible that most people will have an intuitive probability threshold, Pt, beyond which they will be inclined to seek testing. For something as consequential as cancer, we may suppose that for this threshold, Pt < 1/2. Indeed, the authors’ data suggest exactly this. In the no-PSA condition, 16.2% of the men rated their chance of getting prostate cancer above 1/2, but 24.5% of them said they would get a biopsy. Therefore, assuming that all of the 16.2% are included in those who opt for a biopsy, that leaves 8.3% of them in the below-1/2 part of the sample who also opt for a biopsy. An interpolation (see the Technical Bits section below) yields Pt = .38 (group average, of course).
The finding that vexed Sah, et al. is that 39.5% for those whose result was “inconclusive” opted for a biopsy, compared to 24.5% in the no-PSA condition. To begin, let’s assume that the “inconclusive” sample’s Pt also is .38 (after all, the authors find no significant differences among the four samples’ prior probabilities of getting prostate cancer). In the “inconclusive” sample 10.8% rated their chances of getting prostate cancer above 1/2 and 18.9% rated it between .26 and .5. So, our estimate of the percentage whose prior probability is .38 or more is 10.8% + 18.9%/2 = 20.3%. This is the percentage of people in the “inconclusive” sample who would have gone for a biopsy if they had no test, given Pt = .38. That leaves 19.2% to account for in the boost from 20.3% to 39.5% after receiving the inconclusive test result. Now, we can assume that 9.5% are in the .25-.50 range because that’s the total percentage of this sample in that range, so the remaining 9.7% must fall in the 0-.25 range. There are 70.3% altogether in that range, so a linear interpolation gives us a lowest probability for the 9.7% we need to account for of .25 (1 – 9.7/70.3) = .2155.
Now we need to compute the maximum relative weight of the test required to raise a subjective probability of .2155 to the threshold .38. From our formula in the Technical Bits section, we have
That is, the test would have to be given at most about 1.37 times the weight that each person gives to their own prior subjective probability of prostate cancer. Weighting the test 1.37 times more than one’s prior probability doesn’t seem like giving outlandish weight to the test. And therefore, a plausible case has been put that the tendency for more people to opt for further testing after an inconclusive test result might not be due to psychological “momentum” at all, but instead the product of rational thought. I’m not claiming that the 727 men in the study actually are doing Bayesian calculations—I’m just pointing out that the authors’ findings can be just as readily explained by attributing Bayesian rationality to them as by inferring that they are in the thrall of some kind of psychological “momentum”. The key to understanding all this intuitively is the distinction (introduced by Keynes in 1921) between the weight vs strength (or extremity) of evidence. An inconclusive test is not extreme, i.e., favouring neither the hypothesis that one has the disease or is clear of it. Nevertheless, it still adds its own weight.
The Pt = .38 result is obtained by a simple linear interpolation. There is 8.3% of the no-PSA sample that fall in the next range below 1/2, which in the authors’ table is a range from .26 to .50. All up, 16% of this sample are in that range, so assuming that the 8.3% are the top-raters in that bunch, our estimate of their lowest probability is .50 – 8.3(.50-.26)/16 = .38.
The relative weight of the test is determined by assuming that the participant is reasoning like a Bayesian agent. Assuming that P(cancer|ambiguous test) = 1/2, we may determine when these conditions could occur for a Bayesian agent whose subjective probability, P(cancer|prior belief) , has a Beta() distribution and therefore a mean of . To begin, the first condition would be satisfied if > Pt. Now denoting the relative weight of the test by , the posterior distribution of P(cancer|prior belief and ambiguous test) would be Beta(). So, the second condition would be satisfied if > Pt. Solving for the weight, we get
This makes intuitive sense in two respects. First, the numerator is positive only if < Pt, i.e., if the first condition hasn’t already been satisfied. The further is below Pt, the greater the weight needs to be in order for the above inequality to be satisfied. Second, the denominator tells us that the further Pt is below ½, the less weight needs to be given to the test for that inequality to be satisfied.
The “precision” of a Beta() distribution is . The greater this sum, the tighter the distribution is around its mean. So the precision can be used as a proxy for the weight of evidence associated by an agent to their prior belief. The test adds to the precision because it is additional evidence: So now we can compare the weight of the test relative to the precision of the prior: Given the minimal weight required in the previous equation, we get
where Pp is the agent’s mean prior probability of getting cancer.