ignorance and uncertainty

All about unknowns and uncertainties

Posts Tagged ‘Andrew Wiles

Expertise on Expertise

with 5 comments

Hi, I’m back again after a few weeks’ travel (presenting papers at conferences). I’ve already posted material on this blog about the “ignorance explosion.” Numerous writings have taken up the theme that there is far too much relevant information for any of us to learn and process and the problem is worsening, despite the benefits of the internet and effective search-engines. We all have had to become more hyper-specialized and fragmented in our knowledge-bases than our forebears, and many of us find it very difficult as a result to agree with one another about the “essential” knowledge that every child should receive in their education and that every citizen should possess.

Well, here is a modest proposal for one such essential: We should all become expert about experts and expertise. That is, we should develop meta-expertise.

We can’t know everything, but knowing an expert when we see one, being able to tell the difference between an expert and an impostor, and knowing what it takes to become an expert can guide our search for assistance in all things about which we’re ignorant. A meta-expert should:

  1. Know the broad parameters of and requirements for attaining expertise;
  2. Be able to distinguish a genuine expert from a pretender or a charlatan;
  3. Know whether expertise is and when it is not attainable in a given domain;
  4. Possess effective criteria for evaluating expertise, within reasonable limits; and
  5. Be aware of the limitations of specialized expertise.

Let’s start with that strongly democratic source of expertise: Wikipedia’s take on experts:

“In many domains there are objective measures of performance capable of distinguishing experts from novices: expert chess players will almost always win games against recreational chess players; expert medical specialists are more likely to diagnose a disease correctly; etc.”

That said, the Wikipedia entry also raises a potentially vexing point, namely that “expertise” may come down to merely a matter of consensus, often dictated by the self-same “experts.” Examples readily spring to mind in areas where objective measures are hard to come by, such as the arts. But consider also domains where objective measures may be obtainable but not assessable by laypeople. Higher mathematics is a good example. Only a tiny group of people on the planet were capable of assessing whether Andrew Wiles really had proven Fermat’s Theorem. The rest of us have to take their word for it.

A crude but useful dichotomy splits views about expertise into two camps: Constructivist and performative. The constructivist view emphasizes the influence of communities of practice in determining what expertise is and who is deemed to have it. The performative view portrays expertise as a matter of learning through deliberative practice. Both views have their points, and many domains of expertise have elements of both. Even domains where objective indicators of expertise are available can have constructivist underpinnings. A proficient modern-day undergraduate physics student would fail late 19th-century undergraduate physics exams; and experienced medical practitioners emigrating from one country to another may find their qualifications and experience unrecognized by their adopted country.

What are the requirements for attaining deep expertise? Two popular criteria are talent and deliberative practice. Re deliberate practice, a much-discussed rule of thumb is the “10,000 hour rule.” This rule was popularized in Malcolm Gladwell’s book Outliers and some authors misattribute it to him. It actually dates back to studies of chess masters in the 1970’s (see Ericsson, K. A., R. Th. Krampe, and C. Tesch-Römer, 1993), and its generalizability to other domains still is debatable. Nevertheless, the 10K rule has some merit, and unfortunately it has been routinely ignored in many psychological studies comparing “experts” with novices, where the “experts” often are undergraduates who have been given a few hours’ practice on a relatively trivial task.

The 10K rule can be a useful guide but there’s an important caveat. It may be a necessary but it is by no means a sufficient condition for guaranteeing deep expertise. At least three other conditions have to be met: Deliberative and effective practice in a domain where deep expertise is attainable. Despite this quite simple line of reasoning, plenty of published authors have committed the error of viewing the 10K rule as both necessary and sufficient. Gladwell didn’t make this mistake, but Jane McGonigal’s recent book on video and computer games devotes considerable space to the notion that because gamers are spending upwards of 10K hours playing games they must be attaining deep “expertise” of some kind. Perhaps some may be, provided they are playing games of sufficient depth. But many will not. (BTW, McGonigal’s book is worth a read despite her over-the-top optimism about how games can save the world—and take a look at her game-design collaborator Bogost’s somewhat dissenting review of her book).

Back to the caveats. First, no deliberation makes practice useless. Having spent approximately 8 hours every day sleeping for the past 61 years (178,120 hours) hasn’t made me an expert on sleep. Likewise, deliberative but ineffective practice methods deny us top-level expertise. Early studies of Morse Code experts demonstrated that mere deliberative practice did not guarantee best performance results; specific training regimes were required instead. Autodidacts with insight and aspirations to attain the highest performative levels in their domains eventually realise how important getting the “right” coaching or teaching is.

Finally, there is the problem of determining whether effective, deliberative practice yields deep expertise in any domain. The domain may simply not be “deep” enough. In games of strategy, tic-tac-toe is a clear example of insufficient depth, checkers is a less obvious but still clear example, whereas chess and go clearly have sufficient depth.

Tic-tac-toe aside, are there domains that possess depth where deep expertise nevertheless is unattainable? There are, at least, some domains that are deeply complex where “experts” perform no better then less trained individuals or simple algorithms. Psychotherapy is one such domain. There is a plethora of studies demonstrating that clinical psychologists’ predictions of patient outcomes are worse than simple linear regression models (cf. Dawes’ searing indictment in his 1994 book) and that sometimes experts’ decisions are no more accurate than beginners’ decisions and simple decision aids. Similar results have been reported for financial planners and political experts. In Philip Tetlock’s 2005 book on so-called “expert” predictions, he finds that many so-called experts perform no better than chance in predicting political events, financial trends, and so on.

What can explain the absence of deep expertise in these instances? Tetlock attributes experts’ poor performance to two factors, among others: Hyperspecialization and overconfidence. “We reach the point of diminishing marginal predictive returns for knowledge disconcertingly quickly,” he reports. “In this age of academic hyperspecialization, there is no reason for supposing that contributors to top journals—distinguished political scientists, area study specialists, economists, and so on—are any better than journalists or attentive readers of the New York Times in ‘reading’ emerging situations.” And the more famous the forecaster the more overblown the forecasts. “Experts in demand,” Tetlock says, “were more overconfident than their colleagues who eked out existences far from the limelight.” Tetlock also claims that cognitive style counts: “Foxes” tend to outperform “hedgehogs.” These terms are taken from Isaiah Berlin’s popular essay: Foxes know a little about lots of things, whereas hedgehogs know one big thing.

Another contributing factor may be a lack of meta-cognitive insight on the part of the experts. A hallmark of expertise is ignoring (not ignorance). This proposition may sound less counter-intuitive if it’s rephrased to say that experts know what to ignore. In an earlier post I mentioned Mary Omodei and her colleagues’ chapter in a 2005 book on professionals’ decision making in connection with this claim. Their chapter opens with the observation of a widespread assumption that domain experts also know how to optimally allocate their cognitive resources when making judgments or decisions in their domain. Their research with expert fire-fighting commanders cast doubt on this assumption.

The key manipulations in the Omodei simulated fire-fighting experiments determined the extent to which commanders had unrestricted access to “complete” information about the fires, weather conditions, and other environmental matters. They found that commanders performed more poorly when information access was unrestricted than when they had to request information from subordinates. They also found that commanders performed more poorly when they believed all available information was reliable than when they believed that some of it was unreliable. The disquieting implication of these findings is that domain expertise doesn’t include meta-cognitive expertise.

Cognitive biases and styles aside, another contributing set of factors may be the characteristics of the complex, deep domains themselves that render deep expertise very difficult to attain. Here is a list of tests you can apply to such domains by way of evaluating their potential for the development of genuine expertise:

  1. Stationarity? Is the domain stable enough for generalizable methods to be derived? In chaotic systems long-range prediction is impossible because of initial-condition sensitivity. In human history, politics and culture, the underlying processes may not be stationary at all.
  2. Rarity? When it comes to prediction, rare phenomena simply are difficult to predict (see my post on making the wrong decisions most of the time for the right reasons).
  3. Observability? Can the outcomes of predictions or decisions be directly or immediately observed? For example in psychology, direct observation of mental states is nearly impossible, and in climatology the consequences of human interventions will take a very long time to unfold.
  4. Objective or even impartial criteria? For instance, what is “good,” “beautiful,” or even “acceptable” in domains such as music, dance or the visual arts? Are such domains irreducibly subjective and culture-bound?
  5. Testability? Are there clear criteria for when an expert has succeeded or failed? Or is there too much “wiggle-room” to be able to tell?

Finally, here are a few tests that can be used to evaluate the “experts” in your life:

  1. Credentials: Does the expert possess credentials that have involved testable criteria for demonstrating proficiency?
  2. Walking the walk: Is the expert an active practitioner in their domain (versus being a critic or a commentator)?
  3. Overconfidence: Ask your expert to make yes-no predictions in their domain of expertise, and before any of these predictions can be tested ask them to estimate the percentage of time they’re going to be correct. Compare that estimate with the resulting percentage correct. If their estimate was too high then your expert may suffer from over-confidence.
  4. Confirmation bias: We’re all prone to this, but some more so than others. Is your expert reasonably open to evidence or viewpoints contrary to their own views?
  5. Hedgehog-Fox test: Tetlock found that Foxes were better-calibrated and more able to entertain self-disconfirming counterfactuals than hedgehogs, but allowed that hedgehogs can occasionally be “stunningly right” in a way that foxes cannot. Is your expert a fox or a hedgehog?
  6. Willingness to own up to error: Bad luck is a far more popular explanation for being wrong than good luck is for being right. Is your expert balanced, i.e., equally critical, when assessing their own successes and failures?

Written by michaelsmithson

August 11, 2011 at 11:26 am

When Is It Folly to Be Wise?

with 3 comments

There are things we’d rather not know. Some of these are temporary; we’d like to know them eventually but not just now. Others, less common, are things we never want(ed) to know.

In this post I’ll focus on the temporary kind. Temporary ignorance has many uses, some of which are not immediately obvious. I’ve already mentioned a few of them in earlier posts. One of these is entertainment. Many forms of entertainment require temporary audience ignorance, including all forms of story-telling and jokes. No unknowns? No mysteries? No surprises? Then no entertainment.

Games are an example of entertainment where uncertainty has a key role even in games of skill. A game that is a foregone conclusion is not very entertaining. Games of skill are like tests but more fun. Why? Partly because games have more uncertainty built into them than tests do, and so they tease us with a mix of outcomes due to skill and sheer luck. More than 25 years ago, a clinical neuropsychologist working in a large hospital told me how he ended up exploiting this connection between games and tests. One of his chief duties was to assess the state and recovery of cognitive functions of patients in a head trauma unit—Often victims of automobile accidents or strokes. The well-established tests of memory, motor control and sustained attention had good psychometric properties but they were boring. Some patients refused to take them; others complied but only with a desultory effort.

Then inspiration struck: My colleague noticed that anyone who could manage it would head down the ward corridor to play Space Invaders. Here was a ready-made test of attention and motor control built into a game. Moreover, repeatedly playing the game actually would facilitate patients’ recovery, so unlike the standard cognitive tests this “test” had a therapeutic effect. He attached a computer to the back of the game, established benchmark measures such as how long players would last if they did nothing or moved the joystick randomly, and started recording individual patients’ results. The results were a clinician’s dream—Meaningful data tracking patients’ recovery and a therapeutic exercise.

Some psychologists who should know better (e.g., Gudykunst and Nishida 2001) have declared that the emotional accompaniment of uncertainty is anxiety. Really? What about thrill, excitement, anticipation, or hope? We can’t feel thrill, excitement, or anticipation without the unknowns that compel them. And as for hope, if there’s no uncertainty then there’s no hope. These positive emotions aren’t merely permitted under uncertainty, they require uncertainty. To my knowledge, no serious investigation has been made into the emotional concomitants of omniscience, but in fact, there is only one human emotional state I associate with omniscience (aside from smugness)—Boredom.

We don’t just think we’re curious or interested; we feel curious or interested. Curiosity and interest have an emotional cutting-edge. Intellectuals, artists and researchers have a love-hate emotional relationship with their own ignorance. On the one hand, they are in the business of vanquishing ignorance and resolving uncertainties. On the other, they need an endless supply of the unknowns, uncertainties, riddles, problems and even paradoxes that are the oxygen of the creative mind. One of the hallmarks of scientists’ reactions to Horgan’s (1996) book, “The End of Science,” was their distress at Horgan’s message that science might be running out of things to discover. Moreover, artists are not attracted to obvious ideas, nor scientists to easy problems. They want their unknowns to be knowable and problems to be solvable, but also interesting and challenging.

Recently an Honours student undertaking her first independent research project came to me for some statistical advice. She sounded frustrated and upset. Gradually it became apparent that hardly any of her experimental work had turned out as expected, and the standard techniques she’d been taught were not helping her to analyze her data and interpret her findings. I explained that she might have to learn about another technique that could help here. She asked me, “Is research always this difficult?” I replied with Piet Hein’s aphorism, “Problems worthy of attack prove their worth by fighting back.” Her eyes narrowed. “Well, now that you put it that way…” Immediately I knew that this student had the makings of a researcher.

A final indication of the truly ambivalent relationship creative folk have with their favorite unknowns is that they miss them once they’ve been dispatched. Andrew Wiles, the mathematician who proved Fermat’s Last Theorem, spoke openly of his sense of loss for the problem that had possessed him for more than 7 years.

And finally, let’s take one more step to reach a well-known but often forgotten observation: Freedom is positively labeled uncertainty about the future. There isn’t much more to it than that. No future uncertainties in your life? Everything about your future is fore-ordained? Then you have no choices and therefore no freedom. As with intellectuals and their unknowns, we want many of our future unknowns to be ultimately knowable but not foreordained. We crave at least some freedom of choice.

People are willing to make sacrifices for their freedom, and here I am not referring only to a choice between freedom and a dreadful confinement or tyrannical oppression. Instead, I have in mind tradeoffs between freedom and desirable, even optimal but locked-in outcomes. People will cling to their freedom to choose even if it means refusing excellent choices.

A 2004 paper by Jiwoong Shin and Daniel Ariely, described in Ariely’s entertaining book “Predictably Irrational” (2008, pp. 145-153) reports the results of experimental evidence for this claim. Shin and Ariely set up an online game with 3 clickable doors, each of which yielded a range of payoffs (e.g. between 1 and 10 cents). The object of the game was to make as much money as possible in 100 clicks. There was a twist: Every time one door was clicked, the others would shrink by a certain amount, and if unchosen for sufficiently many times a door would disappear altogether. Shin and Ariely found that even bright university (MIT) students would forgo top earnings in order to keep all the doors open. Shin and Ariely tried providing the participants with the exact monetary payoffs from each door (so they would know which door offered the most) and they even modified the game so that a disappeared door could be “reincarnated” with a single click. It made no difference; participants continued to refuse to close any doors. For them, the opportunity costs of closed doors loomed larger than the payoffs they could have had by sticking with the best door.

So here we have one of the key causes of indecision, namely a strong desire to “keep our options open,” i.e., to maintain positively labeled uncertainty. If achieving certainty is framed in terms of closing off options, we strive to avoid it. If uncertainty is framed as keeping our options open we try to maintain it, even if that entails missing out on an excellent choice. This tendency is illustrated by a folk-wisdom stereotype in the wild and wonderful world of dating-and-mating. He and she are in love and their relationship has been thriving for more than a year. She’d like to make it permanent, but he’s still reluctant to commit. Why? Because someone “better” might come along…

What could drive us to keep our options open, refusing to commit even when we end up letting our best opportunities pass us by? Could it be the way we think about probabilities? Try this rather grim thought-experiment: First, choose an age beyond your current age (for me, say, 75). Then, think of the probability that you’ll get cancer before you reach that age. Now, think of the probability that you’ll get cancer of the stomach. Think of the probability you’ll get lung cancer. The probability you’ll get bone cancer. Or cancer of the brain. Or breast cancer (if you’re a woman) or prostate cancer (if you’re a man). Or skin cancer. Or pancreatic cancer… If you’re like most people, unpacking “cancer” into a dozen or so varieties will make it seem more likely that you’ll get it than considering “cancer” in one lump—It just seems more probable that you’d end up with at least one of those varieties. The more ways we can think of something happening, the more likely we think it is.  Cognitive psychologists have found experimental evidence for this effect (for the curious, take a look at papers by Tversky and Kohler 1994 and Tversky and Rottenstreich 1997),

An even more startling effect was uncovered in a paper by Kirkpatrick and Epstein (1992). They offered people a choice between drawing a ticket from a lottery of 10 tickets, 9 losing and 1 winning, and a lottery with 100 tickets, 90 losing and 10 winning. The participants confirmed that they knew the probability of winning either lottery was .1, so there was no effect on their probability judgments. Nevertheless, when asked which they preferred most chose the 100-ticket lottery. Why? Because that lottery gave them 10 ways of winning whereas the other gave them only 1 way.

The more options we keep open, the more “winning tickets” we think we hold and the greater the hope we might get lucky. When we’ve committed to one of those options we may have gained certitude, but luck and hope have vanished.

Written by michaelsmithson

November 3, 2010 at 10:50 am