Problems with the elicitation of uncertainty

Abstract

The Oil and Gas industry has a poor record when it comes to accurately assessing uncertainties. An assessment of the risks that arise from these uncertainties is a major factor influencing investment decisions in the O&G industry. Thus, throughout the industry, experts' understanding of the probabilities of uncertain events are elicited via a range of methods. However, industry sources continue to report ‘surprise’ values (outside the P(80) range) far more often than the 20% indicated by this interval. This suggests problems in the elicitation process: either experts' beliefs do not capture the distributions they are attempting to define or the processes used in the industry are failing to elicit the experts' subjective probability distributions (SPDs) accurately. The authors discuss the ways in which experts' subjective beliefs about the probability of events are commonly elicited in the industry, and the biases expected to be observed therein, in light of psychological findings and theories of decision-making. Special note is made of the limitations imposed on decision-making by the nature of human short-term memory and predictable biases resulting from reliance on heuristic (rule-of-thumb) reasoning techniques. The results of an experiment comparing two commonly used elicitation techniques with one, designed by the authors, which utilizes heuristic reasoning as a strength rather than a weakness, are presented and discussed. Accuracy, both in terms of subjective and objective measures, was improved on our experimental tasks by use of the new, More-or-Less Elicitation (MOLE ) technique over both of the currently used methods. We conclude that an approach to elicitation that recognizes the tendency of people to use heuristic reasoning, rather than forces them to reason in probabilistic, non-intuitive ways, may yield superior results. Finally, we discuss possible refinements to the MOLE process and further research required to elucidate the impacts of various biases on elicited distributions.