More-or-less elicitation (MOLE): Testing a heuristic elicitation method

Abstract

Elicitation of people’s knowledge is a central methodological challenge for psychology, with important impacts in many technical disciplines and industrial settings. The need to convert an expert’s beliefs into a useable format is of particular importance when judgments and decisions are made under uncertainty. Simply asking a person for their best estimate or to estimate a range is subject to many biases – e.g., overconfidence – and methods for eliciting information that avoid these effects are required. This paper presents a heuristic-based elicitation method, More-Or-Less Elicitation (MOLE) which, rather than requiring people make absolute judgments, asks them to make repeated relative judgments. MOLE uses these, along with confidence statements, to construct probability distributions (pdfs) representing a person’s beliefs. We evaluate MOLE by comparing these subjective pdfs with ranges elicited using traditional methods. The central finding is that use of MOLE greatly improves the accuracy and precision of elicited ranges, thereby reducing overconfidence. The benefits of this and other possible heuristic-based methods of elicitation are discussed.