Constructed value of information with iterative scoring and parametric uncertainty to identify management-relevant research priorities for a declining raptor species

Abstract

Constructed value of information (CVoI) is an expert elicitation decision-analytic tool used to prioritize sources of uncertainty based on their potential to improve decision outcomes if resolved. Despite increased application of CVoI, the robustness of CVoI prioritization of sources of uncertainty relative to differences in expert elicitation and scoring methods has not been evaluated. We engaged a group of species experts in a decision-analytic process to elicit uncertainties, framed as alternative hypotheses, about current population declines of the American kestrel ( Falco sparverius ) in the United States. Participants scored 13 hypotheses across 3 CVoI criteria, which are defined as constructed scales. Rather than experts selecting a single score per criterion, we used a likelihood point method to incorporate parametric uncertainty in the scoring process, in which experts were given 100 points to distribute across possible score categories within the criterion-specific constructed scale. Experts provided scores over 2 scoring rounds, with an opportunity to review and discuss initial scores between rounds. We used a Shannon entropy calculation to quantify how evenly participants allotted their points. We used simulation to evaluate the robustness of our prioritization results relative to a scoring method in which participants selected a single score category for each criterion. Participants often spread their points across 2 adjacent scores, reflecting parametric uncertainty. For one third of the hypothesis-scoring round combinations, the prioritization results differed in approximately 50% of simulations. The highest scoring hypotheses related to how the use of artificial versus natural nest cavities affects fecundity or survival, whether winter roosting sites are a limiting factor for population growth, and whether gamebird habitat management may benefit kestrel populations. Our CVoI prioritization framework can be used to develop collaborative research that is directly relevant to a management decision and is an advance in eliciting more representative expert beliefs.