TY - GEN

T1 - Binary Scoring Rules that Incentivize Precision

AU - Neyman, Eric

AU - Noarov, Georgy

AU - Weinberg, S. Matthew

N1 - Publisher Copyright:
© 2021 ACM.

PY - 2021/7/18

Y1 - 2021/7/18

N2 - All proper scoring rules incentivize an expert to predict accurately (report their true estimate), but not all proper scoring rules equally incentivize precision. Rather than treating the expert's belief as exogenously given, we consider a model where a rational expert can endogenously refine their belief by repeatedly paying a fixed cost, and is incentivized to do so by a proper scoring rule. Specifically, our expert aims to predict the probability that a biased coin flipped tomorrow will land heads, and can flip the coin any number of times today at a cost of per flip. Our first main result defines an incentivization index for proper scoring rules, and proves that this index measures the expected error of the expert's estimate (where the number of flips today is chosen adaptively to maximize the predictor's expected payoff). Our second main result finds the unique scoring rule which optimizes the incentivization index over all proper scoring rules. We also consider extensions to minimizing the lh moment of error, and again provide an incentivization index and optimal proper scoring rule. In some cases, the resulting scoring rule is differentiable, but not infinitely differentiable. In these cases, we further prove that the optimum can be uniformly approximated by polynomial scoring rules. Finally, we compare common scoring rules via our measure, and include simulations confirming the relevance of our measure even in domains outside where it provably applies.

AB - All proper scoring rules incentivize an expert to predict accurately (report their true estimate), but not all proper scoring rules equally incentivize precision. Rather than treating the expert's belief as exogenously given, we consider a model where a rational expert can endogenously refine their belief by repeatedly paying a fixed cost, and is incentivized to do so by a proper scoring rule. Specifically, our expert aims to predict the probability that a biased coin flipped tomorrow will land heads, and can flip the coin any number of times today at a cost of per flip. Our first main result defines an incentivization index for proper scoring rules, and proves that this index measures the expected error of the expert's estimate (where the number of flips today is chosen adaptively to maximize the predictor's expected payoff). Our second main result finds the unique scoring rule which optimizes the incentivization index over all proper scoring rules. We also consider extensions to minimizing the lh moment of error, and again provide an incentivization index and optimal proper scoring rule. In some cases, the resulting scoring rule is differentiable, but not infinitely differentiable. In these cases, we further prove that the optimum can be uniformly approximated by polynomial scoring rules. Finally, we compare common scoring rules via our measure, and include simulations confirming the relevance of our measure even in domains outside where it provably applies.

KW - information elicitation

KW - proper scoring rules

UR - http://www.scopus.com/inward/record.url?scp=85112021655&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85112021655&partnerID=8YFLogxK

U2 - 10.1145/3465456.3467639

DO - 10.1145/3465456.3467639

M3 - Conference contribution

AN - SCOPUS:85112021655

T3 - EC 2021 - Proceedings of the 22nd ACM Conference on Economics and Computation

SP - 718

EP - 733

BT - EC 2021 - Proceedings of the 22nd ACM Conference on Economics and Computation

PB - Association for Computing Machinery, Inc

T2 - 22nd ACM Conference on Economics and Computation, EC 2021

Y2 - 18 July 2021 through 23 July 2021

ER -