Inspired by Philip E. Tetlock and Dan Gardner’s Superforecasting, I propose a SDMB forecasting tournament with the following rules:
All forecasts are for events within the next 18 months, ideally in a range from 1-18 months, beginning with events after October 1, 2017.
Each question must have a concrete, binary, and publicly known answer.
Questions will involve important matters in a variety of subject areas.
There shall be 15 questions, to reduce the influence of luck and without too much time commitment.
Each answer will be in the form of a probability (0-100%).
For ease of administration, there will be no updating of answers after the answer period closes on September 15, 2017.
The winner will be the forecaster with the lowest brier score.
I will not participate, will select the questions, and will judge the answers (though that part should be not open to judgment if the questions are properly crafted).
In this thread, I want to gauge interest in participation and also formulate questions.
Here’s some ideas for questions:
The stock price of AMZN will be above 1000 on October 15, 2017.
The metacritic.com score for Blade Runner 2049 will exceed 85 as of October 31, 2017.
The U.S. House of Representatives will pass legislation authorizing the use of military force against North Korea on or before December 31, 2017.
Roy Moore will be seated as a U.S. Senator from Alabama before February 1, 2018.
Nicolas Maduro will be President of Venezuela on November 1, 2018.
Thinking about how the scoring works, we’ll want questions that could go either way. Ideally, we’d have a large pool of questions, and then down-select to those where the probability’s mean over the group were closest to 50%. I think that’s too much work and we’ll simply need smart choices.
Is there a way to check what the Metacritic score for movie was on a particular date? If not, then there is no way to confirm if the question was yes or no in the future, since the score can change over time.
I’d be even more specific: on or between 1 Oct 2017 and 31 Dec 2017. To avoid debate about whether legislation passed during 1950s satisfies the question.
Will a grand jury convened by Robert Mueller issue an indictment on or between 1 Oct 2017 and 31 Jul 2018, and publicly reveal it on or before 31 Jul 2018? (That’s about one year after it was convened.)
Will Teresa May be Prime Minister of the United Kingdom on any part of the day of 13 Jul 2018? (Two year anniversary of becoming PM.)
At the closing of the Winter Olympics on 25 Feb 2018, will the United States team have won more than 28 medals? (That’s how many won in 2014.)
A space vehicle will survive impact with the moon on or between 1 Oct 2017 and 31 Dec 2018. (There are several missions planned by varying states and companies.)
Will the winning score on the SDMB Challenge: Best Forecaster be equal or better than 0.01? (Corresponding to 90% correct.)
They should be phrased as answers to questions, because it makes it unambiguous what gets a 1 value in the brier score and what gets a 0 value. Either the statement is true (1) or false (0).
[strike]January 2019 is too far out. The limit is 18 months.[/strike] The second question is probably amendable to a closer date, but I fear “implicate” may be too vague.
Edit: Yikes! I don’t know why I mentally translated that to January 2020. Forgive me.
Sure, we can, but how will anyone after that date check if the prediction was correct or not?
Thinking about it more, I’d like to suggest the following scoring formula: S = 1 - sqrt( sum( (f_t - o_t)^2 ) / N ), which is just the complement of the root-mean-square error. The advantage is a perfect score will be 100% and the worst score 0%, with a linear scaling in between. It won’t change the ordering compared to a standard Brier score, but is easier for the typical person to understand.
The winning score is almost certain to be better than 50%, simply because of random chance and multiple contestants. To make an interesting question, we need to move the threshold higher.
Also note, a score of 0.02 corresponds to a 86% correct. If you’re 50% correct on average, you’ll get a score of 0.25.
~ ~ ~ ~ ~
More suggested questions…
Will Taylor Swift have the number one song on the Billboard Top 200 Chart anytime in the month of December 2017?
Will the State of Texas will execute ten or more persons in the year 2018?
Will an American win any tennis Grand Slam singles competition in the year 2018?
Will Janet Yellen be Chair or acting Chair of the Board of Governors of the Federal Reserve System on February 4, 2018? (That’s one day after her current term as Chair ends.)
On 31 Dec 2017 will market capitalization of Amazon.com Inc be greater than $500 billion? (Currently at about $475B.)
I presume that I (or someone else) will report the answer in this thread. We can do a screenshot, if necessary, to preserve it for posterity.
I meant that .02 would make the mean answering guess closer to 50%, not that the person getting that score would get 50% correct on average. In other words, .02 strikes me as closer to the line where we might guess that whether the winner exceeds that score is a coin-flip.
I have no particular opinion about whether to use a brier score or your modified version. My concern is just that people will confuse that final score with a mean of their guesses, given the similarity in form and magnitude.
Excellent. Is there some reason to prefer a question about market cap over a question about stock price?
