by Dardo Ferreiro

In two previous posts (part1 and part2), we showed the parts of results from a recent forecasting experiment. There, we were happy to show a consistent tendency in 3 of our forecasting questions : consensus discussions in groups improved the accuracy of predictions. In this post we report the outcomes of another 3 questions (Q4, Q5 and Q6).

Before diving into these new results, it is relevant to remember a specific comment from a crowdcognition friend, Keyvan Mallahi Karai (KMK), quoted in our previous post.

KMK said: ‘I think for something like this to work one needs each participant to have some clue about what they were supposed to predict. If this is the case I can see how the aggregation of these can lead to a better prediction. But if they are all completely clueless, there is really no way that the aggregate prediction is any better than each one. As an example, there is no way that a group does better than an individual in predicting a coin flip or anything that is as random as a coin flip to them.’

The three question that we are reporting on today are:

Q4 – Will there be more cases of measles in the USA in 2019 than in 2018?

Q5 – Will the price of regular gasoline in the USA per gallon on december 30th 2019 be higher than 2.7 USD?

Q6 – Will the movie ‘Jumanji: The Next Level’ (2019) have a higher IMDb (International Movie Database) rating than the original Jumanji movie (1995) on december 30th 2019?

Since we run the experiment in Europe with international participants, these predictions are harder than the ones we reported previously. Below we show the Forecasting Error (Brier scores) for the three harder questions. As in the previous posts: Lower values mean less error, and a better prediction. Dotted line shows the median of the Baseline = i1. Bars show the median of Brier scores for those who did not discuss the question (left bar), group consensus after discussions (middle bar) and individual final forecasts after Discussion (right bar). Error bars show the median absolute deviation.

Will there be more cases of measles in the USA in 2019 than in 2018?

Will the price of regular gasoline in the USA per gallon on december 30th 2019 be higher than 2.7 USD?

Will the movie ‘Jumanji: The Next Level’ (2019) have a higher IMDb (International Movie Database) rating than the original Jumanji movie (1995) on december 30th 2019?

In these three questions, group discussions did not improve the predictions at all. If anything, they worsened them. Given that our participants most likely had no precise idea how many liters a gallon is, were not familiar with what is a reasonable IMDb score, and were probably not up to date on USA epidemiological reports, these results confirm KMK’s intuition (one is tempted to use the word prophecy).

Finally, we calculated the group results for the whole six questions, and superimposed the result for each individual question, including previous ones in which group discussions did improve the predictions (for clarity, only median values are shown):