Forecasts for 2025 - how'd we do? (part)

I was mostly wrong about 2025. But so was everybody else! Well, at least with regards to the predictions I'm looking at today.

Start of the year, I made my assessment of the predictions from Lightcone hosts. Ten months in, things would be going great for me, if I hadn't also risked a few predictions myself. Guess that did keep me humble.

What I want to say is it's time to start measuring! While things aren't over til the year is over, I figured I'd start defining how we measure the success of each prediction. This gives me some time to refine methodology, and also to explain in unnecessary detail how I'll keep track of prediction performance. Let's dive in.

How we measure

While there are many ways to measure how well forecasts are doing, I'll assume a couple things to simplify the measurement process.

My forecasts will be categorical.

That is, there will always be a discrete number of categories in which the outcome can fall. Unless explicitly stated, outcomes will just be a binary.

If a forecast requires a number, I'll create categories by splitting the possible values into a range.

I'll assign likelihoods to my own forecasts. Typically, as a category, which I then translate into a value.

This just makes it easier to communicate. Values are fixed for each category.

With this, we can calculate a loss function, which will tell us how poorly we're doing. The lower the loss, the better. We'll use Brier Score as our loss.

Batch, predict, cut, measure

My process will always consist of these four steps:

Batch: prepare a group of things I want to think about. These will be treated as a group with the purpose of measuring performance, but it also forces me to think about what subjects I'm talking about - which is good!

Predict: when making forecasts, follow good practice. Categorical predictions, with a likelihood, with a target date.

Cut: "freeze" the state of the world at the target date.

Measure: determine in which category the outcome fell, providing a rationale for it. Then, update our performance metrics for the group and whole.

A note on breaking my own rules

If you read the original article, you've noticed I've made a mistake: I didn't say how certain I was of each of my forecasts. I just want to say I'll do better next time, but whenever this happens I'll assume that I assigned 100% likelihood to the outcome.

Lightcone predictions for 2025

We have four claims to review here,

Diana Hu: AI research to win another Nobel Prize

Harj Taggar: Crypto goes mainstream (via stablecoins)

Garry Tan: The fate of DOGE is linked to the fate of DOGE

Jared Friedman: Zoom call with an AI

Some are easier claims to measure, and we like them for it, some harder, and we'll need a lot of assumptions and simplifications.

My own predictions for 2025

Oh, this will be a lot of fun. I was very wrong. But I'll get around to my own predictions when the year ends. By then, it'll be time to be wrong about the future again, with predictions for 2026!

The scoreboard, or taking myself seriously

This is our scoreboard for now.

It isn't much, but it's a start. From now on, we know how we'll measure the accuracy of our predictions. I just need to keep making silly forecasts, and in a few years we'll have a decently-sized sample to look at. It'll help us tell whether or not I'm a good forecaster. Honestly, I'm looking forward to it.

As an aside, did you know Google Sheets has a query function with it's own syntax? The results table was generated in one go with the query below. Pretty silly that you have to repeat the select statement in the "label" statement to rename the columns. The "AS" keyword has existed for a while. Anyways.

=QUERY(
Forecasts!1:1000,
"select A, count(A), sum(H) / count(H)
where A is not null
group by A
label count(A) 'Count', sum(H) / count(H) 'Brier Score'",
1)