I was mostly wrong about 2025. But so was everybody else! Well, at least with regards to the predictions I'm looking at today.
Start of the year, I made my assessment of the predictions from Lightcone hosts. Ten months in, things would be going great for me, if I hadn't also risked a few predictions myself. Guess that did keep me humble.
What I want to say is it's time to start measuring! While things aren't over til the year is over, I figured I'd start defining how we measure the success of each prediction. This gives me some time to refine methodology, and also to explain in unnecessary detail how I'll keep track of prediction performance. Let's dive in.
How we measure
While there are many ways to measure how well forecasts are doing, I'll assume a couple things to simplify the measurement process.
My forecasts will be categorical.
That is, there will always be a discrete number of categories in which the outcome can fall. Unless explicitly stated, outcomes will just be a binary.
If a forecast requires a number, I'll create categories by splitting the possible values into a range.
I'll assign likelihoods to my own forecasts. Typically, as a category, which I then translate into a value.
This just makes it easier to communicate. Values are fixed for each category.
With this, we can calculate a loss function, which will tell us how poorly we're doing. The lower the loss, the better. We'll use Brier Score as our loss.
Batch, predict, cut, measure
My process will always consist of these four steps:
Batch: prepare a group of things I want to think about. These will be treated as a group with the purpose of measuring performance, but it also forces me to think about what subjects I'm talking about - which is good!
Predict: when making forecasts, follow good practice. Categorical predictions, with a likelihood, with a target date.
Cut: "freeze" the state of the world at the target date.
Measure: determine in which category the outcome fell, providing a rationale for it. Then, update our performance metrics for the group and whole.
A note on breaking my own rules
If you read the original article, you've noticed I've made a mistake: I didn't say how certain I was of each of my forecasts. I just want to say I'll do better next time, but whenever this happens I'll assume that I assigned 100% likelihood to the outcome.
Lightcone predictions for 2025
We have four claims to review here,
Diana Hu: AI research to win another Nobel Prize
Harj Taggar: Crypto goes mainstream (via stablecoins)
Garry Tan: The fate of DOGE is linked to the fate of DOGE
Jared Friedman: Zoom call with an AI
Some are easier claims to measure, and we like them for it, some harder, and we'll need a lot of assumptions and simplifications.
Lightcone forecast breakdown
It's hard to predict the future, even in broad strokes.
Let's take a look at each statement.
AI Research will win another Nobel Prize: not this year
It didn't happen this year. Same for the Abel Prize. Plenty of research happening that isn't related to AI, not much to discuss here, honestly.
My assessment was "Very Unlikely", so one in the bag, I guess.
Crypto will become mainstream: tough year for the haters, still no
I broke this claim into three smaller ones that are easier to judge.
Cross-border payments: Likely
Direct payments (in the US): Very Unlikely
Direct payments (global): Not Gonna Happen
Let's start from the easiest. Crypto didn't take over the world for payments. Instant payment systems are thriving, but they don't use stablecoins. Sure, crypto is booming, but nobody buys a coffee in USDC.
For cross-border, things change a bit. It still isn't going mainstream, in my opinion, but it sure seems to be making waves. You've got big players like Circle moving in (1, 2), and all the players I cited in the original post are doing pretty well.
It still wasn't this year, though. By any measure, banks and Wise are still the only mainstream channels for remittance.Direct payments: hit. Cross-border: miss. We're 3-1 so far.
Direct payments: hit. Cross-border: miss. We're 3-1 so far.
If DOGE works, DOGE will go up: two negatives don't make a positive
DOGE isn't going up.
From an objective viewpoint, DOGE didn't achieve it's declared goal of 1T USD savings. Vivek was out 2 days after the department was officially created, and Musk didn't last 6 months. It's not clear how much money the department is saving. That's a failure in my book. It's certainly more complicated than that, and smarter people than me are thinking about it, but there's no way to claim this was a solid success.
Either way, the mechanism just didn't work. The department was operating with Musk in charge up to May, and that's precisely when dogecoin went down! Even with lots of favorable crypto news throughout the year, it's still down 37% YTD.
I think the mechanism at play here isn't financial or regulatory. It's just optics. It's the typical "buy the hype, sell the news" cycle. It's a shitcoin, for Christ's sake. And the narrative broke.
When DOGE was announced, Musk was all the hype. They were gonna slash trillions in wasteful expenses, government was gonna have so much money, everyone would get a personal Optimus robot, interests rates would go down, whatever.
If we look at the graph from Aug 19 2024, when the "DOGE" name came to life as a tweet, to the actual date it was confirmed, Nov 12, DOGE went up 243%. In financial terms, that's a shitload.
Then the news comes in on Jan 20th (-11% from confirmation), the department is created, and now people are looking at how these cuts are gonna be made, and what's actually gonna be cut, and by the time Musk leaves the narrative has burst into flames (-43% from Jan20, or -49% from confirmation).
All that said, I'm going with a no, which is sadly a loss for me again, as I said this was "Likely". 3-2.
Zoom call with an AI: with huge caveats, no
AI for video made some large strides this year. It's now incredibly easy to violate copyright, create deepfakes of famous or anonymous people, get a job working remotely from North Korea or make brainrot. It's pretty good at generating content too, I guess.
This prediction, however, talked about a real-time conversation with a counterparty that was 100% AI (text, voice, video), and felt natural. We're not quite there yet,
That doesn't mean the world isn't getting crazier. We're way past the point for non-real-time video, with AI-generated videos for Instagram and TikTok, and great videos for scamming, and serious discussion of AI in the film industry, and AI ads. But it's not real-time yet, at least not without a person on the other side acting as a "basis" for the video.
Which sucks, by the way, for my prediction specifically, even though it's good for the world as a whole. I marked this as "Very Likely" - my most optimistic assessment! And another miss. 3-3. The future is hard.
A summary, and some comments
None of the 4 predictions came to pass. In the interest of consistency, I'll take my assessments for the crypto forecast separately, as 3 statements. That said, we've got 6 statements to evaluate.
Things didn't go as planned for me, but they sure went worse for the Lightcone 2025 forecasts. All in all, we've got a Brier Score of 0.43 on this one.
My own predictions for 2025
Oh, this will be a lot of fun. I was very wrong. But I'll get around to my own predictions when the year ends. By then, it'll be time to be wrong about the future again, with predictions for 2026!
The scoreboard, or taking myself seriously
This is our scoreboard for now.
It isn't much, but it's a start. From now on, we know how we'll measure the accuracy of our predictions. I just need to keep making silly forecasts, and in a few years we'll have a decently-sized sample to look at. It'll help us tell whether or not I'm a good forecaster. Honestly, I'm looking forward to it.
As an aside, did you know Google Sheets has a query function with it's own syntax? The results table was generated in one go with the query below. Pretty silly that you have to repeat the select statement in the "label" statement to rename the columns. The "AS" keyword has existed for a while. Anyways.
=QUERY(
Forecasts!1:1000,
"select A, count(A), sum(H) / count(H)
where A is not null
group by A
label count(A) 'Count', sum(H) / count(H) 'Brier Score'",
1)