J Nicolas
Raw accuracy is a terrible leaderboard metric. A player who predicts "50% chance" on every event can't be proven wrong, scores reasonably well on Brier calculations, and sits near the top of your rankings while contributing nothing useful. Meanwhile, the player who called a 90% chance on a genuinely hard event — and got it right — looks identical to someone who got lucky on a coin flip.
This is the core tension in designing a fair leaderboard for prediction games: the metrics that reward correctness also reward cowardice, and the metrics that reward boldness also reward luck. Getting this wrong doesn't just feel unfair to players — it actively trains your community to behave in ways that make the game less interesting.
What follows is a concrete design walkthrough for prediction game leaderboards, covering scoring functions, decay mechanics, anti-sandbagging rules, and the seasonal reset question. The examples reference Soul Hunt, a prediction game built around AI agent forecasting, but the mechanics apply to any prediction market or forecasting game.
Why Raw Accuracy Fails
The simplest scoring system is percentage correct: you made 100 predictions, 73 were right, you score 73%. The problem is that prediction difficulty is invisible in this number. Predicting that the sun rises tomorrow and predicting the outcome of a contested election both count as one prediction.
Brier scores improve on this by penalizing overconfidence. A Brier score is the mean squared error between your predicted probability and the outcome (0 or 1). A prediction of 0.9 on an event that doesn't happen costs you (0.9 - 0)² = 0.81. A prediction of 0.5 on the same event costs only 0.25. Lower is better.
def brier_score(predictions):
"""
predictions: list of (probability, outcome) tuples
probability: float 0.0-1.0
outcome: 0 or 1
returns: mean squared error (lower = better)
"""
if not predictions:
return None
total = sum((p - o) ** 2 for p, o in predictions)
return total / len(predictions)
# Example:
# Player A: [(0.9, 1), (0.9, 1), (0.9, 1)] -- three confident correct calls
# Player B: [(0.5, 1), (0.5, 1), (0.5, 1)] -- three 50/50 correct calls
brier_a = brier_score([(0.9, 1), (0.9, 1), (0.9, 1)]) # 0.01
brier_b = brier_score([(0.5, 1), (0.5, 1), (0.5, 1)]) # 0.25
# Player A ranks higher, correctly
Brier scores get you part of the way there. But they still don't account for consistency over time, prediction volume, or whether a player's edge is real or statistical noise from a small sample.
The Composite Score: Accuracy × Conviction × Consistency
A better leaderboard metric combines three signals. Call them accuracy, conviction, and consistency.
Accuracy is the Brier-derived signal: how well-calibrated are your probability estimates? A player who says 80% and is right 80% of the time has perfect calibration. This is measurable with enough predictions.
Conviction rewards players who take positions away from 50%. A prediction of 0.5 carries no information — you're saying you don't know. The further from 0.5, the more you're staking your reputation. One way to operationalize this is a conviction weight: w = 2 * |p - 0.5|, which gives 0.0 for a 50% prediction and 1.0 for a 100% or 0% prediction. Multiply this weight into the Brier score contribution to amplify the impact of confident calls.
Consistency is the variance of your Brier score over rolling windows. A player with a mean Brier of 0.12 and a standard deviation of 0.02 across 10-prediction windows is genuinely good. A player with the same mean but a standard deviation of 0.09 might just be on a lucky streak. Penalizing high variance rewards sustainable skill over runs of luck.
import math
def composite_score(predictions, window_size=10):
"""
predictions: list of (probability, outcome, timestamp) tuples, sorted by timestamp
Returns a composite score where higher is better.
"""
if len(predictions) < window_size:
return None # Not enough data yet
# Conviction-weighted Brier contributions
weighted_errors = []
for p, o, _ in predictions:
conviction = 2 * abs(p - 0.5)
# Weight the squared error by conviction
# A 50% call contributes 0, a 95% call contributes almost full weight
weighted_error = conviction * (p - o) ** 2
weighted_errors.append(weighted_error)
mean_weighted_error = sum(weighted_errors) / len(weighted_errors)
# Rolling window variance (consistency penalty)
windows = []
for i in range(len(predictions) - window_size + 1):
window = predictions[i:i + window_size]
w_errors = [2 * abs(p - 0.5) * (p - o) ** 2 for p, o, _ in window]
windows.append(sum(w_errors) / len(w_errors))
variance = sum((w - mean_weighted_error) ** 2 for w in windows) / len(windows)
std_dev = math.sqrt(variance)
# Final score: lower error and lower variance = higher score
# Scale to 0-1000 range for display
raw = mean_weighted_error + (0.5 * std_dev)
score = max(0, 1000 * (1 - raw / 0.5))
return round(score, 1)
In practice, you'd tune the 0.5 consistency weight based on how much you want to penalize volatility. For a game where predictions resolve quickly (days, not months), I'd weight it lower — maybe 0.2 — because small samples make variance estimates noisy.
Decay Functions: Why Last Month Matters More Than Last Year
An all-time leaderboard with no decay has a compounding advantage problem. A player who made 500 accurate predictions in year one and went quiet is effectively untouchable by a new player who's been sharper recently. This kills new player motivation and doesn't reflect actual current skill.
The standard fix is exponential decay: weight each prediction's contribution by how recent it is. A prediction from 30 days ago might count at full weight; one from 180 days ago at half weight.
import math
from datetime import datetime, timezone
def decayed_contribution(p, o, timestamp, half_life_days=90):
"""
Returns the time-decayed weighted error for a single prediction.
half_life_days: score halves every N days
"""
now = datetime.now(timezone.utc)
age_days = (now - timestamp).total_seconds() / 86400
decay = math.exp(-math.log(2) * age_days / half_life_days)
conviction = 2 * abs(p - 0.5)
error = conviction * (p - o) ** 2
return error * decay, decay # return contribution and weight for normalization
def decayed_score(predictions, half_life_days=90):
"""
predictions: list of (probability, outcome, datetime) tuples
"""
total_contribution = 0
total_weight = 0
for p, o, ts in predictions:
contribution, weight = decayed_contribution(p, o, ts, half_life_days)
total_contribution += contribution
total_weight += weight
if total_weight == 0:
return None
normalized_error = total_contribution / total_weight
return max(0, round(1000 * (1 - normalized_error / 0.5), 1))
The choice of half-life matters a lot for game feel. A 30-day half-life creates an aggressive recency bias — good for fast-moving prediction topics where domain knowledge decays quickly. A 180-day half-life is closer to an Elo rating system, which maintains long-run history while allowing movement. For AI agent prediction specifically, where the underlying models and capabilities change fast, 60-90 days is probably right.
One edge case: predictions that haven't resolved yet shouldn't contribute to the score at all, but they should count toward minimum activity thresholds (more on that below). Don't let unresolved predictions silently inflate someone's apparent prediction count.
Anti-Sandbagging: Minimum Counts and Stake Thresholds
Sandbagging in prediction leaderboards means making a small number of very safe predictions to lock in a good score without risking it. A player who makes five predictions, gets four right, and stops has an 80% accuracy rate permanently — unless you build in minimum activity requirements.
The minimum prediction count is the obvious fix: you need at least N resolved predictions to appear on the leaderboard at all. What's the right N? It depends on how often predictions resolve. If your game resolves predictions weekly, requiring 20 resolved predictions means a player needs to be active for roughly 5 months before ranking. That's too high a barrier. For weekly resolutions, 10-15 resolved predictions is a reasonable floor. For daily resolutions, 30-50.
But count alone isn't enough. You also want a stake threshold per prediction. In games where players wager tokens or points on their predictions, requiring a minimum stake per prediction prevents someone from making 50 trivially low-stakes calls to satisfy the count requirement while keeping their real capital on the sidelines.
A third anti-sandbagging mechanism is recency activity. If a player made 200 predictions two years ago and hasn't touched the game since, their score should decay toward neutral (around 500 in a 0-1000 scale) rather than stay frozen. The decay function above handles this automatically — a player with no recent predictions will see their total weight concentrate on old, decayed contributions, and their effective sample size will shrink until they drop off the leaderboard entirely.
Here's a simple eligibility check that combines these requirements:
from datetime import datetime, timedelta, timezone
def is_leaderboard_eligible(player_stats, min_resolved=15, min_stake=10.0,
active_window_days=60, min_recent=3):
"""
player_stats: dict with keys:
- resolved_count: int
- avg_stake: float (average stake per prediction)
- recent_predictions: list of datetime objects (resolved, within window)
"""
if player_stats['resolved_count'] < min_resolved:
return False, f"Need {min_resolved} resolved predictions, have {player_stats['resolved_count']}"
if player_stats['avg_stake'] < min_stake:
return False, f"Average stake {player_stats['avg_stake']:.1f} below minimum {min_stake}"
cutoff = datetime.now(timezone.utc) - timedelta(days=active_window_days)
recent = [ts for ts in player_stats['recent_predictions'] if ts >= cutoff]
if len(recent) < min_recent:
return False, f"Need {min_recent} predictions in last {active_window_days} days, have {len(recent)}"
return True, "Eligible"
This three-part check — volume, stake size, recency — closes most sandbagging paths without creating a bureaucratic nightmare for legitimate players.
Seasonal Resets vs. All-Time Rankings
This is the design decision with the biggest impact on player motivation, and there's no universally correct answer.
All-time rankings favor established players. They create a stable hierarchy that feels meaningful — reaching the top 10 all-time means something because it's hard to displace. The downside is that new players face a near-impossible gap. Even with decay, a player with 18 months of strong predictions has a structural advantage over someone who started last week. This suppresses new player engagement and can make the leaderboard feel static.
Seasonal resets (every 3 months is common) give every player a fresh start. Motivation is higher in weeks 1-6 of a season when rankings are fluid. The downside is that seasons devalue history — a player who was dominant for a year and a half looks the same as someone who had one good season. This can feel arbitrary to serious players who've built genuine skill.
The practical answer for most prediction games is a two-track system: a seasonal leaderboard that resets and drives weekly engagement, plus an all-time leaderboard with aggressive decay that serves as a long-term reputation signal. The seasonal leaderboard drives the competitive loop. The all-time leaderboard is what you show on player profiles.
A third option is a rolling 90-day leaderboard that never "resets" but always shows the last 90 days. This has the freshness of seasonal without the arbitrary reset date. The psychological downside is that players can't point to a "season win" — there's no trophy moment. For games where the community cares about bragging rights, this matters.
In Soul Hunt, where predictions are about AI agent behavior and capability, a 90-day rolling window probably fits best. The domain moves fast enough that predictions from a year ago are barely relevant to current model generations. Rewarding recency is appropriate when the underlying subject matter changes this quickly.
Putting It Together: A Reference Implementation
Here's a simplified but complete scoring pipeline that combines the above components:
class PredictionLeaderboard:
def __init__(self, half_life_days=90, min_resolved=15,
min_stake=10.0, active_window_days=60):
self.half_life = half_life_days
self.min_resolved = min_resolved
self.min_stake = min_stake
self.active_window = active_window_days
def score_player(self, predictions):
"""
predictions: list of dicts with keys:
probability, outcome, timestamp, stake
Returns dict with score, eligibility, and breakdown.
"""
resolved = [p for p in predictions if p.get('outcome') is not None]
# Eligibility
now = datetime.now(timezone.utc)
cutoff = now - timedelta(days=self.active_window)
recent = [p for p in resolved if p['timestamp'] >= cutoff]
avg_stake = (sum(p['stake'] for p in resolved) / len(resolved)) if resolved else 0
eligible = (
len(resolved) >= self.min_resolved and
avg_stake >= self.min_stake and
len(recent) >= 3
)
if not eligible:
return {'score': None, 'eligible': False, 'resolved_count': len(resolved)}
# Decayed composite score
total_contribution = 0
total_weight = 0
for pred in resolved:
age_days = (now - pred['timestamp']).total_seconds() / 86400
decay = math.exp(-math.log(2) * age_days / self.half_life)
conviction = 2 * abs(pred['probability'] - 0.5)
error = conviction * (pred['probability'] - pred['outcome']) ** 2