Extract
Complete sessions were reviewed in a previously fixed order.
Audit
Retrospective audit · 2005–2018
What remains once the future has happened?
Cybersecurity conferences saw the themes coming. They did not always know how to measure them.
We turned 120 explicit predictions into tests with a deadline, threshold, and evidence. This is not a ranking of gurus: it is an audit of what future-facing talk can — and cannot — promise.
120predictions
with an expired horizon
with an expired horizon
36sessions
across three conferences
across three conferences
138S3 sessions
reviewed
reviewed
Black Hat USA · Chaos Communication Congress · Virus Bulletin
01 The finding
Among resolvable predictions
60.7%
cleared the threshold set before the outcome was examined.
That is 54 fulfilled predictions out of 89 resolvable cases. The other 31 were not automatically turned into failures: the available evidence could not apply the threshold without forcing a conclusion.
See the full distribution02 What we measured
A prediction is not a memorable line. It is a claim that can be wrong.
Before seeking outcomes, each row froze the verbatim quote, atomic restatement, deadline, indicator, thresholds, and admissible source types.
Freeze
The corpus was locked before outcome searches were enabled.
Check
Official, technical, and contemporary sources were applied to each threshold.
Keep the doubt
Where evidence could not resolve a case, it remained indeterminate.
The extraction register and outcome evaluation were kept as separate files, with verifiable SHA-256 hashes.
03 Full result
The result is a distribution, not a single headline.
Indeterminate cases matter: they show the cost of forecasting with vague metrics or incomplete historical sources.
Fulfilled5445.0%
Not fulfilled3529.2%
Indeterminate3125.8%
Sensitivity to uncertainty
45.0%
if every indeterminate case were a failure
60.7%
main analysis: indeterminate cases are excluded from the denominator
70.8%
if every indeterminate case were a success
The main analysis has a descriptive Wilson interval of 50.3–70.2%. In an exploratory session-clustered check, the approximate interval widens to 47.7–73.7%.
04 The useful reading
Getting that something will happen is not the same as…
…getting how, how much, or when right.
01
Theme emergence
Did at least one case occur? This is the dimension the corpus resolves most often.
02
Specific mechanism
Did it arrive through the technical architecture that was anticipated?
03
Scale and prevalence
Did it reach the promised share, volume, or diffusion?
04
Timing
Did it happen within the stated window, rather than years later?
Emergence
Mobile worms outside the lab
The 2006 prediction required autonomous spread on real devices. The threshold was met.45-CCC-2005-006-02Mechanism
Fingerprint matching did not move mostly to the cloud
The topic appeared, but the mechanism forecast for 2014 phones did not reach its threshold.34-CCC-2013-034-10Prevalence and timing
eCall was not widespread in 2016
Later adoption does not rescue the specific deadline and market-share claim.92-CCC-2014-045-0105 A concentrated result
One series contributes
73 of 120 predictions.
Security Nightmares reached 66.0% among resolvable cases; the rest of the corpus reached 53.8%. That gap does not prove that its speakers were better: it can also reflect format, topic type, and base rates.
06 Limits that change the reading
The result is useful
because it does not
pretend to be certain.
- A non-probability sample shaped by documentary availability.
- 115 of 120 predictions are class B: they required operationalization.
- One coder; the planned double coding was not completed.
- An internal preregistration, not deposited externally.
- No baseline or probabilities: there is no skill score.
- Proving a historical absence is often harder than proving an event.
Data and traceability
Every result should lead back to its row.
The research retains the text, threshold, and evidence for each prediction. This page is a doorway to that record, not a replacement for it.