2025 · last edited May 2026
Why even a superintelligence couldn’t answer every question we’d want to ask.
There is a common fantasy about superintelligence: that a sufficiently powerful mind could answer any question we could think to ask. We ask how a protein folds, what a market will do, which theory of physics is right, what a person will decide next year, and the machine answers. The fantasy treats intelligence as if it were not merely better cognition, but a kind of universal access to consequence.
That picture is wrong. A superintelligence would be much better than we are at finding structure, noticing which variables matter, choosing representations, proving theorems, and designing experiments. But intelligence is not omniscience. It is the ability to exploit structure, and, just as essentially, to find out whether there is structure left to exploit, often only by trying. A superintelligence would find the shortcuts that exist; it could not conjure the ones that don’t; and it could not always tell, in advance, which case it faced [1]. Where the structure is absent, irrelevant to the question, unavailable in the data, or computationally too expensive to unfold, even a superintelligence faces a barrier.
One reason this is hard to see is that the best examples of intelligence look like compression. Newton compressed falling apples and planetary motion into one law. Maxwell compressed electricity, magnetism, and light into a few equations. Darwin compressed a mass of biological facts into a mechanism. When intelligence succeeds, it often replaces a large, irregular-looking collection of cases with a smaller object that explains them. A law, a proof, a model, a sufficient statistic, an invariant: these are all ways of making the world smaller without losing what we care about.
That last phrase — “without losing what we care about” — does a great deal of work. There is no single compression of the world; there are many compressions, each preserving different information for different questions. A subway map, excellent if you want to get from Brooklyn to Queens, is useless if you want to know where the hills are. A thermodynamic description of a gas, excellent if you want pressure and temperature, is useless if you want to know which molecule will hit a particular patch of glass at noon. A theory can be a magnificent compression and still be useless for a question that asks for details the compression intentionally threw away.
Understanding, then, is not just the act of making things smaller; it is the act of making them smaller in a way that preserves the answer to some class of queries. There may be one compression that lets you predict the average behavior of a system, another that lets you control it, another that explains how it fails, and no compression that lets you recover its exact microscopic future. Superintelligence would be very good at discovering such representations, and would find compressions we never imagined. Even so, a compression is always indexed to a purpose.
Consider the strongest version of the fantasy. Suppose a superintelligence appears tomorrow and discovers the correct laws of physics. Not an approximation. Not a useful effective theory. The real rules. It writes them down, perhaps in a form too abstract for humans to read, but finite and exact. At which point many would say the future has now been solved.
It has not. The machine has the rules, not all the consequences of the rules.
A law of physics is like a program, and the future is its output; knowing the program does not always let you know that output without running it. Sometimes it does; sometimes there are invariants, conservation laws, symmetries, limiting distributions, mean-field approximations, attractors, or other handles that let you answer the question cheaply. Which is why you do not compute every collision in a gas to estimate its pressure. Good science often consists in finding exactly these handles.
But sometimes the handle breaks off. The system may have to be unfolded step by step. If you want a billionth decimal place, or the identity of one molecule, or the exact configuration after a long chain of unstable interactions, the useful abstraction no longer preserves the thing you asked for. At that point the problem is no longer “find the law.” It is “run the law.” This is the simulation barrier: the most vivid of several distinct walls, but not the only one.
That barrier is not merely that our computers are slow. More hardware helps when the computation can be parallelized, approximated, or shortened. But some computations, however much hardware you throw at them, appear to admit no shortcut of the kind we want. The only way to know the answer is to perform, or simulate, the process that produces it. This is computational irreducibility [2]: some processes do not admit a faster route to their results than the computation itself. The point is operational, not metaphysical: the future can be determined by laws and still be practically inaccessible.
Chaos is the most familiar wall of a different kind [3]: here the obstacle is not a missing shortcut but the unattainable precision of the inputs. Chaotic systems can be perfectly deterministic; the equations stay exact. Yet they are so badly conditioned that a tiny change in the present becomes, after enough time, a large change in the outcome. Michael Berry’s argument is the sharpest version [4]: a box of gas perturbed only by the gravitational pull of a single electron at the edge of the observable universe loses predictability after roughly fifty molecular collisions; bring the perturbation as close as people standing near a billiard table, and ideal balls go uncertain in six or seven [5]. The point removes a common escape route. Exact prediction would require exact isolation and exact initial conditions, and we can have neither. In an unstable system, a tiny perturbation is a seed.
This is the part the usual superintelligence fantasy misses: exactness in the laws does not propagate to exact inputs, to cheap consequence, or to useful consequence when the question hangs on a detail no abstraction preserves.
A superintelligence could still be extraordinarily powerful in such a world. It could extend horizons, calibrate distributions, find interventions robust across many possible futures, and tell you when an event resists detailed prediction even though its aggregate shape is stable. These are large abilities. They are not oracles.
The same gap appears the moment we leave physics for the body. Only now the unfolded consequence is a life. Of the roughly eleven thousand diseases catalogued in humans, only about six hundred, some five percent, have an approved treatment [6]; for the thousands we class as rare, the gap is starker still. Much of that is ordinary unfinished science: the right target not yet found, the trial not yet run. But part of it is structural. A drug’s effect is not a local fact about a single protein; it is the long-run behavior of a whole body, a system of feedbacks across scales. Even granting that we knew every molecular rule, the consequence at the level we care about often cannot be deduced from it; it has to be observed. Not by simulating a body atom by atom, which no one ever will, but by running a trial: when a consequence cannot be deduced, the cheapest way to learn it is to let it play out and measure, and even that cannot be hurried past the biology it waits on. Which diseases conceal a clean shortcut and which must be unfolded this way, we usually cannot say in advance. We find out by trying.
And the body, at least, does not revise itself because we published a prediction about it. Human systems do [7]. Human affairs contain physics, biology, cognition, institutions, incentives, and feedback. They also contain publication effects: a prediction about the system can become part of the system. A weather forecast does not negotiate with the cloud. A market forecast can move the market. A political forecast can change turnout. A public prediction about a person can alter that person’s incentives, self-conception, or constraints. In these cases the act of prediction is not outside the causal graph.
That does not make prediction impossible; it makes the question more specific — predict what, at what scale, conditional on what information, under what intervention, and against what standard of success? Whether a cure for a given disease can be built at all is something we may learn only after years of looking, if we ever learn it. Whether that cure would work in you, in particular, you might discover only by taking it. And how the market will receive a product you are about to launch is a question your own launch helps answer, so no study run beforehand can fully settle it. None of these is a failure of intelligence; they are different kinds of question, which is why “solved” and “unsolved” is the wrong way to sort them.
If opacity can survive even in a world we built ourselves, then it is not merely a symptom of nature’s messiness. Chess is that world: finite, fully known, every rule written by us. A perfect oracle exists in principle, a table assigning every position its true value, but it is astronomically large; the real question is whether some compact function could read a position’s worth without searching. Empirically, it cannot. The strongest search-free model yet built, a transformer trained on a top engine’s own evaluations, reaches grandmaster-level blitz and still falls short of the engine it imitated; its strength emerges only at large scale, with no compact evaluator in sight [8]. One cannot prove the shortcut is absent; chess is finite, so in principle it is there. But the evidence points to a wall, and no one could have certified that wall in advance: you learn it by building the evaluator and watching it fall short. What you pay instead is search: the unfolding, move by move, that no compact evaluator has yet been able to replace.
It is worth being honest about the status of that claim. The results behind it are real but general. They are theorems about the worst case, not about your case: they bound what any method can decide across all processes, not whether the one in front of you has a shortcut. For all they prove, the world might consist entirely of the tractable cases. The reason to doubt that is not a proof but a pattern: in the finite game and in the living body alike, the shortcuts that would make consequence cheap have not appeared, and we have usually learned this only by looking for them and failing. The wager here is that a superintelligence, for all its reach, would keep meeting the same pattern: not a proof that an answer is out of reach, but the discovery, made the hard way, that there was no shortcut after all.
Not every barrier is the same barrier, and most of the work is telling them apart.
Some are intelligence-limited: they wait for the right abstraction, proof, search method, or experiment design, and here a superintelligence is very strong. Many mathematical problems, engineering designs, and scientific theories fall partly in this class. The answer has structure; we have not yet found it.
Others are resource-limited. We know the form of the computation or experiment; it is simply too expensive: more compute, better instruments, more time, a larger dataset. A superintelligence can cut waste and improve design, but it cannot make atoms free.
Some are data-limited: the relevant information was never measured, was destroyed, or is so entangled with unobserved causes that many histories produce the same evidence. A superintelligence can infer more from traces than we can, but it cannot recover bits the world no longer contains.
A fourth kind is preference-limited. “What should we do?” is not always a hidden physics question. A superintelligence can expose tradeoffs and design mechanisms for bargaining or aggregation, but it cannot turn plural values into a theorem unless we supply a procedure for resolving them.
And some are simulation-limited. The rules may be known, the initial state known well enough, the question well-posed, and still the consequence has to be unfolded. If the system is chaotic, the horizon is short relative to the precision you can buy; if the computation is truly irreducible, there is no shortcut to find; if it is merely vast, as in chess, a shortcut may exist yet stay forever out of reach; and if the question asks which molecule hits which patch of glass at noon, every useful summary of the gas has already thrown that molecule away.
After all this, the practical value of superintelligence becomes clearer. It would sort questions better than we do, recognizing which barrier each query faces and choosing tools accordingly. But which barrier a question faces is not stamped on its surface; often the only way to learn that a problem had no shortcut is to have looked for one and failed. The taxonomy is in part a map drawn in hindsight. Even so, that alone would change almost everything. Much human effort is wasted by mismatching tools to problems: treating preference conflicts as technical puzzles, missing data as a regression failure, chaotic forecasts as a confidence failure, simulation barriers as a failure of imagination.
The future is not hidden only because we are stupid. Parts of it are hidden because the relevant computation has not happened yet. Parts are hidden because the world did not preserve the information needed to reconstruct them. Parts are hidden because every compression that answers one question throws away what another question would need. Parts are hidden because asking the question changes the system being asked about. And some of it is hidden in a way that hides itself: you cannot always tell, in advance, which kind of hidden you face.
A superintelligence would not answer every question we could ask. It would show us the shape of the ones it cannot.
[1] The claim that one often cannot tell in advance whether a shortcut exists has a precise backbone. There is no general procedure that decides whether a process can be compressed or sped up: the shortest description of a string — its Kolmogorov complexity — is uncomputable, and by Rice’s theorem the nontrivial properties of what a program computes cannot in general be decided from its description. Proving that no shortcut exists — a genuine lower bound — is among the hardest open problems in mathematics; we cannot even prove that P ≠ NP, that the canonical “hard” problems really are hard. So “no shortcut known” is usually a status, not a verdict.
[2] I use “computational irreducibility” in the standard informal sense: some computations cannot be sped up by a shortcut, so the only way to determine the answer is to perform or simulate the computation. That no shortcut exists is, for any given natural system, an empirical expectation rather than a proven theorem (see note [1]). (MathWorld)
[3] The chaos discussion follows the standard characterization of chaotic systems as deterministic, nonlinear systems exhibiting sensitive dependence on initial conditions and aperiodic behavior. (Stanford Encyclopedia of Philosophy)
[4] Berry’s example is stronger than the classroom “atom a light-year away” version. In “The Electron at the End of the Universe,” he argues that a single electron at the observable limit of the universe can destroy predictability in a gas after about fifty molecular collisions, and that nearby people can make ideal billiard-ball motion uncertain after about six or seven collisions.
[5] I first met this argument as an undergraduate at Wharton, in a statistics class where Dean P. Foster was substituting for Michael Steele. He set up the billiard table as a deliberately ideal universe — perfectly flat, perfectly rigid balls, no air resistance, no spin, perfectly known initial positions and velocities — and then asked how precisely the initial conditions would have to be known to predict the trajectory over some number of collisions. The arithmetic was simple enough to do on the board: if a tiny error in impact angle grows by a factor of ten per collision, then an initial angular error of one part in a googol, 10⁻¹⁰⁰, becomes order one after about a hundred collisions; more generally, the error after n collisions is roughly 10ⁿ⁻¹⁰⁰. Add even small external gravitational perturbations and the horizon collapses to a handful. The exact multipliers depend on table geometry, restitution, spin, cushion physics, and what counts as failure — none of which mattered for the point Foster was making, which I still find counterintuitive: exponential error growth converts absurd precision into ordinary ignorance.
[6] The count of roughly eleven thousand catalogued human diseases, of which only about six hundred (some five percent) have an approved treatment, follows analyses based on the Human Disease Ontology. The figure that around ninety-five percent of recognized rare diseases — variously counted at seven thousand to ten thousand — lack an approved treatment is standard in the rare-disease literature (National Organization for Rare Disorders; the U.S. Government Accountability Office’s 2024 report on rare-disease drugs, GAO-25-106774, puts the count at up to ten thousand, with about five percent treated). The word throughout is “treatment,” not “cure.”
[7] Tolstoy made an early version of this argument for history. In the Second Epilogue of War and Peace, and in the shorter 1868 essay “Some Words about War and Peace,” he contends that historians err by attributing events to the decisions of great men; the actual causes are the integral of countless small acts of will, and any genuine science of history would have to operate on infinitesimals it has no way to collect. The shape of the claim is the simulation barrier applied to human time: even granting laws of human nature, the consequences cannot be read off without inputs no one can recover.
[8] The search-free result is DeepMind’s “Grandmaster-Level Chess Without Search” (Ruoss et al., 2024): a 270-million-parameter transformer trained on Stockfish 16’s evaluations reached a Lichess blitz rating near 2900 with no search at play time, while remaining short of the search-based engine it learned from. The authors report that strong play emerges only at sufficient model and data scale.