The constraint on the self-improvement of artificial systems is not the capacity to generate variants, but the possibility of selecting among them. And selection compresses only where an arbiter exists.
There is a question that recurs in discussions of the self-improvement of artificial systems, and it is almost always badly posed. People ask how powerful a system has to become before it can improve itself. But power is a matter of degree, and degrees do not produce a threshold on their own. The more useful question concerns something else: what decides whether one variant is better than the one before.
Biological evolution kept two phases apart. Variation was blind, with no direction anticipated. Selection emerged slowly from the consequences those variations met in the environment, across generations.
Machine learning can compress the first phase drastically. In many training procedures variation is no longer blind: an error signal steers the search through the space of possibilities. But the second phase does not compress as easily, because selecting quickly requires knowing quickly whether what has been produced is better.
This is where the most common mistake hides. One observes the acceleration of search and concludes that the whole evolutionary engine has been accelerated. It has not. Variation can be made abundant and cheap in almost any domain; selection compresses only where a verification criterion exists that is fast, reliable and cheap enough to enter the loop.
The constraint on self-improvement, then, is not — at least in the first instance — the capacity to generate variants; that can become almost unlimited. The constraint is the possibility of discriminating among them without reintroducing, at every step, a slow and expensive human judge.
From this a bifurcation arises: not a mere difference of speed along a single continuum, but different regimes, bound to produce different dynamics.
Offensive security offers a particularly clean case.
On InterCode-CTF, a benchmark of security challenges, earlier results reached about 40% with GPT-4o. A later elicitation effort reached 92% with the same base model in about five weeks; the overall result then rose to 95% using prompting, tools and multiple attempts. Google Project Zero's Project Naptime showed something similar on CyberSecEval 2: by changing the agent architecture around the model, success on the buffer overflow tests went from 5% to 100%, and on the advanced memory corruption problems from 24% to 76%.
The point is not that the model learned to be more intelligent in five weeks. The capability was in part already there: it was the system built around the model that could not yet extract it.
And offensive security has a decisive property: many of its checks close the loop without ambiguity. The exploit works or it does not. The flag is captured or it is not. The program runs or it fails. There is no need to ask anyone whether the result seems better.
The general progress of the field is consistent with this structure. The UK AI Security Institute reports that models now complete apprentice-level cyber tasks about 50% of the time, against little more than 10% at the start of 2024, and that in 2025 the first model was tested that could complete expert-level tasks, those that normally require more than ten years of human experience. The length of the tasks models can carry through unassisted doubles roughly every eight months. Stanford's AI Index 2026 observes more generally that benchmarks designed to stay hard for years are sometimes saturated within months.
Here, though, one has to stop before concluding too much. The five weeks of InterCode-CTF are not a demonstration of autonomous recursive self-improvement: it was human beings who experimented with prompts, tools, number of attempts and agent structure. They demonstrate something narrower and, for this thesis, sufficient: when the verification loop is strong and cheap, even relatively small changes to the process that generates the variants can yield enormous gains. It is the ground on which any recursive self-improvement would meet the least resistance. Not proof that the loop has already closed, but proof that in certain domains it can.
The symmetrical case would be a domain that lacks a check of this kind. But first it is worth examining one that seems to have it, because the mistake is instructive.
The automatic detection of AI-generated text has, at least apparently, the ideal arbiter. The ground truth is binary and can be known without annotation: if a system generates the text, whoever runs the experiment knows its provenance by construction.
And some contemporary detectors work far better than the widespread belief that detection is simply impossible would suggest. A 2025 NBER working paper compared several commercial detectors on human and artificial texts of varying length and provenance; the best achieved error rates close to zero in many conditions and kept performing well even against generic "humanising" tools. This should be said, because it is inconvenient for the prevailing narrative.
That does not, however, solve the problem.
To begin with, a detector's performance on one distribution does not guarantee the same performance on all the others. Liang and colleagues showed in 2023 that the detectors then available misclassified as artificial more than half of the TOEFL essays written by non-native English speakers, while behaving almost perfectly on native texts; the mechanism was associated with the lower perplexity and lower linguistic variety of those texts. There is no need to assume that this result describes all future detectors for ever. The theoretical point is a different one.
A detector does not directly verify the property of interest. It looks for features statistically associated with that property.
In offensive security, in many cases, passing the test is succeeding. If the local objective is to obtain a given flag by exploiting the system, the appearance of the flag is not a statistical clue that the exploit may have worked: it is part of the definition of success. Call this a constitutive arbiter.
In the detection of artificial text the situation is different. A certain distribution of words, a statistical signature, an internal representation may be strongly correlated with the provenance of the text, but they do not coincide with it. Call this a correlational arbiter.
The difference appears as soon as the criterion becomes a target of optimisation. Optimising against a constitutive arbiter tends to produce what the test defines as success. Optimising against a correlational arbiter can instead destroy the very relation that made the test useful. This is the structure described by Goodhart's law: when a measure becomes the direct target of optimisation, it may cease to be a good measure.
In text detection this produces a dynamic unlike that of cyber. The detector is improved; the generator learns to avoid the signals the detector uses; the detector has to adapt to the new distribution. Not necessarily convergence, then, but maintenance. It is worth noting that the authors of the NBER study themselves expect a continuing race between detectors, models and evasion tools, and recommend periodic audits on the model of bank stress tests — that is, they describe a regime of maintenance, not of accumulation.
Here the second part of the bifurcation emerges.
In a domain with a constitutive arbiter, an acquired technique can enter the system's cumulative stock. A faster algorithm stays faster. A formally verified proof stays a proof. A correctly exploited vulnerability remains one, even if the outside world may later change and make it necessary to find new ones. The target can move, but the local criterion of success is not destroyed by being optimised.
With correlational arbiters this is not guaranteed. If the system learns exactly which features trip the detector and removes them while its artificial origin stays unchanged, it has improved its performance against the criterion without moving a millimetre closer to the property the criterion was supposed to represent. One runs to stay still. And in some cases one may even run in the wrong direction.
In the constitutive regime, progress is deposited. In the correlational regime, it has to be defended.
If the distinction holds, one can predict how correlational domains will react under pressure: not by improving the proxy, but by replacing it with a predicate that is harder to manipulate.
In the case of content provenance the direction is already visible, and it is cryptographic attestation. Instead of trying to infer statistically whether a text is human, one verifies whether it comes from a signed chain. The move works because it changes the predicate. "Was this written by a human being?" remains a property that is hard to infer from the text alone. "Was this produced by this identity through this attested chain?" can become a verifiable property.
But the second predicate is not the first. A constitutive arbiter has been manufactured artificially where no natural one existed, and to obtain it the question has been changed.
Meanwhile, where not even this is possible, the cost does not disappear: it moves onto whoever selects. Medium chose not to entrust the decision on generated texts to automatic detection and transferred a substantial share of the selection burden to the editors of its own publications. The choice was probably dictated by costs and relations with authors more than by a technical impossibility, but its form is the one the thesis predicts: when selection is not automated, it stays human, and it is paid for.
The regress does not, then, end in verification. It ends in the institution that guarantees the signature, or in the person who reads and chooses.