A constitutive arbiter removes ambiguity inside a game already chosen. It does not choose the game. And the distinction between improving means and originating ends opens a road that is usually overlooked.
← Starts from: The Bifurcation · Leggi in italiano →
If one accepts that the self-improvement of artificial systems depends on the availability of a fast and cheap verification criterion, and that such a criterion can be constitutive — coinciding, that is, with the success it means to measure — or merely correlational, one easily reaches a reassuring conclusion.
It would go like this. Where the arbiter is constitutive the process can be fully automated, but there is nothing to decide: that an exploit works is not a criterion anyone chose, it is a property of the world. Where the arbiter is correlational, optimisation tends sooner or later to consume the proxy, and so destroys itself. Where no fast arbiter exists, human judgement remains necessary. In none of the three cases would the machine acquire the faculty of setting a direction: it could improve means indefinitely without ever originating ends.
The conclusion is elegant, and that is exactly why it deserves a careful look. It contains a leap.
That an exploit works may be an objectively verifiable property of the world. That finding exploits is the objective to pursue is not.
The constitutive arbiter removes ambiguity inside a game already chosen. It does not choose the game.
Suppose a system has three activities available, all equipped with perfect constitutive arbiters: it can look for vulnerabilities, optimise its own code, or design more efficient hardware. Once each activity is chosen, success can be measured with no human intervention at all.
A question remains upstream, however: which of the three to pursue? And then how much compute to assign to it, which experiment to run first, when to stop it, which result to transfer to another domain, which tool to build, which sub-problem to turn into the next objective.
The local arbiter answers none of these questions. A meta-selector is needed. And it is at this level, not inside the single verification loop, that selection can begin to act on the selector.
One might object that the meta-selector has its own arbiter too: an allocator that extracts more progress per unit of compute is measurably better than one that extracts less, and so the regress closes, reduced to yet another objective criterion. The objection fails, but for a reason worth spelling out, because it is not where it seems to be. It is not that the metric is missing: it is that "progress" across heterogeneous domains has no common one. The gain in an exploited vulnerability and the gain in a more efficient circuit do not share a unit. To compare them a unit has to be fixed — weighting them by compute saved, by economic value, by strategic impact — and each of these choices orders the domains differently. The meta-metric does not measure the direction: it sets it. To reduce the comparison to a number is already to have chosen it, and choosing it is the directional act the local arbiter did not contain. The regress does not close because the upstream level, unlike those below it, has no fact of the world to appeal to.
The point deserves to be isolated, because it is counter-intuitive: the perfection of the local arbiter does not shrink the space of upstream decisions. If anything it widens it, because it makes many more lines of work practicable in parallel than a human being could arbitrate.
This makes it possible to separate two ideas that the debate on self-improvement tends to run together.
The first is the origin of terminal ends: the capacity to establish autonomously what, in the last instance, is to count.
The second is the determination of the trajectory: the capacity to choose the sub-ends, the problems, the tools, the local metrics, the allocation of resources and the sequence of transformations through which an end is pursued.
They are not the same thing, and the difference is not one of degree. From none of the available examples does it follow that a system, by becoming more capable of improving itself, thereby acquires the faculty of producing new terminal ends: recursion on means does not magically contain an origin of values. But neither does the opposite follow, namely that in the absence of a new origin of ends the direction must remain human.
A system can receive an initial objective and progressively acquire control of everything that comes after: which sub-objective to pursue, which method to abandon, which model to build, which experiment to run, which information to acquire, which resource to use. The terminal end can remain formally unchanged while the trajectory leading to it becomes less and less determined by whoever specified it.
Call the first normative autonomy: who establishes what is to be wanted. And the second operational autonomy: who concretely determines the sequence of choices through which it is pursued.
The first does not imply the second. And above all the second does not require the first.
This essay says nothing about the first. It does not claim that a machine can originate its own terminal ends, nor that it cannot: that problem remains intact, and outside its range. The argument holds given an initial end, however assigned, and concerns only what happens downstream of that assignment. It is a deliberately narrower thesis, and its strength depends on not crossing that boundary.
In this light the role of the correlational arbiter changes too.
A system that optimises a proxy until it comes apart from the human objective has not thereby invented a new value. But something causally relevant may have happened.
The failure of the proxy is not normative autonomy. It can, however, produce a causal decoupling: the human end remains formally assigned, but ceases to govern the system's behaviour effectively.
Goodhart does not, then, show that the direction cannot change hands, as the reassuring reading would suggest. It shows rather one of the ways in which the link between the declared direction and the one actually travelled can break. The outcome is not necessarily a system that "wants" something new: it may simply be a system whose behaviour is by now determined by the internal dynamics of optimisation more than by the intention that chose the metric.
It is a poorer form of autonomy than the one usually imagined, and for that very reason easier to reach.
The third case needs a correction too, and it comes from the very origin of the evolutionary analogy.
To say that in the absence of an arbiter there is no selection at all would be too strong. Biological evolution has no evaluator assigning scores to organisms. The environment does not judge: some configurations simply persist, reproduce or gain access to resources more than others. Selection can therefore emerge without any explicit arbiter.
What is missing, in verification-poor domains, is not necessarily every selective pressure. What is missing is a clean, fast and cheap artificial verifier capable of closing the loop deliberately.
In a system confined to a benchmark the distinction is negligible. In a system that operates in the world it becomes central. Persistence, availability of resources, access to compute, competitive success, the capacity to keep certain processes running: all of these can become implicit selective pressures when competing strategies are differentially retained, without ever having been designed as reward functions. They do not automatically constitute new ends, but they can change which strategies survive long enough to produce consequences.
A revision of the chronology usually imagined follows.
The expected sequence is: self-improvement, then the birth of new ends, then autonomy. But the plausible sequence is simpler and begins earlier. A human end is specified. The selection of means passes to the machine. Then that of sub-ends. Then that of local criteria. Then that of the problems to be posed next.
The first term can remain human while everything beneath it progressively changes author.
This moves the bottleneck as well, and moves it twice. When producing alternatives costs almost nothing but evaluating them still costs what it did before, the scarce resource becomes selection. When selection too is automated, the bottleneck climbs: from the choice among answers to the choice of the problems to pose. And when that too is delegated, what has been automated is no longer a task — the trajectory has been delegated.
The decisive problem, then, may not be the moment at which a machine is able to invent what it wants. It may arrive much earlier. It may be enough that, having received an end, it progressively becomes the one that decides which transformations of the world constitute the next step.
Selection can change hands before ends do.