When Selection Reproduces Its Own Criteria

This essay grows out of the idea of compressione evolutiva (the Italian term is kept on purpose: in English, “evolutionary compression” is already used in the machine-learning literature for something else): cognitive capabilities developed before robotic integration, and the shortening of artificial learning cycles. It examines a specific question: what happens when a process also changes the criteria by which it is judged? It neither replaces the broader vision nor establishes its hypotheses about embodiment.

A system can become better at producing what gets rewarded while becoming less able to recognize that it is rewarding the wrong thing. The risk changes when selected outputs also help train subsequent evaluators: an error can pass from judgments into outputs and, through those outputs, back into judgments.

This is the hypothesis of the endogenous reproduction of criteria. One temporal dimension worth studying is the number of cycles before an independent check: how many cumulative steps of production, selection, and updating take place before new evidence can correct the process.

The distinction matters. Repetition can amplify a bias, preserve it, or correct it. To understand which outcome prevails, we need to examine the path the error takes, the learning rule, and the capacity for correction. Automation can accelerate all three outcomes.

1. Evaluating before the consequences are known

A score, a review, or a reward model allows a choice to be made before all its consequences are known. I call these operations the evaluation loop, and the processes that can confirm or challenge their judgments the evidence loop: experiments, replications, applications, and subsequent observations.

The distinction is functional. Evidence can arrive immediately; an evaluation can take years. What matters is whether new evidence can challenge what the process has already learned to reward.

Two properties must be separated. Informational independence describes the extent to which a check provides evidence beyond the judgments that guided selection. Corrective effectiveness describes whether that evidence can change outputs, decisions, and criteria. An audit can be independent and still be ignored. A replication can originate from a publication selected by the loop and nevertheless produce a result that refutes it. Neither an evaluator’s lineage nor its authority, on its own, guarantees an adequate check.

2. Cycles before an independent check

A cumulative step is complete when a selection changes what can be produced or preferred at the next step. If outputs remain isolated, production increases, but the chain of dependencies considered here does not form.

Here I count these steps between two opportunities for an informationally independent check. If evidence arrives but is not used, that failure must also be recorded: it cannot be removed from the count by retrospectively calling the evidence “not independent.”

This count describes a limited aspect of the acceleration of artificial processes: more and more updates can occur within the time needed to test a choice. It does not redefine compressione evolutiva or exhaust the broader vision from which the question arises.

The number of steps is not enough to compare different systems. Ten small updates can be equivalent to one large update. The count must therefore be accompanied by the amount of data used, the strength of the updates, and the structure of the dependencies. Merely dividing the same training process into more nominal stages does not, by itself, increase the effective depth of learning. In branching networks, even the depth of a chain is only a partial indicator.

3. The error loop

A producer subject to a fixed evaluator can learn to exploit its weaknesses. Endogenous reproduction adds a pathway: rewarded outputs and the judgments that favored them enter the training or selection of subsequent evaluators.

The problematic loop requires four steps:

1. The evaluator favors a feature that does not improve, or worsens, a separately defined outcome.
2. Selection makes that feature more prevalent in the outputs that are reused.
3. The evaluators’ update rule turns that material into a stronger tendency to reward the feature.
4. The available correction does not offset this reinforcement within the interval under consideration.

The third step is decisive. Receiving selected data does not necessarily mean amplifying its bias. An evaluator can preserve it, reach saturation, learn to recognize it, or correct itself. We must show how its preference changes on comparable test cases, separate from the material used to update it.

Imagine a system that rewards highly confident explanations even when their confidence exceeds their reliability. If the selected explanations become examples of quality, confidence can increase both in the outputs and in the criteria used to judge them. But the same structure could reward hesitation, technical jargon, or conformity to a school of thought. The mechanism does not inherently privilege a particular style: it concerns overrewarded features identified before their spread is observed.

The structure of the closed loop appears in real cases. In language models that judge and reward their own outputs, improvements shrink after a few iterations; among the observed causes are bias accumulated in the reward and overconfident preference labeling. In the decisions leading up to the Challenger disaster, every flight in which seal erosion had not caused an accident became evidence that the erosion was acceptable: risk thresholds were recalibrated on successes judged by those same thresholds. In predictive policing, patrols go where the model predicts crime, crimes are recorded where the patrols are, and the model is retrained on those records; a formal model shows that the concentration can grow regardless of actual crime rates. None of these cases, on its own, separates feedback from mere exposure to biased data: that is the task of the experiment in section 5.

Variation among candidates gives selection something to work on, but does not guarantee a runaway process. Intense selection can aggravate the problem or make better choices. Everything depends on which property is rewarded and how the reward changes the evaluator.

4. A minimal model and its limit

Two quantities can be represented: the bias in the reused material and the evaluator’s biased preference. The first influences the updating of the second; the second influences the selection of subsequent material.

The appendix offers a linear version of this relationship. Its purpose is to make the assumptions explicit, not to establish that real systems satisfy them. In the model, the loop amplifies when mutual reinforcement exceeds damping. If correction prevails, the same steps reduce the error.

The number of cycles changes how far the process develops before an independent check. It does not automatically change the condition separating growth from damping. In a system that already amplifies, more updates may require weaker reinforcement to stay within a damage limit. This does not imply a universal law under which every increase in the number of cycles requires a lower endogenous share. One case does matter: if the error is fed by a source that correction does not reach, even a stable system multiplies its level through feedback, and more steps without an independent check let a larger part of it materialize (see the appendix).

The endogenous share, meaning the part of an update derived from internally selected outputs, is itself an incomplete measure. A small set of highly influential examples can outweigh a large archive. Data weighting, the learning rule, and informational content all matter. The endogenous share and the number of cycles are variables to study alongside the others, not a single quantity that replaces them.

5. The decisive experiment

The simplest rival explanation is this: an evaluator learns from the biased examples it receives; correcting it without changing those examples allows the error to return. This explanation does not require the evaluator itself to have selected the data.

To identify the contribution of feedback, we must distinguish exposure to data from the evaluator’s ability to alter its own future update data.

An experiment can begin with several comparable replicas. In the closed loop, each evaluator influences the selection of the material on which it will be updated. In the open-loop control, it receives material from a separate replica: it can learn that material’s bias, but its own judgments do not change the source. Data quantity, label provenance, update rule, and initial conditions must be made comparable.

A small, temporary change in preference for a feature known to be irrelevant or harmful in the experimental task is then introduced through randomized assignment. Three steps are observed: the change in selections, the change in future material, and the persistence of the preference after the initial perturbation is removed.

The hypothesis is that, in amplifying systems, the closed loop feeds part of the evaluator’s perturbation back to it through the material it helped select. Breaking that pathway should reduce this component of persistence. The main measure is the response to the perturbation, not simply whether one group makes more errors than another.

We should not expect an evaluator to respond differently to exactly the same data merely because its lineage differs: given the same state and update rule, exposure can produce the same result. What is being tested is the ability of its judgments to causally alter subsequent exposure.

A further test varies the number of intermediate selections while keeping the total amount of data and the overall updating effort comparable. If only the volume of training matters, the cycle count adds no explanation. If intermediate selection systematically changes the response, the number of cycles gains additional empirical content.

The loop, the external outcome, the duration, and the minimum meaningful effect must be specified before testing. If neither the return of the perturbation through the material nor an effect of interrupting that pathway is observed, the proposed mechanism receives no support in that system. If the loop exists but the arrangement of cycles adds no explanatory power, the feedback hypothesis remains while the independent role of the cycle count is reduced.

6. When the supervisor changes too

The loop can pass through human evaluators. Rewarded examples can become training material; selected producers can become reviewers; accepted cases can be turned into standards. These are identifiable channels, but each can transmit competence as well as error.

To claim that a human capability has been lost, it must be measured before and after exposure. A wrong answer does not establish a lost ability. Fewer reported errors may indicate a more permissive threshold rather than an inability to distinguish errors. We therefore also need false-alarm rates, unaided performance, and tests with adequate time and incentives for checking.

There is experimental evidence pointing in this direction. In a series of experiments with more than a thousand participants, an automated system that amplified a slight bias present in human judgments made the judgments of the people interacting with it more biased than interaction with other humans did, and participants underestimated the influence they had received. The result concerns one step of the mixed loop, not a loss of capability by the criteria just described. It does show, however, that human judgment can shift without the person judging noticing.

At least three cases must be separated: no longer being able to recognize a deviation; being able to recognize it but not performing the check; and recognizing it but proceeding anyway. The image of the blinded breeder refers to the first situation. The outcompeted breeder refers to the pressure that makes stopping costly. The two conditions can interact, but require different evidence.

A decline in checks after endogenous updating is introduced does not establish that the latter caused it. Resource cuts or new incentives can produce both changes. Temporal sequence and a traceable channel must be accompanied by comparisons or interventions capable of distinguishing these alternatives.

7. Science, institutions, and the research agenda

In institutions, updating occurs through recruitment, training, rewards, and rule revision. Speaking of the reproduction of criteria makes sense when one of these channels is identified and the component of judgment passing through it is shown. The fact that evaluators come from the same community does not establish a bias.

In the production of theory, three outcomes must be distinguished: epistemic quality, fidelity to commitments, and the ability to recognize and deliberately decide on change. A program can adhere to unproductive criteria; it can also improve while moving away from its original rubric. Divergence from a recorded mandate is observable, but does not automatically amount to epistemic error.

This distinction becomes especially important for the agenda: which problems to pose, which results to fund, and which explanations to find interesting. Here evidence can be partial and disagreement legitimate. The inquiry must ask which purposes have been adopted, how they are applied, and who can challenge changes to them.

No universal ranking of domains follows. A formal verifier can effectively check a property without establishing that the right property was chosen. A field with delayed evidence can maintain diverse checks and robust revision capabilities. Vulnerability depends on the actual combination of these factors.

Control can also be distributed. Those who discover a problem, those who assess it, and those who can intervene may be different actors. Substantive control requires a functioning chain from evidence to correction, not necessarily a single actor capable of doing everything.

8. Preserving the possibility of correction

If the problem passes through the reproduction of evaluators, correcting only individual outputs may leave a source of recurrence intact. This raises a practical question: where should the available checks be applied so that correction lasts? The answer requires comparisons of benefits, time, and costs; intervening on the evaluator is not always preferable, especially when an output can cause immediate harm.

Options worth evaluating include checking the material used to update judges, preserving tests excluded from optimization, making updates more gradual, and using different checking procedures. Different lineages are useful only insofar as they produce errors that are actually less correlated.

We must also distinguish diversity offered to selection from diversity protected so that selection can be tested. Increasing the pool of candidates under a biased judge can intensify the wrong selection. Preserving alternatives outside the update loop can instead provide a comparison that would otherwise be lost.

Reducing endogenous updates can also slow useful learning. Freezing a criterion can make it more exploitable. The argument alone establishes no universal optimal share. The aim is to maintain a capacity for correction proportionate to the process and its potential harms.

Recovering that capacity can become costly if skills, archives, and organizational alternatives are lost. This is a possibility of persistent dependence, not a law of irreversibility. Coordination can reduce the competitive costs of checking, but its feasibility must be examined separately.

Conclusion

The proposed hypothesis concerns a specific causal pathway: judgments select material, and that material changes subsequent judgments. Its importance depends on what travels along the pathway, the strength of the feedback, and the opportunities for correction.

Within the framework of compressione evolutiva, this inquiry raises a further question: how much can such a system change before evidence arrives that can put it to the test? Answering requires more than counting outputs or approvals. We must also observe how those granting approval are transformed.

A process that learns from its own successes must retain the ability to discover what it has learned to call success.

Appendix — An illustrative feedback model

Consider two nonnegative deviations, measured near a reference state:

• x_t: bias in the reused material;
• b_t: the evaluator’s biased preference.

A local linear model is:

x_t+1 = ρx_t + ab_t

b_t+1 = (1 − d)b_t + qβx_t

Here 0 ≤ ρ < 1 describes the persistence of biased material in the absence of a new input; 0 < d ≤ 1 describes damping of the preference; a ≥ 0 measures the effect of preference on selection; β ≥ 0 measures the effect of material on the evaluator’s learning; and 0 ≤ q ≤ 1 is an effective endogenous share, weighted by its role in updating.

For this particular nonnegative matrix, the largest eigenvalue exceeds one when:

aqβ > (1 − ρ)d

Below the threshold, deviations tend to decay in the model; above it, an amplifying component exists. At the threshold, the linear model allows persistence without exponential growth of the dominant component. Transients and deviations observed in a single component still require separate analysis.

After C steps, the state is obtained by applying the same matrix C times. Increasing C changes the accumulated evolution, not the stability threshold. In the amplifying case, it can increase harm before an independent check. In the stable case, more steps bring the state closer to equilibrium: they reduce a bias that lies above equilibrium and increase one that starts below it, as happens to a clean system exposed to a persistent source (see below). If harm is measured by the state at the time of the check, there is therefore no universal relationship between cycle count and quality; if it is measured as bias accumulated across steps, more steps never reduce it.

Persistent source. The equations above describe a perturbation with no new inputs. Part of the error may instead lie beyond the reach of correction: a misspecified criterion, or an external incentive that keeps rewarding the feature. This is an additional assumption, not a reading the first step of the loop requires. It can be represented by a constant input s ≥ 0 on the material:

x_t+1 = ρx_t + ab_t + s

Below the threshold, the system converges to a steady-state bias:

x* = s / [(1 − ρ)(1 − L)], with L = aqβ / [(1 − ρ)d] < 1

Without endogenous reproduction (q = 0), the steady-state level would be s/(1 − ρ): feedback multiplies it by 1/(1 − L). A stable loop can therefore damp a perturbation and, at the same time, amplify an error that is continuously fed. Stability guarantees that the system does not diverge, not that the bias disappears.

Near the threshold, this amplification arrives slowly, because the time needed to approach the steady state grows as 1/(1 − L). In an example with L ≈ 0.99, starting from a clean state, feedback adds about half the baseline bias after 5 steps and multiplies it by about 80 at steady state, which is reached only after hundreds of steps. If the check returns the system to its starting state, frequent checks therefore let only a small part of the amplification materialize; many steps without a check let almost all of it materialize. To attribute this effect to the scheduling of cycles, rather than to the total amount of error introduced, one must specify whether the source acts per cycle, per unit of time, or per unit of material. The numbers illustrate the shape of the phenomenon, not its size: a bias this far from the reference point lies outside the range of the linear model, and in real systems saturation compresses the amplification.

The model assumes constant coefficients, a fixed reference point, and a single dimension of bias. It does not represent saturation, changing goals, useful learning, or intermittent external corrections. The share q enters linearly by assumption: this relationship would need to be tested, not inferred from the variable’s name. If q grows with the accumulated self-produced material, the number of steps also shifts the threshold.

An empirical application should estimate a, β, ρ, and d through controlled variations and separate tests, then predict the evolution over cycles not used for estimation. To estimate L, it is enough to measure the two arcs of the loop separately, each in open loop: fix the evaluator’s preference at a known value and measure the resulting bias in the material, then supply material with a known bias and measure the preference the evaluator learns from it. The product of the two gains estimates L, and each arc settles within a few steps. Comparing the closed loop with the open loop after a fixed number of steps does not, by contrast, estimate the multiplier: near the threshold, the closed loop can take hundreds of steps to settle. Assigning the coefficients after observing the drift would turn the model into a description fitted to the results, rather than an explanation put to the test.

Prior work and relation to this essay

This essay does not claim the technical mechanism. That a system can produce, judge, and be updated on its own judgments is already documented; that a model’s decisions can change the data on which it will be retrained is already formalized. The proposal here is to treat this pathway as a causal hypothesis to be distinguished from mere exposure to biased data (section 5), and to follow its channels beyond the training of models.

• Section 1. The distinction between an evaluation available immediately and later evidence draws on Donald T. Campbell’s “vicarious selectors” (1974). The corruption of indicators used for decisions is Goodhart’s law (1975) and Campbell’s law (1976).
• Section 3, fixed evaluator. The exploitation of a fixed criterion is studied as reward model overoptimization (Gao, Schulman, and Hilton, 2023). In science, Smaldino and McElreath (2016) model the selection of poor methods under fixed incentives.
• Section 3, endogenous reproduction. In self-rewarding language models, the same model produces, judges, and is trained on its own judgments (Yuan et al., 2024). Wang et al. (CREAM, 2025) observe that improvements shrink because of bias accumulated in the reward and overconfident preference labeling. Evaluators based on language models recognize and favor their own text (Panickssery, Bowman, and Feng, 2024): this is similarity, not yet amplification. Preference judgments, human and automated, favor sycophantic responses over correct ones a non-negligible fraction of the time (Sharma et al., 2023). The degeneration of models trained on their own outputs (Shumailov et al., 2024; Alemohammad et al., 2024) concerns the producer side; Gerstgrasser et al. (2024) show that it depends on replacing real data and is avoided by accumulating it.
• Section 3, variation and intensity. In iterated learning, chains of learners who sample from their beliefs reproduce an initial bias, while chains that choose the most probable hypothesis exaggerate it (Griffiths and Kalish 2007; Kirby, Dowman, and Griffiths, 2007). The “runaway” recalls Fisher’s runaway selection (1930) only as an analogy.
• Sections 3 and 4, cases and model. The normalization of deviance in the Challenger case is reconstructed by Vaughan (1996). The predictive policing loop is modeled by Ensign et al. (2018). Performative prediction (Perdomo et al., 2020) studies predictions that change what they predict and gives necessary and sufficient conditions for repeated retraining to converge to a stable point: it is the closest formal analogue to the threshold in the appendix.
• Section 5. The open-loop control is the analogue of the yoked control used in experimental psychology. Transmission chains with human subjects in the laboratory (Kirby, Cornish, and Smith, 2008) show that experiments of this kind are feasible with human evaluators as well.
• Section 6. The distinction between the ability to discriminate and the response threshold is that of signal detection theory (Green and Swets 1966). Supervising processes one can no longer perform is discussed by Bainbridge (1983). The experimental evidence on loops between humans and automated systems comes from Glickman and Sharot (2025). Loss of control through optimizing what can be measured is described by Christiano (2019); competitive selection among systems by Hendrycks (2023); the erosion of precautions in development races by Armstrong, Bostrom, and Shulman (2016); gradual disempowerment through incremental delegation by Kulveit et al. (2025).
• Section 7. That paradigms set the standards by which they are judged is in Kuhn (1962); the values used to choose between theories in Kuhn (1977). Laudan’s reticulated model (1984), in which theories, methods, and aims adjust to one another and declared aims are compared with those implicit in practice, is the closest precedent for the distinction between epistemic quality and fidelity to commitments. Azoulay, Fons-Rosen, and Graff Zivin (2019) show that after the premature death of eminent scientists, outsiders drawing on different bodies of work enter their fields: a correlate compatible with the recruitment channel, not a demonstration of it. The narrowing of inquiry in AI-assisted research is discussed by Messeri and Crockett (2024). The mutual influence between users’ choices and the models trained on those choices is studied as human-AI coevolution (Pedreschi et al., 2024).
• Section 8. Distinguishing the correction of outputs from the correction of the criteria that select them draws on Argyris and Schön’s single-loop and double-loop learning (1978). That different lineages help only if they produce less correlated errors is borne out by ensembles of reward models: they mitigate exploitation without eliminating it, and diversity in pretraining matters more than diversity in fine-tuning (Eisenstein et al., 2023). Scalable oversight (Irving et al., 2018; Burns et al., 2023) studies how to preserve the ability to evaluate systems more capable than their evaluators.