As AI begins helping to develop and evaluate its own successors, flawed assumptions and biases could multiply with each generation, fuelling an ‘intelligence explosion’ in which increasingly capable systems become harder to govern. To avoid this, stronger independent evaluation and oversight are essential, argues Vendan Ananda Kumararajah

An AI system helps researchers design a more capable successor, and that successor contributes more effectively to developing the system that follows, with each round being able to accelerate the next.

This is the prospect behind an ‘intelligence explosion’: a reinforcing process in which better AI helps produce better AI. Whether it accelerates dramatically will depend on constraints that intelligence alone cannot remove. But the possibility raises a question that deserves attention before any explosion occurs: what, precisely, should count as ‘improvement’?

A successor might write better software, solve harder problems or accelerate scientific research. Those would be real gains in capability. Yet they would not settle whether its knowledge was better supported by evidence, its judgement less distorted or its wider agency more legitimate.

The distinctive challenge arises when AI participates in the process through which successors are produced and judged. Intelligence then helps shape the conditions under which future intelligence is recognised as better.

Consider a hypothetical development cycle. An AI helps propose a new architecture, generates training material, designs tests and interprets results. Human researchers retain responsibility but increasingly depend on its contributions when deciding what deserves further investment.

The successor performs better, yet part of that apparent progress may reflect assumptions shared by the system generating material, the tests assessing it and the explanations interpreting its success. What looks like independent confirmation may partly be repetition of the same underlying assumptions.

AI-assisted research therefore requires independent and adequate evaluation, especially where a system helps construct the measures against which its successor excels.

Human research already contains incentives, blind spots and contested definitions of progress. Recursive AI development could, therefore, accelerate both discovery and the reproduction of unexamined assumptions. A faster cycle may expose mistakes sooner or embed them more deeply before they are recognised.

The question is therefore larger than whether a successor passes its tests. It needs to be whether the development process can reveal what those tests fail to recognise, and change accordingly.

AI is already used to generate code, assist scientific enquiry, produce synthetic data, identify model failures and automate portions of evaluation. None of these uses alone constitutes an intelligence explosion. Together, however, they indicate why the process by which systems are improved matters as much as the performance gains eventually reported.

If a system contributes to research directions, training data, evaluation procedures and interpretations of results, then improvement becomes partly recursive. AI is increasingly involved in the machinery that determines what assessment means, as well as being the object assessed.

My A3 framework approaches intelligence in action through three simultaneous, mutually conditioning primitives – elements that operate together and shape one another: Aram, ethical coherence; Aanavam, endogenous systemic distortion, or distortion arising within the system itself; and Adhikaram, legitimate agency that must be earned, remain conditional and be capable of revision or withdrawal.

Applied to recursive AI development, these change the object of improvement. Assessment must concern not only a successor’s outputs but also the purposes, distortions and authority through which it is produced.

Aram asks whether the direction of improvement is ethically coherent: which problems are important, whose interests shape objectives, and what consequences would make apparent success unjustifiable. These questions belong within the choice and revision of objectives, before a performance gain is declared desirable.

Aanavam identifies endogenous distortion. Targets, training material, incentives and evaluation methods can narrow what becomes visible as knowledge. A successor may become increasingly effective at satisfying a measure while the development process becomes less able to recognise what that measure excludes.

This doesn’t mean that every model is trapped in a closed loop or that human evaluation is free of distortion but that recursive development gives special importance to whether the process can disclose and correct the assumptions through which it judges itself.

Adhikaram asks what agency such changing conditions justify. Authorisation to assist research does not automatically authorise a system to select objectives, certify outcomes or deploy successors. Greater competence may warrant wider discretion, but only where the grounds for that discretion can be established, challenged and revised.

These conditions operate together. Ethical coherence depends on what can be known and how knowing may be distorted. Legitimate agency depends on both, while its exercise creates consequences that require renewed judgement.

A3’s Knowledge Continuum likewise treats Arivu, Vidya and Gnanam as distinct, co-present modes of knowing – different ways of understanding that coexist and inform one another. Arivu concerns what is known in a particular situation; Vidya concerns knowledge developed through organised and disciplined methods; and Gnanam concerns understanding how knowledge fits together, where its limits lie and what acting on it might mean. 

They don’t form successive stages, and the relevance here is simple: information processing and task performance cannot exhaust intelligence. Assessment must also concern the grounds for claims, the position and limitations of whoever or whatever makes them, and the consequences of acting on what is claimed to be known. Technical improvement remains real. A more accurate model is more accurate, a faster research process is faster, and a system that solves a harder problem has achieved something significant. But technical improvement is an incomplete account of improvement in intelligence in action.

If successive systems change how research is conducted and evidence is evaluated, their governing conditions must be reconsidered alongside those changes.

A model authorised to recommend experiments may develop the capacity to execute them. A system helping to evaluate successors may become influential in deciding which successor is deployed. Each transition changes what is at stake, what evidence is needed and who must remain answerable.

Alignment to specified objectives cannot alone settle whether those objectives, evaluation procedures and permissions remain legitimate after such transformations. The institutions governing development must be able to revise their assumptions and authority structures, as well as the systems’ mandates.

This is where A3’s account of absorption and closure come into their own. ‘Absorption’ means learning from experience in ways that change what future systems can treat as reliable knowledge and what distortions they must address. ‘Closure’ means carrying those lessons through into the rules and authority governing what happens next. Discovering a weakness should change what the next system can treat as warranted knowledge, which distortions require correction and what agency remains justified.

Suppose an AI-assisted successor achieves better diagnostic accuracy. Investigation then reveals that its evaluations underrepresent particular patients and that clinicians struggle to challenge its confident recommendations. Absorbing that experience requires changes to evidence, testing, clinical responsibility and deployment authority. A higher aggregate score cannot resolve those findings.

The successor may need a narrower remit while independent evidence is gathered. Its development process may require different evaluators. The institution may need to change who can approve deployment and how patients or clinicians can challenge decisions. Closure reaches the authority governing succession itself.

This is the crucial point. A successor does not inherit its predecessor’s authority as though authority were a software dependency. It may be more capable yet have a weaker claim to broad discretion because it operates in a new domain, relies on a different evidential basis or acts at a speed and scale that existing oversight cannot govern adequately.

Equally, a carefully demonstrated capability gain may justify a bounded expansion of agency where new arrangements for evidence, accountability and review have been established. The principle is that the relationship between agency and capability must remain contestable.

An intelligence explosion would compress the time available for these judgements. It could make inherited permissions obsolete more quickly and allow unexamined assumptions to travel through several generations of systems.

Developers, boards and public bodies must therefore distinguish capability advances from justified expansions of agency. They need evidence not only that a successor performs better but that uncertainty and distortion can be exposed, contested and corrected within the development and deployment process.

They must also recognise improvement that entails relinquishing authority. A more capable system may deserve less discretion in a particular setting because its effects exceed the evidence or institutional arrangements supporting its use.

The future of AI will not be determined only by whether systems can build more capable successors but by whether each successor improves the conditions under which intelligence knows, judges and acts.

Vendan Ananda Kumararajah is an internationally recognised transformation architect and systems thinker. The originator of the A3 Model—a new-order cybernetic framework uniting ethics, distortion awareness, and agency in AI and governance—he bridges ancient Tamil philosophy with contemporary systems science. A Member of the Chartered Management Institute and author of Navigating Complexity and System Challenges: Foundations for the A3 Model (2025), Vendan is redefining how intelligence, governance, and ethics interconnect in an age of autonomous technologies.

READ MORE: Who governs 900 billion AI agents?. Huawei’s forecasts of an agentic future expose a question deeper than computing capacity: how can institutions ensure that autonomous systems retain legitimate authority as they learn, coordinate and act, asks Vendan Ananda Kumararajah.

Do you have news to share or expertise to contribute? The European welcomes insights from business leaders and sector specialists. Get in touch with our editorial team to find out more.

_{Main Image: panumas nikhomkhai / Pexels}