AI governance has become increasingly sophisticated, with governments, regulators and technology companies building ever more elaborate systems of oversight. But Vendan Ananda Kumararajah argues that one question remains largely unanswered: who determines whether the institutions governing artificial intelligence are themselves still fit to exercise that authority?

Across governments, international institutions, standards bodies and technology companies, the architecture of AI oversight has become increasingly sophisticated. We now have risk-management frameworks, responsible-AI standards, safety evaluations, incident reporting, stakeholder consultation, transparency requirements and increasingly detailed controls over deployment.

Most governance systems, however, begin after authority has already been assigned. They tell governments, companies, regulators and developers how AI should be governed, but are much less likely to ask whether those actors retain the legitimate authority to govern in the first place.

In my previous article, AI governance has mapped the risks but who decides when AI is fit to act?, I argued that increasingly capable AI requires us to distinguish technical capability from legitimate agency. The same principle has an implication for the institutions overseeing it: if legitimacy is important for the system being governed, it is also important for the governor.

That is the question at the centre of my recent comparative study, The State of AI Governance: From Risk Management and Responsible AI to Recursive Legitimacy. The study examines national, international and corporate governance frameworks including NIST, OECD, Singapore, China, the United States, Canada, Microsoft, Anthropic and others.

Much of contemporary AI governance is impressively developed. The recurring blind spot is that the governing actor itself is rarely scrutinised in the same way as the AI system it oversees.

Consider a regulator. It may possess statutory powers to govern AI while lacking the technical knowledge, institutional independence or adaptive capacity required to exercise those powers effectively.

A corporation may have an elaborate responsible-AI programme while simultaneously being the developer, beneficiary, assessor and principal authoriser of the systems being governed. A government may possess formidable implementation capacity while organising that capacity around assumptions that are themselves never seriously reopened to challenge.

None of this means those institutions are illegitimate, but legitimacy cannot simply be assumed from position, capability or formal mandate.

My comparative analysis found precisely this distinction. Singapore demonstrates substantial implementation capacity. Microsoft documents extensive governance processes, including executive oversight, red-teaming, pre-deployment review, monitoring and feedback-driven change.  Yet operational sophistication and institutional capacity still leave a more fundamental question unanswered: when should the authority of the governing institution itself be reconsidered? That is the higher-order governance problem: governing the governor rather than just the AI.

One of the most important findings from the comparison is the distinction between instrumental reflexivity and ethical reflexivity. In plain English, instrumental reflexivity means improving how a system pursues its existing goals; ethical reflexivity means being able to question whether those goals, incentives and authority remain legitimate.

Many sophisticated AI-governance systems are highly capable of learning. They evaluate performance, monitor incidents, improve security, refine controls and respond to changing technical conditions. A governance system can therefore become better at achieving its objectives without ever questioning whether those objectives, incentives or authority structures remain legitimate. It can learn extensively while remaining normatively closed – able to improve its methods without questioning the values and assumptions that govern them.

A developer operating under intense commercial pressure may improve its safety mechanisms while never reopening the question of whether a particular capability should be deployed at all. A national strategy may become increasingly effective at accelerating AI adoption without revisiting the distribution of power, benefit or risk created by that acceleration. A regulator may refine its oversight procedures without recognising that information asymmetry or institutional capture has begun to shape what it regards as acceptable evidence. 

The important question is whether feedback can reach – and alter – authority itself. This is where my A3 Model – a framework I developed to assess whether governance remains ethically coherent, resistant to distortion and fit to exercise consequential authority – introduces the idea of recursive legitimacy. A3 does not seek to replace NIST, OECD, regulation, safety evaluation or corporate controls. Those frameworks provide essential specialist knowledge and mechanisms. Instead, A3 asks whether the actor applying those mechanisms remains fit and legitimately authorised to do so.

It examines that question through three interacting conditions: ethical coherence, systemic distortion and legitimate adaptive agency. In practice, that means asking whether the institution is still acting consistently with its ethical purpose, whether incentives or failures of information are distorting its judgement and whether it remains competent and legitimately authorised to exercise power.

A3 treats authority as challengeable, revisable and, where necessary, revocable rather than permanent. Instead of assuming that a regulator, board, government or developer is entitled to govern and then asking only whether it follows the correct process, the A3 approach – recursive governance – also asks whether new experience, evidence, consequences for those affected or emerging distortion should alter that actor’s authority.

In practical terms, governance must therefore contain mechanisms for correcting the AI system and, when ordinary correction is no longer sufficient, for constraining, withdrawing or reconstituting the arrangements governing it – in other words, limiting or withdrawing authority or fundamentally changing how the system is governed.

In frontier AI – the most advanced AI systems – a company may design the model, determine its risk thresholds, conduct evaluations, interpret the evidence, decide whether mitigation is sufficient and ultimately authorise deployment. Each individual mechanism may be rigorous yet the architecture still contains a structural question: who determines whether the institution making all of those judgements should continue to possess that authority?

Independent evaluation, regulation and external scrutiny all help but recursive legitimacy requires something stronger than additional observers: the possibility that evidence can alter the distribution of decision rights themselves.

Affected stakeholders must sometimes possess more than consultation rights. Regulators must themselves remain accountable for whether they remain competent and free from capture by the interests they regulate. Boards must be capable of losing decision-making discretion and developers must encounter boundaries they cannot simply reinterpret internally. Without those constraints, governance risks becoming an increasingly sophisticated form of self-authorisation.

The first phase of AI governance concentrated on trustworthy AI – making AI systems themselves safe, reliable and worthy of trust – and that work remains indispensable. 

But increasingly autonomous and consequential systems now create another requirement: trustworthy governance of AI – examining the institutions authorised to determine what risks society should accept, whose interests count and when deployment should proceed alongside the models, outputs and risks themselves.

AI governance has become sophisticated enough that its principal problem extends beyond a lack of rules to whether the institutions applying those rules can themselves be governed. Consideration needs to be given to who is fit to exercise responsibility for governing AI, how that fitness is demonstrated and what happens when it deteriorates.

For as artificial intelligence acquires greater agency, the legitimacy of those controlling it will matter every bit as much as the behaviour of the technology itself. 

If we are serious about governing AI, then, we must also be prepared to govern the governors.

Vendan Ananda Kumararajah is an internationally recognised transformation architect and systems thinker. The originator of the A3 Model—a new-order cybernetic framework uniting ethics, distortion awareness, and agency in AI and governance—he bridges ancient Tamil philosophy with contemporary systems science. A Member of the Chartered Management Institute and author of Navigating Complexity and System Challenges: Foundations for the A3 Model (2025), Vendan is redefining how intelligence, governance, and ethics interconnect in an age of autonomous technologies.

READ MORE: ‘AI governance has mapped the risks but who decides when AI is fit to act?‘. A major MIT study has mapped how serious the risks from artificial intelligence could become. But identifying the dangers is only part of the problem. Vendan Ananda Kumararajah argues that as AI systems gain greater autonomy, governance must also determine whether the systems – and the institutions controlling them – possess legitimate authority to act.

Do you have news to share or expertise to contribute? The European welcomes insights from business leaders and sector specialists. Get in touch with our editorial team to find out more.

_{Main Image: Leandro Alamino/Pexels}