AI governance: a complete framework for steering your autonomous agents

88% of organisations use AI in at least one business function (Deloitte, State of AI, 2025) and fewer than one in five has a formal governance framework for its autonomous agents. The gap matters, because an agent does not just answer: it acts on real systems. This guide sets out what AI governance is, what the arrival of autonomous agents changes, the five steps to put it in place and the five pillars of a framework that holds.

What is AI governance?

The definition and scope of AI governance

AI governance refers to the set of rules, responsibilities and processes that structure the rollout, oversight and responsible use of AI systems in an organisation, throughout their life cycle. It defines who can do what, within what clear framework of responsible governance and according to which criteria of responsible use. For each risk level, the required human oversight is precisely calibrated. That way, every system is steered reliably and transparently. Its scope covers all AI systems: language models, autonomous agents and automation systems.

Did you know? 88% of organisations use AI in at least one business function (Deloitte State of AI 2025). Yet fewer than one in five has a formal governance framework to ensure the responsible use of its AI agents. Good practice for effective governance means clearly defined human intervention for each type of agent.

Classic AI governance vs agentic AI governance: why the distinction matters

Traditional artificial intelligence governance oversees a model's outputs: bias, accuracy, compliance of answers. It is passive governance, focused on data and results. For a company using only non-agentic LLMs, this level of oversight can be enough.

Agentic AI governance is of a radically different nature. An autonomous AI agent does not analyse: it acts. It executes workflows, accesses internal systems and makes decisions without direct human oversight. Governing an agent means controlling its actions in real time, not just auditing its outputs.

This requires specific tools: an agent registry to inventory every agent in production, a policy engine to apply rules on the fly and a kill switch to stop any non-compliant behaviour immediately. Agentic control mechanisms operate at a granularity that classic governance tools do not cover.

What changes in 2026 for AI governance?

Regulatory and competitive pressure, combined, makes 2026 a year nobody can sidestep.

On the regulatory side, the European timetable has shifted. The transparency obligations in Article 50 of Regulation (EU) 2024/1689 have applied since 2 August 2026, while compliance for the high-risk systems in Annex III has been postponed to 2 December 2027 by the digital omnibus regulation that entered into force on 27 July 2026. The postponement buys time; it does not change the obligations themselves: they remain on the books.

On the market side, organisations that structure their AI governance now gain a lasting competitive lead. They secure their AI innovation while keeping their speed of execution. Responsible innovation and regulatory compliance become their competitive advantage.

Expert tip: Do not wait until July 2026 to map your AI agents. Regulatory compliance takes 3 to 6 months on average for a mid-sized company. The organisations that start now are the ones that will avoid penalties and have operational AI governance in place before the deadline.

The three levels of AI governance: individual, organisational, regulatory

An effective AI governance framework operates on three complementary levels at once:

1. Individual level: user training and making teams accountable for good practice when using AI agents, for example the rules for using HR or sales agents
2. Organisational level: AI committee, internal policy, approval processes and oversight bodies for each deployment
3. Regulatory level: regulatory compliance (AI Act, GDPR) and applicable sector standards

These three governance frameworks work as a cascade: a robust regulatory framework imposes organisational rules, which in turn translate into individual practices.

Who leads AI governance: CISO, CAIO, CFO, IT director

AI governance is a cross-functional topic that involves several executive functions. Four key roles carry operational responsibility:

• CISO: oversight of agent security, data protection, OWASP Agentic AI compliance
• CAIO / CIO: AI strategy, use case selection, weighing risks against benefits
• CFO: financial attribution of AI costs per agent and per team, ROI validation
• IT director: integrating agents into the existing information system and implementing governance tools

The central governance body, usually an AI committee, coordinates these four functions on a monthly basis. It approves new deployments and rules on anomalies.

Which indicators show that governance works?

Governance that holds is verified on facts, not on stated intent. Four indicators are enough to reveal areas of residual risk:

• The share of agents actually inventoried in the central registry, compared with those discovered later by chance
• The effective time to stop a faulty agent, measured in a real exercise and not on a written procedure
• The share of anomalies documented with root cause analysis, rather than closed without explanation
• The traceability of decisions made by an autonomous agent: what, when, on which sources

None of these four indicators can be inferred from an internal questionnaire. They are read from the execution logs.

The critical stakes of agentic AI governance

Why is traditional AI governance no longer enough for autonomous agents?

Tools designed to govern LLMs are not suited to autonomous AI agents. The scope of control is fundamentally different.

The table above illustrates the structural gap between the two approaches. An AI agent can access your CRM, send emails, approve orders and modify files, without any human validating each action. Governing that requires an operational kill switch, an up-to-date agent registry and a policy engine that enforces rules in real time. Human oversight can no longer be manual at this scale: it has to be tooled.

Security risks and exposure of sensitive data

A poorly governed AI agent is a significant attack vector, because of its level of access to critical systems. Without guardrails, it can access internal databases without restriction, bypass SSO access controls, exfiltrate sensitive data via the Model Context Protocol (MCP) or compromise the confidentiality of customer data.

The OWASP Agentic AI Top 10, published in December 2025, lists the ten priority attack vectors specific to autonomous agents. A security strategy suited to autonomous agents builds on this reference. Some examples of priority risks: privilege escalation through an agent's tools, malicious prompt injection and guardrail bypass through combined sequences of actions.

Security risk management starts with a risk assessment of each agent according to its level of access to systems and data. An agent with administrator access to the information system requires far stricter guardrails than a read-only reporting agent.

Algorithmic bias and the company's legal liability

When an AI agent makes decisions that affect people (recruitment, credit approval, sales targeting), the company incurs legal liability if algorithmic bias is proven.

AI Act Art. 14 requires effective human oversight for high-risk AI systems. On documentation, identified biases and corrective measures must be traceable and auditable. Regular audits of the datasets and models used by your agents are part of good responsible governance practice.

Any use of an autonomous AI agent in your company is as much an ethical obligation as a legal one. That way, your organisation stays in control of its automated processes, and accountable for their impacts.

Visibility and regaining control over AI usage

Agents deployed by business teams without going through the IT department exist in most organisations. This is not a discipline problem: it is a sign that the business need was real and the official channel was too slow. The problem starts when nobody knows how many there are.

Did you know? In the estates we analyse, 20 to 40% of paid AI licences go unused. The usages that matter most are rarely the ones leadership had anticipated. This is an engagement finding, not a market statistic.

AI governance turns this risk into an asset. Rather than banning business-led AI initiatives, it puts a framework around them: a clear usage policy, a lightweight approval process and automatic control tools make it possible to secure innovation without blocking it.

What concrete obligations does the AI Act impose on high-risk systems?

Regulation (EU) 2024/1689 imposes five obligations on the high-risk systems in Annex III, whose compliance date is now set at 2 December 2027:

1. A documented risk management system (Art. 9): continuous risk assessment throughout the life cycle
2. Exhaustive technical documentation (Art. 11): specifications, architecture, capabilities and limits of the system
3. Effective human oversight (Art. 14): human oversight mechanisms on high-impact actions
4. CE marking and registration in the EUDAMED-AI European database before deployment
5. Serious incident reports sent to the authorities within 15 working days

The postponed deadline exempts no one from anything: it moves the date, not the content. For organisations deploying agents in HR, financial or medical processes, these five obligations should be treated as a documentation workstream, not as a box to tick the day before.

Loss of human oversight and a traceability gap

When AI agents make decisions without human oversight and without auditable logs, the company loses control of its own processes. And when something fails, it cannot trace back the sequence of actions.

The traceability of an agent's actions relies on three mechanisms: auditable logs (recording each tool call and each decision), prompt versioning (the history of instructions given to the agent) and OpenTelemetry traces exported to a centralised observability system.

AI process monitoring is not only a compliance matter: it is what lets you detect behavioural drift before it has a business impact.

Financial drift: AI costs with no attribution and no steering

Without monitoring, LLM API calls pile up in an opaque IT budget. A single HR agent calling GPT-4 Turbo 100,000 times a month generates a cost of several thousand euros, invisible until a financial attribution system tracks it.

Managing AI agents well depends on the ability to attribute each API call to a specific agent, a specific team and a specific use case. This has to be designed in from the start: this quality of steering is the distinguishing mark of mature governance. This granularity lets the CFO make trade-offs, the IT director size the infrastructure and teams measure the real ROI of each deployed agent.

Financial performance indicators for AI governance naturally complement security and compliance metrics to give a 360° view of your agents in production.

Data sovereignty and GDPR compliance

AI agents process personal data at scale: emails, contracts, HR files, customer data. Each processing operation engages your liability under the GDPR (Art. 83: fines of up to 4% of worldwide turnover for non-compliant processing).

Privacy protection for your users and customers rests on three requirements: minimising the data collected by agents, limiting retention periods and securing the flows between the agent and third-party systems.

Sovereignty over sensitive data comes down to the choice of infrastructure: on-premise deployment or a certified European cloud for agents that process high-risk data. The processing rules and policies must be documented in your DPO's record of processing activities.

Putting AI governance in place in 5 steps

Step 1: Map and classify all your AI systems

The AI inventory is the non-negotiable starting point. A company that does not know its artificial intelligence systems cannot govern them.

There are two categories to record: the declared inventory, meaning the tools approved by the IT department, and unrecorded usage, meaning the agents put into service by business teams without going through that channel. The second category is detected by analysing outbound calls and through team interviews, never by either method alone.

For each system identified, apply the AI Act risk classification:

• Unacceptable risk: prohibited systems (social scoring, behavioural manipulation)
• High risk: Annex III systems (HR, credit, medical, critical infrastructure), with the heaviest regulatory obligations
• Limited risk: chatbots, assistants with a transparency obligation
• Minimal risk: productivity tools, recommendations with no operational impact

This risk classification determines the level of governance and the regulatory obligations that apply to each agent. According to BPIFrance (2025), more than 35% of French SMEs and mid-sized companies use AI without a structured formal framework.

Step 2: Define governance roles and bodies

Governance without clearly named owners stays theoretical. The second step in putting in place an operational framework is to name the people involved and define their mandates.

Appoint an AI Officer, or give the role to the CISO or the IT director depending on your size. Create a cross-functional AI committee bringing together IT, Security, Legal, AI Ethics and business representatives. This collective intelligence ensures that every governance decision takes business, technical and ethical stakes into account. The committee meets monthly to approve new deployments and handle incidents. Its collective intelligence allows fast, informed trade-offs.

RBAC (Role-Based Access Control) structures access through 4 governance roles:

• Owner: responsible for the agent's life cycle, holds the kill switch
• Admin: configures policies and guardrails
• Contributor: deploys and updates agents within their scope
• Reader: views metrics and logs without changing configurations

This formalised governance body is the backbone of your framework. It ensures responsible governance over time.

Step 3: Formalise the AI policy and usage rules

The AI policy is the founding document of your governance. It must be approved by the executive committee, communicated to all teams and reviewed at least once a year.

Four mandatory elements make up a robust AI charter:

1. Permitted and prohibited use cases: the domains where AI agents may operate and those that are excluded (autonomous medical decisions, discrimination in recruitment)
2. Data that agents may process: classification of accessible data by sensitivity level
3. Human oversight requirements: for each type of agent, the level of human validation required before action
4. Incident reporting procedure: channel, timeframe, owner and traceability for each incident

These rules and policies turn ethical best practices into repeatable, verifiable procedures.

Step 4: Deploy control and observability tools

Tools make your AI governance operational in real time. Without technical infrastructure, the policy remains a document with no effect.

Three categories of tools structure your augmented human oversight:

1. Agent registry: a centralised catalogue of all your agents in production (version, owner, risk level, policies applied). It is your single system of record for AI Act compliance.
2. Policy engine: a real-time rule enforcement engine. It intercepts each agent action before execution to check that it complies with the defined policies. Latency: under 10ms, so performance is not affected. Enforcing it ensures that every agent operates within the defined limits.
3. Observability: traces, logs and metrics exported via OpenTelemetry to your monitoring tools (Langfuse, LangSmith, Arize Phoenix). AI process monitoring becomes continuous and automated.

Step 5: Audit, measure and improve continuously

AI governance is not a project with an end date: it is a collective effort and a living organisational process. A quarterly governance audit spots drift before it turns into anomalies.

These four indicators are read from the execution logs, at regular intervals:

The continuous improvement loop runs in three stages: anomalies recorded → root cause analysis → update of the policy and guardrails. This cycle makes your effective governance a lasting ethical and competitive advantage.

What does governance that has taken hold look like?

Four milestones show that a programme has moved beyond the stage of intent:

• The agents in service are inventoried and classified against the regulatory framework. An agent that pre-screens job applications counts as high risk within the meaning of Annex III; an internal writing assistant does not
• The usage policy is approved at executive level and known to teams, not just published
• The four execution indicators are read from the logs, not self-reported
• Anomalies are documented with their cause and the policy evolves accordingly

These milestones are not just compliance indicators. They reflect stakeholder trust (customers, partners, investors and regulators) in your ability to deploy AI responsibly. AI governance then becomes a commercial argument in its own right.

Common mistakes to avoid when setting it up

Four mistakes sabotage even the most ambitious AI governance programmes:

• Governing only in-house agents, and forgetting that Copilot, ServiceNow and Agentforce are also third-party agents that need the same oversight
• Treating security as governance, when security ≠ governance: security protects the perimeter; governance steers what happens inside it
• Setting up an AI committee without an executive mandate: without budget and decision-making authority, the committee is advisory but powerless
• Neglecting incident documentation: every undocumented event is a lost opportunity for improvement and a regulatory risk in the event of an inspection

The 5 pillars of a robust AI governance framework

Pillar 1, ethics: transparency, explainability and bias control

Ethics is the first pillar because it defines the values that guide the other four. Without an explicit ethical foundation, governance decisions become arbitrary.

Three dimensions structure this pillar:

• Transparency: the decisions made by your AI agents are documented and understandable by the stakeholders concerned. No decision-making black box without documentation.
• Explainability of AI decisions: every output with significant impact can be justified by a traceable causal chain. Critical for regulated sectors (banking, insurance, HR).
• Bias control: regular audits of the training datasets and of the outputs your agents produce. Proactive detection of gender, origin or age-group bias in automated decisions.

Responsible governance means your organisation publicly stands by these three commitments, not just internally.

Pillar 2, regulation: AI Act, GDPR and European compliance

The European regulatory framework imposes three cumulative sets of obligations on agentic AI systems:

Regulatory compliance is not an annual box-ticking exercise. It is a continuous state, maintained by your governance processes, your control tools and your documentation.

Pillar 3, organisation: AI committee, RBAC and decision bodies

A solid organisational framework turns policy into practice. Three concrete structures make up this pillar:

The AI committee brings the key stakeholders together every month (CISO, IT director, business representatives, legal counsel). It approves high-risk deployments, handles significant events and evolves the AI policy.

RBAC (Role-Based Access Control) applies the principle of least privilege to every agent and every user. The four roles, owner, admin, contributor and reader, ensure that nobody holds more rights than their role requires.

The governance body is the central decision-making organ. It holds an explicit executive mandate: a dedicated budget, the power to block non-compliant deployments and a direct reporting line to the executive committee.

Pillar 4, technology: observability, emergency stop and runtime guardrails

The technical pillar makes governance operational in real time. Three components are non-negotiable:

1. Kill switch: a mechanism to stop an agent in production immediately, which the owner or the administrator can trigger in under a minute. Essential to contain an anomaly before it spreads.
2. Runtime guardrails: they intercept each agent action before it goes out: personal data detection, defence against instruction injection, topic restrictions, allow lists and deny lists. The real constraint is latency: a guardrail that slows the business down ends up switched off.
3. Observability: full instrumentation via OpenTelemetry with export to Langfuse, Arize Phoenix or LangSmith. Every trace, every decision and every tool call is recorded and queryable.

Effective control of autonomous agents is only possible with these three components deployed in production, not in a test environment.

Pillar 5, finance: cost attribution per agent and measurable ROI

Financial attribution is the least visible pillar but one of the most strategic. Without it, your organisation invests in AI without knowing what each agent really costs.

Financial attribution makes it possible to calculate, for each deployed agent, the cost of LLM API calls, infrastructure costs and maintenance time, broken down by team and by use case. Every company can then decide its AI investments on real data, without approximation.

A simple AI ROI formula: number of people handling the task manually × their unit cost / total cost of the agent. If the ratio is > 1, the deployment pays off. This strategic approach to the financial pillar makes every production release a data-driven choice. Clear operational intelligence on the real costs of each agent is essential for making sound trade-offs.

Reference frameworks and standards (ISO/IEC 42001, NIST AI RMF, OECD)

Three international governance frameworks command consensus among leadership teams:

• ISO/IEC 42001:2023: the first international standard dedicated to artificial intelligence management systems. It defines the requirements for establishing, implementing, maintaining and improving an AI management system within a company, whatever its size. This governance technology applies to any organisation deploying AI agents.
• NIST AI Risk Management Framework: a US framework adopted globally to assess and manage AI risks. It is built around four functions: govern, map, measure and manage.
• OECD Principles: seven principles for responsible governance, namely inclusive growth, safety, transparency, robustness, accountability, respect for privacy and international cooperation.

Several vendors also publish template kits under open licences to structure an internal governance framework. They save time on the documentary form, not on the inventory of the estate, which remains the work to be done.

The tool ecosystem for AI governance

The AI governance tool ecosystem is organised in three complementary layers:

• MLOps and model registries: MLflow to version the AI models your agents use, DVC for dataset traceability, Weights & Biases for training monitoring
• LLM observability: Langfuse and LangSmith to trace every LLM call, OpenTelemetry for standard instrumentation, Arize Phoenix for drift detection
• Governance, risk and compliance adapted to AI: explainability and bias detection tools, tracking of autonomous agents, a registry of the policies applied

The choice depends on the agents already in service. And that is precisely the starting point: the list of tools is useless as long as the estate is unknown.

Why does the inventory of the estate come before tooling?

A governance framework is built on an inventory. As long as the list of agents in service, assistants created and subscriptions taken out remains incomplete, every written rule applies only to the visible part of the estate. It is the most expensive blind spot: you govern what you have recorded seriously and the rest not at all.

Where to start: measure the estate before governing it

What vendor dashboards do not tell you

Each vendor provides a dashboard for its own scope. That is useful and it is not enough: what escapes, by design, is what teams do elsewhere and the comparison of the same usage from one tool to the next. Yet it is precisely this view, across all tools, that lets you decide what to put a framework around first.

What we measure and how

We analyse real AI usage across the estate, from the logs and exports your platforms already produce, on the tools you administer, without installing anything on workstations. Exports handed over or a read-only connection: the exact setup is decided at the scoping meeting.

Interviews and data, never one in place of the other. Logs give the facts: who uses what, how often, with what error rate. Interviews give the intent and the business constraint. Results are aggregated by team and by tool, never by person; the exact status of the processing is validated with your data protection officer. We analyse usage, never people.

No scanning of workstations. No source code analysis. The analysis covers usage, from the logs.

The three deliverables

The report comes with an appendix of what usually goes unsaid: the scope actually covered, what we could not see and the figures we refuse to put forward. Work that does not state its blind spots asks to be taken on trust.

What we put in place after the measurement

Measuring governs nothing on its own. Three tooling levers shift usage, from the least costly to the most committing: integrations that bring the agent to where the work already happens, shared assets that turn an individual find into a team capability and common agents that take on recurring tasks. Tooling, not discipline: you do not fix a usage by asking teams to do better.

The validation steps this work requires (informing employee representatives, consulting the data protection officer, read-only access and nothing installed on workstations) are not a brake. They are what make the result defensible internally.

Targeted and measured at your organisation: 30 to 50% lower cost on the targeted usages, errors divided by 2 to 4. These are engagement objectives, not results guaranteed in advance.

Idun Group has been working on these topics since late 2023. At LVMH, for all the perfume and cosmetics brands, an Idun engagement on their generative AI platform in production. At SNCF Immobilier, the audit run in its social-housing subsidiary uncovered 50 critical use cases.