IVA IDEAS & RESEARCH · THE FIVE-STAGE GOVERNANCE CYCLE · PART 1 OF 5

Design: Governance Is Decided Where Work Is Organized

Why the most consequential AI governance decisions concern who does what, what gets automated and whose knowledge sets the boundary — and a worksheet for making them.

Iva Bruni · 2026 · Part 1 of 5 in the series The Five-Stage Governance Cycle

01 Design · 02 Enable · 03 Demonstrate · 04 Learn · 05 Redesign

AI governance becomes operational through decisions about how work is organized, what is automated, where human authority remains, and whose knowledge shapes those decisions.

Policy sets the limits; design decides what happens

Law and policy constrain what can be built, but rarely say which tasks a system should take on, where its authority stops, or who steps in at the edge. Those choices are made when obligations are turned into operating boundaries and decision rights.

Established frameworks place this work early. The NIST AI Risk Management Framework’s Map function establishes the context in which an AI system’s risks are framed, and its GOVERN 3.2 subcategory calls for policies and procedures that define and differentiate roles and responsibilities for human–AI configurations and oversight (NIST, n.d.). But what does that mean when a manager is deciding which tasks to automate, who remains in control and when someone should intervene?

Four design decisions

1. Should this task be automated at all?

The first decision has four honest answers: automate, assist (the system supports a person who decides), defer (the evidence or capability is not there yet), or do not automate.

Canada amended its Directive on Automated Decision-Making, effective 25 April 2023, so that the Algorithmic Impact Assessment — which must be published before a system launches — now includes questions about the reasons for automation (Government of Canada, 2024). Workers’ own view of the question matters too. A Stanford study of about 1,500 workers, covering 844 tasks in 104 occupations, found mismatches between the tasks workers want automated and what AI can currently do (Shao et al., 2025).

In practice: Robodebt. Australia’s social security agency used tax-office income data, “evenly apportioning that income over fortnightly increments,” to compare against what recipients had reported, and an automated system sent letters when the figures differed (Prygodicz, 2021, [2], [39]). Recipients who supplied no further information were treated as if they had earned the tax-office amounts in equal fortnightly parts ([40]). The Commonwealth later admitted that it “did not have a proper legal basis to raise, demand or recover asserted debts” — at least $1.763 billion asserted against about 433,000 people ([4]). The judge called it “a shameful chapter in the administration of the Commonwealth social security system” ([5]). The Royal Commission that later examined the scheme’s wider institutional failures recommended designing policies and processes with an emphasis on the people they are meant to serve; documenting the criteria for excluding groups from compliance activity; and, for each compliance program, documenting the categories of vulnerable recipients it may affect and how their needs will be accommodated, in consultation with stakeholders including advocacy bodies (Royal Commission into the Robodebt Scheme, 2023, Recommendations 10.1, 11.1 and 11.4). The question the scheme skipped was not how to automate the calculation, but whether that calculation could lawfully be automated at all.

2. Where does the automation boundary sit?

Once a task is in scope, the next decision is where the system’s authority ends.

In practice: a tiered IT agent. Singapore’s agentic AI framework includes a deployment case that tiers an IT agent’s actions by severity, reversibility and the feasibility of human oversight. Low-risk, reversible tasks such as password resets are fully automated and audited biweekly. Moderate-risk fixes need human approval, and high-risk permission changes are off-limits to the agent (IMDA, 2026). It is a documented design illustration rather than an independently evaluated result, but it shows the boundary decision made deliberately.

The important question isn’t what the technology can do. It’s what could go wrong, whether the action can be undone and whether a person has a realistic chance to intervene. Those are the considerations that should determine where automation stops. Reversibility also needs precision: some actions can be undone directly, while others can only be offset by a compensating correction.

3. Who holds decision rights on each side of the boundary?

A boundary is only as clear as the authority attached to it. For each side, the design should say who decides, who is consulted, who can stop or contain the system, and who answers for the outcome. It should also say, separately, who may change the system’s rules and who may change its limits. Singapore’s framework states that humans remain ultimately accountable (IMDA, 2026); the design has to name which humans.

In practice: Air Canada’s chatbot. A customer asked the airline’s website chatbot about bereavement fares and was told he could apply “within 90 days of the date your ticket was issued” (Moffatt, 2024, ¶15). Another page of the airline’s website gave different information. Before the tribunal, Air Canada argued it could not be held liable for information provided by its agents or representatives, including a chatbot. The tribunal read this as suggesting “the chatbot is a separate legal entity that is responsible for its own actions,” and rejected it: “It should be obvious to Air Canada that it is responsible for all the information on its website” (¶27). It found that the airline “did not take reasonable care to ensure its chatbot was accurate” (¶28), and that there was “no reason why Mr. Moffatt should know that one section of Air Canada’s webpage is accurate, and another is not” (¶29). The system had been allowed to speak for the company on refunds, but no one had decided who answered for what it said.

4. How do suppliers and third parties fit in?

Many AI capabilities arrive through vendors. The UK Government’s AI Playbook asks teams to work with commercial colleagues from the start and to hold bought systems to the same responsible-AI expectations as systems built in-house (UK Government, 2025). A design that ignores the supplier leaves a boundary undrawn: what happens when the vendor changes the model underneath, and how that change is held back until someone has reviewed it.

In practice: a vendor sepsis model. A proprietary sepsis prediction model was “implemented at hundreds of US hospitals.” When researchers at one health system validated it independently across 38,455 hospitalizations, it achieved an area under the curve of 0.63, “substantially worse than the performance reported by its developer.” It missed 67% of patients with sepsis while alerting on 18% of all hospitalized patients (Wong et al., 2021). The study’s authors concluded that its “widespread adoption despite poor performance raises fundamental concerns.” What was done well is the validation itself: one organization tested the vendor’s claim in its own setting. What the case shows more broadly is how far a bought model can travel on its developer’s word.

Illustrative Scenario: the hotel front office

This scenario is hypothetical. It is constructed to show the mechanism, not drawn from a specific organization, and it continues through all five parts. Amounts, times and volumes are illustrative.

A full-service hotel plans to deploy a vendor-supplied AI agent to handle guest compensation. Management proposes that the agent approve everything below a cost threshold and send the rest to a manager. Front-desk staff point out what the cost data cannot show: some larger compensation is time-sensitive service recovery, decided while the guest is still at the desk, and an approval step there can undermine it. The design is revised into three functions:

	Function	Who decides	Governance relationship	Accountable overseer
F1	Routine billing corrections of errors the agent can verify — duplicate charges, rate-code mismatches, misapplied tax — within per-item and per-folio caps	The agent, within rules and caps; every action logged	Human oversight of AI, after the fact	Front Office Manager (FOM): samples corrections weekly, can reverse an eligible correction or initiate a compensating adjustment where necessary, change rules within the caps and pause the function
F2	Service-recovery suggestions: a proposed type and amount, with reasons, when a front-desk agent opens a case	The front-desk agent facing the guest, within a written authority limit	Human oversight of AI, before the outcome	The front-desk agent, who accepts, modifies or rejects each suggestion
F3	Logging every compensation decision, by the agent or by staff, and flagging unusual patterns	No one — F3 only flags	AI-assisted review of people’s decisions: a form of employee monitoring	FOM, who reviews flags

Changing the caps needs the General Manager (GM), and the duty manager can stop the agent on any shift. Because F3 helps managers review what staff do, the hotel treats it as employee monitoring: its purpose, the data it records, who can see it and how long records are kept are set in writing with staff, and no automated decision about an employee rests on a flag. The supplier contract requires 14 days’ written notice of any change to the model, thresholds or features, with a right to defer activation and a staging environment for retesting. The expected gains are less manager time on approvals, faster service recovery and fewer billing errors; the costs include about three hours a week of FOM review.

The technology did not change. The allocation of authority did — because of what the people doing the work knew.

Participation with decision rights intact

Involving people does not mean handing them every decision. This series keeps three roles apart: who contributes evidence, who is consulted, and who decides. In the hotel, staff challenged an assumption; management still authorized the design.

In practice: screenwriters. The 2023 agreement between the Writers Guild of America and the studios draws an automation boundary through collective bargaining. Under it, “AI can’t write or rewrite literary material,” AI-generated material “will not be considered source material,” the company “must disclose to the writer if any materials given to the writer have been generated by AI,” and the company “can’t require the writer to use AI software” (Writers Guild of America, 2023). The people who do the work set the limits on what the system may do to it — a Design decision made with participation at its center. The agreement establishes negotiated contractual boundaries; it does not by itself show how well each safeguard works in practice.

The form participation takes depends on context. An ILO review of social dialogue on AI and algorithmic management across five world regions found that collective worker voice supports a “high road” approach, in which technology complements workers’ skills rather than replacing them (Doellgast et al., 2025). Applying that lesson to workplaces without unions is an analogy, not the study’s finding. In some jurisdictions participation is a legal obligation. Germany’s Works Constitution Act requires employers to inform and consult works councils about planned working procedures, including the use of artificial intelligence (§90). German law goes further where a system can monitor employees’ behavior or performance, as the hotel’s logging and flagging function can. Works councils then have a co-determination right over the introduction and use of such devices (§87(1) no. 6) (Federal Republic of Germany, 2001). Whether such rules apply, and how, depends on the jurisdiction and the workplace.

Where there is no formal representation, as in much of US hospitality, participation has to be built. That can mean input sessions, frontline review of proposed boundaries, and a way to flag a decision the system got wrong. During change processes, the UK Government’s AI Playbook asks that staff be given sufficient time, information and tools to identify and adapt to emerging risks (UK Government, 2025).

Two rules keep participation honest:

• It should be proportionate. The test is whether relevant perspectives can influence the decision, not how many meetings took place.
• It should leave evidence of influence. Record what people contributed, which assumptions they challenged, and whether the decision changed — or why it did not.

What the cases show

None of these cases tells the whole governance story. Each brings a different part of it into focus. The Singapore agent and the screenwriters’ agreement illustrate Design done deliberately. The sepsis validation is a Demonstrate step, taken by one health system after the model was already in wide use. In Robodebt and Air Canada, external legal proceedings exposed or addressed failures in automated or AI-assisted decisions. Their public records illustrate the consequences of leaving important questions of legality and organizational responsibility unresolved. Taken together, these cases illustrate why isolated governance measures are insufficient. The Five-Stage Governance Cycle proposes a way to connect design decisions, usable oversight, testing, operational learning and redesign. This is our reading of a handful of public records, not a survey.

The responses are not new. Organizations already have ways to define authority, test controls, involve workers and respond to problems. The challenge is making those practices work together:

Gap seen in the cases	Where the cycle addresses it	Established practice it draws on
A task automated without asking whether it should be (Robodebt)	Design: the eligibility decision, with reasons recorded	Reasons for automation in Canada’s impact assessment (Government of Canada, 2024)
A system speaking for the organization with no one answerable for it (Air Canada)	Design: boundary and decision rights; Enable: overseers who can act	Defined human–AI roles (NIST, n.d., GOVERN 3.2)
A bought model trusted on its developer’s claims (sepsis model)	Demonstrate: test the configuration in its own setting before relying on it	Design versus operating effectiveness (PCAOB, n.d.)
Poor performance found only by outside study (sepsis model)	Learn: compare operating evidence with what was expected	Plan-Do-Study-Act (The W. Edwards Deming Institute, n.d.)
Failures exposed by external legal proceedings (Robodebt, Air Canada)	Redesign: a governed decision triggered by the organization’s own evidence	Management of change (OSHA, 2009); continual improvement (ISO, 2023)
Workers’ knowledge left out of the boundary — or, done well, built into it (screenwriters)	Design and Learn: participation with evidence of influence	Social dialogue on AI (Doellgast et al., 2025)

The rest of the series takes these stages in turn. Each part pairs one stage with a one-page tool, so that the remedies can be used together rather than one at a time.

Scale the design work to the stakes

One rule runs through the series: the depth of governance work should scale with the consequences of getting it wrong. Public frameworks work the same way: Canada uses four impact levels with proportionate mitigations, and Singapore uses tiers of autonomy (Government of Canada, n.d.; IMDA, 2026). A low-impact internal tool may need a paragraph per design decision; a system that decides about customers, money or employees needs the full worksheet, real participation and a named owner for every boundary. Part 5 sets out a lighter path for low-consequence uses.

Use it in your next automation decision

A record that makes design decisions open to question — not a scoring model or a validated assessment.

What comes next

A design governs nothing until people can operate it. Part 2, Enable, asks whether the front-desk agent and the Front Office Manager can actually exercise the authority this design gives them — and keep it as the work changes.