Technical checks, legal reviews and vendor assurances tell us what an AI system is supposed to do. They rarely tell us what it does to the people it decides about, or who has the authority to stop it.
From “Joan Is Awful” to Workday - who answers for the outcome?
“Joan Is Awful” is well-trodden ground in the AI debate. But there’s a question behind that Black Mirror episode I keep coming back to.
I run a business built around the people who buy, sell and govern AI, cybersecurity, privacy and risk-management solutions. The conversations we host on stage are more assured than the ones we have in the bar afterwards.
On stage, we talk about frameworks and safeguards. The question I want us to spend more time on is whether we’d be comfortable being on the receiving end of the systems we approve.
The risk isn’t just AI going rogue. It’s a system doing exactly what it was designed to do, with consequences the people deploying it haven’t properly considered.
Take the Workday case, covered by Reuters this week. Job applicants say its hiring tools unfairly screened out Black people, women, people over 40 and people with disabilities. Workday disputes that, saying its AI looks at job qualifications, not characteristics such as race, age or disability. The applicants are asking to bring those claims as a class action, but the court hasn’t decided that discrimination occurred.
That isn’t the same story as “Joan Is Awful”. But it raises a related leadership question: how do we check what a system actually does to people, rather than simply accepting what we’re told it does?
Technical checks matter. So do legal reviews and vendor assurances. But I’d want to know who is testing the outcomes, who can challenge a decision and who has the authority to stop the system when something looks wrong.
Likewise, getting someone to click “agree” shouldn’t end the conversation about whether they understand what will happen to their data.
For me, that’s where responsible AI becomes a leadership responsibility, not something we can simply hand to legal or IT.
So alongside “is it compliant?” and “has legal signed it off?”, I’d ask one more question:
If you were the applicant being rejected, the customer being profiled or the person whose life became the content, would you still approve it? And if not, what needs to change before anyone else is put in that position?
Join the debate at #RISK Expo Europe
#RISK Expo Europe
10–11 November 2026, ExCeL London
The questions in this article — who tests the outcomes, who can challenge a decision, who has the authority to stop a system, are on the agenda at #RISK Expo Europe.
Across two days and three stages, risk, compliance, security and audit leaders from organisations including Aviva, John Lewis Partnership, Euroclear, HSBC and NCC Group will work through how AI is governed in practice rather than in policy:
- AI in the Three Lines of Defence — who owns the risk when automation runs through risk, compliance and audit at once
- Ian Rae, John Lewis Partnership, on the use cases and limits of AI-enabled GRC
- Arcangelo Leone de Castris, Aviva, on turning ethical AI governance into the operating model
- Michael Rasmussen on responsible AI governance and where GRC goes next
On day two, the co-located PrivSec AI Governance theatre brings Max Schrems, Lord Chris Holmes, and Michael Charles Borrelli of AI & Partners on EU AI Act enforcement.
If the harder questions usually get asked in the bar afterwards, this is the room to ask them in.
No comments yet