AI polarisation is getting more intense. Discussions often descend into binary debates of pro-AI and anti-AI. There’s a more nuanced path: Deciding the right role for AI in your processes and knowing when not to use AI at all.
For a while AI trainees have been asking me when and when not to rely on AI. I had been putting off writing up my best practices, but then I recently read Constantine Papas’s write-up of research done by UXtweak Research and the Slovak University of Technology in Bratislava. Presented in the paper Distorted Perspectives of LLM-Simulated Preferences: Can AI Mislead Design?, in Constantine’s words,
“[The research] isolates the single most common way a design team actually reaches for synthetic feedback. Not political polling. Not consumer surveys. The A/B. The ‘which of these two screens do people prefer, and why.’”
Karen McGrane, a pioneering content strategist, UX leader, and author of the books Content Strategy for Mobile and Going Responsive, posted Constantine’s article on LinkedIn, saying, “Finally, I have an article I can share to once and for all shut down all debate about synthetic users.”
Screenshot of Karen McGrane’s LinkedIn post about Constantine Papas’s article, Synthetic Users Are Influencing Your Design Decisions. New Research Says They’re Right About as Often as a Coin Flip.
But I think it only makes the debate more interesting.
It raises a recurring issue that’s common not just in design and user experience work, but in all kinds of strategy, knowledge, and content work. The key issue I see is: People are asking AIs to do their jobs for them. They’re inviting AI to replace them.
The people who should worry about AI taking their jobs are the ones who ask it to do their jobs.
I was talking to a friend – a lawyer, neither an AI champion nor detractor – who looked at me squarely and said, “Isn’t asking an AI to take a decision between designs the absolute last thing a designer should be asking AI to do?”
Exactly.
After years of every year of work getting more and more intertwined with AI, my current position is: Don’t ask it to do late-stage critical decision-making. Ever. In short, don’t let it do your job.
If you’re a designer, don’t ask it to do the design.
If you’re a strategist, don’t ask it strategise.
If you’re a writer, don’t ask it to write.
There’s a difference between getting help, offloading tasks, and it doing the core of your job.
AIs are not humans. They’re tools.
In my last article on AI and personas in strategy, research, and design work, I made the distinction between
- working with your data using AI, vs
- working with AI instead of with your data.
The dialogue in the comments was lively! But it kept coming back to the same question: “Why would we use AI instead of people?”
That question assumes that AI is the product, the result – or worse, a stand-in for humans at every stage of a work process. It assumes that the moment you use AI, the intent is to replace human contact.
We can augment, accelerate, facilitate, enhance, critique, and more, all without ever coming into the universe of replacement.
AIs are not humans. They’re tools. We need to shift our mental model for AI’s role in a business workflow. We need to learn when to use AI tools and when not to.
Moving from AI fears to a place of AI safety
Full disclosure: I work with AI and I train people on AI. For decades I’ve been doing mainly strategy, design, knowledge, and content work, so I built a whole persona and journey mapping methodology, which when ChatGPT 4 came out, I accelerated with AI.
I could see that our process was good, but the labour involved was prohibitive, and practical realities of humans meant opportunity was being left on the table. AI could be a threat and a gift at the same time.
Based on data they collected in their 2026 survey of tech workers, Noam Segal and Lenny Rachitsky at Lenny’s Newsletter found that 22% of respondents worry about losing their job to AI. But it showed that respondents’ main “underlying fear is of being overworked.” They worry about being expected to do more for the same pay (51%), getting trapped in an unsustainable pace (46%), and seeing the quality of their work going down (41%).
The fear is real. That is the expectation.
For example, if I employ a bunch of people to dig holes, and we switch from shovels to industrial diggers, I’m expecting way more holes for not much more pay. But I need to acknowledge that industrial diggers are both more powerful and more dangerous than shovels. So how do we balance speed, quality, and safety?
In my work, AI helped me greatly increase the speed and quality of my persona and journey mapping methodology (because AI makes the results traceable in ways that would be totally unfeasible with traditional methods). But I also had to keep my work with AI safe and keep humans in the driver’s seat. It’s only with both that you can move on from these fears.
The AI modes
I’ve identified a spectrum of AI modes that notably correspond with workflow stages and how much AI contributes to each stage.
From most to least safe, they are:
- Exploration mode: AI suggests things you’ll verify.
- Aggregation mode: AI brings together and presents back your facts.
- Expert mode: AI supplies its own knowledge.
- Replacement mode: AI makes your decisions for you.
AI helps with brainstorming, divergent thinking, and option exploration.
A wrong item on the list of all possibilities has little risk (happens all the time with real humans).
This works, with guardrails.
AI retrieves, arranges, and renders them.
This works especially well when every claim is traced back to a source.
(Note: You can watch my video on LinkedIn to learn more about tracing claims to sources and to see some examples.)
AI is retrieving encoded knowledge, not guessing about real people.
Domain expertise has right and wrong answers, and those answers are richly represented in training data.
This works, but verify critical items you rely on.
To make this relevant to the nuances of your own users/customers, you may feed AI the research you have done with those particular users or the specifics of your business context. In those cases you may have to judge if augmenting your nuanced user understanding with general knowledge refines or dilutes your research.
Here the model does something worse than just making a mistake: it manufactures false balance to play it safe, giving false confidence or creating more work when a human could have just made a call.
High risk. Avoid or treat with extreme caution.
Consider a knowledge graph to control high-value factual queries that link to complex and nuanced underlying data.
The use cases in each mode
The use cases in each mode are ordered from safest to most dangerous. For each use case, I give a verdict on whether AI is
- ✅ useful
- ⚠️ useful with guardrails
- ❌ high risk
There are guardrails for the use cases whose verdicts are either ✅ or ⚠️.
- Verify load-bearing claims. The output of AI playing an Expert mode is often right, but “often” isn’t a foundation for key decisions. Anything that influences a decision should be checked against a primary source or a human expert.
- Label the epistemic status. Output from Exploration mode or Expert mode is hypothetical or review input, never evidence. Don’t let generated edge cases leak into research repositories unlabelled; provenance is what keeps the line visible.
- Watch for mode drift. Most misuse starts in a legitimate mode and drifts: a terminology check becomes “Would they like it?”; objection generation becomes “So which objection matters most?” The moment the question becomes what real people prefer, feel, or would do, you’ve crossed into Replacement mode.
- Keep the friction. If AI output never disagrees with you, never dislikes anything, and finds every option “clear and intuitive,” you’re reading RLHF politeness, not analysis. Prompt explicitly for refutation and worst-case readings.
- Don’t individuate persona simulations. They’re not individual humans. Never ask one for specific personal preferences (“What’s your favourite movie?”, “Which design looks best?”). The Distorted Perspectives study’s single-persona condition was the worst result of all: 91% divergence, near-deterministic answers, collapsed exploration. Aggregate framing (“People in this segment tend to…”) at least triggers the model’s crowd-diversity priors; individual framing produces a confident fictional person.
Exploration mode: AI is a possibility generator
Verdict
✅
Wrong guesses cost nothing; real research downstream is the filter. This is what “synthetic sufficient” legitimately means.
✅
You want the coverage checklist, not evidence. Every item still gets verified by a human or a real test.
✅
Generating possible objections is exploration. Predicting which objection will actually kill the deal is a decision. Don’t mix these up.
✅
The human is the thing being tested, so the participant being fake is fine. The simulation only needs to be plausible, not accurate.
Experts correct a wrong artifact far more readily than they fill a blank page, so provocation works. But validation is everything: people are biased to go along with whatever they’re presented, so ask reviewers to falsify the draft rather than approve it, and never let unreviewed synthetic scaffolding pass downstream as findings.
Good for mechanics and comprehension traps. Never use the model’s answers to pilot expected distributions.
Useful only if the persona is a thin layer over real research with traceable claims. A persona improvised from model priors teaches the team a fiction with confidence.
Can be useful if considered to be a variant of brainstorming. Helpful as a thought-starter for humans to chew on.
Why is “What’s wrong with this?” ok when “Which is better?” isn’t?
The answer is falsifiability. Each flagged defect can be checked in seconds by a human looking at the artifact, so the loop can be completed between humans and AI.
Four conditions to keep in mind:
- Frame it as an expert review, not a persona simulation. “Review this against usability heuristics/accessibility standards/your database expertise” is Exploration mode. “As a busy nurse, do you see anything wrong?” reintroduces Replacement mode.
- Treat “no issues found” as the least trustworthy output, not a pass. The model’s “reinforcement learning from human feedback” (RLHF) politeness and lack of actual human taste means it may be letting things through that a human wouldn’t.
- Ignore the model’s severity and priority rankings. “Which of these problems matters most?” is preference-shaped judgment sneaking back in – humans rank, the model lists.
- Don’t let the list full constrain the review. We’re all biased to go along with what we’re presented, so the model’s findings anchor attention. Verify each finding and still look for gaps in AI’s responses.
Aggregation mode: AI is an aggregation and access layer
In this mode, there’s no simulation. The facts are human and the AI aggregates, arranges, and retrieves. This is the strongest-evidence mode. In it, AI applies research best practices at all points, using AI as a search and aggregation tool to surface the data you’ve collected.
Verdict
✅
Why
The facts are real. The AI does the aggregation and arrangement that no human can hold in their head.
Journey maps are narratives, and narrative-completion is what large language models (LLMs) do well. They can infer transitions between evidenced stages, emotional states, and smooth the sequence for coherence, while retaining the source data.
Guardrails: every stage and claim should trace to a source; unsourced items should be labelled as inference or assumption; gaps in the journey are findings (which could be the topic of your next study), not blanks to fill.
✅
Why
The personas are assembled by AI, not generated by it. Every attribute can cite a transcript, ticket, or study; anything without a citation is visibly labelled as an assumption.
✅
Why
AI provides the purest thin layer. It’s surfacing data you’ve collected, with citations.
The failure state is hallucination, where the LLM is answering from its training data or just making something up. There are several methods to improve anchoring in the data, such as forcing models to offer citations you can look up in the research (with anchor quotes) or redirecting it with a statement like “That data isn’t in the research” to prevent it from making assumptions.
For industrial strength, knowledge graphs + AI are better than AIs alone.
✅
Why
This is a useful form of gap-plugging, and the legitimate twin of the “prioritisation” row in the Replacement mode table. A research-grounded persona can tell you “participants asked for a voice option and it’s not in your spec”. That’s retrieval of what the research said, with a citation. What it can’t do is rank your list or pick a favourite. That’s opinion. Same persona, same conversation, opposite epistemic status: “What did the research say that I’ve missed?” works; “What do you want most?” doesn’t.
✅
Style transfer is something models are genuinely good at. If you give it real quotes, it can adopt the register, vocabulary, and frustrations of the people behind them. The success criteria is humans feeling the research come to life, which isn’t a hard pass/fail job.
AI can make research felt, not just read, which is what personas were always for in the first place. Note that the model can perform tone from verbatims; it cannot judge tone in place of a real human (see the ❌ “judging of copy tone” row in Replacement mode).
Expert mode: AI as an expert reviewer (using model as a domain expert)
Verdict
Encoded expert knowledge has checkable right answers. The model has effectively “read the manuals.” Verify load-bearing claims, but this is a core AI use case.
Adds a knowledge filter you personally lack. Two different prompting frames (generator vs critic) also catch different errors.
This converts one bland recommendation into a visible set of trade-offs, and every argument is a falsifiable claim you can check.
Caveats:
Mine the transcript for arguments, never the conclusion. The same engine that won’t take a strong stand in preferences or decisions can stage a suspiciously balanced debate, often ending without a real winner. You judge the duel, not the AI.
Rule-based domains are documented and stable. Note that this is an expert audit, not a user simulation. “Check this against WCAG 2.2” works; “Pretend to be a blind user” doesn’t (see Replacement mode).
A grade-8 reading level is a known equation. Grammar and spelling rules are not taste-based. Tone is different from grammatical correctness.
Useful cheap first pass; expect false positives and treat findings as leads for human review, not verdicts. You’re simulating an expert method, not a user population.
Related to A/B but a different beast, and the difference is the unit of output. A preference verdict is one unfalsifiable aggregate judgment; a defect list is discrete, individually checkable claims. This is genuinely human-in-the-loop compatible: the human verifies each finding against the artifact.
Vocabulary is knowledge; it’s reliable. But “Will the audience like this term?” is preference. Don’t let the questions drift from looking up to decision-making.
Replacement mode: Synthetic users as a decision-making tool (producing “data” about real people)
Verdict
Why
The “Distorted Perspectives” study’s exact scenario (section 5.1): 53% agreement with real users’ top pick, distributions genuinely different in 44% of tasks, and the dominant failure is a manufactured tie that greenlights your existing bias.
Why
The justifications are where the lack of a mind shows most: praise for nondifferences, hallucinated tasks, faked memories, zero genuine friction (no dislike, no refusal, no gut aversion).
Why
Frequencies among real populations are contingent facts the model doesn’t have. Flattened entropy means it can’t even get the shape of a distribution right.
Why
These are high-stakes decisions with money attached – the most consequential version of the same failure.
Why
A preference test in roadmap clothing — ranking a list is opinion, and opinion is where the false-balance failure lives. The useful reframe is “What have I forgotten?” (see Aggregation mode): same conversation, opposite epistemic status.
Why
The study found that models miss tonal nuance between same-meaning variants entirely — the exact register where preference lives. Don’t confuse judging tone with adopting tone (see Aggregation mode): performing a voice from real verbatims is a strength; evaluating which tone lands is a decision.
Why
Simulating the lived experience requires a decision and is unearned. Return to the Expert mode and then test with real users.
Moving forward
The “AI is magic!” phase is starting to wane. Thank god.
Imagine my delight at hearing Sam Anthony, a senior data scientist at IBM, say, “AI is not magic and it’s not the best solution for everything.” Sam’s video (which I had no idea about until after I wrote most of this article, I promise!) Knowing When Not to Use AI: AI Agents vs Rules vs ML is a bit techie, but it ends on a message that should be a sigh of relief:
“Great systems are built by making disciplined choices about how decisions get made across humans, rules, models, and machines. The most successful AI systems likely won’t be the ones using the most AI. It will be the ones who consistently choose the right kind of intelligence for the problem.”
The video highlights that humans are always needed for high-stakes decisions, ownership, and dealing with the ambiguity of real life. GenAI is good at tasks like dealing with unstructured information at scale, transformation, and flexibility, where “some amount of error” is acceptable. Wise words.
What I’m putting forward is that AI can help you explore what you already know so you can then figure out what you could do next. What AI shouldn’t do is tell you what you should do next.
0 Comments