Eight entirely fictional examples

When AI Gets It Wrong

AI errors can sound calm, detailed and authoritative. Practise spotting the problem before an answer reaches real work.

AI invented a NICE recommendation

AI output: ‘NICE recommends weekly cognitive screening for every person entering community drug treatment.’

Spot the problem

The wording sounds formal and plausible, but no source has been supplied and the recommendation may not exist.

Reveal what went wrong

What went wrong?

A convincing organisation name is not evidence. Generative AI can fabricate a recommendation or combine separate guidance.

Safer approach

Ask for the exact document, open NICE guidance directly, search the text and confirm scope, date and wording.

Key lesson

A reference must exist and support the claim attributed to it.

AI used outdated drug-law information

AI output: The draft gives a confident legal classification without stating a source date or jurisdiction.

Spot the problem

Drug law changes and an answer may mix UK and overseas information.

Reveal what went wrong

What went wrong?

The response failed to identify the relevant UK nation, date and primary legal or government source.

Safer approach

Check current GOV.UK and legislation sources immediately before publication and state the review date.

Key lesson

Time-sensitive legal information needs primary-source verification.

AI removed an important safety warning

AI output: A plain-English rewrite is shorter, but the emergency signs and uncertainty have disappeared.

Spot the problem

The text reads more smoothly while becoming less safe.

Reveal what went wrong

What went wrong?

Summarisation can treat qualifications as expendable detail.

Safer approach

Specify warnings that must remain, then compare the rewrite with the source line by line.

Key lesson

Clarity must not come at the expense of safety or accuracy.

AI created a stereotyped fictional client

AI output: The scenario links poverty, offending and poor motivation as though they naturally belong together.

Spot the problem

The fictional case reproduces stigma and narrows discussion before it begins.

Reveal what went wrong

What went wrong?

Patterns in training data can reproduce familiar social and professional stereotypes.

Safer approach

Prompt for strengths, social context, ambiguity and stereotype checks; then review with appropriate people.

Key lesson

Fictional does not automatically mean fair or harmless.

AI invented a research paper

AI output: The title, journal, authors and DOI look credible, but the paper cannot be found.

Spot the problem

A fabricated citation may enter a briefing or training pack as if it were evidence.

Reveal what went wrong

What went wrong?

Language models can construct references from common academic patterns.

Safer approach

Search the DOI, journal and title; open the paper and check the claim in context.

Key lesson

Never cite a paper you have not located and read sufficiently.

The case was still identifiable

AI output: A name was removed, but age, rare circumstances, location and dates remained.

Spot the problem

Colleagues or community members could still recognise the person.

Reveal what went wrong

What went wrong?

Anonymisation is about the whole information combination, not simply deleting a name.

Safer approach

Use a fully fictional case or follow a formally approved anonymisation and processing route.

Key lesson

‘No name’ does not mean ‘not identifiable’.

AI sounded certain when evidence was weak

AI output: The answer states that an intervention ‘will reduce relapse’ although the supplied study was small and observational.

Spot the problem

The conclusion is stronger than the research design allows.

Reveal what went wrong

What went wrong?

AI may smooth uncertainty into decisive prose and confuse association with cause.

Safer approach

Ask for design, sample, limitations and alternative explanations, then inspect the original paper.

Key lesson

Confidence should reflect evidence strength, not writing style.

AI mixed UK and US guidance

AI output: The answer combines UK service terminology with US law, treatment pathways and emergency numbers.

Spot the problem

The final resource looks coherent but is unsafe and unusable in context.

Reveal what went wrong

What went wrong?

Generative AI does not reliably keep jurisdictional boundaries unless they are supplied and checked.

Safer approach

State the UK jurisdiction and source set, prohibit outside claims and verify every pathway and contact.

Key lesson

Always check jurisdiction, not only subject matter.