The biggest risk with AI in agencies is not that the output is bad. It is that it is almost good. It reads well, sounds confident, and looks polished. But buried inside are hallucinated statistics, subtle tone shifts, and claims that fall apart under scrutiny.
If you are scaling AI-assisted delivery across your agency, you need a QA process that is just as scalable. Without one, you are building speed on a foundation of risk. This is one of the key skills gaps we see holding agencies back.
Why AI output needs review
Three categories of failure show up repeatedly in AI-generated agency work.
Hallucinations. AI invents facts. It will cite studies that do not exist, reference statistics with no source, and name tools or features that were never released. This is not occasional. It is a structural feature of how large language models work. If your agency publishes a press release with a fabricated statistic, the client takes the reputational hit.
Tone drift. AI has a default voice: slightly corporate, moderately enthusiastic, generically professional. Over the course of a long piece or multiple pieces, the output drifts away from your client’s brand voice. One blog post might be fine. Twenty blog posts over three months will read like they were written by a different organisation.
Factual errors and outdated information. AI models have training data cutoffs. They will state things as current that are months or years out of date. Pricing, product features, regulatory requirements, and market statistics all shift. AI does not know what has changed since it was trained.
Building a review framework
Your QA process should answer one question at every stage: is this output ready for the client to see?
Here is a framework that works.
Layer 1: Automated checks. Before any human sees the output, run it through automated tools. Grammarly or LanguageTool catches spelling, grammar, and style issues. A custom prompt that asks a second AI model to fact-check claims against known sources catches the most obvious hallucinations. Plagiarism checkers (Copyscape, Originality.ai) verify the content is not too close to existing published work.
Layer 2: Junior review. A junior team member checks for structural issues: does the piece follow the brief, does it hit the right word count, does it include the required keywords or talking points, does it flow logically? This is pattern-matching work. It does not require deep expertise, but it does require attention to detail.
Layer 3: Senior review. A senior team member or subject matter expert reviews for substance. Are the claims accurate? Does the strategic angle make sense? Would this advice actually work? Is this something your agency would stand behind? This is the layer that catches the “almost good” problem, the output that reads well but says nothing useful or, worse, says something wrong.
Layer 4: Brand voice check. For client-facing content, someone who knows the client’s voice reads it purely for tone. Not accuracy, not structure, just voice. Does it sound like the client? Would it fit alongside their existing content? This can be the same person as the senior reviewer, but it should be a distinct pass.
Tiered review: what needs senior eyes
Not every piece of AI output requires the full four-layer treatment. Use a tiered system based on risk.
Tier 1 (full review): High-stakes, public-facing work. Press releases, strategy documents, client presentations, published articles, anything with the client’s name on it. Every layer applies. If you are using AI to draft client reports, the data accuracy check is non-negotiable.
Tier 2 (light review): Internal and operational content. Meeting summaries, internal documentation, project briefs, status updates. Junior review plus a quick senior scan. The stakes are lower, so the process can be faster.
Tier 3 (automated only): Disposable and iterative content. Social media draft options, brainstorming outputs, internal research summaries, first-pass keyword lists. Automated checks only. A human will use these as inputs, not outputs.
The tier system is critical for scaling. If you apply the same rigour to a social media caption that you apply to a press release, AI saves you no time at all.
Tools for automated QA
Build a QA toolkit that runs before human review begins.
- Grammarly Business or LanguageTool: Style, grammar, and readability. Set up custom style guides for each client.
- Custom fact-checking prompts: Use a separate AI instance (not the one that generated the content) to challenge claims. Prompt: “Review the following text. Identify any statistics, dates, product names, or factual claims. For each, assess whether it is likely accurate or potentially fabricated. Flag anything you cannot verify.”
- Originality.ai: AI detection and plagiarism checking. Useful for client reassurance even if you are transparent about AI usage.
- Brand voice scorecards: Create a checklist for each client (formal vs informal, technical vs accessible, specific vocabulary to use or avoid) and score each piece against it.
If you have built prompt engineering into your workflow, your QA burden decreases. Better prompts produce better first drafts, which means less correction downstream.
Maintaining brand voice at scale
Brand voice is where AI-assisted content production most commonly falls down. The solution is not better prompting alone. It is a combination of inputs and checks.
Feed examples, not descriptions. Instead of telling AI “write in a friendly, professional tone,” give it three examples of existing content that nail the client’s voice. Pattern matching from examples works better than interpretation from adjectives.
Create a voice reference document for each client. Include: words they use, words they avoid, sentence length preferences, whether they use contractions, how technical they get, and sample paragraphs that represent the ideal voice. Attach this to every prompt.
Rotate reviewers. If the same person reviews every piece, voice drift becomes invisible. A fresh pair of eyes catches shifts that daily exposure misses.
When to reject vs edit
This is a judgement call, but here are clear guidelines.
Reject and regenerate when:
- The factual foundation is wrong (incorrect statistics, fabricated sources, outdated information that changes the conclusion)
- The structure does not match the brief
- The tone is fundamentally off (formal when it should be casual, or vice versa)
- The piece is generic filler with no genuine insight
Edit when:
- The structure and substance are sound but the voice needs adjustment
- Specific sentences need tightening or clarifying
- Data points need updating or verifying
- The piece needs the addition of original insight, case studies, or data that AI cannot provide
The ratio matters. If you are editing more than 40% of every AI output, your prompts need work, not your editing process. Go back to the brief and the prompt, add more context, and regenerate.
The cost of skipping QA
Agencies that skip QA on AI output learn the hard way. A fabricated statistic in a client report. A press release with a competitor’s messaging baked in. A blog series where every post sounds the same, and nothing sounds like the client.
The time you save by generating content with AI evaporates if you spend it on damage control instead.
Build the QA process now, before you need it. Scale it with tiers so it does not become a bottleneck. And never let AI output reach a client without a human confirming it is worth sending.
This is part of Delivery Notes, a series on implementing AI inside your agency. Subscribe to the newsletter to get new articles weekly.
