alexandre.

← Notes

Maker ≠ checker: why my agents have a sceptic

An LLM on its own makes up references. On a multi-agent system producing scientific briefs, separating the one who writes from the one who verifies — then wiring in programmatic verification — delivered zero hallucinated citations.

  • agents
  • ADK
  • guardrails
  • evaluation

An LLM drafting a scientific brief produces fluent, well-structured, convincing text. And some of its references don’t exist.

I ran into the problem on a concrete case: preparing outreach briefs for scientific key opinion leaders (KOLs). Done by hand, the work is time-consuming: reading a researcher’s recent literature, extracting a credible angle of approach, sourcing every claim. Automated naively, it becomes dangerous — one invented citation in a brief that lands on the desk of the very expert you’re trying to approach, and the team’s credibility is gone for good. The cost of an error is not proportional to its frequency.

Why a single agent hallucinates

This isn’t a one-off slip; it’s an expected failure mode. A model is trained to produce a plausible sequence of tokens; post-training partly teaches it to abstain, but nothing gives it access to the world. And a bibliographic reference is precisely the easiest object to make plausible: a credible author name, a journal that exists, a consistent year, a well-formed DOI. In the generated text, the real citation and its counterfeit are indistinguishable: you tell them apart by looking elsewhere, not by reading more carefully.

The intuitive fix — adding ‘check your sources’ to the prompt — runs into a structural problem: it is the same model, in the same context, with the same biases, being asked to doubt what it has just written. It rereads its own output having already accepted its premises.

The architecture: whoever writes doesn’t verify

In finance and in industry, the ‘maker ≠ checker’ rule is as old as auditing itself: the person who produces a deliverable is never the one who signs it off. I apply the same rule to agents — at the level of the architecture, not the prompt.

The system is a multi-agent pipeline built with Google ADK on Vertex AI, combining three types of orchestration: Sequential to chain collection, drafting, verification and delivery; Parallel to work several leads at once instead of one after another; LoopAgent to run the drafting ↔ verification loop until it converges.

Architecture: writer, skeptic and Europe PMC gate in the same loop The collected sources enter a LoopAgent that chains three sub-agents: the writer drafts, the skeptic challenges, then a programmatic verification gate queries Europe PMC. That gate, and it alone, decides whether to exit the loop: if a citation cannot be found or an objection remains open, the next pass goes back to the writer; otherwise the brief is assembled. sources LoopAgent · stops on escalate, or on max_iterations writer drafts, cites skeptic challenges citation_gate Europe PMC no LLM brief escalate citation not found or open objection → next pass max_iterations reached without escalate → failure, not a green light Architecture: writer, skeptic and Europe PMC gate in the same loop The collected sources enter a LoopAgent that chains three sub-agents: the writer drafts, the skeptic challenges, then a programmatic verification gate queries Europe PMC. That gate, and it alone, decides whether to exit the loop: if a citation cannot be found or an objection remains open, the next pass goes back to the writer; otherwise the brief is assembled. sources LoopAgent · stops on escalate, or on max_iterations writer drafts, cites skeptic challenges citation_gate Europe PMC no LLM escalate brief citation not found or open objection → next pass max_iterations reached without escalate → failure, not a green light

At the heart of the loop, two separate roles. The writer drafts and cites its sources. The sceptic reviews with a single instruction: find what doesn’t hold up. It rewrites nothing — it challenges. The asymmetry is deliberate: asking a model to improve a text and asking it to attack one don’t elicit the same behaviour, and I’d rather have an agent whose objective is explicitly adversarial than a polite one.

The code, simplified

Here is the skeleton of the pipeline. It is simplified: collect_sources, assemble_dossier and extract_dois are omitted, and the constants (model, iteration cap, timeout) are there to illustrate, not to document the actual tuning. The rest runs.

from typing import AsyncGenerator

import httpx
from google.adk.agents import BaseAgent, LlmAgent, LoopAgent, SequentialAgent
from google.adk.agents.invocation_context import InvocationContext
from google.adk.events import Event, EventActions
from pydantic import BaseModel

EUROPE_PMC = "https://www.ebi.ac.uk/europepmc/webservices/rest/search"


class Objections(BaseModel):
    items: list[str]


async def unindexed_dois(dois: list[str]) -> list[str] | None:
    """DOIs missing from the Europe PMC index. No LLM here.

    Returns None if Europe PMC is unreachable: "I don't know" is not
    "all good", and the gate has to treat the two differently.
    """
    missing: list[str] = []
    try:
        async with httpx.AsyncClient(timeout=10) as client:
            for doi in dois:
                response = await client.get(
                    EUROPE_PMC,
                    params={"query": f'DOI:"{doi}"', "format": "json", "pageSize": 1},
                )
                response.raise_for_status()
                if response.json().get("hitCount", 0) == 0:
                    missing.append(doi)
    except httpx.HTTPError:
        return None
    return missing


writer = LlmAgent(
    name="writer",
    model="gemini-flash-latest",
    instruction=(
        "Write the dossier section from {sources}. "
        "Every claim carries a DOI taken from {sources}. "
        # `{key?}` rather than `{key}`: on the first pass, objections doesn't
        # exist in session yet, and a missing key raises an error.
        "If {objections?} is not empty, fix only the listed points."
    ),
    output_key="draft",
)

skeptic = LlmAgent(
    name="skeptic",
    model="gemini-flash-latest",
    instruction=(
        "You rewrite nothing, you challenge. List the flaws in {draft}: "
        "a claim not supported by {sources}, an extrapolation, "
        "a DOI that doesn't match the point being made."
    ),
    # Without output_schema, output_key stores raw text: "No objections."
    # would be a non-empty string, hence truthy, and the loop would never exit.
    output_schema=Objections,
    output_key="objections",
)


class CitationGate(BaseAgent):
    """Deterministic gate. The only thing that decides whether the loop exits."""

    async def _run_async_impl(
        self, ctx: InvocationContext
    ) -> AsyncGenerator[Event, None]:
        state = ctx.session.state
        missing = await unindexed_dois(extract_dois(state.get("draft", "")))
        objections = (state.get("objections") or {}).get("items", [])
        # `missing == []` rather than `not missing`: None (API unreachable) is
        # not the same as an empty list (everything verified).
        verified = missing == [] and not objections
        yield Event(
            author=self.name,
            actions=EventActions(
                escalate=verified,
                state_delta={"verified": verified, "unindexed_dois": missing},
            ),
        )


review_loop = LoopAgent(
    name="review_loop",
    sub_agents=[writer, skeptic, CitationGate(name="citation_gate")],
    max_iterations=3,
)

pipeline = SequentialAgent(
    name="kol_dossier",
    # assemble_dossier refuses to assemble if state["verified"] is false.
    sub_agents=[collect_sources, review_loop, assemble_dossier],
)

Two details carry everything else.

The first: the loop exit is decided by CitationGate, not by a model. A LoopAgent hands back control when a sub-agent emits escalate=True, or when max_iterations is reached. Handing that signal to an LLM means asking it to authorise its own publication.

The second: reaching max_iterations is not a success. That is the trap in this set-up: the loop terminates normally, the next agent runs, and an unverified brief goes out. Hence the verified flag in session, which the assembly step reads before assembling. The guarantee doesn’t come from the loop; it comes from the fact that only one branch leads to the deliverable.

What worked, what isn’t enough

What isn’t enough: the sceptic alone. A second model reviewing a first one catches a great deal: extrapolations, unsourced claims, drift between what a reference says and what it is made to say. But on whether a reference exists, it is exactly as helpless as the writer: all it has in front of it is text, and an invented DOI is well-formed text. This isn’t a measured failure; it’s a limit of principle — a checker can only rule on what it can observe, and this one observes nothing.

What worked: leaving the LLM for the final step. Every citation is checked programmatically against Europe PMC. No interpretation, no confidence score: one call, one verdict. A citation that can’t be found is rejected and sent back to the writer with the offending identifier — so with something to fix, not just a reproach.

What I watch on every set-up like this: the loop that doesn’t converge. A drafting ↔ verification loop can spin without making progress. On the Statements of Work (SoW) agent, where a manager ↔ agent review loop is supervised by an LLM judge, there is explicit detection of incomplete answers and an anti-loop mechanism that re-prompts in a targeted way instead of repeating the same request. That case is, incidentally, the exception to my own rule: the exit there depends on a model’s judgement, because ‘is the manager’s answer complete?’ has no deterministic oracle. When no source of truth exists, a bounded LLM judge is better than nothing; when one does exist, doing without it is a choice made by default, not a constraint.

The metric, and what it doesn’t say

Zero hallucinated citations across the briefs produced. It is the only metric I claim here, and it has to be seen for what it is: less a score measured after the fact than a property of the gate. No brief is assembled until every DOI has been found — the zero is the consequence of the guardrail, not a performance of the model.

What it doesn’t say, better to spell it out before I’m asked. It measures neither editorial relevance, nor the completeness of the collection, nor whether the citation actually supports the point it is attached to. And ‘not found in Europe PMC’ is not ‘does not exist’: a valid DOI outside the index’s scope — a preprint, a book chapter, a non-indexed journal, a publication that is too recent — will be wrongly rejected. The gate is tuned to produce false positives rather than false negatives: that is the right way round for the risk here, but it is a setting, not a truth. Checking that a reference is indexed is a membership test, not a correctness test. It’s the floor, not the ceiling — but it’s the floor that was missing.

Checklist: adding a checker to your agent

  1. Name the deliverable and its failure mode. ‘An invented citation in a brief sent to an expert’ is a failure mode. ‘Hallucinations’ is not.
  2. Look for the source of truth. An API, an internal reference dataset, a product database, a written business rule. If one exists, it has the final say, not the model.
  3. Make the checker a separate agent. Not a sentence added to the writer’s prompt: an agent with its own objective, its own model if need be, and no permission to rewrite.
  4. Give it an instruction to attack, not to improve. ‘Find what doesn’t hold up’ and ‘improve this text’ don’t aim at the same thing.
  5. Make the guardrail blocking. A warning at the bottom of the page gets ignored. A rejected revision doesn’t ship. If your control is advisory, you don’t have a control.
  6. Take the decision out of the LLM, whenever an oracle exists. And when none exists, bound the judge and log its verdicts instead of trusting it.
  7. Treat ‘I don’t know’ as a failure, not a green light. Unreachable API, timeout, quota: the most dangerous case for a verifier is the one where it doesn’t answer and the code reads its silence as consent.
  8. Check what your iteration cap actually does. In most frameworks, hitting the cap ends the loop without signalling anything. If the next step doesn’t reread an explicit flag, your guardrail can be bypassed by exhaustion.
  9. Return the error with what’s needed to fix it. The offending identifier, the rule violated, the source passage. A rejection without a diagnosis makes the loop spin for nothing.

What I look at first when I come in

Faced with an existing agent, the first question is simple: who verifies? If the answer is ‘the same model, in the same prompt’, the risk is not under control, however good the demo looks. The second question is just as simple: what happens when the verifier says no — and when it says nothing at all?

Separating the maker from the checker, then wiring in deterministic verification wherever a source of truth exists, is the change with the best effort-to-risk ratio I know. The same principle runs on other cases: blocking guardrails stop a legal agent from proposing a revision without a source passage. That’s always where I start.