NOTES
Short notes, verified facts.
Short, factual notes on GenAI agents in production: evaluation, guardrails, observability, productionisation on Google Cloud.
-
Two memories, not one: the lawyer’s preferences and the team’s doctrine
A well-injected playbook answers ‘what does the rule say?’, never ‘how does this team work?’. On a contract review agent, I separated each lawyer’s personal memory from the team doctrine, approved by an admin. Two levels, two prompts, and a model that proposes but never writes.
-
Agent observability: one trace per task, one cost per step
An untraced agent is a black box that sends you a bill. The trace model I use in production on a legal agent: Langfuse for LLM calls, cost calculated alongside, and tracing that can’t bring production down.
-
A legal agent doesn’t deliver text, it delivers Word revisions
A lawyer doesn’t review a rewritten paragraph: they review a diff in their own document. How to produce native, word-level OOXML tracked changes, and why Google Docs suggestions take a detour.
-
Maker ≠ checker: why my agents have a sceptic
An LLM on its own makes up references. On a multi-agent system producing scientific briefs, separating the one who writes from the one who verifies — then wiring in programmatic verification — delivered zero hallucinated citations.