AI / Experiment
Agentic Reporting Assistant
An agent that drafts campaign performance narratives from structured data — with hard guardrails so it reports what the numbers say and nothing more.
Started 2025-02-10 / Updated 2025-09-14
Reporting is the part of marketing that eats the most hours and produces the least new thinking. The question I wanted to answer was narrow: can a language model turn a clean performance table into a defensible narrative without inventing causality?
The experiment is deliberately unglamorous. No dashboards, no chat UI. A scheduled job, a structured payload, a prompt contract and an evaluation pass that fails loudly.
What I'm testing
Whether constraining the model's input to a typed metric object — rather than raw exports — removes the hallucinated conclusions that make AI reporting unusable in a real team.
- Only pre-computed metrics and deltas enter the prompt; no free text.
- Every claim in the draft must map back to a metric key.
- Anything the data cannot support is written as an open question, not an insight.
How it is wired
A scheduled task pulls platform data into a normalised table, computes period-over-period deltas and significance flags, then hands a compact JSON object to the model. The response is validated against a schema before anyone sees it.
What I've learned so far
Accuracy is mostly an input-design problem, not a model problem. The moment the payload included raw campaign names and notes, the model started attributing performance to causes nobody had measured.
The useful output is not the narrative. It is the list of questions the agent could not answer — that list is a surprisingly good agenda for the weekly review.
Setup
- Domain
- AI
- Cadence
- Weekly scheduled run
- Guardrail
- Schema-validated output
- Failure mode
- Fails loudly, never silently
Want the detail behind this experiment?
If this overlaps with something you're building, I'm happy to share what worked and what didn't.