Greenlight

A PM should not need a data analyst to know if a test worked.

Every team running experiments hits the same wall. The test is live, the numbers are trickling in, and the person who most needs an answer, the PM, cannot get one without either learning SQL or waiting for someone who can write it. By the time the analysis comes back, the moment to act on it has often passed.

The Copilot exists to close that gap. Not another chart to stare at, a verdict, with the reasoning attached, the moment enough data exists to trust it.

Dashboards show you numbers. They do not tell you what to do.

Most experimentation tools stop at here is the data. A PM still has to notice the right delta, know whether it is statistically meaningful, and decide what to do about it. Three separate skills, none of which is run the business.

• Generic dashboards report metrics configured up front, and someone has to remember to configure the right one.
• A raw p-value means nothing to most PMs without someone translating it into a decision.
• By the time an analyst is looped in to sanity-check a test, days have usually passed.
• Every new test subject needs its own custom integration before anyone can even look at its data.

One agent, with a hard line between what it decides and what it computes.

The Copilot is a single LangGraph agent a PM talks to like a colleague. The design choice that makes it trustworthy, not just impressive, is a strict boundary: the agent can look, reason, and decide which metric matters, but it never computes the statistics itself. The z-test and the Scale, Continue, Stop, or Rollback rule live in plain, deterministic code the model cannot touch. Ask it the same question on the same data twice and the number never moves, only the explanation might read a little differently.

It never guesses the metric. It looks first.

The interesting engineering problem was not the statistics, it was refusing to hardcode what success means. A hardcoded conversion equals checkout_completed breaks the moment you point the Copilot at a different product. So every analysis runs through the same widening chain.

1. 1Inspect

   A real headless browser opens both variant URLs and captures what is actually on the page.

2. 2Infer

   The agent compares the two DOM snapshots against the event names that genuinely exist in the data, and picks the one metric that reflects what changed.

3. 3Query

   A dedicated read-only data sub-agent writes and runs the SQL to pull that metric per variant.

4. 4Decide

   A fixed, testable rule set turns the numbers into Scale, Continue, Stop, or Rollback.

A recommendation you can check, never a black box.

A PM acting on a Scale or Rollback call is making a real business decision, so it cannot be something they take on faith. The statistics module runs a two-proportion z-test against the exact counts the data sub-agent pulled, and the verdict rules are a fixed table, not a judgment the model makes fresh each time. Every decision comes back with the p-value, the uplift, and the actual SQL that ran, so a skeptical PM can check the work in seconds.

Keeping the agent honest when it is easier for it to fake the work.

The hardest problem was not statistics, it was language models. Left alone, a model will sometimes narrate what a SQL query would return instead of actually calling the tool to run it, especially under pressure in a multi-step reasoning chain. That is invisible in a demo and dangerous in production: a confident answer built on a query that never ran.

The fix was architectural, not prompt tuning. The data-fetching step was pulled into its own sub-agent with a narrow job and a hard requirement to call the tool before answering, and the main agent's prompt walks the pipeline as an explicit, ordered checklist rather than trusting the model to remember the right order. It is the kind of failure you only catch by watching an agent work, not by reading its final answer.

The agent that reads your production data cannot be trusted to also write to it.

This is the part most AI analytics tools skip. The Copilot's SQL access runs as a dedicated read-only Postgres role, not the app's normal database user, with three independent layers stopping it from doing damage. The query has to start with SELECT or it is rejected before running. A second check blocks any write keyword even if it slips past the first. A row limit is enforced even when the agent does not ask for one. Underneath all of it, the role itself has no write grants, so even a bug in the app layer cannot turn into a mutation.

Started in a hackathon, still being hardened.

The Copilot began as a hackathon build and I have kept building on it since, using Claude as an engineering partner throughout, from designing the agent's tool boundaries to writing the FastAPI and React around it. The decisions that mattered most, refusing to let the model touch the statistics, sandboxing its database access at the role level, forcing it to look at the actual pages instead of trusting a config value, were product and security calls before they were code.

It is honest about where it still is. Today the agent's database access sits on the same Postgres instance as the test subject's data, separated by role, not by network. Event ingestion is a direct synchronous write rather than a queued one. Both are active work: moving the agent onto its own database, and decoupling ingestion behind a proper event queue, so any test subject can plug in without ever handing over credentials.