A LinkedIn post recently described a 7-person startup that deleted their entire product feedback loop. PostHog captures every user behavior on the site. Five AI agents read the event stream, narrate what happened in each session, extract bugs and product gaps, prioritize them, open GitHub issues, and ship fixes. Humans review the code before production, but the path from “a user struggled” to “the fix is in review” runs without a single status meeting.
Every agent in that chain is a frontier LLM doing end-to-end reasoning. We recognized the pattern immediately. It is the same architecture problem we solved in our autonomous bug-fixing demo, applied to a different domain: expensive models doing domain-specific pattern recognition they have to relearn on every call.
So we built the analysis layer as a template you run on your own data. Three fine-tunable SLM tools that analyze PostHog sessions and produce structured, actionable output. Each one is a small model you train on distil labs and host yourself, so the analysis runs on your own infrastructure instead of a frontier API. This post walks through how.
Three tools, three contracts
The pipeline has three stages. Each one is a standalone tool with a strict input/output schema defined in code. The schemas are the contracts that distil labs trains against, so the downstream consumer never has to parse free-form text.
Narrator takes a raw PostHog session (an array of timestamped events: pageviews, clicks, custom events) and compresses it into a 3-sentence narrative. “User navigated to dashboard, clicked Export CSV twice, received a 500 error both times, and left the page.” The narrative strips noise from the raw event stream and surfaces the behavioral signal.
Extractor takes the narrative and produces structured findings: an array of objects, each with a kind (either a bug or a gap, where a gap is something the user wanted to do but couldn’t), a severity, a title, and supporting evidence. Duplicate findings across sessions are collapsed using embedding similarity, so the same bug surfaced by 50 different users shows up once with 50 pieces of evidence rather than appearing 50 separate times.
Prioritizer takes the deduplicated findings and returns a ranked list with a rationale for each ranking. The output is a JSON array the orchestrator can hand directly to a dashboard, a Notion page, or a downstream agent that opens GitHub issues.
The LinkedIn post also describes downstream agents for GitHub issue creation and PR coding. Those are deliberately outside this repo. They are application code that lives downstream of the analysis. The distil labs contribution is the analysis itself: turning raw event streams into prioritized, structured product intelligence.
Why three narrow tools instead of one big prompt
The obvious approach is to give a frontier model the full PostHog session and ask it to narrate, extract bugs, and prioritize in one shot. That works for a demo. It breaks at scale for the same reason monolithic orchestrators break in our bug-fixing work.
Each tool in the pipeline has a different reasoning profile. The narrator needs to compress noisy event sequences into behavioral summaries. The extractor needs to classify issues and match them against known categories. The prioritizer needs to weigh severity, frequency, and user impact. A single model prompted to do all three produces variable output because it is juggling three distinct cognitive tasks in one inference call.
Splitting them means each tool gets a narrow contract that a small model can learn to fulfill reliably. It also means each tool trains independently, deploys independently, and updates independently. If the extractor’s categories need to change (say you add “accessibility issue” as a new kind), you retrain one model and leave the other two untouched.
Train your own specialists
The three tools are meant to run on SLMs you own. Getting those models is the main path, and it does not require a frontier key. The repo ships seed training data for all three tools in examples/seeds/: hand-authored, schema-valid input/output pairs you hand straight to distil labs: 25 narrator pairs, 25 extractor pairs, and 22 prioritizer batches. distil labs expands them into thousands of synthetic examples (about 10,000 per tool in our run) and returns a fine-tuned model per tool.
When you want the models to reflect your own product rather than the generic seeds, the repo also generates training data from your PostHog sessions. Here a frontier model plays teacher, once, to label your data:
bun run collect-training --limit 200For each tool, this runs the teacher over your cached sessions, validates every output against the tool’s schema, and writes clean training pairs to JSONL in the same shape as the committed seeds. That is the only place a frontier key is involved, and only if you choose this path over the seeds. It reasons about your event patterns from scratch every time, which is exactly the work you are distilling away.
Either way, you get back a fine-tuned model per tool, trained on your data so the students learn your product’s intents, features, and failure patterns rather than a generic demo’s. Deploying each one is a single environment variable:
# pin each tool to the weights distil labs delivers for your data
TOOL_NARRATOR_MODEL=<your-narrator-model>
TOOL_EXTRACTOR_MODEL=<your-extractor-model>
TOOL_PRIORITIZER_MODEL=<your-prioritizer-model>No code change. Same loop. The routing layer reads the model name from the environment and dispatches accordingly. Convert the returned weights to GGUF and the students run on the default Ollama path, or serve the safetensors behind any OpenAI-compatible runtime.
Fine-tuning works here because each tool has a narrow, well-defined contract. A small model can learn to fulfill it more reliably than a generalist prompted to approximate it, since the training narrows the model to exactly the output format the harness needs instead of preserving general-purpose capability it will never use.
Results
We ran the full loop on the repo’s own committed seeds. One student per tool (Qwen3-0.6B for the narrator, Qwen3-1.7B for the extractor and prioritizer), trained on the distil labs platform. Every model below was scored on the same held-out test set by an LLM judge. Cells show judge passes out of the held-out n, with ROUGE in parentheses; the test sets are small (5, 5, and 4 examples), which is why we back the table with live testing below:
| Tool | Untrained student | Teacher (gpt-oss-120b) | Tuned student |
|---|---|---|---|
| narrator (Qwen3-0.6B, n=5) | 0/5 (39.2) | 5/5 (62.1) | 5/5 (64.0) |
| extractor (Qwen3-1.7B, n=5) | 2/5 (60.3) | 4/5 (67.6) | 4/5 (68.9) |
| prioritizer (Qwen3-1.7B, n=4) | 3/4 (39.0) | 3/4 (47.2) | 3/4 (50.3) |
The tuned students match the 120B teacher’s judge score on all three tools and edge it on ROUGE, at a fraction of the size and at zero inference cost on your own hardware.
The untrained column is the point. Before training, the base model does not just lose on quality, it breaks the contract: in our live runs it violated the 3-sentence narration format on roughly a third of sessions and invented facts, like reporting a successful login in a session where the user never got past a failed password reset. After training, the same 0.6B held the format on all ten test sessions in both runs, citing real queries, button labels, and error codes.
The prioritizer’s judge is pass/fail on a hard constraint (every finding id ranked exactly once) and saturates for base and student alike, so live testing is the sharper lens there: our first 0.6B attempt dropped an id when a batch contained near-duplicate findings, while the 1.7B student we ship held exact coverage on every batch we tested, including that adversarial one.
The repo ships an eval so you can score your own students against the teacher:
bun run eval --sample 20The eval targets the extractor, the schema-critical middle of the pipeline: it runs the student and the teacher on the same narrations and has a frontier judge score both on precision, recall, and severity calibration. Our run against gpt-5-mini as teacher and judge: 3 ties, 1 student win, 1 teacher win across the 5 demo narrations, with the student running locally at $0 per call. The same pattern held in our autonomous bug-fixing demo: a 0.6B student outperformed its 120B teacher by 29 percentage points on the target task, because a model fine-tuned to one contract beats a generalist approximating it. Run the eval on your sessions and you get your own version of that comparison, not ours.
What it costs
Teacher mode (frontier models doing all three tools): the demo runs about $0.10 to $0.30 for 5 sample sessions, roughly $0.02 to $0.06 per session across the three calls. That scales close to linearly: around $20 to $60 a day at 1,000 sessions, and $200 to $600 a day at 10,000, before any downstream agents touch the output. bun run report gives you the exact spend per tool and per model after any run.
After distillation (self-hosted SLMs): the models run on your own GPU or CPU, so inference cost becomes a fixed infrastructure line item rather than a per-call API charge.
The economics change for the same reason as the bug-fixing demo. Domain reasoning moves out of the expensive model. The teacher’s job during data collection is to demonstrate what correct output looks like for each input. Once the student learns that mapping, the teacher is no longer in the loop. You stop paying a frontier model to re-read your product’s UX patterns on every session.
Try it yourself
The repo ships committed seed data, so you can get to a trained model without running a frontier teacher at all. examples/seeds/ contains hand-authored, schema-valid training pairs for all three tools. Hand them to distil labs, host the models you get back, pin them per tool, and run the pipeline on your own hardware:
bun install
cp .env.example .env
# pin the models distil labs trains from the seeds:
# TOOL_NARRATOR_MODEL=<your-narrator-model>
# TOOL_EXTRACTOR_MODEL=<your-extractor-model>
# TOOL_PRIORITIZER_MODEL=<your-prioritizer-model>
bun run demo # 5 bundled sample sessions, on the SLM engine, no API keyThe demo ships with bundled sample sessions, so you do not need a PostHog instance to see the loop. Five sessions go through narrator, extractor, and prioritizer (duplicate findings merge as occurrence evidence on the way), and the findings land in a SQLite database. bun run report turns them into a markdown rollup and bun run dashboard serves a live view at localhost:8787. Our full run on the trained models ends at Total USD: $0.000000.
You can also skip training entirely. The three models we trained from the committed seeds are published on Hugging Face: distil-qwen3-0.6b-posthog-narrator, distil-qwen3-1.7b-posthog-extractor, and distil-qwen3-1.7b-posthog-prioritizer. Pull them, pin them, and the demo runs the trained pipeline out of the box. To watch the loop with a generic model first, ollama pull qwen2.5:7b and run the same command against the placeholder.
To train on your own product’s behavior instead of the seeds, point collect-training at your PostHog sessions and let the teacher label them. And if you just want to see the pipeline run on a frontier model, bun run demo --engine llm is the optional teacher path. Neither is required to get started: the seeds plus your distil labs models are the whole loop, key-free.
Source: GitHub
How the pieces fit together
distil labs trains the SLM tools. Your PostHog event patterns become fine-tuned specialists that produce structured output on demand. The platform handles synthetic data expansion, training, and model delivery.
PostHog provides the event stream. The pipeline ingests sessions via PostHog’s API and processes them through the three-tool chain.
The runtime is pluggable. Ollama for local development, vLLM or LM Studio for on-premises hosting, or a Cloudflare Worker fronting your distilled weights for edge deployment. The same code runs against any OpenAI-compatible HTTP endpoint.
This is the same intelligent harness pattern from our autonomous bug-fixing series, applied to a domain every product team understands. Instead of crash logs that get auto-diagnosed, it is user sessions that get auto-classified into prioritized, actionable product intelligence.