Support assistant
A language-model assistant that drafts replies from a company's help docs and past tickets. A human reads and approves every send; the model never talks to a customer directly.
- Client
- B2B SaaS, Series A
- Year
- Role
- Sole engineer, embedded with the support team
- Timeline
- 6 weeks
- Stack
- TypeScript, Postgres with pgvector, OpenAI API, Zendesk
The brief
A four-person support team answering the same two hundred questions in slightly different words, with a median first response of four hours. Leadership wanted "AI support". The team wanted to not be replaced by a chatbot that made things up.
The brief I agreed with both sides: make the team faster, keep a person on every reply, and be honest about what the model doesn't know.
What I built
A sidebar inside the inbox the team already used. When a ticket arrives, the assistant retrieves the most relevant help articles and resolved tickets, drafts a reply, and shows its sources. The agent edits or discards it, then sends.
- Retrieval over help docs and 18 months of resolved tickets, re-indexed nightly.
- Drafts with inline citations, so the agent can check the claim before sending it.
- A "don't know" path: if retrieval confidence is low, the draft says so rather than guessing.
- Feedback loop: every edit the team makes is logged and reviewed weekly to tune prompts and sources.
- Evaluation set of 300 real tickets, run on every change, so quality regressions are caught before deploy.
The first two weeks were spent on the evaluation set, before any model was called. It is the reason the rest of the project went quickly.
Outcome
Median first response fell from four hours to twenty minutes. Around 70% of drafts were sent with light edits or none. Nobody on the team was replaced; two of them now spend part of the week improving the help docs the assistant draws from.
The company has since asked for the same approach on their sales inbox.