Problem
The SaaS company, a B2B business with a rapidly growing customer base, was drowning in support tickets. Their support team of eight was handling a high and growing volume of tickets each month as the product scaled. The average first-response time had crept up to 24 hours, and customer satisfaction scores were trending downward — a dangerous trajectory for a company whose retention metrics directly drove revenue.
The root cause was that the vast majority of tickets were repetitive: password resets, billing questions, integration configuration, and “how do I…” feature questions that were all answered in the company’s documentation. Skilled support engineers were spending their days copy-pasting answers from the knowledge base instead of investigating the complex, high-value technical issues that actually required their expertise. The team was burning out, and the company was considering hiring four additional agents — at significant cost — just to keep up.
The leadership team wanted a solution that could automatically resolve the repetitive majority of tickets while escalating the complex minority to human agents with full context. They were adamant that the AI agent must not hallucinate answers or give customers incorrect configuration guidance, as that would be worse than no answer at all.
Solution
We built a retrieval-augmented AI support agent that grounds every response in the company’s own documentation, help center articles, API reference, and historical resolved tickets. The agent is built on LangChain and uses Pinecone as its vector database to store embeddings of all support content. When a customer submits a ticket, the agent retrieves the most relevant documentation chunks, synthesizes a response, and — critically — includes citations linking back to the source articles so the customer can verify the guidance.
The agent handles the full ticket lifecycle autonomously for resolvable issues: it reads the customer’s message, asks clarifying questions if needed, provides a step-by-step solution, and confirms whether the issue is resolved. If the customer confirms resolution, the agent closes the ticket and logs the resolution. If the issue is complex, ambiguous, or the customer indicates the solution did not work, the agent escalates to a human agent with a complete summary of the conversation, the documentation it referenced, and its recommended next steps — so the human agent picks up exactly where the AI left off.
A FastAPI service hosts the agent and exposes it through the same ticketing interface that customers already use, so there is no new tool to learn. The agent identifies itself transparently as an AI assistant and offers a one-click option to escalate to a human at any point, which preserved customer trust and complied with the company’s transparency values.
Architecture
The architecture follows a retrieval-augmented generation (RAG) pattern with three layers: ingestion, retrieval, and generation. The ingestion pipeline runs as a nightly job that scrapes the help center, parses the API documentation, and processes newly resolved tickets, then generates embeddings with OpenAI and stores them in Pinecone. This ensures the agent’s knowledge is always current with the latest product updates and documentation changes.
The retrieval layer uses Pinecone for semantic search. When a ticket arrives, the agent converts the customer’s message into an embedding, queries Pinecone for the most relevant content chunks, and applies a relevance threshold — if no chunks meet the threshold, the agent immediately escalates to a human rather than guessing. This guardrail is what prevented hallucinations and built customer trust.
The generation layer is a LangChain agent hosted in FastAPI. It receives the retrieved context, the conversation history, and the customer’s message, and produces a grounded response with citations. The agent uses a tool-calling pattern to perform actions like checking account status, verifying billing details, or testing API credentials — all through internal APIs — before responding, so its answers are not just text but verified, account-specific guidance.
Workflow Diagram
Customer submits ticket
│
▼
FastAPI Agent Service
│
├──► Embed customer message ──► Pinecone (semantic search)
│ │
│ ▼
│ Relevant docs (above threshold?)
│ │
├──► LangChain Agent ──► Synthesize response + citations
│ │
│ ├──► Resolved? ──► Close ticket, log resolution
│ │
│ └──► Escalate ──► Human agent (with full context + summary)
│
└──► Nightly ingestion: docs + resolved tickets ──► PineconeThe diagram shows both the real-time ticket resolution flow and the nightly ingestion pipeline that keeps the agent’s knowledge base current. The relevance threshold gate is a critical design decision: it ensures the agent only responds when it has high-confidence supporting material, and escalates otherwise.
Technologies
LangChain provides the agent framework, including the tool-calling primitives that let the agent interact with internal APIs (billing, account status, feature flags) to provide account-specific answers rather than generic documentation. The agent’s multi-turn memory allows it to handle clarifying questions naturally.
Pinecone serves as the vector database for semantic retrieval. Its managed infrastructure handled the scaling requirements without any operational burden on the company’s team, and its low-latency query performance kept the agent’s response time under two minutes even as the knowledge base grew to over 10,000 documents.
FastAPI hosts the agent service and provides the high-throughput, low-latency API layer needed to handle thousands of concurrent tickets. OpenAI powers both the embedding generation for the ingestion pipeline and the response generation for the agent, with careful prompt engineering to ensure responses are grounded, cited, and appropriately scoped.
Results
The agent now auto-resolves the majority of all incoming tickets without human intervention — a large number of tickets each month that previously required a human agent’s time. These are the repetitive, documentation-grounded questions that were burning out the support team, and their automated resolution freed the human agents to focus on complex technical investigations and relationship-building with key enterprise customers.
First-response time dropped from 24 hours to under 2 minutes for all tickets, including those that are ultimately escalated to humans, because the agent provides an immediate first response and context summary. For the tickets that are auto-resolved, the customer experience went from a day-long wait to an instant, accurate, cited answer — a transformation that the company’s customer satisfaction scores reflected within the first month.
The financial impact was significant: the company avoided hiring additional support agents, generating meaningful cost savings in fully loaded staffing costs. More importantly, the existing team’s morale improved measurably — they were no longer copy-pasting documentation and could instead focus on the challenging technical work that made the job rewarding.
Lessons Learned
The single most important lesson was that grounding is everything. The first prototype of the agent, built without the relevance threshold and citation system, occasionally produced plausible-sounding but incorrect answers. Once we added the threshold gate — where the agent escalates rather than guesses when retrieval confidence is low — and required citations for every response, customer trust in the agent rose sharply and escalation rates from unsatisfied customers dropped to near zero.
A second lesson was that transparency about AI involvement builds rather than erodes trust. The company was initially nervous about telling customers they were talking to an AI. In practice, customers were not only accepting of it but appreciative — they valued the instant response and the easy option to escalate to a human. Hiding the AI’s nature would have backfired; embracing it transparently was the right call.
Finally, we learned that the nightly ingestion pipeline was the unsung hero of the system. The agent’s accuracy depended entirely on its knowledge being current. By automating the ingestion of new documentation, product updates, and resolved tickets, the agent stayed accurate as the product evolved, with zero manual maintenance — which is the only reason the system remained effective months after launch.