Home / Case studies / Production RAG pipeline for enterprise knowledge
Deepskilling Case studies
Production RAG pipeline for enterprise knowledge
Workshop-shaped capstone: retrieval, evaluation, and guardrails on AWS
Industry: Enterprise knowledge / ML platform
Challenge
A professional services team needed accurate answers over internal playbooks and tickets. Prompt-only chatbots hallucinated policy details and could not cite sources for auditors.
Approach
- Scoped a single workflow: policy Q&A with mandatory citations and refusal when retrieval confidence is low.
- Built hybrid retrieval (metadata filters + embeddings) over chunked playbooks with ownership tags.
- Added an evaluation harness for faithfulness and relevance on a golden question set.
- Deployed behind IAM-controlled endpoints with prompt/retrieval logging for incident review.
Outcomes
- Groundedness scores rose above the team’s acceptance threshold on the golden set before wider rollout.
- Average answer latency stayed within the agreed p95 budget for internal users.
- Support escalations for “wrong policy” answers dropped after refusal behaviour shipped.
Stack
AWS Bedrock · OpenSearch / vector index · Python · Evaluation harness · CloudWatch
Lessons
- Chunking and metadata beat model swaps for early quality gains.
- Ship refusals and eval gates before expanding to more document types.
- Treat retrieval logs as production telemetry, not debug leftovers.
Expert
Rajeev ChandranResearch citations
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Lewis, P. et al. (2020). NeurIPS. Foundational RAG formulation pairing parametric models with non-parametric memory.