Overview
Most RAG demos stop at "it answers questions". A service people can trust needs more: answers grounded in approved documents, citations, safe failure behavior, and a way to measure quality. This article walks through a FastAPI service built on Amazon Bedrock that grows step by step toward that goal.
It is a learning project, not a commercial production system. The name reflects the engineering practices it tries to follow, and the roadmap at the end is explicit about what is still missing.
Step 1: Learn RAG by building it by hand
Before using a managed service, the project implements retrieval manually so each moving part is visible:
- Chunking: splitting a sample lending-policy document into passages.
- Embeddings: converting chunks and questions to vectors.
- Similarity search: ranking chunks by cosine similarity.
- Generation: passing the top chunks to the model as context.
Doing this once makes the managed alternative much easier to evaluate, because you know what it is doing for you.
Step 2: Move to Bedrock Knowledge Bases
The managed version uses Amazon Bedrock Knowledge Bases with Titan Text Embeddings V2 and S3 Vectors as the vector store. An ingestion script loads the documents, and the API calls Bedrock's RetrieveAndGenerate operation, so retrieval and generation happen in one managed call.
The API
| Endpoint | Purpose |
|---|---|
GET /health | Liveness check |
POST /ai/v1/ask | Direct model call through the Bedrock Converse API, with no retrieval |
POST /ai/v1/rag | Knowledge Base question answering with citations |
Keeping the plain model endpoint beside the RAG endpoint makes the difference easy to demonstrate: the same question gets a generic answer from one and a policy-based answer from the other.
Grounding and abstention
The prompt instructs the model to use only the retrieved search results, not outside knowledge, and not to invent policies, thresholds or approvals. When the sources are insufficient, it must respond with a fixed sentence:
I don't know based on the available policy information.A fixed abstention phrase matters because it is predictable for users and testable in automation.
Citations and defensive API design
- Citations: the response includes the source document locations, deduplicated and sorted.
- Validation: Pydantic models enforce request shape and input length limits.
- Controlled errors: AWS exceptions are translated into generic service errors so internal details are not exposed to callers.
- Configuration: settings come from environment variables;
.envis excluded from git and Gitleaks runs as a pre-commit hook.
Evaluating RAG with a golden dataset
The most valuable part of the project is a small evaluation set. Each case defines a question, the expected policy, expected keywords, and whether the system should abstain:
- Answerable questions such as missed repayments, authorization levels for large financing, and debt-to-income thresholds. The expected policy should be retrieved and the answer should contain the expected keywords.
- Unanswerable questions such as minimum credit score or minimum age, which the documents do not cover. The correct behavior is to abstain.
The evaluation script measures retrieval, generation and the overall pipeline separately. That separation shows whether a bad answer came from missing context or from the model ignoring it. The set is small, so treat it as a pattern to expand rather than a benchmark.
What is still missing
- Containerization and a runtime on AWS
- IAM workload identity, API Gateway and authentication
- Bedrock Guardrails
- Observability, retries and scaling behavior
- Larger evaluation sets and security testing
- CI/CD, infrastructure as code and cost controls
Part of this infrastructure work is being explored separately in the enterprise platform project.
Key takeaways
- Build retrieval by hand once; it makes managed services easier to judge.
- Make "I don't know" a first-class, testable behavior.
- Return citations so answers can be verified.
- Evaluate retrieval and generation separately, and include questions that should be refused.
Explore the project
For the source code, setup steps and the current roadmap, visit the production-ai-platform repository on GitHub.