Self-hosted document QA with citation validation, refusal behavior and usage budgets. An engineering reference project: inspect the implementation, tests and evaluation limits below. Citation verification does not guarantee that every answer is correct.
Monorepo — Turborepo + pnpm. Python backend (
packages/core), Next.js frontend (apps/web).
This is a public reference implementation, not a client case study or a claim of compliance. For a short technical review:
| Question | Inspect |
|---|---|
| What happens to an answer with invalid or missing citations? | Grounding tests |
| How are per-query and daily budgets checked? | Budget tests |
| How is answer quality evaluated? | Evaluation artifacts and their context |
| Which checks actually ran? | GitHub Actions |
The numbers below describe particular recorded evaluations, not universal accuracy. Check the model, dataset and mode before comparing results. Offline evaluations cannot substitute for a deployment-specific assessment.
Work with Josué on an existing backend or AI integration
Every RAG demo looks great on a single document. Production requires that the system say "I don't know" when the document doesn't say it, that every answer carries a citation back to a real page, and that you can prove the system works by running the same golden questions any time and getting the same pass/fail result.
That's what this project delivers.
flowchart LR
Web[apps/web<br/>Next.js 15 + shadcn]
MCP[MCP Client<br/>Claude / Cursor / etc.]
User[User] --> Web
MCP --> API
Web -->|HTTP /v1| API[packages/core<br/>FastAPI :8800]
API --> Svc[RagService]
Svc --> Embed[Embedder]
Svc --> Ret[Hybrid Retrieval]
Svc --> LLM[OpenRouter<br/>gpt-4o-mini]
Svc --> Budget[BudgetLedger<br/>$5/day, $0.10/query]
LLM --> Ground[Cite-or-refuse]
Ground --> Cite[Answer + citations]
production-rag/
├── packages/
│ └── core/ # Python — FastAPI + FastMCP + RAG pipeline (ragcore)
│ ├── src/ragcore/ # service, api, mcp_server, retrieval, generation, budget, security, evals
│ ├── tests/
│ ├── alembic/ # 0001 initial schema (pgvector + FTS)
│ └── pyproject.toml
├── apps/
│ ├── web/ # Next.js 15 — chat UI, ingest, budget, niche switcher
│ ├── medical/ # PubMed niche (:8810)
│ ├── legal/ # CourtListener niche (:8820)
│ └── accounting/ # SEC EDGAR niche (:8830)
├── infra/ # Dockerfile + docker-compose (pgvector)
├── docs/
└── package.json
| Metric | Value |
|---|---|
| pass_rate | 86.4% (19/22) |
| answered_rate | 81.2% (13/16 in-scope) |
| refusal_rate | 100.0% (6/6 out-of-scope) |
| avg_latency | 4.2s |
| cost | $0.00 (free OpenRouter model) |
| model | nvidia/nemotron-3-super-120b-a12b:free |
Full breakdown: docs/eval_results_v0.3.0.md · baseline JSON: packages/core/evals/results/baseline_core_live.json
Reproduce: make eval-live (needs OPENROUTER_API_KEY + OPENAI_API_KEY)
Offline CI gate (no network): core 95.5% · regulatory 94.0% — make eval-gate
VECTOR_BACKEND=memory|postgres— Postgres path uses pgvector HNSW + FTS (Alembic0001)- Response cache (TTL), security gate (rate limit / injection / PII mask)
- Fixed + semantic chunkers; flag-gated rerank (lexical default, optional cross-encoder)
- Multipart
POST /v1/documents/upload+ browser drag-and-drop in the web UI - Niche apps via
create_app(RagService(prompt_builder=...))— no monkey-patch GET /v1/documents/{id}+ MCPget_document_status- CI: memory unit suite + Postgres migrate smoke + offline eval regression gate
- Golden sets: 22 core PDF cases + 50 regulatory (PT-BR BACEN/CVM/LGPD/Simples + EN ASC 606/842)
- v0.3: Qdrant backend + bench dual-write, always-on rerank (cross-encoder w/ lexical fallback),
OTel→Tempo/Prometheus/Grafana compose, offline config matrix (
make eval-matrix)
/v1/health · /v1/ask · /v1/search · /v1/documents · /v1/documents/batch · /v1/documents/upload · /v1/budget · /v1/metrics
ask_documents · search_documents · ingest_document · health · budget
/— Chat with citation rail/ingest— PDF ingest (single + batch)/budget— Daily ledger dashboard
Requires Python 3.12+, uv, Node 20+ and pnpm 9+. Start with the deterministic checks; live model calls require provider credentials and may incur charges.
git clone https://github.com/joshbarros/production-rag.git
cd production-rag
make install
make test
make eval-gateFor the local API and web UI, copy .env.example to .env at the repository root and configure the providers you intend to use. The default configuration uses OpenRouter for generation and OpenAI for embeddings. Choose a local-development API token; the web client uses a browser-visible token, so this setup is not a production authentication boundary.
cp .env.example .env
# Edit .env locally. Never commit credentials.
# Set API_TOKEN in your shell to match .env before starting both services.
make devUse make help for individual API/web, infrastructure and evaluation commands. make eval-live calls configured providers; offline gates do not establish live-provider accuracy. The commands above are aligned with the current Makefile and CI configuration; see the linked CI run for executed checks.
Editorial / Linear-app style. Dark surface, monospace for data, serif for emphasis, sharp micro-interactions. Full spec in docs/design_system.md.
- Type: Inter (sans) · Instrument Serif (display, italic) · JetBrains Mono (data)
- Palette:
#0A0A0Bbg ·#7C3AED → #06B6D4accent gradient (rationed) - Surface: hairline borders, no shadows
- Motion: 150–250ms ease-out, no springs
| Phase | Feature | Status |
|---|---|---|
| 2 | SQLAlchemy models, Alembic, OpenTelemetry stages | ✓ |
| 3 | PyMuPDF ingest, recursive chunking, hybrid retrieval, cite-or-refuse | ✓ |
| 4 | FastMCP server with structured tool errors + auth | ✓ |
| 5 | Docling OCR fallback for scanned PDFs, batch ingest, doc status | ✓ |
| 6 | Budget ledger, daily/query caps, rejected_budget response |
✓ |
| 7 | Golden-set eval suite (8 cases), scorer, runner | ✓ |
| 7.5 | Fuzzy citation verification (≥70% word overlap) | ✓ |
| 8 | Polish + ship | ✓ |
| 9 | Monorepo (Turborepo + pnpm) + Next.js frontend | ✓ |
What's deferred (in design, not fully built):
- 500k-chunk scale validation
- Committed live OpenRouter baseline numbers (
make eval-livethen commit JSON) - Public deploy smoke of free-tier pin (Caddy + demo corpus)
See docs/limits.md for graceful-degradation behavior.
make up # postgres + qdrant + otel + prometheus + tempo + grafana
make obs # print URLs
# Grafana :3001 (admin/admin) · Prometheus :9090 · Qdrant :6333 · OTLP :4317make eval-matrix
# → packages/core/evals/results/matrix_core_offline.md# start pgvector
docker compose -f infra/docker-compose.yml up -d postgres
# migrate
cd packages/core && uv run alembic upgrade head
# run with durable store
VECTOR_BACKEND=postgres API_TOKEN=changeme \
uv run uvicorn ragcore.api.app:app --port 8800