Skip to content

About

Document QA reference implementation: citation validation, refusal tests, usage budgets and reproducible evaluations. Python / FastAPI.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

Production RAG

Self-hosted document QA with citation validation, refusal behavior and usage budgets. An engineering reference project: inspect the implementation, tests and evaluation limits below. Citation verification does not guarantee that every answer is correct.

Monorepo — Turborepo + pnpm. Python backend (packages/core), Next.js frontend (apps/web).


Engineering evidence

This is a public reference implementation, not a client case study or a claim of compliance. For a short technical review:

Question Inspect
What happens to an answer with invalid or missing citations? Grounding tests
How are per-query and daily budgets checked? Budget tests
How is answer quality evaluated? Evaluation artifacts and their context
Which checks actually ran? GitHub Actions

The numbers below describe particular recorded evaluations, not universal accuracy. Check the model, dataset and mode before comparing results. Offline evaluations cannot substitute for a deployment-specific assessment.

Work with Josué on an existing backend or AI integration

Why this exists

Every RAG demo looks great on a single document. Production requires that the system say "I don't know" when the document doesn't say it, that every answer carries a citation back to a real page, and that you can prove the system works by running the same golden questions any time and getting the same pass/fail result.

That's what this project delivers.


Architecture

flowchart LR
    Web[apps/web<br/>Next.js 15 + shadcn]
    MCP[MCP Client<br/>Claude / Cursor / etc.]
    User[User] --> Web
    MCP --> API
    Web -->|HTTP /v1| API[packages/core<br/>FastAPI :8800]
    API --> Svc[RagService]
    Svc --> Embed[Embedder]
    Svc --> Ret[Hybrid Retrieval]
    Svc --> LLM[OpenRouter<br/>gpt-4o-mini]
    Svc --> Budget[BudgetLedger<br/>$5/day, $0.10/query]
    LLM --> Ground[Cite-or-refuse]
    Ground --> Cite[Answer + citations]
Loading

Monorepo layout

production-rag/
├── packages/
│   └── core/            # Python — FastAPI + FastMCP + RAG pipeline (ragcore)
│       ├── src/ragcore/ # service, api, mcp_server, retrieval, generation, budget, security, evals
│       ├── tests/
│       ├── alembic/     # 0001 initial schema (pgvector + FTS)
│       └── pyproject.toml
├── apps/
│   ├── web/             # Next.js 15 — chat UI, ingest, budget, niche switcher
│   ├── medical/         # PubMed niche (:8810)
│   ├── legal/           # CourtListener niche (:8820)
│   └── accounting/       # SEC EDGAR niche (:8830)
├── infra/               # Dockerfile + docker-compose (pgvector)
├── docs/
└── package.json

Latest eval — v0.3.0 (live)

Metric Value
pass_rate 86.4% (19/22)
answered_rate 81.2% (13/16 in-scope)
refusal_rate 100.0% (6/6 out-of-scope)
avg_latency 4.2s
cost $0.00 (free OpenRouter model)
model nvidia/nemotron-3-super-120b-a12b:free

Full breakdown: docs/eval_results_v0.3.0.md · baseline JSON: packages/core/evals/results/baseline_core_live.json
Reproduce: make eval-live (needs OPENROUTER_API_KEY + OPENAI_API_KEY)

Offline CI gate (no network): core 95.5% · regulatory 94.0% — make eval-gate

v0.2.x highlights

  • VECTOR_BACKEND=memory|postgres — Postgres path uses pgvector HNSW + FTS (Alembic 0001)
  • Response cache (TTL), security gate (rate limit / injection / PII mask)
  • Fixed + semantic chunkers; flag-gated rerank (lexical default, optional cross-encoder)
  • Multipart POST /v1/documents/upload + browser drag-and-drop in the web UI
  • Niche apps via create_app(RagService(prompt_builder=...)) — no monkey-patch
  • GET /v1/documents/{id} + MCP get_document_status
  • CI: memory unit suite + Postgres migrate smoke + offline eval regression gate
  • Golden sets: 22 core PDF cases + 50 regulatory (PT-BR BACEN/CVM/LGPD/Simples + EN ASC 606/842)
  • v0.3: Qdrant backend + bench dual-write, always-on rerank (cross-encoder w/ lexical fallback), OTel→Tempo/Prometheus/Grafana compose, offline config matrix (make eval-matrix)

Surfaces

HTTP API (port 8800)

/v1/health · /v1/ask · /v1/search · /v1/documents · /v1/documents/batch · /v1/documents/upload · /v1/budget · /v1/metrics

MCP (port 8801, streamable-http)

ask_documents · search_documents · ingest_document · health · budget

Web UI (port 3000)

  • / — Chat with citation rail
  • /ingest — PDF ingest (single + batch)
  • /budget — Daily ledger dashboard

Quickstart

Requires Python 3.12+, uv, Node 20+ and pnpm 9+. Start with the deterministic checks; live model calls require provider credentials and may incur charges.

git clone https://github.com/joshbarros/production-rag.git
cd production-rag
make install
make test
make eval-gate

For the local API and web UI, copy .env.example to .env at the repository root and configure the providers you intend to use. The default configuration uses OpenRouter for generation and OpenAI for embeddings. Choose a local-development API token; the web client uses a browser-visible token, so this setup is not a production authentication boundary.

cp .env.example .env
# Edit .env locally. Never commit credentials.
# Set API_TOKEN in your shell to match .env before starting both services.
make dev

Use make help for individual API/web, infrastructure and evaluation commands. make eval-live calls configured providers; offline gates do not establish live-provider accuracy. The commands above are aligned with the current Makefile and CI configuration; see the linked CI run for executed checks.


Design system

Editorial / Linear-app style. Dark surface, monospace for data, serif for emphasis, sharp micro-interactions. Full spec in docs/design_system.md.

  • Type: Inter (sans) · Instrument Serif (display, italic) · JetBrains Mono (data)
  • Palette: #0A0A0B bg · #7C3AED → #06B6D4 accent gradient (rationed)
  • Surface: hairline borders, no shadows
  • Motion: 150–250ms ease-out, no springs

What's in the box

Phase Feature Status
2 SQLAlchemy models, Alembic, OpenTelemetry stages ✓
3 PyMuPDF ingest, recursive chunking, hybrid retrieval, cite-or-refuse ✓
4 FastMCP server with structured tool errors + auth ✓
5 Docling OCR fallback for scanned PDFs, batch ingest, doc status ✓
6 Budget ledger, daily/query caps, rejected_budget response ✓
7 Golden-set eval suite (8 cases), scorer, runner ✓
7.5 Fuzzy citation verification (≥70% word overlap) ✓
8 Polish + ship ✓
9 Monorepo (Turborepo + pnpm) + Next.js frontend ✓

What's deferred (in design, not fully built):

  • 500k-chunk scale validation
  • Committed live OpenRouter baseline numbers (make eval-live then commit JSON)
  • Public deploy smoke of free-tier pin (Caddy + demo corpus)

See docs/limits.md for graceful-degradation behavior.

Observability (compose)

make up     # postgres + qdrant + otel + prometheus + tempo + grafana
make obs    # print URLs
# Grafana :3001 (admin/admin) · Prometheus :9090 · Qdrant :6333 · OTLP :4317

Config matrix

make eval-matrix
# → packages/core/evals/results/matrix_core_offline.md

Postgres mode

# start pgvector
docker compose -f infra/docker-compose.yml up -d postgres
# migrate
cd packages/core && uv run alembic upgrade head
# run with durable store
VECTOR_BACKEND=postgres API_TOKEN=changeme \
  uv run uvicorn ragcore.api.app:app --port 8800

About

Document QA reference implementation: citation validation, refusal tests, usage budgets and reproducible evaluations. Python / FastAPI.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages