Skip to content

[Docs] Add RAG failure modes & debug checklist (#223)#224

Merged
ignorejjj merged 2 commits intoRUC-NLPIR:mainfrom
onestardao:main
Mar 1, 2026
Merged

[Docs] Add RAG failure modes & debug checklist (#223)#224
ignorejjj merged 2 commits intoRUC-NLPIR:mainfrom
onestardao:main

Conversation

@onestardao
Copy link
Copy Markdown
Contributor

Motivation

Many users are now building RAG pipelines directly on top of FlashRAG.

In practice, people often find that benchmarks look good, but the end-to-end system still fails in subtle ways (hallucinated answers, context drift, retrieval that looks fine but does not match the user’s intent, etc.). This PR proposes a small, docs-only checklist page that helps FlashRAG users reason about these common failure modes.

The checklist is written from a FlashRAG-centric perspective and can live entirely inside this repo. At the bottom of the page there is a short note that the table is adapted from an open-source “16 problem map” for RAG debugging (WFGY), but readers do not need to visit that project for the page to be useful.

What this PR does

  • adds a new docs page: docs/rag_failure_modes_and_debug_checklist.md
    • a single table of 16 common failure modes (retrieval, reasoning, state, infra)
    • each row includes: short name, what typically breaks, and how FlashRAG components/logs can help
  • links this page from the Additional FAQs section in README.md:
    • “How can I debug common RAG failure modes when using FlashRAG?”

Scope

  • documentation only
  • no code changes
  • no changes to datasets, configs, or evaluation scripts

The goal is to give FlashRAG users a light-weight debugging companion when they start running their own RAG experiments on top of your framework.

Fixes #223.

This document provides a debugging checklist for teams using FlashRAG, outlining failure modes and a structured approach to identify issues in RAG pipelines.
@ignorejjj ignorejjj merged commit 6aca76d into RUC-NLPIR:main Mar 1, 2026
@onestardao
Copy link
Copy Markdown
Contributor Author

Really appreciate you including WFGY in FlashRAG.

It’s great to see the 16-mode failure map being used as a structured debugging lens. Thank you for making it part of the repo.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Proposal: add a small “RAG failure modes & debug checklist” doc (docs only)

2 participants