Notebook idea: RAG debugging cookbook using WFGY 16 Problem Map

[onestardao]

Hi, thanks for maintaining the Fireworks cookbook. The existing notebooks are very helpful for people learning how to build RAG systems on top of Fireworks.

I wanted to propose an extra notebook idea that many users seem to need once they move beyond “hello world”:

Debugging RAG pipelines with a failure-mode map

What I have in mind

There is an open source project called WFGY Problem Map that defines 16 common failure modes (No.1–No.16) for LLM + RAG systems, such as:

• retrieval hallucination despite correct context
• vector store ingestion / index issues
• bootstrap ordering / infra races between app and DB
• config / secrets drift first appearing in production

Problem Map README:

• https://github.com/onestardao/WFGY/tree/main/ProblemMap#readme

A cookbook notebook could:

• build a simple Fireworks-based RAG example
• deliberately trigger 2–3 of the Problem Map failure modes
• show how to recognise each mode from logs / behaviour
• apply a minimal structural fix

This would give users a concrete debugging playbook instead of just tuning prompts or swapping models.

If this sounds interesting, I am happy to help draft an outline or example prompts. If it feels outside the scope of the cookbook, no problem, I know you already have a lot of material to maintain.

[onestardao]

Hi, just a quick follow-up here.

Since this issue was opened, the WFGY 2.0 “RAG 16 Problem Map” has been adopted or referenced by more than 20 repositories in the RAG / LLM ecosystem, including:

• RAGFlow
• LlamaIndex
• ToolUniverse (Harvard MIMS Lab)
• Rankify (University of Innsbruck)
• Multimodal RAG Survey (QCRI LLM Lab)

In addition, several curated “awesome” lists and LLM resource collections have included it as a structured RAG failure-mode checklist.

Across these projects, WFGY is not framed as a competing framework. Instead, it acts as a vendor-neutral diagnostic and evaluation layer — providing a consistent failure taxonomy for RAG drift, retrieval errors, reasoning collapse, and long-horizon instability.

Given that this list curates resources around LLM systems, I wanted to check again whether including WFGY as a debugging / evaluation reference would make sense here.

If aligned with your scope, I’m happy to submit a minimal MVP PR — just a one-line entry, fully aligned with your formatting and style.

If not, totally understandable. Just wanted to reconfirm before proceeding.

Appreciate the work you put into maintaining this list.