[QUESTION] Exception encountered when running Pathway RAG App in WSL

[robertjak]

Hello,

I was following the tutorial Real-Time AI Pipeline with DeepSeek, Ollama and Pathway trying to run the Pathway RAG App without Docker container directly on the WSL (Windows 11) when I after a prompt to the http://0.0.0.0:8000/v1/pw_ai_answer encountered the following error in the app.py script:

2025-02-14 12:13:35 pathway_engine.engine.dataflow ERROR JSONDecodeError: Expecting ':' delimiter: line 3 column 32 (char 61) in operator 154. Occurred here:
Line: answer = answer.select(answer=extract_answer(pw.this.answer))
File: /home/rjak/.local/lib/python3.12/site-packages/pathway/xpacks/llm/question_answering.py:56
Function: _query_chat_strict_json
2025-02-14 12:13:35 pathway_engine.engine.dataflow ERROR Error value encountered in filter condition, skipping the row in operator 163.
2025-02-14 12:13:35 pathway_engine.engine.dataflow ERROR Error value encountered in filter condition, skipping the row in operator 177.

The Docker container works fine. Please help.

[berkecanrizai]

Hi, I assume you are running the Adaptive RAG that is referenced at the bottom. This exception is thrown from the Adaptive RAG while attempting to parse the model response.

In the case of Adaptive RAG, model responses need to be JSON parsable strings. Can you make sure that your LiteLLM chat instance has the format key in the constructor, or in the YAML?

For example:

$llm_model: "ollama/mistral"

$llm: !pw.xpacks.llm.llms.LiteLLMChat
  model: $llm_model
  retry_strategy: !pw.udfs.ExponentialBackoffRetryStrategy
    max_retries: 6
  cache_strategy: !pw.udfs.DiskCache
  temperature: 0
  top_p: 1
  format: "json"
  api_base: "http://localhost:11434"

Here, format: "json" will force model to give JSON outputs. Let us know if the issue persists.

[szymondudycz]

Hey, can you check again if your app.yaml is the same as required in this step: https://pathway.com/blog/deepseek-ollama#_4-modifying-the-appyaml? In particular did you change question_answerer to be:

question_answerer: !pw.xpacks.llm.question_answering.BaseRAGQuestionAnswerer
  llm: $llm
  indexer: $document_store
  search_topk: 8  # number of retrieved chunks

I think that AdaptiveRAG could also work, but for that remove the line:

strict_prompt: true

it asks the LLM to return json, and not every chat works well with that.

[robertjak]

Hi, thanks for your quick response. Unfortunately, neither suggestion helped. Using the second suggestion I get the following error when posting a prompt:

2025-02-19 10:21:49 pathway_engine.engine.dataflow ERROR APIConnectionError: litellm.APIConnectionError: 'name'
Traceback (most recent call last):
File "/home/rjak/llm-app/examples/pipelines/private-rag/myenv/lib/python3.12/site-packages/litellm/main.py", line 2512, in completion
generator = ollama.get_ollama_response(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rjak/llm-app/examples/pipelines/private-rag/myenv/lib/python3.12/site-packages/litellm/llms/ollama.py", line 294, in get_ollama_response
"name": function_call["name"],
~~~~~~~~~~~~~^^^^^^^^
KeyError: 'name'
in operator 139.

But, as I said, it works in a Docker container on Windows with the same app.yaml configuration.

[berkecanrizai]

@robertjak issue seems to be with the litellm, I think it is the same with this one.

You can try updating the litellm with pip install -U litellm on your WSL. If that also fails, you can open a new shell in the docker and check the litellm version that works.

If none of the above works, you may try installing an older version with pip install "litellm==1.42.0". However, keep in mind that this may be unsafe and is not recommended.

[robertjak]

thanks, but I'm still getting the same error, even when using the same version as in the docker (1.44.28). It seems to be an issue with Litellm.