Issue while trying to use RAGAS to evaluate RAG application

[adhsay]

Hello,

If you can kindly help with the issue I am seeing.
I am trying to evaluate my RAG application using giskard and RAGAS,but on trying to do so I am getting the following error:

Giskard_Testing\giskard\rag\metrics\correctness.py", line 77, in call
out = llm_client.complete(
^^^^^^^^^^^^^^^^^^^
AttributeError: 'AzureChatOpenAI' object has no attribute 'complete'

I am using as seen above, AzureChatOpenAI as LLM.I see the same error even if I use ChatOpenAI.

Can you kindly please help on probable solution to this or where am I going wrong.
Thanks in advance.

[pierlj]

Hi, did you follow the instructions from this page to setup your LLM client?

Also, to better understand your issue, can you provide the code you ran to get this error?

[adhsay]

Hello @pierlj , thank you for your reply.

This is the code snippet I am trying to use:

os.environ['AZURE_OPENAI_API_KEY'] = 'key'
os.environ['AZURE_OPENAI_ENDPOINT'] = 'endpoint'
os.environ['OPENAI_API_VERSION'] = 'version'

#openAI
#giskard.llm.set_llm_api("openai")
#oc = OpenAIClient(model="gpt-4-turbo-preview")
#giskard.llm.set_default_client(oc)

You'll need to provide the name of the model that you've deployed

Beware, the model provided must be capable of using function calls

set_llm_model('gpt-35-turbo-16k')

from giskard.rag import generate_testset, KnowledgeBase

import chardet

r_file = open("state_of_the_union.txt", 'rb').read()
result = chardet.detect(r_file)
charenc = result['encoding']
print(charenc)

df = pd.read_csv("state_of_the_union.txt", sep="\t",encoding='Windows-1252')
knowledge_base = KnowledgeBase.from_pandas(df)

Generate a testset with 10 questions & answers for each question types (this will take a while)

testset = generate_testset(
knowledge_base,
num_questions=2,
agent_description="A customer support chatbot for company X", # helps generating better questions
)

Save the generated testset

testset.save("my_testset.json")

Convert it to a pandas dataframe

df = testset.to_pandas()
#df.to_csv("my_testset.csv")
#print(df)
#print(df["question"])

##########################################################################################################

pages = []
loader = TextLoader("./state_of_the_union.txt")

loader.load_and_split()

local_pages = loader.load_and_split() # load and split the pdf into pages
print(len(local_pages))
pages.extend(local_pages)

split the pages into chunks

text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=50)
splits = text_splitter.split_documents(pages)

Create a vector store for the sample data

persist_directory = 'data/chroma/'
#!rm -rf ./data/chroma # remove old database files if any

#embedding = OpenAIEmbeddings()
embedding = AzureOpenAIEmbeddings(
openai_api_version="version",
api_key="key",
azure_endpoint="endpoint",
azure_deployment="text-embedding-ada-002",
)

vectordb = Chroma.from_documents(
documents=splits,
embedding=embedding,
persist_directory=persist_directory
)
vectordb.persist()

retriever = vectordb.as_retriever()

Define LLM

#llm = ChatOpenAI(model_name="gpt-3.5-turbo", temperature=0)

llm = AzureChatOpenAI(
openai_api_version="version",
api_key="key",
azure_endpoint="endpoint",
azure_deployment="gpt-4",
)

Define prompt template

prompt_template = """You are an assistant for question-answering tasks.
Use the following pieces of retrieved context to answer the question.
If you don't know the answer, just say that you don't know.
Question: {question}
Context: {context}
Answer:
"""

prompt = ChatPromptTemplate.from_template(prompt_template)

retrieval = RunnableParallel(
{"context": retriever, "question": RunnablePassthrough()}
)

chain = retrieval | prompt | llm | StrOutputParser()

from giskard.rag import evaluate
answers = []
context = []
queries = df["question"].tolist()
for query in queries:
#question = str(df["question"])
answers.append(chain.invoke(query))
context.append([docs.page_content for docs in retriever.get_relevant_documents(query)])
#print(f"answer is: {answers}")

Run the evaluation and get a report

report = evaluate(answers, testset=testset, knowledge_base=knowledge_base)
report.correctness_by_topic()
report.correctness_by_question_type()
report.failures
report.to_html("rag_eval_report.html")

from giskard.rag.metrics.ragas_metrics import ragas_context_recall, ragas_faithfulness, ragas_context_precision, ragas_answer_relevancy
#ground_truth = df['ground_truth'].tolist()

from giskard.rag.base import AgentAnswer

aa= AgentAnswer(message=answers,documents=context)

report1 = evaluate(
answers,
testset=testset,
knowledge_base=knowledge_base,
#metrics=[ragas_answer_relevancy],
llm_client=llm,
)
report1.to_html("ragas_eval_report_giskard.html")

[pierlj]

You get this error because you pass an instance of AzureChatOpenAI to the evaluate function. This should instead be an instance of giskard LLMClient. You set the default model to gpt-35-turbo-16k above, so I guess you want to use gpt 4 only for the evaluation. To do so, you can create a new LLMClient as follows:

from giskard.llm.client.openai import OpenAIClient
from openai import AzureOpenAI

llm_client = OpenAIClient("gpt-4", AzureOpenAI())

Since you configured the azure env variable properly, this should work. Another possibility is to change the default model used by giskard library with set_default_llm and don't pass anything to the evaluate function, but it changes the model globally, so if you want to use it anywhere else, it might not be convenient.

[adhsay]

Thank you @pierlj for your kind help on this, it works now.

[adhsay]

One more query please @pierlj , can you kindly give an example of how I can use "retrieved_documents" to evaluate RAGAS instead of using functions like get_answer_fn.Thanks in advance.

[pierlj]

To compute the ragas metrics, you need to define an answer_fn that returns an instance of AgentAnswer. The retrieved_document argument from the evaluate function is deprecated, but you can pass a list of AgentAnswer in place of a list of strings as the answers. I will update the docs as it is still mentioned there.

Here is an example you can follow:

from giskard.rag import AgentAnwer

def get_answer_fn(question: str, history=None) -> str:
    """A function representing your RAG agent."""
    # Format appropriately the history for your RAG agent
    messages = history if history else []
    messages.append({"role": "user", "content": question})

    # Get the answer and the documents
    agent_output = get_answer_from_agent(messages)

    # Following llama_index syntax, you can get the answer and the retrieved documents
    answer = agent_output.text
    documents = agent_output.source_nodes

    # Instead of returning a simple string, we return the AgentAnswer object which
    # allows us to specify the retrieved context which is used by RAGAS metrics
    return AgentAnswer(
        message=answer,
        documents=documents
    )

[oussamaJmaaa]

hello , your answer is giving a validation error saying an str type expected (type=type_error.str) .

""""
from giskard.rag import evaluate, RAGReport
from llama_index.core.base.llms.types import ChatMessage, MessageRole
from giskard.rag.metrics.ragas_metrics import (
ragas_context_recall,
ragas_context_precision,
)

report = evaluate(
get_answer_fn,
testset=testset,
knowledge_base=knowledge_base,
metrics=[ragas_context_recall, ragas_context_precision],
)
""""
this is the function that i'm using for evaluation ,i followed the documentation provided for llamaindex integration . maybe it's a version problem ? can you please help me to solve this issue ?

[pierlj]

Hello, can you provide the full stack trace for your issue? Also, what version of giskard and llama-index are you running?

[oussamaJmaaa]

hello , thank you for your response .
this is the full stack trace :
"""
{
"name": "ValidationError",
"message": "1 validation error for Task
input
str type expected (type=type_error.str)",
"stack": "---------------------------------------------------------------------------
ValidationError Traceback (most recent call last)
Cell In[36], line 18
14 return answer
17 # Run the evaluation and get a report
---> 18 report = evaluate(get_answer_fn, testset=testset, knowledge_base=knowledge_base)

File c:\Users\oussa\anaconda3\envs\testENV\lib\site-packages\giskard\rag\evaluate.py:79, in evaluate(answer_fn, testset, knowledge_base, llm_client, agent_description, metrics)
73 if testset is None:
74 testset = generate_testset(knowledge_base)
76 model_outputs = (
77 [_cast_to_agent_answer(ans) for ans in answer_fn]
78 if isinstance(answer_fn, Sequence)
---> 79 else _compute_answers(answer_fn, testset)
80 )
82 llm_client = llm_client or get_default_client()
84 # @todo: improve this

File c:\Users\oussa\anaconda3\envs\testENV\lib\site-packages\giskard\rag\evaluate.py:141, in _compute_answers(answer_fn, testset)
138 if needs_history:
139 kwargs[ANSWER_FN_HISTORY_PARAM] = sample.conversation_history
--> 141 answer = answer_fn(sample.question, **kwargs)
142 model_outputs.append(_cast_to_agent_answer(answer))
144 return model_outputs

Cell In[36], line 11, in get_answer_fn(question, history)
8 messages.append({"role": "user", "content": question})
10 # Get the answer
---> 11 answer = get_answer_from_agent(messages)
12 answer = str(answer).replace("assistant:","")
13 print(type(answer))

Cell In[19], line 9, in get_answer_from_agent(question)
1 def get_answer_from_agent(question):
3 answer = get_agent_worker(
4 index,
5 prefix_system_prompt=AGENT_SYSTEM_PROMPT,
6 query_engine_tool_desc=QUERY_ENGINE_TOOL_DESCRIPTION,
7 agent_model_temp=AGENT_MODEL_TEMPERATURE,
8 agent_llm=AGENT_MODEL,
----> 9 ).as_agent().chat(question)
10 return answer

File c:\Users\oussa\anaconda3\envs\testENV\lib\site-packages\llama_index\core\instrumentation\dispatcher.py:274, in Dispatcher.span..wrapper(func, instance, args, kwargs)
270 self.span_enter(
271 id_=id_, bound_args=bound_args, instance=instance, parent_id=parent_id
272 )
273 try:
--> 274 result = func(*args, **kwargs)
275 except BaseException as e:
276 self.event(SpanDropEvent(span_id=id_, err_str=str(e)))

File c:\Users\oussa\anaconda3\envs\testENV\lib\site-packages\llama_index\core\callbacks\utils.py:41, in trace_method..decorator..wrapper(self, *args, **kwargs)
39 callback_manager = cast(CallbackManager, callback_manager)
40 with callback_manager.as_trace(trace_id):
---> 41 return func(self, *args, **kwargs)

File c:\Users\oussa\anaconda3\envs\testENV\lib\site-packages\llama_index\core\agent\runner\base.py:644, in AgentRunner.chat(self, message, chat_history, tool_choice)
639 tool_choice = self.default_tool_choice
640 with self.callback_manager.event(
641 CBEventType.AGENT_STEP,
642 payload={EventPayload.MESSAGES: [message]},
643 ) as e:
--> 644 chat_response = self._chat(
645 message=message,
646 chat_history=chat_history,
647 tool_choice=tool_choice,
648 mode=ChatResponseMode.WAIT,
649 )
650 assert isinstance(chat_response, AgentChatResponse)
651 e.on_end(payload={EventPayload.RESPONSE: chat_response})

File c:\Users\oussa\anaconda3\envs\testENV\lib\site-packages\llama_index\core\instrumentation\dispatcher.py:274, in Dispatcher.span..wrapper(func, instance, args, kwargs)
270 self.span_enter(
271 id_=id_, bound_args=bound_args, instance=instance, parent_id=parent_id
272 )
273 try:
--> 274 result = func(*args, **kwargs)
275 except BaseException as e:
276 self.event(SpanDropEvent(span_id=id_, err_str=str(e)))

File c:\Users\oussa\anaconda3\envs\testENV\lib\site-packages\llama_index\core\agent\runner\base.py:568, in AgentRunner._chat(self, message, chat_history, tool_choice, mode)
566 if chat_history is not None:
567 self.memory.set(chat_history)
--> 568 task = self.create_task(message)
570 result_output = None
571 dispatch_event(AgentChatWithStepStartEvent(user_msg=message))

File c:\Users\oussa\anaconda3\envs\testENV\lib\site-packages\llama_index\core\agent\runner\base.py:319, in AgentRunner.create_task(self, input, **kwargs)
316 extra_state = self.init_task_state_kwargs
318 callback_manager = kwargs.pop("callback_manager", self.callback_manager)
--> 319 task = Task(
320 input=input,
321 memory=self.memory,
322 extra_state=extra_state,
323 callback_manager=callback_manager,
324 **kwargs,
325 )
326 # # put input into memory
327 # self.memory.put(ChatMessage(content=input, role=MessageRole.USER))
328
329 # get initial step from task, and put it in the step queue
330 initial_step = self.agent_worker.initialize_step(task)

File c:\Users\oussa\anaconda3\envs\testENV\lib\site-packages\pydantic\v1\main.py:341, in BaseModel.init(pydantic_self, **data)
339 values, fields_set, validation_error = validate_model(pydantic_self.class, data)
340 if validation_error:
--> 341 raise validation_error
342 try:
343 object_setattr(pydantic_self, 'dict', values)

ValidationError: 1 validation error for Task
input
str type expected (type=type_error.str)"
}
""""
i'm using the latest version of giksard 2.14.0 and my llamaindex version is 0.10.36

[pierlj]

The issue is that you pass a list of dict (i.e. a list of messages) to the chat function on llama index in your get_answer_from_agent function. It expects a string instead. Inside the get_answer_from_agent, you should split the list of messages to get the user question (the last message of the list) and the chat history and pass this separately to the llama index chat function.