[Question]: Nodes from PineconeVectorStore is empty.

[kilimchoi]

Question Validation

• I have searched both the documentation and discord for an answer.

Question

First, I load the documents into Pinecone with this code:

PINECONE_INDEX_NAME ="name"
vector_store = PineconeVectorStore(pinecone_index=pinecone.Index(PINECONE_INDEX_NAME))
storage_context = StorageContext.from_defaults(vector_store=vector_store)
documents = SimpleDirectoryReader(data_dir).load_data()
VectorStoreIndex.from_documents(
    documents, storage_context=storage_context
)

Then I try to load the load the documents from the same vector store using the code below,

index = VectorStoreIndex.from_vector_store(vector_store)
index_nodes = index.docstore.docs
index_nodes <- returns {} 

Do you see anything here that cause the vector store to give an empty response?

[dosubot]

🤖

Hello @kilimchoi! I'm Dosu, a bot designed to assist with bug troubleshooting and answer questions related to the LlamaIndex repository. I'm here to lend a hand while we wait for a human maintainer. Let's get started on your issue!

Based on the information you've provided and the context from the LlamaIndex repository, there could be several reasons why the vector store is returning an empty response.

Firstly, it's possible that there are no matching documents in the index. This could be due to the specific query you're using, or it could be that the index is empty or doesn't contain the documents you're expecting. You might want to check the state of the index and the documents you're trying to load.

Secondly, the query method in the PineconeVectorStore class, which is responsible for querying the index for the most similar nodes, supports filtering. If the filters are too restrictive, they could be excluding all the matching documents, resulting in an empty response. You might want to check the filters you're using.

Thirdly, if there's an error in the query parameters, such as an invalid query mode or an invalid alpha value, the query might fail and return an empty response. You might want to check the query parameters you're using.

Lastly, if the add_sparse_vector attribute is set to True and there's an error in the tokenizer or the sparse vector generation, the query might fail and return an empty response. You might want to check the tokenizer and the sparse vector generation.

In a similar issue found in the LlamaIndex repository, an "Empty Response" was usually because the token counting broke. The user confirmed that setting context_window = 350 resolved the issue. Making it less than 300 or bigger than 400 resulted in errors. You might want to try this solution and see if it resolves your issue. Here is the link to the issue for your reference: [Bug]: "Empty Response" by running official documents.

Without more specific information about the query you're using and the state of the index, it's hard to say exactly what's causing the problem. You might want to consider adding logging or error handling to the query method to help diagnose the issue.

As for the expected behavior of the VectorStoreIndex.from_vector_store(vector_store) method, I wasn't able to find specific information within the repository. It might be helpful if you could provide more details about the behavior you're expecting or any errors you're encountering.

I hope this helps! If you have any more questions or need further clarification, feel free to ask.

Sources

• [Bug]: "Empty Response" by running official documents.
• llama_index/vector_stores/pinecone.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[logan-markewich]

the docstore is disabled by default when using vector db integrations, to simplify storage

you can override this, but then you need to save and manage the docstore too

[logan-markewich]

index = VectorStoreIndex.from_documents(..., store_nodes_override=True)

[kilimchoi]

the docstore is disabled by default when using vector db integrations, to simplify storage

you can override this, but then you need to save and manage the docstore too

ok. In that case, is there a way to fetch documents from an external vector store index without having to manage the docstore? Only reason I'm trying to get the documents is because I want to create a dict to pass it to RecursiveRetriever

[logan-markewich]

only way is to use the vector store API directly (i.e. using pinecone directly), or setting a high top-k and retrieving all the nodes stored

retriever = index.as_retriever(similarity_top_k=10000)
source_nodes = retriever.retrieve("fake")
nodes = [x.node for x in source_nodes]