Skip to content

Add extensive note about "the indexing warning" in README #38

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Jun 26, 2024
Merged

Add extensive note about "the indexing warning" in README #38

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
9018893
added a note about the indexing warning
Jun 11, 2024
ddcb0a7
rephrasing of the indexing warning
Jun 24, 2024
401bc76
warning text links to the readme section on indexing; syntax fix
Jun 25, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 27 additions & 3 deletions libs/astradb/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ message_history = AstraDBChatMessageHistory(
)
```

## LLM Cache
### LLM Cache

```python
from langchain_astradb import AstraDBCache
Expand All @@ -53,7 +53,7 @@ cache = AstraDBCache(
)
```

## Semantic LLM Cache
### Semantic LLM Cache

```python
from langchain_astradb import AstraDBSemanticCache
Expand All @@ -65,7 +65,7 @@ cache = AstraDBSemanticCache(
)
```

## Document loader
### Document loader

```python
from langchain_astradb import AstraDBLoader
Expand Down Expand Up @@ -100,3 +100,27 @@ store = AstraDBByteStore(
token="AstraCS:...",
)
```

## Warnings about indexing

When creating an Astra DB object in LangChain, such as an `AstraDBVectorStore`, you may see a warning similar to the following:

> Astra DB collection '...' is detected as having indexing turned on for all fields (either created manually or by older versions of this plugin). This implies stricter limitations on the amount of text each string in a document can store. Consider reindexing anew on a fresh collection to be able to store longer texts.

The reason for the warning is that the requested collection already exists on the database, and it is configured to [index all of its fields for search](https://docs.datastax.com/en/astra-db-serverless/api-reference/collections.html#the-indexing-option), possibly implicitly, by default. When the LangChain object tries to create it, it attempts to enforce, instead, an indexing policy tailored to the prospected usage. For example, the LangChain vector store will index the metadata but leave the textual content out: this is both to enable storing very long texts and to avoid indexing fields that will never be used in filtering a search (indexing those would also have a slight performance cost for writes).

Typically there are two reasons why you may encounter the warning:

1. you have created a collection by other means than letting the `AstraDBVectorStore` do it for you: for example, through the Astra UI, or using AstraPy's `create_collection` method of class `Database` directly;
2. you have created the collection with a version of the Astra DB plugin that is not up-to-date (i.e. prior to the `langchain-astradb` partner package).

Keep in mind that this is a warning and your application will continue running just fine, as long as you don't store very long texts.
Should you need to add to a vector store, for example, a `Document` whose `page_content` exceeds ~8K in length, you will receive an indexing error from the database.

### Remediation

You have several options:

- you can ignore the warning because you know your application will never need to store very long textual contents;
- you can ignore the warning and explicitly instruct the plugin _not to_ create the collection, assuming it exists already (which suppresses the warning): `store = AstraDBVectorStore(..., setup_mode=langchain_astradb.utils.astradb.SetupMode.OFF)`. In this case the collection will be used as-is, no (indexing) questions asked;
- if you can afford populating the collection anew, you can drop it and re-run the LangChain application: the collection will be created with the optimized indexing settings. **This is the recommended option, when possible**.
11 changes: 7 additions & 4 deletions libs/astradb/langchain_astradb/utils/astradb.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -317,8 +317,11 @@ def _validate_indexing_policy(
"fields (either created manually or by older "
"versions of this plugin). This implies stricter "
"limitations on the amount of text each string in a "
"document can store. Consider reindexing anew on a "
"fresh collection to be able to store longer texts."
"document can store. Consider indexing anew on a "
"fresh collection to be able to store longer texts. "
"See https://github.com/langchain-ai/langchain-"
"datastax/blob/main/libs/astradb/README.md#"
"warnings-about-indexing for more details."
),
UserWarning,
stacklevel=2,
Expand All @@ -330,7 +333,7 @@ def _validate_indexing_policy(
"fields (either created manually or by older "
"versions of this plugin). This is incompatible with "
"the requested indexing policy for this object. "
"Consider reindexing anew on a fresh "
"Consider indexing anew on a fresh "
"collection with the requested indexing "
"policy, or alternatively leave the indexing "
"settings for this object to their defaults "
Expand All @@ -348,7 +351,7 @@ def _validate_indexing_policy(
"detected as having the following indexing policy: "
f"{options_json}{default_desc}. This is incompatible "
"with the requested indexing policy for this object. "
"Consider reindexing anew on a fresh "
"Consider indexing anew on a fresh "
"collection with the requested indexing "
"policy, or alternatively align the requested "
"indexing settings to the collection to keep using it."
Expand Down