Vector store, autodetect mode #65
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 7 commits
August 29, 2024 16:33
cbornet
reviewed
Aug 31, 2024
cbornet
reviewed
Aug 31, 2024
cbornet
reviewed
Aug 31, 2024
cbornet
reviewed
Aug 31, 2024
libs/astradb/langchain_astradb/utils/vector_store_autodetect.py
Outdated
Show resolved
Hide resolved
cbornet
reviewed
Aug 31, 2024
cbornet
reviewed
Aug 31, 2024
cbornet
reviewed
Aug 31, 2024
cbornet
reviewed
Aug 31, 2024
cbornet
reviewed
Aug 31, 2024
cbornet
reviewed
Aug 31, 2024
added 6 commits
September 5, 2024 09:54
…everted docstring on setup_mode
Vector Store "autodetect mode", preliminary guideSummaryThe newly-introduced "autodetect" mode for the Astra DB Vector Store assumes Upon initialization, the vector store class figures out the collection and Tested scenarios:All the following have been tested to work with this init mode.
Note that the vector stores created through LangFlow fall in the first two cases, Usage tipsIf you anticipate the collection to be populated by ingestion pipelines other than Note: autodetect will fall back to "native" mode, also for back-compatibility, in these Typical usageYou can initialize the store with the store = AstraDBVectorStore(
collection_name="my_collection",
token=os.environ["ASTRA_DB_APPLICATION_TOKEN"],
api_endpoint=os.environ["ASTRA_DB_API_ENDPOINT"],
autodetect_collection=True,
)Note that most other init parameters can also be passed (see below for details). Additional parameters:
(Note: both parameters can also be used outside of the autodetect mode -- except for Forbidden parameters:The following parameters, pertaining to how the collection should be created, are not permitted Tested scenarios, detailsHere is a minimal script to try a basic similarity search with an autodetected store: import os
import logging
logging.basicConfig(level=5)
from langchain_astradb import AstraDBVectorStore
ad_store = AstraDBVectorStore(
collection_name=os.environ["AUTODETECT_COLLECTION_NAME"],
token=os.environ["ASTRA_DB_APPLICATION_TOKEN"],
api_endpoint=os.environ["ASTRA_DB_API_ENDPOINT"],
autodetect_collection=True,
)
for doc, sco in ad_store.similarity_search_with_score(
"Tell me so and so",
k=2,
):
print(f"\n\n{'=' * 80}")
print(f"Score = {sco}")
print(f"Page content = {doc.page_content}")The following outlines the preparation steps preliminary to running the above "native" non-vectorize storeSimply use a regular vector store with client-side embeddings and have it Afterwards, try the autodetect script above on the same collection (but see the following Note). Note 1: in this case, you need to create the correct embedding, e.g.: from langchain_openai import OpenAIEmbeddings
embedding = OpenAIEmbeddings(
api_key=os.environ["OPENAI_API_KEY"],
model="text-embedding-ada-002",
)and then pass it as an additional argument ( (Note 2: in this case, "native" vectorize-based storeUse a regular vectorize-based vector store and have it Afterwards, try the autodetect script above on the same collection. non-vectorize with imported CSVYou can create a Collection in the Astra UI with "Bring your own embedding", setting An example CSV is as follows (requires a dimension of 1536): Once the collection is created and populated, try the autodetect script given above, but check the following Note. Note 1: in this case, you need to create the correct embedding, e.g.: from langchain_openai import OpenAIEmbeddings
embedding = OpenAIEmbeddings(
api_key=os.environ["OPENAI_API_KEY"],
model="text-embedding-ada-002",
)and then pass it as an additional argument ( Note 2: if you leave vectorize collection with imported CSV, "$vectorize" columnCreate an empty collection with vectorize and upload a CSV with a column explicitly named Then you can simply run the autodetect mini script above as is. vectorize collection with imported CSV, marked columnCreate an empty collection with vectorize and upload a CSV with no In this example it would be Then you can simply run the autodetect mini script above as is. vectorize collection with PDFCreate an empty collection with vectorize and choose the PDF format in the Upload wizard. Then you can simply run the autodetect mini script above as is. |
cbornet
reviewed
Sep 6, 2024
cbornet
approved these changes
Sep 6, 2024
|
LGTM, great work! |
cbornet
reviewed
Sep 7, 2024
| `metadata_indexing_exclude` | ||
| `collection_indexing_policy` | ||
| `collection_vector_service_options` | ||
| `metric`, `setup_mode`, `metadata_indexing_include`, |
There was a problem hiding this comment.
I think in rst, highlighting a word is done with 2 back-ticks.
Nit as it was already with single ticks.
There was a problem hiding this comment.
Thanks for the info!
I am making all backticks in vectorstore docstring into double backticks right now.
added 2 commits
September 9, 2024 11:07
cbornet
reviewed
Sep 9, 2024
| Note: | ||
| For concurrency in synchronous :meth:`~add_texts`:, as a rule of thumb, on a | ||
| typical client machine it is suggested to keep the quantity | ||
| For concurrency in synchronous :meth:``~add_texts``:, as a rule of thumb, |
There was a problem hiding this comment.
Here it should remain single tick 😉
There was a problem hiding this comment.
ouch! ahahah. Fixing again :)
cbornet
reviewed
Sep 9, 2024
| (specifically, how often a write is an update of an existing id). | ||
| Remember you can pass concurrency settings to individual calls to | ||
| :meth:`~add_texts` and :meth:`~add_documents` as well. | ||
| :meth:``~add_texts`` and :meth:``~add_documents`` as well. |
cbornet
reviewed
Sep 9, 2024
| Completely delete the collection from the database (as opposed | ||
| to :meth:`~clear`, which empties it only). | ||
| to :meth:``~clear``, which empties it only). |
cbornet
reviewed
Sep 9, 2024
| Completely delete the collection from the database (as opposed | ||
| to :meth:`~aclear`, which empties it only). | ||
| to :meth:``~aclear``, which empties it only). |
cbornet
reviewed
Sep 9, 2024
| **kwargs: you can pass any argument that you would | ||
| to :meth:`~add_texts` and/or to the 'AstraDBVectorStore' constructor | ||
| (see these methods for details). These arguments will be | ||
| to :meth:``~add_texts`` and/or to the |
cbornet
reviewed
Sep 9, 2024
| to :meth:`~aadd_texts` and/or to the 'AstraDBVectorStore' constructor | ||
| (see these methods for details). These arguments will be | ||
| routed to the respective methods as they are. | ||
| to :meth:``~aadd_texts`` and/or to the ``AstraDBVectorStore`` |
There was a problem hiding this comment.
Single tick for aadd_texts
cbornet
reviewed
Sep 9, 2024
| """Create an Astra DB vectorstore from a document list. | ||
| Utility method that defers to 'from_texts' (see that one). | ||
| Utility method that defers to ``from_texts`` (see that one). |
There was a problem hiding this comment.
NIT: could use :meth: here for better reference.
cbornet
reviewed
Sep 9, 2024
| Args: see 'from_texts', except here you have to supply 'documents' | ||
| in place of 'texts' and 'metadatas'. | ||
| Args: see ``from_texts``, except here you have to supply 'documents' |
There was a problem hiding this comment.
Not related to this PR: incorrect syntax for Args
There was a problem hiding this comment.
Doing this while I'm at it ...
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I will add some explanations these days (it's too late now, sorry).