You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Also, you can just load a partition into a resource group and have its replicas distributed among several resource groups. The following assumes that a collection named `Books` already exists and it has a partition named `Novels`.
156
159
157
160
```python
158
-
collection = Collection("Books")
161
+
collection ="Books"
162
+
partition ="Novels"
159
163
160
164
# Use the load method of a collection to load one of its partition
Note that `_resource_groups`is an optional parameter, and leaving it unspecified have Milvus load the replicas onto the query nodes in the default resource group.
@@ -180,8 +180,8 @@ All code samples on this page are in PyMilvus 2.5.3. Upgrade your PyMilvus insta
print(f"Succeeded in moving {num_replicas} replica(s) of {collection_name} from {source} to {target}.")
185
185
exceptException:
186
186
print("Something went wrong while moving replicas.")
187
187
@@ -193,17 +193,18 @@ All code samples on this page are in PyMilvus 2.5.3. Upgrade your PyMilvus insta
193
193
You can drop a resource group that hold no query node (`limits.node_num = 0`) at any time. In this guide, resource group `rg` now has one query node. You need to change the configuration `limits.node_num` of resource group into zero first.
print(f"Something went wrong while dropping {source}.")
207
+
print(f"Something went wrong while dropping {resource_group}.")
207
208
```
208
209
209
210
For more details, please refer to the [relevant examples in pymilvus](https://github.com/milvus-io/pymilvus/blob/v2.4.3/examples/resource_group_declarative_api.py)
@@ -219,34 +220,33 @@ Here is a good practice for managing QueryNodes in a cloud environment:
219
220
Here is an example setup:
220
221
221
222
```python
222
-
from pymilvus import utility
223
223
from pymilvus.client.types import ResourceGroupConfig
224
224
225
225
_PENDING_NODES_RESOURCE_GROUP="__pending_nodes"
226
226
227
227
def init_cluster(node_num: int):
228
228
print(f"Init cluster with {node_num} nodes, all nodes will be put in default resource group")
229
229
# create a pending resource group, which can used to hold the pending nodes that do not hold any data.
Copy file name to clipboardExpand all lines: v2.5.x/site/en/integrations/kafka-connect-milvus.md
+15-4Lines changed: 15 additions & 4 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,13 +1,24 @@
1
1
---
2
2
id: kafka-connect-milvus.md
3
-
summary: In this quick start guide we show how to setup open source kafka and Zilliz Cloud to ingest vector data.
4
-
title: Integrate Milvus with WhyHow
3
+
summary: Apache Kafka is integrated with Milvus and Zilliz Cloud to stream vector data. Learn how to use Kafka-Milvus connector to build real-time pipelines for semantic search, recommendation systems, and AI-driven analytics.
4
+
title: Connect Apache Kafka® with Milvus/Zilliz Cloud for Real-Time Vector Data Ingestion
5
5
---
6
6
7
-
# Connect Kafka with Milvus
7
+
# Connect Apache Kafka® with Milvus/Zilliz Cloud for Real-Time Vector Data Ingestion
8
8
9
9
In this quick start guide we show how to setup open source kafka and Zilliz Cloud to ingest vector data.
10
10
11
+
This tutorial explains how to use Apache Kafka® to stream and ingest vector data into Milvus vector database and Zilliz Cloud (fully-managed Milvus), enabling advanced real-time applications such as semantic search, recommendation systems, and AI-powered analytics.
12
+
13
+
Apache Kafka is a distributed event streaming platform designed for high-throughput, low-latency pipelines. It is widely used to collect, store, and process real-time data streams from sources like databases, IoT devices, mobile apps, and cloud services. Kafka’s ability to handle large volumes of data makes it an important data source of vector databases like Milvus or Zilliz Cloud.
14
+
15
+
For example, Kafka can capture real-time data streams—such as user interactions, sensor readings, together with their embeddings from machine learning models—and publish these streams directly to Milvus or Zilliz Cloud. Once in the vector database, this data can be indexed, searched, and analyzed efficiently.
16
+
17
+
The Kafka integration with Milvus and Zilliz Cloud provides a seamless way to build powerful pipelines for unstructured data workflows. The connector works for both open-source Kafka deployment and hosted services such as [Confluent](https://www.confluent.io/hub/zilliz/kafka-connect-milvus) and [StreamNative](https://docs.streamnative.io/hub/connector-kafka-connect-milvus-sink-v0.1).
18
+
19
+
In this tutorial we use Zilliz Cloud as a demostration:
20
+
21
+
11
22
## Step 1: Download the kafka-connect-milvus plugin
12
23
13
24
Complete the following steps to download the kafka-connect-milvus plugin.
@@ -116,4 +127,4 @@ Ensure you have Kafka and Zilliz Cloud setup and properly configured.
116
127
117
128
### Support
118
129
119
-
If you require any assistance or have questions regarding the Kafka Connect Milvus Connector, please feel free to reach out to our support team: **Email:** [[email protected]](mailto:[email protected])
130
+
If you require any assistance or have questions regarding the Kafka Connect Milvus Connector, please feel free to reach out to the maintainer of the connector: **Email:** [[email protected]](mailto:[email protected])
0 commit comments