feat: impl StructArray -- support embedding searches embeddings in embedding list with element level filter expression by SpadeA-Tang · Pull Request #45830 · milvus-io/milvus

SpadeA-Tang · 2025-11-25T07:05:36Z

For a vector field inside a STRUCT, since a STRUCT can only appear as the element type of an ARRAY field, the vector field in STRUCT is effectively an array of vectors, i.e. an embedding list.
Milvus already supports searching embedding lists with metrics whose names start with the prefix MAX_SIM_.

This PR allows Milvus to search embeddings inside an embedding list using the same metrics as normal embedding fields. Each embedding in the list is treated as an independent vector and participates in ANN search.

Further, since STRUCT may contain scalar fields that are highly related to the embedding field, this PR introduces an element-level filter expression to refine search results.
The grammar of the element-level filter is:

element_filter(structFieldName, $[subFieldName] == 3)

where $[subFieldName] refers to the value of subFieldName in each element of the STRUCT array structFieldName.

It can be combined with existing filter expressions, for example:

"varcharField == 'aaa' && element_filter(struct_field, $[struct_int] == 3)"

A full example:

struct_schema = milvus_client.create_struct_field_schema()
struct_schema.add_field("struct_str", DataType.VARCHAR, max_length=65535)
struct_schema.add_field("struct_int", DataType.INT32)
struct_schema.add_field("struct_float_vec", DataType.FLOAT_VECTOR, dim=EMBEDDING_DIM)

schema.add_field(
    "struct_field",
    datatype=DataType.ARRAY,
    element_type=DataType.STRUCT,
    struct_schema=struct_schema,
    max_capacity=1000,
)
...

filter = "varcharField == 'aaa' && element_filter(struct_field, $[struct_int] == 3 && $[struct_str] == 'abc')"
res = milvus_client.search(
    COLLECTION_NAME,
    data=query_embeddings,
    limit=10,
    anns_field="struct_field[struct_float_vec]",
    filter=filter,
    output_fields=["struct_field[struct_int]", "varcharField"],
)

TODO:

When an element_filter expression is used, a regular filter expression must also be present. Remove this restriction.
Implement element_filter expressions in the query.

Signed-off-by: SpadeA <[email protected]>

sre-ci-robot · 2025-11-25T07:06:24Z

[ci-v2-notice]
Notice: We are gradually rolling out the new ci-v2 system.

Legacy CI jobs remain unaffected, you can just ignore ci-v2 if you don't want to run it.
Additional "ci-v2/*" checkers will run for this PR to ensure the new ci-v2 system is working as expected.
For tests that exist in both v1 and v2, passing in either system is considered PASS.

To rerun ci-v2 checks, comment with:

/ci-rerun-code-check // for ci-v2/code-check
/ci-rerun-build // for ci-v2/build
/ci-rerun-ut-integration // for ci-v2/ut-integration
/ci-rerun-ut-go // for ci-v2/ut-go
/ci-rerun-ut-cpp // for ci-v2/ut-cpp
/ci-rerun-ut // for all ci-v2/ut-integration, ci-v2/ut-go, ci-v2/ut-cpp
/ci-rerun-e2e-arm // for ci-v2/e2e-arm [master branch only]
/ci-rerun-e2e-default // for ci-v2/e2e-default [master branch only]

If you have any questions or requests, please contact @zhikunyao.

mergify · 2025-11-25T07:23:05Z

@SpadeA-Tang go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify · 2025-11-25T07:29:07Z

@SpadeA-Tang cpu-e2e job failed, comment /run-cpu-e2e can trigger the job again.

Signed-off-by: SpadeA <[email protected]>

mergify · 2025-11-25T12:39:33Z

@SpadeA-Tang go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify · 2025-11-25T14:09:31Z

@SpadeA-Tang cpu-e2e job failed, comment /run-cpu-e2e can trigger the job again.

Signed-off-by: SpadeA <[email protected]>

mergify · 2025-11-26T04:36:19Z

@SpadeA-Tang go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify · 2025-11-26T05:23:38Z

@SpadeA-Tang cpu-e2e job failed, comment /run-cpu-e2e can trigger the job again.

Signed-off-by: SpadeA <[email protected]>

mergify · 2025-11-26T07:21:13Z

@SpadeA-Tang go-sdk check failed, comment rerun go-sdk can trigger the job again.

Signed-off-by: SpadeA <[email protected]>

mergify · 2025-11-26T08:08:03Z

@SpadeA-Tang go-sdk check failed, comment rerun go-sdk can trigger the job again.

Signed-off-by: SpadeA <[email protected]>

mergify · 2025-11-26T09:07:12Z

@SpadeA-Tang go-sdk check failed, comment rerun go-sdk can trigger the job again.

codecov · 2025-12-02T05:31:43Z

Codecov Report

❌ Patch coverage is 85.26265% with 303 lines in your changes missing coverage. Please review.
✅ Project coverage is 76.00%. Comparing base (3f063a2) to head (06b62f7).
⚠️ Report is 27 commits behind head on master.

Files with missing lines	Patch %	Lines
internal/core/src/segcore/SegmentGrowingImpl.cpp	38.67%	65 Missing ⚠️
internal/core/src/query/PlanProto.cpp	77.22%	41 Missing ⚠️
internal/proxy/search_reduce_util.go	20.00%	21 Missing and 3 partials ⚠️
internal/querynodev2/segments/search_reduce.go	20.00%	18 Missing and 6 partials ⚠️
internal/core/src/common/ArrayOffsets.cpp	86.66%	18 Missing ⚠️
internal/core/src/plan/PlanNode.h	55.00%	18 Missing ⚠️
internal/parser/planparserv2/parser_visitor.go	88.18%	10 Missing and 5 partials ⚠️
internal/proxy/task_index.go	0.00%	15 Missing ⚠️
...ernal/util/indexparamcheck/vector_index_checker.go	0.00%	15 Missing ⚠️
internal/core/src/segcore/reduce/Reduce.cpp	52.17%	11 Missing ⚠️
... and 18 more

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #45830      +/-   ##
==========================================
+ Coverage   73.29%   76.00%   +2.70%     
==========================================
  Files        1369     1905     +536     
  Lines      213893   298568   +84675     
==========================================
+ Hits       156778   226913   +70135     
- Misses      49633    64160   +14527     
- Partials     7482     7495      +13

Components	Coverage Δ
Client	`78.18% <ø> (ø)`
Core	`82.88% <88.91%> (∅)`
Go	`73.93% <56.83%> (-0.02%)`	⬇️

Files with missing lines	Coverage Δ
internal/core/src/common/ArrayOffsets.h	`100.00% <100.00%> (ø)`
internal/core/src/common/ArrayOffsetsTest.cpp	`100.00% <100.00%> (ø)`
internal/core/src/common/QueryInfo.h	`100.00% <ø> (ø)`
internal/core/src/common/QueryResult.h	`98.55% <100.00%> (ø)`
internal/core/src/common/Schema.h	`98.14% <100.00%> (ø)`
internal/core/src/common/Utils.h	`97.43% <ø> (ø)`
internal/core/src/common/VectorArray.h	`82.44% <100.00%> (ø)`
internal/core/src/exec/Driver.cpp	`83.91% <100.00%> (ø)`
internal/core/src/exec/QueryContext.h	`88.88% <100.00%> (ø)`
...ernal/core/src/exec/expression/BinaryRangeExpr.cpp	`72.96% <100.00%> (ø)`
... and 47 more

... and 526 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Signed-off-by: SpadeA <[email protected]>

internal/core/src/common/ArrayOffsets.h

internal/core/src/common/ArrayOffsets.cpp

internal/querynodev2/segments/search_reduce.go

internal/core/unittest/test_element_filter.cpp

internal/core/src/common/ArrayOffsets.h

internal/core/src/exec/expression/ColumnExpr.cpp

internal/core/src/exec/operator/ElementFilterNode.h

internal/core/src/query/SubSearchResult.h

internal/core/src/query/Plan.cpp

internal/core/src/query/SearchOnGrowing.cpp

internal/core/src/segcore/ChunkedSegmentSealedImpl.cpp

Signed-off-by: SpadeA <[email protected]>

zhengbuqian · 2025-12-03T02:32:14Z

internal/core/src/common/Schema.cpp

+const FieldMeta&
+Schema::GetFirstArrayFieldInStruct(const std::string& struct_name) const {
+    // Check cache first
+    auto cache_it = struct_array_field_cache_.find(struct_name);


I was thinking of caching it in constructor. using this method we need to worry about concurrency safty.

Schema is is inited by default constructor and add fields one by one. So it's unable to handle this in constructor. I think add a mutext for struct_array_field_cache_ is fine.

Signed-off-by: SpadeA <[email protected]>

zhengbuqian · 2025-12-03T08:38:32Z

internal/core/src/common/ArrayOffsets.h

+    BuildFromSegment(const void* segment, const FieldMeta& field_meta);
+
+ private:
+    const std::vector<int32_t> element_row_ids_;


remember to report this portion of memory usage to cachinglayer. reach out to @sparknack for details

Signed-off-by: SpadeA <[email protected]>

sparknack · 2025-12-09T07:22:13Z

For the resource management section, LGTM.

Signed-off-by: SpadeA <[email protected]>

sre-ci-robot · 2025-12-09T09:58:43Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: SpadeA-Tang, zhengbuqian

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [zhengbuqian]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Signed-off-by: SpadeA <[email protected]>

mergify · 2025-12-10T03:39:22Z

@SpadeA-Tang go-sdk check failed, comment rerun go-sdk can trigger the job again.

Signed-off-by: SpadeA <[email protected]>

mergify · 2025-12-10T04:35:40Z

@SpadeA-Tang go-sdk check failed, comment rerun go-sdk can trigger the job again.

Signed-off-by: SpadeA <[email protected]>

SpadeA-Tang · 2025-12-10T09:48:53Z

/ci-rerun-e2e-default

Signed-off-by: SpadeA <[email protected]>

SpadeA-Tang · 2025-12-15T02:21:54Z

/ci-rerun-e2e-default

zhagnlu · 2025-12-15T04:00:49Z

/lgtm

impl element filter g4

8af7490

Signed-off-by: SpadeA <[email protected]>

sre-ci-robot added area/compilation area/dependency Pull requests that update a dependency file size/XXL Denotes a PR that changes 1000+ lines. labels Nov 25, 2025

sre-ci-robot requested review from bigsheeper and congqixia November 25, 2025 07:05

mergify bot added dco-passed DCO check passed. kind/feature Issues related to feature request from users labels Nov 25, 2025

SpadeA-Tang force-pushed the element-filter branch from b8c0742 to cc5ba77 Compare November 25, 2025 09:06

segcore impl doen

3a8c0c8

Signed-off-by: SpadeA <[email protected]>

SpadeA-Tang force-pushed the element-filter branch from cc5ba77 to 3a8c0c8 Compare November 25, 2025 11:43

SpadeA-Tang added 2 commits November 25, 2025 19:49

rebase

39ff54b

Signed-off-by: SpadeA <[email protected]>

reserve space

eb04891

Signed-off-by: SpadeA <[email protected]>

SpadeA-Tang added 2 commits November 26, 2025 11:54

revert change for depricated stream reduce

769262e

Signed-off-by: SpadeA <[email protected]>

rebase

7aaad05

Signed-off-by: SpadeA <[email protected]>

update proto

7b3e467

Signed-off-by: SpadeA <[email protected]>

sre-ci-robot added area/test sig/testing labels Nov 26, 2025

format fix

2e5478f

Signed-off-by: SpadeA <[email protected]>

fix package management

9e39345

Signed-off-by: SpadeA <[email protected]>

fill value consider ElementFilter

0e81b7a

Signed-off-by: SpadeA <[email protected]>

zhengbuqian reviewed Dec 2, 2025

View reviewed changes

internal/core/src/segcore/ChunkedSegmentSealedImpl.cpp Outdated Show resolved Hide resolved

address comments

672701d

Signed-off-by: SpadeA <[email protected]>

zhengbuqian reviewed Dec 3, 2025

View reviewed changes

address comment

ca7c9c6

Signed-off-by: SpadeA <[email protected]>

zhengbuqian reviewed Dec 3, 2025

View reviewed changes

SpadeA-Tang added 2 commits December 4, 2025 10:43

resource charge

f3563ca

Signed-off-by: SpadeA <[email protected]>

more robust handle of resource charge

98df6e5

Signed-off-by: SpadeA <[email protected]>

remove lock

e2d54df

Signed-off-by: SpadeA <[email protected]>

zhengbuqian approved these changes Dec 9, 2025

View reviewed changes

sre-ci-robot added the approved label Dec 9, 2025

rebase

245a024

Signed-off-by: SpadeA <[email protected]>

SpadeA-Tang added 2 commits December 10, 2025 12:10

fix compile

0a57506

Signed-off-by: SpadeA <[email protected]>

add lock back

88ba686

Signed-off-by: SpadeA <[email protected]>

SpadeA-Tang added 2 commits December 10, 2025 15:07

fix compile issue

047cdbf

Signed-off-by: SpadeA <[email protected]>

fix test

ccc1bfb

Signed-off-by: SpadeA <[email protected]>

fix tests

06b62f7

Signed-off-by: SpadeA <[email protected]>

mergify bot added the ci-passed label Dec 15, 2025

sre-ci-robot assigned zhagnlu Dec 15, 2025

sre-ci-robot added the lgtm label Dec 15, 2025

sre-ci-robot merged commit f6f716b into milvus-io:master Dec 15, 2025
16 of 19 checks passed

Comments

Conversation

SpadeA-Tang commented Nov 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sre-ci-robot commented Nov 25, 2025

Uh oh!

mergify bot commented Nov 25, 2025

Uh oh!

mergify bot commented Nov 25, 2025

Uh oh!

mergify bot commented Nov 25, 2025

Uh oh!

mergify bot commented Nov 25, 2025

Uh oh!

mergify bot commented Nov 26, 2025

Uh oh!

mergify bot commented Nov 26, 2025

Uh oh!

mergify bot commented Nov 26, 2025

Uh oh!

mergify bot commented Nov 26, 2025

Uh oh!

mergify bot commented Nov 26, 2025

Uh oh!

codecov bot commented Dec 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zhengbuqian Dec 3, 2025

Choose a reason for hiding this comment

Uh oh!

SpadeA-Tang Dec 3, 2025

Choose a reason for hiding this comment

Uh oh!

zhengbuqian Dec 3, 2025

Choose a reason for hiding this comment

Uh oh!

sparknack commented Dec 9, 2025

Uh oh!

sre-ci-robot commented Dec 9, 2025

Uh oh!

mergify bot commented Dec 10, 2025

Uh oh!

mergify bot commented Dec 10, 2025

Uh oh!

SpadeA-Tang commented Dec 10, 2025

Uh oh!

SpadeA-Tang commented Dec 15, 2025

Uh oh!

zhagnlu commented Dec 15, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

SpadeA-Tang commented Nov 25, 2025 •

edited

Loading

codecov bot commented Dec 2, 2025 •

edited

Loading