From c855ea31710ebec69c4f9c48733905795d5f534a Mon Sep 17 00:00:00 2001 From: yah01 Date: Fri, 7 Apr 2023 18:26:29 +0800 Subject: [PATCH] MEP for search by primary keys (#23193) Signed-off-by: yah01 --- docs/design_docs/20230403-search_by_pk.md | 77 +++++++++++++++++++++++ 1 file changed, 77 insertions(+) create mode 100644 docs/design_docs/20230403-search_by_pk.md diff --git a/docs/design_docs/20230403-search_by_pk.md b/docs/design_docs/20230403-search_by_pk.md new file mode 100644 index 0000000000..f7f523bbfc --- /dev/null +++ b/docs/design_docs/20230403-search_by_pk.md @@ -0,0 +1,77 @@ +# MEP: Search By Primary Keys + +Current state: Under Discussion + +ISSUE: [[Feature]: Support to search by primary keys #23184](https://github.com/milvus-io/milvus/issues/23184) + +Keywords: Search, ANN + +Released: v2.3.0 + +## Summary + +Support to search (ANNS) by the query vectors corresponding to the given primary keys. + +## Motivation + +For now, Milvus requires passing the query vectors to do anns, we have to fetch the vectors first if they are in the collection, which is complex and slow. + +We need a way to do anns directly for the corresponding vectors of given primary keys, which should be more efficient. + +## Public Interfaces + +Add new field `primary_keys` in `SearchRequest`, all SDKs adds new method: +```golang +func SearchByPK( + ctx context.Context, + collName string, + partitions []string, + expr string, + outputFields []string, + primaryKeys []entity.PrimaryKey, + vectorField string, + metricType entity.MetricType, + topK int, + sp entity.SearchParam, + opts ...SearchQueryOptionFunc, +) ([]SearchResult, error) +``` + +## Design Details + +Proxy fetches the vectors by primary keys from QueryNodes first, and then search with these vectors. + +For better performance, we will add a new RPC interface for QueryNode: +```proto +rpc Fetch(FetchRequest) returns (FetchResponse) {} +``` +which only supports to fetch by primary keys, this will be more efficient than the present `Query` interface. + + +## Compatibility, Deprecation, and Migration Plan + +None + +## Test Plan + +### Unit Tests + +- Test for fetching in segcore +- Test for fetching in QueryNode +- Test for searching by primary keys in Proxy + + +### E2E Tests +| Test Cases | Expected Behavior | +| :------------------------------------------: | :--------------------------------: | +| search by non-existed primary keys | report error | +| search by existed primary keys | return topk results for each query | +| all present tests cases of search by vectors | the same | + +## Rejected Alternatives + +Implement this without adding the `Fetch` interface, using `Query` to retrieve the vectors and then search by vectors. + +## References + +None