admin/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2025-12-07 17:48:29 +08:00
Code Issues Packages Projects Releases Wiki Activity
23,410 Commits 133 Branches 183 Tags
Commit Graph

42 Commits

Author SHA1 Message Date
wei liu
9dddc380a4
fix: Handle empty FieldsData in reduce/rerank for requery scenario (#44917) 
issue: #44909

When requery optimization is enabled, search results contain IDs
but empty FieldsData. During reduce/rerank operations, if the
first shard has empty FieldsData while others have data,
PrepareResultFieldData initializes an empty array, causing
AppendFieldData to panic when accessing array indices.

Changes:
- Find first non-empty FieldsData as template in 3 functions:
  reduceAdvanceGroupBy, reduceSearchResultDataWithGroupBy,
  reduceSearchResultDataNoGroupBy
- Add length check before 2 AppendFieldData calls in reduce
  functions to prevent panic
- Improve newRerankOutputs to find first non-empty fieldData
  using len(FieldsData) check instead of GetSizeOfIDs
- Add length check in appendResult before AppendFieldData
- Add comprehensive unit tests for empty and partial empty
  FieldsData scenarios in both reduce and rerank functions

This fix handles both pure requery (all empty) and mixed
scenarios (some empty, some with data) without breaking normal
search flow. The key improvement is checking FieldsData length
directly rather than IDs, as requery may have IDs but empty
FieldsData.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2025-10-21 19:42:03 +08:00
PjJinchen
d8efe8a6fb
feat: support ali qwen rerank (#44363) (#44364) 
issue: #44363

---------

Signed-off-by: PjJinchen <6268414+pj1987111@users.noreply.github.com>
 2025-09-30 23:19:52 +08:00
aoiasd
294282f1d2
enhance: support use nullable field as bm25 function input field (#44586) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2025-09-29 10:25:05 +08:00
congqixia
99598ae5ec
enhance: Add param item for hybrid search requery policy (#44466) 
Related to #39757

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2025-09-24 17:32:04 +08:00
junjiejiangjjj
f07979f91d
enhance: add support for controlling function output field insertion (#44162) 
#44053

Signed-off-by: junjie.jiang <junjie.jiang@zilliz.com>
 2025-09-24 17:26:04 +08:00
junjiejiangjjj
71563d5d0e
enhance: optimize decay function with configurable score merging and … (#44066) 
…normalization

- Add configurable score merge functions (max, avg, sum) for decay
reranking
- Introduce norm_score parameter to control score normalization behavior
- Refactor score normalization logic into reusable utility functions

#44051

Signed-off-by: junjie.jiang <junjie.jiang@zilliz.com>
 2025-09-23 14:18:06 +08:00
aoiasd
f98dd84dec
fix: add utf8 check before bm25 functoin run (#44220) 
relate: https://github.com/milvus-io/milvus/issues/44219

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2025-09-08 14:43:56 +08:00
aoiasd
208a345a3d
enhance: package analyzer code in Go and fix named analyzer as tokenizer (#43694) 
relate: https://github.com/milvus-io/milvus/issues/43687

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2025-08-27 10:59:52 +08:00
Tianx
c0d62268ac
feat: add timesatmptz data type (#44005) 
issue: https://github.com/milvus-io/milvus/issues/27467
>
https://github.com/milvus-io/milvus/issues/27467#issuecomment-3092211420
> * [x]  M1 Create collection with timestamptz field
> * [x]  M2 Insert timestamptz field data
> * [x]  M3 Retrieve timestamptz field data
> * [x]  M4 Implement handoff[ ]  

The second PR of issue:
https://github.com/milvus-io/milvus/issues/27467, which completes M1-M4
described above.

---------

Signed-off-by: xtx <xtianx@smail.nju.edu.cn>
 2025-08-26 15:59:53 +08:00
junjiejiangjjj
f1ce84996d
enhance: refactor model service configuration and environment variables (#44036) 
- Add enable configuration for all model service providers
- Migrate environment variables from MILVUSAI_* to MILVUS_* prefix with
backward compatibility
- Unify model service enable/disable logic using configuration
- Add tests for environment variable parsing with fallback support

#35856

Signed-off-by: junjie.jiang <junjie.jiang@zilliz.com>
 2025-08-26 10:49:52 +08:00
junjiejiangjjj
f3d7e47227
feat: Supports more rerankers (#43270) 
https://github.com/milvus-io/milvus/issues/35856

Signed-off-by: junjiejiangjjj <junjie.jiang@zilliz.com>
 2025-08-22 17:29:47 +08:00
aoiasd
dcf04a58b8
feat: support use score function on segment search and use filter (#43868) 
relate: https://github.com/milvus-io/milvus/issues/43867
Support boost function score, multiply by the weight if match filter.

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2025-08-19 16:15:45 +08:00
junjiejiangjjj
4db877f76c
fix: Fix weighted rerank (#43503) 
#43478

Signed-off-by: junjie.jiang <junjie.jiang@zilliz.com>
 2025-07-23 14:54:53 +08:00
junjiejiangjjj
77f3a1f213
enhance: Add search post pipeline (#43065) 
https://github.com/milvus-io/milvus/issues/35856

Signed-off-by: junjiejiangjjj <junjie.jiang@zilliz.com>
 2025-07-21 11:10:52 +08:00
junjiejiangjjj
9865d672f7
fix: Model rerank supports Truncate (#42643) 
https://github.com/milvus-io/milvus/issues/42632

Signed-off-by: junjie.jiang <junjie.jiang@zilliz.com>
 2025-06-19 15:02:41 +08:00
junjiejiangjjj
f1a4526bac
enhance: refactor rrf and weighted rerank (#42154) 
https://github.com/milvus-io/milvus/issues/35856

Signed-off-by: junjie.jiang <junjie.jiang@zilliz.com>
 2025-06-10 18:08:35 +08:00
aoiasd
13330bd466
fix: add concurrency and close protect for bm25 function (#42597) 
relate: https://github.com/milvus-io/milvus/issues/42576

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2025-06-10 11:36:34 +08:00
aoiasd
2eb24fbe7c
fix: analyzer memory leak because function runner not close (#41839) 
relate: https://github.com/milvus-io/milvus/issues/41213

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2025-06-05 14:24:40 +08:00
aoiasd
2ae4d80120
enhance: support run analyzer by loaded collection field (#42113) 
relate: https://github.com/milvus-io/milvus/issues/42094

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2025-05-29 10:54:30 +08:00
junjiejiangjjj
4202c775ba
feat: Support vllm and tei rerank (#41947) 
https://github.com/milvus-io/milvus/issues/35856

Signed-off-by: junjie.jiang <junjie.jiang@zilliz.com>
 2025-05-28 19:18:28 +08:00
junjiejiangjjj
0bbbf98a5b
enhance: refactor decay rerank (#41778) 
https://github.com/milvus-io/milvus/issues/35856
1. Optimizing decay function
2. Since the decay function is larger, the more similar it is， the
smaller the L2/JACCARD/HAMMING metrics scores the more similar they are.
For these metrics, the decay function regenerates new scores.

Signed-off-by: junjie.jiang <junjie.jiang@zilliz.com>
 2025-05-20 15:22:24 +08:00
junjiejiangjjj
f337d2989b
enhance : New decay function (#41634) 
https://github.com/milvus-io/milvus/issues/35856

Signed-off-by: junjie.jiang <junjie.jiang@zilliz.com>
 2025-05-08 17:24:54 +08:00
junjiejiangjjj
bb7df40fc1
feat: Http interface supports rerank (#41486) 
https://github.com/milvus-io/milvus/issues/35856

Signed-off-by: junjie.jiang <junjie.jiang@zilliz.com>
 2025-04-28 23:02:50 +08:00
junjiejiangjjj
e56adc121b
enhance: refactor embedding credentials manager (#41442) 
https://github.com/milvus-io/milvus/issues/35856

Signed-off-by: junjie.jiang <junjie.jiang@zilliz.com>
 2025-04-24 14:34:38 +08:00
junjiejiangjjj
f23df95a77
feat : Support decay rerank (#41223) 
https://github.com/milvus-io/milvus/issues/35856
#41312

Signed-off-by: junjie.jiang <junjie.jiang@zilliz.com>
 2025-04-23 20:48:39 +08:00
aoiasd
f52c2909c4
feat: support multi analyzer for bm25 function (#41351) 
relate: https://github.com/milvus-io/milvus/issues/41213

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2025-04-23 18:22:38 +08:00
SimFG
91d40fa558
fix: Update logging context and upgrade dependencies (#41318) 
- issue: #41291

---------

Signed-off-by: SimFG <bang.fu@zilliz.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 2025-04-23 10:52:38 +08:00
junjiejiangjjj
fe81c7baae
feat: Add function config (#40534) 
#35856 
1. Add function-related configuration in milvus.yaml
2. Add null and empty value check to TextEmbeddingFunction

Signed-off-by: junjie.jiang <junjie.jiang@zilliz.com>
 2025-03-25 10:06:24 +08:00
junjiejiangjjj
359e7efd8e
feat: Add function running monitoring (#40358) 
#35856 
#40004 
1. Optimize model verification logic
2. Add profiling code

Signed-off-by: junjie.jiang <junjie.jiang@zilliz.com>
 2025-03-10 22:28:05 +08:00
junjiejiangjjj
b2e630b1a1
feat: Support TEI serving and support int8 embedding (#40199) 
#35856

Signed-off-by: junjie.jiang <junjie.jiang@zilliz.com>
 2025-03-05 10:18:00 +08:00
junjiejiangjjj
162d241063
feat: Add siliconflow text embedding (#39867) 
https://github.com/milvus-io/milvus/issues/35856

Signed-off-by: junjie.jiang <junjie.jiang@zilliz.com>
 2025-02-26 11:01:56 +08:00
congqixia
cb7f2fa6fd
enhance: Use v2 package name for pkg module (#39990) 
Related to #39095

https://go.dev/doc/modules/version-numbers

Update pkg version according to golang dep version convention

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2025-02-22 23:15:58 +08:00
Patrick Weizhi Xu
04fff74a56
feat: introduce Text data type (#39874) 
issue: https://github.com/milvus-io/milvus/issues/39818

This PR mimics Varchar data type, allows insert, search, query, delete,
full-text search and others.
Functionalities related to filter expressions are disabled temporarily. 

Storage changes for Text data type will be in the following PRs.

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
 2025-02-19 11:04:51 +08:00
junjiejiangjjj
09b9131321
feat: Add cohere text embedding (#39581) 
#35856

Signed-off-by: junjie.jiang <junjie.jiang@zilliz.com>
 2025-02-13 18:00:16 +08:00
junjiejiangjjj
16cbdfb3b1
feat: Add Text Embedding Function (#36366) 
https://github.com/milvus-io/milvus/issues/35856

Signed-off-by: junjie.jiang <junjie.jiang@zilliz.com>
 2025-01-24 14:23:06 +08:00
aoiasd
f16232f834
fix: sparse inc mast less than uint32 max (#38250) 
relate: https://github.com/milvus-io/milvus/issues/35853

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2024-12-06 15:14:41 +08:00
Buqian Zheng
00edec2ebd
fix: incorrect dim set when creating BM25 doc sparse array (#37717) 
issue: https://github.com/milvus-io/milvus/issues/35853

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
 2024-11-16 16:36:30 +08:00
aoiasd
1c5b5e1e3d
feat: Add chinese and english analyzer with refactor jieba tokenizer (#37494) 
relate: https://github.com/milvus-io/milvus/issues/35853

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2024-11-14 10:34:31 +08:00
aoiasd
12951f0abb
enhance: rename tokenizer to analyzer and check analyzer params (#37478) 
relate: https://github.com/milvus-io/milvus/issues/35853

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2024-11-10 16:12:26 +08:00
aoiasd
d67853fa89
feat: Tokenizer support build with params and clone for concurrency (#37048) 
relate: https://github.com/milvus-io/milvus/issues/35853
https://github.com/milvus-io/milvus/issues/36751

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2024-11-06 17:48:24 +08:00
aoiasd
db34572c56
feat: support load and query with bm25 metric (#36071) 
relate: https://github.com/milvus-io/milvus/issues/35853

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2024-10-11 10:23:20 +08:00
aoiasd
139787371e
feat: support embedding bm25 sparse vector and flush bm25 stats log (#36036) 
relate: https://github.com/milvus-io/milvus/issues/35853

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2024-09-19 10:57:12 +08:00