36 Commits

Author SHA1 Message Date
aoiasd
f98dd84dec
fix: add utf8 check before bm25 functoin run (#44220)
relate: https://github.com/milvus-io/milvus/issues/44219

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2025-09-08 14:43:56 +08:00
aoiasd
208a345a3d
enhance: package analyzer code in Go and fix named analyzer as tokenizer (#43694)
relate: https://github.com/milvus-io/milvus/issues/43687

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2025-08-27 10:59:52 +08:00
Tianx
c0d62268ac
feat: add timesatmptz data type (#44005)
issue: https://github.com/milvus-io/milvus/issues/27467
>
https://github.com/milvus-io/milvus/issues/27467#issuecomment-3092211420
> * [x]  M1 Create collection with timestamptz field
> * [x]  M2 Insert timestamptz field data
> * [x]  M3 Retrieve timestamptz field data
> * [x]  M4 Implement handoff[ ]  

The second PR of issue:
https://github.com/milvus-io/milvus/issues/27467, which completes M1-M4
described above.

---------

Signed-off-by: xtx <xtianx@smail.nju.edu.cn>
2025-08-26 15:59:53 +08:00
junjiejiangjjj
f1ce84996d
enhance: refactor model service configuration and environment variables (#44036)
- Add enable configuration for all model service providers
- Migrate environment variables from MILVUSAI_* to MILVUS_* prefix with
backward compatibility
- Unify model service enable/disable logic using configuration
- Add tests for environment variable parsing with fallback support

#35856

Signed-off-by: junjie.jiang <junjie.jiang@zilliz.com>
2025-08-26 10:49:52 +08:00
junjiejiangjjj
f3d7e47227
feat: Supports more rerankers (#43270)
https://github.com/milvus-io/milvus/issues/35856

Signed-off-by: junjiejiangjjj <junjie.jiang@zilliz.com>
2025-08-22 17:29:47 +08:00
aoiasd
dcf04a58b8
feat: support use score function on segment search and use filter (#43868)
relate: https://github.com/milvus-io/milvus/issues/43867
Support boost function score, multiply by the weight if match filter.

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2025-08-19 16:15:45 +08:00
junjiejiangjjj
4db877f76c
fix: Fix weighted rerank (#43503)
#43478

Signed-off-by: junjie.jiang <junjie.jiang@zilliz.com>
2025-07-23 14:54:53 +08:00
junjiejiangjjj
77f3a1f213
enhance: Add search post pipeline (#43065)
https://github.com/milvus-io/milvus/issues/35856

Signed-off-by: junjiejiangjjj <junjie.jiang@zilliz.com>
2025-07-21 11:10:52 +08:00
junjiejiangjjj
9865d672f7
fix: Model rerank supports Truncate (#42643)
https://github.com/milvus-io/milvus/issues/42632

Signed-off-by: junjie.jiang <junjie.jiang@zilliz.com>
2025-06-19 15:02:41 +08:00
junjiejiangjjj
f1a4526bac
enhance: refactor rrf and weighted rerank (#42154)
https://github.com/milvus-io/milvus/issues/35856

Signed-off-by: junjie.jiang <junjie.jiang@zilliz.com>
2025-06-10 18:08:35 +08:00
aoiasd
13330bd466
fix: add concurrency and close protect for bm25 function (#42597)
relate: https://github.com/milvus-io/milvus/issues/42576

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2025-06-10 11:36:34 +08:00
aoiasd
2eb24fbe7c
fix: analyzer memory leak because function runner not close (#41839)
relate: https://github.com/milvus-io/milvus/issues/41213

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2025-06-05 14:24:40 +08:00
aoiasd
2ae4d80120
enhance: support run analyzer by loaded collection field (#42113)
relate: https://github.com/milvus-io/milvus/issues/42094

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2025-05-29 10:54:30 +08:00
junjiejiangjjj
4202c775ba
feat: Support vllm and tei rerank (#41947)
https://github.com/milvus-io/milvus/issues/35856

Signed-off-by: junjie.jiang <junjie.jiang@zilliz.com>
2025-05-28 19:18:28 +08:00
junjiejiangjjj
0bbbf98a5b
enhance: refactor decay rerank (#41778)
https://github.com/milvus-io/milvus/issues/35856
1. Optimizing decay function
2. Since the decay function is larger, the more similar it is, the
smaller the L2/JACCARD/HAMMING metrics scores the more similar they are.
For these metrics, the decay function regenerates new scores.

Signed-off-by: junjie.jiang <junjie.jiang@zilliz.com>
2025-05-20 15:22:24 +08:00
junjiejiangjjj
f337d2989b
enhance : New decay function (#41634)
https://github.com/milvus-io/milvus/issues/35856

Signed-off-by: junjie.jiang <junjie.jiang@zilliz.com>
2025-05-08 17:24:54 +08:00
junjiejiangjjj
bb7df40fc1
feat: Http interface supports rerank (#41486)
https://github.com/milvus-io/milvus/issues/35856

Signed-off-by: junjie.jiang <junjie.jiang@zilliz.com>
2025-04-28 23:02:50 +08:00
junjiejiangjjj
e56adc121b
enhance: refactor embedding credentials manager (#41442)
https://github.com/milvus-io/milvus/issues/35856

Signed-off-by: junjie.jiang <junjie.jiang@zilliz.com>
2025-04-24 14:34:38 +08:00
junjiejiangjjj
f23df95a77
feat : Support decay rerank (#41223)
https://github.com/milvus-io/milvus/issues/35856
#41312

Signed-off-by: junjie.jiang <junjie.jiang@zilliz.com>
2025-04-23 20:48:39 +08:00
aoiasd
f52c2909c4
feat: support multi analyzer for bm25 function (#41351)
relate: https://github.com/milvus-io/milvus/issues/41213

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2025-04-23 18:22:38 +08:00
SimFG
91d40fa558
fix: Update logging context and upgrade dependencies (#41318)
- issue: #41291

---------

Signed-off-by: SimFG <bang.fu@zilliz.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2025-04-23 10:52:38 +08:00
junjiejiangjjj
fe81c7baae
feat: Add function config (#40534)
#35856 
1. Add function-related configuration in milvus.yaml
2. Add null and empty value check to TextEmbeddingFunction

Signed-off-by: junjie.jiang <junjie.jiang@zilliz.com>
2025-03-25 10:06:24 +08:00
junjiejiangjjj
359e7efd8e
feat: Add function running monitoring (#40358)
#35856 
#40004 
1. Optimize model verification logic
2. Add profiling code

Signed-off-by: junjie.jiang <junjie.jiang@zilliz.com>
2025-03-10 22:28:05 +08:00
junjiejiangjjj
b2e630b1a1
feat: Support TEI serving and support int8 embedding (#40199)
#35856

Signed-off-by: junjie.jiang <junjie.jiang@zilliz.com>
2025-03-05 10:18:00 +08:00
junjiejiangjjj
162d241063
feat: Add siliconflow text embedding (#39867)
https://github.com/milvus-io/milvus/issues/35856

Signed-off-by: junjie.jiang <junjie.jiang@zilliz.com>
2025-02-26 11:01:56 +08:00
congqixia
cb7f2fa6fd
enhance: Use v2 package name for pkg module (#39990)
Related to #39095

https://go.dev/doc/modules/version-numbers

Update pkg version according to golang dep version convention

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-02-22 23:15:58 +08:00
Patrick Weizhi Xu
04fff74a56
feat: introduce Text data type (#39874)
issue: https://github.com/milvus-io/milvus/issues/39818

This PR mimics Varchar data type, allows insert, search, query, delete,
full-text search and others.
Functionalities related to filter expressions are disabled temporarily. 

Storage changes for Text data type will be in the following PRs.

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
2025-02-19 11:04:51 +08:00
junjiejiangjjj
09b9131321
feat: Add cohere text embedding (#39581)
#35856

Signed-off-by: junjie.jiang <junjie.jiang@zilliz.com>
2025-02-13 18:00:16 +08:00
junjiejiangjjj
16cbdfb3b1
feat: Add Text Embedding Function (#36366)
https://github.com/milvus-io/milvus/issues/35856

Signed-off-by: junjie.jiang <junjie.jiang@zilliz.com>
2025-01-24 14:23:06 +08:00
aoiasd
f16232f834
fix: sparse inc mast less than uint32 max (#38250)
relate: https://github.com/milvus-io/milvus/issues/35853

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-12-06 15:14:41 +08:00
Buqian Zheng
00edec2ebd
fix: incorrect dim set when creating BM25 doc sparse array (#37717)
issue: https://github.com/milvus-io/milvus/issues/35853

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-11-16 16:36:30 +08:00
aoiasd
1c5b5e1e3d
feat: Add chinese and english analyzer with refactor jieba tokenizer (#37494)
relate: https://github.com/milvus-io/milvus/issues/35853

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-11-14 10:34:31 +08:00
aoiasd
12951f0abb
enhance: rename tokenizer to analyzer and check analyzer params (#37478)
relate: https://github.com/milvus-io/milvus/issues/35853

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-11-10 16:12:26 +08:00
aoiasd
d67853fa89
feat: Tokenizer support build with params and clone for concurrency (#37048)
relate: https://github.com/milvus-io/milvus/issues/35853
https://github.com/milvus-io/milvus/issues/36751

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-11-06 17:48:24 +08:00
aoiasd
db34572c56
feat: support load and query with bm25 metric (#36071)
relate: https://github.com/milvus-io/milvus/issues/35853

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-10-11 10:23:20 +08:00
aoiasd
139787371e
feat: support embedding bm25 sparse vector and flush bm25 stats log (#36036)
relate: https://github.com/milvus-io/milvus/issues/35853

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-09-19 10:57:12 +08:00