admin/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2026-01-07 19:31:51 +08:00
Code Issues Packages Projects Releases Wiki Activity
22,784 Commits 137 Branches 188 Tags
Commit Graph

10870 Commits

Author SHA1 Message Date
yihao.dai
f65e6b7c6e
enhance: Optimize datacoord meta mutex (#40552) 
Use a separate collection mutex.

issue: https://github.com/milvus-io/milvus/issues/40551

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2025-03-25 13:46:25 +08:00
zhagnlu
7fdb2e144f
enhance:change multi or expr to in expr (#40757) 
#40752

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2025-03-25 11:06:18 +08:00
Chun Han
eee68e9139
fix: mmap properties failed to apply when creating collection(#40511) (#40512) 
related: #40511

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
 2025-03-25 10:08:19 +08:00
junjiejiangjjj
fe81c7baae
feat: Add function config (#40534) 
#35856 
1. Add function-related configuration in milvus.yaml
2. Add null and empty value check to TextEmbeddingFunction

Signed-off-by: junjie.jiang <junjie.jiang@zilliz.com>
 2025-03-25 10:06:24 +08:00
SimFG
16efcda5c4
feat: support to deny dll according to database property (#40764) 
- issue: #40762

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2025-03-23 11:18:28 +08:00
congqixia
1953676ee0
fix: Set task queue time before enqueue (#40815) 
Related to #40814

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2025-03-21 19:36:15 +08:00
Ted Xu
688505ab1c
enhance: cleanup lint check exclusions (#40829) 
See: #40828

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
 2025-03-21 18:12:14 +08:00
Bingyi Sun
ecea17c576
enhance: Set field name as json path if not specified (#40419) 
issue: #35528

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2025-03-21 11:26:13 +08:00
cai.zhang
a41cb942f6
fix: Do not delete the centroids file when sampling fails instead wait GC (#40701) 
issue: #40700

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2025-03-21 10:32:12 +08:00
congqixia
9824615efb
enhance: Close component in topological order when unsub channel (#40796) 
Related to #40795

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2025-03-21 10:22:11 +08:00
sthuang
d7df78a6c9
feat: Storage v2 compaction (#40667) 
- Feat: Support Mix compaction. Covering tests include compatibility and
rollback ability.
  - Read v1 segments and compact with v2 format.
  - Read both v1 and v2 segments and compact with v2 format.
  - Read v2 segments and compact with v2 format.
  - Compact with duplicate primary key test.
  - Compact with bm25 segments.
  - Compact with merge sort segments.
  - Compact with no expiration segments.
  - Compact with lack binlog segments.
  - Compact with nullable field segments.
- Feat: Support Clustering compaction. Covering tests include
compatibility and rollback ability.
  - Read v1 segments and compact with v2 format.
  - Read both v1 and v2 segments and compact with v2 format.
  - Read v2 segments and compact with v2 format.
  - Compact bm25 segments with v2 format.
  - Compact with memory limit.
- Enhance: Use serdeMap serialize in BuildRecord function to support all
Milvus data types.
related: #39173

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
 2025-03-21 10:16:12 +08:00
congqixia
77515310b9
enhance: Resolve existing lint issues (#40797) 
This PR
- Format import issues in datacoord & datanode
- Fix JSONPathIndex naming issue

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2025-03-20 19:36:11 +08:00
congqixia
e23f429299
fix: Check whether index type support mmap even param enabled (#40614) 
Related to #40537

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2025-03-20 15:52:13 +08:00
Zhen Ye
4dc7c11bbe
fix: remove dead code at broadcaster (#40788) 
issue: #38399

Signed-off-by: chyezh <chyezh@outlook.com>
 2025-03-20 14:20:59 +08:00
congqixia
096f58f314
enhance: Add an extra parameter for slow log (#40758) 
Related to #40756

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2025-03-20 00:34:13 +08:00
Bingyi Sun
5a6b4e56d5
fix: Fix tasks will panic if one of them throw an exception. (#40691) 
issue: https://github.com/milvus-io/milvus/issues/40690

the variable rcm will be dangling if a future throws an exception and
return.

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2025-03-19 16:52:09 +08:00
aoiasd
92bdf7a0c1
enhance: support run anayser return detaild token (#40458) 
relate: https://github.com/milvus-io/milvus/issues/39705

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2025-03-19 15:48:15 +08:00
zhagnlu
6c55db44f1
enhance: reorder sub expr for conjunct expr (#39872) 
two point:
 (1) reoder conjucts expr's subexpr, postpone heavy operations
sequence: int(column) -> index(column) -> string(column) -> light
conjuct
...... -> json(column) -> heavy conjuct -> two_column_compare
(2) support pre filter for expr execute, skip scan raw data that had
been skipped
     because of preceding expr result.

#39869

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2025-03-19 14:50:14 +08:00
Zhen Ye
8db708f67d
enhance: enable memory prof based on jemalloc (#40731) 
issue: #40730

also see: https://github.com/milvus-io/cgosymbolizer/pull/2

After these PR, at linux:

- the milvus will always enable jemalloc by default.
- jemalloc will always compiled with --enable-prof options.
- all image will always enable the jemalloc prof by default.
- a pprof http service for jemalloc at `/debug/jemalloc/` will be
registered into restful.
- `jeprof` can remote profile the memory of milvus.

Signed-off-by: chyezh <chyezh@outlook.com>
 2025-03-19 14:46:18 +08:00
Zhen Ye
b119ac5d30
enhance: add wal access mode options (#40617) 
issue: #40532

Signed-off-by: chyezh <chyezh@outlook.com>
 2025-03-19 14:02:11 +08:00
Buqian Zheng
03b63bf982
fix: use NewInsertDataWithFunctionOutputField when importing binlog file (#40741) 
issue: https://github.com/milvus-io/milvus/issues/40740

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
 2025-03-19 10:50:14 +08:00
aoiasd
e5c12421f7
enhance: remain raw token for audit log (#40637) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2025-03-18 18:02:20 +08:00
Buqian Zheng
c12abf4e2a
enhance: improve sparse query nnz metric (#40713) 
add query type and field id label; add metric for hybrid search

issue: https://github.com/milvus-io/milvus/issues/35853

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
 2025-03-18 17:20:16 +08:00
congqixia
4ca1a7a76d
fix: Protect getSegmentIndexes with rlock (#40719) 
Related to #40718

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2025-03-18 16:40:14 +08:00
XuanYang-cn
015a8f7631
fix: L0 brings its own start pos when syncing (#40663) 
See also: #40388, #40207

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2025-03-18 15:56:16 +08:00
Ted Xu
be86d31ea3
feat: compaction to support add field (#40415) 
See: #39718

---------

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
 2025-03-18 11:32:12 +08:00
aoiasd
f943a8c78b
enhance: accesslog support fetch hybrid search expr and field (#40167) 
relate: https://github.com/milvus-io/milvus/issues/40166

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2025-03-17 18:08:10 +08:00
congqixia
94a859c028
enhance: Add buffer forwarder for stream delta loading (#40559) 
See also #40558
Related to #35303 & #38066 as well

This PR:
- Add `BufferedForward` to limit memory usage forwarding stream delete
- Add `UseLoad` flag to determine `Delete` shall use `segment.Delete` or
`segment.LoadDelta`
- Fix delegator accidentally use always true candidate while load
streaming delta

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2025-03-17 15:24:10 +08:00
zhagnlu
7ebe3d7038
enhance: refine chunk access logic and add some comment on data (#40618) 
#40367

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2025-03-16 22:20:08 +08:00
cai.zhang
123b6588b6
feat: Support get segment binlogs info with new interface GetSegmentsInfo (#40464) 
issue: #40341

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2025-03-16 21:04:07 +08:00
groot
9fbfcda48e
fix: Fix a crash issue of bulkinsert (#40331) 
issue: https://github.com/milvus-io/milvus/issues/40291
pr: https://github.com/milvus-io/milvus/pull/40304

Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2025-03-14 18:14:07 +08:00
Bingyi Sun
6249335859
fix: Catch invalid json pointer error (#40625) 
issue: #35528

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2025-03-14 16:56:08 +08:00
Bingyi Sun
d3adab15ac
fix: Build double index for all json numeric field (#40619) 
issue: #35528

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2025-03-14 16:52:11 +08:00
Bingyi Sun
8f66a0432d
fix: json path should be empty string if it's root (#40611) 
issue: #35528

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2025-03-14 16:50:16 +08:00
Bingyi Sun
a7cff3873b
fix: Verify json_cast_type (#40604) 
issue: https://github.com/milvus-io/milvus/issues/40420

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2025-03-14 16:48:26 +08:00
zhagnlu
8f85acc944
Revert "fix:reset default auto index type for scalar (#37086)" (#39819) 
This reverts commit eeb67a38450732eb8ab3a4eb4aaac63401426627.
#32900

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2025-03-14 16:16:11 +08:00
Bingyi Sun
8fbacf3583
fix: Null expr does not work for json field (#40456) 
issue: https://github.com/milvus-io/milvus/issues/40455

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2025-03-14 16:06:08 +08:00
Spade A
001fc992df
enhance: get doc ids by batch (#40608) 
issue: #40607

tantivy change: https://github.com/zilliztech/tantivy/pull/3

Benchmarks:
Test Envrioment: CPU 9900K
The data is insert by:
```
for i in 0..N {
    for j in 0..UNIQUE {
        let key = format!("hello{}", j);
        index_writer.add_string(&key, i * UNIQUE + j).unwrap();
    }
}
```
So the unique influences the locality of the matched docs.
The latency is the avg latency over 1000 repeate quries.
The result shows 22.5%-34.8% latency reduction.

![image](https://github.com/user-attachments/assets/dd8af75a-ddc3-445d-92df-50d354dd5645)

---------

Signed-off-by: SpadeA <tangchenjie1210@gmail.com>
 2025-03-14 15:48:09 +08:00
cai.zhang
6dbe5d475e
enhance: Refine task meta with key lock (#40613) 
issue: #39101

2.5 pr: #40146 #40353

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2025-03-14 15:44:22 +08:00
SimFG
bf4fc6a8c6
feat: add DDLDB rate type and related quota configurations (#40651) 
- issue: #40650

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2025-03-14 15:38:09 +08:00
Spade A
f36d1562bd
enhance: add metrics for random sample (#40634) 
issue: #39541

Signed-off-by: SpadeA <tangchenjie1210@gmail.com>
 2025-03-13 21:42:11 +08:00
yihao.dai
bab30a41bf
enhance: Improve import error msgs (#40567) 
issue: https://github.com/milvus-io/milvus/issues/40208

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2025-03-13 21:02:07 +08:00
Zhen Ye
f6fb4bc442
fix: backoff will retry infinitely after reaching max elapse (#40589) 
issue: #40588

Signed-off-by: chyezh <chyezh@outlook.com>
 2025-03-13 16:24:06 +08:00
yihao.dai
b2a8694686
enhance: Merge IndexNode and DataNode (#40272) 
Merge DataNode and IndexNode into DataNode.

issue: https://github.com/milvus-io/milvus/issues/39115

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2025-03-13 14:26:11 +08:00
Ted Xu
df4285c9ef
enhance: API integration with storage v2 in clustering-compactions (#40133) 
See #39173

---------

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
 2025-03-13 14:12:06 +08:00
Zhen Ye
5735c3ef19
fix: too many memory usage of streaming node (#40606) 
issue: #40592

Signed-off-by: chyezh <chyezh@outlook.com>
 2025-03-13 07:10:07 +08:00
Xiaofan
fb48b3c7ac
fix: empty sparse row in importer (#40585) 
fix #40584

parquet bulk writer can not finish 0 dim sparse vector.

Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
 2025-03-13 01:29:41 +08:00
Spade A
9f3bd55755
fix: avoid panic when field not exists in schema in query node (#40541) 
ref #40473

This PR is a workaround to avoid the panic described in the issue.

---------

Signed-off-by: SpadeA <tangchenjie1210@gmail.com>
 2025-03-12 22:44:08 +08:00
jaime
c8a96377bb
enhance: move object storage client creation to pkg package (#40440) 
issue: #40439

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2025-03-12 20:38:07 +08:00
yihao.dai
27c7cbbc72
fix: Fix QueryNodeNumEntities metric (#40602) 
fix QueryNodeNumEntities metric introduced by pr
https://github.com/milvus-io/milvus/pull/39536

issue: https://github.com/milvus-io/milvus/issues/38162

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2025-03-12 19:08:05 +08:00