admin/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2026-01-04 09:52:30 +08:00
Code Issues Packages Projects Releases Wiki Activity
22,018 Commits 138 Branches 188 Tags
Commit Graph

10305 Commits

Author SHA1 Message Date
Bingyi Sun
5a6b4e56d5
fix: Fix tasks will panic if one of them throw an exception. (#40691) 
issue: https://github.com/milvus-io/milvus/issues/40690

the variable rcm will be dangling if a future throws an exception and
return.

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2025-03-19 16:52:09 +08:00
aoiasd
92bdf7a0c1
enhance: support run anayser return detaild token (#40458) 
relate: https://github.com/milvus-io/milvus/issues/39705

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2025-03-19 15:48:15 +08:00
zhagnlu
6c55db44f1
enhance: reorder sub expr for conjunct expr (#39872) 
two point:
 (1) reoder conjucts expr's subexpr, postpone heavy operations
sequence: int(column) -> index(column) -> string(column) -> light
conjuct
...... -> json(column) -> heavy conjuct -> two_column_compare
(2) support pre filter for expr execute, skip scan raw data that had
been skipped
     because of preceding expr result.

#39869

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2025-03-19 14:50:14 +08:00
Zhen Ye
8db708f67d
enhance: enable memory prof based on jemalloc (#40731) 
issue: #40730

also see: https://github.com/milvus-io/cgosymbolizer/pull/2

After these PR, at linux:

- the milvus will always enable jemalloc by default.
- jemalloc will always compiled with --enable-prof options.
- all image will always enable the jemalloc prof by default.
- a pprof http service for jemalloc at `/debug/jemalloc/` will be
registered into restful.
- `jeprof` can remote profile the memory of milvus.

Signed-off-by: chyezh <chyezh@outlook.com>
 2025-03-19 14:46:18 +08:00
Zhen Ye
b119ac5d30
enhance: add wal access mode options (#40617) 
issue: #40532

Signed-off-by: chyezh <chyezh@outlook.com>
 2025-03-19 14:02:11 +08:00
Buqian Zheng
03b63bf982
fix: use NewInsertDataWithFunctionOutputField when importing binlog file (#40741) 
issue: https://github.com/milvus-io/milvus/issues/40740

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
 2025-03-19 10:50:14 +08:00
aoiasd
e5c12421f7
enhance: remain raw token for audit log (#40637) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2025-03-18 18:02:20 +08:00
Buqian Zheng
c12abf4e2a
enhance: improve sparse query nnz metric (#40713) 
add query type and field id label; add metric for hybrid search

issue: https://github.com/milvus-io/milvus/issues/35853

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
 2025-03-18 17:20:16 +08:00
congqixia
4ca1a7a76d
fix: Protect getSegmentIndexes with rlock (#40719) 
Related to #40718

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2025-03-18 16:40:14 +08:00
XuanYang-cn
015a8f7631
fix: L0 brings its own start pos when syncing (#40663) 
See also: #40388, #40207

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2025-03-18 15:56:16 +08:00
Ted Xu
be86d31ea3
feat: compaction to support add field (#40415) 
See: #39718

---------

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
 2025-03-18 11:32:12 +08:00
aoiasd
f943a8c78b
enhance: accesslog support fetch hybrid search expr and field (#40167) 
relate: https://github.com/milvus-io/milvus/issues/40166

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2025-03-17 18:08:10 +08:00
congqixia
94a859c028
enhance: Add buffer forwarder for stream delta loading (#40559) 
See also #40558
Related to #35303 & #38066 as well

This PR:
- Add `BufferedForward` to limit memory usage forwarding stream delete
- Add `UseLoad` flag to determine `Delete` shall use `segment.Delete` or
`segment.LoadDelta`
- Fix delegator accidentally use always true candidate while load
streaming delta

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2025-03-17 15:24:10 +08:00
zhagnlu
7ebe3d7038
enhance: refine chunk access logic and add some comment on data (#40618) 
#40367

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2025-03-16 22:20:08 +08:00
cai.zhang
123b6588b6
feat: Support get segment binlogs info with new interface GetSegmentsInfo (#40464) 
issue: #40341

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2025-03-16 21:04:07 +08:00
groot
9fbfcda48e
fix: Fix a crash issue of bulkinsert (#40331) 
issue: https://github.com/milvus-io/milvus/issues/40291
pr: https://github.com/milvus-io/milvus/pull/40304

Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2025-03-14 18:14:07 +08:00
Bingyi Sun
6249335859
fix: Catch invalid json pointer error (#40625) 
issue: #35528

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2025-03-14 16:56:08 +08:00
Bingyi Sun
d3adab15ac
fix: Build double index for all json numeric field (#40619) 
issue: #35528

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2025-03-14 16:52:11 +08:00
Bingyi Sun
8f66a0432d
fix: json path should be empty string if it's root (#40611) 
issue: #35528

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2025-03-14 16:50:16 +08:00
Bingyi Sun
a7cff3873b
fix: Verify json_cast_type (#40604) 
issue: https://github.com/milvus-io/milvus/issues/40420

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2025-03-14 16:48:26 +08:00
zhagnlu
8f85acc944
Revert "fix:reset default auto index type for scalar (#37086)" (#39819) 
This reverts commit eeb67a38450732eb8ab3a4eb4aaac63401426627.
#32900

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2025-03-14 16:16:11 +08:00
Bingyi Sun
8fbacf3583
fix: Null expr does not work for json field (#40456) 
issue: https://github.com/milvus-io/milvus/issues/40455

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2025-03-14 16:06:08 +08:00
Spade A
001fc992df
enhance: get doc ids by batch (#40608) 
issue: #40607

tantivy change: https://github.com/zilliztech/tantivy/pull/3

Benchmarks:
Test Envrioment: CPU 9900K
The data is insert by:
```
for i in 0..N {
    for j in 0..UNIQUE {
        let key = format!("hello{}", j);
        index_writer.add_string(&key, i * UNIQUE + j).unwrap();
    }
}
```
So the unique influences the locality of the matched docs.
The latency is the avg latency over 1000 repeate quries.
The result shows 22.5%-34.8% latency reduction.

![image](https://github.com/user-attachments/assets/dd8af75a-ddc3-445d-92df-50d354dd5645)

---------

Signed-off-by: SpadeA <tangchenjie1210@gmail.com>
 2025-03-14 15:48:09 +08:00
cai.zhang
6dbe5d475e
enhance: Refine task meta with key lock (#40613) 
issue: #39101

2.5 pr: #40146 #40353

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2025-03-14 15:44:22 +08:00
SimFG
bf4fc6a8c6
feat: add DDLDB rate type and related quota configurations (#40651) 
- issue: #40650

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2025-03-14 15:38:09 +08:00
Spade A
f36d1562bd
enhance: add metrics for random sample (#40634) 
issue: #39541

Signed-off-by: SpadeA <tangchenjie1210@gmail.com>
 2025-03-13 21:42:11 +08:00
yihao.dai
bab30a41bf
enhance: Improve import error msgs (#40567) 
issue: https://github.com/milvus-io/milvus/issues/40208

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2025-03-13 21:02:07 +08:00
Zhen Ye
f6fb4bc442
fix: backoff will retry infinitely after reaching max elapse (#40589) 
issue: #40588

Signed-off-by: chyezh <chyezh@outlook.com>
 2025-03-13 16:24:06 +08:00
yihao.dai
b2a8694686
enhance: Merge IndexNode and DataNode (#40272) 
Merge DataNode and IndexNode into DataNode.

issue: https://github.com/milvus-io/milvus/issues/39115

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2025-03-13 14:26:11 +08:00
Ted Xu
df4285c9ef
enhance: API integration with storage v2 in clustering-compactions (#40133) 
See #39173

---------

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
 2025-03-13 14:12:06 +08:00
Zhen Ye
5735c3ef19
fix: too many memory usage of streaming node (#40606) 
issue: #40592

Signed-off-by: chyezh <chyezh@outlook.com>
 2025-03-13 07:10:07 +08:00
Xiaofan
fb48b3c7ac
fix: empty sparse row in importer (#40585) 
fix #40584

parquet bulk writer can not finish 0 dim sparse vector.

Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
 2025-03-13 01:29:41 +08:00
Spade A
9f3bd55755
fix: avoid panic when field not exists in schema in query node (#40541) 
ref #40473

This PR is a workaround to avoid the panic described in the issue.

---------

Signed-off-by: SpadeA <tangchenjie1210@gmail.com>
 2025-03-12 22:44:08 +08:00
jaime
c8a96377bb
enhance: move object storage client creation to pkg package (#40440) 
issue: #40439

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2025-03-12 20:38:07 +08:00
yihao.dai
27c7cbbc72
fix: Fix QueryNodeNumEntities metric (#40602) 
fix QueryNodeNumEntities metric introduced by pr
https://github.com/milvus-io/milvus/pull/39536

issue: https://github.com/milvus-io/milvus/issues/38162

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2025-03-12 19:08:05 +08:00
wei liu
0420dc1eb1
fix: use correct delete checkpoint to prevent premature data cleanup (#40366) 
issue: #40292
related to #39552

- Fix incorrect delete checkpoint usage in SyncDistribution
- Change checkpoint parameter from action.GetCheckpoint() to
action.GetDeleteCP() in SyncTargetVersion call
- This resolves the issue where delete buffer data was being cleaned
prematurely due to wrong checkpoint reference

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2025-03-12 15:00:08 +08:00
sthuang
c0e03b6ca4
fix: rbac star privilege return empty when listing policy (#40553) 
related: https://github.com/milvus-io/milvus/issues/40547

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
 2025-03-12 14:16:05 +08:00
Spade A
95e2680a36
fix: ref collection for search/query (#40549) 
ref https://github.com/milvus-io/milvus/issues/40473

Collection is got without ref which means the collection could be
releases and the struct could be freed during the search which leads
schema inconsistency.

Signed-off-by: SpadeA <tangchenjie1210@gmail.com>
 2025-03-12 11:30:07 +08:00
Bingyi Sun
0698d04f7d
enhance: Upgrade simdjson version (#40538) 
issue: https://github.com/milvus-io/milvus/issues/40519
simdjson returns better error code in newer version.

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2025-03-11 15:04:05 +08:00
cai.zhang
e5f50076ec
enhance: Only check element type with not null array (#40446) 
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2025-03-11 14:58:07 +08:00
yihao.dai
a33c9372ce
fix: Fix channel not balance on datanodes (#40422) 
1. Prevent channels from being assigned to only one datanode during
datacoord startup.
2. Optimize the channel assignment policy by considering newly assigned
channels.
3. Make msgdispatcher manager lock-free.

issue: https://github.com/milvus-io/milvus/issues/40421,
https://github.com/milvus-io/milvus/issues/37630

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2025-03-11 14:56:16 +08:00
Bingyi Sun
a729bb84ba
enhance: add json path escape and replace $meta with dynamic field name (#40407) 
issue: #35528

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2025-03-11 14:00:05 +08:00
Zhen Ye
d9fe8f0dcf
fix: [skip e2e] wab unittest may failure (#40470) 
issue: #38399

Signed-off-by: chyezh <chyezh@outlook.com>
 2025-03-11 11:34:06 +08:00
congqixia
3899b0f0d4
fix: Add duplicated type/index params check creating collection (#40462) 
Related to #40461

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2025-03-11 10:14:11 +08:00
junjiejiangjjj
359e7efd8e
feat: Add function running monitoring (#40358) 
#35856 
#40004 
1. Optimize model verification logic
2. Add profiling code

Signed-off-by: junjie.jiang <junjie.jiang@zilliz.com>
 2025-03-10 22:28:05 +08:00
Bingyi Sun
0a7e692b6f
fix: Fix null offset loading in inverted index (#40523) 
issue: #40516

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2025-03-10 22:12:04 +08:00
Cai Yudong
2bd2cca04a
enhance: Truly support multi vector data types in SearchBruteForce (#40499) 
Issue: #38666

Signed-off-by: CaiYudong <yudong.cai@zilliz.com>
 2025-03-10 18:36:03 +08:00
yihao.dai
2ca2e2dbc8
fix: Fix parsing import endTs (#40332) 
Parsing import beginTs, endTs as a hybrid timestamp.

issue: https://github.com/milvus-io/milvus/issues/40326

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2025-03-10 17:38:04 +08:00
XuanYang-cn
e6c46a25ea
enhance: Use correct counter metrics for overall wa calculation (#40394) 
- Use CounterVec to calculate sum of increase during a time period.
- Use entries number instead of binlog size

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2025-03-10 16:34:06 +08:00
congqixia
391804c7fb
enhance: Add channel seal policy based on blocking l0 (#40505) 
Related to #40502

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2025-03-10 16:28:04 +08:00