admin/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2026-02-02 01:06:41 +08:00
Code Issues Packages Projects Releases Wiki Activity
21,883 Commits 146 Branches 196 Tags
Commit Graph

1147 Commits

Author SHA1 Message Date
yihao.dai
27ea5d14dc
fix: [2.5] Fix delete data loss due to duplicate binlogID (#40976) 
With concurrenct L0 compaction
(https://github.com/milvus-io/milvus/pull/36816), delta logs might be
written to the same L1 segment, causing logID duplication when using the
incremental beginLogID. This PR removes the beginLogID mechanism and
instead passes a log ID range, where the number of IDs in the range
equals the number of compaction segment binlogs multiplied by an
expansion factor.

issue: https://github.com/milvus-io/milvus/issues/40207

pr: https://github.com/milvus-io/milvus/pull/40960

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2025-03-28 14:34:21 +08:00
wei liu
b64bb63e77
enhance: [2.5] Add trigger interval config for auto balance (#39154) (#39918) 
issue: #39156
pr: #39154

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2025-03-27 16:40:23 +08:00
congqixia
2411c184a8
enhance: [2.5] Support detailed manual compaction criterion (#40892) (#40924) 
Cherry-pick from master
pr: #40892
Related to #40866

This PR:
- update go-api/v2 and support partition id/channel/segment level manual
compaction
- refines the compaction trigger implementation
- unify the compaction signal usage

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2025-03-27 10:36:22 +08:00
cai.zhang
e8970e8bf4
fix: [2.5] Update milvus key words (#40870) 
issue: #40290 
master pr: #40869

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2025-03-26 19:44:32 +08:00
Zhen Ye
53438c751d
fix: skip consuming from streaming service message (#40879) 
issue: #40532
pr: #40877

Signed-off-by: chyezh <chyezh@outlook.com>
 2025-03-25 15:22:23 +08:00
cai.zhang
ec0c25aaec
fix: [2.5] As the segment size increases the number of slots should also increase (#40862) 
issue: #40858 
master pr: #39084

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2025-03-25 14:42:24 +08:00
yihao.dai
9b7d66d9da
fix: [2.5] Fix channel not balance on datanodes (#40854) 
1. Prevent channels from being assigned to only one datanode during
datacoord startup.
2. Optimize the channel assignment policy by considering newly assigned
channels.

issue: https://github.com/milvus-io/milvus/issues/40421,
https://github.com/milvus-io/milvus/issues/37630

pr: https://github.com/milvus-io/milvus/pull/40422

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2025-03-25 10:06:24 +08:00
yihao.dai
b534c9d804
enhance: [2.5] Introduce batch subscription in msgdispatcher (#40596) 
Introduce a batch subscription mechanism in msgdispatcher: the
msgdispatcher now includes a vchannel watch task queue, where all
vchannels in the queue will subscribe to the MQ only once and pull
messages from the oldest vchannel checkpoint to the latest.

issue: https://github.com/milvus-io/milvus/issues/39862

pr: https://github.com/milvus-io/milvus/pull/39863

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2025-03-24 10:18:17 +08:00
cai.zhang
d703d8dac8
fix: [2.5] Set correct default configuration for task slots (#40821) 
issue: #39101

master pr: #39084

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2025-03-23 21:28:19 +08:00
SimFG
1263505808
feat: [2.5] support to deny dll according to database property (#40784) 
- issue: #40762
- pr: #40764

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2025-03-23 11:18:28 +08:00
cqy123456
7c716d61dc
fix:[2.5]wrong p.InterimIndexRefineQuantType default value and reduce ut run time (#40830) 
issue: https://github.com/milvus-io/milvus/issues/27678

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2025-03-21 19:10:13 +08:00
Buqian Zheng
cff0e82f57
enhance: [2.5] improve sparse query nnz metric (#40714) 
add query type and field id label; add metric for hybrid search

issue: https://github.com/milvus-io/milvus/issues/35853
pr: https://github.com/milvus-io/milvus/pull/40713

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
 2025-03-20 14:14:21 +08:00
XuanYang-cn
1a6761ac69
enhance: [cp25]Replace currRows with NumOfRows (#40074) (#40681) 
See also: #40068
pr: #40074

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2025-03-20 11:24:13 +08:00
congqixia
cd8d7efc96
enhance: [2.5] Add an extra parameter for slow log (#40759) 
Cherry-pick from master
pr: #40758
Related to #40756

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2025-03-19 18:10:13 +08:00
cai.zhang
0cfe835cac
enhance: [2.5] Support slot for index task and stats task (#40288) 
issue: #39101 

master pr: #39084

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2025-03-19 16:50:11 +08:00
congqixia
e49294cd16
enhance: [2.5] Add buffer forwarder for stream delta loading (#40559) (#40699) 
Cherry-pick from master
pr: #40559
See also #40558
Related to #35303 & #38066 as well

This PR:
- Add `BufferedForward` to limit memory usage forwarding stream delete
- Add `UseLoad` flag to determine `Delete` shall use `segment.Delete` or
`segment.LoadDelta`
- Fix delegator accidentally use always true candidate while load
streaming delta

---------

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2025-03-19 10:28:14 +08:00
yihao.dai
aca9dd9105
fix: [2.5] Fix messages loss during seek (#40736) 
issue: https://github.com/milvus-io/milvus/issues/40733

pr: https://github.com/milvus-io/milvus/pull/40735

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2025-03-18 21:56:33 +08:00
congqixia
6d25fc56aa
enhance: Bump milvus & proto version to v2.5.7 (#40728) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2025-03-18 19:58:15 +08:00
congqixia
8de5174e4d
fix: [2.5] Remove lag targets after main dispatcher (#40717) 
Related to #40716

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2025-03-18 19:10:34 +08:00
Xianhui Lin
705b3c90a5
fix: Failed to rolling upgrade from v2.5.6 to new 2.5 version when enable JsonKeyStats (#40661) 
fix: Failed to rolling upgrade from v2.5.6 to new 2.5 version when
enable JsonKeyStats.The reason is that the file path of the jsonkeyindex
has changed.
issue: https://github.com/milvus-io/milvus/issues/40649
，https://github.com/milvus-io/milvus/issues/40669
https://github.com/milvus-io/milvus/issues/40707
master-pr: https://github.com/milvus-io/milvus/pull/38039

---------

Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
 2025-03-18 17:32:16 +08:00
Zhen Ye
167d058682
enhance: remove redundant resource key watch operation, just keep consistency of wal (#40503) 
issue: #38399
pr: #40235

related PR: #39522

- Just implement exclusive broadcaster between broadcast message with
same resource key to keep same order in different wal.
- After simplify the broadcast model, original watch-based broadcast is
too complicated and redundant, remove it.
- Add metrics for broadcast.

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 2025-03-18 15:46:25 +08:00
cqy123456
b8abd91962
enhance[2.5]: add more config for interimIndex to support more refine mode (#40429) 
issue: https://github.com/milvus-io/milvus/issues/27678

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2025-03-17 22:04:12 +08:00
XuanYang-cn
f455923ac9
enhance: Use correct counter metrics for overall wa calculation (#40394) (#40679) 
pr: #40394

- Use CounterVec to calculate sum of increase during a time period.
- Use entries number instead of binlog size

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2025-03-17 15:06:19 +08:00
Gao
09f7a60fe1
enhance: [2.5] make segment prune config refreshable (#40632) 
pr: #40539

Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2025-03-14 16:12:09 +08:00
SimFG
a3c60a03b5
feat: [2.5] add DDLDB rate type and related quota configurations (#40653) 
- issue: #40650
- pr: #40651

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2025-03-14 15:36:17 +08:00
zhagnlu
6b9e141ada
enhance: reorder sub expr for conjunct expr (#40186) 
pr:#39872

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2025-03-14 15:16:08 +08:00
congqixia
d8a2c1a907
enhance: [2.5] Add channel seal policy based on blocking l0 (#40505) (#40535) 
Cherry-pick from 2.5
pr: #40505
Related to #40502

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2025-03-14 10:02:12 +08:00
cai.zhang
bdc0e68aaf
feat: [2.5] Support get segment binlogs info with new interface GetSegmentsInfo (#40466) 
issue: #40341 

master pr: #40464

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2025-03-14 00:14:08 +08:00
Xianhui Lin
f5e9dea2aa
fix: [2.5]fix the garbage cleanup logic of jsonkey stats && improve json key stats filer (#40039) 
fix: fix the garbage collection cleanup logic of jsonkey stats &&
improve json key stats filer
issue: https://github.com/milvus-io/milvus/issues/36995
https://github.com/milvus-io/milvus/issues/40034
https://github.com/milvus-io/milvus/issues/40041
https://github.com/milvus-io/milvus/issues/40106
https://github.com/milvus-io/milvus/issues/40138
pr: https://github.com/milvus-io/milvus/pull/38039

---------

Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
 2025-03-13 20:18:10 +08:00
yihao.dai
cd8f1fe0e4
enhance: [2.5] Remove unnecessary collection and partition label from the metrics (#40593) 
/kind improvement

pr: https://github.com/milvus-io/milvus/pull/39536

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2025-03-13 12:24:08 +08:00
Bingyi Sun
683b26ffb7
feat: cherry pick json path index (#40313) 
issue: #35528 
pr: #36750 
this pr includes json path index pr and some related prs:
1. update tantivy version #39253 
2. json path index #36750 
3. fall back to brute force #40076 
4. term filter #40140 
5. bug fix #40336

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2025-03-10 22:14:05 +08:00
aoiasd
ebf3416245
fix: main dispatcher not terminate cause messsage stream leak. (#40061) (#40351) 
Main dispatcher will leak when we remove solo dispatcher in the end.
relate: https://github.com/milvus-io/milvus/issues/40046
pr: https://github.com/milvus-io/milvus/pull/40061

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2025-03-05 20:14:02 +08:00
yihao.dai
7cbd40c7a1
fix: [2.5] Fix parsing import endTs (#40333) 
Parsing import beginTs, endTs as a hybrid timestamp.

issue: https://github.com/milvus-io/milvus/issues/40326

pr: https://github.com/milvus-io/milvus/pull/40332

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2025-03-05 11:40:13 +08:00
congqixia
4272703de4
enhance: Bump milvus & proto version to v2.5.6 (#40293) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2025-03-03 16:52:00 +08:00
SimFG
a945345110
fix: [2.5] use the different msg type for the OperatePrivilegeV2 api (#40193) 
- issue: #40178
- pr: #40192

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2025-03-03 10:20:01 +08:00
cai.zhang
4a75fdac03
enhance: Remove task meta when task is no need to do (#40146) 
issue: #39911 

master pr: #39084

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2025-02-25 10:35:57 +08:00
Xianhui Lin
a4eb2ce224
fix: [2.5]Revert qc statschecker for json key stats (#40125) 
Revert qc statschecker for json key stats
issue:https://github.com/milvus-io/milvus/issues/36995
pr:https://github.com/milvus-io/milvus/pull/39876

Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
 2025-02-24 13:31:55 +08:00
cai.zhang
9cbab98382
enhance: [2.5] Export request timeout interval in config (#40118) 2025-02-23 13:47:12 +08:00
congqixia
709594f158
enhance: [2.5] Use v2 package name for pkg module (#40117) 
Cherry-pick from master
pr: #39990
Related to #39095

https://go.dev/doc/modules/version-numbers

Update pkg version according to golang dep version convention

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2025-02-23 00:46:01 +08:00
Xianhui Lin
c1de61ff7c
fix: [2.5]Replace the position of EnabledJSONKeyStats (#40108) 
Replace the position of EnabledJSONKeyStats
issue: https://github.com/milvus-io/milvus/issues/36995
pr: https://github.com/milvus-io/milvus/pull/38039

---------

Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
 2025-02-22 14:35:54 +08:00
congqixia
cb0dc6293c
enhance: [2.5] Sync proxy.maxTaskNum default value to 1024 (#40073) 
Cherry-pick from master
pr: #40072
Sync milvus.yaml value to default value for maxTaskNum

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2025-02-22 11:31:53 +08:00
cai.zhang
0caad5fe37
enhance: [2.5] Add metrics for proxy queue (#40071) 
master pr: #40070

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2025-02-21 21:39:54 +08:00
yihao.dai
b8a758b6c4
enhance: [2.5] Add get vector latency metric and refine request limit error message (#40085) 
issue: https://github.com/milvus-io/milvus/issues/40078

pr: https://github.com/milvus-io/milvus/pull/40083

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2025-02-21 20:19:55 +08:00
congqixia
cae8f75e47
enhance: Bump milvus version to 2.5.5 (#40017) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2025-02-19 18:48:54 +08:00
wei liu
3c2d8c1419
enhance: [2.5] Add management api to check querycoord balance status (#37784) (#39909) 
issue: #37783
pr: #37784

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2025-02-19 10:56:49 +08:00
Xianhui Lin
04175d8549
enhance: [2.5]compress jsonstatslog path only save filename (#39979) 
enhance: compress jsonstatslog path only save filename
issue: https://github.com/milvus-io/milvus/issues/36995
pr: https://github.com/milvus-io/milvus/pull/38039

Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
 2025-02-19 10:20:52 +08:00
yihao.dai
7a92121c92
fix: [2.5] Fix failed to seek to earliest position (#39966) 
If it is the earliest message ID, skip the seek to prevent failure.

issue: https://github.com/milvus-io/milvus/issues/39964

pr: https://github.com/milvus-io/milvus/pull/39965

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2025-02-18 20:36:53 +08:00
zhagnlu
de8abe7970
fix:reset default auto index for varchar (#39908) 
pr: #39819

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2025-02-17 20:55:26 +08:00
cai.zhang
52434ccc78
enhance: [2.5] Limit the speed of the generating stats task (#39645) 
master pr: #39644

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2025-02-17 16:06:17 +08:00
XuanYang-cn
8067113133
enhance: [cp25]Enable to observe write amplification (#39661) (#39743) 
pr: #39661

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2025-02-17 16:00:17 +08:00