22406 Commits

Author SHA1 Message Date
yihao.dai
59a6eef774
fix: Fix compaction getting stuck (#42087)
Reset `isCompacting` flag after JSONStats and BM25 task finished.

issue: https://github.com/milvus-io/milvus/issues/42083

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-05-27 10:26:27 +08:00
hckex
a20500b3ed
doc: Fix small typo in internal/datanode/README.md (#42089)
This PR fixes a minor typo in the README file of the datanode module.
Corrected "imformation" to "information".

Signed-off-by: hckex <33862757+hckex@users.noreply.github.com>
2025-05-27 10:16:27 +08:00
Zhen Ye
212e17c4c5
fix: modify param to use less memory when flush and sync (#42102)
issue: #42097

Signed-off-by: chyezh <chyezh@outlook.com>
2025-05-27 10:12:27 +08:00
Linkwei
40bea4fc21
enhance: [skip e2e] Enhance monitoring by adding metrics related to Milvus 2.5. (#42099)
enhance: [skip e2e] Enhance monitoring by adding metrics related to
Milvus 2.5. https://github.com/milvus-io/milvus/issues/41652

Signed-off-by: Linkwei <link.xie@zilliz.com>
2025-05-27 10:00:27 +08:00
sre-ci-robot
53abf2cc79 Update all contributors
Signed-off-by: sre-ci-robot <sre-ci-robot@zilliz.com>
2025-05-26 12:01:04 +00:00
zhuwenxing
fa04fcbc3a
test: Update default tei endpoint (#42079)
/kind improvement

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2025-05-26 19:06:31 +08:00
aoiasd
0fafb706ba
enhance: add segment bm25 stats local cache (#41775)
relate: https://github.com/milvus-io/milvus/issues/41424

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2025-05-26 18:44:27 +08:00
wei liu
b6723296b2
enhance: [skip e2e] Improve update-milvus-api command for version consistency (#42074)
issue: #42073 
Enhanced the update-milvus-api Makefile command to update milvus-proto
version across all 4 go.mod files in the project:

- Update main go.mod (root module)
- Update client/go.mod (client module)
- Update pkg/go.mod (pkg module)
- Update tests/go_client/go.mod (test client module)

Improvements:
- Add intelligent version detection (git tag vs commit ID)
- Add comprehensive error handling and validation
- Add detailed progress feedback and success reporting
- Add proper directory existence checks
- Add UPDATE_MILVUS_API.md documentation with usage examples

This ensures all modules use the same milvus-proto version, improving
project consistency and maintainability.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2025-05-26 17:08:47 +08:00
zhuwenxing
bfa948c2d4
test: Add int8 vector field support in search checker (#42047)
/kind improvement

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2025-05-26 16:18:49 +08:00
wei liu
f84650ece0
enhance: Reduce session TTL from 30s to 10s for faster failure detection (#42050)
Optimize session management by reducing the TTL (Time To Live) value for
service registration from 30 seconds to 10 seconds. This change improves
the system's ability to detect service failures more quickly and
enhances overall cluster responsiveness.

Changes include:
- Update default session TTL from 30s to 10s in milvus.yaml
- Adjust DefaultSessionTTL constant from 30 to 10 seconds
- Update SessionTTL default value from 60 to 10 seconds
- Maintain consistent TTL values across configuration files

This optimization reduces the time required for the system to detect
when services become unavailable, leading to faster failover and
improved cluster stability during node failures or network issues.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2025-05-26 12:04:26 +08:00
cai.zhang
80fe573c76
enhance: Pass the compaction configuration through request parameters (#41979)
issue: #41123

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2025-05-26 11:52:27 +08:00
Bingyi Sun
e9c0756dc6
enhance: return index params for restful api (#42016)
issue: #42015

Signed-off-by: sunby <sunbingyi1992@gmail.com>
2025-05-26 11:50:28 +08:00
Chun Han
d1cfa58a0a
feature: support compact expiry data(#41336) (#42056)
related: #41336

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2025-05-25 16:46:31 +08:00
yihao.dai
194b492f05
fix: Fix proxy panic at shard client manager (#42025)
issue: https://github.com/milvus-io/milvus/issues/42024

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-05-23 22:44:26 +08:00
cai.zhang
344d002346
fix: Don't create index for unsorted importing segment when enable stats (#42044)
issue: #41863

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2025-05-23 20:46:26 +08:00
Xianhui Lin
a72492169f
feat: add NotifyDropPartition in mixcoord for droppartition in dc (#42029)
add NotifyDropPartition in mixcoord for droppartition in dc
issue:https://github.com/milvus-io/milvus/issues/41976
https://github.com/milvus-io/milvus/issues/41542

Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
2025-05-23 18:32:26 +08:00
Zhen Ye
38c804fb01
fix: more stable recovery graceful closing and stable unittest (#42013)
issue: #41544

Signed-off-by: chyezh <chyezh@outlook.com>
2025-05-23 17:52:26 +08:00
XuanYang-cn
252d49d01e
fix: ChannelManager double assignment (#41837)
See also: #41876

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2025-05-23 14:16:29 +08:00
yihao.dai
f71930e8db
enhance: Enhance import context (#42021)
Rename `imeta` to `importMeta` to improve readability, and enhance
import related context usage.

issue: https://github.com/milvus-io/milvus/issues/41123

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-05-23 12:58:27 +08:00
yihao.dai
83c9527e70
enhance: Use QuerySlot interface for tasks (#41989)
Use `QuerySlot` rpc instead of `QueryTask` for querying slot.

issue: https://github.com/milvus-io/milvus/issues/41123

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-05-23 10:30:28 +08:00
cai.zhang
48419da4d2
fix: Re add json stats trigger (#41967)
issue: #41123

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2025-05-23 10:02:27 +08:00
binbin
eea6b50fbb
test: add test cases for int8 vector (#41957)
Signed-off-by: binbin lv <binbin.lv@zilliz.com>
2025-05-23 09:24:28 +08:00
tinswzy
1735f557ca
fix sn oom issue during small file loading in wp (#41946)
#41846  #41894 
Resolve SN OOM issue during small file loading in Woodpecker; 
Correct WP fence/close execution order;

Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
2025-05-23 01:30:28 +08:00
presburger
e878fe588e
enhance: Set the default GPU version autoindex to use the CAgra index (#41906)
issue:  #41907

Signed-off-by: yusheng.ma <yusheng.ma@zilliz.com>
2025-05-23 01:20:28 +08:00
yihao.dai
e04e5b41ca
enhance: Add task version monitoring (#42023)
issue: https://github.com/milvus-io/milvus/issues/41123

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-05-22 23:24:28 +08:00
congqixia
244aa30076
fix: Lock before reading flusher cp sampling truncate cp (#42019)
Related to #42018

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-05-22 21:38:28 +08:00
Zhen Ye
c9b0748ff9
enhance: add delete rows into delete msg header and more metric (#41952)
issue: #41544

- add delete rows into delete messsage header
- add more insert/delete metrics
- fix non-broadcast message has broadcast header

Signed-off-by: chyezh <chyezh@outlook.com>
2025-05-22 20:28:26 +08:00
congqixia
9fb0257bfa
enhance: Set thread name for segcore thread pool (#42017)
Thread name could be helpful when debugging thread explosion issues

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-05-22 19:06:27 +08:00
zhuwenxing
c937da5234
test: add multi analyzer test (#41578)
/kind improvement

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2025-05-22 17:46:24 +08:00
congqixia
22ef03b040
fix: [AddField] Expire metacache with rc task timestamp (#42010)
Related to #39718 #41681

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-05-22 17:20:25 +08:00
binbin
804b175771
test: add test cases for rerank (#42012)
Signed-off-by: binbin lv <binbin.lv@zilliz.com>
2025-05-22 17:04:25 +08:00
Zhen Ye
458ab86894
fix: stop retry if collection not found too much when get recovery from coord (#41980)
issue: #41966

Signed-off-by: chyezh <chyezh@outlook.com>
2025-05-22 16:22:24 +08:00
Buqian Zheng
2e3539319d
feat: vector field raw data to mmap by default (#41975)
issue: https://github.com/milvus-io/milvus/issues/41435

should address https://github.com/milvus-io/milvus/issues/41774

this PR also: 
* added caching layer memory overhead metric
* re-enable TextMatch.GrowingLoadData test

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2025-05-22 11:56:25 +08:00
wei liu
78010262f0
enhance: Optimize shard serviceable mechanism (#41937)
issue: https://github.com/milvus-io/milvus/issues/41690
- Merge leader view and channel management into ChannelDistManager,
allowing a channel to have multiple delegators.
- Improve shard leader switching to ensure a single replica only has one
shard leader per channel. The shard leader handles all resource loading
and query requests.
- Refine the serviceable mechanism: after QC completes loading, sync the
query view to the delegator. The delegator then determines its
serviceable status based on the query view.
- When a delegator encounters forwarding query or deletion failures,
mark the corresponding segment as offline and transition it to an
unserviceable state.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2025-05-22 11:38:24 +08:00
congqixia
f021b3f26a
fix: [AddField] Add protection logic inserting old data into new schema (#41978)
Related to #39718

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-05-22 11:30:24 +08:00
wei liu
dad43a3894
fix: cost metrics collection logic for replica selection (#41965)
issue: #41621
- Deprecate EnableWorkerSQCostMetrics parameter
- Always collect cost metrics from all search and retrieve results
- Update code with comments explaining the changes rationale

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2025-05-22 10:20:25 +08:00
Buqian Zheng
8a85bc4213
fix: fixes async warmup deadlock (#41995)
issue: https://github.com/milvus-io/milvus/issues/41993

also updated cachinglayer metrics

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2025-05-22 09:54:24 +08:00
yanliang567
f20e085b22
test: Update ivf_rabitq error msg and groupby support nullable field (#41997)
related issue: #41898

Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
2025-05-21 22:20:25 +08:00
wei liu
4e1208f4f6
enhance: support balancing multiple collections in single trigger (#41875)
issue: #41874
- Optimize balance_checker to support balancing multiple collections
simultaneously
- Add new parameters for segment and channel balancing batch sizes
- Add enableBalanceOnMultipleCollections parameter
- Update tests for balance checker

This change improves resource utilization by allowing the system to
balance multiple collections in a single trigger with configurable batch
sizes.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2025-05-21 21:38:25 +08:00
SimFG
9f866dd7c3
enhance: add privilege group message types and corresponding tests (#41939)
- issue: #41938

Signed-off-by: SimFG <bang.fu@zilliz.com>
2025-05-21 11:12:28 +08:00
yihao.dai
3f85d376dd
fix: Fix ineffective log level setting in ThreadWatcher (#41886)
issue: https://github.com/milvus-io/milvus/issues/41781

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-05-20 21:22:24 +08:00
yihao.dai
142bd2fc05
enhance: Pooling for data tasks (#41256)
1. Add global scheduler for datacoord.
2. Define and implement new CreateTask, QueryTask, DropTask interfaces.
3. Refine Import, Compaction, Stats, Index task.

issue: https://github.com/milvus-io/milvus/issues/41123

Co-authored-by: Cai Zhang <cai.zhang@zilliz.com>
2025-05-20 21:06:24 +08:00
yanliang567
d475d93a3d
test: Add ivf_rabitq index tests (#41914)
related issue: #41760

---------

Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
2025-05-20 19:28:24 +08:00
Zhen Ye
59ab274dbe
fix: use flusher and recovery checkpoint together to determine the truncate position (#41934)
issue: #41544

- unify the log field of message
- use the minimum one of flusher and recovery storage checkpoint as the
truncate position

Signed-off-by: chyezh <chyezh@outlook.com>
2025-05-20 16:10:24 +08:00
congqixia
b099926f24
fix: Read group by field in sparse format in it (#41943)
Related to #41942

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-05-20 16:02:23 +08:00
junjiejiangjjj
1cc5fa8be9
fix: Fix rrf score (#41857)
https://github.com/milvus-io/milvus/issues/41835

Signed-off-by: junjie.jiang <junjie.jiang@zilliz.com>
2025-05-20 15:24:24 +08:00
junjiejiangjjj
0bbbf98a5b
enhance: refactor decay rerank (#41778)
https://github.com/milvus-io/milvus/issues/35856
1. Optimizing decay function
2. Since the decay function is larger, the more similar it is, the
smaller the L2/JACCARD/HAMMING metrics scores the more similar they are.
For these metrics, the decay function regenerates new scores.

Signed-off-by: junjie.jiang <junjie.jiang@zilliz.com>
2025-05-20 15:22:24 +08:00
foxspy
3dbad0306a
fix: Add bypass thread pool mode to avoid growing indexes blocking insert/load (#41012)
issue: #40825

Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
2025-05-20 14:30:24 +08:00
aoiasd
b6f251718c
enhance: support print nq and params for search and query (#41802)
relate: https://github.com/milvus-io/milvus/issues/41801

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2025-05-20 14:10:23 +08:00
congqixia
f2a8330f87
fix: [StorageV2] Use correct group building index (#41925)
Related to #39173 #41534

This pr fixes an issue that building mem index may report datatype not
match error when collection split fields into multiple groups

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-05-20 13:26:23 +08:00