admin/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2025-12-08 10:08:42 +08:00
Code Issues Packages Projects Releases Wiki Activity
22,154 Commits 133 Branches 183 Tags
Commit Graph

698 Commits

Author SHA1 Message Date
wei liu
b08d9efe69
fix: Prevent delegator unserviceable due to shard leader change (#42689) (#43309) 
issue: #42098 #42404
pr: #42689
Fix critical issue where concurrent balance segment and balance channel
operations cause delegator view inconsistency. When shard leader
switches between load and release phases of segment balance, it results
in loading segments on old delegator but releasing on new delegator,
making the new delegator unserviceable.

The root cause is that balance segment modifies delegator views, and if
these modifications happen on different delegators due to leader change,
it corrupts the delegator state and affects query availability.

Changes include:
- Add shardLeaderID field to SegmentTask to track delegator for load
- Record shard leader ID during segment loading in move operations
- Skip release if shard leader changed from the one used for loading
- Add comprehensive unit tests for leader change scenarios

This ensures balance segment operations are atomic on single delegator,
preventing view corruption and maintaining delegator serviceability.

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2025-07-15 17:46:51 +08:00
wei liu
33e9b873de
enhance: add logs for count(*) (#43001) (#43249) 
pr: #43001

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2025-07-11 19:14:48 +08:00
Chun Han
bfa9688da3
enhance: supporting separate chunk cache pool(#42803) (#42901) 
related: #42803

1. add a new thread pools using folly::CPUThreadPoolExecutor, named by
FThreadPools
2. reading vectors from chunkcache will use the separated
CHUNKCACHE_POOL to avoid being influenced by load collection
3. Note. For safety on cloud side on 2.5.x, only read-chunk-cache
operations is using this newly created thread pools other caller points
for threadpool will be mutated in the near future
4. master-branch doesn't need this pr as caching layer unified the chunk
cache behaviour

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
 2025-06-26 15:52:43 +08:00
aoiasd
7feeeabca5
enhance: [2.5] bm25 stats local cache use local storage path (#42924) 
relate: https://github.com/milvus-io/milvus/pull/42923

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2025-06-25 13:44:46 +08:00
Zhen Ye
30b2a66f59
fix: rocksmq cannot graceful stop (#42843) 
issue: #40532
pr: #42841

Signed-off-by: chyezh <chyezh@outlook.com>
 2025-06-19 22:18:40 +08:00
aoiasd
98fe28d0ef
fix: [2.5] flow graph should free function resource after all node close (#42731) (#42775) 
pr: https://github.com/milvus-io/milvus/pull/42731

relate: https://github.com/milvus-io/milvus/issues/42730

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2025-06-17 14:40:39 +08:00
aoiasd
5110130b2e
enhance: add segment bm25 stats local cache (#41775) (#42646) 
relate: https://github.com/milvus-io/milvus/issues/41424
pr: https://github.com/milvus-io/milvus/pull/41775

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2025-06-13 16:50:37 +08:00
aoiasd
40ecaa6061
fix:[2.5] add concurrency and close protect for bm25 function (#42599) 
relate: https://github.com/milvus-io/milvus/issues/42576
pr: https://github.com/milvus-io/milvus/pull/42597

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2025-06-10 14:18:35 +08:00
Zhen Ye
a55c371261
fix: querynode upgrade from 2.5 get stucked (#42503) 
issue: #42492
pr: #42502

- querynode graceful stop can be done if there's only L0 segment exists.

Signed-off-by: chyezh <chyezh@outlook.com>
 2025-06-05 15:56:33 +08:00
aoiasd
198ff1f150
enhance: [2.5] support run analyzer by loaded collection field (#42119) 
relate: https://github.com/milvus-io/milvus/issues/42094
pr: https://github.com/milvus-io/milvus/pull/42113

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2025-05-29 10:26:30 +08:00
Xianhui Lin
843c4228df
fix: pass the ttl duration in the search request for ttl filter (#42121) 
fix:  pass the TTL duration in the search request for TTL filter
issue:https://github.com/milvus-io/milvus/issues/41959
pr:https://github.com/milvus-io/milvus/pull/42122

Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
 2025-05-28 09:42:28 +08:00
Xianhui Lin
0490344442
fix: support TTL expiration with queries returning no results (#42103) 
support TTL expiration with queries returning no results
issue:https://github.com/milvus-io/milvus/issues/41959
pr:https://github.com/milvus-io/milvus/pull/42086

---------

Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
 2025-05-27 15:18:28 +08:00
Xianhui Lin
0574fc7b7b
enhance: support TTL expiration with queries returning no results (#41960) 
support TTL expiration with queries returning no results
issue:https://github.com/milvus-io/milvus/issues/41959
pr:https://github.com/milvus-io/milvus/pull/41720

---------

Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
 2025-05-26 15:52:28 +08:00
aoiasd
daf745ffa3
fix: [2.5] hybird search sub requset not set analyzer name (#41897) 
relate: https://github.com/milvus-io/milvus/issues/41213
pr: https://github.com/milvus-io/milvus/pull/41896

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2025-05-16 17:58:22 +08:00
aoiasd
bb562c6a7e
fix:[2.5] analyzer memory leak because function runner not close (#41840) 
relate: https://github.com/milvus-io/milvus/issues/41213
pr:https://github.com/milvus-io/milvus/pull/41839

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2025-05-15 15:48:23 +08:00
cai.zhang
6278fb3b56
fix: [2.5] Collect entites num group by collection instead of partition (#41789) 
issue: #41787

master pr: #41788

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2025-05-15 11:40:23 +08:00
zhagnlu
5b8ea84d38
fix: add params to ignore config type exception (#41777) 
pr: #41776

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2025-05-13 11:28:57 +08:00
congqixia
8b026f93a9
enhance: [2.5] Add mutex preventing concurrent plugin.Open (#41761) (#41764) 
Cherry pick from master
pr: #41761 
Concurrent calling plugin.Open might cause empty pluginpath issue

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2025-05-12 23:20:57 +08:00
Chun Han
69a80b9ce3
enhance: resize high priority wqthreadpool dynamically(#40838) (#41549) 
related: #40838

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
 2025-04-28 00:44:39 +08:00
SimFG
6e18ededab
fix: [2.5] mockery too unavailable after upgrade golang version (#41522) 
- issue: ##41291
- pr: #41481

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2025-04-25 14:40:40 +08:00
SimFG
18eb627533
fix: [2.5] Update logging context and upgrade dependencies (#41319) 
- issue: #41291
- pr: #41318

---------

Signed-off-by: SimFG <bang.fu@zilliz.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 2025-04-24 23:50:40 +08:00
aoiasd
87b9cbffaf
fix:[2.5] bm25 search failed when avgdl == nan (#41503) 
relate: https://github.com/milvus-io/milvus/issues/41490
pr: https://github.com/milvus-io/milvus/pull/41502

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2025-04-24 23:16:38 +08:00
Xiaowei Shi
2f78f25c7f
fix: correct wrong querynode metric labels (#41344) (#41422) 
cherry-pick from master: https://github.com/milvus-io/milvus/pull/41344
related: https://github.com/milvus-io/milvus/issues/41343

Signed-off-by: Xiaowei Shi <shallwe.shih@gmail.com>
 2025-04-24 21:40:40 +08:00
aoiasd
8b3353cdab
feat: [2.5] Support run analyzer and more tokenizer. (#41444) 
relate: https://github.com/milvus-io/milvus/issues/39659
https://github.com/milvus-io/milvus/issues/39705
pr: https://github.com/milvus-io/milvus/pull/40416
https://github.com/milvus-io/milvus/pull/40458
https://github.com/milvus-io/milvus/pull/39723
https://github.com/milvus-io/milvus/pull/40813
https://github.com/milvus-io/milvus/pull/39854
https://github.com/milvus-io/milvus/pull/39660

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2025-04-24 10:48:39 +08:00
aoiasd
544493e3e2
feat:[2.5] support multi analyzer for bm25 function (#41456) 
relate: https://github.com/milvus-io/milvus/issues/41213
pr: https://github.com/milvus-io/milvus/pull/41351

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2025-04-23 20:52:39 +08:00
aoiasd
8f2ccc25ab
fix: [2.5] bm25 stats idf oracle leak (#41426) 
relate: https://github.com/milvus-io/milvus/issues/41424
pr: https://github.com/milvus-io/milvus/pull/41425

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2025-04-22 19:58:37 +08:00
Chun Han
a89b611b2a
fix: solve incompitable problem for none-encoding index (#41346) 
related: https://github.com/milvus-io/milvus/issues/40838
pr: https://github.com/milvus-io/milvus/pull/40839

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
 2025-04-16 21:04:52 +08:00
Spade A
699f8a6ada
fix: [2.5] change log info to debug for collection ref (#41269) 
master: https://github.com/milvus-io/milvus/pull/41267
issue: https://github.com/milvus-io/milvus/issues/41268

---------

Signed-off-by: SpadeA <tangchenjie1210@gmail.com>
 2025-04-15 17:28:31 +08:00
liliu-z
cb0f984155
enhance: Revert "separate for index completed (#40873)" (#41152) 
This reverts commit 23e579e3240a30397f05f5b308be687f6f16b013. #40873

issue: #39519

Signed-off-by: Li Liu <li.liu@zilliz.com>
 2025-04-08 17:36:30 +08:00
wei liu
ea2b40a79d
fix: remove metrics reset calls to ensure accurate reporting (#41081) 
issue: #41048
pr: #41049
Fixes issue introduced in PR #33522 where metric resets caused
incomplete data collection by monitoring systems.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2025-04-08 11:38:36 +08:00
Chun Han
23e579e324
separate for index completed (#40873) 
related: https://github.com/milvus-io/milvus/issues/40781

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
 2025-04-05 10:20:24 +08:00
Gao
9c41636a95
fix: [2.5] correct WithGroupSize while reducing (#40920) 
issue: #40887 
pr: #40888

Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2025-03-26 12:02:20 +08:00
yihao.dai
b534c9d804
enhance: [2.5] Introduce batch subscription in msgdispatcher (#40596) 
Introduce a batch subscription mechanism in msgdispatcher: the
msgdispatcher now includes a vchannel watch task queue, where all
vchannels in the queue will subscribe to the MQ only once and pull
messages from the oldest vchannel checkpoint to the latest.

issue: https://github.com/milvus-io/milvus/issues/39862

pr: https://github.com/milvus-io/milvus/pull/39863

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2025-03-24 10:18:17 +08:00
congqixia
0f0cb4c189
enhance: [2.5] Close component in topological order when unsub channel (#40796) (#40819) 
Cherry-pick from master
pr: #40796

Related to #40795

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2025-03-21 19:12:15 +08:00
Buqian Zheng
cff0e82f57
enhance: [2.5] improve sparse query nnz metric (#40714) 
add query type and field id label; add metric for hybrid search

issue: https://github.com/milvus-io/milvus/issues/35853
pr: https://github.com/milvus-io/milvus/pull/40713

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
 2025-03-20 14:14:21 +08:00
congqixia
e49294cd16
enhance: [2.5] Add buffer forwarder for stream delta loading (#40559) (#40699) 
Cherry-pick from master
pr: #40559
See also #40558
Related to #35303 & #38066 as well

This PR:
- Add `BufferedForward` to limit memory usage forwarding stream delete
- Add `UseLoad` flag to determine `Delete` shall use `segment.Delete` or
`segment.LoadDelta`
- Fix delegator accidentally use always true candidate while load
streaming delta

---------

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2025-03-19 10:28:14 +08:00
Xianhui Lin
705b3c90a5
fix: Failed to rolling upgrade from v2.5.6 to new 2.5 version when enable JsonKeyStats (#40661) 
fix: Failed to rolling upgrade from v2.5.6 to new 2.5 version when
enable JsonKeyStats.The reason is that the file path of the jsonkeyindex
has changed.
issue: https://github.com/milvus-io/milvus/issues/40649
，https://github.com/milvus-io/milvus/issues/40669
https://github.com/milvus-io/milvus/issues/40707
master-pr: https://github.com/milvus-io/milvus/pull/38039

---------

Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
 2025-03-18 17:32:16 +08:00
zhagnlu
6b9e141ada
enhance: reorder sub expr for conjunct expr (#40186) 
pr:#39872

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2025-03-14 15:16:08 +08:00
Xianhui Lin
f5e9dea2aa
fix: [2.5]fix the garbage cleanup logic of jsonkey stats && improve json key stats filer (#40039) 
fix: fix the garbage collection cleanup logic of jsonkey stats &&
improve json key stats filer
issue: https://github.com/milvus-io/milvus/issues/36995
https://github.com/milvus-io/milvus/issues/40034
https://github.com/milvus-io/milvus/issues/40041
https://github.com/milvus-io/milvus/issues/40106
https://github.com/milvus-io/milvus/issues/40138
pr: https://github.com/milvus-io/milvus/pull/38039

---------

Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
 2025-03-13 20:18:10 +08:00
yihao.dai
cd8f1fe0e4
enhance: [2.5] Remove unnecessary collection and partition label from the metrics (#40593) 
/kind improvement

pr: https://github.com/milvus-io/milvus/pull/39536

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2025-03-13 12:24:08 +08:00
Spade A
bcec7c5087
fix: [2.5] ref collection for search/query (#40550) 
ref https://github.com/milvus-io/milvus/issues/40473
cherry-pick: https://github.com/milvus-io/milvus/pull/40549

---------

Signed-off-by: SpadeA <tangchenjie1210@gmail.com>
 2025-03-12 20:12:09 +08:00
Bingyi Sun
683b26ffb7
feat: cherry pick json path index (#40313) 
issue: #35528 
pr: #36750 
this pr includes json path index pr and some related prs:
1. update tantivy version #39253 
2. json path index #36750 
3. fall back to brute force #40076 
4. term filter #40140 
5. bug fix #40336

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2025-03-10 22:14:05 +08:00
Xianhui Lin
a4eb2ce224
fix: [2.5]Revert qc statschecker for json key stats (#40125) 
Revert qc statschecker for json key stats
issue:https://github.com/milvus-io/milvus/issues/36995
pr:https://github.com/milvus-io/milvus/pull/39876

Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
 2025-02-24 13:31:55 +08:00
congqixia
709594f158
enhance: [2.5] Use v2 package name for pkg module (#40117) 
Cherry-pick from master
pr: #39990
Related to #39095

https://go.dev/doc/modules/version-numbers

Update pkg version according to golang dep version convention

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2025-02-23 00:46:01 +08:00
Xianhui Lin
c1de61ff7c
fix: [2.5]Replace the position of EnabledJSONKeyStats (#40108) 
Replace the position of EnabledJSONKeyStats
issue: https://github.com/milvus-io/milvus/issues/36995
pr: https://github.com/milvus-io/milvus/pull/38039

---------

Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
 2025-02-22 14:35:54 +08:00
congqixia
b7c631f0b7
enhance: [2.5] Remove frequent json stats distribution log (#40027) 
Info log is too frequent only printing the json stats & index info

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2025-02-20 13:33:57 +08:00
Zhen Ye
56c1a8d462
fix: ignore growing option is lost at hibridsearch (#39900) 
issue: #39892
pr: #39799

Signed-off-by: chyezh <chyezh@outlook.com>
 2025-02-17 15:30:15 +08:00
Xianhui Lin
f0964f769d
enhance: [2.5]Add json key inverted index in stats for optimization (#39876) 
Add json key inverted index in stats for optimization
issue: https://github.com/milvus-io/milvus/issues/36995
pr: https://github.com/milvus-io/milvus/pull/38039

---------

Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2025-02-16 20:12:15 +08:00
cqy123456
d121ac3a7a
enhance: [2.5]intermin index support different index type and more data type(fp16/bf16) (#39180) 
issue: https://github.com/milvus-io/milvus/issues/27678
related: https://github.com/milvus-io/milvus/pull/39753
some raw data status will change:
Intermin index has raw data: 
<html xmlns:v="urn:schemas-microsoft-com:vml"
xmlns:o="urn:schemas-microsoft-com:office:office"
xmlns:x="urn:schemas-microsoft-com:office:excel"
xmlns="http://www.w3.org/TR/REC-html40">

<head>

<meta name=ProgId content=Excel.Sheet>
<meta name=Generator content="Microsoft Excel 15">
<link id=Main-File rel=Main-File

href="file:////Users/cqy/Library/Group%20Containers/UBF8T346G9.Office/TemporaryItems/msohtmlclip/clip.htm">
<link rel=File-List

href="file:////Users/cqy/Library/Group%20Containers/UBF8T346G9.Office/TemporaryItems/msohtmlclip/clip_filelist.xml">
<!--table
	{mso-displayed-decimal-separator:"\.";
	mso-displayed-thousand-separator:"\,";}
@page
	{margin:.75in .7in .75in .7in;
	mso-header-margin:.3in;
	mso-footer-margin:.3in;}
.font5
	{color:windowtext;
	font-size:9.0pt;
	font-weight:400;
	font-style:normal;
	text-decoration:none;
	font-family:等线;
	mso-generic-font-family:auto;
	mso-font-charset:134;}
tr
	{mso-height-source:auto;
	mso-ruby-visibility:none;}
col
	{mso-width-source:auto;
	mso-ruby-visibility:none;}
br
	{mso-data-placement:same-cell;}
td
	{padding-top:1px;
	padding-right:1px;
	padding-left:1px;
	mso-ignore:padding;
	color:black;
	font-size:12.0pt;
	font-weight:400;
	font-style:normal;
	text-decoration:none;
	font-family:等线;
	mso-generic-font-family:auto;
	mso-font-charset:134;
	mso-number-format:General;
	text-align:general;
	vertical-align:middle;
	border:none;
	mso-background-source:auto;
	mso-pattern:auto;
	mso-protection:locked visible;
	white-space:nowrap;
	mso-rotate:0;}
ruby
	{ruby-align:left;}
rt
	{color:windowtext;
	font-size:9.0pt;
	font-weight:400;
	font-style:normal;
	text-decoration:none;
	font-family:等线;
	mso-generic-font-family:auto;
	mso-font-charset:134;
	mso-char-type:none;
	display:none;}
-->
</head>

<body link="#0563C1" vlink="#954F72">


sparse vector | growing segment | sealed segment
-- | -- | --
BM25 | no | no
IP | yes | no
  |   |  
dense vector | growing segment | sealed segment
ivf flat cc | yes | yes
scann_dvr | no | no



</body>

</html>

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2025-02-13 09:56:47 +08:00
congqixia
1b14f5a7ab
fix: [2.5] Segments return both growing&sealed result (#39789) 
Cherry-pick from master
pr: #39787
Previous PR #38311
Logic bug found in #38348

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2025-02-12 14:44:48 +08:00