admin/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2026-02-02 01:06:41 +08:00
Code Issues Packages Projects Releases Wiki Activity
21,878 Commits 146 Branches 196 Tags
Commit Graph

450 Commits

Author SHA1 Message Date
cai.zhang
ec0c25aaec
fix: [2.5] As the segment size increases the number of slots should also increase (#40862) 
issue: #40858 
master pr: #39084

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2025-03-25 14:42:24 +08:00
yihao.dai
b534c9d804
enhance: [2.5] Introduce batch subscription in msgdispatcher (#40596) 
Introduce a batch subscription mechanism in msgdispatcher: the
msgdispatcher now includes a vchannel watch task queue, where all
vchannels in the queue will subscribe to the MQ only once and pull
messages from the oldest vchannel checkpoint to the latest.

issue: https://github.com/milvus-io/milvus/issues/39862

pr: https://github.com/milvus-io/milvus/pull/39863

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2025-03-24 10:18:17 +08:00
cai.zhang
d703d8dac8
fix: [2.5] Set correct default configuration for task slots (#40821) 
issue: #39101

master pr: #39084

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2025-03-23 21:28:19 +08:00
cqy123456
7c716d61dc
fix:[2.5]wrong p.InterimIndexRefineQuantType default value and reduce ut run time (#40830) 
issue: https://github.com/milvus-io/milvus/issues/27678

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2025-03-21 19:10:13 +08:00
congqixia
cd8d7efc96
enhance: [2.5] Add an extra parameter for slow log (#40759) 
Cherry-pick from master
pr: #40758
Related to #40756

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2025-03-19 18:10:13 +08:00
cai.zhang
0cfe835cac
enhance: [2.5] Support slot for index task and stats task (#40288) 
issue: #39101 

master pr: #39084

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2025-03-19 16:50:11 +08:00
congqixia
e49294cd16
enhance: [2.5] Add buffer forwarder for stream delta loading (#40559) (#40699) 
Cherry-pick from master
pr: #40559
See also #40558
Related to #35303 & #38066 as well

This PR:
- Add `BufferedForward` to limit memory usage forwarding stream delete
- Add `UseLoad` flag to determine `Delete` shall use `segment.Delete` or
`segment.LoadDelta`
- Fix delegator accidentally use always true candidate while load
streaming delta

---------

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2025-03-19 10:28:14 +08:00
Xianhui Lin
705b3c90a5
fix: Failed to rolling upgrade from v2.5.6 to new 2.5 version when enable JsonKeyStats (#40661) 
fix: Failed to rolling upgrade from v2.5.6 to new 2.5 version when
enable JsonKeyStats.The reason is that the file path of the jsonkeyindex
has changed.
issue: https://github.com/milvus-io/milvus/issues/40649
，https://github.com/milvus-io/milvus/issues/40669
https://github.com/milvus-io/milvus/issues/40707
master-pr: https://github.com/milvus-io/milvus/pull/38039

---------

Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
 2025-03-18 17:32:16 +08:00
cqy123456
b8abd91962
enhance[2.5]: add more config for interimIndex to support more refine mode (#40429) 
issue: https://github.com/milvus-io/milvus/issues/27678

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2025-03-17 22:04:12 +08:00
Gao
09f7a60fe1
enhance: [2.5] make segment prune config refreshable (#40632) 
pr: #40539

Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2025-03-14 16:12:09 +08:00
SimFG
a3c60a03b5
feat: [2.5] add DDLDB rate type and related quota configurations (#40653) 
- issue: #40650
- pr: #40651

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2025-03-14 15:36:17 +08:00
zhagnlu
6b9e141ada
enhance: reorder sub expr for conjunct expr (#40186) 
pr:#39872

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2025-03-14 15:16:08 +08:00
congqixia
d8a2c1a907
enhance: [2.5] Add channel seal policy based on blocking l0 (#40505) (#40535) 
Cherry-pick from 2.5
pr: #40505
Related to #40502

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2025-03-14 10:02:12 +08:00
Xianhui Lin
f5e9dea2aa
fix: [2.5]fix the garbage cleanup logic of jsonkey stats && improve json key stats filer (#40039) 
fix: fix the garbage collection cleanup logic of jsonkey stats &&
improve json key stats filer
issue: https://github.com/milvus-io/milvus/issues/36995
https://github.com/milvus-io/milvus/issues/40034
https://github.com/milvus-io/milvus/issues/40041
https://github.com/milvus-io/milvus/issues/40106
https://github.com/milvus-io/milvus/issues/40138
pr: https://github.com/milvus-io/milvus/pull/38039

---------

Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
 2025-03-13 20:18:10 +08:00
cai.zhang
9cbab98382
enhance: [2.5] Export request timeout interval in config (#40118) 2025-02-23 13:47:12 +08:00
congqixia
709594f158
enhance: [2.5] Use v2 package name for pkg module (#40117) 
Cherry-pick from master
pr: #39990
Related to #39095

https://go.dev/doc/modules/version-numbers

Update pkg version according to golang dep version convention

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2025-02-23 00:46:01 +08:00
Xianhui Lin
c1de61ff7c
fix: [2.5]Replace the position of EnabledJSONKeyStats (#40108) 
Replace the position of EnabledJSONKeyStats
issue: https://github.com/milvus-io/milvus/issues/36995
pr: https://github.com/milvus-io/milvus/pull/38039

---------

Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
 2025-02-22 14:35:54 +08:00
congqixia
cb0dc6293c
enhance: [2.5] Sync proxy.maxTaskNum default value to 1024 (#40073) 
Cherry-pick from master
pr: #40072
Sync milvus.yaml value to default value for maxTaskNum

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2025-02-22 11:31:53 +08:00
Xianhui Lin
04175d8549
enhance: [2.5]compress jsonstatslog path only save filename (#39979) 
enhance: compress jsonstatslog path only save filename
issue: https://github.com/milvus-io/milvus/issues/36995
pr: https://github.com/milvus-io/milvus/pull/38039

Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
 2025-02-19 10:20:52 +08:00
zhagnlu
de8abe7970
fix:reset default auto index for varchar (#39908) 
pr: #39819

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2025-02-17 20:55:26 +08:00
cai.zhang
52434ccc78
enhance: [2.5] Limit the speed of the generating stats task (#39645) 
master pr: #39644

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2025-02-17 16:06:17 +08:00
XuanYang-cn
ee25af4c9b
enhance: Add configs for compaction schedule (#39010) (#39511) 
pr: #39010

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2025-02-17 11:44:15 +08:00
Xianhui Lin
f0964f769d
enhance: [2.5]Add json key inverted index in stats for optimization (#39876) 
Add json key inverted index in stats for optimization
issue: https://github.com/milvus-io/milvus/issues/36995
pr: https://github.com/milvus-io/milvus/pull/38039

---------

Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2025-02-16 20:12:15 +08:00
zhagnlu
16ea4fcb90
Revert "fix:reset default auto index type for scalar (#37086)" (#39820) 
This reverts commit eeb67a38450732eb8ab3a4eb4aaac6340142662.
pr: #39819

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2025-02-13 17:36:14 +08:00
wei liu
a50249e0bb
enhance: [2.5] dump pprof info if component stop progress timeout (#39760) 
issue: #39735
pr: #39726

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2025-02-13 15:30:47 +08:00
cqy123456
d121ac3a7a
enhance: [2.5]intermin index support different index type and more data type(fp16/bf16) (#39180) 
issue: https://github.com/milvus-io/milvus/issues/27678
related: https://github.com/milvus-io/milvus/pull/39753
some raw data status will change:
Intermin index has raw data: 
<html xmlns:v="urn:schemas-microsoft-com:vml"
xmlns:o="urn:schemas-microsoft-com:office:office"
xmlns:x="urn:schemas-microsoft-com:office:excel"
xmlns="http://www.w3.org/TR/REC-html40">

<head>

<meta name=ProgId content=Excel.Sheet>
<meta name=Generator content="Microsoft Excel 15">
<link id=Main-File rel=Main-File

href="file:////Users/cqy/Library/Group%20Containers/UBF8T346G9.Office/TemporaryItems/msohtmlclip/clip.htm">
<link rel=File-List

href="file:////Users/cqy/Library/Group%20Containers/UBF8T346G9.Office/TemporaryItems/msohtmlclip/clip_filelist.xml">
<!--table
	{mso-displayed-decimal-separator:"\.";
	mso-displayed-thousand-separator:"\,";}
@page
	{margin:.75in .7in .75in .7in;
	mso-header-margin:.3in;
	mso-footer-margin:.3in;}
.font5
	{color:windowtext;
	font-size:9.0pt;
	font-weight:400;
	font-style:normal;
	text-decoration:none;
	font-family:等线;
	mso-generic-font-family:auto;
	mso-font-charset:134;}
tr
	{mso-height-source:auto;
	mso-ruby-visibility:none;}
col
	{mso-width-source:auto;
	mso-ruby-visibility:none;}
br
	{mso-data-placement:same-cell;}
td
	{padding-top:1px;
	padding-right:1px;
	padding-left:1px;
	mso-ignore:padding;
	color:black;
	font-size:12.0pt;
	font-weight:400;
	font-style:normal;
	text-decoration:none;
	font-family:等线;
	mso-generic-font-family:auto;
	mso-font-charset:134;
	mso-number-format:General;
	text-align:general;
	vertical-align:middle;
	border:none;
	mso-background-source:auto;
	mso-pattern:auto;
	mso-protection:locked visible;
	white-space:nowrap;
	mso-rotate:0;}
ruby
	{ruby-align:left;}
rt
	{color:windowtext;
	font-size:9.0pt;
	font-weight:400;
	font-style:normal;
	text-decoration:none;
	font-family:等线;
	mso-generic-font-family:auto;
	mso-font-charset:134;
	mso-char-type:none;
	display:none;}
-->
</head>

<body link="#0563C1" vlink="#954F72">


sparse vector | growing segment | sealed segment
-- | -- | --
BM25 | no | no
IP | yes | no
  |   |  
dense vector | growing segment | sealed segment
ivf flat cc | yes | yes
scann_dvr | no | no



</body>

</html>

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2025-02-13 09:56:47 +08:00
SimFG
f542c866b3
fix: [2.5] root check the role list if rootShouldBindRole is true (#39714) 
- issue: #39712
- pr: #39713

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2025-02-11 10:10:51 +08:00
jaime
ddc5b299ad
enhance: expose more metrics data (#39466) 
issue: #36621 #39417
pr: #39456
1. Adjust the server-side cache size.
2. Add source information for configurations.
3. Add node ID for compaction and indexing tasks.
4. Resolve localhost access issues to fix health check failures for
etcd.

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2025-02-07 11:48:45 +08:00
congqixia
c32ec451b0
fix: [2.5] Use param item formatter to avoid SetConfig to overlay (#39597) (#39636) 
Cherry-pick from master
pr: #39597 
Related to #39596

When updating the build param configuration, the `Formatter` could be
used to do so and completed avoid touching the `overlay` config items

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2025-02-07 10:02:44 +08:00
Gao
f62eca09a0
enhance: set materializedView.enabled default to true (#39450) 
pr: #39449

Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2025-01-20 19:55:04 +08:00
congqixia
bb32bda423
enhance: [2.5] Sync multipleChunkEnabled default value & milvus yaml (#39373) 
Cherry-pick from master
pr: #39372 
The default value and yaml have different values which may cause
confusion when upgrading from older version.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2025-01-17 12:07:02 +08:00
foxspy
e7520599df
fix: [2.5] remove param num_build_thread (#39341) 
issue: #39090 
pr: #39340

Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
 2025-01-16 18:05:01 +08:00
yihao.dai
6773fb10a8
enhance: [2.5] Read metadata concurrently to accelerate recovery (#38900) 
Read metadata such as segments, binlogs, and partitions concurrently at
the collection level.

issue: https://github.com/milvus-io/milvus/issues/37630

pr: https://github.com/milvus-io/milvus/pull/38403

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2025-01-16 17:53:01 +08:00
yihao.dai
29dad64341
fix: [2.5] Fix consume blocked due to too many consumers (#38915) 
This PR limits the maximum number of consumers per pchannel to 10 for
each QueryNode and DataNode.

issue: https://github.com/milvus-io/milvus/issues/37630

pr: https://github.com/milvus-io/milvus/pull/38455

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2025-01-16 15:19:03 +08:00
SimFG
4cb47b3f67
feat: [2.5] root privileges can be customized (#39324) 
- issue: #39184
- pr: #39191

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2025-01-16 14:27:11 +08:00
congqixia
b658467107
enhance: [2.5] Unify LoadStateLock RLock & PinIf (#39206) (#39255) 
Cherry-pick from master
pr: #39206 #39308
Related to #39205

This PR merge `RLock` & `PinIfNotReleased` into `PinIf` function
preventing segment being released before any Read operation finished.

---------

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2025-01-16 00:33:03 +08:00
yihao.dai
5b0bb4c04e
enhance: [2.5] Reduce memory usage of BF in DataNode and QueryNode (#38913) 
1. DataNode: Skip generating BF during the insert phase (BF will be
regenerated during the sync phase).
2. QueryNode: Skip generating or maintaining BF for growing segments;
deletion checks will be handled in the segcore.

issue: https://github.com/milvus-io/milvus/issues/37630

pr: https://github.com/milvus-io/milvus/pull/38129

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2025-01-15 13:59:00 +08:00
yihao.dai
b91c0a8079
enhance: [2.5] Optimize GetLocalDiskSize and segment loader mutex (#38907) 
1. Make the segment loader lock protect only the resource.
2. Optimize GetDiskUsage to avoid excessive overhead.

issue: https://github.com/milvus-io/milvus/issues/37630

pr: https://github.com/milvus-io/milvus/pull/38599

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2025-01-14 20:43:06 +08:00
sthuang
4a47f760b3
fix: [2.5] rbac custom group privilege level check (#39164) (#39224) 
cherry-pick from master: https://github.com/milvus-io/milvus/pull/39164
related: https://github.com/milvus-io/milvus/issues/39086

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
 2025-01-14 16:50:59 +08:00
Zhen Ye
54036bcafd
enhance: add broadcast operation for msgstream (#39119) 
issue: #38399
pr: #39040

- make broadcast service available for msgstream by reusing the
architecture streaming service

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 2025-01-14 15:11:00 +08:00
Patrick Weizhi Xu
1d55ad6a1d
enhance: [2.5] add param for tuning max VARCHAR length and restore limit to 65535 (#38883) 
issue: #38882
pr: https://github.com/milvus-io/milvus/pull/38884

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
(cherry picked from commit a64a7377ea8258efe36c74bd1cf9db556726ba2d)
 2025-01-02 13:56:54 +08:00
sthuang
a4901ef7ec
fix: [2.5] fix privilege group list and list collections (#38738) 
cherry-pick from: https://github.com/milvus-io/milvus/pull/38684
related: https://github.com/milvus-io/milvus/issues/37031

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
 2024-12-25 18:06:50 +08:00
Zhen Ye
52de43dbeb
enhance: add multiply factor when loading index (#38722) 
issue: #38715
pr: #38716

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-12-25 10:50:58 +08:00
jaime
78438ef41e
fix: revert optimize CPU usage for CheckHealth requests (#35589) (#38555) 
issue: #35563

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-12-19 00:38:45 +08:00
jaime
29e620fa6d
fix: sync task still running after DataNode has stopped (#38377) 
issue: #38319

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-12-17 18:06:44 +08:00
jaime
28fdbc4e30
enhance: optimize CPU usage for CheckHealth requests (#35589) 
issue: #35563
1. Use an internal health checker to monitor the cluster's health state,
storing the latest state on the coordinator node. The CheckHealth
request retrieves the cluster's health from this latest state on the
proxy sides, which enhances cluster stability.
2. Each health check will assess all collections and channels, with
detailed failure messages temporarily saved in the latest state.
3. Use CheckHealth request instead of the heavy GetMetrics request on
the querynode and datanode

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-12-17 11:02:45 +08:00
SimFG
2afe2eaf3e
feat: support to replicate collection when the services contains the system tt msg (#37559) 
- issue: #37105

---------

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-12-17 09:08:46 +08:00
yihao.dai
a514f839b2
enhance: Speed up meta recovery (#38285) 
Increase the batchSize in WalkWithPrefix operations to 10000.

issue: https://github.com/milvus-io/milvus/issues/37630

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-12-12 10:58:43 +08:00
Ted Xu
dc85d8e968
enhance: improve mix compaction performance by removing max segment limitations (#38344) 
See #37234

---------

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
 2024-12-11 20:38:42 +08:00
wei liu
e279ccf109
enhance: Enable score based balance channel policy (#38143) 
issue: #38142
current balance channel policy only consider current collection's
distribution, so if all collections has 1 channel, and all channels has
been loaded on same querynode, after querynode num increase, balance
channel won't be triggered.

This PR enable score based balance channel policy, to achieve:
1. distribute all channels evenly across multiple querynodes
2. distribute each collection's channel evenly across multiple
querynodes.

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-12-11 17:20:43 +08:00