admin/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2026-02-02 01:06:41 +08:00
Code Issues Packages Projects Releases Wiki Activity
21,702 Commits 146 Branches 196 Tags
Commit Graph

1100 Commits

Author SHA1 Message Date
zhagnlu
de8abe7970
fix:reset default auto index for varchar (#39908) 
pr: #39819

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2025-02-17 20:55:26 +08:00
cai.zhang
52434ccc78
enhance: [2.5] Limit the speed of the generating stats task (#39645) 
master pr: #39644

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2025-02-17 16:06:17 +08:00
XuanYang-cn
8067113133
enhance: [cp25]Enable to observe write amplification (#39661) (#39743) 
pr: #39661

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2025-02-17 16:00:17 +08:00
Zhen Ye
56c1a8d462
fix: ignore growing option is lost at hibridsearch (#39900) 
issue: #39892
pr: #39799

Signed-off-by: chyezh <chyezh@outlook.com>
 2025-02-17 15:30:15 +08:00
XuanYang-cn
ee25af4c9b
enhance: Add configs for compaction schedule (#39010) (#39511) 
pr: #39010

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2025-02-17 11:44:15 +08:00
Xianhui Lin
f0964f769d
enhance: [2.5]Add json key inverted index in stats for optimization (#39876) 
Add json key inverted index in stats for optimization
issue: https://github.com/milvus-io/milvus/issues/36995
pr: https://github.com/milvus-io/milvus/pull/38039

---------

Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2025-02-16 20:12:15 +08:00
zhagnlu
16ea4fcb90
Revert "fix:reset default auto index type for scalar (#37086)" (#39820) 
This reverts commit eeb67a38450732eb8ab3a4eb4aaac6340142662.
pr: #39819

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2025-02-13 17:36:14 +08:00
wei liu
a50249e0bb
enhance: [2.5] dump pprof info if component stop progress timeout (#39760) 
issue: #39735
pr: #39726

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2025-02-13 15:30:47 +08:00
cqy123456
d121ac3a7a
enhance: [2.5]intermin index support different index type and more data type(fp16/bf16) (#39180) 
issue: https://github.com/milvus-io/milvus/issues/27678
related: https://github.com/milvus-io/milvus/pull/39753
some raw data status will change:
Intermin index has raw data: 
<html xmlns:v="urn:schemas-microsoft-com:vml"
xmlns:o="urn:schemas-microsoft-com:office:office"
xmlns:x="urn:schemas-microsoft-com:office:excel"
xmlns="http://www.w3.org/TR/REC-html40">

<head>

<meta name=ProgId content=Excel.Sheet>
<meta name=Generator content="Microsoft Excel 15">
<link id=Main-File rel=Main-File

href="file:////Users/cqy/Library/Group%20Containers/UBF8T346G9.Office/TemporaryItems/msohtmlclip/clip.htm">
<link rel=File-List

href="file:////Users/cqy/Library/Group%20Containers/UBF8T346G9.Office/TemporaryItems/msohtmlclip/clip_filelist.xml">
<!--table
	{mso-displayed-decimal-separator:"\.";
	mso-displayed-thousand-separator:"\,";}
@page
	{margin:.75in .7in .75in .7in;
	mso-header-margin:.3in;
	mso-footer-margin:.3in;}
.font5
	{color:windowtext;
	font-size:9.0pt;
	font-weight:400;
	font-style:normal;
	text-decoration:none;
	font-family:等线;
	mso-generic-font-family:auto;
	mso-font-charset:134;}
tr
	{mso-height-source:auto;
	mso-ruby-visibility:none;}
col
	{mso-width-source:auto;
	mso-ruby-visibility:none;}
br
	{mso-data-placement:same-cell;}
td
	{padding-top:1px;
	padding-right:1px;
	padding-left:1px;
	mso-ignore:padding;
	color:black;
	font-size:12.0pt;
	font-weight:400;
	font-style:normal;
	text-decoration:none;
	font-family:等线;
	mso-generic-font-family:auto;
	mso-font-charset:134;
	mso-number-format:General;
	text-align:general;
	vertical-align:middle;
	border:none;
	mso-background-source:auto;
	mso-pattern:auto;
	mso-protection:locked visible;
	white-space:nowrap;
	mso-rotate:0;}
ruby
	{ruby-align:left;}
rt
	{color:windowtext;
	font-size:9.0pt;
	font-weight:400;
	font-style:normal;
	text-decoration:none;
	font-family:等线;
	mso-generic-font-family:auto;
	mso-font-charset:134;
	mso-char-type:none;
	display:none;}
-->
</head>

<body link="#0563C1" vlink="#954F72">


sparse vector | growing segment | sealed segment
-- | -- | --
BM25 | no | no
IP | yes | no
  |   |  
dense vector | growing segment | sealed segment
ivf flat cc | yes | yes
scann_dvr | no | no



</body>

</html>

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2025-02-13 09:56:47 +08:00
cai.zhang
667c84740c
enhance: [2.5] Add metrics for parse expression (#39716) 
master pr: #39654

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2025-02-12 10:30:46 +08:00
yihao.dai
c2986d4ab9
fix: [2.5] Fix concurrent map (#39776) 
issue: https://github.com/milvus-io/milvus/issues/39778

pr: https://github.com/milvus-io/milvus/pull/39775

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2025-02-11 22:12:46 +08:00
SimFG
f542c866b3
fix: [2.5] root check the role list if rootShouldBindRole is true (#39714) 
- issue: #39712
- pr: #39713

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2025-02-11 10:10:51 +08:00
yihao.dai
4a8e6fc59c
fix: [2.5] Fix flowgraph leak (#39686) 
Consider lag targets when determining whether to remove the dispatcher
manager.

issue: https://github.com/milvus-io/milvus/issues/39642

pr: https://github.com/milvus-io/milvus/pull/39656

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2025-02-08 15:40:45 +08:00
Zhen Ye
858dc10ef9
enhance: broadcast with event-based notification (#39550) 
issue: #38399
pr: #39522

- broadcast message can carry multi resource key now.
- implement event-based notification for broadcast messages
- broadcast message use broadcast id as a unique identifier in message
- broadcasted message on vchannels keep the broadcasted vchannel now.
- broadcasted message and broadcast message have a common broadcast
header now.

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 2025-02-07 11:50:50 +08:00
jaime
ddc5b299ad
enhance: expose more metrics data (#39466) 
issue: #36621 #39417
pr: #39456
1. Adjust the server-side cache size.
2. Add source information for configurations.
3. Add node ID for compaction and indexing tasks.
4. Resolve localhost access issues to fix health check failures for
etcd.

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2025-02-07 11:48:45 +08:00
congqixia
c32ec451b0
fix: [2.5] Use param item formatter to avoid SetConfig to overlay (#39597) (#39636) 
Cherry-pick from master
pr: #39597 
Related to #39596

When updating the build param configuration, the `Formatter` could be
used to do so and completed avoid touching the `overlay` config items

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2025-02-07 10:02:44 +08:00
SimFG
30411d6d3a
fix: [2.5] deny to set the mmap param for the alter index api (#39520) 
- issue: #39517
- pr: #39518

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2025-01-22 23:55:06 +08:00
Xianhui Lin
20c6de156e
feat: update milvusproto version to 2.5.5 (#39478) 
feat: update milvusproto version to 2.5
issue: https://github.com/milvus-io/milvus/issues/39093

Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
 2025-01-21 15:11:05 +08:00
Xianhui Lin
3b3959c625
enhance: [2.5]Add schema update time verification for insert and upsert to use cache (#39382) 
enhance: Add schema update time verification for insert and upsert to
use cache
issue: https://github.com/milvus-io/milvus/issues/39093
Related to pr: https://github.com/milvus-io/milvus/pull/39096

---------

Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
 2025-01-21 10:19:11 +08:00
aoiasd
15d60c60c2
enhance: [2.5] remain token after authorize (#39455) 
relate: https://github.com/milvus-io/milvus/issues/39285

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2025-01-20 20:27:04 +08:00
Gao
f62eca09a0
enhance: set materializedView.enabled default to true (#39450) 
pr: #39449

Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2025-01-20 19:55:04 +08:00
Duc Le Tu
3893f24323
fix: apply custom producer config for kafkaHealthCheck (#39364) 
issue: https://github.com/milvus-io/milvus/issues/39287
pr: https://github.com/milvus-io/milvus/pull/39283

Signed-off-by: DLT1412 <tuduc93@gmail.com>
 2025-01-20 17:27:04 +08:00
congqixia
bb32bda423
enhance: [2.5] Sync multipleChunkEnabled default value & milvus yaml (#39373) 
Cherry-pick from master
pr: #39372 
The default value and yaml have different values which may cause
confusion when upgrading from older version.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2025-01-17 12:07:02 +08:00
foxspy
e7520599df
fix: [2.5] remove param num_build_thread (#39341) 
issue: #39090 
pr: #39340

Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
 2025-01-16 18:05:01 +08:00
yihao.dai
6773fb10a8
enhance: [2.5] Read metadata concurrently to accelerate recovery (#38900) 
Read metadata such as segments, binlogs, and partitions concurrently at
the collection level.

issue: https://github.com/milvus-io/milvus/issues/37630

pr: https://github.com/milvus-io/milvus/pull/38403

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2025-01-16 17:53:01 +08:00
yihao.dai
9d2a0e775c
fix: [2.5] Fix slow dist handle and slow observe (#38905) 
1. Provide partition&channel level indexing in the collection target.
2. Make SegmentAction not wait for distribution.
3. Remove scheduler and target manager mutex
4. Optimize logging to reduce CPU overhead.

issue: https://github.com/milvus-io/milvus/issues/37630

pr: https://github.com/milvus-io/milvus/pull/38566

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2025-01-16 17:07:02 +08:00
congqixia
e68d161e60
enhance: Bump milvus & proto version to v2.5.4 (#39330) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2025-01-16 15:39:03 +08:00
yihao.dai
29dad64341
fix: [2.5] Fix consume blocked due to too many consumers (#38915) 
This PR limits the maximum number of consumers per pchannel to 10 for
each QueryNode and DataNode.

issue: https://github.com/milvus-io/milvus/issues/37630

pr: https://github.com/milvus-io/milvus/pull/38455

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2025-01-16 15:19:03 +08:00
SimFG
4cb47b3f67
feat: [2.5] root privileges can be customized (#39324) 
- issue: #39184
- pr: #39191

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2025-01-16 14:27:11 +08:00
smellthemoon
1602390734
enhance: support null expr(#38772) (#38931) 
pr: #38772 
issue: #31728

---------

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2025-01-16 10:55:07 +08:00
congqixia
b658467107
enhance: [2.5] Unify LoadStateLock RLock & PinIf (#39206) (#39255) 
Cherry-pick from master
pr: #39206 #39308
Related to #39205

This PR merge `RLock` & `PinIfNotReleased` into `PinIf` function
preventing segment being released before any Read operation finished.

---------

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2025-01-16 00:33:03 +08:00
yihao.dai
5b0bb4c04e
enhance: [2.5] Reduce memory usage of BF in DataNode and QueryNode (#38913) 
1. DataNode: Skip generating BF during the insert phase (BF will be
regenerated during the sync phase).
2. QueryNode: Skip generating or maintaining BF for growing segments;
deletion checks will be handled in the segcore.

issue: https://github.com/milvus-io/milvus/issues/37630

pr: https://github.com/milvus-io/milvus/pull/38129

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2025-01-15 13:59:00 +08:00
Zhen Ye
74246d8167
fix: wrong streaming mockery package name (#39261) 
issue: #39095
pr: #39260

Signed-off-by: chyezh <chyezh@outlook.com>
 2025-01-15 11:44:59 +08:00
cai.zhang
4270174899
fix: [2.5] Add scalar index engine version for compatibility (#39236) 
issue: #39203 

master pr: #39204

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2025-01-14 21:01:01 +08:00
yihao.dai
b91c0a8079
enhance: [2.5] Optimize GetLocalDiskSize and segment loader mutex (#38907) 
1. Make the segment loader lock protect only the resource.
2. Optimize GetDiskUsage to avoid excessive overhead.

issue: https://github.com/milvus-io/milvus/issues/37630

pr: https://github.com/milvus-io/milvus/pull/38599

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2025-01-14 20:43:06 +08:00
sthuang
4a47f760b3
fix: [2.5] rbac custom group privilege level check (#39164) (#39224) 
cherry-pick from master: https://github.com/milvus-io/milvus/pull/39164
related: https://github.com/milvus-io/milvus/issues/39086

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
 2025-01-14 16:50:59 +08:00
zhenshan.cao
99a8274326
enhance: Add primary field name in SearchResult and QueryResults (#39222) 
pr: https://github.com/milvus-io/milvus/pull/39220
issue: https://github.com/milvus-io/milvus/issues/39219

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2025-01-14 16:10:58 +08:00
Zhen Ye
54036bcafd
enhance: add broadcast operation for msgstream (#39119) 
issue: #38399
pr: #39040

- make broadcast service available for msgstream by reusing the
architecture streaming service

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 2025-01-14 15:11:00 +08:00
SimFG
7c8db6f61f
fix: [2.5] use the object heap to keep the min ddl ts order (#39195) 
- issue: #39002
- pr: #39118

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2025-01-14 14:23:05 +08:00
Zhen Ye
adfc3f945e
enhance: record memory size (uncompressed) item for index (#38844) 
issue: #38715 
pr: #38770

- Current milvus use a serialized index size(compressed) for estimate
resource for loading.
- Add a new field MemSize (before compressing) for index to estimate
resource.

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 2025-01-14 10:33:06 +08:00
congqixia
ac730be5aa
enhance: Bump milvus & proto version to v2.5.3 (#39198) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2025-01-13 12:56:57 +08:00
tinswzy
39fa00fc1a
enhance: add intent ctx to rootcoord init (#38948) 
issue: #38439 
cherry pick "enhance: add intent ctx to rootcoord init"

Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
 2025-01-11 23:54:57 +08:00
congqixia
6b127d4f5f
fix: [2.5] Return io error other than NotExist refreshing config (#38924) (#39072) 
Cherry pick from master
pr: #38924
Related to #38923

This PR:

- Check whether `os.Stat` config file error is io.ErrNotExist
- Panic when get config return error during Milvus initialization

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2025-01-10 14:09:04 +08:00
Zhen Ye
95809ca767
enhance: make new go package to manage proto (#39128) 
issue: #39095
pr: #39114

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 2025-01-10 10:53:01 +08:00
jaime
0693634f62
enhance: add db name in replica description (#38673) 
issue: #36621
pr: #38672

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2025-01-09 19:43:04 +08:00
Zhen Ye
f2e581e2d7
enhance: add broadcast for streaming service (#39047) 
issue: #38399
pr: #39020

- Add new rpc for transfer broadcast to streaming coord
- Add broadcast service at streaming coord to make broadcast message
sent automicly

also cherry pick the pr #38400

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 2025-01-09 16:46:56 +08:00
Zhen Ye
6f1febe881
enhance: move streaming coord from datacoord to rootcoord (#39009) 
issue: #38399
pr: #39007

We want to support broadcast operation for both streaming and msgstream.
But msgstream can be only sent message from rootcoord and proxy.
So this pr move the streamingcoord to rootcoord to make easier
implementation.

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 2025-01-07 17:56:56 +08:00
XuanYang-cn
b457c2f415
enhance: [2.5]Add missing delete metrics (#38634) (#38747) 
Add 2 counter metrics:
- Total delete entries from deltalog:
milvus_datanode_compaction_delete_count
- Total missing deletes: milvus_datanode_compaction_missing_delete_count

See also: #34665
pr: #38634

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2025-01-07 11:20:56 +08:00
congqixia
6433f76c61
enhance: [2.5] Bump golang/x/net to v0.33 fixing security alarm (#38850) (#38932) 
Cherry-pick from master
pr: #38850
Related to https://github.com/milvus-io/milvus/security/dependabot/106

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2025-01-03 14:50:54 +08:00
jaime
5fb8313d6f
enhance: excessive disk usage logs generated if the path does not exist (#38822) 
issue: #38820
pr: #38821

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2025-01-02 14:28:53 +08:00