admin/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2025-12-07 17:48:29 +08:00
Code Issues Packages Projects Releases Wiki Activity
21,848 Commits 133 Branches 183 Tags
Commit Graph

664 Commits

Author SHA1 Message Date
Buqian Zheng
cff0e82f57
enhance: [2.5] improve sparse query nnz metric (#40714) 
add query type and field id label; add metric for hybrid search

issue: https://github.com/milvus-io/milvus/issues/35853
pr: https://github.com/milvus-io/milvus/pull/40713

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
 2025-03-20 14:14:21 +08:00
congqixia
e49294cd16
enhance: [2.5] Add buffer forwarder for stream delta loading (#40559) (#40699) 
Cherry-pick from master
pr: #40559
See also #40558
Related to #35303 & #38066 as well

This PR:
- Add `BufferedForward` to limit memory usage forwarding stream delete
- Add `UseLoad` flag to determine `Delete` shall use `segment.Delete` or
`segment.LoadDelta`
- Fix delegator accidentally use always true candidate while load
streaming delta

---------

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2025-03-19 10:28:14 +08:00
Xianhui Lin
705b3c90a5
fix: Failed to rolling upgrade from v2.5.6 to new 2.5 version when enable JsonKeyStats (#40661) 
fix: Failed to rolling upgrade from v2.5.6 to new 2.5 version when
enable JsonKeyStats.The reason is that the file path of the jsonkeyindex
has changed.
issue: https://github.com/milvus-io/milvus/issues/40649
，https://github.com/milvus-io/milvus/issues/40669
https://github.com/milvus-io/milvus/issues/40707
master-pr: https://github.com/milvus-io/milvus/pull/38039

---------

Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
 2025-03-18 17:32:16 +08:00
zhagnlu
6b9e141ada
enhance: reorder sub expr for conjunct expr (#40186) 
pr:#39872

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2025-03-14 15:16:08 +08:00
Xianhui Lin
f5e9dea2aa
fix: [2.5]fix the garbage cleanup logic of jsonkey stats && improve json key stats filer (#40039) 
fix: fix the garbage collection cleanup logic of jsonkey stats &&
improve json key stats filer
issue: https://github.com/milvus-io/milvus/issues/36995
https://github.com/milvus-io/milvus/issues/40034
https://github.com/milvus-io/milvus/issues/40041
https://github.com/milvus-io/milvus/issues/40106
https://github.com/milvus-io/milvus/issues/40138
pr: https://github.com/milvus-io/milvus/pull/38039

---------

Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
 2025-03-13 20:18:10 +08:00
yihao.dai
cd8f1fe0e4
enhance: [2.5] Remove unnecessary collection and partition label from the metrics (#40593) 
/kind improvement

pr: https://github.com/milvus-io/milvus/pull/39536

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2025-03-13 12:24:08 +08:00
Spade A
bcec7c5087
fix: [2.5] ref collection for search/query (#40550) 
ref https://github.com/milvus-io/milvus/issues/40473
cherry-pick: https://github.com/milvus-io/milvus/pull/40549

---------

Signed-off-by: SpadeA <tangchenjie1210@gmail.com>
 2025-03-12 20:12:09 +08:00
Bingyi Sun
683b26ffb7
feat: cherry pick json path index (#40313) 
issue: #35528 
pr: #36750 
this pr includes json path index pr and some related prs:
1. update tantivy version #39253 
2. json path index #36750 
3. fall back to brute force #40076 
4. term filter #40140 
5. bug fix #40336

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2025-03-10 22:14:05 +08:00
Xianhui Lin
a4eb2ce224
fix: [2.5]Revert qc statschecker for json key stats (#40125) 
Revert qc statschecker for json key stats
issue:https://github.com/milvus-io/milvus/issues/36995
pr:https://github.com/milvus-io/milvus/pull/39876

Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
 2025-02-24 13:31:55 +08:00
congqixia
709594f158
enhance: [2.5] Use v2 package name for pkg module (#40117) 
Cherry-pick from master
pr: #39990
Related to #39095

https://go.dev/doc/modules/version-numbers

Update pkg version according to golang dep version convention

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2025-02-23 00:46:01 +08:00
Xianhui Lin
c1de61ff7c
fix: [2.5]Replace the position of EnabledJSONKeyStats (#40108) 
Replace the position of EnabledJSONKeyStats
issue: https://github.com/milvus-io/milvus/issues/36995
pr: https://github.com/milvus-io/milvus/pull/38039

---------

Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
 2025-02-22 14:35:54 +08:00
congqixia
b7c631f0b7
enhance: [2.5] Remove frequent json stats distribution log (#40027) 
Info log is too frequent only printing the json stats & index info

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2025-02-20 13:33:57 +08:00
Zhen Ye
56c1a8d462
fix: ignore growing option is lost at hibridsearch (#39900) 
issue: #39892
pr: #39799

Signed-off-by: chyezh <chyezh@outlook.com>
 2025-02-17 15:30:15 +08:00
Xianhui Lin
f0964f769d
enhance: [2.5]Add json key inverted index in stats for optimization (#39876) 
Add json key inverted index in stats for optimization
issue: https://github.com/milvus-io/milvus/issues/36995
pr: https://github.com/milvus-io/milvus/pull/38039

---------

Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2025-02-16 20:12:15 +08:00
cqy123456
d121ac3a7a
enhance: [2.5]intermin index support different index type and more data type(fp16/bf16) (#39180) 
issue: https://github.com/milvus-io/milvus/issues/27678
related: https://github.com/milvus-io/milvus/pull/39753
some raw data status will change:
Intermin index has raw data: 
<html xmlns:v="urn:schemas-microsoft-com:vml"
xmlns:o="urn:schemas-microsoft-com:office:office"
xmlns:x="urn:schemas-microsoft-com:office:excel"
xmlns="http://www.w3.org/TR/REC-html40">

<head>

<meta name=ProgId content=Excel.Sheet>
<meta name=Generator content="Microsoft Excel 15">
<link id=Main-File rel=Main-File

href="file:////Users/cqy/Library/Group%20Containers/UBF8T346G9.Office/TemporaryItems/msohtmlclip/clip.htm">
<link rel=File-List

href="file:////Users/cqy/Library/Group%20Containers/UBF8T346G9.Office/TemporaryItems/msohtmlclip/clip_filelist.xml">
<!--table
	{mso-displayed-decimal-separator:"\.";
	mso-displayed-thousand-separator:"\,";}
@page
	{margin:.75in .7in .75in .7in;
	mso-header-margin:.3in;
	mso-footer-margin:.3in;}
.font5
	{color:windowtext;
	font-size:9.0pt;
	font-weight:400;
	font-style:normal;
	text-decoration:none;
	font-family:等线;
	mso-generic-font-family:auto;
	mso-font-charset:134;}
tr
	{mso-height-source:auto;
	mso-ruby-visibility:none;}
col
	{mso-width-source:auto;
	mso-ruby-visibility:none;}
br
	{mso-data-placement:same-cell;}
td
	{padding-top:1px;
	padding-right:1px;
	padding-left:1px;
	mso-ignore:padding;
	color:black;
	font-size:12.0pt;
	font-weight:400;
	font-style:normal;
	text-decoration:none;
	font-family:等线;
	mso-generic-font-family:auto;
	mso-font-charset:134;
	mso-number-format:General;
	text-align:general;
	vertical-align:middle;
	border:none;
	mso-background-source:auto;
	mso-pattern:auto;
	mso-protection:locked visible;
	white-space:nowrap;
	mso-rotate:0;}
ruby
	{ruby-align:left;}
rt
	{color:windowtext;
	font-size:9.0pt;
	font-weight:400;
	font-style:normal;
	text-decoration:none;
	font-family:等线;
	mso-generic-font-family:auto;
	mso-font-charset:134;
	mso-char-type:none;
	display:none;}
-->
</head>

<body link="#0563C1" vlink="#954F72">


sparse vector | growing segment | sealed segment
-- | -- | --
BM25 | no | no
IP | yes | no
  |   |  
dense vector | growing segment | sealed segment
ivf flat cc | yes | yes
scann_dvr | no | no



</body>

</html>

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2025-02-13 09:56:47 +08:00
congqixia
1b14f5a7ab
fix: [2.5] Segments return both growing&sealed result (#39789) 
Cherry-pick from master
pr: #39787
Previous PR #38311
Logic bug found in #38348

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2025-02-12 14:44:48 +08:00
congqixia
c0cc8a59f7
fix: [2.5] Return early when skip load pk index (#39763) 
Cherry pick from master
pr: #39762
Previous PR #39437 only print log and add index while load operation is
still executed. This PR return early when segment decides not to load PK
index.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2025-02-11 11:04:50 +08:00
cqy123456
758ac5a492
fix: [2.5]Adjust segment loader's memory estimate for intermin indexes (#39509) 
issue: https://github.com/milvus-io/milvus/issues/27678
related 2.4 pr: https://github.com/milvus-io/milvus/pull/39508
related 2.5 pr: https://github.com/milvus-io/milvus/pull/39509
related master pr: https://github.com/milvus-io/milvus/pull/39507

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2025-01-24 11:19:14 +08:00
Bingyi Sun
8475d8b193
fix: cherry pick warmup async (#39402) (#39474) 
related pr: https://github.com/milvus-io/milvus/pull/38690
issue: https://github.com/milvus-io/milvus/issues/38692

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2025-01-21 15:25:05 +08:00
aoiasd
f912cbf86d
enhance:[Cherry-pick] skip load bm25 sparse row data (#39207) 
pr: https://github.com/milvus-io/milvus/pull/39078

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2025-01-21 11:05:04 +08:00
congqixia
8e2e828a78
enhance: [2.5] Skip loading pk index for sorted segment in loader (#39438) 
Cherry pick from master
pr: #39437

Related to #39339
Previous PR #39389 only skips append index into segment

Also related to #39428

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2025-01-20 18:07:04 +08:00
yihao.dai
c945efa853
enhance: [2.5] Remove mutex from SegmentManger in querynode (#39051) (#39282) 
Remove mutex from SegmentManger in querynode to prevent mutex
contention.

issue: https://github.com/milvus-io/milvus/issues/37630

pr: https://github.com/milvus-io/milvus/pull/39051

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2025-01-16 13:27:02 +08:00
congqixia
21df11b37c
enhance: [2.5] Add try-catch and return CStatus for NewCollection (#39279) (#39303) 
Cherry pick from master
pr: #39279 
Related to #28795

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2025-01-16 10:13:06 +08:00
congqixia
b658467107
enhance: [2.5] Unify LoadStateLock RLock & PinIf (#39206) (#39255) 
Cherry-pick from master
pr: #39206 #39308
Related to #39205

This PR merge `RLock` & `PinIfNotReleased` into `PinIf` function
preventing segment being released before any Read operation finished.

---------

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2025-01-16 00:33:03 +08:00
yihao.dai
5b0bb4c04e
enhance: [2.5] Reduce memory usage of BF in DataNode and QueryNode (#38913) 
1. DataNode: Skip generating BF during the insert phase (BF will be
regenerated during the sync phase).
2. QueryNode: Skip generating or maintaining BF for growing segments;
deletion checks will be handled in the segcore.

issue: https://github.com/milvus-io/milvus/issues/37630

pr: https://github.com/milvus-io/milvus/pull/38129

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2025-01-15 13:59:00 +08:00
cai.zhang
4270174899
fix: [2.5] Add scalar index engine version for compatibility (#39236) 
issue: #39203 

master pr: #39204

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2025-01-14 21:01:01 +08:00
yihao.dai
b91c0a8079
enhance: [2.5] Optimize GetLocalDiskSize and segment loader mutex (#38907) 
1. Make the segment loader lock protect only the resource.
2. Optimize GetDiskUsage to avoid excessive overhead.

issue: https://github.com/milvus-io/milvus/issues/37630

pr: https://github.com/milvus-io/milvus/pull/38599

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2025-01-14 20:43:06 +08:00
Zhen Ye
adfc3f945e
enhance: record memory size (uncompressed) item for index (#38844) 
issue: #38715 
pr: #38770

- Current milvus use a serialized index size(compressed) for estimate
resource for loading.
- Add a new field MemSize (before compressing) for index to estimate
resource.

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 2025-01-14 10:33:06 +08:00
yihao.dai
8fadcde403
fix: [2.5] Fix incorrect memory estimation for small segments (#38909) 
Skip estimation index memory logic for segments without index file.

issue: https://github.com/milvus-io/milvus/issues/37630

pr: https://github.com/milvus-io/milvus/pull/38813

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2025-01-13 21:49:12 +08:00
Zhen Ye
95809ca767
enhance: make new go package to manage proto (#39128) 
issue: #39095
pr: #39114

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 2025-01-10 10:53:01 +08:00
jaime
0693634f62
enhance: add db name in replica description (#38673) 
issue: #36621
pr: #38672

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2025-01-09 19:43:04 +08:00
cai.zhang
b9da6a58e2
enhance: [2.5]Return collection not loaded rather than not found on querynode (#38593) (#38918) 
issue: #38586

master pr: #38593

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2025-01-05 18:55:00 +08:00
Gao
018fbb48b1
fix: search req transfer correctly (#38954) 
issue: #37899 
pr: #38955

Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2025-01-03 10:02:54 +08:00
Zhen Ye
04f07f00b7
fix: the retrieve plan on heap is used after free when reduce (#38841) 
issue: #38731
pr: #38840

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-12-30 15:52:51 +08:00
Zhen Ye
52de43dbeb
enhance: add multiply factor when loading index (#38722) 
issue: #38715
pr: #38716

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-12-25 10:50:58 +08:00
jaime
78438ef41e
fix: revert optimize CPU usage for CheckHealth requests (#35589) (#38555) 
issue: #35563

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-12-19 00:38:45 +08:00
jaime
28fdbc4e30
enhance: optimize CPU usage for CheckHealth requests (#35589) 
issue: #35563
1. Use an internal health checker to monitor the cluster's health state,
storing the latest state on the coordinator node. The CheckHealth
request retrieves the cluster's health from this latest state on the
proxy sides, which enhances cluster stability.
2. Each health check will assess all collections and channels, with
detailed failure messages temporarily saved in the latest state.
3. Use CheckHealth request instead of the heavy GetMetrics request on
the querynode and datanode

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-12-17 11:02:45 +08:00
SimFG
2afe2eaf3e
feat: support to replicate collection when the services contains the system tt msg (#37559) 
- issue: #37105

---------

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-12-17 09:08:46 +08:00
congqixia
9c8c1b3bb7
enhance: Remove levelZeroMut totally (#38473) 
The level zero mutex could be remove since all operations are guarded by
segment manager mutex

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-12-16 14:38:45 +08:00
tinswzy
27229f7907
enhance: refine exists log print with ctx (#38080) 
issue: #35917 
Refines exists log print with ctx

Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
 2024-12-14 22:36:44 +08:00
congqixia
10460ed3f8
enhance: Simplify querynode tsafe & reduce goroutine number (#38416) 
Related to #37630

TSafe manager is too complex for current implementation and each
delegator need one goroutine waiting for tsafe update event.

Tsafe updating could be executed in pipeline. This PR remove tsafe
manager and simplify the entire logic of tsafe updating.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-12-13 10:56:43 +08:00
Buqian Zheng
75e64b993f
enhance: add metrics for counting number of nun-zeros/tokens of sparse/FTS search (#38329) 
sparse vectors may have arbitrary number of non zeros and it is hard to
optimize without knowing the actual distribution of nnz. this PR adds a
metric for analyzing that.

issue: https://github.com/milvus-io/milvus/issues/35853

comparing with https://github.com/milvus-io/milvus/pull/38328, this
includes also metric for FTS in query node delegator

also fixed a bug of sparse when searching by pk

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
 2024-12-12 16:22:43 +08:00
Gao
8977454311
enhance: support recall estimation (#38017) 
issue: #37899 
Only `search` api will be supported

---------

Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2024-12-11 20:40:48 +08:00
congqixia
7c55649585
enhance: Refine querynode collection number metrics (#38350) 
Related to #37630

Previously the loaded collection metrics was calculated via scanning all
loaded segment in segment manager, which is slow and buggy
implementation.

This PR:

- Move collection num metrics to collection manager
- Remove deprecated loaded partition metrics update logic

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-12-11 11:36:43 +08:00
congqixia
7ea9c983d2
enhance: Add mockery package config for QC&QN (#38340) 
Related to #38339

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-12-10 19:18:42 +08:00
congqixia
b8e3795374
enhance: Add secondary index for querynode segment manager (#38311) 
Related to #37630

Add secondary index with vchannel to reduce `GetBy` rlock holding time
when segment number is large.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-12-10 18:18:42 +08:00
wei liu
28c5189c6d
enhance: Refine the error msg for filter node (#38278) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-12-06 18:22:40 +08:00
jaime
8ed019735c
enhance: add disk stats within system metrics (#38033) 
issue: ##36621

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-12-06 16:32:41 +08:00
congqixia
051bc280dd
enhance: Make dynamic load/release partition follow targets (#38059) 
Related to #37849

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-12-05 16:24:40 +08:00
congqixia
618f0cb728
enhance: Put release segment and other misc cgo call into pool (#38186) 
Related to #30273

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-12-05 11:04:40 +08:00