admin/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2026-02-02 01:06:41 +08:00
Code Issues Packages Projects Releases Wiki Activity
22,355 Commits 146 Branches 196 Tags
Commit Graph

363 Commits

Author SHA1 Message Date
sijie-ni-0214
13d7f6f86c
fix: Fix textlog load fail 2.5 (#46704) 
issue: https://github.com/milvus-io/milvus/issues/46678
master pr: https://github.com/milvus-io/milvus/pull/46697

Signed-off-by: sijie-ni-0214 <sijie.ni@zilliz.com>
 2026-01-05 15:53:23 +08:00
sijie-ni-0214
9d8edf9497
fix: [2.5] etcd RPC size limit exceeded when dropping collection (#46580) 
issue: https://github.com/milvus-io/milvus/issues/46410
master pr: https://github.com/milvus-io/milvus/pull/46414

Signed-off-by: sijie-ni-0214 <sijie.ni@zilliz.com>
 2025-12-29 15:57:21 +08:00
zhagnlu
78d70db6fd
fix: support skip load json stats when disable jsonstats (#45098) 
pr: #45101

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2025-10-28 10:50:11 +08:00
jiaqizho
00ef6032c6
enhance:[2.5] Introduce sparse filter in query (#44347) (#44790) 
pr: #44347

Signed-off-by: jiaqizho <jiaqi.zhou@zilliz.com>
 2025-10-14 15:02:01 +08:00
congqixia
7514eece4c
enhance: [2.5] Add mutex and range check preventing concurrent del (#44128) (#44202) 
Cherry-pick from master
pr: #44128
This PR adds a mutex prevent concurrent applying delete on same segment
and check latestDeltaTimestamp to skip overlapping delete range

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2025-09-04 10:35:54 +08:00
zhagnlu
6c29689ca2
enhance: support expr result cache (#43882) 
cherry-pick from pr: #43923

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2025-08-26 11:19:57 +08:00
Xianhui Lin
7141f96f00
fix: [2.5]add segment lock in LoadTextIndex and LoadJSONKeyIndex (#43815) 
fix: add segment lock in LoadTextIndex and LoadJSONKeyIndex
issue:https://github.com/milvus-io/milvus/issues/43572
pr: https://github.com/milvus-io/milvus/pull/43811

Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
 2025-08-18 10:21:57 +08:00
Chun Han
f033294dc1
fix: try to get span raw data for variable length data type(#43544) (#43703) 
related: #43544
pr: https://github.com/milvus-io/milvus/pull/43705

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
 2025-08-04 11:25:39 +08:00
wei liu
33e9b873de
enhance: add logs for count(*) (#43001) (#43249) 
pr: #43001

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2025-07-11 19:14:48 +08:00
Zhen Ye
a55c371261
fix: querynode upgrade from 2.5 get stucked (#42503) 
issue: #42492
pr: #42502

- querynode graceful stop can be done if there's only L0 segment exists.

Signed-off-by: chyezh <chyezh@outlook.com>
 2025-06-05 15:56:33 +08:00
SimFG
6e18ededab
fix: [2.5] mockery too unavailable after upgrade golang version (#41522) 
- issue: ##41291
- pr: #41481

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2025-04-25 14:40:40 +08:00
SimFG
18eb627533
fix: [2.5] Update logging context and upgrade dependencies (#41319) 
- issue: #41291
- pr: #41318

---------

Signed-off-by: SimFG <bang.fu@zilliz.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 2025-04-24 23:50:40 +08:00
Spade A
699f8a6ada
fix: [2.5] change log info to debug for collection ref (#41269) 
master: https://github.com/milvus-io/milvus/pull/41267
issue: https://github.com/milvus-io/milvus/issues/41268

---------

Signed-off-by: SpadeA <tangchenjie1210@gmail.com>
 2025-04-15 17:28:31 +08:00
liliu-z
cb0f984155
enhance: Revert "separate for index completed (#40873)" (#41152) 
This reverts commit 23e579e3240a30397f05f5b308be687f6f16b013. #40873

issue: #39519

Signed-off-by: Li Liu <li.liu@zilliz.com>
 2025-04-08 17:36:30 +08:00
Chun Han
23e579e324
separate for index completed (#40873) 
related: https://github.com/milvus-io/milvus/issues/40781

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
 2025-04-05 10:20:24 +08:00
Xianhui Lin
705b3c90a5
fix: Failed to rolling upgrade from v2.5.6 to new 2.5 version when enable JsonKeyStats (#40661) 
fix: Failed to rolling upgrade from v2.5.6 to new 2.5 version when
enable JsonKeyStats.The reason is that the file path of the jsonkeyindex
has changed.
issue: https://github.com/milvus-io/milvus/issues/40649
，https://github.com/milvus-io/milvus/issues/40669
https://github.com/milvus-io/milvus/issues/40707
master-pr: https://github.com/milvus-io/milvus/pull/38039

---------

Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
 2025-03-18 17:32:16 +08:00
Xianhui Lin
f5e9dea2aa
fix: [2.5]fix the garbage cleanup logic of jsonkey stats && improve json key stats filer (#40039) 
fix: fix the garbage collection cleanup logic of jsonkey stats &&
improve json key stats filer
issue: https://github.com/milvus-io/milvus/issues/36995
https://github.com/milvus-io/milvus/issues/40034
https://github.com/milvus-io/milvus/issues/40041
https://github.com/milvus-io/milvus/issues/40106
https://github.com/milvus-io/milvus/issues/40138
pr: https://github.com/milvus-io/milvus/pull/38039

---------

Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
 2025-03-13 20:18:10 +08:00
yihao.dai
cd8f1fe0e4
enhance: [2.5] Remove unnecessary collection and partition label from the metrics (#40593) 
/kind improvement

pr: https://github.com/milvus-io/milvus/pull/39536

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2025-03-13 12:24:08 +08:00
Bingyi Sun
683b26ffb7
feat: cherry pick json path index (#40313) 
issue: #35528 
pr: #36750 
this pr includes json path index pr and some related prs:
1. update tantivy version #39253 
2. json path index #36750 
3. fall back to brute force #40076 
4. term filter #40140 
5. bug fix #40336

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2025-03-10 22:14:05 +08:00
congqixia
709594f158
enhance: [2.5] Use v2 package name for pkg module (#40117) 
Cherry-pick from master
pr: #39990
Related to #39095

https://go.dev/doc/modules/version-numbers

Update pkg version according to golang dep version convention

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2025-02-23 00:46:01 +08:00
Xianhui Lin
f0964f769d
enhance: [2.5]Add json key inverted index in stats for optimization (#39876) 
Add json key inverted index in stats for optimization
issue: https://github.com/milvus-io/milvus/issues/36995
pr: https://github.com/milvus-io/milvus/pull/38039

---------

Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2025-02-16 20:12:15 +08:00
cqy123456
d121ac3a7a
enhance: [2.5]intermin index support different index type and more data type(fp16/bf16) (#39180) 
issue: https://github.com/milvus-io/milvus/issues/27678
related: https://github.com/milvus-io/milvus/pull/39753
some raw data status will change:
Intermin index has raw data: 
<html xmlns:v="urn:schemas-microsoft-com:vml"
xmlns:o="urn:schemas-microsoft-com:office:office"
xmlns:x="urn:schemas-microsoft-com:office:excel"
xmlns="http://www.w3.org/TR/REC-html40">

<head>

<meta name=ProgId content=Excel.Sheet>
<meta name=Generator content="Microsoft Excel 15">
<link id=Main-File rel=Main-File

href="file:////Users/cqy/Library/Group%20Containers/UBF8T346G9.Office/TemporaryItems/msohtmlclip/clip.htm">
<link rel=File-List

href="file:////Users/cqy/Library/Group%20Containers/UBF8T346G9.Office/TemporaryItems/msohtmlclip/clip_filelist.xml">
<!--table
	{mso-displayed-decimal-separator:"\.";
	mso-displayed-thousand-separator:"\,";}
@page
	{margin:.75in .7in .75in .7in;
	mso-header-margin:.3in;
	mso-footer-margin:.3in;}
.font5
	{color:windowtext;
	font-size:9.0pt;
	font-weight:400;
	font-style:normal;
	text-decoration:none;
	font-family:等线;
	mso-generic-font-family:auto;
	mso-font-charset:134;}
tr
	{mso-height-source:auto;
	mso-ruby-visibility:none;}
col
	{mso-width-source:auto;
	mso-ruby-visibility:none;}
br
	{mso-data-placement:same-cell;}
td
	{padding-top:1px;
	padding-right:1px;
	padding-left:1px;
	mso-ignore:padding;
	color:black;
	font-size:12.0pt;
	font-weight:400;
	font-style:normal;
	text-decoration:none;
	font-family:等线;
	mso-generic-font-family:auto;
	mso-font-charset:134;
	mso-number-format:General;
	text-align:general;
	vertical-align:middle;
	border:none;
	mso-background-source:auto;
	mso-pattern:auto;
	mso-protection:locked visible;
	white-space:nowrap;
	mso-rotate:0;}
ruby
	{ruby-align:left;}
rt
	{color:windowtext;
	font-size:9.0pt;
	font-weight:400;
	font-style:normal;
	text-decoration:none;
	font-family:等线;
	mso-generic-font-family:auto;
	mso-font-charset:134;
	mso-char-type:none;
	display:none;}
-->
</head>

<body link="#0563C1" vlink="#954F72">


sparse vector | growing segment | sealed segment
-- | -- | --
BM25 | no | no
IP | yes | no
  |   |  
dense vector | growing segment | sealed segment
ivf flat cc | yes | yes
scann_dvr | no | no



</body>

</html>

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2025-02-13 09:56:47 +08:00
congqixia
1b14f5a7ab
fix: [2.5] Segments return both growing&sealed result (#39789) 
Cherry-pick from master
pr: #39787
Previous PR #38311
Logic bug found in #38348

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2025-02-12 14:44:48 +08:00
congqixia
c0cc8a59f7
fix: [2.5] Return early when skip load pk index (#39763) 
Cherry pick from master
pr: #39762
Previous PR #39437 only print log and add index while load operation is
still executed. This PR return early when segment decides not to load PK
index.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2025-02-11 11:04:50 +08:00
cqy123456
758ac5a492
fix: [2.5]Adjust segment loader's memory estimate for intermin indexes (#39509) 
issue: https://github.com/milvus-io/milvus/issues/27678
related 2.4 pr: https://github.com/milvus-io/milvus/pull/39508
related 2.5 pr: https://github.com/milvus-io/milvus/pull/39509
related master pr: https://github.com/milvus-io/milvus/pull/39507

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2025-01-24 11:19:14 +08:00
Bingyi Sun
8475d8b193
fix: cherry pick warmup async (#39402) (#39474) 
related pr: https://github.com/milvus-io/milvus/pull/38690
issue: https://github.com/milvus-io/milvus/issues/38692

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2025-01-21 15:25:05 +08:00
aoiasd
f912cbf86d
enhance:[Cherry-pick] skip load bm25 sparse row data (#39207) 
pr: https://github.com/milvus-io/milvus/pull/39078

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2025-01-21 11:05:04 +08:00
congqixia
8e2e828a78
enhance: [2.5] Skip loading pk index for sorted segment in loader (#39438) 
Cherry pick from master
pr: #39437

Related to #39339
Previous PR #39389 only skips append index into segment

Also related to #39428

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2025-01-20 18:07:04 +08:00
yihao.dai
c945efa853
enhance: [2.5] Remove mutex from SegmentManger in querynode (#39051) (#39282) 
Remove mutex from SegmentManger in querynode to prevent mutex
contention.

issue: https://github.com/milvus-io/milvus/issues/37630

pr: https://github.com/milvus-io/milvus/pull/39051

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2025-01-16 13:27:02 +08:00
congqixia
21df11b37c
enhance: [2.5] Add try-catch and return CStatus for NewCollection (#39279) (#39303) 
Cherry pick from master
pr: #39279 
Related to #28795

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2025-01-16 10:13:06 +08:00
congqixia
b658467107
enhance: [2.5] Unify LoadStateLock RLock & PinIf (#39206) (#39255) 
Cherry-pick from master
pr: #39206 #39308
Related to #39205

This PR merge `RLock` & `PinIfNotReleased` into `PinIf` function
preventing segment being released before any Read operation finished.

---------

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2025-01-16 00:33:03 +08:00
yihao.dai
5b0bb4c04e
enhance: [2.5] Reduce memory usage of BF in DataNode and QueryNode (#38913) 
1. DataNode: Skip generating BF during the insert phase (BF will be
regenerated during the sync phase).
2. QueryNode: Skip generating or maintaining BF for growing segments;
deletion checks will be handled in the segcore.

issue: https://github.com/milvus-io/milvus/issues/37630

pr: https://github.com/milvus-io/milvus/pull/38129

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2025-01-15 13:59:00 +08:00
yihao.dai
b91c0a8079
enhance: [2.5] Optimize GetLocalDiskSize and segment loader mutex (#38907) 
1. Make the segment loader lock protect only the resource.
2. Optimize GetDiskUsage to avoid excessive overhead.

issue: https://github.com/milvus-io/milvus/issues/37630

pr: https://github.com/milvus-io/milvus/pull/38599

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2025-01-14 20:43:06 +08:00
Zhen Ye
adfc3f945e
enhance: record memory size (uncompressed) item for index (#38844) 
issue: #38715 
pr: #38770

- Current milvus use a serialized index size(compressed) for estimate
resource for loading.
- Add a new field MemSize (before compressing) for index to estimate
resource.

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 2025-01-14 10:33:06 +08:00
yihao.dai
8fadcde403
fix: [2.5] Fix incorrect memory estimation for small segments (#38909) 
Skip estimation index memory logic for segments without index file.

issue: https://github.com/milvus-io/milvus/issues/37630

pr: https://github.com/milvus-io/milvus/pull/38813

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2025-01-13 21:49:12 +08:00
Zhen Ye
95809ca767
enhance: make new go package to manage proto (#39128) 
issue: #39095
pr: #39114

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 2025-01-10 10:53:01 +08:00
Zhen Ye
04f07f00b7
fix: the retrieve plan on heap is used after free when reduce (#38841) 
issue: #38731
pr: #38840

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-12-30 15:52:51 +08:00
Zhen Ye
52de43dbeb
enhance: add multiply factor when loading index (#38722) 
issue: #38715
pr: #38716

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-12-25 10:50:58 +08:00
SimFG
2afe2eaf3e
feat: support to replicate collection when the services contains the system tt msg (#37559) 
- issue: #37105

---------

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-12-17 09:08:46 +08:00
tinswzy
27229f7907
enhance: refine exists log print with ctx (#38080) 
issue: #35917 
Refines exists log print with ctx

Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
 2024-12-14 22:36:44 +08:00
Gao
8977454311
enhance: support recall estimation (#38017) 
issue: #37899 
Only `search` api will be supported

---------

Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2024-12-11 20:40:48 +08:00
congqixia
7c55649585
enhance: Refine querynode collection number metrics (#38350) 
Related to #37630

Previously the loaded collection metrics was calculated via scanning all
loaded segment in segment manager, which is slow and buggy
implementation.

This PR:

- Move collection num metrics to collection manager
- Remove deprecated loaded partition metrics update logic

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-12-11 11:36:43 +08:00
congqixia
b8e3795374
enhance: Add secondary index for querynode segment manager (#38311) 
Related to #37630

Add secondary index with vchannel to reduce `GetBy` rlock holding time
when segment number is large.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-12-10 18:18:42 +08:00
congqixia
051bc280dd
enhance: Make dynamic load/release partition follow targets (#38059) 
Related to #37849

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-12-05 16:24:40 +08:00
congqixia
618f0cb728
enhance: Put release segment and other misc cgo call into pool (#38186) 
Related to #30273

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-12-05 11:04:40 +08:00
Zhen Ye
c6dcef7b84
enhance: move segcore codes of segment into one package (#37722) 
issue: #33285

- move most cgo opeartions related to search/query into segcore package
for reusing for streamingnode.
- add go unittest for segcore operations.

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-11-29 10:22:36 +08:00
congqixia
cb6542339e
enhance: Mark cgo thread with tag name (#38000) 
Related to #37999

This PR add `SetThreadName` API for marking cgo thread and utilize it
when initializing cgo worker.

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-11-26 11:22:35 +08:00
congqixia
1ed686783f
enhance: Use PrimaryKeys to replace interface slice for segment delete (#37880) 
Related to #35303

Reduce temporary memory usage for PK interface for segment delete.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-11-22 11:52:33 +08:00
congqixia
92e6ee6285
enhance: Use load pool for CreateTextIndex (#37898) 
Related to #37895

Only resolves the starving issue which caused goroutine leakage

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-11-22 10:06:33 +08:00
congqixia
ee54a98578
enhance: Add cgo call metrics for load/write API (#37405) 
Cgo API cost is not observerable since not metrics is related to them.
This PR add metrics for some sync cgo call related to load & write

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-11-07 10:06:25 +08:00