admin/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2025-12-07 01:28:27 +08:00
Code Issues Packages Projects Releases Wiki Activity
21,848 Commits 133 Branches 183 Tags
Commit Graph

96 Commits

Author SHA1 Message Date
Xianhui Lin
705b3c90a5
fix: Failed to rolling upgrade from v2.5.6 to new 2.5 version when enable JsonKeyStats (#40661) 
fix: Failed to rolling upgrade from v2.5.6 to new 2.5 version when
enable JsonKeyStats.The reason is that the file path of the jsonkeyindex
has changed.
issue: https://github.com/milvus-io/milvus/issues/40649
，https://github.com/milvus-io/milvus/issues/40669
https://github.com/milvus-io/milvus/issues/40707
master-pr: https://github.com/milvus-io/milvus/pull/38039

---------

Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
 2025-03-18 17:32:16 +08:00
zhagnlu
6b9e141ada
enhance: reorder sub expr for conjunct expr (#40186) 
pr:#39872

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2025-03-14 15:16:08 +08:00
Xianhui Lin
f5e9dea2aa
fix: [2.5]fix the garbage cleanup logic of jsonkey stats && improve json key stats filer (#40039) 
fix: fix the garbage collection cleanup logic of jsonkey stats &&
improve json key stats filer
issue: https://github.com/milvus-io/milvus/issues/36995
https://github.com/milvus-io/milvus/issues/40034
https://github.com/milvus-io/milvus/issues/40041
https://github.com/milvus-io/milvus/issues/40106
https://github.com/milvus-io/milvus/issues/40138
pr: https://github.com/milvus-io/milvus/pull/38039

---------

Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
 2025-03-13 20:18:10 +08:00
congqixia
709594f158
enhance: [2.5] Use v2 package name for pkg module (#40117) 
Cherry-pick from master
pr: #39990
Related to #39095

https://go.dev/doc/modules/version-numbers

Update pkg version according to golang dep version convention

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2025-02-23 00:46:01 +08:00
Xianhui Lin
c1de61ff7c
fix: [2.5]Replace the position of EnabledJSONKeyStats (#40108) 
Replace the position of EnabledJSONKeyStats
issue: https://github.com/milvus-io/milvus/issues/36995
pr: https://github.com/milvus-io/milvus/pull/38039

---------

Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
 2025-02-22 14:35:54 +08:00
Xianhui Lin
f0964f769d
enhance: [2.5]Add json key inverted index in stats for optimization (#39876) 
Add json key inverted index in stats for optimization
issue: https://github.com/milvus-io/milvus/issues/36995
pr: https://github.com/milvus-io/milvus/pull/38039

---------

Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2025-02-16 20:12:15 +08:00
cqy123456
d121ac3a7a
enhance: [2.5]intermin index support different index type and more data type(fp16/bf16) (#39180) 
issue: https://github.com/milvus-io/milvus/issues/27678
related: https://github.com/milvus-io/milvus/pull/39753
some raw data status will change:
Intermin index has raw data: 
<html xmlns:v="urn:schemas-microsoft-com:vml"
xmlns:o="urn:schemas-microsoft-com:office:office"
xmlns:x="urn:schemas-microsoft-com:office:excel"
xmlns="http://www.w3.org/TR/REC-html40">

<head>

<meta name=ProgId content=Excel.Sheet>
<meta name=Generator content="Microsoft Excel 15">
<link id=Main-File rel=Main-File

href="file:////Users/cqy/Library/Group%20Containers/UBF8T346G9.Office/TemporaryItems/msohtmlclip/clip.htm">
<link rel=File-List

href="file:////Users/cqy/Library/Group%20Containers/UBF8T346G9.Office/TemporaryItems/msohtmlclip/clip_filelist.xml">
<!--table
	{mso-displayed-decimal-separator:"\.";
	mso-displayed-thousand-separator:"\,";}
@page
	{margin:.75in .7in .75in .7in;
	mso-header-margin:.3in;
	mso-footer-margin:.3in;}
.font5
	{color:windowtext;
	font-size:9.0pt;
	font-weight:400;
	font-style:normal;
	text-decoration:none;
	font-family:等线;
	mso-generic-font-family:auto;
	mso-font-charset:134;}
tr
	{mso-height-source:auto;
	mso-ruby-visibility:none;}
col
	{mso-width-source:auto;
	mso-ruby-visibility:none;}
br
	{mso-data-placement:same-cell;}
td
	{padding-top:1px;
	padding-right:1px;
	padding-left:1px;
	mso-ignore:padding;
	color:black;
	font-size:12.0pt;
	font-weight:400;
	font-style:normal;
	text-decoration:none;
	font-family:等线;
	mso-generic-font-family:auto;
	mso-font-charset:134;
	mso-number-format:General;
	text-align:general;
	vertical-align:middle;
	border:none;
	mso-background-source:auto;
	mso-pattern:auto;
	mso-protection:locked visible;
	white-space:nowrap;
	mso-rotate:0;}
ruby
	{ruby-align:left;}
rt
	{color:windowtext;
	font-size:9.0pt;
	font-weight:400;
	font-style:normal;
	text-decoration:none;
	font-family:等线;
	mso-generic-font-family:auto;
	mso-font-charset:134;
	mso-char-type:none;
	display:none;}
-->
</head>

<body link="#0563C1" vlink="#954F72">


sparse vector | growing segment | sealed segment
-- | -- | --
BM25 | no | no
IP | yes | no
  |   |  
dense vector | growing segment | sealed segment
ivf flat cc | yes | yes
scann_dvr | no | no



</body>

</html>

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2025-02-13 09:56:47 +08:00
cai.zhang
4270174899
fix: [2.5] Add scalar index engine version for compatibility (#39236) 
issue: #39203 

master pr: #39204

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2025-01-14 21:01:01 +08:00
yihao.dai
b91c0a8079
enhance: [2.5] Optimize GetLocalDiskSize and segment loader mutex (#38907) 
1. Make the segment loader lock protect only the resource.
2. Optimize GetDiskUsage to avoid excessive overhead.

issue: https://github.com/milvus-io/milvus/issues/37630

pr: https://github.com/milvus-io/milvus/pull/38599

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2025-01-14 20:43:06 +08:00
jaime
0693634f62
enhance: add db name in replica description (#38673) 
issue: #36621
pr: #38672

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2025-01-09 19:43:04 +08:00
tinswzy
27229f7907
enhance: refine exists log print with ctx (#38080) 
issue: #35917 
Refines exists log print with ctx

Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
 2024-12-14 22:36:44 +08:00
congqixia
10460ed3f8
enhance: Simplify querynode tsafe & reduce goroutine number (#38416) 
Related to #37630

TSafe manager is too complex for current implementation and each
delegator need one goroutine waiting for tsafe update event.

Tsafe updating could be executed in pipeline. This PR remove tsafe
manager and simplify the entire logic of tsafe updating.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-12-13 10:56:43 +08:00
Zhen Ye
c6dcef7b84
enhance: move segcore codes of segment into one package (#37722) 
issue: #33285

- move most cgo opeartions related to search/query into segcore package
for reusing for streamingnode.
- add go unittest for segcore operations.

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-11-29 10:22:36 +08:00
jaime
7bbfe86bcd
enhance: add list index and segment index retrieval API for WebUI (#37861) 
issue: #36621

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-11-22 16:58:34 +08:00
Zhen Ye
fca946dee1
enhance: move the search utilities from querynode into new module (#37531) 
issue: #33285

- The search utilities will be shared between query node and streaming
node.

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-11-11 11:10:27 +08:00
jaime
f348bd9441
feat: add segment,pipeline, replica and resourcegroup api for WebUI (#37344) 
issue: #36621

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-11-07 11:52:25 +08:00
jaime
9d16b972ea
feat: add tasks page into management WebUI (#37002) 
issue: #36621

1. Add API to access task runtime metrics, including:
  - build index task
  - compaction task
  - import task
- balance (including load/release of segments/channels and some leader
tasks on querycoord)
  - sync task
2. Add a debug model to the webpage by using debug=true or debug=false
in the URL query parameters to enable or disable debug mode.

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-10-28 10:13:29 +08:00
Yinzuo Jiang
3628593d20
feat: Implement custom function module in milvus expr (#36560) 
OSPP 2024 project:
https://summer-ospp.ac.cn/org/prodetail/247410235?list=org&navpage=org

Solutions:

- parser (planparserv2)
    - add CallExpr in planparserv2/Plan.g4
    - update parser_visitor and show_visitor
- grpc protobuf
    - add CallExpr in plan.proto
- execution (`core/src/exec`)
- add `CallExpr` `ValueExpr` and `ColumnExpr` (both logical and
physical) for function call and function parameters
- function factory (`core/src/exec/expression/function`)
    - create a global hashmap when starting milvus (see server.go)
- the global hashmap stores function signatures and their function
pointers, the CallExpr in execution engine can get the function pointer
by function signature.
- custom functions
    - empty(string)
    - starts_with(string, string)
- add cpp/go unittests and E2E tests

closes: #36559

Signed-off-by: Yinzuo Jiang <jiangyinzuo@foxmail.com>
 2024-10-25 15:25:30 +08:00
yihao.dai
763fd0dfc5
enhance: Use a separate mmap config for chunk cache (#36276) 
issue: https://github.com/milvus-io/milvus/issues/35273

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-09-15 16:23:09 +08:00
aoiasd
c22a2cebb6
fix: split stream query result to avoid grpc response too large error (#36090) 
relate: https://github.com/milvus-io/milvus/issues/36089

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2024-09-13 15:07:09 +08:00
Zhen Ye
a773836b89
enhance: optimize milvus core building (#35610) 
issue: #35549,#35611,#35633

- remove milvus_segcore milvus_indexbuilder..., add libmilvus_core
- core building only link once
- move opendal compilation into cmake
- fix odr

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-08-23 12:35:02 +08:00
SimFG
731d45abbe
enhance: provide more general configuration to control mmap behavior (#35359) 
- issue: #35273

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-08-21 00:22:54 +08:00
congqixia
c64a078458
enhance: Support proxy/delegator qn client pooling (#35194) 
See also #35196

Add param item for proxy/delegator query node client pooling and
implement pooling logic

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-08-02 11:24:19 +08:00
zhenshan.cao
aa247f192d
enhance: remove unused code for StorageV2 (#35132) 
issue: https://github.com/milvus-io/milvus/issues/34168

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2024-08-01 12:08:13 +08:00
cai.zhang
2372452fac
enhance: Optimized the GC logic to ensure that memory is released in time (#34949) 
issue: #34703

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-07-28 23:53:47 +08:00
cqy123456
32f685ff12
enhance: growing segment support mmap (#32633) 
issue: https://github.com/milvus-io/milvus/issues/32984

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2024-06-18 14:42:00 +08:00
aoiasd
59a7a46904
enhance: Merge query stream result for reduce delete task (#32855) 
relate: https://github.com/milvus-io/milvus/issues/32854

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2024-05-27 18:15:43 +08:00
wei liu
39f56678a0
enhance: Reduce bloom filter lock contention between insert and delete in query coord (#32643) 
issue: #32530

cause ProcessDelete need to check whether pk exist in bloom filter, and
ProcessInsert need to update pk to bloom filter, when execute
ProcessInsert and ProcessDelete in parallel, it will cause race
condition in segment's bloom filter

This PR execute ProcessInsert and ProcessDelete in serial to avoid block
each other

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-05-22 19:11:40 +08:00
wei liu
f1c9986974
enhance: Skip return data distribution if no change happen (#32814) 
issue: #32813

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-05-17 10:11:37 +08:00
foxspy
f6777267e3
enhance: add score compute consistency config for knowhere (#32997) 
issue: https://github.com/milvus-io/milvus/issues/32583
related: #32584

Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
 2024-05-13 14:21:31 +08:00
Bingyi Sun
fecd9c21ba
feat: LRU cache implementation (#32567) 
issue: https://github.com/milvus-io/milvus/issues/32783
This pr is the implementation of lru cache on branch lru-dev.

Signed-off-by: sunby <sunbingyi1992@gmail.com>
Co-authored-by: chyezh <chyezh@outlook.com>
Co-authored-by: MrPresent-Han <chun.han@zilliz.com>
Co-authored-by: Ted Xu <ted.xu@zilliz.com>
Co-authored-by: jaime <yun.zhang@zilliz.com>
Co-authored-by: wayblink <anyang.wang@zilliz.com>
 2024-05-06 20:29:30 +08:00
wei liu
c0555d4b45
fix: Remove read only node from replica immedaitely after node down (#32666) 
issue: #32665

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-04-28 20:25:25 +08:00
Bingyi Sun
8e661f791a
fix: lazy load index data in cache (#31094) 
issue: https://github.com/milvus-io/milvus/issues/31571

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2024-03-25 15:43:07 +08:00
Bingyi Sun
0fe2ac4c3a
fix: fix mmap enable log (#31439) 
issue: https://github.com/milvus-io/milvus/issues/31443

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2024-03-20 19:17:05 +08:00
jaime
db79be3ae0
fix: ctx cancel should be the last step while stopping server (#31220) 
issue: #31219

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-03-15 10:33:05 +08:00
Chun Han
3298e64bd3
enhance: cache config values for saving cpu cycles to parse config item (#30947) 
related: #30958

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2024-03-12 11:09:04 +08:00
MrPresent-Han
17a2fd048e
feat: support set up knowhere-build-pool-size on querynode(#29650) (#30922) 
related: #29650

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2024-02-29 18:15:00 +08:00
chyezh
0c7474d7e8
enhance: add graceful stop timeout to avoid node stop hang under extreme cases (#30317) 
1. add coordinator graceful stop timeout to 5s
2. change the order of datacoord component while stop
3. change querynode grace stop timeout to 900s, and we should
potentially change this to 600s when graceful stop is smooth

issue: #30310
also see pr: #30306

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-02-29 17:01:50 +08:00
chyezh
77477d6340
fix: wrong context passing into NewClient, error handling lost in session_util (#30817) 
issue: #30799

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-02-28 10:40:09 +08:00
yiwangdr
c6665c2a4c
test: support multiple data/querynodes in integration test (#30618) 
issue: https://github.com/milvus-io/milvus/issues/29507

Signed-off-by: yiwangdr <yiwangdr@gmail.com>
 2024-02-21 11:54:53 +08:00
Bingyi Sun
e1258b8cad
feat: integrate storagev2 into loading segment (#29336) 
issue: #29335

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2024-01-12 18:10:51 +08:00
congqixia
b251c3a682
enhance: add ctx for HandleCStatus and callers (#29517) 
See also #29516

Make `HandleCStatus` print trace id for better logging

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-12-27 16:10:47 +08:00
cqy123456
4c979538a4
enhance: update cagra index params in config and add params check (#29045) 
issue:https://github.com/milvus-io/milvus/issues/29230
this pr do two things about cagra index:
 a.milvus yaml config support gpu memory settings

 b.add cagra-params check

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
Co-authored-by: yusheng.ma <yusheng.ma@zilliz.com>
 2023-12-26 11:04:47 +08:00
SimFG
dd9c61831d
enhance: Support to get the param value in the runtime (#29297) 
/kind improvement
issue: #29299

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-12-22 18:36:44 +08:00
yah01
1997fd150c
enhance: use local path as mmap file path if no mmap path provided (#28992) 
the mmap mode isn't controlled by the config anymore, so we don't
require user to set it when they enabled mmap mode.

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-12-21 18:12:45 +08:00
wei liu
cb0676153f
enhance: add metrics for stopping querynode balance progress (#29201) 
This PR add three metrics to track the stopping balance progress.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-12-21 10:14:52 +08:00
zhagnlu
a602171d06
enhance: Refactor runtime and expr framework (#28166) 
#28165

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2023-12-18 12:04:42 +08:00
yihao.dai
b4353ca4ce
enhance: Remove vector chunk manager (#28569) 
We have implemented the chunkcache (in cpp) to retrieve vectors, hence
rendering the vectorchunkcache (in golang) obsolete.

issue: https://github.com/milvus-io/milvus/issues/28568

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-11-30 18:00:33 +08:00
SimFG
e3b7fdac61
Delay the cancellation of ctx when stopping the node (#28247) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-11-08 03:20:17 +08:00
cqy123456
4fbe3c9142
replace loaded binlog with binlog index for search performance (#27673) 
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2023-11-01 02:20:15 +08:00