admin/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2025-12-29 06:55:27 +08:00
Code Issues Packages Projects Releases Wiki Activity
20,294 Commits 136 Branches 185 Tags
Commit Graph

558 Commits

Author SHA1 Message Date
smellthemoon
b5b5afb63f
fix: [skip e2e]golang lint check fail (#34682) 
#34680

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2024-07-16 10:47:36 +08:00
wei liu
acb33bba4d
enhance: Preserve fixed-size memory in delegator node for growing segment. (#34596) 
issue: #34595
When consuming insert data on the delegator node, QueryCoord will move
out some sealed segments to manage its memory usage. After the growing
segment gets flushed, some sealed segments from other workers will be
moved back to the delegator node. To avoid the frequent movement of
segments, we estimate the maximum growing row count and preserve a
fixed-size memory in the delegator node.

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-07-15 20:51:46 +08:00
chyezh
fda720b880
enhance: streaming service grpc utilities (#34436) 
issue: #33285

- add two grpc resolver (by session and by streaming coord assignment
service)
- add one grpc balancer (by serverID and roundrobin)
- add lazy conn to avoid block by first service discovery
- add some utility function for streaming service

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-07-15 20:49:38 +08:00
congqixia
8b5754f7fe
enhance: Add segment seal proportion jitter (#34636) 
See also #34574

Add jitter for segment seal proportion to avoid seal operation burst in
short period of time.

This PR also fix license header in paramtable pkg.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-07-15 14:47:39 +08:00
congqixia
1a248f2668
enhance: Add param item for segmentFlushInterval (#34629) 
See also #28817

Add paramitem for segment flush interval

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-07-12 18:59:35 +08:00
chyezh
1bc3c0b925
enhance: implement balancer at streaming coord (#34435) 
issue: #33285

- add balancer implementation
- add channel count fair balance policy
- add channel assignment discover grpc service

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-07-11 09:58:48 +08:00
jaime
c332f69dec
enhance: skip orphan channel cp meta when checking cp lag (#34555) 
issue: # #34545

Print warn log instead of check health fail if orphan channel cp meta is
found in health check request.

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-07-11 09:36:56 +08:00
aoiasd
c1e04534c3
enhance:change access log write cache default config (#34354) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2024-07-09 18:18:12 +08:00
congqixia
3333160b8d
enhance: Fix lint issues from recent PRs (#34482) 
See also #34483
Some lint issues are introduced due to lack of static check run. This PR
fixes these problems.

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-07-09 10:06:24 +08:00
Gao
ae6d6f91e6
enhance: change autoindex default metric type (#34261) 
issue: #34304 
cosine is more widely used in float vectors, and cosine and hamming
distance are 'metrics' which have good geometric properties

Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2024-07-08 19:52:24 +08:00
congqixia
145f4f919b
fix: Add nbits parameter check for IVF_PQ (#34451) 
See also #34426

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-07-08 11:16:10 +08:00
jaime
60be454db0
enhance: add disk quota and max collections into db properties (#34368) 
issue: #34385

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-07-05 18:22:17 +08:00
zhagnlu
18f2458385
fix: fix bitmap supported type (#34350) 
#34314

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-07-05 15:50:10 +08:00
jaime
21fc5f5d46
enhance: Remove datanode reporting TT based on MQ implementation (#34421) 
issue: #34420

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-07-05 15:48:09 +08:00
chyezh
7611128e57
enhance: wal adaptor implementation (#34122) 
issue: #33285

- add adaptor to implement walimpls into wal interface.
- implement timetick sorted and filtering scanner.
- add test for wal.

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-07-04 15:23:08 +08:00
XuanYang-cn
8a2be8a457
fix: DataNode might OOM by estimating based on MemorySize (#34201) 
See also: #34136

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-07-04 10:16:17 +08:00
wei liu
9db897985b
enhance: refine misleading param name for bloom filter parallel factor (#34334) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-07-04 10:06:10 +08:00
congqixia
94fb5805d5
fix: Use raw parameter value to perform CAS (#34343) 
See also #34342

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-07-03 16:00:08 +08:00
cai.zhang
8165961223
enhance: Optimeize clustering compaction (#34313) 
issue: #30633

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-07-03 15:14:08 +08:00
Patrick Weizhi Xu
32c943f1bc
enhance: [skip e2e] update the version of MV (#34378) 
issue: #29892

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
 2024-07-03 14:28:16 +08:00
Buqian Zheng
fa8d641ce6
fix: update check for sparse hnsw index (#33713) 
issue: #29419

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
 2024-07-02 21:56:09 +08:00
chyezh
3563136c2a
enhance: timetick interceptor implementation (#34238) 
issue: #33285

- optimize the message package
- add interceptor package to achieve append operation intercepting.
- add timetick interceptor to attach timetick properties for message.
- add timetick background task to send timetick message.

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-07-02 14:42:08 +08:00
cai.zhang
a5be322ab2
enhance: Don't set the timeout for SyncSegments (#34226) 
issue: #32809

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-07-02 14:28:07 +08:00
yihao.dai
ff51c7e628
fix: Fix panic due to failed to seek (#34229) 
Converting the same msgposition's vchannel to a pchannel multiple times
would result in an invalid pchannel, leading to seek failure and panic.
This PR:
1. Make a copy of msgposition in msgdispatcher.
2. Check if channel is already a pchannel, no further channel conversion
is performed.

issue: https://github.com/milvus-io/milvus/issues/34221

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-07-01 16:08:12 +08:00
aoiasd
186757e622
enhance: support mark error as user error (#33498) 
relate: https://github.com/milvus-io/milvus/issues/33492

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2024-07-01 14:56:12 +08:00
jaime
d6afb31b94
enhance: make subfunctions of datanode component modular (#33992) 
issue: #33994

also remove deprecated channel manager based on the etcd implementation

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-07-01 14:46:07 +08:00
jaime
0426390f06
enhance: improve check health (#33800) 
issue: #34264

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-07-01 10:16:06 +08:00
wayblink
73ffc1b424
Refine clustering_compaction_task retry mechanism (#34194) 
#32939

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-06-30 20:22:09 +08:00
XuanYang-cn
dda70aa81b
fix: LegacyVersionWithoutRPCWatch default value to 2.4.1 (#34184) 
See also: #31933

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-06-27 19:36:06 +08:00
cqy123456
51ebe95be5
enhance: remove CheckVecIndexWithDataTypeExist function in pkg and remove some cgo call (#34102) 
issue: https://github.com/milvus-io/milvus/issues/22837
related pr: https://github.com/milvus-io/milvus/pull/34104

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2024-06-26 21:13:44 +08:00
cqy123456
381d43717d
enhance: update new mmap config parmeters version (#34143) 
issue: https://github.com/milvus-io/milvus/issues/32984
related pr: https://github.com/milvus-io/milvus/pull/34145

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2024-06-25 21:20:05 +08:00
wayblink
f9a0f7bb25
Add an option to enable/disable vector field clustering key (#34097) 
#30633

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-06-25 18:52:04 +08:00
jaime
d08cb885ca
enhance: enable flush rate limiter of collection level (#33837) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-06-24 10:52:03 +08:00
Patrick Weizhi Xu
b961767005
enhance: support integral type for MV and skip MV if there is only one category (#33161) 
issue: #29892

---------

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
 2024-06-24 10:20:01 +08:00
Ted Xu
958ecd500b
feat: adding grouped virtual resource allocator (#33669) 
See #33559

---------

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
 2024-06-21 15:58:01 +08:00
congqixia
cc77363b66
enhance: Set maxPartitionNum default value to 1024 (#33949) 
See also #30059

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-20 09:11:59 +08:00
smellthemoon
2a1356985d
enhance: support null in go payload (#32296) 
#31728

---------

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2024-06-19 17:08:00 +08:00
cqy123456
298e50b834
enhance: check index with data type (#33880) 
issue: https://github.com/milvus-io/milvus/issues/22837
related: https://github.com/milvus-io/milvus/pull/33878

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2024-06-19 10:15:59 +08:00
chyezh
2f6f964bc8
enhance: [skip e2e] modify gc configuration document (#33946) 
issue: #31740

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-06-19 09:57:59 +08:00
Gao
a789c60380
enhance: autoindex for multi data type (#33868) 
issue: #22837 

contain https://github.com/milvus-io/milvus/pull/33625
https://github.com/milvus-io/milvus/pull/33867
https://github.com/milvus-io/milvus/pull/33911 which already merged to
2.4 branch

Signed-off-by: chasingegg <chao.gao@zilliz.com>
Co-authored-by: foxspy <xianliang.li@zilliz.com>
 2024-06-18 21:34:01 +08:00
congqixia
3fdaae8792
fix: Return record with largest timestamp for entires with same PK (#33936) 
See also #33883

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-18 15:55:59 +08:00
cqy123456
32f685ff12
enhance: growing segment support mmap (#32633) 
issue: https://github.com/milvus-io/milvus/issues/32984

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2024-06-18 14:42:00 +08:00
wei liu
b72026c859
enhance: Add rbac support on describe/alter database api (#33803) 
issue: #32707

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-06-17 10:07:57 +08:00
zhagnlu
d43ec4db0b
enhance: support array bitmap index (#33527) 
#32900

---------

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-06-16 21:51:58 +08:00
wei liu
4987067375
enhance: Execute bloom filter apply in parallel to speed up segment predict (#33792) 
issue: #33610

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-06-14 11:37:56 +08:00
wei liu
ab93d9c23d
enhance: Use BatchPkExist to reduce bloom filter func call cost (#33611) 
issue:#33610

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-06-13 17:57:56 +08:00
cqy123456
be3559e022
enhance: proxy check hnsw with sparse is legal (#33697) 
issue:https://github.com/milvus-io/milvus/issues/22837
related:https://github.com/milvus-io/milvus/pull/33653

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2024-06-12 20:13:55 +08:00
congqixia
b90999b741
fix: Compare config value then swap when caching param value (#33785) 
See also #33784

This PR change the behavior of `SetCacheValue` of config manager:

- Use mutex and map instead of concurrent map for `configCache`
- Compare config raw value before set cache value

With this implementation, concurrent caching & eviction shall always
have current output:

|time|caching |eviction|config   |cached   |
|----|--------|------- |---------|---------|
|t0  |get     |        |old value|null     |
|t1  |CAS OK  |        |old value|old value|
|t2  |        |update  |new value|old value|
|t3  |        |eviction|new value|null     |

|time|caching |eviction|config   |cached   |
|----|--------|------- |---------|---------|
|t0  |get     |        |old value|null     |
|t1  |        |update  |new value|null     |
|t2  |CAS fail|        |old value|null     |
|t3  |        |eviction|new value|null     |

|time|caching |eviction|config   |cached   |
|----|--------|------- |---------|---------|
|t0  |        |update  |new value|null     |
|t1  |get     |        |new value|null     |
|t2  |CAS OK  |        |new value|new value|
|t3  |        |eviction|new value|null     |

|time|caching |eviction|config   |cached   |
|----|--------|------- |---------|---------|
|t0  |        |update  |new value|null     |
|t1  |get     |        |new value|null     |
|t2  |        |eviction|new value|null     |
|t3  |CAS OK  |        |new value|new value|

|time|caching |eviction|config   |cached   |
|----|--------|------- |---------|---------|
|t0  |        |update  |new value|null     |
|t1  |        |eviction|new value|null     |
|t2  |get     |        |new value|null     |
|t3  |CAS OK  |        |new value|new value|

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-12 18:51:56 +08:00
Chun Han
f7af323d1e
fix: sync partitiion stats blocking balance task(#33741) (#33742) 
related: #33741

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2024-06-11 14:21:56 +08:00
wayblink
a1232fafda
feat: Major compaction (#33620) 
#30633

Signed-off-by: wayblink <anyang.wang@zilliz.com>
Co-authored-by: MrPresent-Han <chun.han@zilliz.com>
 2024-06-10 21:34:08 +08:00