admin/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2025-12-30 23:45:28 +08:00
Code Issues Packages Projects Releases Wiki Activity
20,281 Commits 136 Branches 186 Tags
Commit Graph

310 Commits

Author SHA1 Message Date
congqixia
8b5754f7fe
enhance: Add segment seal proportion jitter (#34636) 
See also #34574

Add jitter for segment seal proportion to avoid seal operation burst in
short period of time.

This PR also fix license header in paramtable pkg.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-07-15 14:47:39 +08:00
congqixia
1a248f2668
enhance: Add param item for segmentFlushInterval (#34629) 
See also #28817

Add paramitem for segment flush interval

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-07-12 18:59:35 +08:00
chyezh
1bc3c0b925
enhance: implement balancer at streaming coord (#34435) 
issue: #33285

- add balancer implementation
- add channel count fair balance policy
- add channel assignment discover grpc service

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-07-11 09:58:48 +08:00
aoiasd
c1e04534c3
enhance:change access log write cache default config (#34354) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2024-07-09 18:18:12 +08:00
Gao
ae6d6f91e6
enhance: change autoindex default metric type (#34261) 
issue: #34304 
cosine is more widely used in float vectors, and cosine and hamming
distance are 'metrics' which have good geometric properties

Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2024-07-08 19:52:24 +08:00
jaime
21fc5f5d46
enhance: Remove datanode reporting TT based on MQ implementation (#34421) 
issue: #34420

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-07-05 15:48:09 +08:00
XuanYang-cn
8a2be8a457
fix: DataNode might OOM by estimating based on MemorySize (#34201) 
See also: #34136

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-07-04 10:16:17 +08:00
wei liu
9db897985b
enhance: refine misleading param name for bloom filter parallel factor (#34334) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-07-04 10:06:10 +08:00
congqixia
94fb5805d5
fix: Use raw parameter value to perform CAS (#34343) 
See also #34342

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-07-03 16:00:08 +08:00
cai.zhang
8165961223
enhance: Optimeize clustering compaction (#34313) 
issue: #30633

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-07-03 15:14:08 +08:00
Patrick Weizhi Xu
32c943f1bc
enhance: [skip e2e] update the version of MV (#34378) 
issue: #29892

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
 2024-07-03 14:28:16 +08:00
cai.zhang
a5be322ab2
enhance: Don't set the timeout for SyncSegments (#34226) 
issue: #32809

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-07-02 14:28:07 +08:00
jaime
d6afb31b94
enhance: make subfunctions of datanode component modular (#33992) 
issue: #33994

also remove deprecated channel manager based on the etcd implementation

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-07-01 14:46:07 +08:00
jaime
0426390f06
enhance: improve check health (#33800) 
issue: #34264

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-07-01 10:16:06 +08:00
XuanYang-cn
dda70aa81b
fix: LegacyVersionWithoutRPCWatch default value to 2.4.1 (#34184) 
See also: #31933

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-06-27 19:36:06 +08:00
cqy123456
381d43717d
enhance: update new mmap config parmeters version (#34143) 
issue: https://github.com/milvus-io/milvus/issues/32984
related pr: https://github.com/milvus-io/milvus/pull/34145

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2024-06-25 21:20:05 +08:00
wayblink
f9a0f7bb25
Add an option to enable/disable vector field clustering key (#34097) 
#30633

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-06-25 18:52:04 +08:00
jaime
d08cb885ca
enhance: enable flush rate limiter of collection level (#33837) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-06-24 10:52:03 +08:00
congqixia
cc77363b66
enhance: Set maxPartitionNum default value to 1024 (#33949) 
See also #30059

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-20 09:11:59 +08:00
chyezh
2f6f964bc8
enhance: [skip e2e] modify gc configuration document (#33946) 
issue: #31740

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-06-19 09:57:59 +08:00
Gao
a789c60380
enhance: autoindex for multi data type (#33868) 
issue: #22837 

contain https://github.com/milvus-io/milvus/pull/33625
https://github.com/milvus-io/milvus/pull/33867
https://github.com/milvus-io/milvus/pull/33911 which already merged to
2.4 branch

Signed-off-by: chasingegg <chao.gao@zilliz.com>
Co-authored-by: foxspy <xianliang.li@zilliz.com>
 2024-06-18 21:34:01 +08:00
cqy123456
32f685ff12
enhance: growing segment support mmap (#32633) 
issue: https://github.com/milvus-io/milvus/issues/32984

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2024-06-18 14:42:00 +08:00
wei liu
4987067375
enhance: Execute bloom filter apply in parallel to speed up segment predict (#33792) 
issue: #33610

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-06-14 11:37:56 +08:00
wei liu
ab93d9c23d
enhance: Use BatchPkExist to reduce bloom filter func call cost (#33611) 
issue:#33610

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-06-13 17:57:56 +08:00
congqixia
b90999b741
fix: Compare config value then swap when caching param value (#33785) 
See also #33784

This PR change the behavior of `SetCacheValue` of config manager:

- Use mutex and map instead of concurrent map for `configCache`
- Compare config raw value before set cache value

With this implementation, concurrent caching & eviction shall always
have current output:

|time|caching |eviction|config   |cached   |
|----|--------|------- |---------|---------|
|t0  |get     |        |old value|null     |
|t1  |CAS OK  |        |old value|old value|
|t2  |        |update  |new value|old value|
|t3  |        |eviction|new value|null     |

|time|caching |eviction|config   |cached   |
|----|--------|------- |---------|---------|
|t0  |get     |        |old value|null     |
|t1  |        |update  |new value|null     |
|t2  |CAS fail|        |old value|null     |
|t3  |        |eviction|new value|null     |

|time|caching |eviction|config   |cached   |
|----|--------|------- |---------|---------|
|t0  |        |update  |new value|null     |
|t1  |get     |        |new value|null     |
|t2  |CAS OK  |        |new value|new value|
|t3  |        |eviction|new value|null     |

|time|caching |eviction|config   |cached   |
|----|--------|------- |---------|---------|
|t0  |        |update  |new value|null     |
|t1  |get     |        |new value|null     |
|t2  |        |eviction|new value|null     |
|t3  |CAS OK  |        |new value|new value|

|time|caching |eviction|config   |cached   |
|----|--------|------- |---------|---------|
|t0  |        |update  |new value|null     |
|t1  |        |eviction|new value|null     |
|t2  |get     |        |new value|null     |
|t3  |CAS OK  |        |new value|new value|

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-12 18:51:56 +08:00
Chun Han
f7af323d1e
fix: sync partitiion stats blocking balance task(#33741) (#33742) 
related: #33741

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2024-06-11 14:21:56 +08:00
wayblink
a1232fafda
feat: Major compaction (#33620) 
#30633

Signed-off-by: wayblink <anyang.wang@zilliz.com>
Co-authored-by: MrPresent-Han <chun.han@zilliz.com>
 2024-06-10 21:34:08 +08:00
XuanYang-cn
1629833060
enhance: Add consts of MsgDispatcher to configs (#33679) 
See also: #33676

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-06-07 14:21:59 +08:00
SimFG
ecee7d90d4
enhance: try to speed up the loading of small collections (#33570) 
- issue: #33569

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-06-07 08:25:53 +08:00
cai.zhang
27cc9f2630
enhance: Support analyze data (#33651) 
issue: #30633

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Co-authored-by: chasingegg <chao.gao@zilliz.com>
 2024-06-06 17:37:51 +08:00
zhenshan.cao
ac4f3997ce
enhance: Reconstructing Compaction to possess persistence capability (#33265) 
issue #33586

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2024-06-05 10:17:50 +08:00
congqixia
2b285e5573
fix: Wrap init segcore tracing with golang timeout (#33494) 
See also #33483

Wrap `C.InitTrace` & `C.SetTrace` with timeout preventing otlp
initializtion hangs forever when endpoint is not set correctly

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-03 19:25:51 +08:00
wei liu
c6a1c49e02
enhance: Use Blocked Bloom Filter instead of basic bloom fitler impl. (#33405) 
issue: #32995
To speed up the construction and querying of Bloom filters, we chose a
blocked Bloom filter instead of a basic Bloom filter implementation.

WARN: This PR is compatible with old version bf impl, but if fall back
to old milvus version, it may causes bloom filter deserialize failed.

In single Bloom filter test cases with a capacity of 1,000,000 and a
false positive rate (FPR) of 0.001, the blocked Bloom filter is 5 times
faster than the basic Bloom filter in both querying and construction, at
the cost of a 30% increase in memory usage.

- Block BF construct time	{"time": "54.128131ms"}
- Block BF size	                {"size": 3021578}
- Block BF Test cost	        {"time": "55.407352ms"}
- Basic BF construct time	{"time": "210.262183ms"}
- Basic BF size	                {"size": 2396308}
- Basic BF Test cost	        {"time": "192.596229ms"}

In multi Bloom filter test cases with a capacity of 100,000, an FPR of
0.001, and 100 Bloom filters, we reuse the primary key locations for all
Bloom filters to avoid repeated hash computations. As a result, the
blocked Bloom filter is also 5 times faster than the basic Bloom filter
in querying.

- Block BF TestLocation cost    {"time": "529.97183ms"}
- Basic BF TestLocation cost	{"time": "3.197430181s"}

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-05-31 17:49:45 +08:00
congqixia
842fb02f74
fix: Use localStorage path to check disk cap for indexnode (#33450) 
See also #30943 #30944

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-05-31 10:15:45 +08:00
cai.zhang
77637180fa
enhance: Periodically synchronize segments to datanode watcher (#33420) 
issue: #32809

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-05-30 13:37:44 +08:00
zhagnlu
589d4dfd82
enhance: optimize bitmap index (#33358) 
#32900

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-05-30 13:09:43 +08:00
yihao.dai
bbb69980ac
enhance: Replace 'off' with 'disable' (#33433) 
YAML will automatically parse "off" as a boolean variable. We should
avoid using "off" in the future.

issue: https://github.com/milvus-io/milvus/issues/32772

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-05-29 12:17:43 +08:00
aoiasd
59a7a46904
enhance: Merge query stream result for reduce delete task (#32855) 
relate: https://github.com/milvus-io/milvus/issues/32854

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2024-05-27 18:15:43 +08:00
yihao.dai
760223f80a
fix: use seperate warmup pool and disable warmup by default (#33348) 
1. use a small warmup pool to reduce the impact of warmup
2. change the warmup pool to nonblocking mode
3. disable warmup by default
4. remove the maximum size limit of 16 for the load pool

issue: https://github.com/milvus-io/milvus/issues/32772

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Co-authored-by: xiaofanluan <xiaofan.luan@zilliz.com>
 2024-05-27 01:25:40 +08:00
aoiasd
1b4e28b97f
enhance: Check by proxy rate limiter when delete get data by query. (#30891) 
relate: https://github.com/milvus-io/milvus/issues/30927

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2024-05-23 20:03:40 +08:00
wei liu
c7be2ce33a
enhance: Decrease bloom filter fp rate to reduce delete impact (#33301) 
when milvus process delete record, it need to find record's corresponded
segment by bloom filter, and higher bloom filter fp rate will cause
delete record forwards to wrong segments.

This PR Decrease bloom filter's default fp to 0.001.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-05-23 18:15:41 +08:00
SimFG
e18d5aceb6
enhance: add config to control whether to init public role permissions (#33165) 
issue: #33164

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-05-21 22:39:46 +08:00
congqixia
c2ac692008
enhance: Add param item to ignore bad message id in checkpoint (#33123) 
See also #33122

This pr add param item `mq.ignoreBadPosition` to control behavior when
mq failed to parse message id from checkpoint

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-05-20 11:29:37 +08:00
yihao.dai
32560263fa
enhance: Query slot for compaction task (#32881) 
Query slot of compaction in datanode, and transfer the control logic for
limiting compaction tasks from datacoord to the datanode.

issue: https://github.com/milvus-io/milvus/issues/32809

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-05-17 18:19:38 +08:00
aoiasd
875ad88d84
enhance: Support access log use stdout with write cache and auto flush as time (#32213) 
relate: https://github.com/milvus-io/milvus/issues/28948

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2024-05-16 10:05:34 +08:00
foxspy
f6777267e3
enhance: add score compute consistency config for knowhere (#32997) 
issue: https://github.com/milvus-io/milvus/issues/32583
related: #32584

Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
 2024-05-13 14:21:31 +08:00
Bingyi Sun
4724779b3b
enhance: remove fallback keys for config generator (#32946) 
Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2024-05-13 13:33:31 +08:00
wei liu
e2332bdc17
enhance: Enable channel exclusive balance policy (#32911) 
issue: #32910  
* split replica's node list to channels when create replicas
 * balance nodes among channels when node change happens
 * implement channel level balance, let balance happens in channel level

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-05-10 17:27:31 +08:00
Bingyi Sun
1e1fba0588
enhance: add lazy load retry configurations (#32848) 
Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2024-05-10 10:55:30 +08:00
aoiasd
54a51b1236
enhance: Support dynamic config for opentelemetry trace (#32169) 
relate: https://github.com/milvus-io/milvus/issues/31940

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2024-05-09 17:43:30 +08:00