admin/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2026-01-03 01:12:25 +08:00
Code Issues Packages Projects Releases Wiki Activity
20,075 Commits 138 Branches 188 Tags
Commit Graph

723 Commits

Author SHA1 Message Date
cqy123456
298e50b834
enhance: check index with data type (#33880) 
issue: https://github.com/milvus-io/milvus/issues/22837
related: https://github.com/milvus-io/milvus/pull/33878

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2024-06-19 10:15:59 +08:00
chyezh
2f6f964bc8
enhance: [skip e2e] modify gc configuration document (#33946) 
issue: #31740

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-06-19 09:57:59 +08:00
Gao
a789c60380
enhance: autoindex for multi data type (#33868) 
issue: #22837 

contain https://github.com/milvus-io/milvus/pull/33625
https://github.com/milvus-io/milvus/pull/33867
https://github.com/milvus-io/milvus/pull/33911 which already merged to
2.4 branch

Signed-off-by: chasingegg <chao.gao@zilliz.com>
Co-authored-by: foxspy <xianliang.li@zilliz.com>
 2024-06-18 21:34:01 +08:00
congqixia
3fdaae8792
fix: Return record with largest timestamp for entires with same PK (#33936) 
See also #33883

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-18 15:55:59 +08:00
cqy123456
32f685ff12
enhance: growing segment support mmap (#32633) 
issue: https://github.com/milvus-io/milvus/issues/32984

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2024-06-18 14:42:00 +08:00
wei liu
b72026c859
enhance: Add rbac support on describe/alter database api (#33803) 
issue: #32707

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-06-17 10:07:57 +08:00
zhagnlu
d43ec4db0b
enhance: support array bitmap index (#33527) 
#32900

---------

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-06-16 21:51:58 +08:00
wei liu
4987067375
enhance: Execute bloom filter apply in parallel to speed up segment predict (#33792) 
issue: #33610

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-06-14 11:37:56 +08:00
congqixia
c689ef4822
fix: Remove eviction operations out of lock (#33834) 
See also #33823

`EvictCacheValueByFormat` may be block by on going `CASCacheValue` and
cause possible deadlock

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-13 21:15:55 +08:00
wei liu
ab93d9c23d
enhance: Use BatchPkExist to reduce bloom filter func call cost (#33611) 
issue:#33610

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-06-13 17:57:56 +08:00
cqy123456
be3559e022
enhance: proxy check hnsw with sparse is legal (#33697) 
issue:https://github.com/milvus-io/milvus/issues/22837
related:https://github.com/milvus-io/milvus/pull/33653

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2024-06-12 20:13:55 +08:00
congqixia
b90999b741
fix: Compare config value then swap when caching param value (#33785) 
See also #33784

This PR change the behavior of `SetCacheValue` of config manager:

- Use mutex and map instead of concurrent map for `configCache`
- Compare config raw value before set cache value

With this implementation, concurrent caching & eviction shall always
have current output:

|time|caching |eviction|config   |cached   |
|----|--------|------- |---------|---------|
|t0  |get     |        |old value|null     |
|t1  |CAS OK  |        |old value|old value|
|t2  |        |update  |new value|old value|
|t3  |        |eviction|new value|null     |

|time|caching |eviction|config   |cached   |
|----|--------|------- |---------|---------|
|t0  |get     |        |old value|null     |
|t1  |        |update  |new value|null     |
|t2  |CAS fail|        |old value|null     |
|t3  |        |eviction|new value|null     |

|time|caching |eviction|config   |cached   |
|----|--------|------- |---------|---------|
|t0  |        |update  |new value|null     |
|t1  |get     |        |new value|null     |
|t2  |CAS OK  |        |new value|new value|
|t3  |        |eviction|new value|null     |

|time|caching |eviction|config   |cached   |
|----|--------|------- |---------|---------|
|t0  |        |update  |new value|null     |
|t1  |get     |        |new value|null     |
|t2  |        |eviction|new value|null     |
|t3  |CAS OK  |        |new value|new value|

|time|caching |eviction|config   |cached   |
|----|--------|------- |---------|---------|
|t0  |        |update  |new value|null     |
|t1  |        |eviction|new value|null     |
|t2  |get     |        |new value|null     |
|t3  |CAS OK  |        |new value|new value|

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-12 18:51:56 +08:00
Chun Han
f7af323d1e
fix: sync partitiion stats blocking balance task(#33741) (#33742) 
related: #33741

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2024-06-11 14:21:56 +08:00
wayblink
a1232fafda
feat: Major compaction (#33620) 
#30633

Signed-off-by: wayblink <anyang.wang@zilliz.com>
Co-authored-by: MrPresent-Han <chun.han@zilliz.com>
 2024-06-10 21:34:08 +08:00
Aldrin
8a4ef1b836
fix: Masked kafka credentials getting printed in logs (#33730) 
issue: https://github.com/milvus-io/milvus/issues/33727

Signed-off-by: Aldrin <imagesai32@gmail.com>
 2024-06-10 00:05:54 +08:00
chyezh
f53ab54c5d
enhance: async cgo utility (#33133) 
issue: #30926, #33132

- implement future-based cgo utility.

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-06-09 22:55:53 +08:00
XuanYang-cn
1629833060
enhance: Add consts of MsgDispatcher to configs (#33679) 
See also: #33676

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-06-07 14:21:59 +08:00
SimFG
ecee7d90d4
enhance: try to speed up the loading of small collections (#33570) 
- issue: #33569

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-06-07 08:25:53 +08:00
cai.zhang
27cc9f2630
enhance: Support analyze data (#33651) 
issue: #30633

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Co-authored-by: chasingegg <chao.gao@zilliz.com>
 2024-06-06 17:37:51 +08:00
aoiasd
b9bafc76b4
enhance: access log support print with consistency level (#33503) 
relate: https://github.com/milvus-io/milvus/issues/33563

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2024-06-05 17:13:50 +08:00
zhenshan.cao
ac4f3997ce
enhance: Reconstructing Compaction to possess persistence capability (#33265) 
issue #33586

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2024-06-05 10:17:50 +08:00
jaime
8858fcb40a
fix: fix loaded entity num is inaccurate (#33521) 
issue: #33520

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-06-04 20:09:54 +08:00
Bingyi Sun
d610fdf033
enhance: change Allocator to generic (#33581) 
Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2024-06-04 19:05:49 +08:00
aoiasd
2422084a29
fix: paramtable cache cause dynamic config non-dynamic (#33473) 
relate: https://github.com/milvus-io/milvus/issues/33461

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2024-06-04 11:39:46 +08:00
Ted Xu
d0a0eac0a4
enhance: adding virtual resource allocator (#33508) 
See #33559

---------

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
 2024-06-03 19:45:46 +08:00
congqixia
2b285e5573
fix: Wrap init segcore tracing with golang timeout (#33494) 
See also #33483

Wrap `C.InitTrace` & `C.SetTrace` with timeout preventing otlp
initializtion hangs forever when endpoint is not set correctly

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-03 19:25:51 +08:00
wei liu
c6a1c49e02
enhance: Use Blocked Bloom Filter instead of basic bloom fitler impl. (#33405) 
issue: #32995
To speed up the construction and querying of Bloom filters, we chose a
blocked Bloom filter instead of a basic Bloom filter implementation.

WARN: This PR is compatible with old version bf impl, but if fall back
to old milvus version, it may causes bloom filter deserialize failed.

In single Bloom filter test cases with a capacity of 1,000,000 and a
false positive rate (FPR) of 0.001, the blocked Bloom filter is 5 times
faster than the basic Bloom filter in both querying and construction, at
the cost of a 30% increase in memory usage.

- Block BF construct time	{"time": "54.128131ms"}
- Block BF size	                {"size": 3021578}
- Block BF Test cost	        {"time": "55.407352ms"}
- Basic BF construct time	{"time": "210.262183ms"}
- Basic BF size	                {"size": 2396308}
- Basic BF Test cost	        {"time": "192.596229ms"}

In multi Bloom filter test cases with a capacity of 100,000, an FPR of
0.001, and 100 Bloom filters, we reuse the primary key locations for all
Bloom filters to avoid repeated hash computations. As a result, the
blocked Bloom filter is also 5 times faster than the basic Bloom filter
in querying.

- Block BF TestLocation cost    {"time": "529.97183ms"}
- Basic BF TestLocation cost	{"time": "3.197430181s"}

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-05-31 17:49:45 +08:00
congqixia
842fb02f74
fix: Use localStorage path to check disk cap for indexnode (#33450) 
See also #30943 #30944

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-05-31 10:15:45 +08:00
Jiquan Long
0c5d8660aa
feat: support inverted index for array (#33452) 
issue: https://github.com/milvus-io/milvus/issues/27704

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2024-05-31 09:47:47 +08:00
Chun Han
416a2cf507
fix: query iterator lack results(#33137) (#33422) 
related: #33137 
adding has_more_result_tag for various level's reduce to rectify
reduce_stop_for_best

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2024-05-30 17:51:44 +08:00
cai.zhang
77637180fa
enhance: Periodically synchronize segments to datanode watcher (#33420) 
issue: #32809

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-05-30 13:37:44 +08:00
zhagnlu
589d4dfd82
enhance: optimize bitmap index (#33358) 
#32900

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-05-30 13:09:43 +08:00
yihao.dai
bbb69980ac
enhance: Replace 'off' with 'disable' (#33433) 
YAML will automatically parse "off" as a boolean variable. We should
avoid using "off" in the future.

issue: https://github.com/milvus-io/milvus/issues/32772

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-05-29 12:17:43 +08:00
wei liu
b13932bb55
enhance: Enable database level replica num and resource groups for loading collection (#33052) 
issue: #30040

This PR introduce two database level props:
1. database.replica.number
2. database.resource_groups

User can set those two database props by AlterDatabase API, then can
load collection without specified replica_num and resource groups. then
it will use database level load param when try to load collections.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-05-29 10:59:43 +08:00
aoiasd
59a7a46904
enhance: Merge query stream result for reduce delete task (#32855) 
relate: https://github.com/milvus-io/milvus/issues/32854

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2024-05-27 18:15:43 +08:00
SimFG
cb99e3db34
enhance: add the includeCurrentMsg param for the Seek method (#33326) 
/kind improvement
- issue: #33325

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-05-27 10:31:41 +08:00
yihao.dai
760223f80a
fix: use seperate warmup pool and disable warmup by default (#33348) 
1. use a small warmup pool to reduce the impact of warmup
2. change the warmup pool to nonblocking mode
3. disable warmup by default
4. remove the maximum size limit of 16 for the load pool

issue: https://github.com/milvus-io/milvus/issues/32772

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Co-authored-by: xiaofanluan <xiaofan.luan@zilliz.com>
 2024-05-27 01:25:40 +08:00
Buqian Zheng
1b67cecd65
enhance: add sparse float vector support to restful v2 (#33231) 
issue: #29419
also re-enabled an e2e test using restful api, which is previously
disabled due to https://github.com/milvus-io/milvus/issues/32214.

In restful api, the accepted json formats of sparse float vector are:

* `{"indices": [1, 100, 1000], "values": [0.1, 0.2, 0.3]}`
* {"1": 0.1, "100": 0.2, "1000": 0.3}

for accepted indice and value range, see
https://milvus.io/docs/sparse_vector.md#FAQ

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
 2024-05-27 00:47:40 +08:00
Xiaofan
36cbce4def
enhance: optimize datanode cpu usage under large collection number (#33267) 
fix #33266 
try to improve cpu usage by refactoring the ttchecker logic and caching
string

Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
 2024-05-25 04:43:41 +08:00
shaoting-huang
3c7d0209d4
enhance: update checker for go version (#33351) 
Signed-off-by: shaoting-huang [shaoting-huang@zilliz.com]

issue: https://github.com/milvus-io/milvus/issues/32982

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
 2024-05-24 14:23:40 +08:00
aoiasd
1b4e28b97f
enhance: Check by proxy rate limiter when delete get data by query. (#30891) 
relate: https://github.com/milvus-io/milvus/issues/30927

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2024-05-23 20:03:40 +08:00
wei liu
c7be2ce33a
enhance: Decrease bloom filter fp rate to reduce delete impact (#33301) 
when milvus process delete record, it need to find record's corresponded
segment by bloom filter, and higher bloom filter fp rate will cause
delete record forwards to wrong segments.

This PR Decrease bloom filter's default fp to 0.001.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-05-23 18:15:41 +08:00
Cai Yudong
fd53cdb103
fix: Fix SparseFloatVector data parse error for json (#33259) 
Issue: #33162

Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
 2024-05-23 14:17:40 +08:00
Buqian Zheng
c5918ffbdb
enhance: mark sparse inverted index as mmap-able (#33281) 
issue: #29419

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
 2024-05-23 14:11:42 +08:00
SimFG
dd0c6d6980
fix: the panic when db isn't existed in the rate limit interceptor (#33244) 
issue: #33243

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-05-22 15:57:39 +08:00
SimFG
e18d5aceb6
enhance: add config to control whether to init public role permissions (#33165) 
issue: #33164

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-05-21 22:39:46 +08:00
congqixia
12e8c6c583
enhance: Try LatestMessageID when checkpoint unmarshal fails (#33158) 
See also #33122

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-05-21 16:59:39 +08:00
Cai Yudong
cb480d17c8
fix: Fix SparseFloatVector data parse error for parquet (#33187) 
Issue: #22837

Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
 2024-05-21 15:09:39 +08:00
aoiasd
f8929cc36a
fix: can't generate traceID when use noop exporter (#33191) 
relate: https://github.com/milvus-io/milvus/issues/33190

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2024-05-21 10:39:39 +08:00
jaime
0d99db23b8
fix: metrics leak on the coord nodes (#33075) 
issue: #32980

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-05-20 22:03:39 +08:00