wei liu
5038036ece
enhance: Reuse hash locations during access bloom fitler ( #32642 )
...
issue: #32530
when try to match segment bloom filter with pk, we can reuse the hash
locations. This PR maintain the max hash Func, and compute hash location
once for all segment, reuse hash location can speed up bf access
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-05-07 06:13:47 -07:00
congqixia
40728ce83d
enhance: Add metautil.Channel to convert string compare to int ( #32749 )
...
See also #32748
This PR:
- Add `metautil.Channel` utiltiy which convert virtual name to physical
channel name, collectionID and shard idx
- Add channel mapper interface & implementation to convert limited
physical channel name into int index
- Apply `metautil.Channel` filter in querynode segment manager logic
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-05-07 19:13:35 +08:00
yihao.dai
9db3aa18bc
enhance: Remove deprecated EnableIndex ( #32704 )
...
/kind improvement
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-05-07 17:11:30 +08:00
chyezh
641f702f64
fix: add request resource timeout for lazy load, refactor context usage in cache ( #32709 )
...
issue: #32663
- Use new param to control request resource timeout for lazy load.
- Remove the timeout parameter of `Do`, remove `DoWait`. use `context`
to control the timeout.
- Use `VersionedNotifier` to avoid notify event lost and broadcast,
remove the redundant goroutine in cache.
related dev pr: #32684
Signed-off-by: chyezh <chyezh@outlook.com>
2024-05-07 16:33:30 +08:00
congqixia
efa0c73c62
fix: Unify querynode metrics cleanup in collection release ( #32805 )
...
Related to #32803
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-05-07 15:41:29 +08:00
aoiasd
31dca3249e
enhance: add type info for payload writer error message and add log when querynode find new collection ( #32522 )
...
relate: https://github.com/milvus-io/milvus/issues/32668
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-05-07 14:45:29 +08:00
Jiquan Long
1f58cda957
enhance: add more trace for search & query ( #32734 )
...
issue: https://github.com/milvus-io/milvus/issues/32728
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-05-07 13:03:29 +08:00
congqixia
7102403a6b
fix: Add Wrapper and Keepalive for CTraceContext ids ( #32746 )
...
See also #32742
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-05-07 10:05:35 +08:00
Bingyi Sun
fecd9c21ba
feat: LRU cache implementation ( #32567 )
...
issue: https://github.com/milvus-io/milvus/issues/32783
This pr is the implementation of lru cache on branch lru-dev.
Signed-off-by: sunby <sunbingyi1992@gmail.com>
Co-authored-by: chyezh <chyezh@outlook.com>
Co-authored-by: MrPresent-Han <chun.han@zilliz.com>
Co-authored-by: Ted Xu <ted.xu@zilliz.com>
Co-authored-by: jaime <yun.zhang@zilliz.com>
Co-authored-by: wayblink <anyang.wang@zilliz.com>
2024-05-06 20:29:30 +08:00
Jiquan Long
c002745902
enhance: retrieve output fields after local reduce ( #32346 )
...
issue: #31822
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-04-25 09:49:26 +08:00
yihao.dai
281a583eda
fix: Correct the negative queryable num entities metric ( #32361 )
...
issue: https://github.com/milvus-io/milvus/issues/32281
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-04-24 15:55:24 +08:00
Cai Yudong
16b8b7b35d
enhance: Add get_vector unittest for float16 & bfloat16 ( #32153 )
...
Issue: #22837
Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2024-04-23 16:15:23 +08:00
chyezh
e19d17076f
fix: delete may lost when enable lru cache, some field should be reset when ReleaseData ( #32012 )
...
issue: #30361
- Delete may be lost when segment is not data-loaded status in lru
cache. skip filtering to fix it.
- `stats_` and `variable_fields_avg_size_` should be reset when
`ReleaseData`
- Remove repeat load delta log operation in lru.
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-04-16 11:17:20 +08:00
wei liu
1a98ce39f5
enhance: Remove useless logic about FromShardLeader ( #32029 )
...
issue: #32047
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-04-10 20:11:19 +08:00
Xiaofan
dbab9c5096
fix: reduce didn't handle offset without limit and reduceStopForBest correctly ( #32089 )
...
fix https://github.com/milvus-io/milvus/issues/32059
this pr fix two issues:
offset is not handled correctly without specify a limit
reduceStopForBest doesn't guarantee to return limit result even if there
are more result when there is small segment
Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
2024-04-10 16:01:18 +08:00
Chun Han
f3f2a5a7e9
fix: evicted segments in the serverlss mode( #31959 ) ( #31961 )
...
related: #31959
1. reset segment index status after evicting to lazyload=true
2. reset num_rows to null_opt
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-04-10 15:15:19 +08:00
SimFG
90bed1caf9
enhance: add the related data size for the read apis ( #31816 )
...
issue: #30436
origin pr: #30438
related pr: #31772
---------
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-04-10 15:07:17 +08:00
chyezh
c9faa6d936
enhance: add more metrics for cache and search ( #31777 )
...
issue: #30931
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-04-10 10:55:17 +08:00
zhenshan.cao
089c805e0a
enhance:Refactor hybrid search ( #32020 )
...
issue: https://github.com/milvus-io/milvus/issues/25639
https://github.com/milvus-io/milvus/issues/31368
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-04-09 14:21:18 +08:00
chyezh
73adf2a5cc
fix: use stateful lock to avoid load and release on LocalSegment concurrently ( #31606 )
...
issue: #31605
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-04-08 17:09:16 +08:00
chyezh
7b400252ff
fix: add configuration disk capacity config for lru and fix some bug ( #31977 )
...
issue: #30361
- Add configurable disk capacity limit
- fix bitset reset logic
- make insert record reinsert after clear
Signed-off-by: chyezh <chyezh@outlook.com>
2024-04-08 15:55:16 +08:00
congqixia
0feee53631
enhance: Add back unit test for compactor and fix some TODOs ( #31829 )
...
This PR adds back compactor "Unhandled" data type unit test and fixes
some TODOs behvaior
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-04-02 20:35:14 +08:00
jaime
bd853be8c7
enhance: Add db label for some usual metrics ( #30956 )
...
issue: #31782
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-04-02 14:27:13 +08:00
chyezh
1ad5ccc50f
enhance: add rg and db interface for segment and db/rg metric label ( #31715 )
...
issue: #30931
Signed-off-by: chyezh <chyezh@outlook.com>
2024-04-01 10:21:21 +08:00
cqy123456
976928ecd1
fix: fix fp16/bf16 some code missing and add more fp16/bf16 test ( #31612 )
...
issue: #31534
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-03-28 14:11:10 +08:00
Bingyi Sun
3d66670619
fix: add lazy load field to mark segment load type ( #31591 )
...
issue: https://github.com/milvus-io/milvus/issues/31673
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-03-28 11:23:10 +08:00
SimFG
b1a1cca10b
feat: add more operation detail info for better allocation ( #30438 )
...
issue: #30436
---------
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-03-28 06:33:11 +08:00
Bingyi Sun
fbff46a005
enhance: add lazyload global config ( #31610 )
...
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-03-27 20:23:10 +08:00
Jiquan Long
4eb4df1e81
fix: predict inverted index resource usage more reasonably ( #31615 )
...
/kind improvement
issue: #31617
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-03-27 19:33:09 +08:00
Bingyi Sun
8e661f791a
fix: lazy load index data in cache ( #31094 )
...
issue: https://github.com/milvus-io/milvus/issues/31571
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-03-25 15:43:07 +08:00
Bingyi Sun
0c868b654d
fix: fix mmap enabled check in resource estimation ( #31536 )
...
issue: https://github.com/milvus-io/milvus/issues/31569
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-03-25 14:29:07 +08:00
congqixia
7b38274cce
fix: Use mutex protection when accessing pks of l0 segments ( #31505 )
...
See also #31504
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-03-22 14:11:06 +08:00
Chun Han
c3264ca3e3
feat: support segment pruner ( #31003 )
...
related: #30376
2024-03-22 13:57:06 +08:00
Ted Xu
09281a07f4
enhance: disk cache to enable size based eviction ( #31469 )
...
See #31262
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-03-21 18:57:07 +08:00
Ted Xu
c13c96e321
feat: adding cache scavenger ( #31264 )
...
See #31262
---------
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-03-21 14:33:06 +08:00
chyezh
9f9ef8ac32
enhance: transfer resource group and dbname to querynode when load ( #30936 )
...
issue: #30931
Signed-off-by: chyezh <chyezh@outlook.com>
2024-03-21 11:59:12 +08:00
groot
c81909bfab
enhance: Support MinIO TLS connection ( #31311 )
...
issue: https://github.com/milvus-io/milvus/issues/30709
pr: #31292
Signed-off-by: yhmo <yihua.mo@zilliz.com>
Co-authored-by: Chen Rao <chenrao317328@163.com>
2024-03-21 11:15:20 +08:00
chyezh
8e293dc1ce
enhance: add resource usage estimate for segment interface ( #31050 )
...
issue: #30931
- move resource estimate function outside from segment loader.
- add load info and collection to base segment.
- add resource usage method for sealed segment.
Signed-off-by: chyezh <chyezh@outlook.com>
2024-03-19 11:53:05 +08:00
Bingyi Sun
bdc70dfc6a
feat: Add global mmap enable configuration ( #31267 )
...
https://github.com/milvus-io/milvus/issues/31279
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-03-18 15:17:10 +08:00
Bingyi Sun
17b4938985
fix: mmap does not work on index load ( #31228 )
...
issue: https://github.com/milvus-io/milvus/issues/31101
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-03-17 23:27:03 +08:00
congqixia
08aba2e05f
fix: Remove QueryNodeEntitiesSize after segment/collection released ( #31290 )
...
See also #31289
This PR:
- Set collection level `QueryNodeEntitiesSize` to zero if all segment
released
- Delete `QueryNodeEntitiesSize` metrics value after collection ref is
zero
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-03-15 15:43:04 +08:00
congqixia
b1ff8e2749
fix: Fix data race on loadSealedSegmentFields opts ( #31230 )
...
See also #31229
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-03-14 10:43:04 +08:00
Buqian Zheng
3c80083f51
feat: [Sparse Float Vector] add sparse vector support to milvus components ( #30630 )
...
add sparse float vector support to different milvus components,
including proxy, data node to receive and write sparse float vectors to
binlog, query node to handle search requests, index node to build index
for sparse float column, etc.
https://github.com/milvus-io/milvus/issues/29419
---------
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-03-13 14:32:54 -07:00
congqixia
57a5e44b88
enhance: Add id,type filter shortcut for QueryNode segment manager ( #31179 )
...
See also #31143
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-03-13 20:15:03 +08:00
Bingyi Sun
7b7187b465
fix: fix mmap load ( #31171 )
...
issue: https://github.com/milvus-io/milvus/issues/31101
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-03-13 10:49:04 +08:00
congqixia
1d96239137
enhance: Add ctx trace for segment load prepare ( #31203 )
...
Add ctx paramter to `prepare` and add trace id in underline logs.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-03-13 10:01:07 +08:00
wei liu
5016031dc7
enhance: add metrics for load segment progress ( #31005 )
...
This PR add metrics for load segment progress:
1. add metrics for load segment/index concurrency
2. add metrics for load index latency
3. change load segment latency's time unit to ms
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-03-08 15:39:02 +08:00
zhagnlu
a65a9ce8a5
fix: fix oom because of wrong load mode for index's raw data ( #31130 )
...
#30962
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-03-07 12:14:26 -08:00
Bingyi Sun
fd17a5f050
fix: check collection lazy load prop using schema ( #30992 )
...
issue: https://github.com/milvus-io/milvus/issues/30361
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-03-06 16:19:01 +08:00
congqixia
30398d4b71
enhance: Fix misleading log content & possible nil panic ( #31021 )
...
- Change load field log from "dy pool" to "load pool"
- Also defer delete when there is no error
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-03-05 16:33:00 +08:00
