jaime
b37d6fa0f9
enhance: decrease cpu overhead during filter segments on datacoord ( #34231 )
...
issue: https://github.com/milvus-io/milvus/issues/33129
pr: #33130
pr: #33373
---------
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-07-01 10:20:08 +08:00
Bingyi Sun
460815ceab
fix: fix partition loaded num metric ( #33316 ) ( #34195 )
...
issue: https://github.com/milvus-io/milvus/issues/32108
related pr: #33316
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-06-28 14:58:05 +08:00
wei liu
18a0efe737
enhance: Avoid search querynode return nil status in response ( #34100 ) ( #34189 )
...
pr: #34100
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-06-26 18:52:04 +08:00
jaime
6423b6c718
enhance: move rocksmq from internal to pkg ( #34165 )
...
pr: https://github.com/milvus-io/milvus/pull/33881
issue: https://github.com/milvus-io/milvus/issues/33956
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-06-26 13:36:05 +08:00
cqy123456
f5344abdaf
enhance: [cherry-pick]growing segment support mmap ( #34110 )
...
issue: issue: https://github.com/milvus-io/milvus/issues/32984
related pr: https://github.com/milvus-io/milvus/pull/32633 ,
https://github.com/milvus-io/milvus/pull/33951 ,
https://github.com/milvus-io/milvus/pull/33993
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-06-25 14:52:07 +08:00
Jiquan Long
22e6807e9a
feat: support inverted index for array ( #33452 ) ( #34053 )
...
pr: https://github.com/milvus-io/milvus/pull/33184
pr: https://github.com/milvus-io/milvus/pull/33452
pr: https://github.com/milvus-io/milvus/pull/33633
issue: https://github.com/milvus-io/milvus/issues/27704
Co-authored-by: xiaocai2333 <cai.zhang@zilliz.com>
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
Co-authored-by: cai.zhang <cai.zhang@zilliz.com>
2024-06-24 10:50:03 +08:00
congqixia
e02a95e3c2
fix: [2.4] Return record with largest timestamp for entires with same PK ( #33936 ) ( #34024 )
...
Cherry-pick from master
pr: #33936
See also #33883
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-21 14:14:01 +08:00
congqixia
891a94ad9e
fix: [2.4] Check nodeID wildcard when removing pkOracle ( #33895 ) ( #34020 )
...
Cherry-pick from master
pr: #33895
See also #33894
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-21 12:04:00 +08:00
wei liu
fbc8fb3cb2
enhance: Skip return data distribution if no change happen ( #32814 ) ( #33985 )
...
issue: #32813
pr: #32814
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-06-21 10:24:12 +08:00
Gao
08c096cf55
enhance: Use primitive type for vectorType ( #33911 )
...
issue: #22837
pr: #33868
Use primitive type instead of proto enum type for queryHook to recognize
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-06-17 19:05:58 +08:00
congqixia
e8071830fa
fix: [2.4] Prevent use captured iteration variable partitionID ( #33907 )
...
Cherry-pick from master
pr: #33906
See also #33902
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-17 19:02:00 +08:00
Gao
5fc1370f6f
enhance: [2.4] autoindex for multi data type ( #33867 )
...
issue: #22837
pr: https://github.com/milvus-io/milvus/pull/33868
- opensource autoindex support
- metric type check for different data types
- autoindex data type for search param
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-06-14 23:26:00 +08:00
chyezh
dd6c982bdb
fix: load operation when segment is on releasing ( #33699 )
...
issue: #30857
pr: #31340
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-06-14 11:47:57 +08:00
wei liu
25d8b74f71
enhance: Execute bloom filter apply in parallel to speed up segment predict ( #33793 )
...
issue: #33610
pr: #33792
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-06-13 14:14:04 +08:00
wei liu
54feef30e7
enhance: Use BatchPkExist to reduce bloom filter func call cost ( #33752 )
...
issue: #33610
pr: #33611
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-06-12 17:45:58 +08:00
SimFG
c331aa4ad3
enhance: [2.4] add the includeCurrentMsg param for the Seek method ( #33743 )
...
/kind improvement
- issue: #33325
- pr: #33326
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-06-11 15:01:55 +08:00
yihao.dai
ed1dee9e38
enhance: Support L0 import ( #33514 ) ( #33712 )
...
issue: https://github.com/milvus-io/milvus/issues/33157
pr: https://github.com/milvus-io/milvus/pull/33514
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-06-08 11:17:52 +08:00
chyezh
93348af5c0
fix: async warmup will be blocked by state lock ( #33687 )
...
issue: #33685
pr: #33686
Signed-off-by: chyezh <chyezh@outlook.com>
2024-06-07 14:23:54 +08:00
Xiaofan
d331b403c3
enhance: Remove l0 delete cache ( #33537 )
...
Cherry pick from master
pr: #32989
remove l0 cache and build delete pk and ts everytime. this reduce the
memory and also increase the code readability
Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
2024-06-06 17:13:50 +08:00
wei liu
0c6354018b
enhance: Avoid load bf in delegator when qn worker has no more memory( #33557 ) ( #33650 )
...
pr: #33557
query coord send load request to delegator, delegator load bf first,
then forward load request to qn worker. but when qn worker has no more
memory, it will return load failed immediatelly. then delegator roll
back the loaded bf. query coord wil retry the load request, and
delegator will load and roll back bf again and again.
this PR delay the loading bf step until load segment succeed in worker.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-06-06 10:41:52 +08:00
Chun Han
627b787aed
fix: query iterator lack results( #33137 ) ( #33422 ) ( #33506 )
...
related: #33137
pr: https://github.com/milvus-io/milvus/pull/33422
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-06-05 18:51:52 +08:00
jaime
0ad55c6c44
fix: fix loaded entity num is inaccurate ( #33522 )
...
issue: #33520
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-06-04 20:09:54 +08:00
yihao.dai
7384bfe3f8
fix: use seperate warmup pool and disable warmup by default ( #33348 ) ( #33349 )
...
1. use a small warmup pool to reduce the impact of warmup
2. change the warmup pool to nonblocking mode
3. disable warmup by default
4. remove the maximum size limit of 16 for the load pool
issue: https://github.com/milvus-io/milvus/issues/32772
pr: https://github.com/milvus-io/milvus/pull/33348
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Co-authored-by: xiaofanluan <xiaofan.luan@zilliz.com>
2024-05-28 19:27:43 +08:00
jaime
8990b8b051
fix: correct error of metrics stats ( #33305 )
...
issue: #32980
cherry pick from master
pr: #33075 #33255
---------
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-05-24 09:15:41 +08:00
Bingyi Sun
d4a146ef1a
enhance: mmap load raw data if scalar index does not have raw data (#… ( #33317 )
...
pr: #33175
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-05-24 09:09:42 +08:00
Jiquan Long
d98e1f6ff5
fix: two-phase retrieval on lru-segment ( #32945 ) ( #33313 )
...
Cherry-pick from master
pr: #32945
issue: #31822
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-05-23 16:25:40 +08:00
wei liu
a988e7cabc
enhance: Reduce bloom filter lock contention between insert and delete in query coord ( #32643 ) ( #33284 )
...
issue: #32530
pr: #32643
cause ProcessDelete need to check whether pk exist in bloom filter, and
ProcessInsert need to update pk to bloom filter, when execute
ProcessInsert and ProcessDelete in parallel, it will cause race
condition in segment's bloom filter
This PR execute ProcessInsert and ProcessDelete in serial to avoid block
each other
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-05-22 20:53:40 +08:00
cai.zhang
6ea7633bd5
enhance: Add memory size for binlog ( #33025 )
...
issue: #33005
1. add `MemorySize` field for insert binlog.
2. `LogSize` means the file size in the storage object.
3. `MemorySize` means the size of the data in the memory.
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2024-05-15 12:59:34 +08:00
SimFG
1d48d0aeb2
enhance: use different value to get related data size according to segment type ( #33017 )
...
issue: #30436
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-05-14 14:59:33 +08:00
Cai Yudong
4fc7915c70
enhance: unify data generation test APIs ( #32955 )
...
Issue: #22837
Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2024-05-14 14:33:33 +08:00
chyezh
96489b814d
fix: remove busy log ( #33042 )
...
issue: #32963
Signed-off-by: chyezh <chyezh@outlook.com>
2024-05-14 14:20:32 +08:00
foxspy
f6777267e3
enhance: add score compute consistency config for knowhere ( #32997 )
...
issue: https://github.com/milvus-io/milvus/issues/32583
related: #32584
Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
2024-05-13 14:21:31 +08:00
chyezh
1c84a1c9b6
fix: lru related issue fixup patch ( #32916 )
...
issue: #32206 , #32801
- search failure with some assertion, segment not loaded and resource
insufficient.
- segment leak when query segments
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-05-10 19:17:30 +08:00
wei liu
25689859a1
fix: Load index metric use wrong time unit ( #32935 )
...
issue:#32899
This PR fix the wrong metric value of load index, which introduced by
pr#32567, use wrong time unit for load index metrics
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-05-10 18:07:30 +08:00
Jiquan Long
0783582e2e
fix: temporarily disable two-phase retrieval when lru is enabled ( #32927 )
...
issue: #31822
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-05-10 14:19:45 +08:00
Bingyi Sun
5cbf081111
fix: fix index resource estimation ( #32842 )
...
issue: #32820
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-05-10 11:53:30 +08:00
Bingyi Sun
17a79f4ca9
enhance: The LRU cache evicts items and retries loading if the disk limit is reached. ( #32819 )
...
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-05-08 14:45:30 +08:00
wei liu
5038036ece
enhance: Reuse hash locations during access bloom fitler ( #32642 )
...
issue: #32530
when try to match segment bloom filter with pk, we can reuse the hash
locations. This PR maintain the max hash Func, and compute hash location
once for all segment, reuse hash location can speed up bf access
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-05-07 06:13:47 -07:00
congqixia
40728ce83d
enhance: Add metautil.Channel to convert string compare to int ( #32749 )
...
See also #32748
This PR:
- Add `metautil.Channel` utiltiy which convert virtual name to physical
channel name, collectionID and shard idx
- Add channel mapper interface & implementation to convert limited
physical channel name into int index
- Apply `metautil.Channel` filter in querynode segment manager logic
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-05-07 19:13:35 +08:00
yihao.dai
9db3aa18bc
enhance: Remove deprecated EnableIndex ( #32704 )
...
/kind improvement
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-05-07 17:11:30 +08:00
chyezh
641f702f64
fix: add request resource timeout for lazy load, refactor context usage in cache ( #32709 )
...
issue: #32663
- Use new param to control request resource timeout for lazy load.
- Remove the timeout parameter of `Do`, remove `DoWait`. use `context`
to control the timeout.
- Use `VersionedNotifier` to avoid notify event lost and broadcast,
remove the redundant goroutine in cache.
related dev pr: #32684
Signed-off-by: chyezh <chyezh@outlook.com>
2024-05-07 16:33:30 +08:00
congqixia
efa0c73c62
fix: Unify querynode metrics cleanup in collection release ( #32805 )
...
Related to #32803
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-05-07 15:41:29 +08:00
aoiasd
31dca3249e
enhance: add type info for payload writer error message and add log when querynode find new collection ( #32522 )
...
relate: https://github.com/milvus-io/milvus/issues/32668
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-05-07 14:45:29 +08:00
Jiquan Long
1f58cda957
enhance: add more trace for search & query ( #32734 )
...
issue: https://github.com/milvus-io/milvus/issues/32728
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-05-07 13:03:29 +08:00
yihao.dai
cf4db3ff4e
enhance: Fix compilation error ( #32797 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-05-06 19:31:49 -07:00
congqixia
7102403a6b
fix: Add Wrapper and Keepalive for CTraceContext ids ( #32746 )
...
See also #32742
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-05-07 10:05:35 +08:00
congqixia
53b5f1be17
enhance: Remove legacy L0 segment if watch failed ( #32725 )
...
Like growing segments, legacy l0 segments shall be removed if watch dml
channel execution fails as well.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-05-07 10:03:42 +08:00
Bingyi Sun
fecd9c21ba
feat: LRU cache implementation ( #32567 )
...
issue: https://github.com/milvus-io/milvus/issues/32783
This pr is the implementation of lru cache on branch lru-dev.
Signed-off-by: sunby <sunbingyi1992@gmail.com>
Co-authored-by: chyezh <chyezh@outlook.com>
Co-authored-by: MrPresent-Han <chun.han@zilliz.com>
Co-authored-by: Ted Xu <ted.xu@zilliz.com>
Co-authored-by: jaime <yun.zhang@zilliz.com>
Co-authored-by: wayblink <anyang.wang@zilliz.com>
2024-05-06 20:29:30 +08:00
Chun Han
ac82cef04d
enhance: disable reload partstats by config ( #32702 )
...
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-04-29 19:11:26 +08:00
wei liu
c0555d4b45
fix: Remove read only node from replica immedaitely after node down ( #32666 )
...
issue: #32665
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-04-28 20:25:25 +08:00
