yihao.dai
de78de7689
fix: [10kcp] Fix consume blocked due to too many consumers ( #38456 )
...
This PR limits the maximum number of consumers per pchannel to 10 for
each QueryNode and DataNode.
issue: https://github.com/milvus-io/milvus/issues/37630
pr: https://github.com/milvus-io/milvus/pull/38455
---------
Signed-off-by: SimFG <bang.fu@zilliz.com>
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Co-authored-by: SimFG <bang.fu@zilliz.com>
2024-12-13 21:20:47 +08:00
yihao.dai
3d490aa158
fix: [10kcp] Replace outer lock with concurrent map ( #38286 )
...
See also: #37493
pr: #37817
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
Co-authored-by: XuanYang-cn <xuan.yang@zilliz.com>
2024-12-09 19:49:20 +08:00
yihao.dai
338ccc9ff9
enhance: [10kcp] Reduce memory usage of BF in DataNode and QueryNode ( #38133 )
...
1. DataNode: Skip generating BF during the insert phase (BF will be
regenerated during the sync phase).
2. QueryNode: Skip generating or maintaining BF for growing segments;
deletion checks will be handled in the segcore.
issue: https://github.com/milvus-io/milvus/issues/37630
pr: https://github.com/milvus-io/milvus/pull/38129
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-12-02 14:41:19 +08:00
smellthemoon
b3e6482367
enhance: add search params in search request in restful( #36304 ) ( #37673 )
...
pr: #36304
pr: #36714
pr: #36448
---------
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-11-15 17:54:30 +08:00
XuanYang-cn
5d5f899274
fix: [cp24]Change memoryCheck write lock to read lock ( #37526 )
...
pr: #37525
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-11-15 14:42:31 +08:00
XuanYang-cn
d23da2db4f
fix: [cp24]Correct varchar primarykey size calculation ( #37619 )
...
See also: #37582
pr: #37617
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-11-14 14:16:38 +08:00
sthuang
9e8b6ace6d
enhance: [2.4] RBAC custom privilege group ( #37560 )
...
Cherry-pick from master
pr: https://github.com/milvus-io/milvus/pull/37087 ,
https://github.com/milvus-io/milvus/pull/37558
issue: #37031
---------
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-11-11 14:20:29 +08:00
wei liu
a9beca44ef
fix: watch channel stuck due to misuse of timer.Reset ( #37433 ) ( #37542 )
...
issue: #37166
pr: #37433
cause the misuse of timer.Reset, which cause dispatcher failed to send
msg to virtual channel buffer, and dispatcher do splitting again and
again, which hold the dispatcher manager's lock, block watching channel
progress.
This PR fix the misuse of timer.Reset
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-11-08 18:46:27 +08:00
SimFG
d0e78cef06
enhance: [2.4] update the expr version to fix the method call error ( #37260 )
...
/kind improvement
- pr: #37259
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-10-31 15:00:23 +08:00
yihao.dai
d30e27e6f9
enhance: Make dataNode.import.maxConcurrentTaskNum dynamic ( #37102 ) ( #37103 )
...
Resize import execution pool when config
`dataNode.import.maxConcurrentTaskNum` update.
issue: https://github.com/milvus-io/milvus/issues/37095
pr: https://github.com/milvus-io/milvus/pull/37102
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-10-25 18:21:29 +08:00
yihao.dai
539f56220f
enhance: Remove bf from datanode ( #36367 ) ( #37027 )
...
Remove bf from datanode:
1. When watching vchannels, skip loading **flushed** segments's bf. For
generating merged bf, we need to keep loading **growing** segments's bf.
2. Bypass bloom filter checks for delete messages, directly writing to
L0 segments.
3. In version 2.4, when dropping a partition, marking segments as
dropped depends on having the full segment list in the DataNode. So, we
need to keep syncing the segments every 10 minutes.
issue: https://github.com/milvus-io/milvus/issues/34585
pr: https://github.com/milvus-io/milvus/pull/35902 ,
https://github.com/milvus-io/milvus/pull/36367 ,
https://github.com/milvus-io/milvus/pull/36592
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-10-22 11:15:28 +08:00
cai.zhang
2bfd22f217
fix: [cherry-pick] Fix clustering compaction task leak ( #36803 )
...
issue: #36686
master pr: #36800
bug reason:
- The clustering compaction tasks on the datanode were never cleaned up.
- The clustering compaction task contains a mapping from clustering key
to buffer, this caused a large memory leak.
fix:
- clean the tasks on datanode by datacoord when clustering compaction
finished.
- reset the mapping that from clustering key to buffer on datanode when
clustering finished.
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-10-17 20:43:30 +08:00
XuanYang-cn
e976b41f97
fix: Remove enableLevelZeroSegment config ( #36507 )
...
See also: #36504
pr: #36535
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-10-11 16:41:21 +08:00
XuanYang-cn
28de6b86ba
fix: [24]fail to init fg clears flushTs so that slows flush ( #36741 )
...
See also: #36709
pr: #36740
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-10-11 16:39:21 +08:00
yihao.dai
a4ef93457d
enhance: Optimize import scheduling and add time cost metric ( #36601 ) ( #36684 )
...
1. Optimize import scheduling strategic:
a. Revise slot weights, calculating them based on the number of files
and segments for both import and pre-import tasks.
b. Ensure that the DN executes tasks in ascending order of task ID.
2. Add time cost metric and log.
issue: https://github.com/milvus-io/milvus/issues/36600 ,
https://github.com/milvus-io/milvus/issues/36518
pr: https://github.com/milvus-io/milvus/pull/36601
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-10-11 10:27:22 +08:00
XuanYang-cn
c69af44472
enhance: [24]Add more info in logs ( #36732 )
...
pr: #36535
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-10-10 17:51:25 +08:00
yihao.dai
c5088b8527
enhance: Add metrics to monitor import throughput and imported rows ( #36519 ) ( #36588 )
...
issue: https://github.com/milvus-io/milvus/issues/36518
pr: https://github.com/milvus-io/milvus/pull/36519
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-09-30 10:03:18 +08:00
cai.zhang
132e4c3ba1
fix: [cherry-pick]Fix data race for cluerting compaction ( #36499 )
...
issue: #36438
master pr: #36440
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-09-28 17:19:21 +08:00
congqixia
eb1602a374
enhance: [2.4] Graceful stop flowgraph manager when stopping datanode ( #36229 ) ( #36358 )
...
Cherry pick from master
pr: #36229
Flowgraph manager is not stopped during datanode stopping procedure
which may lead to unexpect flowgraph behavior during/after datanode stop
progress.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-19 17:39:12 +08:00
XuanYang-cn
6dc7d2041f
fix: Set an empty segment if compaction deleted all inserts ( #36045 )
...
See also: #36038
pr: #36044
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-09-06 20:09:05 +08:00
cai.zhang
b7a0e08dd3
fix: [cherry-pick]Fix data race for clustering compaction writer ( #35958 )
...
issue: #35950
master pr: #35957
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-09-05 04:07:10 +08:00
SimFG
8b706122a8
enhance: [2.4] support to drop the role which is related the privilege list ( #35863 )
...
- issue: #35545
- pr: #35727
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-08-31 21:57:02 +08:00
yihao.dai
be17cf6ff1
fix: Fix panic due to empty candidate import segments ( #35673 ) ( #35674 )
...
issue: https://github.com/milvus-io/milvus/issues/35662
pr: https://github.com/milvus-io/milvus/pull/35673
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-08-27 17:34:59 +08:00
XuanYang-cn
81c4cae2c1
enhance: Enable to write multiple segments in mix compactor ( #35648 )
...
Prevent segments to be written larger than maxSize * expansionRate
See also: #35584
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-08-24 10:00:58 +08:00
congqixia
f3fb37655b
fix: [2.4] Use correct release func when datanode unwatch a channel ( #35657 )
...
Cherry-pick from master
pr: #35655
See also #35654
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-23 14:09:03 +08:00
wei liu
e2542a1bf5
enhance: Update protobuf-go to protobuf-go v2 ( #34394 ) ( #35555 )
...
issue: #34252
pr: #34394 #35072 #35084
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-21 18:50:58 +08:00
cai.zhang
09aea3fbf1
enhance: [cherry-pick] Optimize the use of locks and avoid double flush clustering buffer writer ( #35490 )
...
issue: #35436
master pr: #35486
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-08-16 02:24:59 +08:00
aoiasd
a20cb727eb
enhance:[Cherry-pick] Check by proxy rate limiter when delete get data by query. ( #30891 ) ( #35262 )
...
relate: https://github.com/milvus-io/milvus/issues/30927
pr: https://github.com/milvus-io/milvus/pull/30891
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-08-13 14:32:21 +08:00
wei liu
8cd6718672
enhance: limit getSegmentInfo batch size to avoid excced grpc message limit ( #35432 )
...
issue: #35395
pr: #35394
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-13 11:42:19 +08:00
congqixia
e3cb994ca3
enhance: [2.4] Fix progress always zero due to integer divide ( #35393 )
...
Cherry-pick from master
pr: #35374
See also #35183
Signed-off-by: Congqi.Xia <congqi.xia@zilliz.com>
2024-08-09 17:20:18 +08:00
cai.zhang
4f0c1982d3
fix: [cherry-pick] Fix the issue of missing stats log after clustering compaction ( #35267 )
...
issue: #35265
master pr: #35266
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-08-07 12:08:16 +08:00
congqixia
b70ede4c9b
enhance: [2.4] Set WatchProgress for channel operation progress response ( #35183 ) ( #35231 )
...
Cherry-pick from master
pr: #35183
The watch progress is always zero in CheckChannelOperationProgress
response, which is meaningless and confusing. This PR set progress value
in rpc response to fix this problem.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-02 16:52:15 +08:00
cai.zhang
f7184101e1
fix: [cherry-pick] Fix data race for clustering buffer writer ( #35146 )
...
issue: #34495
master pr: #35145
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-08-01 16:08:13 +08:00
congqixia
d16320705e
enhance: [2.4] Add Segment Level in milvus segment info APIs ( #34763 ) ( #35023 )
...
Cherry-pick from master
pr: #34763
See also #34746
This PR add segment level field in response of
`GetPersistentSegmentInfo` and `GetQuerySegmentInfo`
---------
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-29 10:11:52 +08:00
congqixia
2a43f43916
fix: [2.4] Remove timeout in datanode watch ctx ( #35011 ) ( #35017 )
...
Cherry-pick from master
pr: #35011
See also #35008
Use tickle timeout logic instead of hardcode context timeout
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-26 11:59:46 +08:00
cai.zhang
9cd6dbcbc9
fix: [cherry-pick] Fix bug for block clustering compaction ( #35021 )
...
issue: #34703
master pr: #35019
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-26 11:33:40 +08:00
cai.zhang
74adedf750
enhance: Optimized the GC logic to ensure that memory is released in time ( #34950 )
...
issue: #34703
master pr: #34949
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-24 14:07:43 +08:00
wei liu
c13c48d99a
fix: Failed to unmarshal field stats's bloom filter ( #34922 )
...
pr #34377 introduce this issue, which miss some new changes during the
cherry-pick
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-23 16:45:47 +08:00
cai.zhang
6986dfdd5b
enhance:[cherry-pick]Send flush signal when the water level reaches the high watermark ( #34908 )
...
issue: #30633
master pr: #34907
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-23 10:27:42 +08:00
cai.zhang
4ed62e9dbb
enhance: [cherry-pick] Add integration test for clustering compaction ( #34860 )
...
issue: #34792
master pr: #34881
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2024-07-22 17:49:42 +08:00
wayblink
33bbc614df
enhance: [cherry-pick] add ut for clustering_compactor ( #34817 )
...
issue: #34792
pr: #34852
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-21 19:25:41 +08:00
cai.zhang
323dee2fbc
fix: [cherry-pick] Fix the issue of concurrent packing of the same segment ( #34838 )
...
issue: #34703
master pr: #34840
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-19 18:27:40 +08:00
wayblink
a26e965e6a
enhance:[cherry-pick] Add compaction task slot usage logic ( #34625 )
...
issue: #34544
pr: #34581
---------
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-18 09:55:43 +08:00
cai.zhang
d74a5bdc1d
fix: [cherry-pick] Fix bug where binlogs already flushed with new segment during pack ( #34760 )
...
issue: #34703
master pr: #34762
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-18 09:15:47 +08:00
cai.zhang
19d3606e0d
fix: [cherry-pick] Fix the bug that caused small segment flush frequently ( #34727 )
...
issue: #34703
master pr: #34725
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-17 09:47:43 +08:00
XuanYang-cn
5909a62ca6
fix: Fix accidentlly exit MixCompaction task loop ( #34689 )
...
See also: #33431 , #34460
pr: #34688
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-07-16 16:21:38 +08:00
cai.zhang
0c5aafd2d1
fix:[cherry-pick] Reset flushed row num after pack segment for clustering compaction ( #34704 )
...
issue: #34703
master pr: #34702
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-16 15:57:43 +08:00
SimFG
15adb2feac
enhance: [2.4] add the seal segment when dispatch delete msgs ( #34566 )
...
/kind improvement
- pr: #34565
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-07-10 19:18:51 +08:00
SimFG
737bd7c734
enhance: [2.4] release the record in delete codec and add some log for compaction ( #34506 )
...
/kind improvement
- pr: #34454
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-07-09 15:40:17 +08:00
yihao.dai
0d7ba810b3
enhance: Check segment existence when FlushSegments and add some key logs ( #34438 ) ( #34472 )
...
Check if the segment exists during FlushSegments and add some key logs
in write path.
issue: https://github.com/milvus-io/milvus/issues/34255
pr: https://github.com/milvus-io/milvus/pull/34438
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-08 19:00:13 +08:00
