jaime
ee7dffc758
fix: sync task still running after DataNode has stopped ( #38441 )
...
issue: #38319
pr: #38377
---------
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-12-18 12:38:47 +08:00
SimFG
4c896c628c
fix: [2.4] replicate message exception when the ttMsgEnable config is changed dynamically ( #38440 )
...
- issue: #38177
- pr: #38178
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-12-14 23:24:51 +08:00
cai.zhang
85ade98eba
fix: [2.4]Fix sorting buffer in clustering compaction ( #38418 )
...
issue: #28410
master pr: #38417
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-12-13 10:12:49 +08:00
yihao.dai
9765db2465
fix: [2.4] Fix empty import task result ( #38317 )
...
Ensure the idempotency of import tasks to prevent duplicate tasks in
DataNode.
issue: https://github.com/milvus-io/milvus/issues/38313
pr: https://github.com/milvus-io/milvus/pull/38316
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-12-11 15:42:49 +08:00
wei liu
0c442b6fd6
fix: Datanode stop progress stuck at writer buffer memory check ( #38274 ) ( #38287 )
...
issue: #38273
pr: #38274 #38278
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-12-09 17:36:42 +08:00
cai.zhang
ddc40a7266
enhance: [2.4]Determine the number of buffers based on the resource limits of the DataNode ( #38210 )
...
issue: #28410
master pr: #38209
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-12-08 17:54:41 +08:00
jaime
67a004ca20
fix: invalid rate limit for time tick delay ( #38218 )
...
issue: #38217
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-12-05 14:20:41 +08:00
jaime
319f5494cd
enhance: optimize CPU usage for CheckHealth requests ( #35595 )
...
issue: #35563
pr: #35589
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-12-04 14:26:41 +08:00
congqixia
49fa78c481
enhance: [2.4] Make timeout work for each GetSegmentInfo req ( #38132 )
...
Cherry pick from master
pr: #36026
Relate: #36025
Fix datanode watch channel timeout when segment number is too large
Previous timeout apply for whole process for batching fetch segment
info, when segment number is large one rpc timeout does not work well
for multiple round rpc case
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Co-authored-by: aoiasd <zhicheng.yue@zilliz.com>
2024-12-02 20:14:39 +08:00
XuanYang-cn
8ae7cdd77d
fix: [24]Replace outer lock with concurrent map ( #37817 ) ( #37897 )
...
See also: #37493
pr: #37817
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-11-29 14:26:43 +08:00
XuanYang-cn
40fe656de9
enhance: compaction performance by remove paramtable get ( #37882 )
...
pr: #37163
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-11-28 10:38:37 +08:00
smellthemoon
b3e6482367
enhance: add search params in search request in restful( #36304 ) ( #37673 )
...
pr: #36304
pr: #36714
pr: #36448
---------
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-11-15 17:54:30 +08:00
XuanYang-cn
5d5f899274
fix: [cp24]Change memoryCheck write lock to read lock ( #37526 )
...
pr: #37525
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-11-15 14:42:31 +08:00
XuanYang-cn
d23da2db4f
fix: [cp24]Correct varchar primarykey size calculation ( #37619 )
...
See also: #37582
pr: #37617
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-11-14 14:16:38 +08:00
sthuang
9e8b6ace6d
enhance: [2.4] RBAC custom privilege group ( #37560 )
...
Cherry-pick from master
pr: https://github.com/milvus-io/milvus/pull/37087 ,
https://github.com/milvus-io/milvus/pull/37558
issue: #37031
---------
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-11-11 14:20:29 +08:00
wei liu
a9beca44ef
fix: watch channel stuck due to misuse of timer.Reset ( #37433 ) ( #37542 )
...
issue: #37166
pr: #37433
cause the misuse of timer.Reset, which cause dispatcher failed to send
msg to virtual channel buffer, and dispatcher do splitting again and
again, which hold the dispatcher manager's lock, block watching channel
progress.
This PR fix the misuse of timer.Reset
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-11-08 18:46:27 +08:00
SimFG
d0e78cef06
enhance: [2.4] update the expr version to fix the method call error ( #37260 )
...
/kind improvement
- pr: #37259
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-10-31 15:00:23 +08:00
yihao.dai
d30e27e6f9
enhance: Make dataNode.import.maxConcurrentTaskNum dynamic ( #37102 ) ( #37103 )
...
Resize import execution pool when config
`dataNode.import.maxConcurrentTaskNum` update.
issue: https://github.com/milvus-io/milvus/issues/37095
pr: https://github.com/milvus-io/milvus/pull/37102
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-10-25 18:21:29 +08:00
yihao.dai
539f56220f
enhance: Remove bf from datanode ( #36367 ) ( #37027 )
...
Remove bf from datanode:
1. When watching vchannels, skip loading **flushed** segments's bf. For
generating merged bf, we need to keep loading **growing** segments's bf.
2. Bypass bloom filter checks for delete messages, directly writing to
L0 segments.
3. In version 2.4, when dropping a partition, marking segments as
dropped depends on having the full segment list in the DataNode. So, we
need to keep syncing the segments every 10 minutes.
issue: https://github.com/milvus-io/milvus/issues/34585
pr: https://github.com/milvus-io/milvus/pull/35902 ,
https://github.com/milvus-io/milvus/pull/36367 ,
https://github.com/milvus-io/milvus/pull/36592
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-10-22 11:15:28 +08:00
cai.zhang
2bfd22f217
fix: [cherry-pick] Fix clustering compaction task leak ( #36803 )
...
issue: #36686
master pr: #36800
bug reason:
- The clustering compaction tasks on the datanode were never cleaned up.
- The clustering compaction task contains a mapping from clustering key
to buffer, this caused a large memory leak.
fix:
- clean the tasks on datanode by datacoord when clustering compaction
finished.
- reset the mapping that from clustering key to buffer on datanode when
clustering finished.
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-10-17 20:43:30 +08:00
XuanYang-cn
e976b41f97
fix: Remove enableLevelZeroSegment config ( #36507 )
...
See also: #36504
pr: #36535
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-10-11 16:41:21 +08:00
XuanYang-cn
28de6b86ba
fix: [24]fail to init fg clears flushTs so that slows flush ( #36741 )
...
See also: #36709
pr: #36740
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-10-11 16:39:21 +08:00
yihao.dai
a4ef93457d
enhance: Optimize import scheduling and add time cost metric ( #36601 ) ( #36684 )
...
1. Optimize import scheduling strategic:
a. Revise slot weights, calculating them based on the number of files
and segments for both import and pre-import tasks.
b. Ensure that the DN executes tasks in ascending order of task ID.
2. Add time cost metric and log.
issue: https://github.com/milvus-io/milvus/issues/36600 ,
https://github.com/milvus-io/milvus/issues/36518
pr: https://github.com/milvus-io/milvus/pull/36601
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-10-11 10:27:22 +08:00
XuanYang-cn
c69af44472
enhance: [24]Add more info in logs ( #36732 )
...
pr: #36535
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-10-10 17:51:25 +08:00
yihao.dai
c5088b8527
enhance: Add metrics to monitor import throughput and imported rows ( #36519 ) ( #36588 )
...
issue: https://github.com/milvus-io/milvus/issues/36518
pr: https://github.com/milvus-io/milvus/pull/36519
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-09-30 10:03:18 +08:00
cai.zhang
132e4c3ba1
fix: [cherry-pick]Fix data race for cluerting compaction ( #36499 )
...
issue: #36438
master pr: #36440
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-09-28 17:19:21 +08:00
congqixia
eb1602a374
enhance: [2.4] Graceful stop flowgraph manager when stopping datanode ( #36229 ) ( #36358 )
...
Cherry pick from master
pr: #36229
Flowgraph manager is not stopped during datanode stopping procedure
which may lead to unexpect flowgraph behavior during/after datanode stop
progress.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-19 17:39:12 +08:00
XuanYang-cn
6dc7d2041f
fix: Set an empty segment if compaction deleted all inserts ( #36045 )
...
See also: #36038
pr: #36044
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-09-06 20:09:05 +08:00
cai.zhang
b7a0e08dd3
fix: [cherry-pick]Fix data race for clustering compaction writer ( #35958 )
...
issue: #35950
master pr: #35957
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-09-05 04:07:10 +08:00
SimFG
8b706122a8
enhance: [2.4] support to drop the role which is related the privilege list ( #35863 )
...
- issue: #35545
- pr: #35727
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-08-31 21:57:02 +08:00
yihao.dai
be17cf6ff1
fix: Fix panic due to empty candidate import segments ( #35673 ) ( #35674 )
...
issue: https://github.com/milvus-io/milvus/issues/35662
pr: https://github.com/milvus-io/milvus/pull/35673
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-08-27 17:34:59 +08:00
XuanYang-cn
81c4cae2c1
enhance: Enable to write multiple segments in mix compactor ( #35648 )
...
Prevent segments to be written larger than maxSize * expansionRate
See also: #35584
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-08-24 10:00:58 +08:00
congqixia
f3fb37655b
fix: [2.4] Use correct release func when datanode unwatch a channel ( #35657 )
...
Cherry-pick from master
pr: #35655
See also #35654
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-23 14:09:03 +08:00
wei liu
e2542a1bf5
enhance: Update protobuf-go to protobuf-go v2 ( #34394 ) ( #35555 )
...
issue: #34252
pr: #34394 #35072 #35084
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-21 18:50:58 +08:00
cai.zhang
09aea3fbf1
enhance: [cherry-pick] Optimize the use of locks and avoid double flush clustering buffer writer ( #35490 )
...
issue: #35436
master pr: #35486
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-08-16 02:24:59 +08:00
aoiasd
a20cb727eb
enhance:[Cherry-pick] Check by proxy rate limiter when delete get data by query. ( #30891 ) ( #35262 )
...
relate: https://github.com/milvus-io/milvus/issues/30927
pr: https://github.com/milvus-io/milvus/pull/30891
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-08-13 14:32:21 +08:00
wei liu
8cd6718672
enhance: limit getSegmentInfo batch size to avoid excced grpc message limit ( #35432 )
...
issue: #35395
pr: #35394
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-13 11:42:19 +08:00
congqixia
e3cb994ca3
enhance: [2.4] Fix progress always zero due to integer divide ( #35393 )
...
Cherry-pick from master
pr: #35374
See also #35183
Signed-off-by: Congqi.Xia <congqi.xia@zilliz.com>
2024-08-09 17:20:18 +08:00
cai.zhang
4f0c1982d3
fix: [cherry-pick] Fix the issue of missing stats log after clustering compaction ( #35267 )
...
issue: #35265
master pr: #35266
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-08-07 12:08:16 +08:00
congqixia
b70ede4c9b
enhance: [2.4] Set WatchProgress for channel operation progress response ( #35183 ) ( #35231 )
...
Cherry-pick from master
pr: #35183
The watch progress is always zero in CheckChannelOperationProgress
response, which is meaningless and confusing. This PR set progress value
in rpc response to fix this problem.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-02 16:52:15 +08:00
cai.zhang
f7184101e1
fix: [cherry-pick] Fix data race for clustering buffer writer ( #35146 )
...
issue: #34495
master pr: #35145
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-08-01 16:08:13 +08:00
congqixia
d16320705e
enhance: [2.4] Add Segment Level in milvus segment info APIs ( #34763 ) ( #35023 )
...
Cherry-pick from master
pr: #34763
See also #34746
This PR add segment level field in response of
`GetPersistentSegmentInfo` and `GetQuerySegmentInfo`
---------
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-29 10:11:52 +08:00
congqixia
2a43f43916
fix: [2.4] Remove timeout in datanode watch ctx ( #35011 ) ( #35017 )
...
Cherry-pick from master
pr: #35011
See also #35008
Use tickle timeout logic instead of hardcode context timeout
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-26 11:59:46 +08:00
cai.zhang
9cd6dbcbc9
fix: [cherry-pick] Fix bug for block clustering compaction ( #35021 )
...
issue: #34703
master pr: #35019
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-26 11:33:40 +08:00
cai.zhang
74adedf750
enhance: Optimized the GC logic to ensure that memory is released in time ( #34950 )
...
issue: #34703
master pr: #34949
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-24 14:07:43 +08:00
wei liu
c13c48d99a
fix: Failed to unmarshal field stats's bloom filter ( #34922 )
...
pr #34377 introduce this issue, which miss some new changes during the
cherry-pick
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-23 16:45:47 +08:00
cai.zhang
6986dfdd5b
enhance:[cherry-pick]Send flush signal when the water level reaches the high watermark ( #34908 )
...
issue: #30633
master pr: #34907
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-23 10:27:42 +08:00
cai.zhang
4ed62e9dbb
enhance: [cherry-pick] Add integration test for clustering compaction ( #34860 )
...
issue: #34792
master pr: #34881
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2024-07-22 17:49:42 +08:00
wayblink
33bbc614df
enhance: [cherry-pick] add ut for clustering_compactor ( #34817 )
...
issue: #34792
pr: #34852
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-21 19:25:41 +08:00
cai.zhang
323dee2fbc
fix: [cherry-pick] Fix the issue of concurrent packing of the same segment ( #34838 )
...
issue: #34703
master pr: #34840
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-19 18:27:40 +08:00
