cai.zhang
4c45bc412f
enhance: Add integration test for clustering compaction ( #34881 )
...
issue: #34792
---------
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-23 10:13:43 +08:00
jaime
3cd24f7548
fix: collection meta is not removed after gc in DataCoord ( #34883 )
...
issue: #34847
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-07-22 21:11:48 +08:00
wayblink
52fa668f5d
fix: fix collection is compacting logic ( #34855 )
...
fix : #34861
also related to: #34792
---------
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-22 11:35:41 +08:00
yihao.dai
8aab6cbfac
enhance: Organize the common modules of streamingNode and dataNode ( #34773 )
...
1. Move the common modules of streamingNode and dataNode to flushcommon
2. Add new GetVChannels interface for rootcoord
issue: https://github.com/milvus-io/milvus/issues/33285
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-22 11:33:51 +08:00
wayblink
c2b8b5fe84
enhance: refine clustering compaction configs and logs ( #34784 )
...
#30633
---------
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-21 19:23:40 +08:00
yihao.dai
b22e549844
enhance: Rename config of sealing by growing segmetns size ( #34787 )
...
/kind improvement
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-19 20:27:41 +08:00
wayblink
a11714913b
fix: fix unstable ut of compaction pickSlot ( #34828 )
...
#34827
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-19 17:53:47 +08:00
wayblink
c79d1af390
enhance: Add compaction task slot usage logic ( #34581 )
...
#34544
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-18 10:27:41 +08:00
yihao.dai
4939f82d4f
enhance: Seal by total growing segments size ( #34692 )
...
Seals the largest growing segment if the total size of growing segments
of each shard exceeds the size threshold(default 4GB). Introducing this
policy can help keep the size of growing segments within a suitable
level, alleviating the pressure on the delegator.
issue: https://github.com/milvus-io/milvus/issues/34554
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-17 21:45:41 +08:00
congqixia
67324eb809
enhance: Add l0 segment entry num quota ( #34733 )
...
See also #34670
This PR add quota configuration for l0 segment entry number per
collection. If l0 compaction cannot keep up the insertion/upsertion
rate, this feature could back press the related rate.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-17 17:35:41 +08:00
yihao.dai
ca758c36cc
enhance: Pre-allocate ids for compaction ( #34187 )
...
This PR removes the dependency of compaction on the ID allocator by
pre-allocating the logID and segmentID.
issue: https://github.com/milvus-io/milvus/issues/33957
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-17 13:23:42 +08:00
wayblink
90e765db87
fix: fix bug if datacoord restart when l0 compaction saveSegmentMeta ( #34679 )
...
#34681
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-17 13:03:41 +08:00
XuanYang-cn
fd7221f4be
fix: Fix accidentlly exit MixCompaction task loop ( #34688 )
...
See also: #33431 , #34460
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-07-16 15:57:42 +08:00
smellthemoon
b5b5afb63f
fix: [skip e2e]golang lint check fail ( #34682 )
...
#34680
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-07-16 10:47:36 +08:00
yihao.dai
1495b7a2c9
fix: Fix no L0 compaction tasks are executed ( #34537 )
...
The nodeID for compaction task initialization is 0. This PR adjusts the
task reassignment conditions to allow new compaction tasks to be
reassigned and executed.
issue: https://github.com/milvus-io/milvus/issues/34460
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-15 17:01:37 +08:00
congqixia
8b5754f7fe
enhance: Add segment seal proportion jitter ( #34636 )
...
See also #34574
Add jitter for segment seal proportion to avoid seal operation burst in
short period of time.
This PR also fix license header in paramtable pkg.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-15 14:47:39 +08:00
wayblink
48898cefea
fix: fix meta revert bug when clustering compaction task fail ( #34604 )
...
#34603
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-14 22:29:38 +08:00
cai.zhang
93a500676b
fix: Remove flushed segment in segment manager generated through import ( #34649 )
...
issue: #34648
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-12 23:15:22 +08:00
congqixia
1a248f2668
enhance: Add param item for segmentFlushInterval ( #34629 )
...
See also #28817
Add paramitem for segment flush interval
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-12 18:59:35 +08:00
wayblink
358e9a10d2
enhance: Alter compactTo segments before compactFrom to avoid data loss if crash ( #34513 )
...
#34512
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-12 00:55:34 +08:00
jaime
a08a0c831f
fix: encountering orphan channel-cp meta after DataCoord GC ( #34612 )
...
issue: #34545
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-07-11 23:01:35 +08:00
XuanYang-cn
d7966f46ad
fix: Compaction task l0 state transfer wrong ( #34597 )
...
This bug caused failed L0 compaction tasks never end.
See also: #34460
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-07-11 21:39:36 +08:00
Patrick Weizhi Xu
104d0966b7
feat: support partition key isolation ( #34336 )
...
issue: #34332
---------
Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
2024-07-11 19:01:35 +08:00
XuanYang-cn
d7a3697fb5
enhance: Add back compactionTaskNum metrics ( #34583 )
...
Fix L0 compaction task recover unable to set segment not isCompacting
See also: #34460
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-07-11 17:45:37 +08:00
wayblink
fd3da90089
enhance: Refine clustering compaction latency metrics ( #34528 )
...
#34527
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-11 17:43:43 +08:00
XuanYang-cn
e0b39d8bf4
fix: Milvus panic when compaction disabled and dropping a collection ( #34103 )
...
See also: #31059
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-07-11 14:44:52 +08:00
congqixia
be92147723
enhance: Print reason when seal segments ( #34575 )
...
See also #34574
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-11 11:40:51 +08:00
jaime
c332f69dec
enhance: skip orphan channel cp meta when checking cp lag ( #34555 )
...
issue: # #34545
Print warn log instead of check health fail if orphan channel cp meta is
found in health check request.
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-07-11 09:36:56 +08:00
XuanYang-cn
314f4d995b
enhance: Tidy dc channel manager ( #34515 )
...
See also: #34518
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-07-09 18:26:12 +08:00
wayblink
0c4d82c6d3
fix: fix compaction task startTime is 0 ( #34496 )
...
fix bug: #34497 #34498
---------
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-09 10:22:13 +08:00
congqixia
3333160b8d
enhance: Fix lint issues from recent PRs ( #34482 )
...
See also #34483
Some lint issues are introduced due to lack of static check run. This PR
fixes these problems.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-09 10:06:24 +08:00
Gao
ae6d6f91e6
enhance: change autoindex default metric type ( #34261 )
...
issue: #34304
cosine is more widely used in float vectors, and cosine and hamming
distance are 'metrics' which have good geometric properties
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-07-08 19:52:24 +08:00
XuanYang-cn
467113deab
fix: Accidently exit the check loop ( #34481 )
...
See also: #34460
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-07-08 16:46:13 +08:00
wayblink
efdaed4ac6
fix: fix clustering compaction can't enqueue when compaction queue is full ( #34445 )
...
#30633
---------
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-08 10:02:13 +08:00
yihao.dai
4e5f1d5f75
enhance: Pre-allocate ids for import ( #33958 )
...
The import is dependent on syncTask, which in turn relies on the
allocator. This PR pre-allocate the necessary IDs for import syncTask.
issue: https://github.com/milvus-io/milvus/issues/33957
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-07 21:26:14 +08:00
jaime
21fc5f5d46
enhance: Remove datanode reporting TT based on MQ implementation ( #34421 )
...
issue: #34420
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-07-05 15:48:09 +08:00
congqixia
439fb24b32
fix: Skip l0 segments when syncing segments to datanodes ( #34388 )
...
See also #34387
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-04 18:02:14 +08:00
congqixia
d51d0954bd
enhance: Continue loop when reassign channel fails ( #34331 )
...
Log will be confusing when `Reassign` channel operation failed for both
success & failure log will be printed in row. This PR continue the loop
to avoid this output.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-04 14:20:10 +08:00
cai.zhang
23e2a0e78b
fix: Remove unused code to fix compiling ( #34406 )
...
issue: #34407
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-04 08:48:08 +08:00
jaime
d1f57aa4ba
enhance: remove deprecated code within channel manager ( #34340 )
...
issue: https://github.com/milvus-io/milvus/issues/33994
only remove deprecated code, no additional changes.
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-07-03 19:46:09 +08:00
wayblink
da56880d0f
fix: Avoid datarace in clustering compaction ( #34288 )
...
#34289
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-03 19:08:09 +08:00
wayblink
a501fa11c0
fix: remove isFull check in compaction.enqueue ( #34338 )
...
#30633
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-03 18:10:10 +08:00
cai.zhang
feb13cdf07
enhance: Skip pick worker when task doesn't need to execute actually ( #34348 )
...
issue: #34347
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-03 15:52:09 +08:00
cai.zhang
8165961223
enhance: Optimeize clustering compaction ( #34313 )
...
issue: #30633
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-03 15:14:08 +08:00
wayblink
fb61344dc9
fix: Revert a optimize in clustering compaction ( #34299 )
...
#30633
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-03 10:32:08 +08:00
cai.zhang
a5be322ab2
enhance: Don't set the timeout for SyncSegments ( #34226 )
...
issue: #32809
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-02 14:28:07 +08:00
cai.zhang
4cf1a358ba
fix: Sync the sealed and flushed segments to datanode ( #34301 )
...
issue: #33696
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-01 22:42:08 +08:00
congqixia
e083d99dd9
enhance: Use channel name map finding channel watcher ( #34294 )
...
See also #33235
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-01 18:46:07 +08:00
jaime
d6afb31b94
enhance: make subfunctions of datanode component modular ( #33992 )
...
issue: #33994
also remove deprecated channel manager based on the etcd implementation
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-07-01 14:46:07 +08:00
jaime
0426390f06
enhance: improve check health ( #33800 )
...
issue: #34264
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-07-01 10:16:06 +08:00
