13 Commits

Author SHA1 Message Date
XuanYang-cn
467113deab
fix: Accidently exit the check loop (#34481)
See also: #34460

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-07-08 16:46:13 +08:00
wayblink
efdaed4ac6
fix: fix clustering compaction can't enqueue when compaction queue is full (#34445)
#30633

---------

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-08 10:02:13 +08:00
cai.zhang
23e2a0e78b
fix: Remove unused code to fix compiling (#34406)
issue: #34407

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-04 08:48:08 +08:00
wayblink
a1232fafda
feat: Major compaction (#33620)
#30633

Signed-off-by: wayblink <anyang.wang@zilliz.com>
Co-authored-by: MrPresent-Han <chun.han@zilliz.com>
2024-06-10 21:34:08 +08:00
zhenshan.cao
ac4f3997ce
enhance: Reconstructing Compaction to possess persistence capability (#33265)
issue #33586

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-06-05 10:17:50 +08:00
yihao.dai
7730b910b9
enhance: Decouple compaction from shard (#33138)
Decouple compaction from shard, remove dependencies on shards (e.g.
SyncSegments, injection).

issue: https://github.com/milvus-io/milvus/issues/32809

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-05-24 09:07:41 +08:00
XuanYang-cn
a1386bae7f
fix: Skip to submit l0 tasks when scheduler full (#31270)
See also: #31242

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-03-15 10:21:12 +08:00
XuanYang-cn
def72947c7
fix: Trigger l0 compaction when l0 views don't change (#30729)
Trigger l0 compaction when l0 views don't change

So that leftover l0 segments would be compacted in the end.

1. Refresh LevelZero plans in comactionPlanHandler, remove the meta
dependency
of compaction trigger v2
2. Add ForceTrigger method for CompactionView interface
3. rename mu to taskGuard
4. Add a new TriggerTypeLevelZeroViewIDLE
5. Add an idleTicker for compaction view manager

See also: #30098, #30556

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-03-05 16:37:00 +08:00
XuanYang-cn
0b6beb7e0f
fix: Fill in info in CompactionSegmentBinlogs (#30279)
After #28873, PartitionID and CollectionID should be filled in
CompactionSegmentBinlog so that DataNode can compose
the correct logPath. However There're some places left forgotten to fill
in the information, causing Datanode downloading `xxx/0/0/xxxx/xxxx`
binlogs during compaction

See also: #30213

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-01-26 10:51:02 +08:00
smellthemoon
e52ce370b6
enhance:don't store logPath in meta to reduce memory (#28873)
don't store logPath in meta to reduce memory, when service get
segmentinfo, generate logpath from logid.
#28885

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-01-18 22:06:31 +08:00
XuanYang-cn
2c62bcbbf6
fix: Fix some left comments (#29027)
See also: #28814

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-12-13 18:48:39 +08:00
XuanYang-cn
5bac7f7897
fix: Fix L0 compaction in datacoord (#28814)
See also: #27606

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-12-05 18:44:37 +08:00
XuanYang-cn
9b371067d2
feat: Add Compaction views and triggers (#27906)
- Add Compaction l0 views
- Add Compaction scheduler
- Add Compaction triggerv2
- Add Compaction view manager

See also: #27606

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-11-23 17:30:25 +08:00