mirror of https://gitee.com/milvus-io/milvus.git synced 2026-01-07 11:21:52 +08:00

History

fix: Trigger l0 compaction when l0 views don't change (#30729 )

Trigger l0 compaction when l0 views don't change

So that leftover l0 segments would be compacted in the end.

1. Refresh LevelZero plans in comactionPlanHandler, remove the meta
dependency
of compaction trigger v2
2. Add ForceTrigger method for CompactionView interface
3. rename mu to taskGuard
4. Add a new TriggerTypeLevelZeroViewIDLE
5. Add an idleTicker for compaction view manager

See also: #30098, #30556

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

2024-03-05 16:37:00 +08:00

broker

enhance: index meta use independent rather than global meta lock (#30869 )

2024-03-04 16:56:59 +08:00

allocator_test.go

Decoupling client and server API in types interface (#27186 )

2023-09-26 09:57:25 +08:00

allocator.go

feat: Add import scheduler and manager (#29367 )

2024-03-01 18:31:02 +08:00

build_index_policy.go

Format the code (#27275 )

2023-09-21 09:45:27 +08:00

channel_checker_test.go

Format the code (#27275 )

2023-09-21 09:45:27 +08:00

channel_checker.go

Format the code (#27275 )

2023-09-21 09:45:27 +08:00

channel_manager_factory.go

enhance: Remove not in use policies (#29448 )

2023-12-28 10:38:46 +08:00

channel_manager_test.go

enhance: Change ChannelManager to interface (#29300 )

2023-12-25 19:24:46 +08:00

channel_manager.go

enhance:change some logs (#29579 )

2024-01-05 16:12:48 +08:00

channel_store_test.go

Change channel to Interface (#27839 )

2023-11-13 11:16:18 +08:00

channel_store.go

Change channel to Interface (#27839 )

2023-11-13 11:16:18 +08:00

channel.go

Change channel to Interface (#27839 )

2023-11-13 11:16:18 +08:00

cluster_test.go

enhance: Change ChannelManager to interface (#29300 )

2023-12-25 19:24:46 +08:00

cluster.go

feat: Add import scheduler and manager (#29367 )

2024-03-01 18:31:02 +08:00

compaction_l0_view_test.go

enhance: Add force trigger (#30641 )

2024-02-19 18:40:50 +08:00

compaction_l0_view.go

fix: Trigger l0 compaction when l0 views don't change (#30729 )

2024-03-05 16:37:00 +08:00

compaction_scheduler_test.go

enhance: Add CompactionTaskNum metrics (#29518 )

2023-12-28 15:46:55 +08:00

compaction_scheduler.go

fix: Trigger l0 compaction when l0 views don't change (#30729 )

2024-03-05 16:37:00 +08:00

compaction_test.go

fix: Trigger l0 compaction when l0 views don't change (#30729 )

2024-03-05 16:37:00 +08:00

compaction_trigger_test.go

enhance: Check channel cp lag before generate compaction task (#30997 )

2024-03-05 13:39:01 +08:00

compaction_trigger_v2_test.go

fix: Trigger l0 compaction when l0 views don't change (#30729 )

2024-03-05 16:37:00 +08:00

compaction_trigger_v2.go

fix: Trigger l0 compaction when l0 views don't change (#30729 )

2024-03-05 16:37:00 +08:00

compaction_trigger.go

enhance: Check channel cp lag before generate compaction task (#30997 )

2024-03-05 13:39:01 +08:00

compaction_view_manager_test.go

fix: Trigger l0 compaction when l0 views don't change (#30729 )

2024-03-05 16:37:00 +08:00

compaction_view_manager.go

fix: Trigger l0 compaction when l0 views don't change (#30729 )

2024-03-05 16:37:00 +08:00

compaction_view.go

fix: Trigger l0 compaction when l0 views don't change (#30729 )

2024-03-05 16:37:00 +08:00

compaction.go

fix: Trigger l0 compaction when l0 views don't change (#30729 )

2024-03-05 16:37:00 +08:00

const.go

enhance: pass partition key scalar info if enabled when build vector index (#29931 )

2024-01-24 00:04:55 +08:00

errors_test.go

Format the code (#27275 )

2023-09-21 09:45:27 +08:00

errors.go

Refine DataCoord status (#27262 )

2023-09-26 17:15:27 +08:00

garbage_collector_test.go

fix: starve lock caused by slow GetCompactionTo method when too much segments (#30963 )

2024-03-05 10:04:59 +08:00

garbage_collector.go

enhance: index meta use independent rather than global meta lock (#30869 )

2024-03-04 16:56:59 +08:00

handler.go

enhance: Explicitly pass LevelZero segment ids in vchan info (#29612 )

2024-01-04 16:46:45 +08:00

import_checker_test.go

feat: Add import scheduler and manager (#29367 )

2024-03-01 18:31:02 +08:00

import_checker.go

feat: Add import scheduler and manager (#29367 )

2024-03-01 18:31:02 +08:00

import_job.go

feat: Add import scheduler and manager (#29367 )

2024-03-01 18:31:02 +08:00

import_meta_test.go

feat: Add import scheduler and manager (#29367 )

2024-03-01 18:31:02 +08:00

import_meta.go

feat: Add import scheduler and manager (#29367 )

2024-03-01 18:31:02 +08:00

import_scheduler_test.go

feat: Add import scheduler and manager (#29367 )

2024-03-01 18:31:02 +08:00

import_scheduler.go

feat: Add import scheduler and manager (#29367 )

2024-03-01 18:31:02 +08:00

import_task.go

feat: Add import scheduler and manager (#29367 )

2024-03-01 18:31:02 +08:00

import_util_test.go

feat: Add import scheduler and manager (#29367 )

2024-03-01 18:31:02 +08:00

import_util.go

feat: Add import scheduler and manager (#29367 )

2024-03-01 18:31:02 +08:00

index_builder_test.go

enhance: index meta use independent rather than global meta lock (#30869 )

2024-03-04 16:56:59 +08:00

index_builder.go

enhance: index meta use independent rather than global meta lock (#30869 )

2024-03-04 16:56:59 +08:00

index_engine_version_manager_test.go

Optimize the codec code of session (#27360 )

2023-10-01 10:33:30 +08:00

index_engine_version_manager.go

feat: trigger compaction to handle index version (#28442 )

2023-11-21 09:26:22 +08:00

index_meta_test.go

enhance: index meta use independent rather than global meta lock (#30869 )

2024-03-04 16:56:59 +08:00

index_meta.go

enhance: index meta use independent rather than global meta lock (#30869 )

2024-03-04 16:56:59 +08:00

index_service_test.go

fix: starve lock caused by slow GetCompactionTo method when too much segments (#30963 )

2024-03-05 10:04:59 +08:00

index_service.go

enhance: index meta use independent rather than global meta lock (#30869 )

2024-03-04 16:56:59 +08:00

indexnode_manager_test.go

Refine state check (#27541 )

2023-10-11 21:01:35 +08:00

indexnode_manager.go

Decoupling client and server API in types interface (#27186 )

2023-09-26 09:57:25 +08:00

meta_test.go

fix: starve lock caused by slow GetCompactionTo method when too much segments (#30963 )

2024-03-05 10:04:59 +08:00

meta_util.go

Make VChannelInfo read compatible with 2.0.2 (#17995 )

2022-07-07 21:58:22 +08:00

meta.go

fix: add GetSegments optimization to avoid meta mutex competition (#31025 )

2024-03-05 14:47:00 +08:00

metrics_info_test.go

Refine state check (#27541 )

2023-10-11 21:01:35 +08:00

metrics_info.go

disable auto balance when old node exists (#28191 )

2023-11-07 14:02:20 +08:00

mock_allocator_test.go

enhance: index meta use independent rather than global meta lock (#30869 )

2024-03-04 16:56:59 +08:00

mock_channel_store.go

Change channel to Interface (#27839 )

2023-11-13 11:16:18 +08:00

mock_channelmanager.go

enhance: Add CompactionTaskNum metrics (#29518 )

2023-12-28 15:46:55 +08:00

mock_cluster.go

enhance: index meta use independent rather than global meta lock (#30869 )

2024-03-04 16:56:59 +08:00

mock_compaction_meta.go

enhance: Refine compaction interfaces to support major compaction (#30632 )

2024-02-19 20:52:50 +08:00

mock_compaction_plan_context.go

fix: Separate schedule and check results loop (#28692 )

2023-11-29 10:50:29 +08:00

mock_handler.go

Change channel to Interface (#27839 )

2023-11-13 11:16:18 +08:00

mock_index_engine_version_manager.go

feat: trigger compaction to handle index version (#28442 )

2023-11-21 09:26:22 +08:00

mock_scheduler.go

enhance: Add CompactionTaskNum metrics (#29518 )

2023-12-28 15:46:55 +08:00

mock_segment_manager.go

enhance: index meta use independent rather than global meta lock (#30869 )

2024-03-04 16:56:59 +08:00

mock_session_manager.go

enhance: index meta use independent rather than global meta lock (#30869 )

2024-03-04 16:56:59 +08:00

mock_test.go

feat: Add import scheduler and manager (#29367 )

2024-03-01 18:31:02 +08:00

mock_trigger_manager.go

feat: Add Compaction views and triggers (#27906 )

2023-11-23 17:30:25 +08:00

OWNERS

[skip ci]Update OWNERS files (#11898 )

2021-11-16 15:41:11 +08:00

policy_test.go

enhance: Remove not in use policies (#29448 )

2023-12-28 10:38:46 +08:00

policy.go

enhance: Remove not in use policies (#29448 )

2023-12-28 10:38:46 +08:00

README.md

[skip ci]Change etcd to lowercase (#9983 )

2021-10-15 18:58:37 +08:00

segment_allocation_policy_test.go

Adjust datacoord for L0 Delta (#28021 )

2023-11-06 15:26:16 +08:00

segment_allocation_policy.go

fix: Decoupling importing segment from flush process (#30402 )

2024-02-03 13:01:12 +08:00

segment_info_test.go

fix: starve lock caused by slow GetCompactionTo method when too much segments (#30963 )

2024-03-05 10:04:59 +08:00

segment_info.go

fix: starve lock caused by slow GetCompactionTo method when too much segments (#30963 )

2024-03-05 10:04:59 +08:00

segment_manager_test.go

feat: Add import scheduler and manager (#29367 )

2024-03-01 18:31:02 +08:00

segment_manager.go

fix: add GetSegments optimization to avoid meta mutex competition (#31025 )

2024-03-05 14:47:00 +08:00

segment_operator_test.go

fix: Prevent clone when selecting segments from meta (#30928 )

2024-03-01 13:59:02 +08:00

segment_operator.go

fix: Prevent clone when selecting segments from meta (#30928 )

2024-03-01 13:59:02 +08:00

server_test.go

enhance: index meta use independent rather than global meta lock (#30869 )

2024-03-04 16:56:59 +08:00

server.go

fix: Trigger l0 compaction when l0 views don't change (#30729 )

2024-03-05 16:37:00 +08:00

services_test.go

enhance: index meta use independent rather than global meta lock (#30869 )

2024-03-04 16:56:59 +08:00

services.go

fix: starve lock caused by slow GetCompactionTo method when too much segments (#30963 )

2024-03-05 10:04:59 +08:00

session_manager_test.go

feat: Add import scheduler and manager (#29367 )

2024-03-01 18:31:02 +08:00

session_manager.go

feat: Add import scheduler and manager (#29367 )

2024-03-01 18:31:02 +08:00

session.go

Decoupling client and server API in types interface (#27186 )

2023-09-26 09:57:25 +08:00

util_test.go

feat: Add import scheduler and manager (#29367 )

2024-03-01 18:31:02 +08:00

util.go

enhance: index meta use independent rather than global meta lock (#30869 )

2024-03-04 16:56:59 +08:00

README.md

Data Coordinator

Data cooridnator(datacoord for short) is the component to organize DataNodes and segments allocations.

Dependency

KV store: a kv store has all the meta info datacoord needs to operate. (etcd)
Message stream: a message stream to communicate statistics information with data nodes. (Pulsar)
Root Coordinator: timestamp, id and meta source.
Data Node(s): could be an instance or a cluster, actual worker group handles data modification operations.