mirror of https://gitee.com/milvus-io/milvus.git synced 2026-01-07 19:31:51 +08:00

History

fix: Decoupling importing segment from flush process (#30402 )

This pr decoups importing segment from flush process by:
1. Exclude the importing segment from the flush policy, this approch
avoids notifying the datanode to flush the importing segment, which may
not exist.
2. When RootCoord call Flush, DataCoord directly set the importing
segment state to `Flushed`.

issue: https://github.com/milvus-io/milvus/issues/30359

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

2024-02-03 13:01:12 +08:00

broker

enhance: Move datacoord broker into separate package (#28876 )

2023-12-01 10:22:34 +08:00

allocator_test.go

Decoupling client and server API in types interface (#27186 )

2023-09-26 09:57:25 +08:00

allocator.go

enhance:Remove msgbase unnecessary assignments (#28511 )

2023-11-24 15:02:39 +08:00

build_index_policy.go

Format the code (#27275 )

2023-09-21 09:45:27 +08:00

channel_checker_test.go

Format the code (#27275 )

2023-09-21 09:45:27 +08:00

channel_checker.go

Format the code (#27275 )

2023-09-21 09:45:27 +08:00

channel_manager_factory.go

enhance: Remove not in use policies (#29448 )

2023-12-28 10:38:46 +08:00

channel_manager_test.go

enhance: Change ChannelManager to interface (#29300 )

2023-12-25 19:24:46 +08:00

channel_manager.go

enhance:change some logs (#29579 )

2024-01-05 16:12:48 +08:00

channel_store_test.go

Change channel to Interface (#27839 )

2023-11-13 11:16:18 +08:00

channel_store.go

Change channel to Interface (#27839 )

2023-11-13 11:16:18 +08:00

channel.go

Change channel to Interface (#27839 )

2023-11-13 11:16:18 +08:00

cluster_test.go

enhance: Change ChannelManager to interface (#29300 )

2023-12-25 19:24:46 +08:00

cluster.go

enhance: Change ChannelManager to interface (#29300 )

2023-12-25 19:24:46 +08:00

compaction_l0_view_test.go

fix: Limit L0 Compaction segment size and count (#30374 )

2024-02-01 20:39:03 +08:00

compaction_l0_view.go

fix: Limit L0 Compaction segment size and count (#30374 )

2024-02-01 20:39:03 +08:00

compaction_scheduler_test.go

enhance: Add CompactionTaskNum metrics (#29518 )

2023-12-28 15:46:55 +08:00

compaction_scheduler.go

enhance: Add CompactionTaskNum metrics (#29518 )

2023-12-28 15:46:55 +08:00

compaction_test.go

fix: compact operation on datacoord meta should preform as a transcation (#29775 )

2024-01-26 16:59:00 +08:00

compaction_trigger_test.go

enhance: Set segment.maxSize param to 1024M (#30139 )

2024-01-29 10:17:02 +08:00

compaction_trigger_v2_test.go

fix: Fix L0 compaction in datacoord (#28814 )

2023-12-05 18:44:37 +08:00

compaction_trigger_v2.go

fix: Fill in info in CompactionSegmentBinlogs (#30279 )

2024-01-26 10:51:02 +08:00

compaction_trigger.go

enhance: Set segment.maxSize param to 1024M (#30139 )

2024-01-29 10:17:02 +08:00

compaction_view_manager_test.go

fix: Limit L0 Compaction segment size and count (#30374 )

2024-02-01 20:39:03 +08:00

compaction_view_manager.go

fix: Fix L0 compaction in datacoord (#28814 )

2023-12-05 18:44:37 +08:00

compaction_view.go

fix: Fix some left comments (#29027 )

2023-12-13 18:48:39 +08:00

compaction.go

fix: call injectDone after compaction failed (#30277 )

2024-02-01 14:25:04 +08:00

const.go

enhance: pass partition key scalar info if enabled when build vector index (#29931 )

2024-01-24 00:04:55 +08:00

errors_test.go

Format the code (#27275 )

2023-09-21 09:45:27 +08:00

errors.go

Refine DataCoord status (#27262 )

2023-09-26 17:15:27 +08:00

garbage_collector_test.go

enhance:don't store logPath in meta to reduce memory (#28873 )

2024-01-18 22:06:31 +08:00

garbage_collector.go

enhance:don't store logPath in meta to reduce memory (#28873 )

2024-01-18 22:06:31 +08:00

handler.go

enhance: Explicitly pass LevelZero segment ids in vchan info (#29612 )

2024-01-04 16:46:45 +08:00

index_builder_test.go

enhance: pass partition key scalar info if enabled when build vector index (#29931 )

2024-01-24 00:04:55 +08:00

index_builder.go

enhance: pass partition key scalar info if enabled when build vector index (#29931 )

2024-01-24 00:04:55 +08:00

index_engine_version_manager_test.go

Optimize the codec code of session (#27360 )

2023-10-01 10:33:30 +08:00

index_engine_version_manager.go

feat: trigger compaction to handle index version (#28442 )

2023-11-21 09:26:22 +08:00

index_meta_test.go

feat: Support multiple vector indexes in a collection (#27700 )

2023-12-29 11:44:45 +08:00

index_meta.go

feat: Support multiple vector indexes in a collection (#27700 )

2023-12-29 11:44:45 +08:00

index_service_test.go

fix: report error if the altering index doesn't support mmap (#29832 )

2024-01-17 16:40:54 +08:00

index_service.go

enhance: pass partition key scalar info if enabled when build vector index (#29931 )

2024-01-24 00:04:55 +08:00

indexnode_manager_test.go

Refine state check (#27541 )

2023-10-11 21:01:35 +08:00

indexnode_manager.go

Decoupling client and server API in types interface (#27186 )

2023-09-26 09:57:25 +08:00

meta_test.go

fix: compact operation on datacoord meta should preform as a transcation (#29775 )

2024-01-26 16:59:00 +08:00

meta_util.go

…

meta.go

fix: compact operation on datacoord meta should preform as a transcation (#29775 )

2024-01-26 16:59:00 +08:00

metrics_info_test.go

Refine state check (#27541 )

2023-10-11 21:01:35 +08:00

metrics_info.go

disable auto balance when old node exists (#28191 )

2023-11-07 14:02:20 +08:00

mock_allocator_test.go

Organize all mockery generation commands in Makefile (#26826 )

2023-09-04 21:19:48 +08:00

mock_channel_store.go

Change channel to Interface (#27839 )

2023-11-13 11:16:18 +08:00

mock_channelmanager.go

enhance: Add CompactionTaskNum metrics (#29518 )

2023-12-28 15:46:55 +08:00

mock_cluster.go

enhance: pack datacoord Cluster and SessionManager with interface and mock them (#28869 )

2023-12-11 17:52:37 +08:00

mock_compaction_meta.go

fix: compact operation on datacoord meta should preform as a transcation (#29775 )

2024-01-26 16:59:00 +08:00

mock_compaction_plan_context.go

fix: Separate schedule and check results loop (#28692 )

2023-11-29 10:50:29 +08:00

mock_handler.go

Change channel to Interface (#27839 )

2023-11-13 11:16:18 +08:00

mock_index_engine_version_manager.go

feat: trigger compaction to handle index version (#28442 )

2023-11-21 09:26:22 +08:00

mock_scheduler.go

enhance: Add CompactionTaskNum metrics (#29518 )

2023-12-28 15:46:55 +08:00

mock_session_manager.go

enhance: Add open telemetry tracing for compaction (#30168 )

2024-01-23 10:37:00 +08:00

mock_test.go

feat: Implement DescribeAlias and ListAliases interfaces (#29641 )

2024-01-11 19:12:51 +08:00

mock_trigger_manager.go

feat: Add Compaction views and triggers (#27906 )

2023-11-23 17:30:25 +08:00

OWNERS

…

policy_test.go

enhance: Remove not in use policies (#29448 )

2023-12-28 10:38:46 +08:00

policy.go

enhance: Remove not in use policies (#29448 )

2023-12-28 10:38:46 +08:00

README.md

…

segment_allocation_policy_test.go

Adjust datacoord for L0 Delta (#28021 )

2023-11-06 15:26:16 +08:00

segment_allocation_policy.go

fix: Decoupling importing segment from flush process (#30402 )

2024-02-03 13:01:12 +08:00

segment_info.go

enhance:don't store logPath in meta to reduce memory (#28873 )

2024-01-18 22:06:31 +08:00

segment_manager_test.go

fix: Decoupling importing segment from flush process (#30402 )

2024-02-03 13:01:12 +08:00

segment_manager.go

fix: Decoupling importing segment from flush process (#30402 )

2024-02-03 13:01:12 +08:00

server_test.go

fix: Decoupling importing segment from flush process (#30402 )

2024-02-03 13:01:12 +08:00

server.go

fix: Only use bound indexnodes in bound mode (#30461 )

2024-02-03 11:01:47 +08:00

services_test.go

fix: Decoupling importing segment from flush process (#30402 )

2024-02-03 13:01:12 +08:00

services.go

fix: Decoupling importing segment from flush process (#30402 )

2024-02-03 13:01:12 +08:00

session_manager_test.go

enhance: Change ChannelManager to interface (#29300 )

2023-12-25 19:24:46 +08:00

session_manager.go

enhance: Add open telemetry tracing for compaction (#30168 )

2024-01-23 10:37:00 +08:00

session.go

Decoupling client and server API in types interface (#27186 )

2023-09-26 09:57:25 +08:00

util_test.go

enhance: add l0 metric and fix datacoord no need drop l0 segment when flush (#28373 )

2023-11-24 15:58:24 +08:00

util.go

fix: Use size bucket for compacted segment size metric (#30028 )

2024-01-26 10:53:02 +08:00

README.md

Data Coordinator

Data cooridnator(datacoord for short) is the component to organize DataNodes and segments allocations.

Dependency

KV store: a kv store has all the meta info datacoord needs to operate. (etcd)
Message stream: a message stream to communicate statistics information with data nodes. (Pulsar)
Root Coordinator: timestamp, id and meta source.
Data Node(s): could be an instance or a cluster, actual worker group handles data modification operations.