mirror of https://gitee.com/milvus-io/milvus.git synced 2026-01-04 18:02:08 +08:00

History

enhance: Rename Compaction to CompactionV2 (#33858 )

Due to the removal of injection and syncSegments from the compaction, we
need to ensure that no compaction is successfully executed during the
rolling upgrade. This PR renames Compaction to CompactionV2, with the
following effects:
- New datacoord + old datanode: Utilizes the CompactionV2 interface,
resulting in the datanode error "CompactionV2 not implemented," causing
compaction to fail;
- Old datacoord + new datanode: Utilizes the CompactionV1 interface,
resulting in the datanode error "CompactionV1 not implemented," causing
compaction to fail.

issue: https://github.com/milvus-io/milvus/issues/32809

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

2024-06-16 22:07:57 +08:00

broker

enhance: Improve datacoord segment filtering with collection (#32831 )

2024-05-08 21:37:29 +08:00

allocator_test.go

Decoupling client and server API in types interface (#27186 )

2023-09-26 09:57:25 +08:00

allocator.go

feat: Add import scheduler and manager (#29367 )

2024-03-01 18:31:02 +08:00

analyze_meta_test.go

enhance: Support analyze data (#33651 )

2024-06-06 17:37:51 +08:00

analyze_meta.go

enhance: Support analyze data (#33651 )

2024-06-06 17:37:51 +08:00

build_index_policy.go

Format the code (#27275 )

2023-09-21 09:45:27 +08:00

channel_checker_test.go

Format the code (#27275 )

2023-09-21 09:45:27 +08:00

channel_checker.go

Format the code (#27275 )

2023-09-21 09:45:27 +08:00

channel_manager_factory.go

feat: datacoord/node watch based on rpc (#32036 )

2024-05-07 15:49:30 +08:00

channel_manager_test.go

enhance: Make channel meta able to writer 200k plus segments (#33279 )

2024-05-28 12:33:42 +08:00

channel_manager_v2_test.go

fix: Return error when startup Delete/AddNode fail (#33193 )

2024-05-22 11:17:38 +08:00

channel_manager_v2.go

enhance: Make channel meta able to writer 200k plus segments (#33279 )

2024-05-28 12:33:42 +08:00

channel_manager.go

fix: reduce redundant map operations in datacoord (#33343 )

2024-05-24 12:47:40 +08:00

channel_store_test.go

enhance: Make channel meta able to writer 200k plus segments (#33279 )

2024-05-28 12:33:42 +08:00

channel_store_v2_test.go

enhance: Make channel meta able to writer 200k plus segments (#33279 )

2024-05-28 12:33:42 +08:00

channel_store_v2.go

enhance: Make channel meta able to writer 200k plus segments (#33279 )

2024-05-28 12:33:42 +08:00

channel_store.go

enhance: Make channel meta able to writer 200k plus segments (#33279 )

2024-05-28 12:33:42 +08:00

channel.go

feat: datacoord/node watch based on rpc (#32036 )

2024-05-07 15:49:30 +08:00

cluster_test.go

enhance: Query slot for compaction task (#32881 )

2024-05-17 18:19:38 +08:00

cluster.go

enhance: Query slot for compaction task (#32881 )

2024-05-17 18:19:38 +08:00

compaction_l0_view_test.go

enhance: Add force trigger (#30641 )

2024-02-19 18:40:50 +08:00

compaction_l0_view.go

fix: Trigger l0 compaction when l0 views don't change (#30729 )

2024-03-05 16:37:00 +08:00

compaction_policy_clustering.go

feat: Major compaction (#33620 )

2024-06-10 21:34:08 +08:00

compaction_policy_l0_test.go

enhance: Reconstructing Compaction to possess persistence capability (#33265 )

2024-06-05 10:17:50 +08:00

compaction_policy_l0.go

enhance: Reconstructing Compaction to possess persistence capability (#33265 )

2024-06-05 10:17:50 +08:00

compaction_task_clustering.go

feat: Major compaction (#33620 )

2024-06-10 21:34:08 +08:00

compaction_task_l0_test.go

enhance: Reconstructing Compaction to possess persistence capability (#33265 )

2024-06-05 10:17:50 +08:00

compaction_task_l0.go

feat: Major compaction (#33620 )

2024-06-10 21:34:08 +08:00

compaction_task_meta.go

enhance: Reconstructing Compaction to possess persistence capability (#33265 )

2024-06-05 10:17:50 +08:00

compaction_task_mix_test.go

enhance: Reconstructing Compaction to possess persistence capability (#33265 )

2024-06-05 10:17:50 +08:00

compaction_task_mix.go

feat: Major compaction (#33620 )

2024-06-10 21:34:08 +08:00

compaction_task_test.go

enhance: Reconstructing Compaction to possess persistence capability (#33265 )

2024-06-05 10:17:50 +08:00

compaction_task.go

feat: Major compaction (#33620 )

2024-06-10 21:34:08 +08:00

compaction_test.go

feat: Major compaction (#33620 )

2024-06-10 21:34:08 +08:00

compaction_trigger_test.go

feat: Major compaction (#33620 )

2024-06-10 21:34:08 +08:00

compaction_trigger_v2_test.go

feat: Major compaction (#33620 )

2024-06-10 21:34:08 +08:00

compaction_trigger_v2.go

feat: Major compaction (#33620 )

2024-06-10 21:34:08 +08:00

compaction_trigger.go

feat: Major compaction (#33620 )

2024-06-10 21:34:08 +08:00

compaction_view.go

feat: Major compaction (#33620 )

2024-06-10 21:34:08 +08:00

compaction.go

feat: Major compaction (#33620 )

2024-06-10 21:34:08 +08:00

const.go

enhance: pass partition key scalar info if enabled when build vector index (#29931 )

2024-01-24 00:04:55 +08:00

errors_test.go

Format the code (#27275 )

2023-09-21 09:45:27 +08:00

errors.go

Refine DataCoord status (#27262 )

2023-09-26 17:15:27 +08:00

garbage_collector_test.go

enhance: decrease cpu overhead during filter segments on datacoord (#33130 )

2024-05-28 19:17:43 +08:00

garbage_collector.go

feat: Major compaction (#33620 )

2024-06-10 21:34:08 +08:00

handler.go

fix: sync partitiion stats blocking balance task(#33741 ) (#33742 )

2024-06-11 14:21:56 +08:00

import_checker_test.go

feat: Major compaction (#33620 )

2024-06-10 21:34:08 +08:00

import_checker.go

fix: Fix multiple vector fields import (#33723 )

2024-06-10 21:57:54 +08:00

import_job.go

fix: Fix multiple vector fields import (#33723 )

2024-06-10 21:57:54 +08:00

import_meta_test.go

feat: Add import scheduler and manager (#29367 )

2024-03-01 18:31:02 +08:00

import_meta.go

enhance: Add go-deadlock as unittest only dependency (#33063 )

2024-05-15 16:33:34 +08:00

import_scheduler_test.go

feat: Major compaction (#33620 )

2024-06-10 21:34:08 +08:00

import_scheduler.go

enhance: Support L0 import (#33514 )

2024-06-07 14:17:20 +08:00

import_task.go

enhance: Enhance and modify the return content of ImportV2 (#31192 )

2024-03-13 19:51:03 +08:00

import_util_test.go

fix: Check if the import job exists (#33672 )

2024-06-10 21:51:55 +08:00

import_util.go

fix: Fix multiple vector fields import (#33723 )

2024-06-10 21:57:54 +08:00

index_engine_version_manager_test.go

Optimize the codec code of session (#27360 )

2023-10-01 10:33:30 +08:00

index_engine_version_manager.go

enhance: Add go-deadlock as unittest only dependency (#33063 )

2024-05-15 16:33:34 +08:00

index_meta_test.go

enhance: Support analyze data (#33651 )

2024-06-06 17:37:51 +08:00

index_meta.go

enhance: Support analyze data (#33651 )

2024-06-06 17:37:51 +08:00

index_service_test.go

enhance: decrease cpu overhead during filter segments on datacoord (#33130 )

2024-05-28 19:17:43 +08:00

index_service.go

enhance: Support analyze data (#33651 )

2024-06-06 17:37:51 +08:00

indexnode_manager_test.go

enhance: Support analyze data (#33651 )

2024-06-06 17:37:51 +08:00

indexnode_manager.go

enhance: Support analyze data (#33651 )

2024-06-06 17:37:51 +08:00

meta_test.go

feat: Major compaction (#33620 )

2024-06-10 21:34:08 +08:00

meta_util.go

Make VChannelInfo read compatible with 2.0.2 (#17995 )

2022-07-07 21:58:22 +08:00

meta.go

fix: lost DroppedAt field when DropCollection (#33766 )

2024-06-12 20:21:56 +08:00

metrics_info_test.go

enhance: Use channel manager interface in server_test (#31621 )

2024-04-12 14:59:20 +08:00

metrics_info.go

enhance: Add metrics for segment index files size (#32979 )

2024-05-13 17:59:32 +08:00

mock_allocator_test.go

enhance: index meta use independent rather than global meta lock (#30869 )

2024-03-04 16:56:59 +08:00

mock_channel_store.go

enhance: Remove compaction plans on the datanode (#33548 )

2024-06-05 20:27:51 +08:00

mock_channelmanager.go

enhance: Remove compaction plans on the datanode (#33548 )

2024-06-05 20:27:51 +08:00

mock_cluster.go

enhance: Remove compaction plans on the datanode (#33548 )

2024-06-05 20:27:51 +08:00

mock_compaction_meta.go

feat: Major compaction (#33620 )

2024-06-10 21:34:08 +08:00

mock_compaction_plan_context.go

enhance: Reconstructing Compaction to possess persistence capability (#33265 )

2024-06-05 10:17:50 +08:00

mock_handler.go

fix: fix collectionInfo leak in datacoord (#32175 )

2024-04-15 16:33:19 +08:00

mock_index_engine_version_manager.go

feat: trigger compaction to handle index version (#28442 )

2023-11-21 09:26:22 +08:00

mock_segment_manager.go

enhance: Execute bloom filter apply in parallel to speed up segment predict (#33792 )

2024-06-14 11:37:56 +08:00

mock_session_manager.go

enhance: Remove compaction plans on the datanode (#33548 )

2024-06-05 20:27:51 +08:00

mock_subcluster.go

feat: datacoord/node watch based on rpc (#32036 )

2024-05-07 15:49:30 +08:00

mock_test.go

enhance: Rename Compaction to CompactionV2 (#33858 )

2024-06-16 22:07:57 +08:00

mock_trigger_manager.go

feat: Major compaction (#33620 )

2024-06-10 21:34:08 +08:00

mock_worker_manager.go

enhance: Support analyze data (#33651 )

2024-06-06 17:37:51 +08:00

OWNERS

[skip ci]Update OWNERS files (#11898 )

2021-11-16 15:41:11 +08:00

partition_stats_meta.go

feat: Major compaction (#33620 )

2024-06-10 21:34:08 +08:00

policy_test.go

enhance: Make channel meta able to writer 200k plus segments (#33279 )

2024-05-28 12:33:42 +08:00

policy.go

fix: Enable to balance released standby channel (#32986 )

2024-05-14 11:51:32 +08:00

README.md

[skip ci]Change etcd to lowercase (#9983 )

2021-10-15 18:58:37 +08:00

segment_allocation_policy_test.go

Adjust datacoord for L0 Delta (#28021 )

2023-11-06 15:26:16 +08:00

segment_allocation_policy.go

fix: Decoupling importing segment from flush process (#30402 )

2024-02-03 13:01:12 +08:00

segment_info_test.go

enhance: use WalkWithPrefix api for oss, enable piplined file gc (#31740 )

2024-04-25 20:41:27 +08:00

segment_info.go

feat: Major compaction (#33620 )

2024-06-10 21:34:08 +08:00

segment_manager_test.go

enhance: Support L0 import (#33514 )

2024-06-07 14:17:20 +08:00

segment_manager.go

enhance: Support L0 import (#33514 )

2024-06-07 14:17:20 +08:00

segment_operator_test.go

fix: Prevent clone when selecting segments from meta (#30928 )

2024-03-01 13:59:02 +08:00

segment_operator.go

enhance: Periodically synchronize segments to datanode watcher (#33420 )

2024-05-30 13:37:44 +08:00

server_test.go

feat: Major compaction (#33620 )

2024-06-10 21:34:08 +08:00

server.go

feat: Major compaction (#33620 )

2024-06-10 21:34:08 +08:00

services_test.go

fix: Check if the import job exists (#33672 )

2024-06-10 21:51:55 +08:00

services.go

fix: Check if the import job exists (#33672 )

2024-06-10 21:51:55 +08:00

session_manager_test.go

feat: datacoord/node watch based on rpc (#32036 )

2024-05-07 15:49:30 +08:00

session_manager.go

enhance: Rename Compaction to CompactionV2 (#33858 )

2024-06-16 22:07:57 +08:00

session.go

enhance: Add go-deadlock as unittest only dependency (#33063 )

2024-05-15 16:33:34 +08:00

sync_segments_scheduler_test.go

enhance: Periodically synchronize segments to datanode watcher (#33420 )

2024-05-30 13:37:44 +08:00

sync_segments_scheduler.go

fix: Don't sync L0 segments to channel watcher (#33664 )

2024-06-06 15:59:50 +08:00

task_analyze.go

enhance: Support analyze data (#33651 )

2024-06-06 17:37:51 +08:00

task_index.go

enhance: Support analyze data (#33651 )

2024-06-06 17:37:51 +08:00

task_scheduler_test.go

enhance: Support analyze data (#33651 )

2024-06-06 17:37:51 +08:00

task_scheduler.go

enhance: Support analyze data (#33651 )

2024-06-06 17:37:51 +08:00

types.go

enhance: Support analyze data (#33651 )

2024-06-06 17:37:51 +08:00

util_test.go

enhance: Add memory size for binlog (#33025 )

2024-05-15 12:59:34 +08:00

util.go

enhance: Support analyze data (#33651 )

2024-06-06 17:37:51 +08:00

README.md

Data Coordinator

Data cooridnator(datacoord for short) is the component to organize DataNodes and segments allocations.

Dependency

KV store: a kv store has all the meta info datacoord needs to operate. (etcd)
Message stream: a message stream to communicate statistics information with data nodes. (Pulsar)
Root Coordinator: timestamp, id and meta source.
Data Node(s): could be an instance or a cluster, actual worker group handles data modification operations.