mirror of https://gitee.com/milvus-io/milvus.git synced 2026-01-07 19:31:51 +08:00

History

enhance: Seal by total growing segments size (#34692 )

Seals the largest growing segment if the total size of growing segments
of each shard exceeds the size threshold(default 4GB). Introducing this
policy can help keep the size of growing segments within a suitable
level, alleviating the pressure on the delegator.

issue: https://github.com/milvus-io/milvus/issues/34554

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

2024-07-17 21:45:41 +08:00

broker

enhance: Improve datacoord segment filtering with collection (#32831 )

2024-05-08 21:37:29 +08:00

allocator_test.go

Decoupling client and server API in types interface (#27186 )

2023-09-26 09:57:25 +08:00

allocator.go

feat: Add import scheduler and manager (#29367 )

2024-03-01 18:31:02 +08:00

analyze_meta_test.go

enhance: Support analyze data (#33651 )

2024-06-06 17:37:51 +08:00

analyze_meta.go

enhance: Support analyze data (#33651 )

2024-06-06 17:37:51 +08:00

build_index_policy.go

Format the code (#27275 )

2023-09-21 09:45:27 +08:00

channel_manager_factory.go

enhance: Tidy dc channel manager (#34515 )

2024-07-09 18:26:12 +08:00

channel_manager_test.go

enhance: Tidy dc channel manager (#34515 )

2024-07-09 18:26:12 +08:00

channel_manager.go

enhance: Tidy dc channel manager (#34515 )

2024-07-09 18:26:12 +08:00

channel_store_test.go

enhance: remove deprecated code within channel manager (#34340 )

2024-07-03 19:46:09 +08:00

channel_store.go

enhance: Tidy dc channel manager (#34515 )

2024-07-09 18:26:12 +08:00

channel.go

enhance: Print reason when seal segments (#34575 )

2024-07-11 11:40:51 +08:00

cluster_test.go

enhance: move rocksmq from internal to pkg module (#33881 )

2024-06-25 21:18:15 +08:00

cluster.go

enhance: Query slot for compaction task (#32881 )

2024-05-17 18:19:38 +08:00

compaction_l0_view_test.go

enhance: Add deltaRowCount in l0 compaction (#33997 )

2024-06-20 17:46:01 +08:00

compaction_l0_view.go

enhance: Add deltaRowCount in l0 compaction (#33997 )

2024-06-20 17:46:01 +08:00

compaction_policy_clustering.go

enhance: Print reason when seal segments (#34575 )

2024-07-11 11:40:51 +08:00

compaction_policy_l0_test.go

enhance: Add back compactionTaskNum metrics (#34583 )

2024-07-11 17:45:37 +08:00

compaction_policy_l0.go

enhance: Add back compactionTaskNum metrics (#34583 )

2024-07-11 17:45:37 +08:00

compaction_task_clustering_test.go

enhance: Pre-allocate ids for compaction (#34187 )

2024-07-17 13:23:42 +08:00

compaction_task_clustering.go

enhance: Pre-allocate ids for compaction (#34187 )

2024-07-17 13:23:42 +08:00

compaction_task_l0_test.go

enhance: Pre-allocate ids for compaction (#34187 )

2024-07-17 13:23:42 +08:00

compaction_task_l0.go

enhance: Pre-allocate ids for compaction (#34187 )

2024-07-17 13:23:42 +08:00

compaction_task_meta.go

enhance: Reconstructing Compaction to possess persistence capability (#33265 )

2024-06-05 10:17:50 +08:00

compaction_task_mix_test.go

enhance: Pre-allocate ids for compaction (#34187 )

2024-07-17 13:23:42 +08:00

compaction_task_mix.go

enhance: Pre-allocate ids for compaction (#34187 )

2024-07-17 13:23:42 +08:00

compaction_task_test.go

enhance: Reconstructing Compaction to possess persistence capability (#33265 )

2024-06-05 10:17:50 +08:00

compaction_task.go

fix: Avoid datarace in clustering compaction (#34288 )

2024-07-03 19:08:09 +08:00

compaction_test.go

fix: remove isFull check in compaction.enqueue (#34338 )

2024-07-03 18:10:10 +08:00

compaction_trigger_test.go

enhance: Pre-allocate ids for compaction (#34187 )

2024-07-17 13:23:42 +08:00

compaction_trigger_v2_test.go

fix: fix clustering compaction can't enqueue when compaction queue is full (#34445 )

2024-07-08 10:02:13 +08:00

compaction_trigger_v2.go

enhance: Pre-allocate ids for compaction (#34187 )

2024-07-17 13:23:42 +08:00

compaction_trigger.go

enhance: Pre-allocate ids for compaction (#34187 )

2024-07-17 13:23:42 +08:00

compaction_view.go

enhance: Add deltaRowCount in l0 compaction (#33997 )

2024-06-20 17:46:01 +08:00

compaction.go

enhance: Pre-allocate ids for compaction (#34187 )

2024-07-17 13:23:42 +08:00

const.go

enhance: pass partition key scalar info if enabled when build vector index (#29931 )

2024-01-24 00:04:55 +08:00

errors_test.go

Format the code (#27275 )

2023-09-21 09:45:27 +08:00

errors.go

Refine DataCoord status (#27262 )

2023-09-26 17:15:27 +08:00

garbage_collector_test.go

fix: encountering orphan channel-cp meta after DataCoord GC (#34612 )

2024-07-11 23:01:35 +08:00

garbage_collector.go

fix: encountering orphan channel-cp meta after DataCoord GC (#34612 )

2024-07-11 23:01:35 +08:00

handler.go

fix:load major compaction partial result(#34051 ) (#34052 )

2024-06-28 14:04:05 +08:00

import_checker_test.go

feat: Major compaction (#33620 )

2024-06-10 21:34:08 +08:00

import_checker.go

fix: Fix multiple vector fields import (#33723 )

2024-06-10 21:57:54 +08:00

import_job.go

fix: Fix multiple vector fields import (#33723 )

2024-06-10 21:57:54 +08:00

import_meta_test.go

feat: Add import scheduler and manager (#29367 )

2024-03-01 18:31:02 +08:00

import_meta.go

enhance: Add go-deadlock as unittest only dependency (#33063 )

2024-05-15 16:33:34 +08:00

import_scheduler_test.go

fix: Prevent import from generating orphaned files (#34070 )

2024-06-25 16:58:02 +08:00

import_scheduler.go

fix: Prevent import from generating orphaned files (#34070 )

2024-06-25 16:58:02 +08:00

import_task.go

enhance: Enhance and modify the return content of ImportV2 (#31192 )

2024-03-13 19:51:03 +08:00

import_util_test.go

fix: Check if the import job exists (#33672 )

2024-06-10 21:51:55 +08:00

import_util.go

enhance: Add l0 segment entry num quota (#34733 )

2024-07-17 17:35:41 +08:00

index_engine_version_manager_test.go

Optimize the codec code of session (#27360 )

2023-10-01 10:33:30 +08:00

index_engine_version_manager.go

enhance: Add go-deadlock as unittest only dependency (#33063 )

2024-05-15 16:33:34 +08:00

index_meta_test.go

enhance: change autoindex default metric type (#34261 )

2024-07-08 19:52:24 +08:00

index_meta.go

enhance: change autoindex default metric type (#34261 )

2024-07-08 19:52:24 +08:00

index_service_test.go

enhance: decrease cpu overhead during filter segments on datacoord (#33130 )

2024-05-28 19:17:43 +08:00

index_service.go

enhance: Skip pick worker when task doesn't need to execute actually (#34348 )

2024-07-03 15:52:09 +08:00

indexnode_manager_test.go

enhance: Support analyze data (#33651 )

2024-06-06 17:37:51 +08:00

indexnode_manager.go

enhance: Support analyze data (#33651 )

2024-06-06 17:37:51 +08:00

meta_test.go

enhance: Add l0 segment entry num quota (#34733 )

2024-07-17 17:35:41 +08:00

meta_util.go

…

meta.go

enhance: Add l0 segment entry num quota (#34733 )

2024-07-17 17:35:41 +08:00

metrics_info_test.go

enhance: Use channel manager interface in server_test (#31621 )

2024-04-12 14:59:20 +08:00

metrics_info.go

enhance: Add l0 segment entry num quota (#34733 )

2024-07-17 17:35:41 +08:00

mock_allocator_test.go

enhance: index meta use independent rather than global meta lock (#30869 )

2024-03-04 16:56:59 +08:00

mock_channel_store.go

enhance: Tidy dc channel manager (#34515 )

2024-07-09 18:26:12 +08:00

mock_channelmanager.go

enhance: Remove compaction plans on the datanode (#33548 )

2024-06-05 20:27:51 +08:00

mock_cluster.go

enhance: Remove compaction plans on the datanode (#33548 )

2024-06-05 20:27:51 +08:00

mock_compaction_meta.go

enhance: Refine compaction (#33982 )

2024-06-25 10:08:03 +08:00

mock_compaction_plan_context.go

enhance: Reconstructing Compaction to possess persistence capability (#33265 )

2024-06-05 10:17:50 +08:00

mock_handler.go

fix: fix collectionInfo leak in datacoord (#32175 )

2024-04-15 16:33:19 +08:00

mock_index_engine_version_manager.go

feat: trigger compaction to handle index version (#28442 )

2023-11-21 09:26:22 +08:00

mock_segment_manager.go

enhance: Execute bloom filter apply in parallel to speed up segment predict (#33792 )

2024-06-14 11:37:56 +08:00

mock_session_manager.go

enhance: Remove compaction plans on the datanode (#33548 )

2024-06-05 20:27:51 +08:00

mock_subcluster.go

feat: datacoord/node watch based on rpc (#32036 )

2024-05-07 15:49:30 +08:00

mock_test.go

enhance: Rename Compaction to CompactionV2 (#33858 )

2024-06-16 22:07:57 +08:00

mock_trigger_manager.go

feat: Major compaction (#33620 )

2024-06-10 21:34:08 +08:00

mock_worker_manager.go

enhance: Support analyze data (#33651 )

2024-06-06 17:37:51 +08:00

OWNERS

…

partition_stats_meta.go

feat: Major compaction (#33620 )

2024-06-10 21:34:08 +08:00

policy_test.go

enhance: Tidy dc channel manager (#34515 )

2024-07-09 18:26:12 +08:00

policy.go

enhance: Tidy dc channel manager (#34515 )

2024-07-09 18:26:12 +08:00

README.md

…

segment_allocation_policy_test.go

enhance: Seal by total growing segments size (#34692 )

2024-07-17 21:45:41 +08:00

segment_allocation_policy.go

enhance: Seal by total growing segments size (#34692 )

2024-07-17 21:45:41 +08:00

segment_info_test.go

enhance: use WalkWithPrefix api for oss, enable piplined file gc (#31740 )

2024-04-25 20:41:27 +08:00

segment_info.go

enhance: Seal by total growing segments size (#34692 )

2024-07-17 21:45:41 +08:00

segment_manager_test.go

fix: Remove flushed segment in segment manager generated through import (#34649 )

2024-07-12 23:15:22 +08:00

segment_manager.go

enhance: Seal by total growing segments size (#34692 )

2024-07-17 21:45:41 +08:00

segment_operator_test.go

fix: Prevent clone when selecting segments from meta (#30928 )

2024-03-01 13:59:02 +08:00

segment_operator.go

enhance: Periodically synchronize segments to datanode watcher (#33420 )

2024-05-30 13:37:44 +08:00

server_test.go

enhance: skip orphan channel cp meta when checking cp lag (#34555 )

2024-07-11 09:36:56 +08:00

server.go

fix: Milvus panic when compaction disabled and dropping a collection (#34103 )

2024-07-11 14:44:52 +08:00

services_test.go

enhance: Remove datanode reporting TT based on MQ implementation (#34421 )

2024-07-05 15:48:09 +08:00

services.go

fix: Milvus panic when compaction disabled and dropping a collection (#34103 )

2024-07-11 14:44:52 +08:00

session_manager_test.go

feat: datacoord/node watch based on rpc (#32036 )

2024-05-07 15:49:30 +08:00

session_manager.go

enhance: Don't set the timeout for SyncSegments (#34226 )

2024-07-02 14:28:07 +08:00

session.go

enhance: Add go-deadlock as unittest only dependency (#33063 )

2024-05-15 16:33:34 +08:00

sync_segments_scheduler_test.go

enhance: Periodically synchronize segments to datanode watcher (#33420 )

2024-05-30 13:37:44 +08:00

sync_segments_scheduler.go

fix: Skip l0 segments when syncing segments to datanodes (#34388 )

2024-07-04 18:02:14 +08:00

task_analyze.go

enhance: Skip pick worker when task doesn't need to execute actually (#34348 )

2024-07-03 15:52:09 +08:00

task_index.go

feat: support partition key isolation (#34336 )

2024-07-11 19:01:35 +08:00

task_scheduler_test.go

feat: support partition key isolation (#34336 )

2024-07-11 19:01:35 +08:00

task_scheduler.go

enhance: Skip pick worker when task doesn't need to execute actually (#34348 )

2024-07-03 15:52:09 +08:00

types.go

enhance: Skip pick worker when task doesn't need to execute actually (#34348 )

2024-07-03 15:52:09 +08:00

util_test.go

enhance: Add memory size for binlog (#33025 )

2024-05-15 12:59:34 +08:00

util.go

fix: encountering orphan channel-cp meta after DataCoord GC (#34612 )

2024-07-11 23:01:35 +08:00

README.md

Data Coordinator

Data cooridnator(datacoord for short) is the component to organize DataNodes and segments allocations.

Dependency

KV store: a kv store has all the meta info datacoord needs to operate. (etcd)
Message stream: a message stream to communicate statistics information with data nodes. (Pulsar)
Root Coordinator: timestamp, id and meta source.
Data Node(s): could be an instance or a cluster, actual worker group handles data modification operations.