mirror of https://gitee.com/milvus-io/milvus.git synced 2026-01-07 19:31:51 +08:00

History

fix: Rootcoord stuck at graceful stop progress (#36880 )

issue: #34553
when rootcoord trigger graceful stop progress, it will block until all
rpc finished. for create collection request, rootcoord need to block
until datacoord finish to watch all channels, but datacoord need to call
`rootcoord.Alloc` during watch channel, and rootcoord doesn't respond to
new request anymore. which cause create collection stucks, and graceful
stop progress stucks.

This PR remove the func call `rootcoord.Alloc` to solve the logic dead
lock during graceful stop progress.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>

2024-10-17 12:15:25 +08:00

allocator

fix: fix make generate-mockery panic with go1.22 (#36830 )

2024-10-17 12:11:31 +08:00

broker

fix: fix make generate-mockery panic with go1.22 (#36830 )

2024-10-17 12:11:31 +08:00

session

fix: datacoord stuck at stopping progress (#36852 )

2024-10-17 12:13:37 +08:00

analyze_meta_test.go

fix: Record the nodeID before assigning tasks (#36371 )

2024-09-28 17:21:15 +08:00

analyze_meta.go

fix: Record the nodeID before assigning tasks (#36371 )

2024-09-28 17:21:15 +08:00

build_index_policy.go

Format the code (#27275 )

2023-09-21 09:45:27 +08:00

channel_manager_factory.go

enhance: Tidy dc channel manager (#34515 )

2024-07-09 18:26:12 +08:00

channel_manager_test.go

fix: Rootcoord stuck at graceful stop progress (#36880 )

2024-10-17 12:15:25 +08:00

channel_manager.go

fix: Rootcoord stuck at graceful stop progress (#36880 )

2024-10-17 12:15:25 +08:00

channel_store_test.go

fix: Fill nil schema so that Milvus can watch channel for those upgraded from 2.2 to 2.4 #35695 (#35694 )

2024-08-27 10:36:59 +08:00

channel_store.go

enhance: Update protobuf-go to protobuf-go v2 (#34394 )

2024-07-29 11:31:51 +08:00

channel.go

fix: Fill nil schema so that Milvus can watch channel for those upgraded from 2.2 to 2.4 #35695 (#35694 )

2024-08-27 10:36:59 +08:00

cluster_test.go

enhance: Move datanode/indexnode manager to session pkg (#35634 )

2024-08-22 16:02:56 +08:00

cluster.go

enhance: Move datanode/indexnode manager to session pkg (#35634 )

2024-08-22 16:02:56 +08:00

compaction_l0_view_test.go

enhance: Add deltaRowCount in l0 compaction (#33997 )

2024-06-20 17:46:01 +08:00

compaction_l0_view.go

enhance: Add deltaRowCount in l0 compaction (#33997 )

2024-06-20 17:46:01 +08:00

compaction_policy_clustering_test.go

enhance: Separate allocator pkg in datacoord (#35622 )

2024-08-22 10:06:56 +08:00

compaction_policy_clustering.go

enhance: Separate allocator pkg in datacoord (#35622 )

2024-08-22 10:06:56 +08:00

compaction_policy_l0_test.go

enhance: Separate allocator pkg in datacoord (#35622 )

2024-08-22 10:06:56 +08:00

compaction_policy_l0.go

fix: Remove enableLevelZeroSegment config (#36535 )

2024-10-17 11:59:24 +08:00

compaction_policy_single_test.go

enhance: Separate allocator pkg in datacoord (#35622 )

2024-08-22 10:06:56 +08:00

compaction_policy_single.go

enhance: enable manual compaction for collections without indexes (#36577 )

2024-10-08 19:57:18 +08:00

compaction_queue_test.go

enhance: Datacoord to support prioritization of compaction tasks (#36547 )

2024-10-09 19:11:20 +08:00

compaction_queue.go

enhance: Datacoord to support prioritization of compaction tasks (#36547 )

2024-10-09 19:11:20 +08:00

compaction_task_clustering_test.go

enhance: the estimate method when loading the collection (#36307 )

2024-10-09 17:35:19 +08:00

compaction_task_clustering.go

enhance: Add PreallocatedSegmentIDs for the compaction task (#36734 )

2024-10-13 17:59:21 +08:00

compaction_task_l0_test.go

feat: support load and query with bm25 metric (#36071 )

2024-10-11 10:23:20 +08:00

compaction_task_l0.go

enhance:use proto.clone in compaction tasks to prevent omissions (#36624 )

2024-10-08 11:23:23 +08:00

compaction_task_meta_test.go

fix: fix collection is compacting logic (#34855 )

2024-07-22 11:35:41 +08:00

compaction_task_meta.go

enhance: Add PreallocatedSegmentIDs for the compaction task (#36734 )

2024-10-13 17:59:21 +08:00

compaction_task_mix_test.go

enhance: Enable to write multiple segments in mix compactor (#35705 )

2024-08-30 11:29:01 +08:00

compaction_task_mix.go

enhance: Add PreallocatedSegmentIDs for the compaction task (#36734 )

2024-10-13 17:59:21 +08:00

compaction_task_test.go

enhance: Move datanode/indexnode manager to session pkg (#35634 )

2024-08-22 16:02:56 +08:00

compaction_task.go

enhance: make slotUsage a field of CompactionTask (#36510 )

2024-09-26 20:19:14 +08:00

compaction_test.go

enhance: Datacoord to support prioritization of compaction tasks (#36547 )

2024-10-09 19:11:20 +08:00

compaction_trigger_test.go

enhance: Add PreallocatedSegmentIDs for the compaction task (#36734 )

2024-10-13 17:59:21 +08:00

compaction_trigger_v2_test.go

enhance: Separate allocator pkg in datacoord (#35622 )

2024-08-22 10:06:56 +08:00

compaction_trigger_v2.go

enhance: Add PreallocatedSegmentIDs for the compaction task (#36734 )

2024-10-13 17:59:21 +08:00

compaction_trigger.go

enhance: Add PreallocatedSegmentIDs for the compaction task (#36734 )

2024-10-13 17:59:21 +08:00

compaction_view.go

enhance: Add deltaRowCount in l0 compaction (#33997 )

2024-06-20 17:46:01 +08:00

compaction.go

enhance: Datacoord to support prioritization of compaction tasks (#36547 )

2024-10-09 19:11:20 +08:00

const.go

enhance: pass partition key scalar info if enabled when build vector index (#29931 )

2024-01-24 00:04:55 +08:00

errors_test.go

Format the code (#27275 )

2023-09-21 09:45:27 +08:00

errors.go

Refine DataCoord status (#27262 )

2023-09-26 17:15:27 +08:00

garbage_collector_test.go

feat: Support stats task to sort segment by PK (#35054 )

2024-09-02 14:19:03 +08:00

garbage_collector.go

enhance: enable manual compaction for collections without indexes (#36577 )

2024-10-08 19:57:18 +08:00

go_channel_singleton.go

fix: Fix improper use of offset in HybridSearch (#36244 )

2024-09-13 22:05:15 +08:00

handler_test.go

fix: Remove neighbors if compactTo is unindexed (#36503 )

2024-10-08 14:15:19 +08:00

handler.go

enhance: enable manual compaction for collections without indexes (#36577 )

2024-10-08 19:57:18 +08:00

import_checker_test.go

enhance: Optimize import scheduling and add time cost metric (#36601 )

2024-10-09 14:41:20 +08:00

import_checker.go

enhance: Optimize import scheduling and add time cost metric (#36601 )

2024-10-09 14:41:20 +08:00

import_job.go

enhance: Optimize import scheduling and add time cost metric (#36601 )

2024-10-09 14:41:20 +08:00

import_meta_test.go

feat: Add import scheduler and manager (#29367 )

2024-03-01 18:31:02 +08:00

import_meta.go

enhance: Optimize import scheduling and add time cost metric (#36601 )

2024-10-09 14:41:20 +08:00

import_scheduler_test.go

enhance: Optimize import scheduling and add time cost metric (#36601 )

2024-10-09 14:41:20 +08:00

import_scheduler.go

enhance: Optimize import scheduling and add time cost metric (#36601 )

2024-10-09 14:41:20 +08:00

import_task.go

feat: support bm25 logs mixcompaction (#36072 )

2024-10-14 16:57:22 +08:00

import_util_test.go

fix: Fix import segments leak in segment manager (#36602 )

2024-10-08 10:11:22 +08:00

import_util.go

enhance: Optimize import scheduling and add time cost metric (#36601 )

2024-10-09 14:41:20 +08:00

index_engine_version_manager_test.go

Optimize the codec code of session (#27360 )

2023-10-01 10:33:30 +08:00

index_engine_version_manager.go

enhance: Add go-deadlock as unittest only dependency (#33063 )

2024-05-15 16:33:34 +08:00

index_meta_test.go

fix: Record the nodeID before assigning tasks (#36371 )

2024-09-28 17:21:15 +08:00

index_meta.go

enhance: enable manual compaction for collections without indexes (#36577 )

2024-10-08 19:57:18 +08:00

index_service_test.go

fix: Return all compactTo segments after support split (#36361 )

2024-09-20 14:11:11 +08:00

index_service.go

enhance: Update stats task to optional (#35947 )

2024-09-12 20:37:08 +08:00

job_manager_test.go

feat: add enable_tokenizer params to VarChar field (#36480 )

2024-10-10 20:33:21 +08:00

job_manager.go

feat: Introduce stats task for import (#35868 )

2024-09-15 15:17:08 +08:00

meta_test.go

enhance: Add sorted for segment info (#36469 )

2024-09-30 10:01:16 +08:00

meta_util.go

Make VChannelInfo read compatible with 2.0.2 (#17995 )

2022-07-07 21:58:22 +08:00

meta.go

enhance: Enhance segment log (#36848 )

2024-10-15 20:43:30 +08:00

metrics_info_test.go

enhance: Move datanode/indexnode manager to session pkg (#35634 )

2024-08-22 16:02:56 +08:00

metrics_info.go

fix: metrics stored_index_files_size is never cleared (#36160 )

2024-09-13 20:09:15 +08:00

mock_channel_store.go

fix: fix make generate-mockery panic with go1.22 (#36830 )

2024-10-17 12:11:31 +08:00

mock_channelmanager.go

fix: fix make generate-mockery panic with go1.22 (#36830 )

2024-10-17 12:11:31 +08:00

mock_cluster.go

fix: fix make generate-mockery panic with go1.22 (#36830 )

2024-10-17 12:11:31 +08:00

mock_compaction_meta.go

fix: fix make generate-mockery panic with go1.22 (#36830 )

2024-10-17 12:11:31 +08:00

mock_compaction_plan_context.go

fix: fix make generate-mockery panic with go1.22 (#36830 )

2024-10-17 12:11:31 +08:00

mock_handler.go

fix: fix make generate-mockery panic with go1.22 (#36830 )

2024-10-17 12:11:31 +08:00

mock_index_engine_version_manager.go

fix: fix make generate-mockery panic with go1.22 (#36830 )

2024-10-17 12:11:31 +08:00

mock_job_manager.go

fix: fix make generate-mockery panic with go1.22 (#36830 )

2024-10-17 12:11:31 +08:00

mock_segment_manager.go

fix: fix make generate-mockery panic with go1.22 (#36830 )

2024-10-17 12:11:31 +08:00

mock_subcluster.go

fix: fix make generate-mockery panic with go1.22 (#36830 )

2024-10-17 12:11:31 +08:00

mock_test.go

enhance: Add PreallocatedSegmentIDs for the compaction task (#36734 )

2024-10-13 17:59:21 +08:00

mock_trigger_manager.go

fix: fix make generate-mockery panic with go1.22 (#36830 )

2024-10-17 12:11:31 +08:00

OWNERS

[skip ci]Update OWNERS files (#11898 )

2021-11-16 15:41:11 +08:00

partition_stats_meta_test.go

enhance: add commit time in partitionStats proto (#35125 )

2024-08-02 16:16:14 +08:00

partition_stats_meta.go

enhance: add commit time in partitionStats proto (#35125 )

2024-08-02 16:16:14 +08:00

policy_test.go

fix: Fill nil schema so that Milvus can watch channel for those upgraded from 2.2 to 2.4 #35695 (#35694 )

2024-08-27 10:36:59 +08:00

policy.go

enhance: Solve channel unbalance on datanode (#34984 )

2024-07-26 17:35:47 +08:00

README.md

[skip ci]Change etcd to lowercase (#9983 )

2021-10-15 18:58:37 +08:00

segment_allocation_policy_test.go

enhance: Change the fixed value to a ratio for clustering segment size (#35076 )

2024-08-01 22:04:14 +08:00

segment_allocation_policy.go

enhance: Change the fixed value to a ratio for clustering segment size (#35076 )

2024-08-01 22:04:14 +08:00

segment_info_test.go

fix: Return all compactTo segments after support split (#36361 )

2024-09-20 14:11:11 +08:00

segment_info.go

fix: Remove neighbors if compactTo is unindexed (#36503 )

2024-10-08 14:15:19 +08:00

segment_manager_test.go

fix: Fix import segments leak in segment manager (#36602 )

2024-10-08 10:11:22 +08:00

segment_manager.go

fix: Fix import segments leak in segment manager (#36602 )

2024-10-08 10:11:22 +08:00

segment_operator_test.go

fix: Prevent clone when selecting segments from meta (#30928 )

2024-03-01 13:59:02 +08:00

segment_operator.go

enhance: Update stats task to optional (#35947 )

2024-09-12 20:37:08 +08:00

server_test.go

fix: Rootcoord stuck at graceful stop progress (#36880 )

2024-10-17 12:15:25 +08:00

server.go

fix: Rootcoord stuck at graceful stop progress (#36880 )

2024-10-17 12:15:25 +08:00

services_test.go

feat: Support stats task to sort segment by PK (#35054 )

2024-09-02 14:19:03 +08:00

services.go

enhance: Optimize import scheduling and add time cost metric (#36601 )

2024-10-09 14:41:20 +08:00

stats_task_meta_test.go

fix: Record the nodeID before assigning tasks (#36371 )

2024-09-28 17:21:15 +08:00

stats_task_meta.go

fix: Record the nodeID before assigning tasks (#36371 )

2024-09-28 17:21:15 +08:00

sync_segments_scheduler_test.go

fix: datacoord stuck at stopping progress (#36852 )

2024-10-17 12:13:37 +08:00

sync_segments_scheduler.go

fix: datacoord stuck at stopping progress (#36852 )

2024-10-17 12:13:37 +08:00

task_analyze.go

fix: Record the nodeID before assigning tasks (#36371 )

2024-09-28 17:21:15 +08:00

task_index.go

enhance: remove warnings when there is no partition key set (#36674 )

2024-10-09 10:27:20 +08:00

task_scheduler_test.go

fix: Record the nodeID before assigning tasks (#36371 )

2024-09-28 17:21:15 +08:00

task_scheduler.go

fix: Record the nodeID before assigning tasks (#36371 )

2024-09-28 17:21:15 +08:00

task_stats_test.go

fix: Record the nodeID before assigning tasks (#36371 )

2024-09-28 17:21:15 +08:00

task_stats.go

feat: support bm25 logs mixcompaction (#36072 )

2024-10-14 16:57:22 +08:00

types.go

fix: Record the nodeID before assigning tasks (#36371 )

2024-09-28 17:21:15 +08:00

util_test.go

enhance: Separate allocator pkg in datacoord (#35622 )

2024-08-22 10:06:56 +08:00

util.go

enhance: enable manual compaction for collections without indexes (#36577 )

2024-10-08 19:57:18 +08:00

README.md

Data Coordinator

Data cooridnator(datacoord for short) is the component to organize DataNodes and segments allocations.

Dependency

KV store: a kv store has all the meta info datacoord needs to operate. (etcd)
Message stream: a message stream to communicate statistics information with data nodes. (Pulsar)
Root Coordinator: timestamp, id and meta source.
Data Node(s): could be an instance or a cluster, actual worker group handles data modification operations.