admin/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2025-12-06 17:18:35 +08:00
Code Issues Packages Projects Releases Wiki Activity
milvus/internal/datacoord
History
aoiasd 354ab2f55e
enhance: sync file resource to querynode and datanode (#44480) 
relate:https://github.com/milvus-io/milvus/issues/43687
Support use file resource with sync mode.
Auto download or remove file resource to local when user add or remove
file resource.
Sync file resource to node when find new node session.

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2025-12-04 16:23:11 +08:00
..
allocator
enhance: Pooling for data tasks (#41256)
2025-05-20 21:06:24 +08:00
broker
enhance: Optimize FlushAll performance for multi-table scenarios (#43339)
2025-07-30 15:37:37 +08:00
session
enhance: remove redundant channel manager from datacoord (#44532)
2025-10-09 11:01:57 +08:00
task
fix: Skip running scalar index when segment was compacted (#44690)
2025-10-11 12:05:56 +08:00
.mockery.yaml
fix: Pre-check import message to prevent pipeline block indefinitely (#42415)
2025-06-11 13:40:38 +08:00
analyze_inspector.go
fix: Fix task getting stuck after recovery (#42114)
2025-05-28 12:46:28 +08:00
analyze_meta_test.go
enhance: Use v2 package name for pkg module (#39990)
2025-02-22 23:15:58 +08:00
analyze_meta.go
enhance: Pooling for data tasks (#41256)
2025-05-20 21:06:24 +08:00
build_index_policy.go
Format the code (#27275)
2023-09-21 09:45:27 +08:00
channel.go
enhance: remove redundant channel manager from datacoord (#44532)
2025-10-09 11:01:57 +08:00
compaction_inspector_test.go
enhance: remove redundant channel manager from datacoord (#44532)
2025-10-09 11:01:57 +08:00
compaction_inspector.go
enhance: remove redundant channel manager from datacoord (#44532)
2025-10-09 11:01:57 +08:00
compaction_l0_view_test.go
feat: support mannual compact l0(#44439) (#44440)
2025-09-23 12:44:07 +08:00
compaction_l0_view.go
feat: support mannual compact l0(#44439) (#44440)
2025-09-23 12:44:07 +08:00
compaction_policy_clustering_test.go
fix: Use diskSegmentMaxSize for coll with sparse and dense vectors (#43194)
2025-07-16 18:04:52 +08:00
compaction_policy_clustering.go
feat: Auto add namespace field data if namespace is enabled (#44933)
2025-10-24 18:40:05 +08:00
compaction_policy_l0_test.go
feat: support mannual compact l0(#44439) (#44440)
2025-09-23 12:44:07 +08:00
compaction_policy_l0.go
feat: support mannual compact l0(#44439) (#44440)
2025-09-23 12:44:07 +08:00
compaction_policy_single_test.go
enhance: Move sort stats task to sort compaction (#42562)
2025-07-08 20:22:47 +08:00
compaction_policy_single.go
feat: Auto add namespace field data if namespace is enabled (#44933)
2025-10-24 18:40:05 +08:00
compaction_queue_test.go
enhance: Use v2 package name for pkg module (#39990)
2025-02-22 23:15:58 +08:00
compaction_queue.go
enhance: Use v2 package name for pkg module (#39990)
2025-02-22 23:15:58 +08:00
compaction_task_clustering_test.go
fix: Set task init when worker doesn't have task (#45675)
2025-11-19 18:03:07 +08:00
compaction_task_clustering.go
enhance: integrate Storage V2 FFI interface for unified storage access (#45723)
2025-11-24 19:57:07 +08:00
compaction_task_l0_test.go
fix: Set task init when worker doesn't have task (#45675)
2025-11-19 18:03:07 +08:00
compaction_task_l0.go
enhance: integrate Storage V2 FFI interface for unified storage access (#45723)
2025-11-24 19:57:07 +08:00
compaction_task_meta_test.go
enhance: Use v2 package name for pkg module (#39990)
2025-02-22 23:15:58 +08:00
compaction_task_meta.go
fix: Fix invalid pre-allocated segment IDs (#44350)
2025-09-15 10:27:58 +08:00
compaction_task_mix_test.go
enhance: remove redundant channel manager from datacoord (#44532)
2025-10-09 11:01:57 +08:00
compaction_task_mix.go
enhance: integrate Storage V2 FFI interface for unified storage access (#45723)
2025-11-24 19:57:07 +08:00
compaction_task.go
fix: Accidentally ignored sealed segments in L0 Compaction (#45340)
2025-11-06 16:53:38 +08:00
compaction_trigger_test.go
enhance: Remove timeout for compaction task (#44277)
2025-09-15 11:03:58 +08:00
compaction_trigger_v2_test.go
feat: support mannual compact l0(#44439) (#44440)
2025-09-23 12:44:07 +08:00
compaction_trigger_v2.go
feat: Auto add namespace field data if namespace is enabled (#44933)
2025-10-24 18:40:05 +08:00
compaction_trigger.go
fix: Ignore compaction task when from segment is not healthy (#45534)
2025-11-13 23:07:39 +08:00
compaction_util.go
enhance: Add unified GRPC latency metrics in inteceptor (#44089)
2025-08-28 09:53:51 +08:00
compaction_view.go
feat: support mannual compact l0(#44439) (#44440)
2025-09-23 12:44:07 +08:00
const.go
enhance: pass partition key scalar info if enabled when build vector index (#29931)
2024-01-24 00:04:55 +08:00
create_meta_test.go
enhance: Pooling for data tasks (#41256)
2025-05-20 21:06:24 +08:00
ddl_callbacks_alter_index.go
enhance: support collection and index with WAL-based DDL framework (#45033)
2025-10-30 14:24:08 +08:00
ddl_callbacks_create_index.go
enhance: support collection and index with WAL-based DDL framework (#45033)
2025-10-30 14:24:08 +08:00
ddl_callbacks_drop_index.go
enhance: support collection and index with WAL-based DDL framework (#45033)
2025-10-30 14:24:08 +08:00
ddl_callbacks.go
enhance: support collection and index with WAL-based DDL framework (#45033)
2025-10-30 14:24:08 +08:00
errors_test.go
enhance: Use v2 package name for pkg module (#39990)
2025-02-22 23:15:58 +08:00
errors.go
Refine DataCoord status (#27262)
2023-09-26 17:15:27 +08:00
external_collection_inspector.go
enhance: implement external collection update task with source change detection (#45690)
2025-11-27 15:33:08 +08:00
external_collection_task_meta.go
enhance: implement external collection update task with source change detection (#45690)
2025-11-27 15:33:08 +08:00
file_resource_manager_test.go
enhance: sync file resource to querynode and datanode (#44480)
2025-12-04 16:23:11 +08:00
file_resource_manager.go
enhance: sync file resource to querynode and datanode (#44480)
2025-12-04 16:23:11 +08:00
garbage_collector_test.go
enhance: Check loaded segments before gc (#42639)
2025-06-13 17:44:38 +08:00
garbage_collector.go
feat: Add RESTful api for rolling upgrade support (#44381)
2025-09-16 20:08:00 +08:00
go_channel_singleton.go
fix: Fix improper use of offset in HybridSearch (#36244)
2024-09-13 22:05:15 +08:00
handler_test.go
enhance: support collection and index with WAL-based DDL framework (#45033)
2025-10-30 14:24:08 +08:00
handler.go
fix: remove wrong start timetick to avoid filtering DML whose timetick is less than it. (#44691)
2025-10-09 20:39:58 +08:00
import_checker_test.go
enhance: sync file resource to querynode and datanode (#44480)
2025-12-04 16:23:11 +08:00
import_checker.go
enhance: Skip redundant failure marking for completed import jobs (#45767)
2025-11-25 11:21:07 +08:00
import_inspector_test.go
enhance: sync file resource to querynode and datanode (#44480)
2025-12-04 16:23:11 +08:00
import_inspector.go
enhance: Move sort stats task to sort compaction (#42562)
2025-07-08 20:22:47 +08:00
import_job.go
enhance: Show create time for import job (#45058)
2025-10-27 12:14:08 +08:00
import_meta_test.go
enhance: Pooling for data tasks (#41256)
2025-05-20 21:06:24 +08:00
import_meta.go
enhance: Enhance import context (#42021)
2025-05-23 12:58:27 +08:00
import_task_import_test.go
enhance: Enhance import context (#42021)
2025-05-23 12:58:27 +08:00
import_task_import.go
fix: Handle retry state in import task (#43474)
2025-07-22 14:52:53 +08:00
import_task_preimport_test.go
enhance: Enhance import context (#42021)
2025-05-23 12:58:27 +08:00
import_task_preimport.go
fix: Fix task state is InProgress but payload is nil (#43777)
2025-08-11 14:13:42 +08:00
import_task.go
enhance: Move sort stats task to sort compaction (#42562)
2025-07-08 20:22:47 +08:00
import_util_test.go
enhance: sync file resource to querynode and datanode (#44480)
2025-12-04 16:23:11 +08:00
import_util.go
feat: Support zilliz models (#45168)
2025-11-13 12:55:37 +08:00
index_engine_version_manager_test.go
enhance: Implement rewatch mechanism for etcd failure scenarios (#43829)
2025-08-14 10:31:44 +08:00
index_engine_version_manager.go
enhance: Implement rewatch mechanism for etcd failure scenarios (#43829)
2025-08-14 10:31:44 +08:00
index_inspector_test.go
enhance: override index_type while creating segment index (#45416)
2025-11-11 07:27:36 +08:00
index_inspector.go
enhance: Estimate the taskSlot based on whether scalar or vector index (#45850)
2025-12-04 14:15:10 +08:00
index_meta_test.go
fix: ddl framework bug patch (#45290)
2025-11-04 22:25:33 +08:00
index_meta.go
enhance: override index_type while creating segment index (#45416)
2025-11-11 07:27:36 +08:00
index_service_test.go
fix: support upgrading from 2.6.x to 2.6.5 (#45264)
2025-11-04 18:31:32 +08:00
index_service.go
enhance: override index_type while creating segment index (#45416)
2025-11-11 07:27:36 +08:00
knapsack_test.go
enhance: refine compaction trigger to reduce read/write amplifaction(#41336) (#41728)
2025-06-04 11:24:38 +08:00
knapsack.go
enhance: refine compaction trigger to reduce read/write amplifaction(#41336) (#41728)
2025-06-04 11:24:38 +08:00
meta_test.go
enhance: sync file resource to querynode and datanode (#44480)
2025-12-04 16:23:11 +08:00
meta_util.go
fix: make savebinlogpath idompotent at binlog level (#43615)
2025-07-29 19:47:36 +08:00
meta.go
enhance: sync file resource to querynode and datanode (#44480)
2025-12-04 16:23:11 +08:00
metrics_info_test.go
enhance: sync file resource to querynode and datanode (#44480)
2025-12-04 16:23:11 +08:00
metrics_info.go
enhance: remove redundant channel manager from datacoord (#44532)
2025-10-09 11:01:57 +08:00
mock_compaction_inspector.go
enhance: Move sort stats task to sort compaction (#42562)
2025-07-08 20:22:47 +08:00
mock_compaction_meta.go
enhance: Fail compaction task to prevent data loss (#43545)
2025-07-25 16:24:54 +08:00
mock_handler.go
feat: Support add_function/alter_function/drop_function (#44895)
2025-11-13 20:53:39 +08:00
mock_import_meta.go
fix: Pre-check import message to prevent pipeline block indefinitely (#42415)
2025-06-11 13:40:38 +08:00
mock_index_engine_version_manager.go
enhance: Pooling for data tasks (#41256)
2025-05-20 21:06:24 +08:00
mock_segment_manager.go
feat: add NotifyDropPartition in mixcoord for droppartition in dc (#42029)
2025-05-23 18:32:26 +08:00
mock_stats_job_manager.go
enhance: Use v2 package name for pkg module (#39990)
2025-02-22 23:15:58 +08:00
mock_sub_cluster.go
enhance: Pooling for data tasks (#41256)
2025-05-20 21:06:24 +08:00
mock_test.go
enhance: sync file resource to querynode and datanode (#44480)
2025-12-04 16:23:11 +08:00
mock_trigger_manager.go
feat: support mannual compact l0(#44439) (#44440)
2025-09-23 12:44:07 +08:00
mock_trigger.go
enhance: Pooling for data tasks (#41256)
2025-05-20 21:06:24 +08:00
OWNERS
[skip ci]Update OWNERS files (#11898)
2021-11-16 15:41:11 +08:00
partition_stats_meta_test.go
enhance: Use v2 package name for pkg module (#39990)
2025-02-22 23:15:58 +08:00
partition_stats_meta.go
enhance: Use v2 package name for pkg module (#39990)
2025-02-22 23:15:58 +08:00
README.md
[skip ci]Change etcd to lowercase (#9983)
2021-10-15 18:58:37 +08:00
segment_allocation_policy_test.go
fix: Ignore growing segment without start pos for seal policy (#41130)
2025-04-07 22:16:23 +08:00
segment_allocation_policy.go
fix: Ignore growing segment without start pos for seal policy (#41130)
2025-04-07 22:16:23 +08:00
segment_info_test.go
fix: Fix GetCompactionTo return empty results when segment was GCed (#44270)
2025-09-12 18:11:58 +08:00
segment_info.go
fix: Fix GetCompactionTo return empty results when segment was GCed (#44270)
2025-09-12 18:11:58 +08:00
segment_manager_test.go
feat: add NotifyDropPartition in mixcoord for droppartition in dc (#42029)
2025-05-23 18:32:26 +08:00
segment_manager.go
feat: add NotifyDropPartition in mixcoord for droppartition in dc (#42029)
2025-05-23 18:32:26 +08:00
segment_operator_test.go
enhance: Use v2 package name for pkg module (#39990)
2025-02-22 23:15:58 +08:00
segment_operator.go
enhance: Add json key inverted index in stats for optimization (#38039)
2025-04-10 15:20:28 +08:00
server_test.go
fix: resolve SessionWatcher goroutine leak and unstable UT in querycoordv2 (#45627)
2025-11-21 18:33:06 +08:00
server.go
enhance: sync file resource to querynode and datanode (#44480)
2025-12-04 16:23:11 +08:00
services_test.go
enhance: sync file resource to querynode and datanode (#44480)
2025-12-04 16:23:11 +08:00
services.go
enhance: sync file resource to querynode and datanode (#44480)
2025-12-04 16:23:11 +08:00
stats_inspector_test.go
enhance: Move sort stats task to sort compaction (#42562)
2025-07-08 20:22:47 +08:00
stats_inspector.go
enhance: support json stats with shredding design (#42534)
2025-09-01 10:49:52 +08:00
stats_task_meta_test.go
enhance: Move sort stats task to sort compaction (#42562)
2025-07-08 20:22:47 +08:00
stats_task_meta.go
enhance: Move sort stats task to sort compaction (#42562)
2025-07-08 20:22:47 +08:00
task_analyze_test.go
feat: Add CMEK cipher plugin (#43722)
2025-08-27 11:15:52 +08:00
task_analyze.go
feat: Add CMEK cipher plugin (#43722)
2025-08-27 11:15:52 +08:00
task_index_test.go
enhance: Pooling for data tasks (#41256)
2025-05-20 21:06:24 +08:00
task_index.go
enhance: Resolve issues integrating loon FFI (#45918)
2025-11-28 18:41:08 +08:00
task_stats_test.go
enhance: Move sort stats task to sort compaction (#42562)
2025-07-08 20:22:47 +08:00
task_stats.go
enhance: support json stats with shredding design (#42534)
2025-09-01 10:49:52 +08:00
task_update_external_collection.go
enhance: implement external collection update task with source change detection (#45690)
2025-11-27 15:33:08 +08:00
util_test.go
enhance: Use v2 package name for pkg module (#39990)
2025-02-22 23:15:58 +08:00
util.go
enhance: Estimate the taskSlot based on whether scalar or vector index (#45850)
2025-12-04 14:15:10 +08:00

README.md

Data Coordinator

Data cooridnator(datacoord for short) is the component to organize DataNodes and segments allocations.

Dependency

  • KV store: a kv store has all the meta info datacoord needs to operate. (etcd)
  • Message stream: a message stream to communicate statistics information with data nodes. (Pulsar)
  • Root Coordinator: timestamp, id and meta source.
  • Data Node(s): could be an instance or a cluster, actual worker group handles data modification operations.