mirror of https://gitee.com/milvus-io/milvus.git synced 2026-01-07 19:31:51 +08:00

History

fix: resolve data race in indexMeta (#46763 )

issue: #46762

Copy the fieldIndexes map while holding the read lock to prevent data
race. The original code released the lock before iterating over the map,
which could cause concurrent access issues.

Affected methods:
- GetSegmentIndexState
- GetIndexedSegments
- IsUnIndexedSegment
- GetSegmentIndexedFields

Signed-off-by: Wei Liu <wei.liu@zilliz.com>

2026-01-07 10:33:25 +08:00

allocator

enhance: Pooling for data tasks (#41256 )

2025-05-20 21:06:24 +08:00

broker

feat: Add comprehensive snapshot functionality for collections (#44361 )

2026-01-06 10:15:24 +08:00

session

feat: Add comprehensive snapshot functionality for collections (#44361 )

2026-01-06 10:15:24 +08:00

task

fix: Skip running scalar index when segment was compacted (#44690 )

2025-10-11 12:05:56 +08:00

.mockery.yaml

feat: Add force merge (#45556 )

2025-12-19 18:03:18 +08:00

analyze_inspector.go

fix: Fix task getting stuck after recovery (#42114 )

2025-05-28 12:46:28 +08:00

analyze_meta_test.go

enhance: Use v2 package name for pkg module (#39990 )

2025-02-22 23:15:58 +08:00

analyze_meta.go

enhance: Pooling for data tasks (#41256 )

2025-05-20 21:06:24 +08:00

build_index_policy.go

Format the code (#27275 )

2023-09-21 09:45:27 +08:00

channel.go

enhance: remove redundant channel manager from datacoord (#44532 )

2025-10-09 11:01:57 +08:00

compaction_inspector_test.go

enhance: Set latestDeletePos from L0 segments to bound L1 selection (#46436 )

2025-12-23 11:55:19 +08:00

compaction_inspector.go

fix: Restore the compaction task correctly to ensure it can be properly cleaned up (#46577 )

2025-12-26 18:57:19 +08:00

compaction_l0_view_test.go

enhance: Set latestDeletePos from L0 segments to bound L1 selection (#46436 )

2025-12-23 11:55:19 +08:00

compaction_l0_view.go

enhance: Set latestDeletePos from L0 segments to bound L1 selection (#46436 )

2025-12-23 11:55:19 +08:00

compaction_policy_clustering_test.go

enhance: remove uncessary segment size estimation and make it configurable (#46302 )

2025-12-13 02:58:46 +08:00

compaction_policy_clustering.go

feat: Support ttl field for entity level expiration (#46342 )

2026-01-05 10:27:24 +08:00

compaction_policy_forcemerge_test.go

feat: Add force merge (#45556 )

2025-12-19 18:03:18 +08:00

compaction_policy_forcemerge.go

feat: Support ttl field for entity level expiration (#46342 )

2026-01-05 10:27:24 +08:00

compaction_policy_l0_test.go

enhance: Set latestDeletePos from L0 segments to bound L1 selection (#46436 )

2025-12-23 11:55:19 +08:00

compaction_policy_l0.go

enhance: Set latestDeletePos from L0 segments to bound L1 selection (#46436 )

2025-12-23 11:55:19 +08:00

compaction_policy_single_test.go

enhance: Move sort stats task to sort compaction (#42562 )

2025-07-08 20:22:47 +08:00

compaction_policy_single.go

feat: Support ttl field for entity level expiration (#46342 )

2026-01-05 10:27:24 +08:00

compaction_queue_test.go

enhance: Use v2 package name for pkg module (#39990 )

2025-02-22 23:15:58 +08:00

compaction_queue.go

enhance: Use v2 package name for pkg module (#39990 )

2025-02-22 23:15:58 +08:00

compaction_task_clustering_test.go

fix: Set task init when worker doesn't have task (#45675 )

2025-11-19 18:03:07 +08:00

compaction_task_clustering.go

enhance: integrate Storage V2 FFI interface for unified storage access (#45723 )

2025-11-24 19:57:07 +08:00

compaction_task_l0_test.go

fix: Set task init when worker doesn't have task (#45675 )

2025-11-19 18:03:07 +08:00

compaction_task_l0.go

enhance: integrate Storage V2 FFI interface for unified storage access (#45723 )

2025-11-24 19:57:07 +08:00

compaction_task_meta_test.go

enhance: Use v2 package name for pkg module (#39990 )

2025-02-22 23:15:58 +08:00

compaction_task_meta.go

fix: Skip Finished tasks when recovery with compatibility (#46515 )

2025-12-23 18:09:18 +08:00

compaction_task_mix_test.go

enhance: remove redundant channel manager from datacoord (#44532 )

2025-10-09 11:01:57 +08:00

compaction_task_mix.go

enhance: support compaction with file resource in ref mode (#46399 )

2026-01-06 16:31:31 +08:00

compaction_task.go

fix: Accidentally ignored sealed segments in L0 Compaction (#45340 )

2025-11-06 16:53:38 +08:00

compaction_trigger_test.go

feat: Support ttl field for entity level expiration (#46342 )

2026-01-05 10:27:24 +08:00

compaction_trigger_v2_test.go

enhance: Set latestDeletePos from L0 segments to bound L1 selection (#46436 )

2025-12-23 11:55:19 +08:00

compaction_trigger_v2.go

enhance: Set latestDeletePos from L0 segments to bound L1 selection (#46436 )

2025-12-23 11:55:19 +08:00

compaction_trigger.go

feat: Support ttl field for entity level expiration (#46342 )

2026-01-05 10:27:24 +08:00

compaction_util.go

enhance: tidy code in import plugin context (#46374 )

2026-01-04 16:51:23 +08:00

compaction_view_forcemerge_test.go

fix: Disable VCS to allow pkg tests to run (#46501 )

2025-12-25 14:31:19 +08:00

compaction_view_forcemerge.go

fix: Disable VCS to allow pkg tests to run (#46501 )

2025-12-25 14:31:19 +08:00

compaction_view.go

feat: support mannual compact l0(#44439 ) (#44440 )

2025-09-23 12:44:07 +08:00

const.go

enhance: pass partition key scalar info if enabled when build vector index (#29931 )

2024-01-24 00:04:55 +08:00

copy_segment_checker_test.go

feat: Add comprehensive snapshot functionality for collections (#44361 )

2026-01-06 10:15:24 +08:00

copy_segment_checker.go

feat: Add comprehensive snapshot functionality for collections (#44361 )

2026-01-06 10:15:24 +08:00

copy_segment_inspector_test.go

feat: Add comprehensive snapshot functionality for collections (#44361 )

2026-01-06 10:15:24 +08:00

copy_segment_inspector.go

feat: Add comprehensive snapshot functionality for collections (#44361 )

2026-01-06 10:15:24 +08:00

copy_segment_job_test.go

feat: Add comprehensive snapshot functionality for collections (#44361 )

2026-01-06 10:15:24 +08:00

copy_segment_job.go

feat: Add comprehensive snapshot functionality for collections (#44361 )

2026-01-06 10:15:24 +08:00

copy_segment_meta_test.go

feat: Add comprehensive snapshot functionality for collections (#44361 )

2026-01-06 10:15:24 +08:00

copy_segment_meta.go

feat: Add comprehensive snapshot functionality for collections (#44361 )

2026-01-06 10:15:24 +08:00

copy_segment_task_test.go

feat: Add comprehensive snapshot functionality for collections (#44361 )

2026-01-06 10:15:24 +08:00

copy_segment_task.go

feat: Add comprehensive snapshot functionality for collections (#44361 )

2026-01-06 10:15:24 +08:00

create_meta_test.go

enhance: Pooling for data tasks (#41256 )

2025-05-20 21:06:24 +08:00

ddl_callbacks_alter_index.go

enhance: support collection and index with WAL-based DDL framework (#45033 )

2025-10-30 14:24:08 +08:00

ddl_callbacks_create_index.go

enhance: support collection and index with WAL-based DDL framework (#45033 )

2025-10-30 14:24:08 +08:00

ddl_callbacks_drop_index.go

enhance: support collection and index with WAL-based DDL framework (#45033 )

2025-10-30 14:24:08 +08:00

ddl_callbacks_flushall.go

enhance: Implement new FlushAllMessage and refactor flush all (#45920 )

2025-12-10 19:27:13 +08:00

ddl_callbacks_snapshot_test.go

feat: Add comprehensive snapshot functionality for collections (#44361 )

2026-01-06 10:15:24 +08:00

ddl_callbacks_snapshot.go

feat: Add comprehensive snapshot functionality for collections (#44361 )

2026-01-06 10:15:24 +08:00

ddl_callbacks.go

feat: Add comprehensive snapshot functionality for collections (#44361 )

2026-01-06 10:15:24 +08:00

errors_test.go

enhance: Use v2 package name for pkg module (#39990 )

2025-02-22 23:15:58 +08:00

errors.go

Refine DataCoord status (#27262 )

2023-09-26 17:15:27 +08:00

external_collection_inspector.go

enhance: implement external collection update task with source change detection (#45905 )

2025-12-29 19:53:21 +08:00

external_collection_task_meta.go

enhance: implement external collection update task with source change detection (#45905 )

2025-12-29 19:53:21 +08:00

file_resource_manager_test.go

enhance: support compaction with file resource in ref mode (#46399 )

2026-01-06 16:31:31 +08:00

file_resource_manager.go

enhance: support compaction with file resource in ref mode (#46399 )

2026-01-06 16:31:31 +08:00

garbage_collector_test.go

feat: Add comprehensive snapshot functionality for collections (#44361 )

2026-01-06 10:15:24 +08:00

garbage_collector.go

feat: Add comprehensive snapshot functionality for collections (#44361 )

2026-01-06 10:15:24 +08:00

go_channel_singleton.go

fix: Fix improper use of offset in HybridSearch (#36244 )

2024-09-13 22:05:15 +08:00

handler_test.go

feat: Add comprehensive snapshot functionality for collections (#44361 )

2026-01-06 10:15:24 +08:00

handler.go

feat: Add comprehensive snapshot functionality for collections (#44361 )

2026-01-06 10:15:24 +08:00

import_checker_test.go

feat: Add comprehensive snapshot functionality for collections (#44361 )

2026-01-06 10:15:24 +08:00

import_checker.go

enhance: Valid row count result and avoid misleading warn log for sort compaction (#46746 )

2026-01-06 16:33:25 +08:00

import_inspector_test.go

feat: Add comprehensive snapshot functionality for collections (#44361 )

2026-01-06 10:15:24 +08:00

import_inspector.go

enhance: Move sort stats task to sort compaction (#42562 )

2025-07-08 20:22:47 +08:00

import_job.go

enhance: Show create time for import job (#45058 )

2025-10-27 12:14:08 +08:00

import_meta_test.go

enhance: Pooling for data tasks (#41256 )

2025-05-20 21:06:24 +08:00

import_meta.go

enhance: Enhance import context (#42021 )

2025-05-23 12:58:27 +08:00

import_task_import_test.go

enhance: Enhance import context (#42021 )

2025-05-23 12:58:27 +08:00

import_task_import.go

enhance: Valid row count result and avoid misleading warn log for sort compaction (#46746 )

2026-01-06 16:33:25 +08:00

import_task_preimport_test.go

enhance: Enhance import context (#42021 )

2025-05-23 12:58:27 +08:00

import_task_preimport.go

fix: Fix task state is InProgress but payload is nil (#43777 )

2025-08-11 14:13:42 +08:00

import_task.go

enhance: Move sort stats task to sort compaction (#42562 )

2025-07-08 20:22:47 +08:00

import_util_test.go

feat: Add comprehensive snapshot functionality for collections (#44361 )

2026-01-06 10:15:24 +08:00

import_util.go

enhance: Valid row count result and avoid misleading warn log for sort compaction (#46746 )

2026-01-06 16:33:25 +08:00

index_engine_version_manager_test.go

enhance: Implement rewatch mechanism for etcd failure scenarios (#43829 )

2025-08-14 10:31:44 +08:00

index_engine_version_manager.go

enhance: Implement rewatch mechanism for etcd failure scenarios (#43829 )

2025-08-14 10:31:44 +08:00

index_inspector_test.go

enhance: override index_type while creating segment index (#45416 )

2025-11-11 07:27:36 +08:00

index_inspector.go

enhance: Estimate index task slot using field size instead of segment size (#46275 )

2025-12-23 11:23:22 +08:00

index_meta_test.go

fix: ddl framework bug patch (#45290 )

2025-11-04 22:25:33 +08:00

index_meta.go

fix: resolve data race in indexMeta (#46763 )

2026-01-07 10:33:25 +08:00

index_service_test.go

feat: Add comprehensive snapshot functionality for collections (#44361 )

2026-01-06 10:15:24 +08:00

index_service.go

feat: Add comprehensive snapshot functionality for collections (#44361 )

2026-01-06 10:15:24 +08:00

knapsack_test.go

enhance: refine compaction trigger to reduce read/write amplifaction(#41336 ) (#41728 )

2025-06-04 11:24:38 +08:00

knapsack.go

enhance: refine compaction trigger to reduce read/write amplifaction(#41336 ) (#41728 )

2025-06-04 11:24:38 +08:00

meta_test.go

feat: Add comprehensive snapshot functionality for collections (#44361 )

2026-01-06 10:15:24 +08:00

meta_util.go

fix: make savebinlogpath idompotent at binlog level (#43615 )

2025-07-29 19:47:36 +08:00

meta.go

enhance: Valid row count result and avoid misleading warn log for sort compaction (#46746 )

2026-01-06 16:33:25 +08:00

metrics_info_test.go

enhance: sync file resource to querynode and datanode (#44480 )

2025-12-04 16:23:11 +08:00

metrics_info.go

enhance: remove redundant channel manager from datacoord (#44532 )

2025-10-09 11:01:57 +08:00

mock_collection_topology_querier.go

feat: Add force merge (#45556 )

2025-12-19 18:03:18 +08:00

mock_compaction_inspector.go

enhance: Move sort stats task to sort compaction (#42562 )

2025-07-08 20:22:47 +08:00

mock_compaction_meta.go

enhance: support compaction with file resource in ref mode (#46399 )

2026-01-06 16:31:31 +08:00

mock_handler.go

feat: Add comprehensive snapshot functionality for collections (#44361 )

2026-01-06 10:15:24 +08:00

mock_import_meta.go

fix: Pre-check import message to prevent pipeline block indefinitely (#42415 )

2025-06-11 13:40:38 +08:00

mock_index_engine_version_manager.go

enhance: Pooling for data tasks (#41256 )

2025-05-20 21:06:24 +08:00

mock_segment_manager.go

feat: add NotifyDropPartition in mixcoord for droppartition in dc (#42029 )

2025-05-23 18:32:26 +08:00

mock_stats_job_manager.go

enhance: Use v2 package name for pkg module (#39990 )

2025-02-22 23:15:58 +08:00

mock_sub_cluster.go

enhance: Pooling for data tasks (#41256 )

2025-05-20 21:06:24 +08:00

mock_test.go

feat: Add comprehensive snapshot functionality for collections (#44361 )

2026-01-06 10:15:24 +08:00

mock_trigger_manager.go

feat: Add force merge (#45556 )

2025-12-19 18:03:18 +08:00

mock_trigger.go

enhance: Pooling for data tasks (#41256 )

2025-05-20 21:06:24 +08:00

OWNERS

[skip ci]Update OWNERS files (#11898 )

2021-11-16 15:41:11 +08:00

partition_stats_meta_test.go

enhance: Use v2 package name for pkg module (#39990 )

2025-02-22 23:15:58 +08:00

partition_stats_meta.go

enhance: Use v2 package name for pkg module (#39990 )

2025-02-22 23:15:58 +08:00

README.md

[skip ci]Change etcd to lowercase (#9983 )

2021-10-15 18:58:37 +08:00

segment_allocation_policy_test.go

enhance: remove uncessary segment size estimation and make it configurable (#46302 )

2025-12-13 02:58:46 +08:00

segment_allocation_policy.go

enhance: remove uncessary segment size estimation and make it configurable (#46302 )

2025-12-13 02:58:46 +08:00

segment_info_test.go

enhance: Estimate index task slot using field size instead of segment size (#46275 )

2025-12-23 11:23:22 +08:00

segment_info.go

enhance: Estimate index task slot using field size instead of segment size (#46275 )

2025-12-23 11:23:22 +08:00

segment_manager_test.go

feat: add NotifyDropPartition in mixcoord for droppartition in dc (#42029 )

2025-05-23 18:32:26 +08:00

segment_manager.go

feat: add NotifyDropPartition in mixcoord for droppartition in dc (#42029 )

2025-05-23 18:32:26 +08:00

segment_operator_test.go

enhance: Use v2 package name for pkg module (#39990 )

2025-02-22 23:15:58 +08:00

segment_operator.go

enhance: Add json key inverted index in stats for optimization (#38039 )

2025-04-10 15:20:28 +08:00

server_test.go

feat: Add force merge (#45556 )

2025-12-19 18:03:18 +08:00

server.go

feat: Add comprehensive snapshot functionality for collections (#44361 )

2026-01-06 10:15:24 +08:00

services_test.go

enhance: support compaction with file resource in ref mode (#46399 )

2026-01-06 16:31:31 +08:00

services.go

feat: Add comprehensive snapshot functionality for collections (#44361 )

2026-01-06 10:15:24 +08:00

snapshot_manager_test.go

feat: Add comprehensive snapshot functionality for collections (#44361 )

2026-01-06 10:15:24 +08:00

snapshot_manager.go

feat: Add comprehensive snapshot functionality for collections (#44361 )

2026-01-06 10:15:24 +08:00

snapshot_meta_test.go

feat: Add comprehensive snapshot functionality for collections (#44361 )

2026-01-06 10:15:24 +08:00

snapshot_meta.go

feat: Add comprehensive snapshot functionality for collections (#44361 )

2026-01-06 10:15:24 +08:00

snapshot_test.go

feat: Add comprehensive snapshot functionality for collections (#44361 )

2026-01-06 10:15:24 +08:00

snapshot.go

feat: Add comprehensive snapshot functionality for collections (#44361 )

2026-01-06 10:15:24 +08:00

stats_inspector_test.go

enhance: support compaction with file resource in ref mode (#46399 )

2026-01-06 16:31:31 +08:00

stats_inspector.go

enhance: support compaction with file resource in ref mode (#46399 )

2026-01-06 16:31:31 +08:00

stats_task_meta_test.go

enhance: Move sort stats task to sort compaction (#42562 )

2025-07-08 20:22:47 +08:00

stats_task_meta.go

enhance: Move sort stats task to sort compaction (#42562 )

2025-07-08 20:22:47 +08:00

task_analyze_test.go

feat: Add CMEK cipher plugin (#43722 )

2025-08-27 11:15:52 +08:00

task_analyze.go

feat: Add CMEK cipher plugin (#43722 )

2025-08-27 11:15:52 +08:00

task_index_test.go

enhance: Pooling for data tasks (#41256 )

2025-05-20 21:06:24 +08:00

task_index.go

enhance: Resolve issues integrating loon FFI (#45918 )

2025-11-28 18:41:08 +08:00

task_stats_test.go

enhance: Move sort stats task to sort compaction (#42562 )

2025-07-08 20:22:47 +08:00

task_stats.go

enhance: pass manifest path to stats task for storage v2 support (#46350 )

2025-12-16 11:11:16 +08:00

task_update_external_collection_test.go

enhance: implement external collection update task with source change detection (#45905 )

2025-12-29 19:53:21 +08:00

task_update_external_collection.go

fix: remove redundant log (#46695 )

2025-12-31 15:35:21 +08:00

util_test.go

fix: validate collection TTL property to prevent compaction stuck (#46717 )

2026-01-01 08:13:22 +08:00

util.go

fix: validate collection TTL property to prevent compaction stuck (#46717 )

2026-01-01 08:13:22 +08:00

README.md

Data Coordinator

Data cooridnator(datacoord for short) is the component to organize DataNodes and segments allocations.

Dependency

KV store: a kv store has all the meta info datacoord needs to operate. (etcd)
Message stream: a message stream to communicate statistics information with data nodes. (Pulsar)
Root Coordinator: timestamp, id and meta source.
Data Node(s): could be an instance or a cluster, actual worker group handles data modification operations.