admin/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2025-12-28 14:35:27 +08:00
Code Issues Packages Projects Releases Wiki Activity
milvus/internal/datacoord
History
XuanYang-cn 99b53316e5
enhance: Set latestDeletePos from L0 segments to bound L1 selection (#46436) 
This commit refines L0 compaction to ensure data consistency by properly
setting the delete position boundary for L1 segment selection.

Key Changes:
1. L0 View Trigger Sets latestDeletePos for L1 Selection
2. Filter L0 Segments by Growing Segment Position in policy, not in
views
3. Renamed LevelZeroSegmentsView to LevelZeroCompactionView
4. Renamed fields for semantic clarity: * segments -> l0Segments *
earliestGrowingSegmentPos -> latestDeletePos
5. Update Default Compaction Prioritizer to level

See also: #46434

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2025-12-23 11:55:19 +08:00
..
allocator
enhance: Pooling for data tasks (#41256)
2025-05-20 21:06:24 +08:00
broker
enhance: Optimize FlushAll performance for multi-table scenarios (#43339)
2025-07-30 15:37:37 +08:00
session
enhance: remove redundant channel manager from datacoord (#44532)
2025-10-09 11:01:57 +08:00
task
fix: Skip running scalar index when segment was compacted (#44690)
2025-10-11 12:05:56 +08:00
.mockery.yaml
feat: Add force merge (#45556)
2025-12-19 18:03:18 +08:00
analyze_inspector.go
fix: Fix task getting stuck after recovery (#42114)
2025-05-28 12:46:28 +08:00
analyze_meta_test.go
enhance: Use v2 package name for pkg module (#39990)
2025-02-22 23:15:58 +08:00
analyze_meta.go
enhance: Pooling for data tasks (#41256)
2025-05-20 21:06:24 +08:00
build_index_policy.go
Format the code (#27275)
2023-09-21 09:45:27 +08:00
channel.go
enhance: remove redundant channel manager from datacoord (#44532)
2025-10-09 11:01:57 +08:00
compaction_inspector_test.go
enhance: Set latestDeletePos from L0 segments to bound L1 selection (#46436)
2025-12-23 11:55:19 +08:00
compaction_inspector.go
enhance: Set latestDeletePos from L0 segments to bound L1 selection (#46436)
2025-12-23 11:55:19 +08:00
compaction_l0_view_test.go
enhance: Set latestDeletePos from L0 segments to bound L1 selection (#46436)
2025-12-23 11:55:19 +08:00
compaction_l0_view.go
enhance: Set latestDeletePos from L0 segments to bound L1 selection (#46436)
2025-12-23 11:55:19 +08:00
compaction_policy_clustering_test.go
enhance: remove uncessary segment size estimation and make it configurable (#46302)
2025-12-13 02:58:46 +08:00
compaction_policy_clustering.go
enhance: remove uncessary segment size estimation and make it configurable (#46302)
2025-12-13 02:58:46 +08:00
compaction_policy_forcemerge_test.go
feat: Add force merge (#45556)
2025-12-19 18:03:18 +08:00
compaction_policy_forcemerge.go
feat: Add force merge (#45556)
2025-12-19 18:03:18 +08:00
compaction_policy_l0_test.go
enhance: Set latestDeletePos from L0 segments to bound L1 selection (#46436)
2025-12-23 11:55:19 +08:00
compaction_policy_l0.go
enhance: Set latestDeletePos from L0 segments to bound L1 selection (#46436)
2025-12-23 11:55:19 +08:00
compaction_policy_single_test.go
enhance: Move sort stats task to sort compaction (#42562)
2025-07-08 20:22:47 +08:00
compaction_policy_single.go
feat: Auto add namespace field data if namespace is enabled (#44933)
2025-10-24 18:40:05 +08:00
compaction_queue_test.go
enhance: Use v2 package name for pkg module (#39990)
2025-02-22 23:15:58 +08:00
compaction_queue.go
enhance: Use v2 package name for pkg module (#39990)
2025-02-22 23:15:58 +08:00
compaction_task_clustering_test.go
fix: Set task init when worker doesn't have task (#45675)
2025-11-19 18:03:07 +08:00
compaction_task_clustering.go
enhance: integrate Storage V2 FFI interface for unified storage access (#45723)
2025-11-24 19:57:07 +08:00
compaction_task_l0_test.go
fix: Set task init when worker doesn't have task (#45675)
2025-11-19 18:03:07 +08:00
compaction_task_l0.go
enhance: integrate Storage V2 FFI interface for unified storage access (#45723)
2025-11-24 19:57:07 +08:00
compaction_task_meta_test.go
enhance: Use v2 package name for pkg module (#39990)
2025-02-22 23:15:58 +08:00
compaction_task_meta.go
fix: Fix invalid pre-allocated segment IDs (#44350)
2025-09-15 10:27:58 +08:00
compaction_task_mix_test.go
enhance: remove redundant channel manager from datacoord (#44532)
2025-10-09 11:01:57 +08:00
compaction_task_mix.go
enhance: Add log with segment size for tasks (#46118)
2025-12-05 16:45:11 +08:00
compaction_task.go
fix: Accidentally ignored sealed segments in L0 Compaction (#45340)
2025-11-06 16:53:38 +08:00
compaction_trigger_test.go
feat: Add force merge (#45556)
2025-12-19 18:03:18 +08:00
compaction_trigger_v2_test.go
enhance: Set latestDeletePos from L0 segments to bound L1 selection (#46436)
2025-12-23 11:55:19 +08:00
compaction_trigger_v2.go
enhance: Set latestDeletePos from L0 segments to bound L1 selection (#46436)
2025-12-23 11:55:19 +08:00
compaction_trigger.go
feat: Add force merge (#45556)
2025-12-19 18:03:18 +08:00
compaction_util.go
feat: Enable to backup and reload ez (#46332)
2025-12-16 17:19:16 +08:00
compaction_view_forcemerge_test.go
feat: Add force merge (#45556)
2025-12-19 18:03:18 +08:00
compaction_view_forcemerge.go
feat: Add force merge (#45556)
2025-12-19 18:03:18 +08:00
compaction_view.go
feat: support mannual compact l0(#44439) (#44440)
2025-09-23 12:44:07 +08:00
const.go
enhance: pass partition key scalar info if enabled when build vector index (#29931)
2024-01-24 00:04:55 +08:00
create_meta_test.go
enhance: Pooling for data tasks (#41256)
2025-05-20 21:06:24 +08:00
ddl_callbacks_alter_index.go
enhance: support collection and index with WAL-based DDL framework (#45033)
2025-10-30 14:24:08 +08:00
ddl_callbacks_create_index.go
enhance: support collection and index with WAL-based DDL framework (#45033)
2025-10-30 14:24:08 +08:00
ddl_callbacks_drop_index.go
enhance: support collection and index with WAL-based DDL framework (#45033)
2025-10-30 14:24:08 +08:00
ddl_callbacks_flushall.go
enhance: Implement new FlushAllMessage and refactor flush all (#45920)
2025-12-10 19:27:13 +08:00
ddl_callbacks.go
enhance: Implement new FlushAllMessage and refactor flush all (#45920)
2025-12-10 19:27:13 +08:00
errors_test.go
enhance: Use v2 package name for pkg module (#39990)
2025-02-22 23:15:58 +08:00
errors.go
Refine DataCoord status (#27262)
2023-09-26 17:15:27 +08:00
external_collection_inspector.go
enhance: implement external collection update task with source change detection (#45690)
2025-11-27 15:33:08 +08:00
external_collection_task_meta.go
enhance: implement external collection update task with source change detection (#45690)
2025-11-27 15:33:08 +08:00
file_resource_manager_test.go
enhance: sync file resource to querynode and datanode (#44480)
2025-12-04 16:23:11 +08:00
file_resource_manager.go
enhance: sync file resource to querynode and datanode (#44480)
2025-12-04 16:23:11 +08:00
garbage_collector_test.go
enhance: support pause GC at collection level (#45943)
2025-12-08 19:33:15 +08:00
garbage_collector.go
fix: stack overflow when gc json or json key (#46317)
2025-12-12 20:05:15 +08:00
go_channel_singleton.go
fix: Fix improper use of offset in HybridSearch (#36244)
2024-09-13 22:05:15 +08:00
handler_test.go
enhance: support collection and index with WAL-based DDL framework (#45033)
2025-10-30 14:24:08 +08:00
handler.go
enhance: support AckSyncUp for broadcaster, and enable it in truncate api (#46313)
2025-12-17 16:55:17 +08:00
import_checker_test.go
enhance: sync file resource to querynode and datanode (#44480)
2025-12-04 16:23:11 +08:00
import_checker.go
enhance: Skip redundant failure marking for completed import jobs (#45767)
2025-11-25 11:21:07 +08:00
import_inspector_test.go
enhance: sync file resource to querynode and datanode (#44480)
2025-12-04 16:23:11 +08:00
import_inspector.go
enhance: Move sort stats task to sort compaction (#42562)
2025-07-08 20:22:47 +08:00
import_job.go
enhance: Show create time for import job (#45058)
2025-10-27 12:14:08 +08:00
import_meta_test.go
enhance: Pooling for data tasks (#41256)
2025-05-20 21:06:24 +08:00
import_meta.go
enhance: Enhance import context (#42021)
2025-05-23 12:58:27 +08:00
import_task_import_test.go
enhance: Enhance import context (#42021)
2025-05-23 12:58:27 +08:00
import_task_import.go
enhance: support useLoonFFI flag in import workflow (#46363)
2025-12-17 16:35:16 +08:00
import_task_preimport_test.go
enhance: Enhance import context (#42021)
2025-05-23 12:58:27 +08:00
import_task_preimport.go
fix: Fix task state is InProgress but payload is nil (#43777)
2025-08-11 14:13:42 +08:00
import_task.go
enhance: Move sort stats task to sort compaction (#42562)
2025-07-08 20:22:47 +08:00
import_util_test.go
enhance: sync file resource to querynode and datanode (#44480)
2025-12-04 16:23:11 +08:00
import_util.go
feat: Add force merge (#45556)
2025-12-19 18:03:18 +08:00
index_engine_version_manager_test.go
enhance: Implement rewatch mechanism for etcd failure scenarios (#43829)
2025-08-14 10:31:44 +08:00
index_engine_version_manager.go
enhance: Implement rewatch mechanism for etcd failure scenarios (#43829)
2025-08-14 10:31:44 +08:00
index_inspector_test.go
enhance: override index_type while creating segment index (#45416)
2025-11-11 07:27:36 +08:00
index_inspector.go
enhance: Estimate index task slot using field size instead of segment size (#46275)
2025-12-23 11:23:22 +08:00
index_meta_test.go
fix: ddl framework bug patch (#45290)
2025-11-04 22:25:33 +08:00
index_meta.go
enhance: override index_type while creating segment index (#45416)
2025-11-11 07:27:36 +08:00
index_service_test.go
fix: support upgrading from 2.6.x to 2.6.5 (#45264)
2025-11-04 18:31:32 +08:00
index_service.go
enhance: override index_type while creating segment index (#45416)
2025-11-11 07:27:36 +08:00
knapsack_test.go
enhance: refine compaction trigger to reduce read/write amplifaction(#41336) (#41728)
2025-06-04 11:24:38 +08:00
knapsack.go
enhance: refine compaction trigger to reduce read/write amplifaction(#41336) (#41728)
2025-06-04 11:24:38 +08:00
meta_test.go
feat: support TruncateCollection api to clear collection data (#46167)
2025-12-12 10:31:14 +08:00
meta_util.go
fix: make savebinlogpath idompotent at binlog level (#43615)
2025-07-29 19:47:36 +08:00
meta.go
feat: support TruncateCollection api to clear collection data (#46167)
2025-12-12 10:31:14 +08:00
metrics_info_test.go
enhance: sync file resource to querynode and datanode (#44480)
2025-12-04 16:23:11 +08:00
metrics_info.go
enhance: remove redundant channel manager from datacoord (#44532)
2025-10-09 11:01:57 +08:00
mock_collection_topology_querier.go
feat: Add force merge (#45556)
2025-12-19 18:03:18 +08:00
mock_compaction_inspector.go
enhance: Move sort stats task to sort compaction (#42562)
2025-07-08 20:22:47 +08:00
mock_compaction_meta.go
enhance: Fail compaction task to prevent data loss (#43545)
2025-07-25 16:24:54 +08:00
mock_handler.go
feat: Support add_function/alter_function/drop_function (#44895)
2025-11-13 20:53:39 +08:00
mock_import_meta.go
fix: Pre-check import message to prevent pipeline block indefinitely (#42415)
2025-06-11 13:40:38 +08:00
mock_index_engine_version_manager.go
enhance: Pooling for data tasks (#41256)
2025-05-20 21:06:24 +08:00
mock_segment_manager.go
feat: add NotifyDropPartition in mixcoord for droppartition in dc (#42029)
2025-05-23 18:32:26 +08:00
mock_stats_job_manager.go
enhance: Use v2 package name for pkg module (#39990)
2025-02-22 23:15:58 +08:00
mock_sub_cluster.go
enhance: Pooling for data tasks (#41256)
2025-05-20 21:06:24 +08:00
mock_test.go
feat: impl ComputePhraseMatchSlop for compute min slop for phrase match query (#45892)
2025-12-19 16:03:18 +08:00
mock_trigger_manager.go
feat: Add force merge (#45556)
2025-12-19 18:03:18 +08:00
mock_trigger.go
enhance: Pooling for data tasks (#41256)
2025-05-20 21:06:24 +08:00
OWNERS
[skip ci]Update OWNERS files (#11898)
2021-11-16 15:41:11 +08:00
partition_stats_meta_test.go
enhance: Use v2 package name for pkg module (#39990)
2025-02-22 23:15:58 +08:00
partition_stats_meta.go
enhance: Use v2 package name for pkg module (#39990)
2025-02-22 23:15:58 +08:00
README.md
[skip ci]Change etcd to lowercase (#9983)
2021-10-15 18:58:37 +08:00
segment_allocation_policy_test.go
enhance: remove uncessary segment size estimation and make it configurable (#46302)
2025-12-13 02:58:46 +08:00
segment_allocation_policy.go
enhance: remove uncessary segment size estimation and make it configurable (#46302)
2025-12-13 02:58:46 +08:00
segment_info_test.go
enhance: Estimate index task slot using field size instead of segment size (#46275)
2025-12-23 11:23:22 +08:00
segment_info.go
enhance: Estimate index task slot using field size instead of segment size (#46275)
2025-12-23 11:23:22 +08:00
segment_manager_test.go
feat: add NotifyDropPartition in mixcoord for droppartition in dc (#42029)
2025-05-23 18:32:26 +08:00
segment_manager.go
feat: add NotifyDropPartition in mixcoord for droppartition in dc (#42029)
2025-05-23 18:32:26 +08:00
segment_operator_test.go
enhance: Use v2 package name for pkg module (#39990)
2025-02-22 23:15:58 +08:00
segment_operator.go
enhance: Add json key inverted index in stats for optimization (#38039)
2025-04-10 15:20:28 +08:00
server_test.go
feat: Add force merge (#45556)
2025-12-19 18:03:18 +08:00
server.go
feat: Add force merge (#45556)
2025-12-19 18:03:18 +08:00
services_test.go
enhance: Return FlushAllMsg in response (#46347)
2025-12-16 10:35:16 +08:00
services.go
fix: set ManifestPath in GetRecoveryInfoV2 response (#46470)
2025-12-19 22:21:19 +08:00
stats_inspector_test.go
enhance: Move sort stats task to sort compaction (#42562)
2025-07-08 20:22:47 +08:00
stats_inspector.go
enhance: support json stats with shredding design (#42534)
2025-09-01 10:49:52 +08:00
stats_task_meta_test.go
enhance: Move sort stats task to sort compaction (#42562)
2025-07-08 20:22:47 +08:00
stats_task_meta.go
enhance: Move sort stats task to sort compaction (#42562)
2025-07-08 20:22:47 +08:00
task_analyze_test.go
feat: Add CMEK cipher plugin (#43722)
2025-08-27 11:15:52 +08:00
task_analyze.go
feat: Add CMEK cipher plugin (#43722)
2025-08-27 11:15:52 +08:00
task_index_test.go
enhance: Pooling for data tasks (#41256)
2025-05-20 21:06:24 +08:00
task_index.go
enhance: Resolve issues integrating loon FFI (#45918)
2025-11-28 18:41:08 +08:00
task_stats_test.go
enhance: Move sort stats task to sort compaction (#42562)
2025-07-08 20:22:47 +08:00
task_stats.go
enhance: pass manifest path to stats task for storage v2 support (#46350)
2025-12-16 11:11:16 +08:00
task_update_external_collection.go
enhance: implement external collection update task with source change detection (#45690)
2025-11-27 15:33:08 +08:00
util_test.go
enhance: Use v2 package name for pkg module (#39990)
2025-02-22 23:15:58 +08:00
util.go
enhance: Estimate index task slot using field size instead of segment size (#46275)
2025-12-23 11:23:22 +08:00

README.md

Data Coordinator

Data cooridnator(datacoord for short) is the component to organize DataNodes and segments allocations.

Dependency

  • KV store: a kv store has all the meta info datacoord needs to operate. (etcd)
  • Message stream: a message stream to communicate statistics information with data nodes. (Pulsar)
  • Root Coordinator: timestamp, id and meta source.
  • Data Node(s): could be an instance or a cluster, actual worker group handles data modification operations.