mirror of https://gitee.com/milvus-io/milvus.git synced 2025-12-07 17:48:29 +08:00

History

fix: Accidentally ignored sealed segments in L0 Compaction (#45340 )

When there're no growing segments in the collection, L0 Compaction will
try to choose all L0 segments that hits all L1/L2 segments.

However, if there's Sealed Segment still under flushing in DataNode at
the same time L0 Compaction selects satisfied L1/L2 segments, L0
Compaction will ignore this Segment because it's not in "FlushState",
which is wrong, causing missing deletes on the Sealed Segment.

This quick solution here is to fail this L0 compaction task once
selected a Sealed segment.

See also: #45339

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

2025-11-06 16:53:38 +08:00

allocator

enhance: Pooling for data tasks (#41256 )

2025-05-20 21:06:24 +08:00

broker

enhance: Optimize FlushAll performance for multi-table scenarios (#43339 )

2025-07-30 15:37:37 +08:00

session

enhance: remove redundant channel manager from datacoord (#44532 )

2025-10-09 11:01:57 +08:00

task

fix: Skip running scalar index when segment was compacted (#44690 )

2025-10-11 12:05:56 +08:00

.mockery.yaml

fix: Pre-check import message to prevent pipeline block indefinitely (#42415 )

2025-06-11 13:40:38 +08:00

analyze_inspector.go

fix: Fix task getting stuck after recovery (#42114 )

2025-05-28 12:46:28 +08:00

analyze_meta_test.go

enhance: Use v2 package name for pkg module (#39990 )

2025-02-22 23:15:58 +08:00

analyze_meta.go

enhance: Pooling for data tasks (#41256 )

2025-05-20 21:06:24 +08:00

build_index_policy.go

Format the code (#27275 )

2023-09-21 09:45:27 +08:00

channel.go

enhance: remove redundant channel manager from datacoord (#44532 )

2025-10-09 11:01:57 +08:00

compaction_inspector_test.go

enhance: remove redundant channel manager from datacoord (#44532 )

2025-10-09 11:01:57 +08:00

compaction_inspector.go

enhance: remove redundant channel manager from datacoord (#44532 )

2025-10-09 11:01:57 +08:00

compaction_l0_view_test.go

feat: support mannual compact l0(#44439 ) (#44440 )

2025-09-23 12:44:07 +08:00

compaction_l0_view.go

feat: support mannual compact l0(#44439 ) (#44440 )

2025-09-23 12:44:07 +08:00

compaction_policy_clustering_test.go

fix: Use diskSegmentMaxSize for coll with sparse and dense vectors (#43194 )

2025-07-16 18:04:52 +08:00

compaction_policy_clustering.go

feat: Auto add namespace field data if namespace is enabled (#44933 )

2025-10-24 18:40:05 +08:00

compaction_policy_l0_test.go

feat: support mannual compact l0(#44439 ) (#44440 )

2025-09-23 12:44:07 +08:00

compaction_policy_l0.go

feat: support mannual compact l0(#44439 ) (#44440 )

2025-09-23 12:44:07 +08:00

compaction_policy_single_test.go

enhance: Move sort stats task to sort compaction (#42562 )

2025-07-08 20:22:47 +08:00

compaction_policy_single.go

feat: Auto add namespace field data if namespace is enabled (#44933 )

2025-10-24 18:40:05 +08:00

compaction_queue_test.go

enhance: Use v2 package name for pkg module (#39990 )

2025-02-22 23:15:58 +08:00

compaction_queue.go

enhance: Use v2 package name for pkg module (#39990 )

2025-02-22 23:15:58 +08:00

compaction_task_clustering_test.go

enhance: support collection and index with WAL-based DDL framework (#45033 )

2025-10-30 14:24:08 +08:00

compaction_task_clustering.go

fix: Accidentally ignored sealed segments in L0 Compaction (#45340 )

2025-11-06 16:53:38 +08:00

compaction_task_l0_test.go

enhance: remove redundant channel manager from datacoord (#44532 )

2025-10-09 11:01:57 +08:00

compaction_task_l0.go

fix: Accidentally ignored sealed segments in L0 Compaction (#45340 )

2025-11-06 16:53:38 +08:00

compaction_task_meta_test.go

enhance: Use v2 package name for pkg module (#39990 )

2025-02-22 23:15:58 +08:00

compaction_task_meta.go

fix: Fix invalid pre-allocated segment IDs (#44350 )

2025-09-15 10:27:58 +08:00

compaction_task_mix_test.go

enhance: remove redundant channel manager from datacoord (#44532 )

2025-10-09 11:01:57 +08:00

compaction_task_mix.go

fix: Accidentally ignored sealed segments in L0 Compaction (#45340 )

2025-11-06 16:53:38 +08:00

compaction_task.go

fix: Accidentally ignored sealed segments in L0 Compaction (#45340 )

2025-11-06 16:53:38 +08:00

compaction_trigger_test.go

enhance: Remove timeout for compaction task (#44277 )

2025-09-15 11:03:58 +08:00

compaction_trigger_v2_test.go

feat: support mannual compact l0(#44439 ) (#44440 )

2025-09-23 12:44:07 +08:00

compaction_trigger_v2.go

feat: Auto add namespace field data if namespace is enabled (#44933 )

2025-10-24 18:40:05 +08:00

compaction_trigger.go

feat: Auto add namespace field data if namespace is enabled (#44933 )

2025-10-24 18:40:05 +08:00

compaction_util.go

enhance: Add unified GRPC latency metrics in inteceptor (#44089 )

2025-08-28 09:53:51 +08:00

compaction_view.go

feat: support mannual compact l0(#44439 ) (#44440 )

2025-09-23 12:44:07 +08:00

const.go

enhance: pass partition key scalar info if enabled when build vector index (#29931 )

2024-01-24 00:04:55 +08:00

create_meta_test.go

enhance: Pooling for data tasks (#41256 )

2025-05-20 21:06:24 +08:00

ddl_callbacks_alter_index.go

enhance: support collection and index with WAL-based DDL framework (#45033 )

2025-10-30 14:24:08 +08:00

ddl_callbacks_create_index.go

enhance: support collection and index with WAL-based DDL framework (#45033 )

2025-10-30 14:24:08 +08:00

ddl_callbacks_drop_index.go

enhance: support collection and index with WAL-based DDL framework (#45033 )

2025-10-30 14:24:08 +08:00

ddl_callbacks.go

enhance: support collection and index with WAL-based DDL framework (#45033 )

2025-10-30 14:24:08 +08:00

errors_test.go

enhance: Use v2 package name for pkg module (#39990 )

2025-02-22 23:15:58 +08:00

errors.go

Refine DataCoord status (#27262 )

2023-09-26 17:15:27 +08:00

garbage_collector_test.go

enhance: Check loaded segments before gc (#42639 )

2025-06-13 17:44:38 +08:00

garbage_collector.go

feat: Add RESTful api for rolling upgrade support (#44381 )

2025-09-16 20:08:00 +08:00

go_channel_singleton.go

fix: Fix improper use of offset in HybridSearch (#36244 )

2024-09-13 22:05:15 +08:00

handler_test.go

enhance: support collection and index with WAL-based DDL framework (#45033 )

2025-10-30 14:24:08 +08:00

handler.go

fix: remove wrong start timetick to avoid filtering DML whose timetick is less than it. (#44691 )

2025-10-09 20:39:58 +08:00

import_checker_test.go

enhance: add file resource api (#43766 )

2025-08-08 14:17:41 +08:00

import_checker.go

fix: Fix import fileStats incorrectly set to nil (#43463 )

2025-07-22 12:37:01 +08:00

import_inspector_test.go

enhance: remove redundant channel manager from datacoord (#44532 )

2025-10-09 11:01:57 +08:00

import_inspector.go

enhance: Move sort stats task to sort compaction (#42562 )

2025-07-08 20:22:47 +08:00

import_job.go

enhance: Show create time for import job (#45058 )

2025-10-27 12:14:08 +08:00

import_meta_test.go

enhance: Pooling for data tasks (#41256 )

2025-05-20 21:06:24 +08:00

import_meta.go

enhance: Enhance import context (#42021 )

2025-05-23 12:58:27 +08:00

import_task_import_test.go

enhance: Enhance import context (#42021 )

2025-05-23 12:58:27 +08:00

import_task_import.go

fix: Handle retry state in import task (#43474 )

2025-07-22 14:52:53 +08:00

import_task_preimport_test.go

enhance: Enhance import context (#42021 )

2025-05-23 12:58:27 +08:00

import_task_preimport.go

fix: Fix task state is InProgress but payload is nil (#43777 )

2025-08-11 14:13:42 +08:00

import_task.go

enhance: Move sort stats task to sort compaction (#42562 )

2025-07-08 20:22:47 +08:00

import_util_test.go

enhance: add file resource api (#43766 )

2025-08-08 14:17:41 +08:00

import_util.go

feat: encode cluster id in auto id (#44471 )

2025-09-22 10:40:02 +08:00

index_engine_version_manager_test.go

enhance: Implement rewatch mechanism for etcd failure scenarios (#43829 )

2025-08-14 10:31:44 +08:00

index_engine_version_manager.go

enhance: Implement rewatch mechanism for etcd failure scenarios (#43829 )

2025-08-14 10:31:44 +08:00

index_inspector_test.go

enhance: Pooling for data tasks (#41256 )

2025-05-20 21:06:24 +08:00

index_inspector.go

enhance: Move sort stats task to sort compaction (#42562 )

2025-07-08 20:22:47 +08:00

index_meta_test.go

fix: ddl framework bug patch (#45290 )

2025-11-04 22:25:33 +08:00

index_meta.go

fix: ddl framework bug patch (#45290 )

2025-11-04 22:25:33 +08:00

index_service_test.go

fix: support upgrading from 2.6.x to 2.6.5 (#45264 )

2025-11-04 18:31:32 +08:00

index_service.go

fix: ddl framework bug patch (#45290 )

2025-11-04 22:25:33 +08:00

knapsack_test.go

enhance: refine compaction trigger to reduce read/write amplifaction(#41336 ) (#41728 )

2025-06-04 11:24:38 +08:00

knapsack.go

enhance: refine compaction trigger to reduce read/write amplifaction(#41336 ) (#41728 )

2025-06-04 11:24:38 +08:00

meta_test.go

enhance: [StorageV2] add manifest path support for FFI integration (#44991 )

2025-10-27 19:24:10 +08:00

meta_util.go

fix: make savebinlogpath idompotent at binlog level (#43615 )

2025-07-29 19:47:36 +08:00

meta.go

enhance: [StorageV2] add manifest path support for FFI integration (#44991 )

2025-10-27 19:24:10 +08:00

metrics_info_test.go

enhance: remove redundant channel manager from datacoord (#44532 )

2025-10-09 11:01:57 +08:00

metrics_info.go

enhance: remove redundant channel manager from datacoord (#44532 )

2025-10-09 11:01:57 +08:00

mock_compaction_inspector.go

enhance: Move sort stats task to sort compaction (#42562 )

2025-07-08 20:22:47 +08:00

mock_compaction_meta.go

enhance: Fail compaction task to prevent data loss (#43545 )

2025-07-25 16:24:54 +08:00

mock_handler.go

enhance: remove redundant channel manager from datacoord (#44532 )

2025-10-09 11:01:57 +08:00

mock_import_meta.go

fix: Pre-check import message to prevent pipeline block indefinitely (#42415 )

2025-06-11 13:40:38 +08:00

mock_index_engine_version_manager.go

enhance: Pooling for data tasks (#41256 )

2025-05-20 21:06:24 +08:00

mock_segment_manager.go

feat: add NotifyDropPartition in mixcoord for droppartition in dc (#42029 )

2025-05-23 18:32:26 +08:00

mock_stats_job_manager.go

enhance: Use v2 package name for pkg module (#39990 )

2025-02-22 23:15:58 +08:00

mock_sub_cluster.go

enhance: Pooling for data tasks (#41256 )

2025-05-20 21:06:24 +08:00

mock_test.go

enhance: forbid build analyzer at proxy (#44067 )

2025-10-23 10:58:12 +08:00

mock_trigger_manager.go

feat: support mannual compact l0(#44439 ) (#44440 )

2025-09-23 12:44:07 +08:00

mock_trigger.go

enhance: Pooling for data tasks (#41256 )

2025-05-20 21:06:24 +08:00

OWNERS

[skip ci]Update OWNERS files (#11898 )

2021-11-16 15:41:11 +08:00

partition_stats_meta_test.go

enhance: Use v2 package name for pkg module (#39990 )

2025-02-22 23:15:58 +08:00

partition_stats_meta.go

enhance: Use v2 package name for pkg module (#39990 )

2025-02-22 23:15:58 +08:00

README.md

[skip ci]Change etcd to lowercase (#9983 )

2021-10-15 18:58:37 +08:00

segment_allocation_policy_test.go

fix: Ignore growing segment without start pos for seal policy (#41130 )

2025-04-07 22:16:23 +08:00

segment_allocation_policy.go

fix: Ignore growing segment without start pos for seal policy (#41130 )

2025-04-07 22:16:23 +08:00

segment_info_test.go

fix: Fix GetCompactionTo return empty results when segment was GCed (#44270 )

2025-09-12 18:11:58 +08:00

segment_info.go

fix: Fix GetCompactionTo return empty results when segment was GCed (#44270 )

2025-09-12 18:11:58 +08:00

segment_manager_test.go

feat: add NotifyDropPartition in mixcoord for droppartition in dc (#42029 )

2025-05-23 18:32:26 +08:00

segment_manager.go

feat: add NotifyDropPartition in mixcoord for droppartition in dc (#42029 )

2025-05-23 18:32:26 +08:00

segment_operator_test.go

enhance: Use v2 package name for pkg module (#39990 )

2025-02-22 23:15:58 +08:00

segment_operator.go

enhance: Add json key inverted index in stats for optimization (#38039 )

2025-04-10 15:20:28 +08:00

server_test.go

enhance: support collection and index with WAL-based DDL framework (#45033 )

2025-10-30 14:24:08 +08:00

server.go

enhance: support collection and index with WAL-based DDL framework (#45033 )

2025-10-30 14:24:08 +08:00

services_test.go

enhance: remove resource type from file resource config (#45103 )

2025-11-03 10:15:32 +08:00

services.go

enhance: remove resource type from file resource config (#45103 )

2025-11-03 10:15:32 +08:00

stats_inspector_test.go

enhance: Move sort stats task to sort compaction (#42562 )

2025-07-08 20:22:47 +08:00

stats_inspector.go

enhance: support json stats with shredding design (#42534 )

2025-09-01 10:49:52 +08:00

stats_task_meta_test.go

enhance: Move sort stats task to sort compaction (#42562 )

2025-07-08 20:22:47 +08:00

stats_task_meta.go

enhance: Move sort stats task to sort compaction (#42562 )

2025-07-08 20:22:47 +08:00

task_analyze_test.go

feat: Add CMEK cipher plugin (#43722 )

2025-08-27 11:15:52 +08:00

task_analyze.go

feat: Add CMEK cipher plugin (#43722 )

2025-08-27 11:15:52 +08:00

task_index_test.go

enhance: Pooling for data tasks (#41256 )

2025-05-20 21:06:24 +08:00

task_index.go

fix: Skip running scalar index when segment was compacted (#44690 )

2025-10-11 12:05:56 +08:00

task_stats_test.go

enhance: Move sort stats task to sort compaction (#42562 )

2025-07-08 20:22:47 +08:00

task_stats.go

enhance: support json stats with shredding design (#42534 )

2025-09-01 10:49:52 +08:00

util_test.go

enhance: Use v2 package name for pkg module (#39990 )

2025-02-22 23:15:58 +08:00

util.go

enhance: remove redundant channel manager from datacoord (#44532 )

2025-10-09 11:01:57 +08:00

README.md

Data Coordinator

Data cooridnator(datacoord for short) is the component to organize DataNodes and segments allocations.

Dependency

KV store: a kv store has all the meta info datacoord needs to operate. (etcd)
Message stream: a message stream to communicate statistics information with data nodes. (Pulsar)
Root Coordinator: timestamp, id and meta source.
Data Node(s): could be an instance or a cluster, actual worker group handles data modification operations.