admin/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2025-12-06 09:08:43 +08:00
Code Issues Packages Projects Releases Wiki Activity
milvus/internal/datacoord
History
Bingyi Sun 0c0630cc38
feat: support dropping index without releasing collection (#42941) 
issue: #42942

This pr includes the following changes:
1. Added checks for index checker in querycoord to generate drop index
tasks
2. Added drop index interface to querynode
3. To avoid search failure after dropping the index, the querynode
allows the use of lazy mode (warmup=disable) to load raw data even when
indexes contain raw data.
4. In segcore, loading the index no longer deletes raw data; instead, it
evicts it.
5. In expr, the index is pinned to prevent concurrent errors.

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
2025-09-02 16:17:52 +08:00
..
allocator
enhance: Pooling for data tasks (#41256)
2025-05-20 21:06:24 +08:00
broker
enhance: Optimize FlushAll performance for multi-table scenarios (#43339)
2025-07-30 15:37:37 +08:00
session
fix: Fix retry state with palyload is not nil (#44068)
2025-08-27 18:11:49 +08:00
task
enhance: Fix unbalanced task scheduling (#43581)
2025-07-28 12:58:55 +08:00
.mockery.yaml
fix: Pre-check import message to prevent pipeline block indefinitely (#42415)
2025-06-11 13:40:38 +08:00
analyze_inspector.go
fix: Fix task getting stuck after recovery (#42114)
2025-05-28 12:46:28 +08:00
analyze_meta_test.go
enhance: Use v2 package name for pkg module (#39990)
2025-02-22 23:15:58 +08:00
analyze_meta.go
enhance: Pooling for data tasks (#41256)
2025-05-20 21:06:24 +08:00
build_index_policy.go
Format the code (#27275)
2023-09-21 09:45:27 +08:00
channel_manager_factory.go
fix: drop collection failed if enable streaming service (#37444)
2024-11-07 10:26:26 +08:00
channel_manager_test.go
fix: ChannelManager double assignment (#41837)
2025-05-23 14:16:29 +08:00
channel_manager.go
fix: ChannelManager double assignment (#41837)
2025-05-23 14:16:29 +08:00
channel_store_test.go
fix: ChannelManager double assignment (#41837)
2025-05-23 14:16:29 +08:00
channel_store.go
fix: ChannelManager double assignment (#41837)
2025-05-23 14:16:29 +08:00
channel.go
enhance: support run analyzer by loaded collection field (#42113)
2025-05-29 10:54:30 +08:00
cluster_test.go
enhance: Implement rewatch mechanism for etcd failure scenarios (#43829)
2025-08-14 10:31:44 +08:00
cluster.go
enhance: Implement rewatch mechanism for etcd failure scenarios (#43829)
2025-08-14 10:31:44 +08:00
compaction_inspector_test.go
fix: Handle compaction retry state (#44119)
2025-08-29 13:31:51 +08:00
compaction_inspector.go
enhance: Fail compaction task to prevent data loss (#43545)
2025-07-25 16:24:54 +08:00
compaction_l0_view_test.go
enhance: Use v2 package name for pkg module (#39990)
2025-02-22 23:15:58 +08:00
compaction_l0_view.go
enhance: Use v2 package name for pkg module (#39990)
2025-02-22 23:15:58 +08:00
compaction_policy_clustering_test.go
fix: Use diskSegmentMaxSize for coll with sparse and dense vectors (#43194)
2025-07-16 18:04:52 +08:00
compaction_policy_clustering.go
fix: Only mark segment compacting for sort stats task (#42516)
2025-06-04 22:46:32 +08:00
compaction_policy_l0_test.go
enhance: Pooling for data tasks (#41256)
2025-05-20 21:06:24 +08:00
compaction_policy_l0.go
enhance: Use v2 package name for pkg module (#39990)
2025-02-22 23:15:58 +08:00
compaction_policy_single_test.go
enhance: Move sort stats task to sort compaction (#42562)
2025-07-08 20:22:47 +08:00
compaction_policy_single.go
enhance: Move sort stats task to sort compaction (#42562)
2025-07-08 20:22:47 +08:00
compaction_queue_test.go
enhance: Use v2 package name for pkg module (#39990)
2025-02-22 23:15:58 +08:00
compaction_queue.go
enhance: Use v2 package name for pkg module (#39990)
2025-02-22 23:15:58 +08:00
compaction_task_clustering_test.go
enhance: Move sort stats task to sort compaction (#42562)
2025-07-08 20:22:47 +08:00
compaction_task_clustering.go
fix: Handle compaction retry state (#44119)
2025-08-29 13:31:51 +08:00
compaction_task_l0_test.go
enhance: Fail compaction task to prevent data loss (#43545)
2025-07-25 16:24:54 +08:00
compaction_task_l0.go
fix: Handle compaction retry state (#44119)
2025-08-29 13:31:51 +08:00
compaction_task_meta_test.go
enhance: Use v2 package name for pkg module (#39990)
2025-02-22 23:15:58 +08:00
compaction_task_meta.go
enhance: Use v2 package name for pkg module (#39990)
2025-02-22 23:15:58 +08:00
compaction_task_mix_test.go
enhance: support cchannel for streaming service (#44143)
2025-09-02 10:05:52 +08:00
compaction_task_mix.go
fix: Handle compaction retry state (#44119)
2025-08-29 13:31:51 +08:00
compaction_task.go
enhance: Move sort stats task to sort compaction (#42562)
2025-07-08 20:22:47 +08:00
compaction_trigger_test.go
enhance: Move sort stats task to sort compaction (#42562)
2025-07-08 20:22:47 +08:00
compaction_trigger_v2_test.go
fix: Use diskSegmentMaxSize for coll with sparse and dense vectors (#43194)
2025-07-16 18:04:52 +08:00
compaction_trigger_v2.go
feat: Add CMEK cipher plugin (#43722)
2025-08-27 11:15:52 +08:00
compaction_trigger.go
feat: impl StructArray -- create schema, insert, and retrieve data (#42855)
2025-07-27 01:30:55 +08:00
compaction_util.go
enhance: Add unified GRPC latency metrics in inteceptor (#44089)
2025-08-28 09:53:51 +08:00
compaction_view.go
enhance: Pooling for data tasks (#41256)
2025-05-20 21:06:24 +08:00
const.go
enhance: pass partition key scalar info if enabled when build vector index (#29931)
2024-01-24 00:04:55 +08:00
create_meta_test.go
enhance: Pooling for data tasks (#41256)
2025-05-20 21:06:24 +08:00
errors_test.go
enhance: Use v2 package name for pkg module (#39990)
2025-02-22 23:15:58 +08:00
errors.go
Refine DataCoord status (#27262)
2023-09-26 17:15:27 +08:00
garbage_collector_test.go
enhance: Check loaded segments before gc (#42639)
2025-06-13 17:44:38 +08:00
garbage_collector.go
enhance: support json stats with shredding design (#42534)
2025-09-01 10:49:52 +08:00
go_channel_singleton.go
fix: Fix improper use of offset in HybridSearch (#36244)
2024-09-13 22:05:15 +08:00
handler_test.go
fix: create multiple idential indexes by accident (#40179)
2025-04-08 15:06:25 +08:00
handler.go
fix: dirty cp metrics after drop (#43567)
2025-07-27 23:22:55 +08:00
import_checker_test.go
enhance: add file resource api (#43766)
2025-08-08 14:17:41 +08:00
import_checker.go
fix: Fix import fileStats incorrectly set to nil (#43463)
2025-07-22 12:37:01 +08:00
import_inspector_test.go
enhance: add file resource api (#43766)
2025-08-08 14:17:41 +08:00
import_inspector.go
enhance: Move sort stats task to sort compaction (#42562)
2025-07-08 20:22:47 +08:00
import_job.go
enhance: Use v2 package name for pkg module (#39990)
2025-02-22 23:15:58 +08:00
import_meta_test.go
enhance: Pooling for data tasks (#41256)
2025-05-20 21:06:24 +08:00
import_meta.go
enhance: Enhance import context (#42021)
2025-05-23 12:58:27 +08:00
import_task_import_test.go
enhance: Enhance import context (#42021)
2025-05-23 12:58:27 +08:00
import_task_import.go
fix: Handle retry state in import task (#43474)
2025-07-22 14:52:53 +08:00
import_task_preimport_test.go
enhance: Enhance import context (#42021)
2025-05-23 12:58:27 +08:00
import_task_preimport.go
fix: Fix task state is InProgress but payload is nil (#43777)
2025-08-11 14:13:42 +08:00
import_task.go
enhance: Move sort stats task to sort compaction (#42562)
2025-07-08 20:22:47 +08:00
import_util_test.go
enhance: add file resource api (#43766)
2025-08-08 14:17:41 +08:00
import_util.go
feat: Add CMEK cipher plugin (#43722)
2025-08-27 11:15:52 +08:00
index_engine_version_manager_test.go
enhance: Implement rewatch mechanism for etcd failure scenarios (#43829)
2025-08-14 10:31:44 +08:00
index_engine_version_manager.go
enhance: Implement rewatch mechanism for etcd failure scenarios (#43829)
2025-08-14 10:31:44 +08:00
index_inspector_test.go
enhance: Pooling for data tasks (#41256)
2025-05-20 21:06:24 +08:00
index_inspector.go
enhance: Move sort stats task to sort compaction (#42562)
2025-07-08 20:22:47 +08:00
index_meta_test.go
fix: Use locking to ensure the atomicity of dropping segment indexes (#42075)
2025-05-28 10:00:28 +08:00
index_meta.go
enhance: [DataCoord] Remove full-collection index work from metrics (#43859)
2025-08-29 12:05:50 +08:00
index_service_test.go
feat: support dropping index without releasing collection (#42941)
2025-09-02 16:17:52 +08:00
index_service.go
feat: support dropping index without releasing collection (#42941)
2025-09-02 16:17:52 +08:00
knapsack_test.go
enhance: refine compaction trigger to reduce read/write amplifaction(#41336) (#41728)
2025-06-04 11:24:38 +08:00
knapsack.go
enhance: refine compaction trigger to reduce read/write amplifaction(#41336) (#41728)
2025-06-04 11:24:38 +08:00
meta_test.go
enhance: [DataCoord] Remove full-collection index work from metrics (#43859)
2025-08-29 12:05:50 +08:00
meta_util.go
fix: make savebinlogpath idompotent at binlog level (#43615)
2025-07-29 19:47:36 +08:00
meta.go
enhance: [DataCoord] Remove full-collection index work from metrics (#43859)
2025-08-29 12:05:50 +08:00
metrics_info_test.go
enhance: Refine task meta with key lock (#40613)
2025-03-14 15:44:22 +08:00
metrics_info.go
enhance: [DataCoord] Remove full-collection index work from metrics (#43859)
2025-08-29 12:05:50 +08:00
mock_channel_manager.go
enhance: Pooling for data tasks (#41256)
2025-05-20 21:06:24 +08:00
mock_channel_store.go
fix: ChannelManager double assignment (#41837)
2025-05-23 14:16:29 +08:00
mock_cluster.go
enhance: Pooling for data tasks (#41256)
2025-05-20 21:06:24 +08:00
mock_compaction_inspector.go
enhance: Move sort stats task to sort compaction (#42562)
2025-07-08 20:22:47 +08:00
mock_compaction_meta.go
enhance: Fail compaction task to prevent data loss (#43545)
2025-07-25 16:24:54 +08:00
mock_handler.go
enhance: Check loaded segments before gc (#42639)
2025-06-13 17:44:38 +08:00
mock_import_meta.go
fix: Pre-check import message to prevent pipeline block indefinitely (#42415)
2025-06-11 13:40:38 +08:00
mock_index_engine_version_manager.go
enhance: Pooling for data tasks (#41256)
2025-05-20 21:06:24 +08:00
mock_segment_manager.go
feat: add NotifyDropPartition in mixcoord for droppartition in dc (#42029)
2025-05-23 18:32:26 +08:00
mock_stats_job_manager.go
enhance: Use v2 package name for pkg module (#39990)
2025-02-22 23:15:58 +08:00
mock_sub_cluster.go
enhance: Pooling for data tasks (#41256)
2025-05-20 21:06:24 +08:00
mock_test.go
enhance: add file resource api (#43766)
2025-08-08 14:17:41 +08:00
mock_trigger_manager.go
enhance: Pooling for data tasks (#41256)
2025-05-20 21:06:24 +08:00
mock_trigger.go
enhance: Pooling for data tasks (#41256)
2025-05-20 21:06:24 +08:00
OWNERS
[skip ci]Update OWNERS files (#11898)
2021-11-16 15:41:11 +08:00
partition_stats_meta_test.go
enhance: Use v2 package name for pkg module (#39990)
2025-02-22 23:15:58 +08:00
partition_stats_meta.go
enhance: Use v2 package name for pkg module (#39990)
2025-02-22 23:15:58 +08:00
policy_test.go
fix: Fix channel not balance on datanodes (#40422)
2025-03-11 14:56:16 +08:00
policy.go
fix: Fix channel not balance on datanodes (#40422)
2025-03-11 14:56:16 +08:00
README.md
[skip ci]Change etcd to lowercase (#9983)
2021-10-15 18:58:37 +08:00
segment_allocation_policy_test.go
fix: Ignore growing segment without start pos for seal policy (#41130)
2025-04-07 22:16:23 +08:00
segment_allocation_policy.go
fix: Ignore growing segment without start pos for seal policy (#41130)
2025-04-07 22:16:23 +08:00
segment_info_test.go
enhance: Use v2 package name for pkg module (#39990)
2025-02-22 23:15:58 +08:00
segment_info.go
enhance: Move sort stats task to sort compaction (#42562)
2025-07-08 20:22:47 +08:00
segment_manager_test.go
feat: add NotifyDropPartition in mixcoord for droppartition in dc (#42029)
2025-05-23 18:32:26 +08:00
segment_manager.go
feat: add NotifyDropPartition in mixcoord for droppartition in dc (#42029)
2025-05-23 18:32:26 +08:00
segment_operator_test.go
enhance: Use v2 package name for pkg module (#39990)
2025-02-22 23:15:58 +08:00
segment_operator.go
enhance: Add json key inverted index in stats for optimization (#38039)
2025-04-10 15:20:28 +08:00
server_test.go
enhance: make immutable message as the param of ack operation for cdc (#43900)
2025-09-01 10:21:52 +08:00
server.go
enhance: make immutable message as the param of ack operation for cdc (#43900)
2025-09-01 10:21:52 +08:00
services_test.go
enhance: add file resource api (#43766)
2025-08-08 14:17:41 +08:00
services.go
enhance: add file resource api (#43766)
2025-08-08 14:17:41 +08:00
stats_inspector_test.go
enhance: Move sort stats task to sort compaction (#42562)
2025-07-08 20:22:47 +08:00
stats_inspector.go
enhance: support json stats with shredding design (#42534)
2025-09-01 10:49:52 +08:00
stats_task_meta_test.go
enhance: Move sort stats task to sort compaction (#42562)
2025-07-08 20:22:47 +08:00
stats_task_meta.go
enhance: Move sort stats task to sort compaction (#42562)
2025-07-08 20:22:47 +08:00
task_analyze_test.go
feat: Add CMEK cipher plugin (#43722)
2025-08-27 11:15:52 +08:00
task_analyze.go
feat: Add CMEK cipher plugin (#43722)
2025-08-27 11:15:52 +08:00
task_index_test.go
enhance: Pooling for data tasks (#41256)
2025-05-20 21:06:24 +08:00
task_index.go
feat: Add CMEK cipher plugin (#43722)
2025-08-27 11:15:52 +08:00
task_stats_test.go
enhance: Move sort stats task to sort compaction (#42562)
2025-07-08 20:22:47 +08:00
task_stats.go
enhance: support json stats with shredding design (#42534)
2025-09-01 10:49:52 +08:00
util_test.go
enhance: Use v2 package name for pkg module (#39990)
2025-02-22 23:15:58 +08:00
util.go
fix: make savebinlogpath idompotent at binlog level (#43615)
2025-07-29 19:47:36 +08:00

README.md

Data Coordinator

Data cooridnator(datacoord for short) is the component to organize DataNodes and segments allocations.

Dependency

  • KV store: a kv store has all the meta info datacoord needs to operate. (etcd)
  • Message stream: a message stream to communicate statistics information with data nodes. (Pulsar)
  • Root Coordinator: timestamp, id and meta source.
  • Data Node(s): could be an instance or a cluster, actual worker group handles data modification operations.