admin/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2025-12-28 22:45:26 +08:00
Code Issues Packages Projects Releases Wiki Activity
milvus/internal/datacoord
History
XuanYang-cn 74b4369c5b
fix: Record active collections for l0Policy (#39217) 
By recording the active collection lists, The l0 compaction trigger
of view change and idle won't influence each other.

Also this pr replaces the L0View cache with real L0 segments' change.
Save some memory and make L0 compaction triggers more accurate.

See also: #39187

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2025-01-16 20:19:02 +08:00
..
allocator
enhance: make new go package to manage proto (#39114)
2025-01-10 10:49:01 +08:00
broker
fix: fix make generate-mockery panic with go1.22 (#36830)
2024-10-17 12:11:31 +08:00
session
enhance: make new go package to manage proto (#39114)
2025-01-10 10:49:01 +08:00
analyze_meta_test.go
enhance: make new go package to manage proto (#39114)
2025-01-10 10:49:01 +08:00
analyze_meta.go
enhance: make new go package to manage proto (#39114)
2025-01-10 10:49:01 +08:00
build_index_policy.go
Format the code (#27275)
2023-09-21 09:45:27 +08:00
channel_manager_factory.go
fix: drop collection failed if enable streaming service (#37444)
2024-11-07 10:26:26 +08:00
channel_manager_test.go
enhance: make new go package to manage proto (#39114)
2025-01-10 10:49:01 +08:00
channel_manager.go
fix: Fix consume blocked due to too many consumers (#38455)
2025-01-15 21:37:01 +08:00
channel_store_test.go
enhance: make new go package to manage proto (#39114)
2025-01-10 10:49:01 +08:00
channel_store.go
enhance: make new go package to manage proto (#39114)
2025-01-10 10:49:01 +08:00
channel.go
enhance: make new go package to manage proto (#39114)
2025-01-10 10:49:01 +08:00
cluster_test.go
enhance: make new go package to manage proto (#39114)
2025-01-10 10:49:01 +08:00
cluster.go
enhance: make new go package to manage proto (#39114)
2025-01-10 10:49:01 +08:00
compaction_l0_view_test.go
enhance: make new go package to manage proto (#39114)
2025-01-10 10:49:01 +08:00
compaction_l0_view.go
enhance: Add deltaRowCount in l0 compaction (#33997)
2024-06-20 17:46:01 +08:00
compaction_policy_clustering_test.go
enhance: make new go package to manage proto (#39114)
2025-01-10 10:49:01 +08:00
compaction_policy_clustering.go
enhance: Reduce mutex contention in datacoord meta (#38219)
2025-01-15 01:15:02 +08:00
compaction_policy_l0_test.go
fix: Record active collections for l0Policy (#39217)
2025-01-16 20:19:02 +08:00
compaction_policy_l0.go
fix: Record active collections for l0Policy (#39217)
2025-01-16 20:19:02 +08:00
compaction_policy_single_test.go
enhance: Reduce mutex contention in datacoord meta (#38219)
2025-01-15 01:15:02 +08:00
compaction_policy_single.go
enhance: Reduce mutex contention in datacoord meta (#38219)
2025-01-15 01:15:02 +08:00
compaction_queue_test.go
enhance: make new go package to manage proto (#39114)
2025-01-10 10:49:01 +08:00
compaction_queue.go
enhance: make new go package to manage proto (#39114)
2025-01-10 10:49:01 +08:00
compaction_task_clustering_test.go
enhance: make new go package to manage proto (#39114)
2025-01-10 10:49:01 +08:00
compaction_task_clustering.go
enhance: simplify compaction tasks to reduce their memory overhead (#39121)
2025-01-15 14:51:00 +08:00
compaction_task_l0_test.go
enhance: simplify compaction tasks to reduce their memory overhead (#39121)
2025-01-15 14:51:00 +08:00
compaction_task_l0.go
enhance: simplify compaction tasks to reduce their memory overhead (#39121)
2025-01-15 14:51:00 +08:00
compaction_task_meta_test.go
enhance: make new go package to manage proto (#39114)
2025-01-10 10:49:01 +08:00
compaction_task_meta.go
enhance: make new go package to manage proto (#39114)
2025-01-10 10:49:01 +08:00
compaction_task_mix_test.go
enhance: make new go package to manage proto (#39114)
2025-01-10 10:49:01 +08:00
compaction_task_mix.go
enhance: simplify compaction tasks to reduce their memory overhead (#39121)
2025-01-15 14:51:00 +08:00
compaction_task.go
enhance: simplify compaction tasks to reduce their memory overhead (#39121)
2025-01-15 14:51:00 +08:00
compaction_test.go
fix: [skip e2e] Make test case assigner result deterministic (#39317)
2025-01-16 14:25:02 +08:00
compaction_trigger_test.go
enhance: Reduce mutex contention in datacoord meta (#38219)
2025-01-15 01:15:02 +08:00
compaction_trigger_v2_test.go
fix: Record active collections for l0Policy (#39217)
2025-01-16 20:19:02 +08:00
compaction_trigger_v2.go
fix: Record active collections for l0Policy (#39217)
2025-01-16 20:19:02 +08:00
compaction_trigger.go
enhance: Reduce mutex contention in datacoord meta (#38219)
2025-01-15 01:15:02 +08:00
compaction_view.go
enhance: make new go package to manage proto (#39114)
2025-01-10 10:49:01 +08:00
compaction.go
enhance: simplify compaction tasks to reduce their memory overhead (#39121)
2025-01-15 14:51:00 +08:00
const.go
enhance: pass partition key scalar info if enabled when build vector index (#29931)
2024-01-24 00:04:55 +08:00
errors_test.go
Format the code (#27275)
2023-09-21 09:45:27 +08:00
errors.go
Refine DataCoord status (#27262)
2023-09-26 17:15:27 +08:00
garbage_collector_test.go
enhance: record memory size (uncompressed) item for index (#38770)
2025-01-14 10:33:06 +08:00
garbage_collector.go
enhance: make new go package to manage proto (#39114)
2025-01-10 10:49:01 +08:00
go_channel_singleton.go
fix: Fix improper use of offset in HybridSearch (#36244)
2024-09-13 22:05:15 +08:00
handler_test.go
fix: Record a map to avoid repeatedly traversing the CompactionFrom (#38925)
2025-01-15 10:02:58 +08:00
handler.go
fix: Record a map to avoid repeatedly traversing the CompactionFrom (#38925)
2025-01-15 10:02:58 +08:00
import_checker_test.go
enhance: make new go package to manage proto (#39114)
2025-01-10 10:49:01 +08:00
import_checker.go
enhance: make new go package to manage proto (#39114)
2025-01-10 10:49:01 +08:00
import_job.go
enhance: make new go package to manage proto (#39114)
2025-01-10 10:49:01 +08:00
import_meta_test.go
enhance: make new go package to manage proto (#39114)
2025-01-10 10:49:01 +08:00
import_meta.go
enhance: refine the datacoord meta related interfaces (#37957)
2024-11-26 19:46:34 +08:00
import_scheduler_test.go
enhance: make new go package to manage proto (#39114)
2025-01-10 10:49:01 +08:00
import_scheduler.go
enhance: make new go package to manage proto (#39114)
2025-01-10 10:49:01 +08:00
import_task.go
enhance: make new go package to manage proto (#39114)
2025-01-10 10:49:01 +08:00
import_util_test.go
enhance: make new go package to manage proto (#39114)
2025-01-10 10:49:01 +08:00
import_util.go
enhance: make new go package to manage proto (#39114)
2025-01-10 10:49:01 +08:00
index_engine_version_manager_test.go
fix: Add scalar index engine version for compatibility (#39204)
2025-01-15 12:25:00 +08:00
index_engine_version_manager.go
fix: Add scalar index engine version for compatibility (#39204)
2025-01-15 12:25:00 +08:00
index_meta_test.go
enhance: record memory size (uncompressed) item for index (#38770)
2025-01-14 10:33:06 +08:00
index_meta.go
fix: Add scalar index engine version for compatibility (#39204)
2025-01-15 12:25:00 +08:00
index_service_test.go
fix: Add index param duplication check (#39289)
2025-01-15 19:23:01 +08:00
index_service.go
fix: Add index param duplication check (#39289)
2025-01-15 19:23:01 +08:00
job_manager_test.go
enhance: make new go package to manage proto (#39114)
2025-01-10 10:49:01 +08:00
job_manager.go
enhance: make new go package to manage proto (#39114)
2025-01-10 10:49:01 +08:00
knapsack_test.go
enhance: try compact small segments first if they may compose a full segment (#37709)
2024-12-02 16:12:38 +08:00
knapsack.go
enhance: improve mix compaction performance by removing max segment limitations (#38344)
2024-12-11 20:38:42 +08:00
meta_test.go
enhance: Reduce mutex contention in datacoord meta (#38219)
2025-01-15 01:15:02 +08:00
meta_util.go
enhance: make new go package to manage proto (#39114)
2025-01-10 10:49:01 +08:00
meta.go
enhance: Reduce mutex contention in datacoord meta (#38219)
2025-01-15 01:15:02 +08:00
metrics_info_test.go
enhance: record memory size (uncompressed) item for index (#38770)
2025-01-14 10:33:06 +08:00
metrics_info.go
enhance: make new go package to manage proto (#39114)
2025-01-10 10:49:01 +08:00
mock_channel_store.go
fix: ChannelManager concurret Release and Watch bug (#38590)
2024-12-19 22:50:47 +08:00
mock_channelmanager.go
enhance: make new go package to manage proto (#39114)
2025-01-10 10:49:01 +08:00
mock_cluster.go
enhance: make new go package to manage proto (#39114)
2025-01-10 10:49:01 +08:00
mock_compaction_meta.go
enhance: make new go package to manage proto (#39114)
2025-01-10 10:49:01 +08:00
mock_compaction_plan_context.go
enhance: make new go package to manage proto (#39114)
2025-01-10 10:49:01 +08:00
mock_handler.go
enhance: make new go package to manage proto (#39114)
2025-01-10 10:49:01 +08:00
mock_index_engine_version_manager.go
fix: Add scalar index engine version for compatibility (#39204)
2025-01-15 12:25:00 +08:00
mock_job_manager.go
enhance: make new go package to manage proto (#39114)
2025-01-10 10:49:01 +08:00
mock_segment_manager.go
enhance: Reduce segmentManager lock granularity (#37836)
2024-12-17 14:12:52 +08:00
mock_subcluster.go
enhance: make new go package to manage proto (#39114)
2025-01-10 10:49:01 +08:00
mock_test.go
enhance: make new go package to manage proto (#39114)
2025-01-10 10:49:01 +08:00
mock_trigger_manager.go
fix: fix make generate-mockery panic with go1.22 (#36830)
2024-10-17 12:11:31 +08:00
OWNERS
[skip ci]Update OWNERS files (#11898)
2021-11-16 15:41:11 +08:00
partition_stats_meta_test.go
enhance: make new go package to manage proto (#39114)
2025-01-10 10:49:01 +08:00
partition_stats_meta.go
enhance: make new go package to manage proto (#39114)
2025-01-10 10:49:01 +08:00
policy_test.go
enhance: make new go package to manage proto (#39114)
2025-01-10 10:49:01 +08:00
policy.go
fix: drop collection failed if enable streaming service (#37444)
2024-11-07 10:26:26 +08:00
README.md
[skip ci]Change etcd to lowercase (#9983)
2021-10-15 18:58:37 +08:00
segment_allocation_policy_test.go
enhance: make new go package to manage proto (#39114)
2025-01-10 10:49:01 +08:00
segment_allocation_policy.go
enhance: make new go package to manage proto (#39114)
2025-01-10 10:49:01 +08:00
segment_info_test.go
enhance: make new go package to manage proto (#39114)
2025-01-10 10:49:01 +08:00
segment_info.go
enhance: Reduce mutex contention in datacoord meta (#38219)
2025-01-15 01:15:02 +08:00
segment_manager_test.go
enhance: make new go package to manage proto (#39114)
2025-01-10 10:49:01 +08:00
segment_manager.go
enhance: make new go package to manage proto (#39114)
2025-01-10 10:49:01 +08:00
segment_operator_test.go
enhance: make new go package to manage proto (#39114)
2025-01-10 10:49:01 +08:00
segment_operator.go
enhance: make new go package to manage proto (#39114)
2025-01-10 10:49:01 +08:00
server_test.go
enhance: record memory size (uncompressed) item for index (#38770)
2025-01-14 10:33:06 +08:00
server.go
enhance: Add logs for check health failed (#39208)
2025-01-15 17:31:00 +08:00
services_test.go
enhance: record memory size (uncompressed) item for index (#38770)
2025-01-14 10:33:06 +08:00
services.go
fix: Record active collections for l0Policy (#39217)
2025-01-16 20:19:02 +08:00
stats_task_meta_test.go
enhance: make new go package to manage proto (#39114)
2025-01-10 10:49:01 +08:00
stats_task_meta.go
enhance: make new go package to manage proto (#39114)
2025-01-10 10:49:01 +08:00
sync_segments_scheduler_test.go
enhance: make new go package to manage proto (#39114)
2025-01-10 10:49:01 +08:00
sync_segments_scheduler.go
enhance: make new go package to manage proto (#39114)
2025-01-10 10:49:01 +08:00
task_analyze.go
enhance: make new go package to manage proto (#39114)
2025-01-10 10:49:01 +08:00
task_index.go
fix: Add scalar index engine version for compatibility (#39204)
2025-01-15 12:25:00 +08:00
task_scheduler_test.go
enhance: record memory size (uncompressed) item for index (#38770)
2025-01-14 10:33:06 +08:00
task_scheduler.go
enhance: make new go package to manage proto (#39114)
2025-01-10 10:49:01 +08:00
task_stats_test.go
enhance: make new go package to manage proto (#39114)
2025-01-10 10:49:01 +08:00
task_stats.go
fix: Record active collections for l0Policy (#39217)
2025-01-16 20:19:02 +08:00
types.go
enhance: make new go package to manage proto (#39114)
2025-01-10 10:49:01 +08:00
util_test.go
enhance: make new go package to manage proto (#39114)
2025-01-10 10:49:01 +08:00
util.go
enhance: make new go package to manage proto (#39114)
2025-01-10 10:49:01 +08:00

README.md

Data Coordinator

Data cooridnator(datacoord for short) is the component to organize DataNodes and segments allocations.

Dependency

  • KV store: a kv store has all the meta info datacoord needs to operate. (etcd)
  • Message stream: a message stream to communicate statistics information with data nodes. (Pulsar)
  • Root Coordinator: timestamp, id and meta source.
  • Data Node(s): could be an instance or a cluster, actual worker group handles data modification operations.