mirror of https://gitee.com/milvus-io/milvus.git synced 2025-12-31 16:05:30 +08:00

History

enhance: expose more metrics data (#39456 )

issue: #36621 #39417
1. Adjust the server-side cache size.
2. Add source information for configurations.
3. Add node ID for compaction and indexing tasks.
4. Resolve localhost access issues to fix health check failures for
etcd.

Signed-off-by: jaime <yun.zhang@zilliz.com>

2025-02-07 11:50:50 +08:00

allocator

enhance: make new go package to manage proto (#39114 )

2025-01-10 10:49:01 +08:00

broker

enhance: Use mockery pkg config for datacoord&datanode (#39567 )

2025-01-24 14:25:06 +08:00

session

enhance: Use mockery pkg config for datacoord&datanode (#39567 )

2025-01-24 14:25:06 +08:00

.mockery.yaml

enhance: Use mockery pkg config for datacoord&datanode (#39567 )

2025-01-24 14:25:06 +08:00

analyze_meta_test.go

enhance: make new go package to manage proto (#39114 )

2025-01-10 10:49:01 +08:00

analyze_meta.go

enhance: make new go package to manage proto (#39114 )

2025-01-10 10:49:01 +08:00

build_index_policy.go

Format the code (#27275 )

2023-09-21 09:45:27 +08:00

channel_manager_factory.go

fix: drop collection failed if enable streaming service (#37444 )

2024-11-07 10:26:26 +08:00

channel_manager_test.go

enhance: make new go package to manage proto (#39114 )

2025-01-10 10:49:01 +08:00

channel_manager.go

fix: Fix consume blocked due to too many consumers (#38455 )

2025-01-15 21:37:01 +08:00

channel_store_test.go

enhance: make new go package to manage proto (#39114 )

2025-01-10 10:49:01 +08:00

channel_store.go

enhance: make new go package to manage proto (#39114 )

2025-01-10 10:49:01 +08:00

channel.go

enhance: make new go package to manage proto (#39114 )

2025-01-10 10:49:01 +08:00

cluster_test.go

enhance: make new go package to manage proto (#39114 )

2025-01-10 10:49:01 +08:00

cluster.go

enhance: make new go package to manage proto (#39114 )

2025-01-10 10:49:01 +08:00

compaction_l0_view_test.go

enhance: make new go package to manage proto (#39114 )

2025-01-10 10:49:01 +08:00

compaction_l0_view.go

enhance: Add deltaRowCount in l0 compaction (#33997 )

2024-06-20 17:46:01 +08:00

compaction_policy_clustering_test.go

enhance: make new go package to manage proto (#39114 )

2025-01-10 10:49:01 +08:00

compaction_policy_clustering.go

enhance: Reduce mutex contention in datacoord meta (#38219 )

2025-01-15 01:15:02 +08:00

compaction_policy_l0_test.go

fix: Record active collections for l0Policy (#39217 )

2025-01-16 20:19:02 +08:00

compaction_policy_l0.go

fix: Record active collections for l0Policy (#39217 )

2025-01-16 20:19:02 +08:00

compaction_policy_single_test.go

enhance: Reduce mutex contention in datacoord meta (#38219 )

2025-01-15 01:15:02 +08:00

compaction_policy_single.go

enhance: Reduce mutex contention in datacoord meta (#38219 )

2025-01-15 01:15:02 +08:00

compaction_queue_test.go

enhance: make new go package to manage proto (#39114 )

2025-01-10 10:49:01 +08:00

compaction_queue.go

enhance: make new go package to manage proto (#39114 )

2025-01-10 10:49:01 +08:00

compaction_task_clustering_test.go

fix: Restore the compacting state for stats task during recovery (#39459 )

2025-02-05 17:11:12 +08:00

compaction_task_clustering.go

fix: Restore the compacting state for stats task during recovery (#39459 )

2025-02-05 17:11:12 +08:00

compaction_task_l0_test.go

enhance: simplify compaction tasks to reduce their memory overhead (#39121 )

2025-01-15 14:51:00 +08:00

compaction_task_l0.go

fix: Restore the compacting state for stats task during recovery (#39459 )

2025-02-05 17:11:12 +08:00

compaction_task_meta_test.go

enhance: make new go package to manage proto (#39114 )

2025-01-10 10:49:01 +08:00

compaction_task_meta.go

enhance: expose more metrics data (#39456 )

2025-02-07 11:50:50 +08:00

compaction_task_mix_test.go

enhance: make new go package to manage proto (#39114 )

2025-01-10 10:49:01 +08:00

compaction_task_mix.go

fix: Restore the compacting state for stats task during recovery (#39459 )

2025-02-05 17:11:12 +08:00

compaction_task.go

fix: Restore the compacting state for stats task during recovery (#39459 )

2025-02-05 17:11:12 +08:00

compaction_test.go

fix: Restore the compacting state for stats task during recovery (#39459 )

2025-02-05 17:11:12 +08:00

compaction_trigger_test.go

fix: Restore the compacting state for stats task during recovery (#39459 )

2025-02-05 17:11:12 +08:00

compaction_trigger_v2_test.go

fix: Record active collections for l0Policy (#39217 )

2025-01-16 20:19:02 +08:00

compaction_trigger_v2.go

fix: Record active collections for l0Policy (#39217 )

2025-01-16 20:19:02 +08:00

compaction_trigger.go

enhance: Reduce mutex contention in datacoord meta (#38219 )

2025-01-15 01:15:02 +08:00

compaction_view.go

enhance: make new go package to manage proto (#39114 )

2025-01-10 10:49:01 +08:00

compaction.go

fix: Restore the compacting state for stats task during recovery (#39459 )

2025-02-05 17:11:12 +08:00

const.go

enhance: pass partition key scalar info if enabled when build vector index (#29931 )

2024-01-24 00:04:55 +08:00

errors_test.go

Format the code (#27275 )

2023-09-21 09:45:27 +08:00

errors.go

Refine DataCoord status (#27262 )

2023-09-26 17:15:27 +08:00

garbage_collector_test.go

enhance: Read metadata concurrently to accelerate recovery (#38403 )

2025-01-23 14:27:27 +08:00

garbage_collector.go

enhance: make new go package to manage proto (#39114 )

2025-01-10 10:49:01 +08:00

go_channel_singleton.go

fix: Fix improper use of offset in HybridSearch (#36244 )

2024-09-13 22:05:15 +08:00

handler_test.go

fix: Record a map to avoid repeatedly traversing the CompactionFrom (#38925 )

2025-01-15 10:02:58 +08:00

handler.go

fix: Record a map to avoid repeatedly traversing the CompactionFrom (#38925 )

2025-01-15 10:02:58 +08:00

import_checker_test.go

enhance: Remove unnecessary collection and partition label from the metrics (#39536 )

2025-02-05 11:01:10 +08:00

import_checker.go

enhance: Remove unnecessary collection and partition label from the metrics (#39536 )

2025-02-05 11:01:10 +08:00

import_job.go

enhance: make new go package to manage proto (#39114 )

2025-01-10 10:49:01 +08:00

import_meta_test.go

enhance: make new go package to manage proto (#39114 )

2025-01-10 10:49:01 +08:00

import_meta.go

enhance: expose more metrics data (#39456 )

2025-02-07 11:50:50 +08:00

import_scheduler_test.go

enhance: Read metadata concurrently to accelerate recovery (#38403 )

2025-01-23 14:27:27 +08:00

import_scheduler.go

enhance: Limit number of segments restored and promptly terminate the job (#39344 )

2025-01-21 15:13:13 +08:00

import_task.go

enhance: make new go package to manage proto (#39114 )

2025-01-10 10:49:01 +08:00

import_util_test.go

enhance: Read metadata concurrently to accelerate recovery (#38403 )

2025-01-23 14:27:27 +08:00

import_util.go

enhance: make new go package to manage proto (#39114 )

2025-01-10 10:49:01 +08:00

index_engine_version_manager_test.go

fix: Add scalar index engine version for compatibility (#39204 )

2025-01-15 12:25:00 +08:00

index_engine_version_manager.go

fix: Add scalar index engine version for compatibility (#39204 )

2025-01-15 12:25:00 +08:00

index_meta_test.go

enhance: record memory size (uncompressed) item for index (#38770 )

2025-01-14 10:33:06 +08:00

index_meta.go

enhance: expose more metrics data (#39456 )

2025-02-07 11:50:50 +08:00

index_service_test.go

fix: deny to set the mmap param for the alter index api when enable auto index (#39518 )

2025-01-23 10:01:04 +08:00

index_service.go

enhance: Refine error msg for schema & index checking (#39533 )

2025-01-24 09:57:06 +08:00

job_manager_test.go

enhance: expose more metrics data (#39456 )

2025-02-07 11:50:50 +08:00

job_manager.go

fix: Restore the compacting state for stats task during recovery (#39459 )

2025-02-05 17:11:12 +08:00

knapsack_test.go

enhance: try compact small segments first if they may compose a full segment (#37709 )

2024-12-02 16:12:38 +08:00

knapsack.go

enhance: improve mix compaction performance by removing max segment limitations (#38344 )

2024-12-11 20:38:42 +08:00

meta_test.go

enhance: Read metadata concurrently to accelerate recovery (#38403 )

2025-01-23 14:27:27 +08:00

meta_util.go

enhance: make new go package to manage proto (#39114 )

2025-01-10 10:49:01 +08:00

meta.go

fix: Restore the compacting state for stats task during recovery (#39459 )

2025-02-05 17:11:12 +08:00

metrics_info_test.go

enhance: record memory size (uncompressed) item for index (#38770 )

2025-01-14 10:33:06 +08:00

metrics_info.go

enhance: make new go package to manage proto (#39114 )

2025-01-10 10:49:01 +08:00

mock_channel_manager.go

enhance: Use mockery pkg config for datacoord&datanode (#39567 )

2025-01-24 14:25:06 +08:00

mock_channel_store.go

fix: ChannelManager concurret Release and Watch bug (#38590 )

2024-12-19 22:50:47 +08:00

mock_cluster.go

enhance: make new go package to manage proto (#39114 )

2025-01-10 10:49:01 +08:00

mock_compaction_meta.go

fix: Restore the compacting state for stats task during recovery (#39459 )

2025-02-05 17:11:12 +08:00

mock_compaction_plan_context.go

fix: Restore the compacting state for stats task during recovery (#39459 )

2025-02-05 17:11:12 +08:00

mock_handler.go

enhance: make new go package to manage proto (#39114 )

2025-01-10 10:49:01 +08:00

mock_index_engine_version_manager.go

fix: Add scalar index engine version for compatibility (#39204 )

2025-01-15 12:25:00 +08:00

mock_segment_manager.go

fix: deleted the sealed segment data accidentally (#39421 )

2025-01-20 15:29:02 +08:00

mock_stats_job_manager.go

enhance: Use mockery pkg config for datacoord&datanode (#39567 )

2025-01-24 14:25:06 +08:00

mock_sub_cluster.go

enhance: Use mockery pkg config for datacoord&datanode (#39567 )

2025-01-24 14:25:06 +08:00

mock_test.go

enhance: Read metadata concurrently to accelerate recovery (#38403 )

2025-01-23 14:27:27 +08:00

mock_trigger_manager.go

fix: deleted the sealed segment data accidentally (#39421 )

2025-01-20 15:29:02 +08:00

OWNERS

[skip ci]Update OWNERS files (#11898 )

2021-11-16 15:41:11 +08:00

partition_stats_meta_test.go

enhance: make new go package to manage proto (#39114 )

2025-01-10 10:49:01 +08:00

partition_stats_meta.go

enhance: make new go package to manage proto (#39114 )

2025-01-10 10:49:01 +08:00

policy_test.go

enhance: make new go package to manage proto (#39114 )

2025-01-10 10:49:01 +08:00

policy.go

fix: drop collection failed if enable streaming service (#37444 )

2024-11-07 10:26:26 +08:00

README.md

[skip ci]Change etcd to lowercase (#9983 )

2021-10-15 18:58:37 +08:00

segment_allocation_policy_test.go

enhance: make new go package to manage proto (#39114 )

2025-01-10 10:49:01 +08:00

segment_allocation_policy.go

enhance: make new go package to manage proto (#39114 )

2025-01-10 10:49:01 +08:00

segment_info_test.go

enhance: make new go package to manage proto (#39114 )

2025-01-10 10:49:01 +08:00

segment_info.go

fix: Restore the compacting state for stats task during recovery (#39459 )

2025-02-05 17:11:12 +08:00

segment_manager_test.go

enhance: Read metadata concurrently to accelerate recovery (#38403 )

2025-01-23 14:27:27 +08:00

segment_manager.go

fix: deleted the sealed segment data accidentally (#39421 )

2025-01-20 15:29:02 +08:00

segment_operator_test.go

enhance: make new go package to manage proto (#39114 )

2025-01-10 10:49:01 +08:00

segment_operator.go

enhance: make new go package to manage proto (#39114 )

2025-01-10 10:49:01 +08:00

server_test.go

fix: Restore the compacting state for stats task during recovery (#39459 )

2025-02-05 17:11:12 +08:00

server.go

fix: Restore the compacting state for stats task during recovery (#39459 )

2025-02-05 17:11:12 +08:00

services_test.go

enhance: record memory size (uncompressed) item for index (#38770 )

2025-01-14 10:33:06 +08:00

services.go

enhance: Limit number of segments restored and promptly terminate the job (#39344 )

2025-01-21 15:13:13 +08:00

stats_task_meta_test.go

enhance: make new go package to manage proto (#39114 )

2025-01-10 10:49:01 +08:00

stats_task_meta.go

fix: Restore the compacting state for stats task during recovery (#39459 )

2025-02-05 17:11:12 +08:00

sync_segments_scheduler_test.go

enhance: make new go package to manage proto (#39114 )

2025-01-10 10:49:01 +08:00

sync_segments_scheduler.go

enhance: make new go package to manage proto (#39114 )

2025-01-10 10:49:01 +08:00

task_analyze.go

fix: Restore the compacting state for stats task during recovery (#39459 )

2025-02-05 17:11:12 +08:00

task_index.go

fix: Restore the compacting state for stats task during recovery (#39459 )

2025-02-05 17:11:12 +08:00

task_scheduler_test.go

fix: Restore the compacting state for stats task during recovery (#39459 )

2025-02-05 17:11:12 +08:00

task_scheduler.go

enhance: expose more metrics data (#39456 )

2025-02-07 11:50:50 +08:00

task_stats_test.go

fix: Restore the compacting state for stats task during recovery (#39459 )

2025-02-05 17:11:12 +08:00

task_stats.go

enhance: expose more metrics data (#39456 )

2025-02-07 11:50:50 +08:00

types.go

fix: Restore the compacting state for stats task during recovery (#39459 )

2025-02-05 17:11:12 +08:00

util_test.go

enhance: make new go package to manage proto (#39114 )

2025-01-10 10:49:01 +08:00

util.go

enhance: make new go package to manage proto (#39114 )

2025-01-10 10:49:01 +08:00

README.md

Data Coordinator

Data cooridnator(datacoord for short) is the component to organize DataNodes and segments allocations.

Dependency

KV store: a kv store has all the meta info datacoord needs to operate. (etcd)
Message stream: a message stream to communicate statistics information with data nodes. (Pulsar)
Root Coordinator: timestamp, id and meta source.
Data Node(s): could be an instance or a cluster, actual worker group handles data modification operations.