mirror of https://gitee.com/milvus-io/milvus.git synced 2026-02-02 01:06:41 +08:00

History

enhance: expose more metrics data (#39466 )

issue: #36621 #39417
pr: #39456
1. Adjust the server-side cache size.
2. Add source information for configurations.
3. Add node ID for compaction and indexing tasks.
4. Resolve localhost access issues to fix health check failures for
etcd.

Signed-off-by: jaime <yun.zhang@zilliz.com>

2025-02-07 11:48:45 +08:00

allocator

enhance: make new go package to manage proto (#39128 )

2025-01-10 10:53:01 +08:00

broker

enhance: [2.5] Use mockery pkg config for datacoord&datanode (#39567 ) (#39577 )

2025-01-24 17:21:13 +08:00

session

enhance: [2.5] Use mockery pkg config for datacoord&datanode (#39567 ) (#39577 )

2025-01-24 17:21:13 +08:00

.mockery.yaml

enhance: [2.5] Use mockery pkg config for datacoord&datanode (#39567 ) (#39577 )

2025-01-24 17:21:13 +08:00

analyze_meta_test.go

enhance: make new go package to manage proto (#39128 )

2025-01-10 10:53:01 +08:00

analyze_meta.go

enhance: make new go package to manage proto (#39128 )

2025-01-10 10:53:01 +08:00

build_index_policy.go

Format the code (#27275 )

2023-09-21 09:45:27 +08:00

channel_manager_factory.go

fix: drop collection failed if enable streaming service (#37444 )

2024-11-07 10:26:26 +08:00

channel_manager_test.go

enhance: make new go package to manage proto (#39128 )

2025-01-10 10:53:01 +08:00

channel_manager.go

fix: [2.5] Fix consume blocked due to too many consumers (#38915 )

2025-01-16 15:19:03 +08:00

channel_store_test.go

enhance: make new go package to manage proto (#39128 )

2025-01-10 10:53:01 +08:00

channel_store.go

enhance: make new go package to manage proto (#39128 )

2025-01-10 10:53:01 +08:00

channel.go

enhance: make new go package to manage proto (#39128 )

2025-01-10 10:53:01 +08:00

cluster_test.go

enhance: make new go package to manage proto (#39128 )

2025-01-10 10:53:01 +08:00

cluster.go

enhance: make new go package to manage proto (#39128 )

2025-01-10 10:53:01 +08:00

compaction_l0_view_test.go

enhance: make new go package to manage proto (#39128 )

2025-01-10 10:53:01 +08:00

compaction_l0_view.go

enhance: Add deltaRowCount in l0 compaction (#33997 )

2024-06-20 17:46:01 +08:00

compaction_policy_clustering_test.go

enhance: make new go package to manage proto (#39128 )

2025-01-10 10:53:01 +08:00

compaction_policy_clustering.go

enhance: [2.5] Reduce mutex contention in datacoord meta (#38904 )

2025-01-15 00:55:00 +08:00

compaction_policy_l0_test.go

fix: [cp25]Record active collections for l0Policy (#39217 ) (#39383 )

2025-01-20 10:47:03 +08:00

compaction_policy_l0.go

fix: [cp25]Record active collections for l0Policy (#39217 ) (#39383 )

2025-01-20 10:47:03 +08:00

compaction_policy_single_test.go

enhance: [2.5] Reduce mutex contention in datacoord meta (#38904 )

2025-01-15 00:55:00 +08:00

compaction_policy_single.go

enhance: [2.5] Reduce mutex contention in datacoord meta (#38904 )

2025-01-15 00:55:00 +08:00

compaction_queue_test.go

enhance: make new go package to manage proto (#39128 )

2025-01-10 10:53:01 +08:00

compaction_queue.go

enhance: make new go package to manage proto (#39128 )

2025-01-10 10:53:01 +08:00

compaction_task_clustering_test.go

fix: [2.5]Set isStating to ensuer mutual exclusive between L0 compacting and stats (#39490 )

2025-01-22 10:27:05 +08:00

compaction_task_clustering.go

fix: [2.5]Set isStating to ensuer mutual exclusive between L0 compacting and stats (#39490 )

2025-01-22 10:27:05 +08:00

compaction_task_l0_test.go

enhance: make new go package to manage proto (#39128 )

2025-01-10 10:53:01 +08:00

compaction_task_l0.go

fix: [2.5]Set isStating to ensuer mutual exclusive between L0 compacting and stats (#39490 )

2025-01-22 10:27:05 +08:00

compaction_task_meta_test.go

enhance: make new go package to manage proto (#39128 )

2025-01-10 10:53:01 +08:00

compaction_task_meta.go

enhance: expose more metrics data (#39466 )

2025-02-07 11:48:45 +08:00

compaction_task_mix_test.go

enhance: make new go package to manage proto (#39128 )

2025-01-10 10:53:01 +08:00

compaction_task_mix.go

fix: [2.5]Set isStating to ensuer mutual exclusive between L0 compacting and stats (#39490 )

2025-01-22 10:27:05 +08:00

compaction_task.go

fix: [2.5]Set isStating to ensuer mutual exclusive between L0 compacting and stats (#39490 )

2025-01-22 10:27:05 +08:00

compaction_test.go

fix: [2.5]Set isStating to ensuer mutual exclusive between L0 compacting and stats (#39490 )

2025-01-22 10:27:05 +08:00

compaction_trigger_test.go

enhance: [2.5]Only check L0 compaction with same channel when stating (#39543 )

2025-02-05 17:17:11 +08:00

compaction_trigger_v2_test.go

fix: [cp25]Record active collections for l0Policy (#39217 ) (#39383 )

2025-01-20 10:47:03 +08:00

compaction_trigger_v2.go

fix: [cp25]Record active collections for l0Policy (#39217 ) (#39383 )

2025-01-20 10:47:03 +08:00

compaction_trigger.go

enhance: [2.5] Reduce mutex contention in datacoord meta (#38904 )

2025-01-15 00:55:00 +08:00

compaction_view.go

enhance: make new go package to manage proto (#39128 )

2025-01-10 10:53:01 +08:00

compaction.go

enhance: [2.5]Only check L0 compaction with same channel when stating (#39543 )

2025-02-05 17:17:11 +08:00

const.go

enhance: pass partition key scalar info if enabled when build vector index (#29931 )

2024-01-24 00:04:55 +08:00

errors_test.go

Format the code (#27275 )

2023-09-21 09:45:27 +08:00

errors.go

Refine DataCoord status (#27262 )

2023-09-26 17:15:27 +08:00

garbage_collector_test.go

enhance: [2.5] Read metadata concurrently to accelerate recovery (#38900 )

2025-01-16 17:53:01 +08:00

garbage_collector.go

enhance: make new go package to manage proto (#39128 )

2025-01-10 10:53:01 +08:00

go_channel_singleton.go

fix: Fix improper use of offset in HybridSearch (#36244 )

2024-09-13 22:05:15 +08:00

handler_test.go

fix: [2.5] Record a map to avoid repeatedly traversing the CompactionFrom (#38926 )

2025-01-15 10:31:00 +08:00

handler.go

fix: [2.5] Record a map to avoid repeatedly traversing the CompactionFrom (#38926 )

2025-01-15 10:31:00 +08:00

import_checker_test.go

enhance: [2.5] Limit the maximum number of segments restored and fail the job if saving the binlog fails (#39359 )

2025-01-17 10:27:04 +08:00

import_checker.go

enhance: [2.5] Limit the maximum number of segments restored and fail the job if saving the binlog fails (#39359 )

2025-01-17 10:27:04 +08:00

import_job.go

enhance: make new go package to manage proto (#39128 )

2025-01-10 10:53:01 +08:00

import_meta_test.go

enhance: make new go package to manage proto (#39128 )

2025-01-10 10:53:01 +08:00

import_meta.go

enhance: expose more metrics data (#39466 )

2025-02-07 11:48:45 +08:00

import_scheduler_test.go

enhance: [2.5] Read metadata concurrently to accelerate recovery (#38900 )

2025-01-16 17:53:01 +08:00

import_scheduler.go

enhance: [2.5] Limit the maximum number of segments restored and fail the job if saving the binlog fails (#39359 )

2025-01-17 10:27:04 +08:00

import_task.go

enhance: make new go package to manage proto (#39128 )

2025-01-10 10:53:01 +08:00

import_util_test.go

enhance: [2.5] Read metadata concurrently to accelerate recovery (#38900 )

2025-01-16 17:53:01 +08:00

import_util.go

enhance: make new go package to manage proto (#39128 )

2025-01-10 10:53:01 +08:00

index_engine_version_manager_test.go

fix: [2.5] Add scalar index engine version for compatibility (#39236 )

2025-01-14 21:01:01 +08:00

index_engine_version_manager.go

fix: [2.5] Add scalar index engine version for compatibility (#39236 )

2025-01-14 21:01:01 +08:00

index_meta_test.go

enhance: record memory size (uncompressed) item for index (#38844 )

2025-01-14 10:33:06 +08:00

index_meta.go

enhance: expose more metrics data (#39466 )

2025-02-07 11:48:45 +08:00

index_service_test.go

fix: [2.5] deny to set the mmap param for the alter index api (#39520 )

2025-01-22 23:55:06 +08:00

index_service.go

enhance: [2.5] Refine error msg for schema & index checking (#39533 ) (#39565 )

2025-01-24 13:43:06 +08:00

job_manager_test.go

enhance: expose more metrics data (#39466 )

2025-02-07 11:48:45 +08:00

job_manager.go

fix: [2.5]Restore the compacting state for stats task during recovery (#39460 )

2025-01-21 01:03:05 +08:00

knapsack_test.go

enhance: try compact small segments first if they may compose a full segment (#37709 )

2024-12-02 16:12:38 +08:00

knapsack.go

enhance: improve mix compaction performance by removing max segment limitations (#38344 )

2024-12-11 20:38:42 +08:00

meta_test.go

enhance: [2.5] Read metadata concurrently to accelerate recovery (#38900 )

2025-01-16 17:53:01 +08:00

meta_util.go

enhance: make new go package to manage proto (#39128 )

2025-01-10 10:53:01 +08:00

meta.go

fix: [2.5] Set the stating state correctly (#39514 )

2025-01-22 18:38:29 +08:00

metrics_info_test.go

enhance: record memory size (uncompressed) item for index (#38844 )

2025-01-14 10:33:06 +08:00

metrics_info.go

enhance: make new go package to manage proto (#39128 )

2025-01-10 10:53:01 +08:00

mock_channel_manager.go

enhance: [2.5] Use mockery pkg config for datacoord&datanode (#39567 ) (#39577 )

2025-01-24 17:21:13 +08:00

mock_channel_store.go

fix: ChannelManager concurret Release and Watch bug (#38590 )

2024-12-19 22:50:47 +08:00

mock_cluster.go

enhance: make new go package to manage proto (#39128 )

2025-01-10 10:53:01 +08:00

mock_compaction_meta.go

fix: [2.5]Set isStating to ensuer mutual exclusive between L0 compacting and stats (#39490 )

2025-01-22 10:27:05 +08:00

mock_compaction_plan_context.go

enhance: [2.5]Only check L0 compaction with same channel when stating (#39543 )

2025-02-05 17:17:11 +08:00

mock_handler.go

enhance: make new go package to manage proto (#39128 )

2025-01-10 10:53:01 +08:00

mock_index_engine_version_manager.go

fix: [2.5] Add scalar index engine version for compatibility (#39236 )

2025-01-14 21:01:01 +08:00

mock_segment_manager.go

fix: deleted the sealed segment data accidentally (#39422 )

2025-01-20 17:49:03 +08:00

mock_stats_job_manager.go

enhance: [2.5] Use mockery pkg config for datacoord&datanode (#39567 ) (#39577 )

2025-01-24 17:21:13 +08:00

mock_sub_cluster.go

enhance: [2.5] Use mockery pkg config for datacoord&datanode (#39567 ) (#39577 )

2025-01-24 17:21:13 +08:00

mock_test.go

enhance: [2.5] Read metadata concurrently to accelerate recovery (#38900 )

2025-01-16 17:53:01 +08:00

mock_trigger_manager.go

fix: deleted the sealed segment data accidentally (#39422 )

2025-01-20 17:49:03 +08:00

OWNERS

[skip ci]Update OWNERS files (#11898 )

2021-11-16 15:41:11 +08:00

partition_stats_meta_test.go

enhance: make new go package to manage proto (#39128 )

2025-01-10 10:53:01 +08:00

partition_stats_meta.go

enhance: make new go package to manage proto (#39128 )

2025-01-10 10:53:01 +08:00

policy_test.go

enhance: make new go package to manage proto (#39128 )

2025-01-10 10:53:01 +08:00

policy.go

fix: drop collection failed if enable streaming service (#37444 )

2024-11-07 10:26:26 +08:00

README.md

[skip ci]Change etcd to lowercase (#9983 )

2021-10-15 18:58:37 +08:00

segment_allocation_policy_test.go

enhance: make new go package to manage proto (#39128 )

2025-01-10 10:53:01 +08:00

segment_allocation_policy.go

enhance: make new go package to manage proto (#39128 )

2025-01-10 10:53:01 +08:00

segment_info_test.go

enhance: make new go package to manage proto (#39128 )

2025-01-10 10:53:01 +08:00

segment_info.go

fix: [2.5]Set isStating to ensuer mutual exclusive between L0 compacting and stats (#39490 )

2025-01-22 10:27:05 +08:00

segment_manager_test.go

fix: deleted the sealed segment data accidentally (#39422 )

2025-01-20 17:49:03 +08:00

segment_manager.go

fix: deleted the sealed segment data accidentally (#39422 )

2025-01-20 17:49:03 +08:00

segment_operator_test.go

enhance: make new go package to manage proto (#39128 )

2025-01-10 10:53:01 +08:00

segment_operator.go

enhance: make new go package to manage proto (#39128 )

2025-01-10 10:53:01 +08:00

server_test.go

fix: [2.5]Set isStating to ensuer mutual exclusive between L0 compacting and stats (#39490 )

2025-01-22 10:27:05 +08:00

server.go

fix: [2.5]Set isStating to ensuer mutual exclusive between L0 compacting and stats (#39490 )

2025-01-22 10:27:05 +08:00

services_test.go

enhance: record memory size (uncompressed) item for index (#38844 )

2025-01-14 10:33:06 +08:00

services.go

fix: deleted the sealed segment data accidentally (#39422 )

2025-01-20 17:49:03 +08:00

stats_task_meta_test.go

enhance: make new go package to manage proto (#39128 )

2025-01-10 10:53:01 +08:00

stats_task_meta.go

fix: [2.5]Restore the compacting state for stats task during recovery (#39460 )

2025-01-21 01:03:05 +08:00

sync_segments_scheduler_test.go

enhance: make new go package to manage proto (#39128 )

2025-01-10 10:53:01 +08:00

sync_segments_scheduler.go

enhance: make new go package to manage proto (#39128 )

2025-01-10 10:53:01 +08:00

task_analyze.go

fix: [2.5] Set deltalogs for stats task after set segment stating (#39502 )

2025-01-22 16:29:06 +08:00

task_index.go

fix: [2.5] Set deltalogs for stats task after set segment stating (#39502 )

2025-01-22 16:29:06 +08:00

task_scheduler_test.go

enhance: [2.5]Only check L0 compaction with same channel when stating (#39543 )

2025-02-05 17:17:11 +08:00

task_scheduler.go

enhance: expose more metrics data (#39466 )

2025-02-07 11:48:45 +08:00

task_stats_test.go

enhance: [2.5]Only check L0 compaction with same channel when stating (#39543 )

2025-02-05 17:17:11 +08:00

task_stats.go

enhance: [2.5]Only check L0 compaction with same channel when stating (#39543 )

2025-02-05 17:17:11 +08:00

types.go

fix: [2.5] Set deltalogs for stats task after set segment stating (#39502 )

2025-01-22 16:29:06 +08:00

util_test.go

enhance: make new go package to manage proto (#39128 )

2025-01-10 10:53:01 +08:00

util.go

enhance: make new go package to manage proto (#39128 )

2025-01-10 10:53:01 +08:00

README.md

Data Coordinator

Data cooridnator(datacoord for short) is the component to organize DataNodes and segments allocations.

Dependency

KV store: a kv store has all the meta info datacoord needs to operate. (etcd)
Message stream: a message stream to communicate statistics information with data nodes. (Pulsar)
Root Coordinator: timestamp, id and meta source.
Data Node(s): could be an instance or a cluster, actual worker group handles data modification operations.