mirror of https://gitee.com/milvus-io/milvus.git synced 2026-01-07 19:31:51 +08:00

History

enhance: optimize CPU usage for CheckHealth requests (#35589 )

issue: #35563
1. Use an internal health checker to monitor the cluster's health state,
storing the latest state on the coordinator node. The CheckHealth
request retrieves the cluster's health from this latest state on the
proxy sides, which enhances cluster stability.
2. Each health check will assess all collections and channels, with
detailed failure messages temporarily saved in the latest state.
3. Use CheckHealth request instead of the heavy GetMetrics request on
the querynode and datanode

Signed-off-by: jaime <yun.zhang@zilliz.com>

2024-12-17 11:02:45 +08:00

allocator

fix: fix make generate-mockery panic with go1.22 (#36830 )

2024-10-17 12:11:31 +08:00

broker

fix: fix make generate-mockery panic with go1.22 (#36830 )

2024-10-17 12:11:31 +08:00

session

enhance: optimize CPU usage for CheckHealth requests (#35589 )

2024-12-17 11:02:45 +08:00

analyze_meta_test.go

enhance: refine the datacoord meta related interfaces (#37957 )

2024-11-26 19:46:34 +08:00

analyze_meta.go

enhance: refine the datacoord meta related interfaces (#37957 )

2024-11-26 19:46:34 +08:00

build_index_policy.go

Format the code (#27275 )

2023-09-21 09:45:27 +08:00

channel_manager_factory.go

fix: drop collection failed if enable streaming service (#37444 )

2024-11-07 10:26:26 +08:00

channel_manager_test.go

enhance: Add ctx param to KV operation interfaces (#38154 )

2024-12-05 15:16:41 +08:00

channel_manager.go

feat: support to replicate collection when the services contains the system tt msg (#37559 )

2024-12-17 09:08:46 +08:00

channel_store_test.go

enhance: Add ctx param to KV operation interfaces (#38154 )

2024-12-05 15:16:41 +08:00

channel_store.go

enhance: refine exists log print with ctx (#38080 )

2024-12-14 22:36:44 +08:00

channel.go

feat: support to replicate collection when the services contains the system tt msg (#37559 )

2024-12-17 09:08:46 +08:00

cluster_test.go

enhance: Move datanode/indexnode manager to session pkg (#35634 )

2024-08-22 16:02:56 +08:00

cluster.go

enhance: refine exists log print with ctx (#38080 )

2024-12-14 22:36:44 +08:00

compaction_l0_view_test.go

enhance: Add deltaRowCount in l0 compaction (#33997 )

2024-06-20 17:46:01 +08:00

compaction_l0_view.go

enhance: Add deltaRowCount in l0 compaction (#33997 )

2024-06-20 17:46:01 +08:00

compaction_policy_clustering_test.go

enhance: refine the datacoord meta related interfaces (#37957 )

2024-11-26 19:46:34 +08:00

compaction_policy_clustering.go

enhance: refine the datacoord meta related interfaces (#37957 )

2024-11-26 19:46:34 +08:00

compaction_policy_l0_test.go

enhance: Separate allocator pkg in datacoord (#35622 )

2024-08-22 10:06:56 +08:00

compaction_policy_l0.go

enhance: tidy compaction logs (#37595 )

2024-11-12 21:18:29 +08:00

compaction_policy_single_test.go

enhance: refine the datacoord meta related interfaces (#37957 )

2024-11-26 19:46:34 +08:00

compaction_policy_single.go

enhance: tidy compaction logs (#37595 )

2024-11-12 21:18:29 +08:00

compaction_queue_test.go

fix: Refine compactionTask to avoid data race (#36936 )

2024-10-24 09:55:28 +08:00

compaction_queue.go

fix: Refine compactionTask to avoid data race (#36936 )

2024-10-24 09:55:28 +08:00

compaction_task_clustering_test.go

enhance: refine the datacoord meta related interfaces (#37957 )

2024-11-26 19:46:34 +08:00

compaction_task_clustering.go

enhance: refine exists log print with ctx (#38080 )

2024-12-14 22:36:44 +08:00

compaction_task_l0_test.go

enhance: refine the datacoord meta related interfaces (#37957 )

2024-11-26 19:46:34 +08:00

compaction_task_l0.go

enhance: add end time stats for compaction and index build tasks (#38048 )

2024-12-06 16:28:40 +08:00

compaction_task_meta_test.go

enhance: refine the datacoord meta related interfaces (#37957 )

2024-11-26 19:46:34 +08:00

compaction_task_meta.go

enhance: refine the datacoord meta related interfaces (#37957 )

2024-11-26 19:46:34 +08:00

compaction_task_mix_test.go

enhance: refine the datacoord meta related interfaces (#37957 )

2024-11-26 19:46:34 +08:00

compaction_task_mix.go

enhance: add end time stats for compaction and index build tasks (#38048 )

2024-12-06 16:28:40 +08:00

compaction_task.go

fix: Refine compactionTask to avoid data race (#36936 )

2024-10-24 09:55:28 +08:00

compaction_test.go

fix: Handle the error of the compaction queue being full (#37989 )

2024-11-29 11:10:36 +08:00

compaction_trigger_test.go

enhance: improve mix compaction performance by removing max segment limitations (#38344 )

2024-12-11 20:38:42 +08:00

compaction_trigger_v2_test.go

enhance: Separate allocator pkg in datacoord (#35622 )

2024-08-22 10:06:56 +08:00

compaction_trigger_v2.go

enhance: refine exists log print with ctx (#38080 )

2024-12-14 22:36:44 +08:00

compaction_trigger.go

enhance: refine exists log print with ctx (#38080 )

2024-12-14 22:36:44 +08:00

compaction_view.go

enhance: Add deltaRowCount in l0 compaction (#33997 )

2024-06-20 17:46:01 +08:00

compaction.go

enhance: refine exists log print with ctx (#38080 )

2024-12-14 22:36:44 +08:00

const.go

enhance: pass partition key scalar info if enabled when build vector index (#29931 )

2024-01-24 00:04:55 +08:00

errors_test.go

Format the code (#27275 )

2023-09-21 09:45:27 +08:00

errors.go

Refine DataCoord status (#27262 )

2023-09-26 17:15:27 +08:00

garbage_collector_test.go

enhance: refine the datacoord meta related interfaces (#37957 )

2024-11-26 19:46:34 +08:00

garbage_collector.go

enhance: refine exists log print with ctx (#38080 )

2024-12-14 22:36:44 +08:00

go_channel_singleton.go

fix: Fix improper use of offset in HybridSearch (#36244 )

2024-09-13 22:05:15 +08:00

handler_test.go

enhance: refine the datacoord meta related interfaces (#37957 )

2024-11-26 19:46:34 +08:00

handler.go

enhance: refine the datacoord meta related interfaces (#37957 )

2024-11-26 19:46:34 +08:00

import_checker_test.go

fix: Use the correct RootPath when decompressing binlog in stats task (#38341 )

2024-12-11 16:16:42 +08:00

import_checker.go

fix: Use the correct RootPath when decompressing binlog in stats task (#38341 )

2024-12-11 16:16:42 +08:00

import_job.go

fix: Revert "enhance: Support db for bulkinsert (#37012 )" (#37420 )

2024-11-07 17:02:25 +08:00

import_meta_test.go

enhance: refine the datacoord meta related interfaces (#37957 )

2024-11-26 19:46:34 +08:00

import_meta.go

enhance: refine the datacoord meta related interfaces (#37957 )

2024-11-26 19:46:34 +08:00

import_scheduler_test.go

enhance: refine the datacoord meta related interfaces (#37957 )

2024-11-26 19:46:34 +08:00

import_scheduler.go

enhance: refine exists log print with ctx (#38080 )

2024-12-14 22:36:44 +08:00

import_task.go

enhance: Use internal json(sonic) to replace std json lib (#37708 )

2024-11-18 10:46:31 +08:00

import_util_test.go

fix: Use the correct RootPath when decompressing binlog in stats task (#38341 )

2024-12-11 16:16:42 +08:00

import_util.go

fix: Use the correct RootPath when decompressing binlog in stats task (#38341 )

2024-12-11 16:16:42 +08:00

index_engine_version_manager_test.go

Optimize the codec code of session (#27360 )

2023-10-01 10:33:30 +08:00

index_engine_version_manager.go

enhance: Add go-deadlock as unittest only dependency (#33063 )

2024-05-15 16:33:34 +08:00

index_meta_test.go

enhance: Add ctx param to KV operation interfaces (#38154 )

2024-12-05 15:16:41 +08:00

index_meta.go

enhance: refine exists log print with ctx (#38080 )

2024-12-14 22:36:44 +08:00

index_service_test.go

enhance: alterindex & altercollection supports altering properties (#37437 )

2024-12-11 10:20:42 +08:00

index_service.go

enhance: refine exists log print with ctx (#38080 )

2024-12-14 22:36:44 +08:00

job_manager_test.go

feat: add segment/channel/task/slow query render (#37561 )

2024-11-12 17:44:29 +08:00

job_manager.go

fix: Use the correct RootPath when decompressing binlog in stats task (#38341 )

2024-12-11 16:16:42 +08:00

knapsack_test.go

enhance: try compact small segments first if they may compose a full segment (#37709 )

2024-12-02 16:12:38 +08:00

knapsack.go

enhance: improve mix compaction performance by removing max segment limitations (#38344 )

2024-12-11 20:38:42 +08:00

meta_test.go

enhance: Add ctx param to KV operation interfaces (#38154 )

2024-12-05 15:16:41 +08:00

meta_util.go

…

meta.go

feat: support to replicate collection when the services contains the system tt msg (#37559 )

2024-12-17 09:08:46 +08:00

metrics_info_test.go

enhance: add list index and segment index retrieval API for WebUI (#37861 )

2024-11-22 16:58:34 +08:00

metrics_info.go

enhance: add disk stats within system metrics (#38033 )

2024-12-06 16:32:41 +08:00

mock_channel_store.go

enhance: regenerate datacoord mockery (#37501 )

2024-11-07 17:38:24 +08:00

mock_channelmanager.go

feat: add segment,pipeline, replica and resourcegroup api for WebUI (#37344 )

2024-11-07 11:52:25 +08:00

mock_cluster.go

fix: fix make generate-mockery panic with go1.22 (#36830 )

2024-10-17 12:11:31 +08:00

mock_compaction_meta.go

enhance: refine the datacoord meta related interfaces (#37957 )

2024-11-26 19:46:34 +08:00

mock_compaction_plan_context.go

enhance: refine the datacoord meta related interfaces (#37957 )

2024-11-26 19:46:34 +08:00

mock_handler.go

fix: fix make generate-mockery panic with go1.22 (#36830 )

2024-10-17 12:11:31 +08:00

mock_index_engine_version_manager.go

fix: fix make generate-mockery panic with go1.22 (#36830 )

2024-10-17 12:11:31 +08:00

mock_job_manager.go

fix: Use the correct RootPath when decompressing binlog in stats task (#38341 )

2024-12-11 16:16:42 +08:00

mock_segment_manager.go

fix: Use the correct RootPath when decompressing binlog in stats task (#38341 )

2024-12-11 16:16:42 +08:00

mock_subcluster.go

fix: fix make generate-mockery panic with go1.22 (#36830 )

2024-10-17 12:11:31 +08:00

mock_test.go

enhance: optimize CPU usage for CheckHealth requests (#35589 )

2024-12-17 11:02:45 +08:00

mock_trigger_manager.go

fix: fix make generate-mockery panic with go1.22 (#36830 )

2024-10-17 12:11:31 +08:00

OWNERS

[skip ci]Update OWNERS files (#11898 )

2021-11-16 15:41:11 +08:00

partition_stats_meta_test.go

fix: Set current partition stats version to 0 by default when not present (#37299 )

2024-10-31 12:48:21 +08:00

partition_stats_meta.go

enhance: refine the datacoord meta related interfaces (#37957 )

2024-11-26 19:46:34 +08:00

policy_test.go

fix: Fill nil schema so that Milvus can watch channel for those upgraded from 2.2 to 2.4 #35695 (#35694 )

2024-08-27 10:36:59 +08:00

policy.go

fix: drop collection failed if enable streaming service (#37444 )

2024-11-07 10:26:26 +08:00

README.md

…

segment_allocation_policy_test.go

enhance: Change the fixed value to a ratio for clustering segment size (#35076 )

2024-08-01 22:04:14 +08:00

segment_allocation_policy.go

fix: Fix large growing segment (#37388 )

2024-11-07 17:12:25 +08:00

segment_info_test.go

fix: Return all compactTo segments after support split (#36361 )

2024-09-20 14:11:11 +08:00

segment_info.go

enhance: Remove unnecessary clone in SetState (#37697 )

2024-11-15 15:06:30 +08:00

segment_manager_test.go

enhance: Add ctx param to KV operation interfaces (#38154 )

2024-12-05 15:16:41 +08:00

segment_manager.go

enhance: refine the datacoord meta related interfaces (#37957 )

2024-11-26 19:46:34 +08:00

segment_operator_test.go

fix: Prevent clone when selecting segments from meta (#30928 )

2024-03-01 13:59:02 +08:00

segment_operator.go

enhance: Update stats task to optional (#35947 )

2024-09-12 20:37:08 +08:00

server_test.go

enhance: optimize CPU usage for CheckHealth requests (#35589 )

2024-12-17 11:02:45 +08:00

server.go

enhance: optimize CPU usage for CheckHealth requests (#35589 )

2024-12-17 11:02:45 +08:00

services_test.go

enhance: refine the datacoord meta related interfaces (#37957 )

2024-11-26 19:46:34 +08:00

services.go

enhance: optimize CPU usage for CheckHealth requests (#35589 )

2024-12-17 11:02:45 +08:00

stats_task_meta_test.go

fix: Record the nodeID before assigning tasks (#36371 )

2024-09-28 17:21:15 +08:00

stats_task_meta.go

feat: add segment,pipeline, replica and resourcegroup api for WebUI (#37344 )

2024-11-07 11:52:25 +08:00

sync_segments_scheduler_test.go

fix: datacoord stuck at stopping progress (#36852 )

2024-10-17 12:13:37 +08:00

sync_segments_scheduler.go

enhance: refine the datacoord meta related interfaces (#37957 )

2024-11-26 19:46:34 +08:00

task_analyze.go

enhance: refine the datacoord meta related interfaces (#37957 )

2024-11-26 19:46:34 +08:00

task_index.go

fix:remove unnecessary error logs (#38245 )

2024-12-08 17:20:40 +08:00

task_scheduler_test.go

enhance: refine the datacoord meta related interfaces (#37957 )

2024-11-26 19:46:34 +08:00

task_scheduler.go

enhance: Refine task scheduler logs (#38334 )

2024-12-11 15:00:44 +08:00

task_stats_test.go

enhance: refine the datacoord meta related interfaces (#37957 )

2024-11-26 19:46:34 +08:00

task_stats.go

enhance: refine the datacoord meta related interfaces (#37957 )

2024-11-26 19:46:34 +08:00

types.go

fix: Record the nodeID before assigning tasks (#36371 )

2024-09-28 17:21:15 +08:00

util_test.go

enhance: Separate allocator pkg in datacoord (#35622 )

2024-08-22 10:06:56 +08:00

util.go

enhance: optimize CPU usage for CheckHealth requests (#35589 )

2024-12-17 11:02:45 +08:00

README.md

Data Coordinator

Data cooridnator(datacoord for short) is the component to organize DataNodes and segments allocations.

Dependency

KV store: a kv store has all the meta info datacoord needs to operate. (etcd)
Message stream: a message stream to communicate statistics information with data nodes. (Pulsar)
Root Coordinator: timestamp, id and meta source.
Data Node(s): could be an instance or a cluster, actual worker group handles data modification operations.