mirror of https://gitee.com/milvus-io/milvus.git synced 2026-01-05 02:12:48 +08:00

History

enhance: Use map instead of slice to maintain channel info (#32273 )

See also #32165

`ChannelManager.Match` is a frequent operation for datacoord. When the
collection number is large, iteration over all channels will cost lots
of CPU time and time consuming.

This PR change the data structure storing datanode-channel info to map
avoiding this iteration when checking channel existence.

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

2024-04-16 15:57:19 +08:00

broker

feat: support rate limiter based on db and partition levels (#31070 )

2024-04-12 16:01:19 +08:00

allocator_test.go

Decoupling client and server API in types interface (#27186 )

2023-09-26 09:57:25 +08:00

allocator.go

feat: Add import scheduler and manager (#29367 )

2024-03-01 18:31:02 +08:00

build_index_policy.go

Format the code (#27275 )

2023-09-21 09:45:27 +08:00

channel_checker_test.go

Format the code (#27275 )

2023-09-21 09:45:27 +08:00

channel_checker.go

Format the code (#27275 )

2023-09-21 09:45:27 +08:00

channel_manager_factory.go

enhance: Remove not in use policies (#29448 )

2023-12-28 10:38:46 +08:00

channel_manager_test.go

enhance: Use map instead of slice to maintain channel info (#32273 )

2024-04-16 15:57:19 +08:00

channel_manager.go

enhance: Use map instead of slice to maintain channel info (#32273 )

2024-04-16 15:57:19 +08:00

channel_store_test.go

enhance: Use map instead of slice to maintain channel info (#32273 )

2024-04-16 15:57:19 +08:00

channel_store.go

enhance: Use map instead of slice to maintain channel info (#32273 )

2024-04-16 15:57:19 +08:00

channel.go

Change channel to Interface (#27839 )

2023-11-13 11:16:18 +08:00

cluster_test.go

enhance: Remove import v1 (#31403 )

2024-03-22 15:29:09 +08:00

cluster.go

enhance: Use channel manager interface in server_test (#31621 )

2024-04-12 14:59:20 +08:00

compaction_l0_view_test.go

enhance: Add force trigger (#30641 )

2024-02-19 18:40:50 +08:00

compaction_l0_view.go

fix: Trigger l0 compaction when l0 views don't change (#30729 )

2024-03-05 16:37:00 +08:00

compaction_scheduler_test.go

enhance: Add CompactionTaskNum metrics (#29518 )

2023-12-28 15:46:55 +08:00

compaction_scheduler.go

fix: Trigger l0 compaction when l0 views don't change (#30729 )

2024-03-05 16:37:00 +08:00

compaction_test.go

fix: [cherry-pick]Skip changing meta if nodeID not match with channel (#31672 )

2024-04-10 15:09:18 +08:00

compaction_trigger_test.go

enhance: Prevent the backlog of channelCP update tasks, perform batch updates of channelCPs (#30941 )

2024-03-07 20:39:02 +08:00

compaction_trigger_v2_test.go

fix: Skip to submit l0 tasks when scheduler full (#31270 )

2024-03-15 10:21:12 +08:00

compaction_trigger_v2.go

fix: Skip to submit l0 tasks when scheduler full (#31270 )

2024-03-15 10:21:12 +08:00

compaction_trigger.go

feat: Add global mmap enable configuration (#31267 )

2024-03-18 15:17:10 +08:00

compaction_view_manager_test.go

fix: Correct the last empty l0 views (#31198 )

2024-03-14 10:31:04 +08:00

compaction_view_manager.go

fix: Correct the last empty l0 views (#31198 )

2024-03-14 10:31:04 +08:00

compaction_view.go

fix: Trigger l0 compaction when l0 views don't change (#30729 )

2024-03-05 16:37:00 +08:00

compaction.go

fix: [cherry-pick]Skip changing meta if nodeID not match with channel (#31672 )

2024-04-10 15:09:18 +08:00

const.go

enhance: pass partition key scalar info if enabled when build vector index (#29931 )

2024-01-24 00:04:55 +08:00

errors_test.go

Format the code (#27275 )

2023-09-21 09:45:27 +08:00

errors.go

Refine DataCoord status (#27262 )

2023-09-26 17:15:27 +08:00

garbage_collector_test.go

enhance: Remove import v1 (#31403 )

2024-03-22 15:29:09 +08:00

garbage_collector.go

fix: Use correct ticker for gc work defer (#31456 )

2024-03-20 22:17:06 +08:00

handler.go

fix: fix collectionInfo leak in datacoord (#32175 )

2024-04-15 16:33:19 +08:00

import_checker_test.go

enhance: Ensure ImportV2 waits for the index to be built and refine some logic (#31629 )

2024-04-01 20:09:13 +08:00

import_checker.go

enhance: Ensure ImportV2 waits for the index to be built and refine some logic (#31629 )

2024-04-01 20:09:13 +08:00

import_job.go

enhance: Return more fields in import progress response (#31539 )

2024-03-24 21:57:06 +08:00

import_meta_test.go

feat: Add import scheduler and manager (#29367 )

2024-03-01 18:31:02 +08:00

import_meta.go

feat: Add import scheduler and manager (#29367 )

2024-03-01 18:31:02 +08:00

import_scheduler_test.go

enhance: Ensure ImportV2 waits for the index to be built and refine some logic (#31629 )

2024-04-01 20:09:13 +08:00

import_scheduler.go

enhance: Ensure ImportV2 waits for the index to be built and refine some logic (#31629 )

2024-04-01 20:09:13 +08:00

import_task.go

enhance: Enhance and modify the return content of ImportV2 (#31192 )

2024-03-13 19:51:03 +08:00

import_util_test.go

enhance: Ensure ImportV2 waits for the index to be built and refine some logic (#31629 )

2024-04-01 20:09:13 +08:00

import_util.go

feat: support rate limiter based on db and partition levels (#31070 )

2024-04-12 16:01:19 +08:00

index_builder_test.go

enhance: gather materialized view search info once per request (#31996 )

2024-04-11 15:21:19 +08:00

index_builder.go

enhance: gather materialized view search info once per request (#31996 )

2024-04-11 15:21:19 +08:00

index_engine_version_manager_test.go

Optimize the codec code of session (#27360 )

2023-10-01 10:33:30 +08:00

index_engine_version_manager.go

feat: trigger compaction to handle index version (#28442 )

2023-11-21 09:26:22 +08:00

index_meta_test.go

enhance: Ensure ImportV2 waits for the index to be built and refine some logic (#31629 )

2024-04-01 20:09:13 +08:00

index_meta.go

enhance: Ensure ImportV2 waits for the index to be built and refine some logic (#31629 )

2024-04-01 20:09:13 +08:00

index_service_test.go

enhance: Add ListIndexes API from datacoord (#31104 )

2024-03-07 17:37:01 +08:00

index_service.go

feat: Add global mmap enable configuration (#31267 )

2024-03-18 15:17:10 +08:00

indexnode_manager_test.go

Refine state check (#27541 )

2023-10-11 21:01:35 +08:00

indexnode_manager.go

Decoupling client and server API in types interface (#27186 )

2023-09-26 09:57:25 +08:00

meta_test.go

feat: support rate limiter based on db and partition levels (#31070 )

2024-04-12 16:01:19 +08:00

meta_util.go

Make VChannelInfo read compatible with 2.0.2 (#17995 )

2022-07-07 21:58:22 +08:00

meta.go

fix: fix collectionInfo leak in datacoord (#32175 )

2024-04-15 16:33:19 +08:00

metrics_info_test.go

enhance: Use channel manager interface in server_test (#31621 )

2024-04-12 14:59:20 +08:00

metrics_info.go

feat: support rate limiter based on db and partition levels (#31070 )

2024-04-12 16:01:19 +08:00

mock_allocator_test.go

enhance: index meta use independent rather than global meta lock (#30869 )

2024-03-04 16:56:59 +08:00

mock_channel_store.go

Change channel to Interface (#27839 )

2023-11-13 11:16:18 +08:00

mock_channelmanager.go

enhance: Add CompactionTaskNum metrics (#29518 )

2023-12-28 15:46:55 +08:00

mock_cluster.go

fix: add lazy load field to mark segment load type (#31591 )

2024-03-28 11:23:10 +08:00

mock_compaction_meta.go

enhance: Refine compaction interfaces to support major compaction (#30632 )

2024-02-19 20:52:50 +08:00

mock_compaction_plan_context.go

fix: Separate schedule and check results loop (#28692 )

2023-11-29 10:50:29 +08:00

mock_handler.go

fix: fix collectionInfo leak in datacoord (#32175 )

2024-04-15 16:33:19 +08:00

mock_index_engine_version_manager.go

feat: trigger compaction to handle index version (#28442 )

2023-11-21 09:26:22 +08:00

mock_scheduler.go

enhance: Add CompactionTaskNum metrics (#29518 )

2023-12-28 15:46:55 +08:00

mock_segment_manager.go

enhance: index meta use independent rather than global meta lock (#30869 )

2024-03-04 16:56:59 +08:00

mock_session_manager.go

fix: add lazy load field to mark segment load type (#31591 )

2024-03-28 11:23:10 +08:00

mock_test.go

fix: fix collectionInfo leak in datacoord (#32175 )

2024-04-15 16:33:19 +08:00

mock_trigger_manager.go

feat: Add Compaction views and triggers (#27906 )

2023-11-23 17:30:25 +08:00

OWNERS

[skip ci]Update OWNERS files (#11898 )

2021-11-16 15:41:11 +08:00

policy_test.go

enhance: Use map instead of slice to maintain channel info (#32273 )

2024-04-16 15:57:19 +08:00

policy.go

enhance: Use map instead of slice to maintain channel info (#32273 )

2024-04-16 15:57:19 +08:00

README.md

[skip ci]Change etcd to lowercase (#9983 )

2021-10-15 18:58:37 +08:00

segment_allocation_policy_test.go

Adjust datacoord for L0 Delta (#28021 )

2023-11-06 15:26:16 +08:00

segment_allocation_policy.go

fix: Decoupling importing segment from flush process (#30402 )

2024-02-03 13:01:12 +08:00

segment_info_test.go

fix: starve lock caused by slow GetCompactionTo method when too much segments (#30963 )

2024-03-05 10:04:59 +08:00

segment_info.go

enhance: Remove import v1 (#31403 )

2024-03-22 15:29:09 +08:00

segment_manager_test.go

feat: add more operation detail info for better allocation (#30438 )

2024-03-28 06:33:11 +08:00

segment_manager.go

enhance: Ensure ImportV2 waits for the index to be built and refine some logic (#31629 )

2024-04-01 20:09:13 +08:00

segment_operator_test.go

fix: Prevent clone when selecting segments from meta (#30928 )

2024-03-01 13:59:02 +08:00

segment_operator.go

fix: Prevent clone when selecting segments from meta (#30928 )

2024-03-01 13:59:02 +08:00

server_test.go

fix: Wait StandBy server ready for testcase (#32216 )

2024-04-12 23:49:18 +08:00

server.go

feat: support rate limiter based on db and partition levels (#31070 )

2024-04-12 16:01:19 +08:00

services_test.go

enhance: Use channel manager interface in server_test (#31621 )

2024-04-12 14:59:20 +08:00

services.go

feat: support rate limiter based on db and partition levels (#31070 )

2024-04-12 16:01:19 +08:00

session_manager_test.go

enhance: Use channel manager interface in server_test (#31621 )

2024-04-12 14:59:20 +08:00

session_manager.go

enhance: Use channel manager interface in server_test (#31621 )

2024-04-12 14:59:20 +08:00

session.go

Decoupling client and server API in types interface (#27186 )

2023-09-26 09:57:25 +08:00

util_test.go

feat: Add import scheduler and manager (#29367 )

2024-03-01 18:31:02 +08:00

util.go

enhance: gather materialized view search info once per request (#31996 )

2024-04-11 15:21:19 +08:00

README.md

Data Coordinator

Data cooridnator(datacoord for short) is the component to organize DataNodes and segments allocations.

Dependency

KV store: a kv store has all the meta info datacoord needs to operate. (etcd)
Message stream: a message stream to communicate statistics information with data nodes. (Pulsar)
Root Coordinator: timestamp, id and meta source.
Data Node(s): could be an instance or a cluster, actual worker group handles data modification operations.