mirror of https://gitee.com/milvus-io/milvus.git synced 2026-01-04 01:42:15 +08:00

History

enhance: Prevent the backlog of channelCP update tasks, perform batch updates of channelCPs (#30941 )

This PR includes the following adjustments:
1. To prevent channelCP update task backlog, only one task with the same
vchannel is retained in the updater. Additionally, the lastUpdateTime is
refreshed after the flowgraph submits the update task, rather than in
the callBack function.
2. Batch updates of multiple vchannel checkpoints are performed in the
UpdateChannelCheckpoint RPC (default batch size is 128). Additionally,
the lock for channelCPs in DataCoord meta has been switched from key
lock to global lock.
3. The concurrency of UpdateChannelCheckpoint RPCs in the datanode has
been reduced from 1000 to 10.

issue: https://github.com/milvus-io/milvus/issues/30004

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Co-authored-by: jaime <yun.zhang@zilliz.com>
Co-authored-by: congqixia <congqi.xia@zilliz.com>

2024-03-07 20:39:02 +08:00

allocator

Replace manual composed grpc call with Broker methods (#27676 )

2023-10-13 09:55:34 +08:00

broker

enhance: Prevent the backlog of channelCP update tasks, perform batch updates of channelCPs (#30941 )

2024-03-07 20:39:02 +08:00

importv2

feat: Add import scheduler and manager (#29367 )

2024-03-01 18:31:02 +08:00

enhance: Reduce unnessary log in binlog_io (#30625 )

2024-02-18 16:50:51 +08:00

iterators

enhance: Speed up L0 compaction (#30410 )

2024-02-04 10:49:05 +08:00

metacache

enhance: Use RWMutex and change WLock to RLock (#30557 )

2024-02-06 17:13:56 +08:00

syncmgr

test: support multiple data/querynodes in integration test (#30618 )

2024-02-21 11:54:53 +08:00

writebuffer

fix: Decoupling importing segment from flush process (#30402 )

2024-02-03 13:01:12 +08:00

binlog_io_test.go

enhance: replace binlogIO with io.BinlogIO in datanode (#29725 )

2024-02-20 14:38:51 +08:00

binlog_io.go

fix: Fix binlog_io metric name conflict (#30689 )

2024-03-01 18:13:02 +08:00

cache_test.go

Make duplicate flush recalls success and other logic update (#16755 )

2022-05-06 17:49:51 +08:00

cache.go

Use typeutil.ConcurrentMap instead of sync.Map (#25846 )

2023-07-24 10:23:01 +08:00

channel_checkpoint_updater_test.go

enhance: Prevent the backlog of channelCP update tasks, perform batch updates of channelCPs (#30941 )

2024-03-07 20:39:02 +08:00

channel_checkpoint_updater.go

enhance: Prevent the backlog of channelCP update tasks, perform batch updates of channelCPs (#30941 )

2024-03-07 20:39:02 +08:00

channel_manager_test.go

enhance: Change DN channelmanger into interface (#29307 )

2023-12-27 16:00:48 +08:00

channel_manager.go

enhance: Change DN channelmanger into interface (#29307 )

2023-12-27 16:00:48 +08:00

compaction_executor_test.go

fix: Separate schedule and check results loop (#28692 )

2023-11-29 10:50:29 +08:00

compaction_executor.go

fix: Clear DN unkown compaction tasks (#30850 )

2024-03-01 11:31:00 +08:00

compactor_test.go

enhance: replace binlogIO with io.BinlogIO in datanode (#29725 )

2024-02-20 14:38:51 +08:00

compactor.go

enhance: replace binlogIO with io.BinlogIO in datanode (#29725 )

2024-02-20 14:38:51 +08:00

data_node_test.go

enhance: Add FlowgraphManager interface (#28852 )

2023-11-30 18:42:32 +08:00

data_node.go

enhance: Prevent the backlog of channelCP update tasks, perform batch updates of channelCPs (#30941 )

2024-03-07 20:39:02 +08:00

data_sync_service_test.go

enhance: Prevent the backlog of channelCP update tasks, perform batch updates of channelCPs (#30941 )

2024-03-07 20:39:02 +08:00

data_sync_service.go

test: support multiple data/querynodes in integration test (#30618 )

2024-02-21 11:54:53 +08:00

event_manager_test.go

enhance: Prevent the backlog of channelCP update tasks, perform batch updates of channelCPs (#30941 )

2024-03-07 20:39:02 +08:00

event_manager.go

test: support multiple data/querynodes in integration test (#30618 )

2024-02-21 11:54:53 +08:00

flow_graph_dd_node_test.go

Format the code (#27275 )

2023-09-21 09:45:27 +08:00

flow_graph_dd_node.go

fix: Add more throughput in related metrics (#30038 )

2024-01-19 11:34:54 +08:00

flow_graph_dmstream_input_node_test.go

Support to replicate the mq message (#27240 )

2023-10-20 14:26:09 +08:00

flow_graph_dmstream_input_node.go

fix: add back existing datanode metrics (#29360 )

2023-12-22 14:20:43 +08:00

flow_graph_manager_test.go

enhance: Prevent the backlog of channelCP update tasks, perform batch updates of channelCPs (#30941 )

2024-03-07 20:39:02 +08:00

flow_graph_manager.go

enhance: Remove debug log (#30955 )

2024-03-01 10:02:59 +08:00

flow_graph_message_test.go

Format the code (#27275 )

2023-09-21 09:45:27 +08:00

flow_graph_message.go

Format the code (#27275 )

2023-09-21 09:45:27 +08:00

flow_graph_node.go

Make flow graph retry longer (#24355 )

2023-05-25 09:53:30 +08:00

flow_graph_time_tick_node.go

enhance: Prevent the backlog of channelCP update tasks, perform batch updates of channelCPs (#30941 )

2024-03-07 20:39:02 +08:00

flow_graph_time_ticker.go

enhance: create goroutine only once (#28594 )

2023-11-22 10:26:28 +08:00

flow_graph_write_node.go

enhance: Remove commented code and fix naming issue (#28450 )

2023-11-16 00:22:20 +08:00

flush_task_counter_test.go

Fix flushManager.isFull is too slow (#28141 )

2023-11-03 14:42:17 +08:00

flush_task_counter.go

Fix flushManager.isFull is too slow (#28141 )

2023-11-03 14:42:17 +08:00

io_pool_test.go

Fix copylocks linter errors (#26217 )

2023-08-08 20:41:15 +08:00

io_pool.go

Fix getting incorrect CPU num (#28146 )

2023-11-06 06:02:16 +08:00

l0_compactor_test.go

enhance: Speed up L0 compaction (#30410 )

2024-02-04 10:49:05 +08:00

l0_compactor.go

enhance: Add param item for datanode l0 batch/linear mode memory ratio (#30523 )

2024-02-18 13:02:50 +08:00

meta_service_test.go

Replace manual composed grpc call with Broker methods (#27676 )

2023-10-13 09:55:34 +08:00

meta_service.go

Replace manual composed grpc call with Broker methods (#27676 )

2023-10-13 09:55:34 +08:00

meta_util.go

Use go-api/v2 for milvus-proto (#24770 )

2023-06-09 01:28:37 +08:00

metrics_info.go

enhance: Add FlowgraphManager interface (#28852 )

2023-11-30 18:42:32 +08:00

mock_channelmanager.go

enhance: Change DN channelmanger into interface (#29307 )

2023-12-27 16:00:48 +08:00

mock_fgmanager.go

enhance: separate serializer logic from sync task (#29413 )

2023-12-26 10:40:47 +08:00

mock_test.go

test: support multiple data/querynodes in integration test (#30618 )

2024-02-21 11:54:53 +08:00

OWNERS

[skip ci]Update OWNERS files (#11898 )

2021-11-16 15:41:11 +08:00

rate_collector_test.go

Move some modules from internal to public package (#22572 )

2023-04-06 19:14:32 +08:00

rate_collector.go

Format the code (#27275 )

2023-09-21 09:45:27 +08:00

README.md

[skip ci]Change uppercase of MinIO (#9539 )

2021-10-09 11:22:58 +08:00

services_test.go

enhance: Prevent the backlog of channelCP update tasks, perform batch updates of channelCPs (#30941 )

2024-03-07 20:39:02 +08:00

services.go

feat: Add import scheduler and manager (#29367 )

2024-03-01 18:31:02 +08:00

stats_updater_test.go

Use writebuffer, sync manager refactory in datanode (#28320 )

2023-11-15 15:24:18 +08:00

stats_updater.go

test: support multiple data/querynodes in integration test (#30618 )

2024-02-21 11:54:53 +08:00

timetick_sender_test.go

enhance: Reduce the memory usage of the timeTickSender (#30968 )

2024-03-02 10:13:01 +08:00

timetick_sender.go

enhance: Reduce the memory usage of the timeTickSender (#30968 )

2024-03-02 10:13:01 +08:00

util.go

Format the code (#27275 )

2023-09-21 09:45:27 +08:00

README.md

Data Node

DataNode is the component to write insert and delete messages into persistent blob storage, for example MinIO or S3.

Dependency

KV store: a kv store that persists messages into blob storage.
Message stream: receive messages and publish imformation
Root Coordinator: get the latest unique IDs.
Data Coordinator: get the flush information and which message stream to subscribe.