mirror of https://gitee.com/milvus-io/milvus.git synced 2025-12-07 17:48:29 +08:00

History

fix: Verify sync task target segment and retry if not match (#30500 )

See also #27675 #30469

For a sync task, the segment could be compacted during sync task. In
previous implementation, this sync task will hold only the old segment
id as KeyLock, in which case compaction on compacted to segment may run
in parallel with delta sync of this sync task.

This PR introduces sync target segment verification logic. It shall
check target segment lock it's holding beforing actually syncing logic.
If this check failed, sync task shall return`errTargetSegementNotMatch`
error and make manager re-fetch the current target segment id.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

2024-02-05 11:33:43 +08:00

allocator

Replace manual composed grpc call with Broker methods (#27676 )

2023-10-13 09:55:34 +08:00

broker

enhance:don't store logPath in meta to reduce memory (#28873 )

2024-01-18 22:06:31 +08:00

importv2

enhance: Extend support for varchar autoID to BulkInsertV2 (#30477 )

2024-02-04 16:57:05 +08:00

enhance: Speed up L0 compaction (#30410 )

2024-02-04 10:49:05 +08:00

iterators

enhance: Speed up L0 compaction (#30410 )

2024-02-04 10:49:05 +08:00

metacache

fix: Decoupling importing segment from flush process (#30402 )

2024-02-03 13:01:12 +08:00

syncmgr

fix: Verify sync task target segment and retry if not match (#30500 )

2024-02-05 11:33:43 +08:00

writebuffer

fix: Decoupling importing segment from flush process (#30402 )

2024-02-03 13:01:12 +08:00

binlog_io_test.go

enhance: make compactor use actual buffer size to decide when to sync (#29945 )

2024-01-13 01:32:52 +08:00

binlog_io.go

enhance: Add open telemetry tracing for compaction (#30168 )

2024-01-23 10:37:00 +08:00

cache_test.go

Make duplicate flush recalls success and other logic update (#16755 )

2022-05-06 17:49:51 +08:00

cache.go

Use typeutil.ConcurrentMap instead of sync.Map (#25846 )

2023-07-24 10:23:01 +08:00

channel_checkpoint_updater.go

feat: Introduce channelCheckpointUpdater to reduce goroutine use in ttNode (#28570 )

2023-12-12 13:48:42 +08:00

channel_manager_test.go

enhance: Change DN channelmanger into interface (#29307 )

2023-12-27 16:00:48 +08:00

channel_manager.go

enhance: Change DN channelmanger into interface (#29307 )

2023-12-27 16:00:48 +08:00

compaction_executor_test.go

fix: Separate schedule and check results loop (#28692 )

2023-11-29 10:50:29 +08:00

compaction_executor.go

fix: call injectDone after compaction failed (#30277 )

2024-02-01 14:25:04 +08:00

compactor_test.go

fix: call injectDone after compaction failed (#30277 )

2024-02-01 14:25:04 +08:00

compactor.go

fix: call injectDone after compaction failed (#30277 )

2024-02-01 14:25:04 +08:00

data_node_test.go

enhance: Add FlowgraphManager interface (#28852 )

2023-11-30 18:42:32 +08:00

data_node.go

feat: Add import executor and manager for datanode (#29438 )

2024-01-31 20:45:04 +08:00

data_sync_service_test.go

fix: Use channel cp as the dml&start position for import segments (#30107 )

2024-01-22 14:36:55 +08:00

data_sync_service.go

enhance:don't store logPath in meta to reduce memory (#28873 )

2024-01-18 22:06:31 +08:00

event_manager_test.go

enhance: Add FlowgraphManager interface (#28852 )

2023-11-30 18:42:32 +08:00

event_manager.go

fix: panic with datanode negetive wait group counter (#30135 )

2024-01-30 18:15:04 +08:00

flow_graph_dd_node_test.go

Format the code (#27275 )

2023-09-21 09:45:27 +08:00

flow_graph_dd_node.go

fix: Add more throughput in related metrics (#30038 )

2024-01-19 11:34:54 +08:00

flow_graph_dmstream_input_node_test.go

Support to replicate the mq message (#27240 )

2023-10-20 14:26:09 +08:00

flow_graph_dmstream_input_node.go

fix: add back existing datanode metrics (#29360 )

2023-12-22 14:20:43 +08:00

flow_graph_manager_test.go

enhance: Add FlowgraphManager interface (#28852 )

2023-11-30 18:42:32 +08:00

flow_graph_manager.go

enhance: Add FlowgraphManager interface (#28852 )

2023-11-30 18:42:32 +08:00

flow_graph_message_test.go

Format the code (#27275 )

2023-09-21 09:45:27 +08:00

flow_graph_message.go

Format the code (#27275 )

2023-09-21 09:45:27 +08:00

flow_graph_node.go

Make flow graph retry longer (#24355 )

2023-05-25 09:53:30 +08:00

flow_graph_time_tick_node.go

fix: Skip updating checkpoint after dropcollection (#29220 )

2023-12-15 16:04:45 +08:00

flow_graph_time_ticker.go

enhance: create goroutine only once (#28594 )

2023-11-22 10:26:28 +08:00

flow_graph_write_node.go

enhance: Remove commented code and fix naming issue (#28450 )

2023-11-16 00:22:20 +08:00

flush_task_counter_test.go

Fix flushManager.isFull is too slow (#28141 )

2023-11-03 14:42:17 +08:00

flush_task_counter.go

Fix flushManager.isFull is too slow (#28141 )

2023-11-03 14:42:17 +08:00

io_pool_test.go

Fix copylocks linter errors (#26217 )

2023-08-08 20:41:15 +08:00

io_pool.go

Fix getting incorrect CPU num (#28146 )

2023-11-06 06:02:16 +08:00

l0_compactor_test.go

enhance: Speed up L0 compaction (#30410 )

2024-02-04 10:49:05 +08:00

l0_compactor.go

enhance: Speed up L0 compaction (#30410 )

2024-02-04 10:49:05 +08:00

meta_service_test.go

Replace manual composed grpc call with Broker methods (#27676 )

2023-10-13 09:55:34 +08:00

meta_service.go

Replace manual composed grpc call with Broker methods (#27676 )

2023-10-13 09:55:34 +08:00

meta_util.go

Use go-api/v2 for milvus-proto (#24770 )

2023-06-09 01:28:37 +08:00

metrics_info.go

enhance: Add FlowgraphManager interface (#28852 )

2023-11-30 18:42:32 +08:00

mock_channelmanager.go

enhance: Change DN channelmanger into interface (#29307 )

2023-12-27 16:00:48 +08:00

mock_fgmanager.go

enhance: separate serializer logic from sync task (#29413 )

2023-12-26 10:40:47 +08:00

mock_test.go

enhance: make compactor use actual buffer size to decide when to sync (#29945 )

2024-01-13 01:32:52 +08:00

OWNERS

[skip ci]Update OWNERS files (#11898 )

2021-11-16 15:41:11 +08:00

rate_collector_test.go

Move some modules from internal to public package (#22572 )

2023-04-06 19:14:32 +08:00

rate_collector.go

Format the code (#27275 )

2023-09-21 09:45:27 +08:00

README.md

[skip ci]Change uppercase of MinIO (#9539 )

2021-10-09 11:22:58 +08:00

services_test.go

fix: Use channel cp as the dml&start position for import segments (#30107 )

2024-01-22 14:36:55 +08:00

services.go

fix: Decoupling importing segment from flush process (#30402 )

2024-02-03 13:01:12 +08:00

stats_updater_test.go

Use writebuffer, sync manager refactory in datanode (#28320 )

2023-11-15 15:24:18 +08:00

stats_updater.go

enhance:Remove msgbase unnecessary assignments (#28511 )

2023-11-24 15:02:39 +08:00

timetick_sender_test.go

Refine datanode Timetick Sender (#28393 )

2023-11-13 21:46:20 +08:00

timetick_sender.go

Refine datanode Timetick Sender (#28393 )

2023-11-13 21:46:20 +08:00

util.go

Format the code (#27275 )

2023-09-21 09:45:27 +08:00

README.md

Data Node

DataNode is the component to write insert and delete messages into persistent blob storage, for example MinIO or S3.

Dependency

KV store: a kv store that persists messages into blob storage.
Message stream: receive messages and publish imformation
Root Coordinator: get the latest unique IDs.
Data Coordinator: get the flush information and which message stream to subscribe.