mirror of https://gitee.com/milvus-io/milvus.git synced 2025-12-07 09:38:39 +08:00

History

fix: call injectDone after compaction failed (#30277 )

syncMgr.Block() will lock the segment when executing compaction.

Previous implementation was unable to Unblock thoese segments when
compaction failed. If next compaction of the same segments arrives,
it'll stuck forever and block all later compation tasks.

This PR makes sure compaction executor would Unblock these segments
after a failure compaction.

Apart form that, this PR also refines some logs and clean some codes of
compaction, compactor:

1. Log segment count instead of segmentIDs to avoid logging too many
segments
2. Flush RPC returns L1 segments only, skip L0 and L2
3. CompactionType is checked in `Compaction`, no need to check again
inside compactor
4. Use ligter method to replace `getSegmentMeta`
5. Log information for L0 compaction when encounters an error

See also: #30213

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

2024-02-01 14:25:04 +08:00

allocator

Replace manual composed grpc call with Broker methods (#27676 )

2023-10-13 09:55:34 +08:00

broker

enhance:don't store logPath in meta to reduce memory (#28873 )

2024-01-18 22:06:31 +08:00

importv2

feat: Add import executor and manager for datanode (#29438 )

2024-01-31 20:45:04 +08:00

enhance: Make l0 compactor download files in parallel (#30309 )

2024-01-30 10:07:09 +08:00

iterators

Add float16 approve for multi-type part (#28427 )

2024-01-11 15:48:51 +08:00

metacache

enhance: Remove current stats after RollStats action (#30391 )

2024-01-31 18:55:04 +08:00

syncmgr

feat: Add import executor and manager for datanode (#29438 )

2024-01-31 20:45:04 +08:00

writebuffer

enhance: Add MemoryHighSyncPolicy back to write buffer manager (#29997 )

2024-01-31 19:03:04 +08:00

binlog_io_test.go

enhance: make compactor use actual buffer size to decide when to sync (#29945 )

2024-01-13 01:32:52 +08:00

binlog_io.go

enhance: Add open telemetry tracing for compaction (#30168 )

2024-01-23 10:37:00 +08:00

cache_test.go

Make duplicate flush recalls success and other logic update (#16755 )

2022-05-06 17:49:51 +08:00

cache.go

Use typeutil.ConcurrentMap instead of sync.Map (#25846 )

2023-07-24 10:23:01 +08:00

channel_checkpoint_updater.go

feat: Introduce channelCheckpointUpdater to reduce goroutine use in ttNode (#28570 )

2023-12-12 13:48:42 +08:00

channel_manager_test.go

enhance: Change DN channelmanger into interface (#29307 )

2023-12-27 16:00:48 +08:00

channel_manager.go

enhance: Change DN channelmanger into interface (#29307 )

2023-12-27 16:00:48 +08:00

compaction_executor_test.go

fix: Separate schedule and check results loop (#28692 )

2023-11-29 10:50:29 +08:00

compaction_executor.go

fix: call injectDone after compaction failed (#30277 )

2024-02-01 14:25:04 +08:00

compactor_test.go

fix: call injectDone after compaction failed (#30277 )

2024-02-01 14:25:04 +08:00

compactor.go

fix: call injectDone after compaction failed (#30277 )

2024-02-01 14:25:04 +08:00

data_node_test.go

enhance: Add FlowgraphManager interface (#28852 )

2023-11-30 18:42:32 +08:00

data_node.go

feat: Add import executor and manager for datanode (#29438 )

2024-01-31 20:45:04 +08:00

data_sync_service_test.go

fix: Use channel cp as the dml&start position for import segments (#30107 )

2024-01-22 14:36:55 +08:00

data_sync_service.go

enhance:don't store logPath in meta to reduce memory (#28873 )

2024-01-18 22:06:31 +08:00

event_manager_test.go

enhance: Add FlowgraphManager interface (#28852 )

2023-11-30 18:42:32 +08:00

event_manager.go

fix: panic with datanode negetive wait group counter (#30135 )

2024-01-30 18:15:04 +08:00

flow_graph_dd_node_test.go

Format the code (#27275 )

2023-09-21 09:45:27 +08:00

flow_graph_dd_node.go

fix: Add more throughput in related metrics (#30038 )

2024-01-19 11:34:54 +08:00

flow_graph_dmstream_input_node_test.go

Support to replicate the mq message (#27240 )

2023-10-20 14:26:09 +08:00

flow_graph_dmstream_input_node.go

fix: add back existing datanode metrics (#29360 )

2023-12-22 14:20:43 +08:00

flow_graph_manager_test.go

enhance: Add FlowgraphManager interface (#28852 )

2023-11-30 18:42:32 +08:00

flow_graph_manager.go

enhance: Add FlowgraphManager interface (#28852 )

2023-11-30 18:42:32 +08:00

flow_graph_message_test.go

Format the code (#27275 )

2023-09-21 09:45:27 +08:00

flow_graph_message.go

Format the code (#27275 )

2023-09-21 09:45:27 +08:00

flow_graph_node.go

Make flow graph retry longer (#24355 )

2023-05-25 09:53:30 +08:00

flow_graph_time_tick_node.go

fix: Skip updating checkpoint after dropcollection (#29220 )

2023-12-15 16:04:45 +08:00

flow_graph_time_ticker.go

enhance: create goroutine only once (#28594 )

2023-11-22 10:26:28 +08:00

flow_graph_write_node.go

enhance: Remove commented code and fix naming issue (#28450 )

2023-11-16 00:22:20 +08:00

flush_task_counter_test.go

Fix flushManager.isFull is too slow (#28141 )

2023-11-03 14:42:17 +08:00

flush_task_counter.go

Fix flushManager.isFull is too slow (#28141 )

2023-11-03 14:42:17 +08:00

io_pool_test.go

Fix copylocks linter errors (#26217 )

2023-08-08 20:41:15 +08:00

io_pool.go

Fix getting incorrect CPU num (#28146 )

2023-11-06 06:02:16 +08:00

l0_compactor_test.go

fix: call injectDone after compaction failed (#30277 )

2024-02-01 14:25:04 +08:00

l0_compactor.go

fix: call injectDone after compaction failed (#30277 )

2024-02-01 14:25:04 +08:00

meta_service_test.go

Replace manual composed grpc call with Broker methods (#27676 )

2023-10-13 09:55:34 +08:00

meta_service.go

Replace manual composed grpc call with Broker methods (#27676 )

2023-10-13 09:55:34 +08:00

meta_util.go

Use go-api/v2 for milvus-proto (#24770 )

2023-06-09 01:28:37 +08:00

metrics_info.go

enhance: Add FlowgraphManager interface (#28852 )

2023-11-30 18:42:32 +08:00

mock_channelmanager.go

enhance: Change DN channelmanger into interface (#29307 )

2023-12-27 16:00:48 +08:00

mock_fgmanager.go

enhance: separate serializer logic from sync task (#29413 )

2023-12-26 10:40:47 +08:00

mock_test.go

enhance: make compactor use actual buffer size to decide when to sync (#29945 )

2024-01-13 01:32:52 +08:00

OWNERS

[skip ci]Update OWNERS files (#11898 )

2021-11-16 15:41:11 +08:00

rate_collector_test.go

Move some modules from internal to public package (#22572 )

2023-04-06 19:14:32 +08:00

rate_collector.go

Format the code (#27275 )

2023-09-21 09:45:27 +08:00

README.md

[skip ci]Change uppercase of MinIO (#9539 )

2021-10-09 11:22:58 +08:00

services_test.go

fix: Use channel cp as the dml&start position for import segments (#30107 )

2024-01-22 14:36:55 +08:00

services.go

fix: call injectDone after compaction failed (#30277 )

2024-02-01 14:25:04 +08:00

stats_updater_test.go

Use writebuffer, sync manager refactory in datanode (#28320 )

2023-11-15 15:24:18 +08:00

stats_updater.go

enhance:Remove msgbase unnecessary assignments (#28511 )

2023-11-24 15:02:39 +08:00

timetick_sender_test.go

Refine datanode Timetick Sender (#28393 )

2023-11-13 21:46:20 +08:00

timetick_sender.go

Refine datanode Timetick Sender (#28393 )

2023-11-13 21:46:20 +08:00

util.go

Format the code (#27275 )

2023-09-21 09:45:27 +08:00

README.md

Data Node

DataNode is the component to write insert and delete messages into persistent blob storage, for example MinIO or S3.

Dependency

KV store: a kv store that persists messages into blob storage.
Message stream: receive messages and publish imformation
Root Coordinator: get the latest unique IDs.
Data Coordinator: get the flush information and which message stream to subscribe.