admin/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2026-01-07 19:31:51 +08:00
Code Issues Packages Projects Releases Wiki Activity
milvus/internal/datanode
History
XuanYang-cn e6eb6f2c78
enhance: Speed up L0 compaction (#30410) 
This PR changes the following to speed up L0 compaction and
prevent OOM:

1. Lower deltabuf limit to 16MB by default, so that each L0 segment
would be 4X smaller than before.
2. Add BatchProcess, use it if memory is sufficient
3. Iterator will Deserialize when called HasNext to avoid massive memory
peek
4. Add tracing in spiltDelta

See also: #30191

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-02-04 10:49:05 +08:00
..
allocator
Replace manual composed grpc call with Broker methods (#27676)
2023-10-13 09:55:34 +08:00
broker
enhance:don't store logPath in meta to reduce memory (#28873)
2024-01-18 22:06:31 +08:00
importv2
fix: Decoupling importing segment from flush process (#30402)
2024-02-03 13:01:12 +08:00
io
enhance: Speed up L0 compaction (#30410)
2024-02-04 10:49:05 +08:00
iterators
enhance: Speed up L0 compaction (#30410)
2024-02-04 10:49:05 +08:00
metacache
fix: Decoupling importing segment from flush process (#30402)
2024-02-03 13:01:12 +08:00
syncmgr
feat: Add import executor and manager for datanode (#29438)
2024-01-31 20:45:04 +08:00
writebuffer
fix: Decoupling importing segment from flush process (#30402)
2024-02-03 13:01:12 +08:00
binlog_io_test.go
enhance: make compactor use actual buffer size to decide when to sync (#29945)
2024-01-13 01:32:52 +08:00
binlog_io.go
enhance: Add open telemetry tracing for compaction (#30168)
2024-01-23 10:37:00 +08:00
cache_test.go
Make duplicate flush recalls success and other logic update (#16755)
2022-05-06 17:49:51 +08:00
cache.go
Use typeutil.ConcurrentMap instead of sync.Map (#25846)
2023-07-24 10:23:01 +08:00
channel_checkpoint_updater.go
feat: Introduce channelCheckpointUpdater to reduce goroutine use in ttNode (#28570)
2023-12-12 13:48:42 +08:00
channel_manager_test.go
enhance: Change DN channelmanger into interface (#29307)
2023-12-27 16:00:48 +08:00
channel_manager.go
enhance: Change DN channelmanger into interface (#29307)
2023-12-27 16:00:48 +08:00
compaction_executor_test.go
fix: Separate schedule and check results loop (#28692)
2023-11-29 10:50:29 +08:00
compaction_executor.go
fix: call injectDone after compaction failed (#30277)
2024-02-01 14:25:04 +08:00
compactor_test.go
fix: call injectDone after compaction failed (#30277)
2024-02-01 14:25:04 +08:00
compactor.go
fix: call injectDone after compaction failed (#30277)
2024-02-01 14:25:04 +08:00
data_node_test.go
enhance: Add FlowgraphManager interface (#28852)
2023-11-30 18:42:32 +08:00
data_node.go
feat: Add import executor and manager for datanode (#29438)
2024-01-31 20:45:04 +08:00
data_sync_service_test.go
fix: Use channel cp as the dml&start position for import segments (#30107)
2024-01-22 14:36:55 +08:00
data_sync_service.go
enhance:don't store logPath in meta to reduce memory (#28873)
2024-01-18 22:06:31 +08:00
event_manager_test.go
enhance: Add FlowgraphManager interface (#28852)
2023-11-30 18:42:32 +08:00
event_manager.go
fix: panic with datanode negetive wait group counter (#30135)
2024-01-30 18:15:04 +08:00
flow_graph_dd_node_test.go
Format the code (#27275)
2023-09-21 09:45:27 +08:00
flow_graph_dd_node.go
fix: Add more throughput in related metrics (#30038)
2024-01-19 11:34:54 +08:00
flow_graph_dmstream_input_node_test.go
Support to replicate the mq message (#27240)
2023-10-20 14:26:09 +08:00
flow_graph_dmstream_input_node.go
fix: add back existing datanode metrics (#29360)
2023-12-22 14:20:43 +08:00
flow_graph_manager_test.go
enhance: Add FlowgraphManager interface (#28852)
2023-11-30 18:42:32 +08:00
flow_graph_manager.go
enhance: Add FlowgraphManager interface (#28852)
2023-11-30 18:42:32 +08:00
flow_graph_message_test.go
Format the code (#27275)
2023-09-21 09:45:27 +08:00
flow_graph_message.go
Format the code (#27275)
2023-09-21 09:45:27 +08:00
flow_graph_node.go
Make flow graph retry longer (#24355)
2023-05-25 09:53:30 +08:00
flow_graph_time_tick_node.go
fix: Skip updating checkpoint after dropcollection (#29220)
2023-12-15 16:04:45 +08:00
flow_graph_time_ticker.go
enhance: create goroutine only once (#28594)
2023-11-22 10:26:28 +08:00
flow_graph_write_node.go
enhance: Remove commented code and fix naming issue (#28450)
2023-11-16 00:22:20 +08:00
flush_task_counter_test.go
Fix flushManager.isFull is too slow (#28141)
2023-11-03 14:42:17 +08:00
flush_task_counter.go
Fix flushManager.isFull is too slow (#28141)
2023-11-03 14:42:17 +08:00
io_pool_test.go
Fix copylocks linter errors (#26217)
2023-08-08 20:41:15 +08:00
io_pool.go
Fix getting incorrect CPU num (#28146)
2023-11-06 06:02:16 +08:00
l0_compactor_test.go
enhance: Speed up L0 compaction (#30410)
2024-02-04 10:49:05 +08:00
l0_compactor.go
enhance: Speed up L0 compaction (#30410)
2024-02-04 10:49:05 +08:00
meta_service_test.go
Replace manual composed grpc call with Broker methods (#27676)
2023-10-13 09:55:34 +08:00
meta_service.go
Replace manual composed grpc call with Broker methods (#27676)
2023-10-13 09:55:34 +08:00
meta_util.go
Use go-api/v2 for milvus-proto (#24770)
2023-06-09 01:28:37 +08:00
metrics_info.go
enhance: Add FlowgraphManager interface (#28852)
2023-11-30 18:42:32 +08:00
mock_channelmanager.go
enhance: Change DN channelmanger into interface (#29307)
2023-12-27 16:00:48 +08:00
mock_fgmanager.go
enhance: separate serializer logic from sync task (#29413)
2023-12-26 10:40:47 +08:00
mock_test.go
enhance: make compactor use actual buffer size to decide when to sync (#29945)
2024-01-13 01:32:52 +08:00
OWNERS
…
rate_collector_test.go
Move some modules from internal to public package (#22572)
2023-04-06 19:14:32 +08:00
rate_collector.go
Format the code (#27275)
2023-09-21 09:45:27 +08:00
README.md
…
services_test.go
fix: Use channel cp as the dml&start position for import segments (#30107)
2024-01-22 14:36:55 +08:00
services.go
fix: Decoupling importing segment from flush process (#30402)
2024-02-03 13:01:12 +08:00
stats_updater_test.go
Use writebuffer, sync manager refactory in datanode (#28320)
2023-11-15 15:24:18 +08:00
stats_updater.go
enhance:Remove msgbase unnecessary assignments (#28511)
2023-11-24 15:02:39 +08:00
timetick_sender_test.go
Refine datanode Timetick Sender (#28393)
2023-11-13 21:46:20 +08:00
timetick_sender.go
Refine datanode Timetick Sender (#28393)
2023-11-13 21:46:20 +08:00
util.go
Format the code (#27275)
2023-09-21 09:45:27 +08:00

README.md

Data Node

DataNode is the component to write insert and delete messages into persistent blob storage, for example MinIO or S3.

Dependency

  • KV store: a kv store that persists messages into blob storage.
  • Message stream: receive messages and publish imformation
  • Root Coordinator: get the latest unique IDs.
  • Data Coordinator: get the flush information and which message stream to subscribe.