admin/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2025-12-07 09:38:39 +08:00
Code Issues Packages Projects Releases Wiki Activity
milvus/internal/datanode
History
congqixia a040692129
enhance: Use estimated batch size to initalize BF (#29842) 
See also: #27675

The bloom filter set initialized new BF with fixed configured `n`. This
value is always larger than the actual batch size and causes generated
BF using more memory.

This PR make write buffer to initialize BF with estimated batch size
from schema & configuration value.

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-01-10 20:36:50 +08:00
..
allocator
Replace manual composed grpc call with Broker methods (#27676)
2023-10-13 09:55:34 +08:00
broker
enhance:Remove msgbase unnecessary assignments (#28511)
2023-11-24 15:02:39 +08:00
io
feat: Add levelzero compaction in DN (#28470)
2023-11-30 14:30:28 +08:00
iterators
feat: Add levelzero compaction in DN (#28470)
2023-11-30 14:30:28 +08:00
metacache
enhance: Use estimated batch size to initalize BF (#29842)
2024-01-10 20:36:50 +08:00
syncmgr
enhance: reduce SyncTask AllocID call and refine code (#29701)
2024-01-05 10:04:46 +08:00
writebuffer
enhance: Use estimated batch size to initalize BF (#29842)
2024-01-10 20:36:50 +08:00
binlog_io_test.go
enhance:change some logs (#29579)
2024-01-05 16:12:48 +08:00
binlog_io.go
Use writebuffer, sync manager refactory in datanode (#28320)
2023-11-15 15:24:18 +08:00
cache_test.go
Make duplicate flush recalls success and other logic update (#16755)
2022-05-06 17:49:51 +08:00
cache.go
Use typeutil.ConcurrentMap instead of sync.Map (#25846)
2023-07-24 10:23:01 +08:00
channel_checkpoint_updater.go
feat: Introduce channelCheckpointUpdater to reduce goroutine use in ttNode (#28570)
2023-12-12 13:48:42 +08:00
channel_manager_test.go
enhance: Change DN channelmanger into interface (#29307)
2023-12-27 16:00:48 +08:00
channel_manager.go
enhance: Change DN channelmanger into interface (#29307)
2023-12-27 16:00:48 +08:00
compaction_executor_test.go
fix: Separate schedule and check results loop (#28692)
2023-11-29 10:50:29 +08:00
compaction_executor.go
fix: Separate schedule and check results loop (#28692)
2023-11-29 10:50:29 +08:00
compactor_test.go
fix: missing to support compact for Array type (#29505)
2023-12-28 15:42:48 +08:00
compactor.go
fix: missing to support compact for Array type (#29505)
2023-12-28 15:42:48 +08:00
data_node_test.go
enhance: Add FlowgraphManager interface (#28852)
2023-11-30 18:42:32 +08:00
data_node.go
enhance: Change DN channelmanger into interface (#29307)
2023-12-27 16:00:48 +08:00
data_sync_service_test.go
fix: Unstable TestDataSyncService/TestStartStop unit test (#29291)
2023-12-19 23:16:42 +08:00
data_sync_service.go
enhance: separate serializer logic from sync task (#29413)
2023-12-26 10:40:47 +08:00
event_manager_test.go
enhance: Add FlowgraphManager interface (#28852)
2023-11-30 18:42:32 +08:00
event_manager.go
enhance:change some logs (#29579)
2024-01-05 16:12:48 +08:00
flow_graph_dd_node_test.go
Format the code (#27275)
2023-09-21 09:45:27 +08:00
flow_graph_dd_node.go
fix: Skip updating checkpoint after dropcollection (#29220)
2023-12-15 16:04:45 +08:00
flow_graph_dmstream_input_node_test.go
Support to replicate the mq message (#27240)
2023-10-20 14:26:09 +08:00
flow_graph_dmstream_input_node.go
fix: add back existing datanode metrics (#29360)
2023-12-22 14:20:43 +08:00
flow_graph_manager_test.go
enhance: Add FlowgraphManager interface (#28852)
2023-11-30 18:42:32 +08:00
flow_graph_manager.go
enhance: Add FlowgraphManager interface (#28852)
2023-11-30 18:42:32 +08:00
flow_graph_message_test.go
Format the code (#27275)
2023-09-21 09:45:27 +08:00
flow_graph_message.go
Format the code (#27275)
2023-09-21 09:45:27 +08:00
flow_graph_node.go
Make flow graph retry longer (#24355)
2023-05-25 09:53:30 +08:00
flow_graph_time_tick_node.go
fix: Skip updating checkpoint after dropcollection (#29220)
2023-12-15 16:04:45 +08:00
flow_graph_time_ticker.go
enhance: create goroutine only once (#28594)
2023-11-22 10:26:28 +08:00
flow_graph_write_node.go
enhance: Remove commented code and fix naming issue (#28450)
2023-11-16 00:22:20 +08:00
flush_task_counter_test.go
Fix flushManager.isFull is too slow (#28141)
2023-11-03 14:42:17 +08:00
flush_task_counter.go
Fix flushManager.isFull is too slow (#28141)
2023-11-03 14:42:17 +08:00
io_pool_test.go
Fix copylocks linter errors (#26217)
2023-08-08 20:41:15 +08:00
io_pool.go
Fix getting incorrect CPU num (#28146)
2023-11-06 06:02:16 +08:00
l0_compactor_test.go
feat: Add levelzero compaction in DN (#28470)
2023-11-30 14:30:28 +08:00
l0_compactor.go
enhance: Add compaction type label to metrics (#29485)
2023-12-27 15:56:48 +08:00
meta_service_test.go
Replace manual composed grpc call with Broker methods (#27676)
2023-10-13 09:55:34 +08:00
meta_service.go
Replace manual composed grpc call with Broker methods (#27676)
2023-10-13 09:55:34 +08:00
meta_util.go
Use go-api/v2 for milvus-proto (#24770)
2023-06-09 01:28:37 +08:00
metrics_info.go
enhance: Add FlowgraphManager interface (#28852)
2023-11-30 18:42:32 +08:00
mock_channelmanager.go
enhance: Change DN channelmanger into interface (#29307)
2023-12-27 16:00:48 +08:00
mock_fgmanager.go
enhance: separate serializer logic from sync task (#29413)
2023-12-26 10:40:47 +08:00
mock_test.go
enhance: make SyncManager pool size refreshable (#29224)
2023-12-15 09:58:43 +08:00
OWNERS
[skip ci]Update OWNERS files (#11898)
2021-11-16 15:41:11 +08:00
rate_collector_test.go
Move some modules from internal to public package (#22572)
2023-04-06 19:14:32 +08:00
rate_collector.go
Format the code (#27275)
2023-09-21 09:45:27 +08:00
README.md
[skip ci]Change uppercase of MinIO (#9539)
2021-10-09 11:22:58 +08:00
services_test.go
enhance: Change DN channelmanger into interface (#29307)
2023-12-27 16:00:48 +08:00
services.go
enhance: Change DN channelmanger into interface (#29307)
2023-12-27 16:00:48 +08:00
stats_updater_test.go
Use writebuffer, sync manager refactory in datanode (#28320)
2023-11-15 15:24:18 +08:00
stats_updater.go
enhance:Remove msgbase unnecessary assignments (#28511)
2023-11-24 15:02:39 +08:00
timetick_sender_test.go
Refine datanode Timetick Sender (#28393)
2023-11-13 21:46:20 +08:00
timetick_sender.go
Refine datanode Timetick Sender (#28393)
2023-11-13 21:46:20 +08:00
util.go
Format the code (#27275)
2023-09-21 09:45:27 +08:00

README.md

Data Node

DataNode is the component to write insert and delete messages into persistent blob storage, for example MinIO or S3.

Dependency

  • KV store: a kv store that persists messages into blob storage.
  • Message stream: receive messages and publish imformation
  • Root Coordinator: get the latest unique IDs.
  • Data Coordinator: get the flush information and which message stream to subscribe.