congqixia ed89c6a2ee
enhance: make compactor use actual buffer size to decide when to sync (#29945)
See also: #29657

Datanode Compactor use estimated row number from schema to decide when
to sync the batch of data when executing compaction. This est value
could go way from actual size when the schema contains variable field(
say VarChar, JSON, etc.)

This PR make compactor able to check the actual buffer data size and
make it possible to sync when buffer is actually beyong max binglog
size.

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-01-13 01:32:52 +08:00
..
2023-09-21 09:45:27 +08:00
2021-11-16 15:41:11 +08:00
2023-09-21 09:45:27 +08:00
2023-09-21 09:45:27 +08:00

Data Node

DataNode is the component to write insert and delete messages into persistent blob storage, for example MinIO or S3.

Dependency

  • KV store: a kv store that persists messages into blob storage.
  • Message stream: receive messages and publish imformation
  • Root Coordinator: get the latest unique IDs.
  • Data Coordinator: get the flush information and which message stream to subscribe.