milvus/docs/design_docs/datanode_ddl_flush_design_0519_2021.md

DataNode DDL Flush Design

update: 5.19.2021, by Goose

Background

Data Definition Language (DDL) is a language used to define data structures and modify data¹. In Milvus terminology, for instance, CreateCollection and DropPartition etc. are DDL. In order to recover or redo DD operations, DataNode flushes DDLs into persistent storages.

Before this design, DataNode buffers DDL chunks by collection, flushes all buffered data in manul/auto flush.

Now in DataNode Recovery Design, flowgraph : vchannel = 1 : 1, and insert data of one segment is always in one vchannel. So each flowgraph concerns only about ONE specific collection. For DDL channels, one flowgraph only cares about DDL operations of one collection.

Goals

• Flowgraph knows about which segment/collection to concern.
• DDNode update masPositions once it buffers ddl about the collection
• DDNode buffers binlog Paths generated by auto-flush
• In manul-flush, a background flush-complete goroutinue waits for DDNode and InsertBufferNode both done flushing, waiting for both binlog paths.

Detailed design

1. Redisign of DDL binlog paths and etcd paths for these binlog paths

DDL flushes based on a manul flush of a segment.

Former design

# minIO/S3 ddl binlog paths
${tenant}/data_definition_log/${collection_id}/ts/${log_idx}
${tenant}/data_definition_log/${collection_id}/ddl/${log_idx}

# etcd paths for ddl binlog paths
${prefix}/${collectionID}/${idx}

The minIO/S3 ddl binlog paths seems ok, but etcd paths aren't clear, especially when we want to relate a ddl flush to a certain segment flush.

Redesign

# etcd paths for ddl binlog paths
${prefix}/${collectionID}/${segmentID}/${idx}

message SaveBinlogPathsRequest {
    common.MsgBase base = 1;
    int64 segmentID = 2;
    int64 collectionID = 3;
    ID2PathList field2BinlogPaths = 4;
    repeated DDLBinlogMeta = 5;
    repeated internal.MsgPosition start_positions = 7;  
    repeated internal.MsgPosition end_positions = 8; 
 }

TODOs

1. Refactor auto-flush of ddNode
2. Refactor etcd paths

[1]: techterms.com