63 Commits

Author	SHA1	Message	Date
chyezh	16b0aee97f	enhance: timetick interceptor optimization (#35287) ... issue: #33285 - remove redundant goroutine by using insepctor. - remove the coutinous non-message timetick persistence - periodically push the time tick forward without persistent timetick message. - add 'message type filter' deliver filter. Signed-off-by: chyezh <chyezh@outlook.com>	2024-08-12 18:58:25 +08:00
chyezh	c725416288	enhance: move streaming proto into pkg (#35284) ... issue: #33285 - move streaming related proto into pkg. - add v2 message type and change flush message into v2 message. Signed-off-by: chyezh <chyezh@outlook.com>	2024-08-07 10:34:16 +08:00
chyezh	14051fed7d	enhance: streaming service client (#34656) ... issue: #33285 - implement streaming service client. - implement producing and consuming service client by streaming coord client and streaming node client. Signed-off-by: chyezh <chyezh@outlook.com>	2024-08-05 21:38:15 +08:00
chyezh	9871966415	enhance: segment alloc interceptor (#34996) ... #33285 - add segment alloc interceptor for streamingnode. - add add manual alloc segment rpc for datacoord. --------- Signed-off-by: chyezh <chyezh@outlook.com>	2024-08-04 07:40:15 +08:00
yihao.dai	a4439cc911	enhance: Implement flusher in streamingNode (#34942) ... - Implement flusher to: - Manage the pipelines (creation, deletion, etc.) - Manage the segment write buffer - Manage sync operation (including receive flushMsg and execute flush) - Add a new `GetChannelRecoveryInfo` RPC in DataCoord. - Reorganize packages: `flushcommon` and `datanode`. issue: https://github.com/milvus-io/milvus/issues/33285 --------- Signed-off-by: bigsheeper <yihao.dai@zilliz.com>	2024-08-02 18:30:23 +08:00
chyezh	39c7e06bc5	enhance: add message and msgstream msgpack adaptor (#34874) ... issue: #33285 - make message builder and message conversion type safe - add adaptor type and function to adapt old msgstream msgpack and message interface --------- Signed-off-by: chyezh <chyezh@outlook.com>	2024-07-22 20:59:42 +08:00
chyezh	cc8f7aa110	fix: streaming service related fix patch (#34696) ... issue: #33285 - add idAlloc interface - fix binary unsafe bug for message - fix service discovery lost when repeated address with different server id --------- Signed-off-by: chyezh <chyezh@outlook.com>	2024-07-16 15:49:38 +08:00
chyezh	fda720b880	enhance: streaming service grpc utilities (#34436) ... issue: #33285 - add two grpc resolver (by session and by streaming coord assignment service) - add one grpc balancer (by serverID and roundrobin) - add lazy conn to avoid block by first service discovery - add some utility function for streaming service Signed-off-by: chyezh <chyezh@outlook.com>	2024-07-15 20:49:38 +08:00
chyezh	dfe0416a70	enhance: implement streaming node server service (#34166) ... issue: #33285 - implement producing and consuming server of message - implement management operation for streaming node server --------- Signed-off-by: chyezh <chyezh@outlook.com>	2024-07-09 07:58:14 +08:00
chyezh	ba04981a43	enhance: implement wal managerment on streaming node (#34153) ... issue: #33285 - add lifetime control for wal. - implement distributed-safe wal manager on streaming node. --------- Signed-off-by: chyezh <chyezh@outlook.com>	2024-07-05 14:29:42 +08:00
chyezh	7611128e57	enhance: wal adaptor implementation (#34122) ... issue: #33285 - add adaptor to implement walimpls into wal interface. - implement timetick sorted and filtering scanner. - add test for wal. --------- Signed-off-by: chyezh <chyezh@outlook.com>	2024-07-04 15:23:08 +08:00
chyezh	d2bc4a53be	enhance: implement rmq and pulsar as wal (#34046) ... issue: #33285 - use reader but not consumer for pulsar - advanced test framework - move some streaming related package into pkg --------- Signed-off-by: chyezh <chyezh@outlook.com>	2024-06-27 15:11:05 +08:00
chyezh	b9237280c2	enhance: wal interface definition (#33745) ... issue: #33285 --------- Signed-off-by: chyezh <chyezh@outlook.com>	2024-06-24 10:34:12 +08:00