9 Commits

Author SHA1 Message Date
Zhen Ye
84df80b5e4
enhance: refactor metrics of streaming (#40031)
issue: #38399

- add metrics for broadcaster component.
- add metrics for wal flusher component.
- add metrics for wal interceptors.
- add slow log for wal.
- add more label for some wal metrics. (local or remote/catcup or
tailing...)

Signed-off-by: chyezh <chyezh@outlook.com>
2025-02-25 12:25:56 +08:00
congqixia
cb7f2fa6fd
enhance: Use v2 package name for pkg module (#39990)
Related to #39095

https://go.dev/doc/modules/version-numbers

Update pkg version according to golang dep version convention

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-02-22 23:15:58 +08:00
Zhen Ye
0988807160
enhance: enable write ahead buffer for streaming service (#39771)
issue: #38399

- Make a timetick-commit-based write ahead buffer at write side.
- Add a switchable scanner at read side to transfer the state between
catchup and tailing read

Signed-off-by: chyezh <chyezh@outlook.com>
2025-02-12 20:38:46 +08:00
Zhen Ye
49657c4690
enhance: add create segment message, enable empty segment flush (#37407)
issue: #37172

- add redo interceptor to implement append context refresh. (make new
timetick)
- add create segment handler for flusher.
- make empty segment flushable and directly change it into dropped.
- add create segment message into wal when creating new growing segment.
- make the insert operation into following seq: createSegment -> insert
-> insert -> flushSegment.
- make manual flush into following seq: flushTs -> flushsegment ->
flushsegment -> manualflush.

---------

Signed-off-by: chyezh <chyezh@outlook.com>
2024-11-08 10:16:34 +08:00
Zhen Ye
2ec6e602d6
enhance: add streaming client metrics (#36523)
issue: #33285

---------

Signed-off-by: chyezh <chyezh@outlook.com>
2024-10-08 21:25:19 +08:00
Zhen Ye
4d69898cb2
enhance: support single pchannel level transaction (#35289)
issue: #33285

- support transaction on single wal.
- last confirmed message id can still be used when enable transaction.
- add fence operation for segment allocation interceptor.

---------

Signed-off-by: chyezh <chyezh@outlook.com>
2024-08-19 21:22:56 +08:00
chyezh
39c7e06bc5
enhance: add message and msgstream msgpack adaptor (#34874)
issue: #33285

- make message builder and message conversion type safe
- add adaptor type and function to adapt old msgstream msgpack and
message interface

---------

Signed-off-by: chyezh <chyezh@outlook.com>
2024-07-22 20:59:42 +08:00
chyezh
ba04981a43
enhance: implement wal managerment on streaming node (#34153)
issue: #33285

- add lifetime control for wal.
- implement distributed-safe wal manager on streaming node.

---------

Signed-off-by: chyezh <chyezh@outlook.com>
2024-07-05 14:29:42 +08:00
chyezh
7611128e57
enhance: wal adaptor implementation (#34122)
issue: #33285

- add adaptor to implement walimpls into wal interface.
- implement timetick sorted and filtering scanner.
- add test for wal.

---------

Signed-off-by: chyezh <chyezh@outlook.com>
2024-07-04 15:23:08 +08:00