7 Commits

Author SHA1 Message Date
tinswzy
1427825133
enhance: improve WAL retention strategy (#45350)
issue: #44369 
woodpecker related[ issue:
#59](https://github.com/zilliztech/woodpecker/issues/59)

Refactor the WAL retention logic in Milvus StreamingNode:
- Remove the simple sampling-based truncation mechanism.
- After flush, WAL data is directly truncated.
- The retention control is now delegated to the underlying message queue
(MQ) implementation.

Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
2025-11-23 21:41:05 +08:00
Zhen Ye
593662970b
fix: reuse consumer for backlog clear and use shared consumer (#42822)
issue: #42820

- fix that ro pulsar cannot be closed when upgrading milvus.

Signed-off-by: chyezh <chyezh@outlook.com>
2025-06-19 19:36:48 +08:00
Zhen Ye
1f66b650e9
fix: pulsar cannot work properly if backlog exceed (#42653)
issue: #42649

- the sync operation of different pchannel is concurrent now.
- add a option to notify the backlog clear automatically.
- make pulsar walimpls can be recovered from backlog exceed.

Signed-off-by: chyezh <chyezh@outlook.com>
2025-06-13 14:28:37 +08:00
Zhen Ye
52950ce392
enhance: add pulsar truncate api to protect pulsar unconsumed message (#41724)
issue: #41465

- implement truncate api for pulsar based on durable subscription.
- truncate api can only be called if wal is read-write.

Signed-off-by: chyezh <chyezh@outlook.com>
2025-05-11 20:50:55 +08:00
Zhen Ye
6982f007e2
enhance: add walimpls access mode options (#40591)
issue: #40532

Signed-off-by: chyezh <chyezh@outlook.com>
2025-03-14 10:58:11 +08:00
congqixia
cb7f2fa6fd
enhance: Use v2 package name for pkg module (#39990)
Related to #39095

https://go.dev/doc/modules/version-numbers

Update pkg version according to golang dep version convention

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-02-22 23:15:58 +08:00
chyezh
d2bc4a53be
enhance: implement rmq and pulsar as wal (#34046)
issue: #33285

- use reader but not consumer for pulsar
- advanced test framework
- move some streaming related package into pkg

---------

Signed-off-by: chyezh <chyezh@outlook.com>
2024-06-27 15:11:05 +08:00