tinswzy
6fa68c1f16
enhance: Support Woodpecker as a WAL storage option for Milvus ( #41095 )
...
#40916 Support Woodpecker as a WAL storage option for Milvus
Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
2025-04-20 22:22:42 +08:00
Sungyun Hur
fbd595eae1
fix: correct a minor typo in log message of kafka client ( #41216 )
...
fixes a minor typo that notify users that some special kafka configs are
overwritten in Milvus.
Signed-off-by: lambert <lambert@daangn.com>
2025-04-16 21:14:20 +08:00
SimFG
a308d2c886
fix: get replicate channel position ( #41188 )
...
- issue: #41187
Signed-off-by: SimFG <bang.fu@zilliz.com>
2025-04-10 15:14:28 +08:00
Zhen Ye
b03e60558a
enhance: add proxy and datanode checker when wal balance startup ( #40877 )
...
issue: #40532
- balance should enable only when there's no proxy and datanode which
version is lower than 2.6.0
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2025-04-01 11:24:22 +08:00
yihao.dai
d8d1dcf076
fix: Fix dispatcher deregister and seek ( #40860 )
...
1. Fix deregister dispatcher concurrency. (Keep sure the same logic as
2.5 branch)
2. Fix seek if includeCurrentMsg. (This is only needed by CDC, so
there's no need to pick to 2.5 branch)
issue: issue: https://github.com/milvus-io/milvus/issues/39862
pr: https://github.com/milvus-io/milvus/pull/39863
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-03-31 11:38:19 +08:00
Ted Xu
688505ab1c
enhance: cleanup lint check exclusions ( #40829 )
...
See: #40828
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2025-03-21 18:12:14 +08:00
yihao.dai
8eb537fdd0
fix: Fix messages loss during seek ( #40735 )
...
issue: https://github.com/milvus-io/milvus/issues/40733
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-03-20 20:18:11 +08:00
yihao.dai
a33c9372ce
fix: Fix channel not balance on datanodes ( #40422 )
...
1. Prevent channels from being assigned to only one datanode during
datacoord startup.
2. Optimize the channel assignment policy by considering newly assigned
channels.
3. Make msgdispatcher manager lock-free.
issue: https://github.com/milvus-io/milvus/issues/40421 ,
https://github.com/milvus-io/milvus/issues/37630
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-03-11 14:56:16 +08:00
aoiasd
3afc3a83ff
fix: rocksmq consumer register not concurrent safe ( #39387 )
...
relate: https://github.com/milvus-io/milvus/issues/39336
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2025-03-11 00:56:04 +08:00
Xiaofan
1c9d43ee9f
fix: rockmq race condition ( #40482 )
...
fix #40481
Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
2025-03-10 10:54:04 +08:00
yihao.dai
004a1875dc
enhance: Introduce batch subscription in msgdispatcher ( #39863 )
...
Introduce a batch subscription mechanism in msgdispatcher: the
msgdispatcher now includes a vchannel watch task queue, where all
vchannels in the queue will subscribe to the MQ only once and pull
messages from the oldest vchannel checkpoint to the latest.
issue: https://github.com/milvus-io/milvus/issues/39862
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-03-05 14:38:02 +08:00
Zhen Ye
f47ab31f23
enhance: remove redundant resource key watch operation, just keep consistency of wal ( #40235 )
...
issue: #38399
related PR: #39522
- Just implement exclusive broadcaster between broadcast message with
same resource key to keep same order in different wal.
- After simplify the broadcast model, original watch-based broadcast is
too complicated and redundant, remove it.
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2025-03-03 14:40:05 +08:00
aoiasd
d67d6b7567
fix: main dispatcher not terminate cause messsage stream leak. ( #40061 )
...
Main dispatcher will leak when we remove solo dispatcher in the end.
https://github.com/milvus-io/milvus/issues/40046
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2025-02-27 16:15:58 +08:00
congqixia
cb7f2fa6fd
enhance: Use v2 package name for pkg module ( #39990 )
...
Related to #39095
https://go.dev/doc/modules/version-numbers
Update pkg version according to golang dep version convention
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-02-22 23:15:58 +08:00
yihao.dai
c917fe4782
fix: Fix failed to seek to earliest position ( #39965 )
...
If it is the earliest message ID, skip the seek to prevent failure.
issue: https://github.com/milvus-io/milvus/issues/39964
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-02-19 00:30:51 +08:00
Zhen Ye
ae700e7519
enhance: make compatitle with old msgstream for new streaming service ( #39943 )
...
issue: #38399
Signed-off-by: chyezh <chyezh@outlook.com>
2025-02-18 11:21:08 +08:00
SimFG
047254665d
feat: support to replicate import msg ( #39171 )
...
- issue: #39849
---------
Signed-off-by: SimFG <bang.fu@zilliz.com>
Signed-off-by: chyezh <chyezh@outlook.com>
Co-authored-by: chyezh <chyezh@outlook.com>
2025-02-16 00:08:13 +08:00
aoiasd
24d2bbc441
enhance: unmashall ts msg in dispatcher instead in msgstream ( #38656 )
...
relate: https://github.com/milvus-io/milvus/issues/38655
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2025-02-14 12:04:13 +08:00
yihao.dai
8288c06e3d
fix: Fix flowgraph leak ( #39656 )
...
Consider lag targets when determining whether to remove the dispatcher
manager.
issue: https://github.com/milvus-io/milvus/issues/39642
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-02-07 16:16:44 +08:00
Zhen Ye
a816a03351
fix: unsafe concurrent consuming api of rocksmq ( #39544 )
...
issue: #38966
Signed-off-by: chyezh <chyezh@outlook.com>
2025-02-05 09:51:10 +08:00
Duc Le Tu
2a962ad1ec
fix: apply custom producer config for kafkaHealthCheck ( #39283 )
...
issue: https://github.com/milvus-io/milvus/issues/39287
KafkaHealthCheck init without ProducerExtraConfig. This PR fix that.
---------
Signed-off-by: DLT1412 <tuduc93@gmail.com>
Co-authored-by: DucLT <duc.le1@be.com.vn>
2025-01-16 19:31:02 +08:00
yihao.dai
a5a83a0904
fix: Fix consume blocked due to too many consumers ( #38455 )
...
This PR limits the maximum number of consumers per pchannel to 10 for
each QueryNode and DataNode.
issue: https://github.com/milvus-io/milvus/issues/37630
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-01-15 21:37:01 +08:00
Zhen Ye
bb8d1ab3bf
enhance: make new go package to manage proto ( #39114 )
...
issue: #39095
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2025-01-10 10:49:01 +08:00
tinswzy
11f8f4a378
enhance: add intent ctx to rootcoord init ( #38439 )
...
issue: #35917
add intent ctx to rootcoord initialization process
Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
2024-12-31 18:22:53 +08:00
Zhen Ye
56c5b66619
fix: the close operation of rmq consumer is not sync ( #38734 )
...
issue: #38399
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-12-30 16:04:51 +08:00
Zhen Ye
285289d5b0
enhance: implement kafka for wal ( #38598 )
...
issue: #38399
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-12-25 10:22:50 +08:00
Zhen Ye
69a9fd6ead
enhance: enable rmq for streaming ( #38669 )
...
issue: #38399
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-12-24 20:24:48 +08:00
Zhen Ye
b18a3cfce2
fix: kafka use shared channel to receive produce result ( #38532 )
...
issue: #38531
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-12-17 23:10:45 +08:00
SimFG
2afe2eaf3e
feat: support to replicate collection when the services contains the system tt msg ( #37559 )
...
- issue: #37105
---------
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-12-17 09:08:46 +08:00
SimFG
fa8ac09550
fix: the issue of replicate message exception when the ttMsgEnable config is changed dynamically ( #38178 )
...
- issue: #38177
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-12-14 23:24:51 +08:00
tinswzy
27229f7907
enhance: refine exists log print with ctx ( #38080 )
...
issue: #35917
Refines exists log print with ctx
Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
2024-12-14 22:36:44 +08:00
jaime
37a52286b1
fix: stream connection leak in case of error ( #38320 )
...
issue: #38318
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-12-11 15:38:49 +08:00
jaime
a1e14d62c7
enhance: refine lock granularity for produers in msgstream ( #38262 )
...
issue: #38261
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-12-06 17:14:40 +08:00
jaime
edabfa8fa6
enhance: Remove pool from produer and consumer in kafka client ( #38264 )
...
issue: #38263
Revert "fix: Move init kafka pool into once (#37786 )"
Revert "enhance: Use pool to limit kafka cgo thread number (#37744 )"
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-12-06 14:08:40 +08:00
tinswzy
7944538ade
enhance: Add ctx param to KV operation interfaces ( #38154 )
...
issue: #35917
Refine KV operation interfaces by adding a ctx param
Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
2024-12-05 15:16:41 +08:00
tinswzy
5768dbbb5d
enhance: refine pular related mq interfaces ( #38007 )
...
issue: #35917
Refines the pulsar-related mq APIs to allow the ctx to be passed down
Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
2024-12-04 20:50:39 +08:00
congqixia
97b7ebbc08
fix: Move init kafka pool into once ( #37786 )
...
Introduced by #37744
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-11-18 21:24:48 +08:00
congqixia
3743fc2b00
enhance: Use pool to limit kafka cgo thread number ( #37744 )
...
Related to #37665
Thread number went rocket high when there is lots of kafka consumers on
datanode. Since the internal implementation is CGO, using which directly
will make cgo thread leaked.
This PR add a worker pool for kafka API utilzing CGO calls to limit
thread number.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-11-18 10:42:31 +08:00
congqixia
4820dd76e5
fix: Close leaked health check kafka producer ( #37728 )
...
Related to #36822
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-11-16 11:20:32 +08:00
Zhen Ye
cd181e4c6d
fix: make NewDataSyncService idempotent of dispatcher ( #37576 )
...
issue: #37547
Signed-off-by: chyezh <chyezh@outlook.com>
2024-11-14 11:00:31 +08:00
wei liu
00f6d0ec51
fix: watch channel stuck due to misuse of timer.Reset ( #37433 )
...
issue: #37166
cause the misuse of timer.Reset, which cause dispatcher failed to send
msg to virtual channel buffer, and dispatcher do splitting again and
again, which hold the dispatcher manager's lock, block watching channel
progress.
This PR fix the misuse of timer.Reset
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-11-07 14:34:24 +08:00
jaime
4746f47282
feat: management WebUI homepage ( #36822 )
...
issue: #36784
1. Implement an embedded web server for WebUI access.
2. Complete the homepage development.
Home page demo:
<img width="2177" alt="iShot_2024-10-10_17 57 34"
src="https://github.com/user-attachments/assets/38539917-ce09-4e54-a5b5-7f4f7eaac353 ">
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-10-23 11:29:28 +08:00
SimFG
821750d902
enhance: force to stop buffer message when receiving the drop collection message ( #36916 )
...
/kind improvement
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-10-17 12:53:29 +08:00
Bingyi Sun
6851738fd1
fix: fix make generate-mockery panic with go1.22 ( #36830 )
...
https://github.com/milvus-io/milvus/issues/36831
Fix `make generate-mockery` panic.
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-10-17 12:11:31 +08:00
aoiasd
2c3a8b7dea
enhance: fix skip msg log with empty position info ( #36626 )
...
All skiped message position info was nil, should print BeginTs or
MessageID instead.
relate: https://github.com/milvus-io/milvus/issues/36569
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-10-09 12:01:20 +08:00
Zhen Ye
addced27f9
fix: rocksmq consume too slow when the channel is full ( #36617 )
...
issue: #36569
Signed-off-by: chyezh <chyezh@outlook.com>
2024-10-09 11:59:31 +08:00
smellthemoon
2055df81aa
enhance: upgrade pulsar-client-go to 0.12.1 ( #36615 )
...
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-09-30 17:13:16 +08:00
SimFG
9c1772f659
enhance: avoid to create many timer object in the target ( #36570 )
...
/kind improvement
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-09-29 19:23:16 +08:00
SimFG
ddadefcb62
enhance: get msg type from the msg header to reduce the Unmarshal usage ( #36409 )
...
/kind improvement
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-09-24 10:19:20 +08:00
SimFG
c50fe71163
fix: long buffering causes mq to be unable to receive messages. ( #36420 )
...
- issue: #36397
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-09-23 16:33:18 +08:00
