nish112022
1e704ecf9f
fix: Add Kafka buffer size limit to prevent DataNode OOM ( #44106 )
...
issue: https://github.com/milvus-io/milvus/issues/44105
- I have added support to set this property
**queued.max.messages.kbytes** in kafka consumers from the user side.
- It limits the size (in KB) of the consumer’s local message queue
(buffer) where messages are temporarily stored after being fetched from
Kafka but before your application actually processes them
---------
Signed-off-by: Nischay Yadav <Nischay.Yadav@ibm.com>
2025-09-01 18:19:21 +08:00
Zhen Ye
5bdc593b8a
enhance: use v0.15.1 official pulsar client and add logging for pulsar client ( #43913 )
...
issue: #43785
- pulsar client will print log into milvus logger now.
- pulsar client open the metric by default.
- upgrade the pulsar client to v0.15.1, and use offical repo.
- the fixing of milvus-io/pulsar-client-go is already covered by
official v0.15.1.
Signed-off-by: chyezh <chyezh@outlook.com>
2025-08-26 16:45:53 +08:00
Zhen Ye
2fd8f910b0
fix: data duplicated when msgdispatcher make splitting ( #42827 )
...
issue: #41570
Signed-off-by: chyezh <chyezh@outlook.com>
2025-06-19 16:32:39 +08:00
Zhen Ye
4f5409e1fe
fix: panic when schema change ( #42727 )
...
issue: #42723
Signed-off-by: chyezh <chyezh@outlook.com>
2025-06-13 17:43:07 +08:00
Zhen Ye
ca48603f35
fix: msg dispatcher lost data at streaming service ( #42670 )
...
issue: #41570
Signed-off-by: chyezh <chyezh@outlook.com>
2025-06-13 11:54:36 +08:00
yihao.dai
ee9d08375d
enhance: Accelerate dispatcher building ( #42500 )
...
Reduce check interval to accelerate dispatcher building.
issue: https://github.com/milvus-io/milvus/issues/42067
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-06-05 16:36:32 +08:00
Zhen Ye
b94cee2413
fix: growing segment from old arch is not flushed after upgrading ( #42164 )
...
issue: #42162
- enhance: add read ahead buffer size issue #42129
- fix: rocksmq consumer's close operation may get stucked
- fix: growing segment from old arch is not flushed after upgrading
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2025-05-29 23:00:28 +08:00
SimFG
9f866dd7c3
enhance: add privilege group message types and corresponding tests ( #41939 )
...
- issue: #41938
Signed-off-by: SimFG <bang.fu@zilliz.com>
2025-05-21 11:12:28 +08:00
SimFG
dfd2548c1b
enhance: support more ddl apis to replicate according to cdc ( #41678 )
...
- issue: #41677
Signed-off-by: SimFG <bang.fu@zilliz.com>
2025-05-08 20:02:58 +08:00
Zhen Ye
de8f0af20d
enhance: use dispatcher at delegator when enable streaming ( #41266 )
...
issue: #38399
- add an adaptor type to adapt the streaming service client and
msgstream client to reuse the msgdispatcher.
Signed-off-by: chyezh <chyezh@outlook.com>
2025-05-06 01:12:53 +08:00
SimFG
623c728dac
enhance: add timeout for message reception in mqMsgStream ( #41602 )
...
/kind improvement
Signed-off-by: SimFG <bang.fu@zilliz.com>
2025-05-06 00:50:52 +08:00
SimFG
3bd6268d3c
enhance: Implement OperatePrivilegeV2 message handling and unmarshal support ( #41355 )
...
- issue: #41353
Signed-off-by: SimFG <bang.fu@zilliz.com>
2025-04-30 12:02:53 +08:00
Zhen Ye
9cb5271027
enhance: remove support of embeded nats mq ( #41565 )
...
issue: #41564
Signed-off-by: chyezh <chyezh@outlook.com>
2025-04-28 23:06:49 +08:00
Zhen Ye
5fd47c3c89
fix: mockery too unavailable after upgrade golang version ( #41481 )
...
issue: #41291
pr: #41318
Signed-off-by: chyezh <chyezh@outlook.com>
2025-04-24 10:46:43 +08:00
SimFG
91d40fa558
fix: Update logging context and upgrade dependencies ( #41318 )
...
- issue: #41291
---------
Signed-off-by: SimFG <bang.fu@zilliz.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2025-04-23 10:52:38 +08:00
tinswzy
6fa68c1f16
enhance: Support Woodpecker as a WAL storage option for Milvus ( #41095 )
...
#40916 Support Woodpecker as a WAL storage option for Milvus
Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
2025-04-20 22:22:42 +08:00
Sungyun Hur
fbd595eae1
fix: correct a minor typo in log message of kafka client ( #41216 )
...
fixes a minor typo that notify users that some special kafka configs are
overwritten in Milvus.
Signed-off-by: lambert <lambert@daangn.com>
2025-04-16 21:14:20 +08:00
SimFG
a308d2c886
fix: get replicate channel position ( #41188 )
...
- issue: #41187
Signed-off-by: SimFG <bang.fu@zilliz.com>
2025-04-10 15:14:28 +08:00
Zhen Ye
b03e60558a
enhance: add proxy and datanode checker when wal balance startup ( #40877 )
...
issue: #40532
- balance should enable only when there's no proxy and datanode which
version is lower than 2.6.0
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2025-04-01 11:24:22 +08:00
yihao.dai
d8d1dcf076
fix: Fix dispatcher deregister and seek ( #40860 )
...
1. Fix deregister dispatcher concurrency. (Keep sure the same logic as
2.5 branch)
2. Fix seek if includeCurrentMsg. (This is only needed by CDC, so
there's no need to pick to 2.5 branch)
issue: issue: https://github.com/milvus-io/milvus/issues/39862
pr: https://github.com/milvus-io/milvus/pull/39863
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-03-31 11:38:19 +08:00
Ted Xu
688505ab1c
enhance: cleanup lint check exclusions ( #40829 )
...
See: #40828
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2025-03-21 18:12:14 +08:00
yihao.dai
8eb537fdd0
fix: Fix messages loss during seek ( #40735 )
...
issue: https://github.com/milvus-io/milvus/issues/40733
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-03-20 20:18:11 +08:00
yihao.dai
a33c9372ce
fix: Fix channel not balance on datanodes ( #40422 )
...
1. Prevent channels from being assigned to only one datanode during
datacoord startup.
2. Optimize the channel assignment policy by considering newly assigned
channels.
3. Make msgdispatcher manager lock-free.
issue: https://github.com/milvus-io/milvus/issues/40421 ,
https://github.com/milvus-io/milvus/issues/37630
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-03-11 14:56:16 +08:00
aoiasd
3afc3a83ff
fix: rocksmq consumer register not concurrent safe ( #39387 )
...
relate: https://github.com/milvus-io/milvus/issues/39336
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2025-03-11 00:56:04 +08:00
Xiaofan
1c9d43ee9f
fix: rockmq race condition ( #40482 )
...
fix #40481
Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
2025-03-10 10:54:04 +08:00
yihao.dai
004a1875dc
enhance: Introduce batch subscription in msgdispatcher ( #39863 )
...
Introduce a batch subscription mechanism in msgdispatcher: the
msgdispatcher now includes a vchannel watch task queue, where all
vchannels in the queue will subscribe to the MQ only once and pull
messages from the oldest vchannel checkpoint to the latest.
issue: https://github.com/milvus-io/milvus/issues/39862
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-03-05 14:38:02 +08:00
Zhen Ye
f47ab31f23
enhance: remove redundant resource key watch operation, just keep consistency of wal ( #40235 )
...
issue: #38399
related PR: #39522
- Just implement exclusive broadcaster between broadcast message with
same resource key to keep same order in different wal.
- After simplify the broadcast model, original watch-based broadcast is
too complicated and redundant, remove it.
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2025-03-03 14:40:05 +08:00
aoiasd
d67d6b7567
fix: main dispatcher not terminate cause messsage stream leak. ( #40061 )
...
Main dispatcher will leak when we remove solo dispatcher in the end.
https://github.com/milvus-io/milvus/issues/40046
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2025-02-27 16:15:58 +08:00
congqixia
cb7f2fa6fd
enhance: Use v2 package name for pkg module ( #39990 )
...
Related to #39095
https://go.dev/doc/modules/version-numbers
Update pkg version according to golang dep version convention
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-02-22 23:15:58 +08:00
yihao.dai
c917fe4782
fix: Fix failed to seek to earliest position ( #39965 )
...
If it is the earliest message ID, skip the seek to prevent failure.
issue: https://github.com/milvus-io/milvus/issues/39964
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-02-19 00:30:51 +08:00
Zhen Ye
ae700e7519
enhance: make compatitle with old msgstream for new streaming service ( #39943 )
...
issue: #38399
Signed-off-by: chyezh <chyezh@outlook.com>
2025-02-18 11:21:08 +08:00
SimFG
047254665d
feat: support to replicate import msg ( #39171 )
...
- issue: #39849
---------
Signed-off-by: SimFG <bang.fu@zilliz.com>
Signed-off-by: chyezh <chyezh@outlook.com>
Co-authored-by: chyezh <chyezh@outlook.com>
2025-02-16 00:08:13 +08:00
aoiasd
24d2bbc441
enhance: unmashall ts msg in dispatcher instead in msgstream ( #38656 )
...
relate: https://github.com/milvus-io/milvus/issues/38655
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2025-02-14 12:04:13 +08:00
yihao.dai
8288c06e3d
fix: Fix flowgraph leak ( #39656 )
...
Consider lag targets when determining whether to remove the dispatcher
manager.
issue: https://github.com/milvus-io/milvus/issues/39642
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-02-07 16:16:44 +08:00
Zhen Ye
a816a03351
fix: unsafe concurrent consuming api of rocksmq ( #39544 )
...
issue: #38966
Signed-off-by: chyezh <chyezh@outlook.com>
2025-02-05 09:51:10 +08:00
Duc Le Tu
2a962ad1ec
fix: apply custom producer config for kafkaHealthCheck ( #39283 )
...
issue: https://github.com/milvus-io/milvus/issues/39287
KafkaHealthCheck init without ProducerExtraConfig. This PR fix that.
---------
Signed-off-by: DLT1412 <tuduc93@gmail.com>
Co-authored-by: DucLT <duc.le1@be.com.vn>
2025-01-16 19:31:02 +08:00
yihao.dai
a5a83a0904
fix: Fix consume blocked due to too many consumers ( #38455 )
...
This PR limits the maximum number of consumers per pchannel to 10 for
each QueryNode and DataNode.
issue: https://github.com/milvus-io/milvus/issues/37630
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-01-15 21:37:01 +08:00
Zhen Ye
bb8d1ab3bf
enhance: make new go package to manage proto ( #39114 )
...
issue: #39095
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2025-01-10 10:49:01 +08:00
tinswzy
11f8f4a378
enhance: add intent ctx to rootcoord init ( #38439 )
...
issue: #35917
add intent ctx to rootcoord initialization process
Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
2024-12-31 18:22:53 +08:00
Zhen Ye
56c5b66619
fix: the close operation of rmq consumer is not sync ( #38734 )
...
issue: #38399
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-12-30 16:04:51 +08:00
Zhen Ye
285289d5b0
enhance: implement kafka for wal ( #38598 )
...
issue: #38399
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-12-25 10:22:50 +08:00
Zhen Ye
69a9fd6ead
enhance: enable rmq for streaming ( #38669 )
...
issue: #38399
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-12-24 20:24:48 +08:00
Zhen Ye
b18a3cfce2
fix: kafka use shared channel to receive produce result ( #38532 )
...
issue: #38531
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-12-17 23:10:45 +08:00
SimFG
2afe2eaf3e
feat: support to replicate collection when the services contains the system tt msg ( #37559 )
...
- issue: #37105
---------
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-12-17 09:08:46 +08:00
SimFG
fa8ac09550
fix: the issue of replicate message exception when the ttMsgEnable config is changed dynamically ( #38178 )
...
- issue: #38177
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-12-14 23:24:51 +08:00
tinswzy
27229f7907
enhance: refine exists log print with ctx ( #38080 )
...
issue: #35917
Refines exists log print with ctx
Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
2024-12-14 22:36:44 +08:00
jaime
37a52286b1
fix: stream connection leak in case of error ( #38320 )
...
issue: #38318
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-12-11 15:38:49 +08:00
jaime
a1e14d62c7
enhance: refine lock granularity for produers in msgstream ( #38262 )
...
issue: #38261
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-12-06 17:14:40 +08:00
jaime
edabfa8fa6
enhance: Remove pool from produer and consumer in kafka client ( #38264 )
...
issue: #38263
Revert "fix: Move init kafka pool into once (#37786 )"
Revert "enhance: Use pool to limit kafka cgo thread number (#37744 )"
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-12-06 14:08:40 +08:00
tinswzy
7944538ade
enhance: Add ctx param to KV operation interfaces ( #38154 )
...
issue: #35917
Refine KV operation interfaces by adding a ctx param
Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
2024-12-05 15:16:41 +08:00
