aoiasd
d67d6b7567
fix: main dispatcher not terminate cause messsage stream leak. ( #40061 )
...
Main dispatcher will leak when we remove solo dispatcher in the end.
https://github.com/milvus-io/milvus/issues/40046
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2025-02-27 16:15:58 +08:00
congqixia
cb7f2fa6fd
enhance: Use v2 package name for pkg module ( #39990 )
...
Related to #39095
https://go.dev/doc/modules/version-numbers
Update pkg version according to golang dep version convention
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-02-22 23:15:58 +08:00
yihao.dai
c917fe4782
fix: Fix failed to seek to earliest position ( #39965 )
...
If it is the earliest message ID, skip the seek to prevent failure.
issue: https://github.com/milvus-io/milvus/issues/39964
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-02-19 00:30:51 +08:00
Zhen Ye
ae700e7519
enhance: make compatitle with old msgstream for new streaming service ( #39943 )
...
issue: #38399
Signed-off-by: chyezh <chyezh@outlook.com>
2025-02-18 11:21:08 +08:00
SimFG
047254665d
feat: support to replicate import msg ( #39171 )
...
- issue: #39849
---------
Signed-off-by: SimFG <bang.fu@zilliz.com>
Signed-off-by: chyezh <chyezh@outlook.com>
Co-authored-by: chyezh <chyezh@outlook.com>
2025-02-16 00:08:13 +08:00
aoiasd
24d2bbc441
enhance: unmashall ts msg in dispatcher instead in msgstream ( #38656 )
...
relate: https://github.com/milvus-io/milvus/issues/38655
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2025-02-14 12:04:13 +08:00
yihao.dai
8288c06e3d
fix: Fix flowgraph leak ( #39656 )
...
Consider lag targets when determining whether to remove the dispatcher
manager.
issue: https://github.com/milvus-io/milvus/issues/39642
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-02-07 16:16:44 +08:00
Zhen Ye
a816a03351
fix: unsafe concurrent consuming api of rocksmq ( #39544 )
...
issue: #38966
Signed-off-by: chyezh <chyezh@outlook.com>
2025-02-05 09:51:10 +08:00
Duc Le Tu
2a962ad1ec
fix: apply custom producer config for kafkaHealthCheck ( #39283 )
...
issue: https://github.com/milvus-io/milvus/issues/39287
KafkaHealthCheck init without ProducerExtraConfig. This PR fix that.
---------
Signed-off-by: DLT1412 <tuduc93@gmail.com>
Co-authored-by: DucLT <duc.le1@be.com.vn>
2025-01-16 19:31:02 +08:00
yihao.dai
a5a83a0904
fix: Fix consume blocked due to too many consumers ( #38455 )
...
This PR limits the maximum number of consumers per pchannel to 10 for
each QueryNode and DataNode.
issue: https://github.com/milvus-io/milvus/issues/37630
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-01-15 21:37:01 +08:00
Zhen Ye
bb8d1ab3bf
enhance: make new go package to manage proto ( #39114 )
...
issue: #39095
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2025-01-10 10:49:01 +08:00
tinswzy
11f8f4a378
enhance: add intent ctx to rootcoord init ( #38439 )
...
issue: #35917
add intent ctx to rootcoord initialization process
Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
2024-12-31 18:22:53 +08:00
Zhen Ye
56c5b66619
fix: the close operation of rmq consumer is not sync ( #38734 )
...
issue: #38399
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-12-30 16:04:51 +08:00
Zhen Ye
285289d5b0
enhance: implement kafka for wal ( #38598 )
...
issue: #38399
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-12-25 10:22:50 +08:00
Zhen Ye
69a9fd6ead
enhance: enable rmq for streaming ( #38669 )
...
issue: #38399
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-12-24 20:24:48 +08:00
Zhen Ye
b18a3cfce2
fix: kafka use shared channel to receive produce result ( #38532 )
...
issue: #38531
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-12-17 23:10:45 +08:00
SimFG
2afe2eaf3e
feat: support to replicate collection when the services contains the system tt msg ( #37559 )
...
- issue: #37105
---------
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-12-17 09:08:46 +08:00
SimFG
fa8ac09550
fix: the issue of replicate message exception when the ttMsgEnable config is changed dynamically ( #38178 )
...
- issue: #38177
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-12-14 23:24:51 +08:00
tinswzy
27229f7907
enhance: refine exists log print with ctx ( #38080 )
...
issue: #35917
Refines exists log print with ctx
Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
2024-12-14 22:36:44 +08:00
jaime
37a52286b1
fix: stream connection leak in case of error ( #38320 )
...
issue: #38318
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-12-11 15:38:49 +08:00
jaime
a1e14d62c7
enhance: refine lock granularity for produers in msgstream ( #38262 )
...
issue: #38261
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-12-06 17:14:40 +08:00
jaime
edabfa8fa6
enhance: Remove pool from produer and consumer in kafka client ( #38264 )
...
issue: #38263
Revert "fix: Move init kafka pool into once (#37786 )"
Revert "enhance: Use pool to limit kafka cgo thread number (#37744 )"
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-12-06 14:08:40 +08:00
tinswzy
7944538ade
enhance: Add ctx param to KV operation interfaces ( #38154 )
...
issue: #35917
Refine KV operation interfaces by adding a ctx param
Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
2024-12-05 15:16:41 +08:00
tinswzy
5768dbbb5d
enhance: refine pular related mq interfaces ( #38007 )
...
issue: #35917
Refines the pulsar-related mq APIs to allow the ctx to be passed down
Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
2024-12-04 20:50:39 +08:00
congqixia
97b7ebbc08
fix: Move init kafka pool into once ( #37786 )
...
Introduced by #37744
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-11-18 21:24:48 +08:00
congqixia
3743fc2b00
enhance: Use pool to limit kafka cgo thread number ( #37744 )
...
Related to #37665
Thread number went rocket high when there is lots of kafka consumers on
datanode. Since the internal implementation is CGO, using which directly
will make cgo thread leaked.
This PR add a worker pool for kafka API utilzing CGO calls to limit
thread number.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-11-18 10:42:31 +08:00
congqixia
4820dd76e5
fix: Close leaked health check kafka producer ( #37728 )
...
Related to #36822
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-11-16 11:20:32 +08:00
Zhen Ye
cd181e4c6d
fix: make NewDataSyncService idempotent of dispatcher ( #37576 )
...
issue: #37547
Signed-off-by: chyezh <chyezh@outlook.com>
2024-11-14 11:00:31 +08:00
wei liu
00f6d0ec51
fix: watch channel stuck due to misuse of timer.Reset ( #37433 )
...
issue: #37166
cause the misuse of timer.Reset, which cause dispatcher failed to send
msg to virtual channel buffer, and dispatcher do splitting again and
again, which hold the dispatcher manager's lock, block watching channel
progress.
This PR fix the misuse of timer.Reset
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-11-07 14:34:24 +08:00
jaime
4746f47282
feat: management WebUI homepage ( #36822 )
...
issue: #36784
1. Implement an embedded web server for WebUI access.
2. Complete the homepage development.
Home page demo:
<img width="2177" alt="iShot_2024-10-10_17 57 34"
src="https://github.com/user-attachments/assets/38539917-ce09-4e54-a5b5-7f4f7eaac353 ">
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-10-23 11:29:28 +08:00
SimFG
821750d902
enhance: force to stop buffer message when receiving the drop collection message ( #36916 )
...
/kind improvement
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-10-17 12:53:29 +08:00
Bingyi Sun
6851738fd1
fix: fix make generate-mockery panic with go1.22 ( #36830 )
...
https://github.com/milvus-io/milvus/issues/36831
Fix `make generate-mockery` panic.
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-10-17 12:11:31 +08:00
aoiasd
2c3a8b7dea
enhance: fix skip msg log with empty position info ( #36626 )
...
All skiped message position info was nil, should print BeginTs or
MessageID instead.
relate: https://github.com/milvus-io/milvus/issues/36569
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-10-09 12:01:20 +08:00
Zhen Ye
addced27f9
fix: rocksmq consume too slow when the channel is full ( #36617 )
...
issue: #36569
Signed-off-by: chyezh <chyezh@outlook.com>
2024-10-09 11:59:31 +08:00
smellthemoon
2055df81aa
enhance: upgrade pulsar-client-go to 0.12.1 ( #36615 )
...
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-09-30 17:13:16 +08:00
SimFG
9c1772f659
enhance: avoid to create many timer object in the target ( #36570 )
...
/kind improvement
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-09-29 19:23:16 +08:00
SimFG
ddadefcb62
enhance: get msg type from the msg header to reduce the Unmarshal usage ( #36409 )
...
/kind improvement
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-09-24 10:19:20 +08:00
SimFG
c50fe71163
fix: long buffering causes mq to be unable to receive messages. ( #36420 )
...
- issue: #36397
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-09-23 16:33:18 +08:00
congqixia
3b01b7dc9a
enhance: Produce messages of multiple topics in parallel ( #36344 )
...
Related to #36343
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-23 11:23:11 +08:00
SimFG
07bb071aaf
enhance: add the unmarshal method for the rbac msg ( #35692 )
...
/kind improvement
- issue: #35391
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-08-30 07:23:00 +08:00
SimFG
9dc1311f38
enhance: add the param to control whether to include the current msg ( #35656 )
...
/kind improvement
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-08-23 15:56:57 +08:00
congqixia
2736a8b88c
enhance: Update Makefile to generate mockery ( #35517 )
...
Some mockery cmd is out-of-date and fail to work. This PR update these
commands to match current pkg.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-16 18:44:53 +08:00
SimFG
b2cc4b0776
feat: add the rbac msg and send them to the replicate channel ( #35392 )
...
- issue: #35391
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-08-15 12:06:52 +08:00
yihao.dai
7721a28584
fix: Fix unstable TestDispatchToVchannels ut ( #35163 )
...
issue: https://github.com/milvus-io/milvus/issues/35150
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-08-01 14:56:13 +08:00
congqixia
de8a266d8a
enhance: Enable linux code checker ( #35084 )
...
See also #34483
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-30 15:53:51 +08:00
SimFG
0e41f104c5
enhance: use the collection id to group msg pack in the msg dispatcher ( #34871 )
...
- issue: #34870
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-07-29 11:45:49 +08:00
wei liu
c45f38aa61
enhance: Update protobuf-go to protobuf-go v2 ( #34394 )
...
issue: #34252
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-29 11:31:51 +08:00
yihao.dai
4a3d98d88c
enhance: Fast close msgdispatcher target ( #34803 )
...
/kind improvement
issue: https://github.com/milvus-io/milvus/issues/34075
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-29 00:25:49 +08:00
chyezh
39c7e06bc5
enhance: add message and msgstream msgpack adaptor ( #34874 )
...
issue: #33285
- make message builder and message conversion type safe
- add adaptor type and function to adapt old msgstream msgpack and
message interface
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-07-22 20:59:42 +08:00
chyezh
1bc3c0b925
enhance: implement balancer at streaming coord ( #34435 )
...
issue: #33285
- add balancer implementation
- add channel count fair balance policy
- add channel assignment discover grpc service
Signed-off-by: chyezh <chyezh@outlook.com>
2024-07-11 09:58:48 +08:00
