yihao.dai
8eb537fdd0
fix: Fix messages loss during seek ( #40735 )
...
issue: https://github.com/milvus-io/milvus/issues/40733
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-03-20 20:18:11 +08:00
yihao.dai
004a1875dc
enhance: Introduce batch subscription in msgdispatcher ( #39863 )
...
Introduce a batch subscription mechanism in msgdispatcher: the
msgdispatcher now includes a vchannel watch task queue, where all
vchannels in the queue will subscribe to the MQ only once and pull
messages from the oldest vchannel checkpoint to the latest.
issue: https://github.com/milvus-io/milvus/issues/39862
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-03-05 14:38:02 +08:00
Zhen Ye
f47ab31f23
enhance: remove redundant resource key watch operation, just keep consistency of wal ( #40235 )
...
issue: #38399
related PR: #39522
- Just implement exclusive broadcaster between broadcast message with
same resource key to keep same order in different wal.
- After simplify the broadcast model, original watch-based broadcast is
too complicated and redundant, remove it.
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2025-03-03 14:40:05 +08:00
congqixia
cb7f2fa6fd
enhance: Use v2 package name for pkg module ( #39990 )
...
Related to #39095
https://go.dev/doc/modules/version-numbers
Update pkg version according to golang dep version convention
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-02-22 23:15:58 +08:00
yihao.dai
c917fe4782
fix: Fix failed to seek to earliest position ( #39965 )
...
If it is the earliest message ID, skip the seek to prevent failure.
issue: https://github.com/milvus-io/milvus/issues/39964
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-02-19 00:30:51 +08:00
SimFG
047254665d
feat: support to replicate import msg ( #39171 )
...
- issue: #39849
---------
Signed-off-by: SimFG <bang.fu@zilliz.com>
Signed-off-by: chyezh <chyezh@outlook.com>
Co-authored-by: chyezh <chyezh@outlook.com>
2025-02-16 00:08:13 +08:00
aoiasd
24d2bbc441
enhance: unmashall ts msg in dispatcher instead in msgstream ( #38656 )
...
relate: https://github.com/milvus-io/milvus/issues/38655
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2025-02-14 12:04:13 +08:00
Duc Le Tu
2a962ad1ec
fix: apply custom producer config for kafkaHealthCheck ( #39283 )
...
issue: https://github.com/milvus-io/milvus/issues/39287
KafkaHealthCheck init without ProducerExtraConfig. This PR fix that.
---------
Signed-off-by: DLT1412 <tuduc93@gmail.com>
Co-authored-by: DucLT <duc.le1@be.com.vn>
2025-01-16 19:31:02 +08:00
tinswzy
11f8f4a378
enhance: add intent ctx to rootcoord init ( #38439 )
...
issue: #35917
add intent ctx to rootcoord initialization process
Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
2024-12-31 18:22:53 +08:00
Zhen Ye
285289d5b0
enhance: implement kafka for wal ( #38598 )
...
issue: #38399
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-12-25 10:22:50 +08:00
Zhen Ye
b18a3cfce2
fix: kafka use shared channel to receive produce result ( #38532 )
...
issue: #38531
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-12-17 23:10:45 +08:00
SimFG
2afe2eaf3e
feat: support to replicate collection when the services contains the system tt msg ( #37559 )
...
- issue: #37105
---------
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-12-17 09:08:46 +08:00
SimFG
fa8ac09550
fix: the issue of replicate message exception when the ttMsgEnable config is changed dynamically ( #38178 )
...
- issue: #38177
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-12-14 23:24:51 +08:00
tinswzy
27229f7907
enhance: refine exists log print with ctx ( #38080 )
...
issue: #35917
Refines exists log print with ctx
Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
2024-12-14 22:36:44 +08:00
jaime
a1e14d62c7
enhance: refine lock granularity for produers in msgstream ( #38262 )
...
issue: #38261
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-12-06 17:14:40 +08:00
jaime
edabfa8fa6
enhance: Remove pool from produer and consumer in kafka client ( #38264 )
...
issue: #38263
Revert "fix: Move init kafka pool into once (#37786 )"
Revert "enhance: Use pool to limit kafka cgo thread number (#37744 )"
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-12-06 14:08:40 +08:00
tinswzy
5768dbbb5d
enhance: refine pular related mq interfaces ( #38007 )
...
issue: #35917
Refines the pulsar-related mq APIs to allow the ctx to be passed down
Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
2024-12-04 20:50:39 +08:00
congqixia
97b7ebbc08
fix: Move init kafka pool into once ( #37786 )
...
Introduced by #37744
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-11-18 21:24:48 +08:00
congqixia
3743fc2b00
enhance: Use pool to limit kafka cgo thread number ( #37744 )
...
Related to #37665
Thread number went rocket high when there is lots of kafka consumers on
datanode. Since the internal implementation is CGO, using which directly
will make cgo thread leaked.
This PR add a worker pool for kafka API utilzing CGO calls to limit
thread number.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-11-18 10:42:31 +08:00
congqixia
4820dd76e5
fix: Close leaked health check kafka producer ( #37728 )
...
Related to #36822
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-11-16 11:20:32 +08:00
jaime
4746f47282
feat: management WebUI homepage ( #36822 )
...
issue: #36784
1. Implement an embedded web server for WebUI access.
2. Complete the homepage development.
Home page demo:
<img width="2177" alt="iShot_2024-10-10_17 57 34"
src="https://github.com/user-attachments/assets/38539917-ce09-4e54-a5b5-7f4f7eaac353 ">
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-10-23 11:29:28 +08:00
SimFG
821750d902
enhance: force to stop buffer message when receiving the drop collection message ( #36916 )
...
/kind improvement
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-10-17 12:53:29 +08:00
Bingyi Sun
6851738fd1
fix: fix make generate-mockery panic with go1.22 ( #36830 )
...
https://github.com/milvus-io/milvus/issues/36831
Fix `make generate-mockery` panic.
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-10-17 12:11:31 +08:00
aoiasd
2c3a8b7dea
enhance: fix skip msg log with empty position info ( #36626 )
...
All skiped message position info was nil, should print BeginTs or
MessageID instead.
relate: https://github.com/milvus-io/milvus/issues/36569
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-10-09 12:01:20 +08:00
smellthemoon
2055df81aa
enhance: upgrade pulsar-client-go to 0.12.1 ( #36615 )
...
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-09-30 17:13:16 +08:00
SimFG
ddadefcb62
enhance: get msg type from the msg header to reduce the Unmarshal usage ( #36409 )
...
/kind improvement
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-09-24 10:19:20 +08:00
SimFG
c50fe71163
fix: long buffering causes mq to be unable to receive messages. ( #36420 )
...
- issue: #36397
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-09-23 16:33:18 +08:00
congqixia
3b01b7dc9a
enhance: Produce messages of multiple topics in parallel ( #36344 )
...
Related to #36343
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-23 11:23:11 +08:00
SimFG
07bb071aaf
enhance: add the unmarshal method for the rbac msg ( #35692 )
...
/kind improvement
- issue: #35391
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-08-30 07:23:00 +08:00
SimFG
b2cc4b0776
feat: add the rbac msg and send them to the replicate channel ( #35392 )
...
- issue: #35391
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-08-15 12:06:52 +08:00
congqixia
de8a266d8a
enhance: Enable linux code checker ( #35084 )
...
See also #34483
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-30 15:53:51 +08:00
wei liu
c45f38aa61
enhance: Update protobuf-go to protobuf-go v2 ( #34394 )
...
issue: #34252
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-29 11:31:51 +08:00
chyezh
39c7e06bc5
enhance: add message and msgstream msgpack adaptor ( #34874 )
...
issue: #33285
- make message builder and message conversion type safe
- add adaptor type and function to adapt old msgstream msgpack and
message interface
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-07-22 20:59:42 +08:00
chyezh
1bc3c0b925
enhance: implement balancer at streaming coord ( #34435 )
...
issue: #33285
- add balancer implementation
- add channel count fair balance policy
- add channel assignment discover grpc service
Signed-off-by: chyezh <chyezh@outlook.com>
2024-07-11 09:58:48 +08:00
SimFG
c6e2900cfc
enhance: add the tick log for the tt msgstream seek method ( #34397 )
...
/kind improvement
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-07-04 10:18:09 +08:00
jaime
9630974fbb
enhance: move rocksmq from internal to pkg module ( #33881 )
...
issue: #33956
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-06-25 21:18:15 +08:00
wayblink
a1232fafda
feat: Major compaction ( #33620 )
...
#30633
Signed-off-by: wayblink <anyang.wang@zilliz.com>
Co-authored-by: MrPresent-Han <chun.han@zilliz.com>
2024-06-10 21:34:08 +08:00
Aldrin
8a4ef1b836
fix: Masked kafka credentials getting printed in logs ( #33730 )
...
issue: https://github.com/milvus-io/milvus/issues/33727
Signed-off-by: Aldrin <imagesai32@gmail.com>
2024-06-10 00:05:54 +08:00
SimFG
cb99e3db34
enhance: add the includeCurrentMsg param for the Seek method ( #33326 )
...
/kind improvement
- issue: #33325
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-05-27 10:31:41 +08:00
congqixia
12e8c6c583
enhance: Try LatestMessageID when checkpoint unmarshal fails ( #33158 )
...
See also #33122
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-05-21 16:59:39 +08:00
congqixia
c2ac692008
enhance: Add param item to ignore bad message id in checkpoint ( #33123 )
...
See also #33122
This pr add param item `mq.ignoreBadPosition` to control behavior when
mq failed to parse message id from checkpoint
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-05-20 11:29:37 +08:00
Xiaofan
b6fefee0cf
fix: etcd not connectable when auth enabled ( #31633 )
...
Fix etcd config source didn't respect auth enabled
Also removed pulsar recoverable error when pulsar return ConsumerBusy.
It could happen that pulsar didn't find the original consumer is dead
and recover takes some time.
fix #31631
Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
2024-04-01 15:23:19 +08:00
groot
ba6d33cd57
fix: Support TLS for kafka connection ( #30468 )
...
#27977
Add extra configurations in milvus.yaml to pass certificates for kafka.
Signed-off-by: yhmo <yihua.mo@zilliz.com>
2024-02-28 18:43:07 +08:00
XuanYang-cn
e0f371c99b
fix: [skip e2e]use unique topic name for test pulsarCtl ( #30364 )
...
See also: #30363
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-02-01 11:05:04 +08:00
smellthemoon
e52ce370b6
enhance:don't store logPath in meta to reduce memory ( #28873 )
...
don't store logPath in meta to reduce memory, when service get
segmentinfo, generate logpath from logid.
#28885
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-01-18 22:06:31 +08:00
XuanYang-cn
f9508a4851
fix: Remove logging data when logging skip msg ( #29707 )
...
See also: #29696
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-01-15 11:40:51 +08:00
wei liu
1f759837c4
fix: remove Unnecessary lock in config manager ( #29836 )
...
issue: #29709 #291712
to avoid concurrent recursive RLock and Lock cause deadlock, This PR
remove the unnecessary lock in config manager
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-01-11 13:48:49 +08:00
smellthemoon
1c1f2a1371
enhance:change some logs ( #29579 )
...
related #29588
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-01-05 16:12:48 +08:00
aoiasd
9e6da45497
fix: Use uber atomic instead sync/atomic which only supported after go v1.20 ( #29377 )
...
relate: https://github.com/milvus-io/milvus/issues/29376
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-12-26 16:06:49 +08:00
yah01
a0e1a1eb31
feat: support enable/disable mmap for index ( #29005 )
...
support enable/disable mmap for index, the user could alter the index's
mode by `AlterIndex` method
related: https://github.com/milvus-io/milvus/issues/21866
---------
Signed-off-by: yah01 <yah2er0ne@outlook.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-12-21 18:07:24 +08:00
