admin/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2026-02-02 01:06:41 +08:00
Code Issues Packages Projects Releases Wiki Activity
22,149 Commits 146 Branches 196 Tags
Commit Graph

134 Commits

Author SHA1 Message Date
yihao.dai
72a8777c9d
enhance: [2.5] Accelerate dispatcher building (#42544) 
Reduce check interval to accelerate dispatcher building.

issue: https://github.com/milvus-io/milvus/issues/42067

pr: https://github.com/milvus-io/milvus/pull/42500

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2025-06-05 18:16:33 +08:00
SimFG
00336a8c4c
enhance: [2.5] add privilege group message types and corresponding tests (#41940) 
- issue: #41938
- pr: #41939

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2025-05-21 11:12:29 +08:00
SimFG
4c656e8a7a
enhance: support more ddl apis to replicate according to cdc (#41679) 
- issue: #41677
- pr: #41678

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2025-05-08 20:02:58 +08:00
SimFG
57c472e41f
enhance: [2.5] add timeout for message reception in mqMsgStream (#41603) 
- pr: #41602

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2025-05-06 10:16:52 +08:00
SimFG
b69bf42a04
enhance: [2.5] Implement OperatePrivilegeV2 message handling and unmarshal support (#41594) 
- issue: #41353
- pr: #41355

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2025-04-30 11:52:53 +08:00
SimFG
6e18ededab
fix: [2.5] mockery too unavailable after upgrade golang version (#41522) 
- issue: ##41291
- pr: #41481

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2025-04-25 14:40:40 +08:00
SimFG
18eb627533
fix: [2.5] Update logging context and upgrade dependencies (#41319) 
- issue: #41291
- pr: #41318

---------

Signed-off-by: SimFG <bang.fu@zilliz.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 2025-04-24 23:50:40 +08:00
aoiasd
d49f0b65d1
fix: rocksmq consumer register not concurrent safe (#39387) (#40885) 
relate: https://github.com/milvus-io/milvus/issues/39336
pr: https://github.com/milvus-io/milvus/pull/39387

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2025-03-28 16:08:18 +08:00
Zhen Ye
53438c751d
fix: skip consuming from streaming service message (#40879) 
issue: #40532
pr: #40877

Signed-off-by: chyezh <chyezh@outlook.com>
 2025-03-25 15:22:23 +08:00
yihao.dai
9b7d66d9da
fix: [2.5] Fix channel not balance on datanodes (#40854) 
1. Prevent channels from being assigned to only one datanode during
datacoord startup.
2. Optimize the channel assignment policy by considering newly assigned
channels.

issue: https://github.com/milvus-io/milvus/issues/40421,
https://github.com/milvus-io/milvus/issues/37630

pr: https://github.com/milvus-io/milvus/pull/40422

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2025-03-25 10:06:24 +08:00
yihao.dai
b534c9d804
enhance: [2.5] Introduce batch subscription in msgdispatcher (#40596) 
Introduce a batch subscription mechanism in msgdispatcher: the
msgdispatcher now includes a vchannel watch task queue, where all
vchannels in the queue will subscribe to the MQ only once and pull
messages from the oldest vchannel checkpoint to the latest.

issue: https://github.com/milvus-io/milvus/issues/39862

pr: https://github.com/milvus-io/milvus/pull/39863

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2025-03-24 10:18:17 +08:00
yihao.dai
aca9dd9105
fix: [2.5] Fix messages loss during seek (#40736) 
issue: https://github.com/milvus-io/milvus/issues/40733

pr: https://github.com/milvus-io/milvus/pull/40735

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2025-03-18 21:56:33 +08:00
congqixia
8de5174e4d
fix: [2.5] Remove lag targets after main dispatcher (#40717) 
Related to #40716

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2025-03-18 19:10:34 +08:00
aoiasd
ebf3416245
fix: main dispatcher not terminate cause messsage stream leak. (#40061) (#40351) 
Main dispatcher will leak when we remove solo dispatcher in the end.
relate: https://github.com/milvus-io/milvus/issues/40046
pr: https://github.com/milvus-io/milvus/pull/40061

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2025-03-05 20:14:02 +08:00
congqixia
709594f158
enhance: [2.5] Use v2 package name for pkg module (#40117) 
Cherry-pick from master
pr: #39990
Related to #39095

https://go.dev/doc/modules/version-numbers

Update pkg version according to golang dep version convention

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2025-02-23 00:46:01 +08:00
yihao.dai
7a92121c92
fix: [2.5] Fix failed to seek to earliest position (#39966) 
If it is the earliest message ID, skip the seek to prevent failure.

issue: https://github.com/milvus-io/milvus/issues/39964

pr: https://github.com/milvus-io/milvus/pull/39965

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2025-02-18 20:36:53 +08:00
yihao.dai
4a8e6fc59c
fix: [2.5] Fix flowgraph leak (#39686) 
Consider lag targets when determining whether to remove the dispatcher
manager.

issue: https://github.com/milvus-io/milvus/issues/39642

pr: https://github.com/milvus-io/milvus/pull/39656

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2025-02-08 15:40:45 +08:00
Duc Le Tu
3893f24323
fix: apply custom producer config for kafkaHealthCheck (#39364) 
issue: https://github.com/milvus-io/milvus/issues/39287
pr: https://github.com/milvus-io/milvus/pull/39283

Signed-off-by: DLT1412 <tuduc93@gmail.com>
 2025-01-20 17:27:04 +08:00
yihao.dai
29dad64341
fix: [2.5] Fix consume blocked due to too many consumers (#38915) 
This PR limits the maximum number of consumers per pchannel to 10 for
each QueryNode and DataNode.

issue: https://github.com/milvus-io/milvus/issues/37630

pr: https://github.com/milvus-io/milvus/pull/38455

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2025-01-16 15:19:03 +08:00
tinswzy
39fa00fc1a
enhance: add intent ctx to rootcoord init (#38948) 
issue: #38439 
cherry pick "enhance: add intent ctx to rootcoord init"

Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
 2025-01-11 23:54:57 +08:00
Zhen Ye
b18a3cfce2
fix: kafka use shared channel to receive produce result (#38532) 
issue: #38531

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-12-17 23:10:45 +08:00
SimFG
2afe2eaf3e
feat: support to replicate collection when the services contains the system tt msg (#37559) 
- issue: #37105

---------

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-12-17 09:08:46 +08:00
SimFG
fa8ac09550
fix: the issue of replicate message exception when the ttMsgEnable config is changed dynamically (#38178) 
- issue: #38177

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-12-14 23:24:51 +08:00
tinswzy
27229f7907
enhance: refine exists log print with ctx (#38080) 
issue: #35917 
Refines exists log print with ctx

Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
 2024-12-14 22:36:44 +08:00
jaime
37a52286b1
fix: stream connection leak in case of error (#38320) 
issue: #38318

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-12-11 15:38:49 +08:00
jaime
a1e14d62c7
enhance: refine lock granularity for produers in msgstream (#38262) 
issue: #38261

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-12-06 17:14:40 +08:00
jaime
edabfa8fa6
enhance: Remove pool from produer and consumer in kafka client (#38264) 
issue: #38263

Revert "fix: Move init kafka pool into once (#37786)"

Revert "enhance: Use pool to limit kafka cgo thread number (#37744)"

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-12-06 14:08:40 +08:00
tinswzy
7944538ade
enhance: Add ctx param to KV operation interfaces (#38154) 
issue: #35917 
Refine KV operation interfaces by adding a ctx param

Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
 2024-12-05 15:16:41 +08:00
tinswzy
5768dbbb5d
enhance: refine pular related mq interfaces (#38007) 
issue: #35917 
Refines the pulsar-related mq APIs to allow the ctx to be passed down

Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
 2024-12-04 20:50:39 +08:00
congqixia
97b7ebbc08
fix: Move init kafka pool into once (#37786) 
Introduced by #37744

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-11-18 21:24:48 +08:00
congqixia
3743fc2b00
enhance: Use pool to limit kafka cgo thread number (#37744) 
Related to #37665

Thread number went rocket high when there is lots of kafka consumers on
datanode. Since the internal implementation is CGO, using which directly
will make cgo thread leaked.

This PR add a worker pool for kafka API utilzing CGO calls to limit
thread number.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-11-18 10:42:31 +08:00
congqixia
4820dd76e5
fix: Close leaked health check kafka producer (#37728) 
Related to #36822

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-11-16 11:20:32 +08:00
Zhen Ye
cd181e4c6d
fix: make NewDataSyncService idempotent of dispatcher (#37576) 
issue: #37547

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-11-14 11:00:31 +08:00
wei liu
00f6d0ec51
fix: watch channel stuck due to misuse of timer.Reset (#37433) 
issue: #37166
cause the misuse of timer.Reset, which cause dispatcher failed to send
msg to virtual channel buffer, and dispatcher do splitting again and
again, which hold the dispatcher manager's lock, block watching channel
progress.

This PR fix the misuse of timer.Reset

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-11-07 14:34:24 +08:00
jaime
4746f47282
feat: management WebUI homepage (#36822) 
issue: #36784
1. Implement an embedded web server for WebUI access.  
2. Complete the homepage development.

Home page demo:
<img width="2177" alt="iShot_2024-10-10_17 57 34"
src="https://github.com/user-attachments/assets/38539917-ce09-4e54-a5b5-7f4f7eaac353">

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-10-23 11:29:28 +08:00
SimFG
821750d902
enhance: force to stop buffer message when receiving the drop collection message (#36916) 
/kind improvement

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-10-17 12:53:29 +08:00
Bingyi Sun
6851738fd1
fix: fix make generate-mockery panic with go1.22 (#36830) 
https://github.com/milvus-io/milvus/issues/36831
Fix `make generate-mockery` panic.

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2024-10-17 12:11:31 +08:00
aoiasd
2c3a8b7dea
enhance: fix skip msg log with empty position info (#36626) 
All skiped message position info was nil, should print BeginTs or
MessageID instead.
relate: https://github.com/milvus-io/milvus/issues/36569

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2024-10-09 12:01:20 +08:00
Zhen Ye
addced27f9
fix: rocksmq consume too slow when the channel is full (#36617) 
issue: #36569

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-10-09 11:59:31 +08:00
smellthemoon
2055df81aa
enhance: upgrade pulsar-client-go to 0.12.1 (#36615) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2024-09-30 17:13:16 +08:00
SimFG
9c1772f659
enhance: avoid to create many timer object in the target (#36570) 
/kind improvement

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-09-29 19:23:16 +08:00
SimFG
ddadefcb62
enhance: get msg type from the msg header to reduce the Unmarshal usage (#36409) 
/kind improvement

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-09-24 10:19:20 +08:00
SimFG
c50fe71163
fix: long buffering causes mq to be unable to receive messages. (#36420) 
- issue: #36397

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-09-23 16:33:18 +08:00
congqixia
3b01b7dc9a
enhance: Produce messages of multiple topics in parallel (#36344) 
Related to #36343

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-09-23 11:23:11 +08:00
SimFG
07bb071aaf
enhance: add the unmarshal method for the rbac msg (#35692) 
/kind improvement
- issue: #35391

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-08-30 07:23:00 +08:00
SimFG
9dc1311f38
enhance: add the param to control whether to include the current msg (#35656) 
/kind improvement

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-08-23 15:56:57 +08:00
congqixia
2736a8b88c
enhance: Update Makefile to generate mockery (#35517) 
Some mockery cmd is out-of-date and fail to work. This PR update these
commands to match current pkg.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-08-16 18:44:53 +08:00
SimFG
b2cc4b0776
feat: add the rbac msg and send them to the replicate channel (#35392) 
- issue: #35391

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-08-15 12:06:52 +08:00
yihao.dai
7721a28584
fix: Fix unstable TestDispatchToVchannels ut (#35163) 
issue: https://github.com/milvus-io/milvus/issues/35150

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-08-01 14:56:13 +08:00
congqixia
de8a266d8a
enhance: Enable linux code checker (#35084) 
See also #34483

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-07-30 15:53:51 +08:00