yihao.dai
b4cb8a4b13
enhance: Add UTF-8 string validation for import ( #40694 )
...
issue: https://github.com/milvus-io/milvus/issues/40684
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-04-01 19:04:21 +08:00
Zhen Ye
b03e60558a
enhance: add proxy and datanode checker when wal balance startup ( #40877 )
...
issue: #40532
- balance should enable only when there's no proxy and datanode which
version is lower than 2.6.0
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2025-04-01 11:24:22 +08:00
yihao.dai
5b78ef0a49
fix: Fix delete data loss due to duplicate binlogID ( #40960 )
...
With concurrenct L0 compaction
(https://github.com/milvus-io/milvus/pull/36816 ), delta logs might be
written to the same L1 segment, causing logID duplication when using the
incremental beginLogID. This PR removes the beginLogID mechanism and
instead passes a log ID range, where the number of IDs in the range
equals the number of compaction segment binlogs multiplied by an
expansion factor.
issue: https://github.com/milvus-io/milvus/issues/40207
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-04-01 10:36:22 +08:00
wei liu
c02892e9fb
enhance: Balance the collection with the largest row count first ( #40297 )
...
issue: #37651
this PR enable to balance the collection with largest row count first,
to avoid temporary migration of small table data to new nodes during
their onboarding, only to be moved out again after the large table
balance, which would cause unnecessary load.
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2025-03-31 16:00:19 +08:00
yihao.dai
d8d1dcf076
fix: Fix dispatcher deregister and seek ( #40860 )
...
1. Fix deregister dispatcher concurrency. (Keep sure the same logic as
2.5 branch)
2. Fix seek if includeCurrentMsg. (This is only needed by CDC, so
there's no need to pick to 2.5 branch)
issue: issue: https://github.com/milvus-io/milvus/issues/39862
pr: https://github.com/milvus-io/milvus/pull/39863
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-03-31 11:38:19 +08:00
dependabot[bot]
b1e092673b
enhance: bump github.com/golang-jwt/jwt/v5 from 5.2.1 to 5.2.2 ( #40923 )
...
Bumps [github.com/golang-jwt/jwt/v5](https://github.com/golang-jwt/jwt )
from 5.2.1 to 5.2.2.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/golang-jwt/jwt/releases ">github.com/golang-jwt/jwt/v5's
releases</a>.</em></p>
<blockquote>
<h2>v5.2.2</h2>
<h2>What's Changed</h2>
<ul>
<li>Fixed <a
href="https://github.com/golang-jwt/jwt/security/advisories/GHSA-mh63-6h87-95cp ">https://github.com/golang-jwt/jwt/security/advisories/GHSA-mh63-6h87-95cp </a>
by <a
href="https://github.com/mfridman "><code>@mfridman</code></a></li>
<li>Fixed some typos by <a
href="https://github.com/Ashikpaul "><code>@Ashikpaul</code></a> in <a
href="https://redirect.github.com/golang-jwt/jwt/pull/382 ">golang-jwt/jwt#382</a></li>
<li>build: add go1.22 to ci workflows by <a
href="https://github.com/mfridman "><code>@mfridman</code></a> in <a
href="https://redirect.github.com/golang-jwt/jwt/pull/383 ">golang-jwt/jwt#383</a></li>
<li>Bump golangci/golangci-lint-action from 4 to 5 by <a
href="https://github.com/dependabot "><code>@dependabot</code></a> in <a
href="https://redirect.github.com/golang-jwt/jwt/pull/387 ">golang-jwt/jwt#387</a></li>
<li>Bump golangci/golangci-lint-action from 5 to 6 by <a
href="https://github.com/dependabot "><code>@dependabot</code></a> in <a
href="https://redirect.github.com/golang-jwt/jwt/pull/389 ">golang-jwt/jwt#389</a></li>
<li>chore: bump ci tests to include go1.23 by <a
href="https://github.com/mfridman "><code>@mfridman</code></a> in <a
href="https://redirect.github.com/golang-jwt/jwt/pull/405 ">golang-jwt/jwt#405</a></li>
<li>Fix jwt -show by <a
href="https://github.com/AlexanderYastrebov "><code>@AlexanderYastrebov</code></a>
in <a
href="https://redirect.github.com/golang-jwt/jwt/pull/406 ">golang-jwt/jwt#406</a></li>
<li>docs: typo by <a
href="https://github.com/kvii "><code>@kvii</code></a> in <a
href="https://redirect.github.com/golang-jwt/jwt/pull/407 ">golang-jwt/jwt#407</a></li>
<li>Update SECURITY.md by <a
href="https://github.com/oxisto "><code>@oxisto</code></a> in <a
href="https://redirect.github.com/golang-jwt/jwt/pull/416 ">golang-jwt/jwt#416</a></li>
<li>Update <code>jwt.Parse</code> example to use
<code>jwt.WithValidMethods</code> by <a
href="https://github.com/mattt "><code>@mattt</code></a> in <a
href="https://redirect.github.com/golang-jwt/jwt/pull/425 ">golang-jwt/jwt#425</a></li>
</ul>
<h2>New Contributors</h2>
<ul>
<li><a href="https://github.com/Ashikpaul "><code>@Ashikpaul</code></a>
made their first contribution in <a
href="https://redirect.github.com/golang-jwt/jwt/pull/382 ">golang-jwt/jwt#382</a></li>
<li><a href="https://github.com/kvii "><code>@kvii</code></a> made their
first contribution in <a
href="https://redirect.github.com/golang-jwt/jwt/pull/407 ">golang-jwt/jwt#407</a></li>
<li><a href="https://github.com/mattt "><code>@mattt</code></a> made
their first contribution in <a
href="https://redirect.github.com/golang-jwt/jwt/pull/425 ">golang-jwt/jwt#425</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/golang-jwt/jwt/compare/v5.2.1...v5.2.2 ">https://github.com/golang-jwt/jwt/compare/v5.2.1...v5.2.2 </a></p>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="0951d18428c035977d9ehttps://redirect.github.com/golang-jwt/jwt/issues/425 ">#425</a>)</li>
<li><a
href="bc8bdca5cchttps://redirect.github.com/golang-jwt/jwt/issues/416 ">#416</a>)</li>
<li><a
href="5ec246c074https://redirect.github.com/golang-jwt/jwt/issues/407 ">#407</a>)</li>
<li><a
href="0123f1ad66https://redirect.github.com/golang-jwt/jwt/issues/406 ">#406</a>)</li>
<li><a
href="f961c72abdhttps://redirect.github.com/golang-jwt/jwt/issues/405 ">#405</a>)</li>
<li><a
href="62e504c281https://redirect.github.com/golang-jwt/jwt/issues/389 ">#389</a>)</li>
<li><a
href="1a56dcf532https://redirect.github.com/golang-jwt/jwt/issues/387 ">#387</a>)</li>
<li><a
href="c8043eab61https://redirect.github.com/golang-jwt/jwt/issues/383 ">#383</a>)</li>
<li><a
href="7c3f6dc563https://redirect.github.com/golang-jwt/jwt/issues/382 ">#382</a>)</li>
<li>See full diff in <a
href="https://github.com/golang-jwt/jwt/compare/v5.2.1...v5.2.2 ">compare
view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores )
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
You can disable automated security fix PRs for this repo from the
[Security Alerts
page](https://github.com/milvus-io/milvus/network/alerts ).
</details>
---------
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
2025-03-27 00:56:21 +08:00
cai.zhang
4220e3355d
fix: Update milvus key words ( #40869 )
...
issue: #40290
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2025-03-26 19:44:32 +08:00
Gao
0a1c5f4460
enhance: make segment prune config refreshable ( #40539 )
...
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2025-03-26 18:50:25 +08:00
Zhen Ye
af80a4dac2
fix: auto flush all segment that is not created by streaming service ( #40767 )
...
issue: #40532
Signed-off-by: chyezh <chyezh@outlook.com>
2025-03-26 16:32:22 +08:00
Zhen Ye
0e83a08ffe
enhance: enable cipher for streaming message ( #40659 )
...
issue: #40321
- add cipher plugin for streaming message package.
- add more unittest for streaming message.
- remove redundant code for streaming message.
Signed-off-by: chyezh <chyezh@outlook.com>
2025-03-26 16:30:20 +08:00
congqixia
84e8e141ea
enhance: Support detailed manual compaction criterion ( #40892 )
...
Related to #40866
This PR:
- update go-api/v2 and support partition id/channel/segment level manual
compaction
- refines the compaction trigger implementation
- unify the compaction signal usage
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-03-25 20:06:22 +08:00
junjiejiangjjj
fe81c7baae
feat: Add function config ( #40534 )
...
#35856
1. Add function-related configuration in milvus.yaml
2. Add null and empty value check to TextEmbeddingFunction
Signed-off-by: junjie.jiang <junjie.jiang@zilliz.com>
2025-03-25 10:06:24 +08:00
SimFG
16efcda5c4
feat: support to deny dll according to database property ( #40764 )
...
- issue: #40762
Signed-off-by: SimFG <bang.fu@zilliz.com>
2025-03-23 11:18:28 +08:00
Ted Xu
688505ab1c
enhance: cleanup lint check exclusions ( #40829 )
...
See: #40828
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2025-03-21 18:12:14 +08:00
yihao.dai
8eb537fdd0
fix: Fix messages loss during seek ( #40735 )
...
issue: https://github.com/milvus-io/milvus/issues/40733
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-03-20 20:18:11 +08:00
Zhen Ye
4dc7c11bbe
fix: remove dead code at broadcaster ( #40788 )
...
issue: #38399
Signed-off-by: chyezh <chyezh@outlook.com>
2025-03-20 14:20:59 +08:00
congqixia
096f58f314
enhance: Add an extra parameter for slow log ( #40758 )
...
Related to #40756
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-03-20 00:34:13 +08:00
aoiasd
92bdf7a0c1
enhance: support run anayser return detaild token ( #40458 )
...
relate: https://github.com/milvus-io/milvus/issues/39705
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2025-03-19 15:48:15 +08:00
zhagnlu
6c55db44f1
enhance: reorder sub expr for conjunct expr ( #39872 )
...
two point:
(1) reoder conjucts expr's subexpr, postpone heavy operations
sequence: int(column) -> index(column) -> string(column) -> light
conjuct
...... -> json(column) -> heavy conjuct -> two_column_compare
(2) support pre filter for expr execute, skip scan raw data that had
been skipped
because of preceding expr result.
#39869
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2025-03-19 14:50:14 +08:00
Zhen Ye
8db708f67d
enhance: enable memory prof based on jemalloc ( #40731 )
...
issue: #40730
also see: https://github.com/milvus-io/cgosymbolizer/pull/2
After these PR, at linux:
- the milvus will always enable jemalloc by default.
- jemalloc will always compiled with --enable-prof options.
- all image will always enable the jemalloc prof by default.
- a pprof http service for jemalloc at `/debug/jemalloc/` will be
registered into restful.
- `jeprof` can remote profile the memory of milvus.
Signed-off-by: chyezh <chyezh@outlook.com>
2025-03-19 14:46:18 +08:00
Zhen Ye
b119ac5d30
enhance: add wal access mode options ( #40617 )
...
issue: #40532
Signed-off-by: chyezh <chyezh@outlook.com>
2025-03-19 14:02:11 +08:00
Buqian Zheng
c12abf4e2a
enhance: improve sparse query nnz metric ( #40713 )
...
add query type and field id label; add metric for hybrid search
issue: https://github.com/milvus-io/milvus/issues/35853
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2025-03-18 17:20:16 +08:00
congqixia
94a859c028
enhance: Add buffer forwarder for stream delta loading ( #40559 )
...
See also #40558
Related to #35303 & #38066 as well
This PR:
- Add `BufferedForward` to limit memory usage forwarding stream delete
- Add `UseLoad` flag to determine `Delete` shall use `segment.Delete` or
`segment.LoadDelta`
- Fix delegator accidentally use always true candidate while load
streaming delta
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-03-17 15:24:10 +08:00
cai.zhang
123b6588b6
feat: Support get segment binlogs info with new interface GetSegmentsInfo ( #40464 )
...
issue: #40341
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2025-03-16 21:04:07 +08:00
zhagnlu
8f85acc944
Revert "fix:reset default auto index type for scalar ( #37086 )" ( #39819 )
...
This reverts commit eeb67a38450732eb8ab3a4eb4aaac63401426627.
#32900
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2025-03-14 16:16:11 +08:00
cai.zhang
6dbe5d475e
enhance: Refine task meta with key lock ( #40613 )
...
issue: #39101
2.5 pr: #40146 #40353
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2025-03-14 15:44:22 +08:00
SimFG
bf4fc6a8c6
feat: add DDLDB rate type and related quota configurations ( #40651 )
...
- issue: #40650
Signed-off-by: SimFG <bang.fu@zilliz.com>
2025-03-14 15:38:09 +08:00
Zhen Ye
6982f007e2
enhance: add walimpls access mode options ( #40591 )
...
issue: #40532
Signed-off-by: chyezh <chyezh@outlook.com>
2025-03-14 10:58:11 +08:00
Xiaofan
7210fc9780
feature: add a prefix on environment config ( #40623 )
...
fix #40622
Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
2025-03-13 16:44:07 +08:00
Zhen Ye
f6fb4bc442
fix: backoff will retry infinitely after reaching max elapse ( #40589 )
...
issue: #40588
Signed-off-by: chyezh <chyezh@outlook.com>
2025-03-13 16:24:06 +08:00
yihao.dai
b2a8694686
enhance: Merge IndexNode and DataNode ( #40272 )
...
Merge DataNode and IndexNode into DataNode.
issue: https://github.com/milvus-io/milvus/issues/39115
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-03-13 14:26:11 +08:00
jaime
c8a96377bb
enhance: move object storage client creation to pkg package ( #40440 )
...
issue: #40439
Signed-off-by: jaime <yun.zhang@zilliz.com>
2025-03-12 20:38:07 +08:00
wei liu
0420dc1eb1
fix: use correct delete checkpoint to prevent premature data cleanup ( #40366 )
...
issue: #40292
related to #39552
- Fix incorrect delete checkpoint usage in SyncDistribution
- Change checkpoint parameter from action.GetCheckpoint() to
action.GetDeleteCP() in SyncTargetVersion call
- This resolves the issue where delete buffer data was being cleaned
prematurely due to wrong checkpoint reference
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2025-03-12 15:00:08 +08:00
Zhen Ye
96a010da7a
fix: msgstream adaptor may not gc quickly after comsumed ( #40555 )
...
issue: #40540
Signed-off-by: chyezh <chyezh@outlook.com>
2025-03-12 14:00:04 +08:00
yihao.dai
a33c9372ce
fix: Fix channel not balance on datanodes ( #40422 )
...
1. Prevent channels from being assigned to only one datanode during
datacoord startup.
2. Optimize the channel assignment policy by considering newly assigned
channels.
3. Make msgdispatcher manager lock-free.
issue: https://github.com/milvus-io/milvus/issues/40421 ,
https://github.com/milvus-io/milvus/issues/37630
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-03-11 14:56:16 +08:00
aoiasd
3afc3a83ff
fix: rocksmq consumer register not concurrent safe ( #39387 )
...
relate: https://github.com/milvus-io/milvus/issues/39336
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2025-03-11 00:56:04 +08:00
junjiejiangjjj
359e7efd8e
feat: Add function running monitoring ( #40358 )
...
#35856
#40004
1. Optimize model verification logic
2. Add profiling code
Signed-off-by: junjie.jiang <junjie.jiang@zilliz.com>
2025-03-10 22:28:05 +08:00
yihao.dai
2ca2e2dbc8
fix: Fix parsing import endTs ( #40332 )
...
Parsing import beginTs, endTs as a hybrid timestamp.
issue: https://github.com/milvus-io/milvus/issues/40326
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-03-10 17:38:04 +08:00
XuanYang-cn
e6c46a25ea
enhance: Use correct counter metrics for overall wa calculation ( #40394 )
...
- Use CounterVec to calculate sum of increase during a time period.
- Use entries number instead of binlog size
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2025-03-10 16:34:06 +08:00
congqixia
391804c7fb
enhance: Add channel seal policy based on blocking l0 ( #40505 )
...
Related to #40502
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-03-10 16:28:04 +08:00
XuanYang-cn
4bebca6416
enhance: Replace currRows with NumOfRows ( #40074 )
...
See also: #40068
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2025-03-10 12:16:03 +08:00
Xiaofan
1c9d43ee9f
fix: rockmq race condition ( #40482 )
...
fix #40481
Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
2025-03-10 10:54:04 +08:00
yihao.dai
004a1875dc
enhance: Introduce batch subscription in msgdispatcher ( #39863 )
...
Introduce a batch subscription mechanism in msgdispatcher: the
msgdispatcher now includes a vchannel watch task queue, where all
vchannels in the queue will subscribe to the MQ only once and pull
messages from the oldest vchannel checkpoint to the latest.
issue: https://github.com/milvus-io/milvus/issues/39862
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-03-05 14:38:02 +08:00
sthuang
63a7c4570e
feat: storage v2 sync ( #39663 )
...
related: #39173
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2025-03-05 11:22:15 +08:00
junjiejiangjjj
b2e630b1a1
feat: Support TEI serving and support int8 embedding ( #40199 )
...
#35856
Signed-off-by: junjie.jiang <junjie.jiang@zilliz.com>
2025-03-05 10:18:00 +08:00
Zhen Ye
f47ab31f23
enhance: remove redundant resource key watch operation, just keep consistency of wal ( #40235 )
...
issue: #38399
related PR: #39522
- Just implement exclusive broadcaster between broadcast message with
same resource key to keep same order in different wal.
- After simplify the broadcast model, original watch-based broadcast is
too complicated and redundant, remove it.
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2025-03-03 14:40:05 +08:00
cai.zhang
762a644d76
enhance: Limit the speed of the generating stats task ( #39644 )
...
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2025-02-28 10:27:59 +08:00
cai.zhang
13aff35a83
enhance: Add metrics for parse expression ( #39654 )
...
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2025-02-28 10:07:58 +08:00
cai.zhang
eb04686348
enhance: Add metrics for proxy queue ( #40070 )
...
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2025-02-27 17:30:01 +08:00
aoiasd
d67d6b7567
fix: main dispatcher not terminate cause messsage stream leak. ( #40061 )
...
Main dispatcher will leak when we remove solo dispatcher in the end.
https://github.com/milvus-io/milvus/issues/40046
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2025-02-27 16:15:58 +08:00
