Zhen Ye
5bdc593b8a
enhance: use v0.15.1 official pulsar client and add logging for pulsar client ( #43913 )
...
issue: #43785
- pulsar client will print log into milvus logger now.
- pulsar client open the metric by default.
- upgrade the pulsar client to v0.15.1, and use offical repo.
- the fixing of milvus-io/pulsar-client-go is already covered by
official v0.15.1.
Signed-off-by: chyezh <chyezh@outlook.com>
2025-08-26 16:45:53 +08:00
Tianx
26c5c779bf
feat: temporarily disable Timestamptz collection creation ( #43935 )
...
issue: https://github.com/milvus-io/milvus/issues/27467
Signed-off-by: xtx <xtianx@smail.nju.edu.cn>
2025-08-21 11:17:46 +08:00
tinswzy
68691f3e35
fix: ensure idempotent object upload on timeout retry ( #43947 )
...
#43927
fix logHandle close deadlock bug
fix ensure idempotent object upload on timeout retry
update to v0.1.4
Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
2025-08-20 14:21:45 +08:00
wei liu
d3c95eaa77
enhance: Support partial field updates with upsert API ( #42877 )
...
issue: #29735
Implement partial field update functionality for upsert operations,
supporting scalar, vector, and dynamic JSON fields without requiring all
collection fields.
Changes:
- Add queryPreExecute to retrieve existing records before upsert
- Implement UpdateFieldData function for merging data
- Add IDsChecker utility for efficient primary key lookups
- Fix JSON data creation in tests using proper map marshaling
- Add test cases for partial updates of different field types
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2025-08-19 11:15:45 +08:00
tinswzy
6a342edc5a
fix: empty error returned on append timeout when MinIO is unavailable ( #43926 )
...
#43810
Fixed the issue where the result err returned by append timeout was
empty when objectstorage was unavailable, causing the client to
mistakenly believe that the write was successful.
Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
2025-08-19 10:47:45 +08:00
Xianhui Lin
b98b3b16a3
feat:add BatchDescribeCollection interface ( #43786 )
...
feat:add BatchDescribeCollection interface
issue: https://github.com/milvus-io/milvus/issues/43781
Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
2025-08-18 01:23:45 +08:00
Zhen Ye
7b005c48bf
enhance: support util template generation for messages ( #43881 )
...
issue: #43880
Signed-off-by: chyezh <chyezh@outlook.com>
2025-08-18 01:19:44 +08:00
tinswzy
084f777552
enhance: use wp internal writer without lock ( #43775 )
...
#43638 #43810
add internal writer without session lock;
refactor and unify read state and log entry
refactor data reading related methods;
fix bug where a closed writer is reused for finalize;
Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
2025-08-18 01:15:44 +08:00
aoiasd
eca51ed2c6
enhance: add file resource api ( #43766 )
...
relate: https://github.com/milvus-io/milvus/issues/43687
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2025-08-08 14:17:41 +08:00
congqixia
c8789513eb
enhance: Bump milvus & proto version to v2.6.0 ( #43704 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-08-05 15:27:39 +08:00
tinswzy
75666153e3
enhance: add internal writer without session lock ( #43675 )
...
#43638
- add internal writer without session lock
- modify lastReadState pb type
Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
2025-08-02 09:25:37 +08:00
tinswzy
1fe60520ae
enhance: update wp version v0.1.3 ( #43658 )
...
#43638
update wp to v0.1.3.
- Fix the goroutine leak of adv file reader.
- Refactor the log reader time wait logic.
- The server segment file reuses the reader singleton.
Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
2025-07-31 14:17:37 +08:00
tinswzy
1718b0d141
enhance: update wp version v0.1.2 ( #43636 )
...
#43638
update wp to v0.1.2
fix read failure when minio is killed during data reading. related wp
commit#[aabd1c4eb2](aabd1c4eb2
2025-07-30 14:39:36 +08:00
tinswzy
173efe2b98
enhance: wp metrics and update deps to v0.1.0 ( #43569 )
...
#43574 #43604 #43431 #43603
Fix wp metrics not registered bug;
Update the version dependent on wp to v0.1.2-rc1;
improve advanced reader with concurrent prefetch blks;
add the segment rolling policy based on the number of blocks;
improve concurrent compaction
release lock failed bug
Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
2025-07-29 14:51:35 +08:00
tinswzy
83f6811dbd
fix: local fs incomplete block read bug ( #43444 )
...
#43340 fix log reader bug
#43370 list object goroutine leak ; block flush bug
#43431 #43356 improve read latency
other fix: local FS block CRC fix; incomplete block read bugfix;
multi-segment rolling not complete bug; local fs concurent flush bug
other enhance: log reader EOF-based segment end detection ; revisioned
log/segment meta updates.
Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
2025-07-25 10:30:54 +08:00
tinswzy
26f2de4bcf
fix: fence failure and remove list API usage ( #43365 )
...
#43356 #43370 fence fail ； goroutine leaks
#43313 record too large
Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
2025-07-18 11:22:51 +08:00
congqixia
5d90b65342
enhance: [StorageV2] Add storage version in Data/Query view resp ( #43348 )
...
Related to #39173
Add `storage_version` in data/query view segment info response
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-07-16 15:52:51 +08:00
tinswzy
b5a1937699
fix: wp refuses to write only when both payload and properties are empty ( #43319 )
...
#43313
Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
2025-07-16 14:42:50 +08:00
liliu-z
003c348d6d
enhance: Upgrade go version to 1.24.4 to fix CVEs ( #43019 )
...
Signed-off-by: liliu-z <liliu-z@users.noreply.github.com>
Co-authored-by: liliu-z <liliu-z@users.noreply.github.com>
2025-07-16 11:28:50 +08:00
tinswzy
0aeac94f8a
fix: no such file error was reported when reading an empty segment in local mode ( #43284 )
...
#43185
Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
2025-07-14 19:30:49 +08:00
tinswzy
7da62698e0
enhance: improve WP parallel sync mechanism and fencing logic ( #42892 )
...
related: #42595
improve WP parallel sync mechanism and fencing logic; remove redundant
metrics and labels
Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
2025-07-13 23:04:49 +08:00
tinswzy
754ca58469
enhance: improve WP parameters according to performance testing ( #42666 )
2025-06-13 20:33:41 +08:00
congqixia
9976e8e297
enhance: Bump milvus, proto & client version to 2.6.0-rc.1 ( #42725 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-06-13 17:54:36 +08:00
Spade A
911a8df17c
feat: impl StructArray -- data storage support in segcore ( #42406 )
...
Ref https://github.com/milvus-io/milvus/issues/42148
This PR mainly enables segcore to support array of vector (read and
write, but not indexing). Now only float vector as the element type is
supported.
---------
Signed-off-by: SpadeA <tangchenjie1210@gmail.com>
Signed-off-by: SpadeA-Tang <tangchenjie1210@gmail.com>
2025-06-12 14:38:35 +08:00
tinswzy
36a4b74fc0
enhance: Adjust the default parameters of WP according to performance tests ( #42598 )
...
#42595
Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
2025-06-10 16:30:35 +08:00
tinswzy
80ac6722f6
fix issue where wp fast-fail pending appends leads to client retry duplicated write ( #42566 )
...
#42563
Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
2025-06-09 10:10:33 +08:00
tinswzy
74432db503
fix stuck insert by binding buffer and chan to ensure proper notification ( #42505 )
...
#41918 #42482 #42049 #42513
cherrypick: sn release memory after pop from heap
wp: Encapsulate buffer and chan into a single item for one-to-one
management and cleanup
Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
Co-authored-by: chyezh <chyezh@outlook.com>
2025-06-05 10:02:32 +08:00
tinswzy
f55f900c85
fix insert hang caused by WAL writer writing to a closing logfile ( #42078 )
...
related issue #42049
wp commit
[94de4](94de4cbc60
2025-06-03 09:58:36 +08:00
aoiasd
2ae4d80120
enhance: support run analyzer by loaded collection field ( #42113 )
...
relate: https://github.com/milvus-io/milvus/issues/42094
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2025-05-29 10:54:30 +08:00
groot
c00005bdaa
feat: support to drop properties of field ( #41996 )
...
issue: https://github.com/milvus-io/milvus/issues/41990
Signed-off-by: yhmo <yihua.mo@zilliz.com>
2025-05-27 14:32:34 +08:00
tinswzy
1735f557ca
fix sn oom issue during small file loading in wp ( #41946 )
...
#41846 #41894
Resolve SN OOM issue during small file loading in Woodpecker;
Correct WP fence/close execution order;
Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
2025-05-23 01:30:28 +08:00
yihao.dai
65dd3982d8
fix: Fix ants.Pool goroutine leak ( #41892 )
...
1. Release the pool after it is no longer in use.
2. Upgrade ants.Pool to fix the goroutine leak issue (see [PR
#287 ](https://github.com/panjf2000/ants/pull/287 )).
issue: https://github.com/milvus-io/milvus/issues/41838
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-05-19 17:56:22 +08:00
tinswzy
3d8629de3e
fix memory reuse in woodpecker to prevent streamingNode OOM ( #41918 )
...
#41846
Reduce woodpecker memory allocation frequency through recycled memory
pools, allowing GC to keep up with collection.
related [woodpecker issue 24
](https://github.com/zilliztech/woodpecker/issues/24 )
Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
2025-05-19 15:22:22 +08:00
tinswzy
4edb1bc6f1
fix: resolve wp WALImpls concurrent read/write bug ( #41763 )
...
#41563 #41579 #41842 #41846 #41758
Upgraded the wp dependency to incorporate recent fixes addressing
multiple concurrency bugs in WALImpls.
Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
2025-05-16 12:02:27 +08:00
congqixia
ba8f62a3b2
enhance: Bump x/net fixing CVE-2025-22872 ( #41861 )
...
Related to #41291
Related to CVE-2025-22872
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-05-15 19:04:23 +08:00
SimFG
9fa50e0b1a
enhance: implement authorization checks for DescribeCollection and DescribeDatabase tasks ( #41798 )
...
- issue: #41694
---------
Signed-off-by: SimFG <bang.fu@zilliz.com>
2025-05-15 17:52:23 +08:00
foxspy
1c794be119
enhance: Output index version information in the DescribeIndex interface ( #41847 )
...
issue: https://github.com/milvus-io/milvus/issues/41431
Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
2025-05-15 14:36:22 +08:00
Zhen Ye
0a465bb5b7
enhance: use recovery+shardmanager, remove segment assignment interceptor ( #41824 )
...
issue: #41544
- add lock interceptor into wal.
- use recovery and shardmanager to replace the original implementation
of segment assignment.
- remove redundant implementation and unittest.
- remove redundant proto definition.
- use 2 streamingnode in e2e.
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2025-05-14 23:00:23 +08:00
shaoyue
5e8966ec32
enhance: update golang-jwt to v4.5.2 to fix cve ( #41734 )
...
/cc @congqixia
Signed-off-by: haorenfsa <haorenfsa@gmail.com>
2025-05-13 10:58:56 +08:00
shaoyue
f0451d2f3a
enhance: update go version to fix cve ( #41696 )
...
/cc @congqixia @czs007
Signed-off-by: haorenfsa <haorenfsa@gmail.com>
2025-05-08 16:04:54 +08:00
XuanYang-cn
b6f3fd0de1
enhance: Update to the latest CipherPlugin API ( #41599 )
...
See also: #41585
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2025-04-29 14:50:50 +08:00
tinswzy
b36ed03141
enhance: Add Truncate Interface to WALImpls for Log Retention Control ( #41517 )
...
#41465 Add Truncate Interface to WALImpls for Proactive Log Retention
Management
Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
2025-04-29 14:36:49 +08:00
Zhen Ye
9cb5271027
enhance: remove support of embeded nats mq ( #41565 )
...
issue: #41564
Signed-off-by: chyezh <chyezh@outlook.com>
2025-04-28 23:06:49 +08:00
SimFG
91d40fa558
fix: Update logging context and upgrade dependencies ( #41318 )
...
- issue: #41291
---------
Signed-off-by: SimFG <bang.fu@zilliz.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2025-04-23 10:52:38 +08:00
tinswzy
6fa68c1f16
enhance: Support Woodpecker as a WAL storage option for Milvus ( #41095 )
...
#40916 Support Woodpecker as a WAL storage option for Milvus
Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
2025-04-20 22:22:42 +08:00
dependabot[bot]
b1e092673b
enhance: bump github.com/golang-jwt/jwt/v5 from 5.2.1 to 5.2.2 ( #40923 )
...
Bumps [github.com/golang-jwt/jwt/v5](https://github.com/golang-jwt/jwt )
from 5.2.1 to 5.2.2.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/golang-jwt/jwt/releases ">github.com/golang-jwt/jwt/v5's
releases</a>.</em></p>
<blockquote>
<h2>v5.2.2</h2>
<h2>What's Changed</h2>
<ul>
<li>Fixed <a
href="https://github.com/golang-jwt/jwt/security/advisories/GHSA-mh63-6h87-95cp ">https://github.com/golang-jwt/jwt/security/advisories/GHSA-mh63-6h87-95cp </a>
by <a
href="https://github.com/mfridman "><code>@mfridman</code></a></li>
<li>Fixed some typos by <a
href="https://github.com/Ashikpaul "><code>@Ashikpaul</code></a> in <a
href="https://redirect.github.com/golang-jwt/jwt/pull/382 ">golang-jwt/jwt#382</a></li>
<li>build: add go1.22 to ci workflows by <a
href="https://github.com/mfridman "><code>@mfridman</code></a> in <a
href="https://redirect.github.com/golang-jwt/jwt/pull/383 ">golang-jwt/jwt#383</a></li>
<li>Bump golangci/golangci-lint-action from 4 to 5 by <a
href="https://github.com/dependabot "><code>@dependabot</code></a> in <a
href="https://redirect.github.com/golang-jwt/jwt/pull/387 ">golang-jwt/jwt#387</a></li>
<li>Bump golangci/golangci-lint-action from 5 to 6 by <a
href="https://github.com/dependabot "><code>@dependabot</code></a> in <a
href="https://redirect.github.com/golang-jwt/jwt/pull/389 ">golang-jwt/jwt#389</a></li>
<li>chore: bump ci tests to include go1.23 by <a
href="https://github.com/mfridman "><code>@mfridman</code></a> in <a
href="https://redirect.github.com/golang-jwt/jwt/pull/405 ">golang-jwt/jwt#405</a></li>
<li>Fix jwt -show by <a
href="https://github.com/AlexanderYastrebov "><code>@AlexanderYastrebov</code></a>
in <a
href="https://redirect.github.com/golang-jwt/jwt/pull/406 ">golang-jwt/jwt#406</a></li>
<li>docs: typo by <a
href="https://github.com/kvii "><code>@kvii</code></a> in <a
href="https://redirect.github.com/golang-jwt/jwt/pull/407 ">golang-jwt/jwt#407</a></li>
<li>Update SECURITY.md by <a
href="https://github.com/oxisto "><code>@oxisto</code></a> in <a
href="https://redirect.github.com/golang-jwt/jwt/pull/416 ">golang-jwt/jwt#416</a></li>
<li>Update <code>jwt.Parse</code> example to use
<code>jwt.WithValidMethods</code> by <a
href="https://github.com/mattt "><code>@mattt</code></a> in <a
href="https://redirect.github.com/golang-jwt/jwt/pull/425 ">golang-jwt/jwt#425</a></li>
</ul>
<h2>New Contributors</h2>
<ul>
<li><a href="https://github.com/Ashikpaul "><code>@Ashikpaul</code></a>
made their first contribution in <a
href="https://redirect.github.com/golang-jwt/jwt/pull/382 ">golang-jwt/jwt#382</a></li>
<li><a href="https://github.com/kvii "><code>@kvii</code></a> made their
first contribution in <a
href="https://redirect.github.com/golang-jwt/jwt/pull/407 ">golang-jwt/jwt#407</a></li>
<li><a href="https://github.com/mattt "><code>@mattt</code></a> made
their first contribution in <a
href="https://redirect.github.com/golang-jwt/jwt/pull/425 ">golang-jwt/jwt#425</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/golang-jwt/jwt/compare/v5.2.1...v5.2.2 ">https://github.com/golang-jwt/jwt/compare/v5.2.1...v5.2.2 </a></p>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="0951d18428c035977d9ehttps://redirect.github.com/golang-jwt/jwt/issues/425 ">#425</a>)</li>
<li><a
href="bc8bdca5cchttps://redirect.github.com/golang-jwt/jwt/issues/416 ">#416</a>)</li>
<li><a
href="5ec246c074https://redirect.github.com/golang-jwt/jwt/issues/407 ">#407</a>)</li>
<li><a
href="0123f1ad66https://redirect.github.com/golang-jwt/jwt/issues/406 ">#406</a>)</li>
<li><a
href="f961c72abdhttps://redirect.github.com/golang-jwt/jwt/issues/405 ">#405</a>)</li>
<li><a
href="62e504c281https://redirect.github.com/golang-jwt/jwt/issues/389 ">#389</a>)</li>
<li><a
href="1a56dcf532https://redirect.github.com/golang-jwt/jwt/issues/387 ">#387</a>)</li>
<li><a
href="c8043eab61https://redirect.github.com/golang-jwt/jwt/issues/383 ">#383</a>)</li>
<li><a
href="7c3f6dc563https://redirect.github.com/golang-jwt/jwt/issues/382 ">#382</a>)</li>
<li>See full diff in <a
href="https://github.com/golang-jwt/jwt/compare/v5.2.1...v5.2.2 ">compare
view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores )
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
You can disable automated security fix PRs for this repo from the
[Security Alerts
page](https://github.com/milvus-io/milvus/network/alerts ).
</details>
---------
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
2025-03-27 00:56:21 +08:00
congqixia
84e8e141ea
enhance: Support detailed manual compaction criterion ( #40892 )
...
Related to #40866
This PR:
- update go-api/v2 and support partition id/channel/segment level manual
compaction
- refines the compaction trigger implementation
- unify the compaction signal usage
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-03-25 20:06:22 +08:00
SimFG
16efcda5c4
feat: support to deny dll according to database property ( #40764 )
...
- issue: #40762
Signed-off-by: SimFG <bang.fu@zilliz.com>
2025-03-23 11:18:28 +08:00
aoiasd
92bdf7a0c1
enhance: support run anayser return detaild token ( #40458 )
...
relate: https://github.com/milvus-io/milvus/issues/39705
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2025-03-19 15:48:15 +08:00
Zhen Ye
8db708f67d
enhance: enable memory prof based on jemalloc ( #40731 )
...
issue: #40730
also see: https://github.com/milvus-io/cgosymbolizer/pull/2
After these PR, at linux:
- the milvus will always enable jemalloc by default.
- jemalloc will always compiled with --enable-prof options.
- all image will always enable the jemalloc prof by default.
- a pprof http service for jemalloc at `/debug/jemalloc/` will be
registered into restful.
- `jeprof` can remote profile the memory of milvus.
Signed-off-by: chyezh <chyezh@outlook.com>
2025-03-19 14:46:18 +08:00
