admin/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2026-01-05 10:22:41 +08:00
Code Issues Packages Projects Releases Wiki Activity
22,696 Commits 136 Branches 188 Tags
Commit Graph

22696 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
yihao.dai
1984be646c
fix: Fix storagev2 binlog import (#43221) 
issue: https://github.com/milvus-io/milvus/issues/43218

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2025-07-13 22:52:49 +08:00
Alexander Guzhva
a848c4a8c5
fix: fix incorrect bitset for the division comparison when the right is < 0 (#43179) 
issue: https://github.com/milvus-io/milvus/issues/42900
@sunby Unfortunately, it is not that easy to fix as it was thought in
#43177

Upd: also handles `Inf` and `NaN` values, and the division by zero case
for `fp32` and `fp64`

Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>
 2025-07-11 19:04:49 +08:00
Zhen Ye
15a6631147
enhance: add quota limit based on sn consuming lag (#43105) 
issue: #42995

- The consuming lag at streaming node will be reported to coordinator.
- The consuming lag will trigger the write limit and deny by quota
center.
- Set the ttProtection by default.

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 2025-07-11 14:10:49 +08:00
cai.zhang
c54a04c71c
fix: L2 segments remain as L2 even after sort compaction (#43237) 
issue: #43186

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2025-07-11 11:30:48 +08:00
congqixia
002a325f0f
fix: [GoSDK] Slice valid data parsing nullable data (#43243) 
Related to #43003

When nq > 1, returning nullable data in search result will lead to
parsing error. This patch add slicing valid data logic to make nullable
parsing validation logic could work.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2025-07-11 10:32:48 +08:00
Zhen Ye
f598ca2b4e
fix: block at msgpack adaptor and wrong metrics (#43235) 
issue: #43018

Signed-off-by: chyezh <chyezh@outlook.com>
 2025-07-11 10:14:49 +08:00
congqixia
5a9efb3f81
enhance: [StorageV2] Refine storage rw option usage & validation (#43175) 
Related to #39173

This PR:
- Make all datanode task passes storage config via storage config option
- Remove legacy comments, rootPath & bucketName parameters
- Fix clustering compaction option behavior
- Add validation logic for `rwOptions`
- Use correct storageType from storageConfig
- Add storage config in sync task

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2025-07-11 01:14:48 +08:00
congqixia
6bbed3b019
fix: [AddField] Add shared_lock for insert prevent race (#43229) 
Related to #43113

When schema change happens, insert shall not happen, otherwise:
- Data race may happen causing insertion failure
- Inconsistent data schema

This PR add shared_lock prevent this data race.

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2025-07-10 21:26:48 +08:00
PjJinchen
a90694165b
feat: Supports tracing services that require header-based authentication. (#43211) 
issue: https://github.com/milvus-io/milvus/issues/43082

support tracing services that require header-based authentication.
for example: aliyun SLS, volcengine LogService etc...

[aliyun
SLS](https://help.aliyun.com/zh/sls/import-trace-data-from-golang-applications-to-log-service-by-using-opentelemetry-sdk-for-golang?spm=a2c4g.11186623.help-menu-search-28958.d_1#section-ktk-xxz-8om)

Add a headers config in trace config

```
trace:
  exporter: otlp
  sampleFraction: 1
  otlp:
    endpoint:  milvus-cn-beijing-pre.cn-beijing.log.aliyuncs.com:10010
    method:  # otlp export method, acceptable values: ["grpc", "http"],  using "grpc" by default
    secure: true
    headers:  # base64
  initTimeoutSeconds: 10
```

it is encoded as base64, raw data is json
```
{
    "x-sls-otel-project": "milvus-cn-beijing-pre",
    "x-sls-otel-instance-id": "milvus-cn-beijing-pre",
    "x-sls-otel-ak-id": "xxx",
    "x-sls-otel-ak-secret": "xxx"
}
```

[volcengine
tls](https://www.volcengine.com/docs/6470/812322#grpc-%E5%8D%8F%E8%AE%AE%E5%88%9D%E5%A7%8B%E5%8C%96%E7%A4%BA%E4%BE%8B)

Add a headers config in trace config

```
trace:
  exporter: otlp
  sampleFraction: 1
  otlp:
    endpoint:  xxx
    method:  # otlp export method, acceptable values: ["grpc", "http"],  using "grpc" by default
    secure: true
    headers:  # base64
  initTimeoutSeconds: 10
```

it is encoded as base64, raw data is json
```
{
    "x-tls-otel-region": "cn-beijing",
    "x-tls-otel-tracetopic": "milvus-cn-beijing-pre",
    "x-tls-otel-ak": "xxx",
    "x-tls-otel-sk": "xxx"
}
```

Signed-off-by: PjJinchen <6268414+pj1987111@users.noreply.github.com>
 2025-07-10 17:32:48 +08:00
zhuwenxing
85c8049296
test: add test cases for json modulo operation (#43191) 
/kind improvement

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
 2025-07-10 17:08:48 +08:00
wei liu
b2597c6329
enhance: apply load config changes after QueryCoord restart (#43108) 
issue: #43107 
- Add checkLoadConfigChanges() to apply load config during startup
- Call config check in startQueryCoord() after restart
- Skip auto-updates for collections with user-specified replica numbers
- Add is_user_specified_replica_mode field to preserve user settings
- Add comprehensive unit tests with mockey

Ensures existing collections use latest cluster-level config after
restart.

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2025-07-10 14:28:48 +08:00
cai.zhang
3ffd44f302
fix: Fix remaining issues with Datanode pooling and StorageV2 (#43147) 
issue: #43146

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2025-07-10 14:26:48 +08:00
yihao.dai
ee9a95189a
enhance: Print segments info after import done (#43200) 
issue: https://github.com/milvus-io/milvus/issues/42488

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2025-07-10 12:38:47 +08:00
Chun Han
07745439b5
fix: empty search groupby result causing crash(#43137) (#43214) 
related: #43137

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
 2025-07-10 12:04:48 +08:00
cai.zhang
47144429bf
fix: Fix regeneratePartitionStats failed after restore clusteringCompactionTask (#43205) 
issue: #43186

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2025-07-10 10:40:47 +08:00
Zhen Ye
490c5d5088
fix: lost message version after compatible message modification (#43217) 
issue: #43018

Signed-off-by: chyezh <chyezh@outlook.com>
 2025-07-10 10:36:48 +08:00
Bingyi Sun
13f6e2130b
fix: Fix hybrid search return back empty set if one result is emtpy (#43209) 
issue: https://github.com/milvus-io/milvus/issues/43160

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2025-07-10 10:34:55 +08:00
congqixia
f027eea545
enhance: [AddField] Add log for segcore segment schema change (#43215) 
Related to #39178

This PR add logs for segment schema change operations.

Also fixes the nit comments from PR #42490

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2025-07-10 10:22:47 +08:00
congqixia
7f8c5c9bb8
fix: [GoSDK] Handle insufficient topks in search result (#43126) 
Related to #43118

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2025-07-10 01:30:48 +08:00
aoiasd
97b1c3ed96
enhance: add warn log if some segment's bm25 stats lacks (#43111) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2025-07-09 23:22:47 +08:00
cai.zhang
95e767611a
fix: Fix merge sort loss data when last row in a record is deleted (#43216) 
issue: #43207

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2025-07-09 22:18:48 +08:00
qixuan
1cd593b673
test:add go sdk case for add field (#43033) 
issue: #42126

Signed-off-by: qixuan <673771573@qq.com>
 2025-07-09 18:20:47 +08:00
zhagnlu
21d1fb2aa3
fix: fix move cursor bug for chunk segment with index (#43095) 
#42974

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2025-07-09 17:38:47 +08:00
cai.zhang
41d1c8d6b3
fix: Handle error for invalid function params and prevent panic (#43189) 
issue: #43188

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2025-07-09 12:46:46 +08:00
tinswzy
c4634d861e
fix: v2.6 WebUI metrics response schema change bug (#42957) 
#42919  
fix metrics response schema incompatibility with WebUI v2.6

Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
 2025-07-08 22:56:47 +08:00
cai.zhang
6989e18599
enhance: Move sort stats task to sort compaction (#42562) 
issue: #42560

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2025-07-08 20:22:47 +08:00
zhuwenxing
f1e75085e8
test: add restful api case for rerank function (#42987) 
/kind improvement

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
 2025-07-08 19:36:46 +08:00
aoiasd
54cc0b60f2
fix: dropped segment in excluded segment use wrong excluded ts (#43115) 
cause some excluded growing data insert again
relate: https://github.com/milvus-io/milvus/issues/43114

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2025-07-08 18:04:46 +08:00
Spade A
d41eec6f10
fix: void copy when getting json chunk (#43183) 
fix: https://github.com/milvus-io/milvus/issues/43182

Signed-off-by: SpadeA <tangchenjie1210@gmail.com>
 2025-07-08 15:28:46 +08:00
cai.zhang
8720feeb79
fix: Fix enqueuing when current batch is fully deleted (#43174) 
issue: #43045

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2025-07-08 12:20:46 +08:00
Zhen Ye
ed9aa1d4db
fix: limit GC concurrency as CPU number (#43165) 
issue: #42833

Signed-off-by: chyezh <chyezh@outlook.com>
 2025-07-08 10:46:46 +08:00
Ted Xu
6153272d4b
enhance: disabling max entry limit by default (#43166) 
See: #43055

---------

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
 2025-07-08 10:10:46 +08:00
yihao.dai
9cbd194c6b
fix: Prevent import from generating small binlogs (#43132) 
- Introduce dynamic buffer sizing to avoid generating small binlogs
during import
- Refactor import slot calculation based on CPU and memory constraints
- Implement dynamic pool sizing for sync manager and import tasks
according to CPU core count

issue: https://github.com/milvus-io/milvus/issues/43131

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2025-07-07 21:32:47 +08:00
sthuang
a0ae5bccc9
fix: [StorageV2] load growing segment get dim datatype check (#43168) 
related: https://github.com/milvus-io/milvus/issues/43072

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
 2025-07-07 15:46:47 +08:00
congqixia
ab818dcbca
fix: [StorageV2] Pass storage config for compaction rw (#43167) 
Related to #43148

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2025-07-07 15:32:46 +08:00
sthuang
276c52490d
fix: [StorageV2] missing arrow fs when building index (#43162) 
fix: https://github.com/milvus-io/milvus/issues/43150,
https://github.com/milvus-io/milvus/issues/43149

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
 2025-07-07 15:26:46 +08:00
sthuang
9f361a228e
enhance: storage v2 chunked column memory size from meta (#43130) 
use meta to get chunked column memory size to avoid getting cells
actually from storage.
related: #39173

---------

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
 2025-07-07 14:24:46 +08:00
congqixia
d09764508a
fix: [Storagev2] Close segment readers in mergeSort (#43116) 
Related to #43062

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2025-07-04 23:56:44 +08:00
yanliang567
0017fa8acc
test: Enable storage v2 in ci and nightly runs (#43086) 
related issue: #43020

---------

Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
 2025-07-04 18:38:44 +08:00
junjiejiangjjj
fafd5db43f
fix: rank params bug (#43112) 
https://github.com/milvus-io/milvus/issues/42985

Signed-off-by: junjiejiangjjj <junjie.jiang@zilliz.com>
 2025-07-04 18:28:44 +08:00
Zhen Ye
46b6f1b9e2
fix: panic when logging a old message should be skipped (#43076) 
issue: #43074

- fix: panic when logging a old message should be skipped, #43074 
- fix: make the ack of broadcaster idompotent, #43026
- fix: lost dropping collection when upgrading, #43092
- fix: panic when DropPartition happen after DropCollection, #43027,
#43078

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 2025-07-04 16:04:44 +08:00
zhikunyao
37507e265a
test: mergify migrate to ci-v2 (#43140) 
Signed-off-by: Zhikun Yao <zhikun.yao@zilliz.com>
 2025-07-04 14:32:44 +08:00
groot
1ee8cea35b
enhance: bulkinsert handle nullable/defaultValue/functionOutput fields (#42956) 
issue: https://github.com/milvus-io/milvus/issues/42173

Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2025-07-04 14:20:44 +08:00
congqixia
684f027496
fix: Remove trimming space logic when validating collection name (#43064) 
Related to #43031

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2025-07-04 11:00:45 +08:00
cai.zhang
4133e3b8fd
fix: Enable merge sort and fix sort bug (#43080) 
issue: #42980, #43034

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2025-07-04 10:18:44 +08:00
Spade A
fce0bbe2ae
fix: remove redundant locks for null_offset (#43103) 
Ref: https://github.com/milvus-io/milvus/issues/40308
https://github.com/milvus-io/milvus/pull/40363 add lock for protecting
concurrent read/write for null offset. But we don't need this for sealed
segment.

---------

Signed-off-by: SpadeA <tangchenjie1210@gmail.com>
 2025-07-04 10:10:45 +08:00
Xianhui Lin
ee053fa244
enhance: remove unnecessary sleep in query and upsert test cases (#43102) 
remove unnecessary sleep in query and upsert test cases
https://github.com/milvus-io/milvus/issues/36995

Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
 2025-07-04 10:08:52 +08:00
Zhen Ye
e97e44d56e
enhance: limit the gc concurrency when cpu is high (#43059) 
issue: #42833

Signed-off-by: chyezh <chyezh@outlook.com>
 2025-07-04 09:22:43 +08:00
congqixia
1d9a9a993d
fix: [StorageV2] Use correct template typename for cache_raw_data_to_disk_common (#43104) 
Related to #43099

Previously `cache_raw_data_to_disk_common` used `milvus::DataType`
template typename, which shall be `knowhere::bf16` or other actual
datatype.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2025-07-03 18:50:46 +08:00
Zhen Ye
bbbc7d4517
enhance: collect all cgo calling into metric and log slow cgo call (#43035) 
issue: #42833

- also fix the error metric for async cgo.
- also make sure the roles can be seen when node startup, #43041.

Signed-off-by: chyezh <chyezh@outlook.com>
 2025-07-03 15:00:44 +08:00