3168 Commits

Author SHA1 Message Date
yanliang567
3a951f2160
test: Cherry pick sparse invert index algo tests (#39816)
related issue: https://github.com/milvus-io/milvus/issues/39332
related pr: #39691

---------

Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
2025-02-17 15:06:15 +08:00
Xianhui Lin
f0964f769d
enhance: [2.5]Add json key inverted index in stats for optimization (#39876)
Add json key inverted index in stats for optimization
issue: https://github.com/milvus-io/milvus/issues/36995
pr: https://github.com/milvus-io/milvus/pull/38039

---------

Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2025-02-16 20:12:15 +08:00
smellthemoon
ba0edb8383
enhance: [2.5]return topks when search in restful v2 (#39839)
pr: #39812

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2025-02-14 11:30:14 +08:00
Zhen Ye
f85bf59921
fix: rootcoord graceful stop failed and limit resource of ci (#39793)
issue: #38399
pr: #39781

---------

Signed-off-by: chyezh <chyezh@outlook.com>
2025-02-13 19:34:13 +08:00
congqixia
7ca5e8309b
enhance: [GoSDK][2.5] Sync gosdk commits from master branch (#39823)
Cherry-pick from master
pr: #38791 #38812 #38940 #39515 #39580 #39662 #39665 #39751

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-02-12 18:46:47 +08:00
wei liu
11cba57dc7
fix: [2.5] load collection stucks if compaction/gc happens (#39761)
issue: #39680
pr: #39701
if compaction/gc happens, load collection may stuck due to
SegmentNotFound, we should trigger UpdateNextTarget to get a new data
view to execute loading operation.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2025-02-11 15:48:50 +08:00
Zhen Ye
858dc10ef9
enhance: broadcast with event-based notification (#39550)
issue: #38399
pr: #39522

- broadcast message can carry multi resource key now.
- implement event-based notification for broadcast messages
- broadcast message use broadcast id as a unique identifier in message
- broadcasted message on vchannels keep the broadcasted vchannel now.
- broadcasted message and broadcast message have a common broadcast
header now.

---------

Signed-off-by: chyezh <chyezh@outlook.com>
2025-02-07 11:50:50 +08:00
congqixia
6f7b2b4e75
enhance: [2.5] Refine error msg for schema & index checking (#39533) (#39565)
Cherry-pick from master
pr: #39533

The error message was malformated or missing some meta info, say field
name. This PR recitfies some message format and add field name in error
message when type param check fails.

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-01-24 13:43:06 +08:00
ThreadDao
c1c1f2df4e
test: [2.5] add test cases for template param (#39475)
issue: #33419
pr: #38867

- add case for Get
- add case for TemplateParam

Signed-off-by: ThreadDao <yufen.zong@zilliz.com>
2025-01-22 09:55:24 +08:00
qixuan
a90bd9e417
test:[cp2.5] add search iterator v2 case for milvus client (#39447)
issue: #37548 
pr: #39400

Signed-off-by: qixuan <673771573@qq.com>
2025-01-22 09:53:06 +08:00
Xianhui Lin
20c6de156e
feat: update milvusproto version to 2.5.5 (#39478)
feat: update milvusproto version to 2.5
issue: https://github.com/milvus-io/milvus/issues/39093

Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
2025-01-21 15:11:05 +08:00
zhuwenxing
f704e3edef
test: [skip e2e][cherry-pick]fix pytest_addoption type (#39397)
pr: https://github.com/milvus-io/milvus/pull/38860

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2025-01-17 16:59:02 +08:00
zhuwenxing
b2f4df49d6
test: [cherry-pick]add scalar index for compatibility testing (#39380)
pr: https://github.com/milvus-io/milvus/pull/39291

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2025-01-17 16:11:02 +08:00
congqixia
e68d161e60
enhance: Bump milvus & proto version to v2.5.4 (#39330)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-01-16 15:39:03 +08:00
sthuang
4a47f760b3
fix: [2.5] rbac custom group privilege level check (#39164) (#39224)
cherry-pick from master: https://github.com/milvus-io/milvus/pull/39164
related: https://github.com/milvus-io/milvus/issues/39086

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2025-01-14 16:50:59 +08:00
zhenshan.cao
99a8274326
enhance: Add primary field name in SearchResult and QueryResults (#39222)
pr: https://github.com/milvus-io/milvus/pull/39220
issue: https://github.com/milvus-io/milvus/issues/39219

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2025-01-14 16:10:58 +08:00
Zhen Ye
54036bcafd
enhance: add broadcast operation for msgstream (#39119)
issue: #38399
pr: #39040

- make broadcast service available for msgstream by reusing the
architecture streaming service

---------

Signed-off-by: chyezh <chyezh@outlook.com>
2025-01-14 15:11:00 +08:00
congqixia
ac730be5aa
enhance: Bump milvus & proto version to v2.5.3 (#39198)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-01-13 12:56:57 +08:00
sthuang
3059fae40d
fix: [2.5] fix restore rbac empty meta crash (#39143)
cp from master: https://github.com/milvus-io/milvus/pull/39141
related: https://github.com/milvus-io/milvus/issues/38985

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2025-01-13 12:04:58 +08:00
zhuwenxing
95df38e24f
test: add text match negative test case (#38982)
pr: https://github.com/milvus-io/milvus/pull/38981

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2025-01-11 14:21:03 +08:00
Zhen Ye
95809ca767
enhance: make new go package to manage proto (#39128)
issue: #39095
pr: #39114

---------

Signed-off-by: chyezh <chyezh@outlook.com>
2025-01-10 10:53:01 +08:00
ThreadDao
c2532def02
test: update cases for search params support and groupby issue fix (#38876)
pr: #38852
- update cases for WithSearchParam
- remove skip for groupby search issue
- update output query dynamic case for server changes

Signed-off-by: ThreadDao <yufen.zong@zilliz.com>
2025-01-09 19:12:56 +08:00
yanliang567
76c66f07be
test: [cp2.5]Add search with hints for different expressions (#39062)
issue: https://github.com/milvus-io/milvus/issues/38877
pr: #39048

---------

Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
2025-01-08 16:24:57 +08:00
cai.zhang
e6dd3e5a57
fix: [2.5]Remove valid expressions from invalid expressions (#38999)
issue: #39014 

master pr: #38957 
master pr: #39012

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2025-01-06 18:02:55 +08:00
yanliang567
aceb028e3a
test: [cp2.5]Refactor pymilvus client v2 testbase and add a test for search hint (#39006)
issue: https://github.com/milvus-io/milvus/issues/38877

    refactor pymilvus client v2 testcasebase
    add a test for search hint
    update pymilvus to 2.6

pr: #38939

---------

Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
2025-01-06 17:02:55 +08:00
nico
498b984607
enhance: delete one pod pipeline in nightly and open mmap in standalone(2.5) (#38888)
pr: #38887

Signed-off-by: nico <cheng.yuan@zilliz.com>
2025-01-03 16:38:55 +08:00
congqixia
6433f76c61
enhance: [2.5] Bump golang/x/net to v0.33 fixing security alarm (#38850) (#38932)
Cherry-pick from master
pr: #38850
Related to https://github.com/milvus-io/milvus/security/dependabot/106

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-01-03 14:50:54 +08:00
Patrick Weizhi Xu
1d55ad6a1d
enhance: [2.5] add param for tuning max VARCHAR length and restore limit to 65535 (#38883)
issue: #38882
pr: https://github.com/milvus-io/milvus/pull/38884

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
(cherry picked from commit a64a7377ea8258efe36c74bd1cf9db556726ba2d)
2025-01-02 13:56:54 +08:00
ThreadDao
cf148f9bd2
test: [2.5] add cases for async milvus client (#38699) (#38853)
pr: #38699
issue: #38697

Signed-off-by: ThreadDao <yufen.zong@zilliz.com>
2024-12-31 11:30:52 +08:00
yanliang567
e6c3d40d90
test: [cp2.5]Add alter API tests (#38843)
related issue: https://github.com/milvus-io/milvus/issues/38471
related pr: #38659 

Also add a test for hybrid search with expr templates

---------

Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
2024-12-30 16:52:52 +08:00
foxspy
3d723676ac
enhance: [2.5]update knowhere version (#38785)
/kind branch-feature
release note: https://github.com/zilliztech/knowhere/releases/tag/v2.5.1

Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
2024-12-27 10:24:50 +08:00
congqixia
a67b57275c
enhance: Bump milvus & proto version to v2.5.1 (#38774)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-12-26 16:36:50 +08:00
sthuang
a4901ef7ec
fix: [2.5] fix privilege group list and list collections (#38738)
cherry-pick from: https://github.com/milvus-io/milvus/pull/38684
related: https://github.com/milvus-io/milvus/issues/37031

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-12-25 18:06:50 +08:00
wei liu
b16d04d7cc
fix: Fix update loading collection's load config doesn't work (#38737)
issue: #38594 
pr: #38595

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-12-25 15:02:50 +08:00
Xianhui Lin
3cf9b6bae5
enhance: [2.5]field stringtype maxlength raise to 1M (#38614)
enhance: field stringtype maxlength raise to 1M
issue: https://github.com/milvus-io/milvus/issues/37436
master pr: https://github.com/milvus-io/milvus/pull/38592

Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
2024-12-20 17:08:45 +08:00
congqixia
b132438c03
enhance: Bump milvus & proto verison to v2.5.0 (#38610)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-12-20 14:30:46 +08:00
congqixia
c39db11509
enhance: [GoSDK] Sync API names and add missing APIs (#38603)
Related to #31293

- Rename `UsingDatabase` to `UseDatabase`
- Uncomment default value methods
- Add missing RBAC APIs
- Add some resource group APIs

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-12-20 11:52:46 +08:00
XuanYang-cn
ca7ec23198
enhance: Use partitionID when delete by partitionKey (#38231)
When delete by partition_key, Milvus will generates L0 segments
globally. During L0 Compaction, those L0 segments will touch all
partitions collection wise. Due to the false-positive rate of segment
bloomfilters, L0 compactions will append false deltalogs to completed
irrelevant partitions, which causes *partition deletion amplification.

This PR uses partition_key to set targeted partitionID when producing
deleteMsgs into MsgStreams. This'll narrow down L0 segments scope to
partition level, and remove the false-positive influence
collection-wise.

However, due to DeleteMsg structure, we can only label one partition to
one deleteMsg, so this enhancement fails if user wants to delete over 2
partition_keys in one deletion.

See also: #34665

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-12-20 11:18:46 +08:00
Bingyi Sun
894c203f27
enhance: allow hyphen in partition name (#38494)
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-12-18 15:04:45 +08:00
nico
fb0e689617
test: update sdk version and add cases (#38478)
Signed-off-by: nico <cheng.yuan@zilliz.com>
2024-12-18 10:10:57 +08:00
zhuwenxing
b694e06488
test: add lib for supporting higher python version (#38475)
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-12-17 14:48:50 +08:00
Bingyi Sun
3e2a2f278b
enhance: Handle rust error in c++ (#38113)
https://github.com/milvus-io/milvus/issues/37930

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-12-16 19:40:45 +08:00
ThreadDao
8794ec966e
test: add go case for groupby search (#38411)
issue: #33419

---------

Signed-off-by: ThreadDao <yufen.zong@zilliz.com>
2024-12-16 10:44:43 +08:00
tinswzy
27229f7907
enhance: refine exists log print with ctx (#38080)
issue: #35917 
Refines exists log print with ctx

Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
2024-12-14 22:36:44 +08:00
congqixia
fe79babdb3
enhance: Bump golang/x/crypto for all packages (#38445)
Related to #38446
See also https://github.com/milvus-io/milvus/security/dependabot/103

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-12-13 19:12:42 +08:00
Xianhui Lin
f2d79b3d3a
enhance: altercollection validation logic (#38419)
altercollection validation logic
issue: https://github.com/milvus-io/milvus/issues/37436

---------

Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
2024-12-12 21:42:42 +08:00
ThreadDao
59234a3350
test: add log level for go sdk test and update part cases (#38385)
- add log level for go sdk cases
- update cases for issue #33460 & #37853

---------

Signed-off-by: ThreadDao <yufen.zong@zilliz.com>
2024-12-12 16:58:43 +08:00
Xianhui Lin
74c6e0e94b
enhance: alterindex & altercollection supports altering properties updata pymilvus (#37437) (#38362)
enhance : 

1. alterindex delete properties
We have introduced a new parameter deleteKeys to the alterindex
functionality, which allows for the deletion of properties within an
index. This enhancement provides users with the flexibility to manage
index properties more effectively by removing specific keys as needed.
2. altercollection delete properties
We have introduced a new parameter deleteKeys to the altercollection
functionality, which allows for the deletion of properties within an
collection. This enhancement provides users with the flexibility to
manage collection properties more effectively by removing specific keys
as needed.

3.support altercollectionfield
We currently support modifying the fieldparams of a field in a
collection using altercollectionfield, which only allows changes to the
max-length attribute.
Key Points:
- New Parameter - deleteKeys: This new parameter enables the deletion of
specified properties from an index. By passing a list of keys to
deleteKeys, users can remove the corresponding properties from the
index.

- Mutual Exclusivity: The deleteKeys parameter cannot be used in
conjunction with the extraParams parameter. Users must choose one
parameter to pass based on their requirement. If deleteKeys is provided,
it indicates an intent to delete properties; if extraParams is provided,
it signifies the addition or update of properties.

issue: https://github.com/milvus-io/milvus/issues/37436

---------

Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>

---------

Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
2024-12-11 13:00:44 +08:00
wei liu
a118ca14a7
fix: Fix role be dropped when grant still exist. (#38342)
issue: #38325
the old impl only to check grant in default db before drop role, which
may cause role be dropped when grant still exist.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-12-11 11:24:42 +08:00
nico
d9a151e9e4
test: update test cases (#38330)
Signed-off-by: nico <cheng.yuan@zilliz.com>
2024-12-10 18:46:42 +08:00