3099 Commits

Author SHA1 Message Date
Yinzuo Jiang
5a06faca39
feat: fp32 vector to fp16/bf16 vector conversion for RESTful API (#37556)
RESTful API. The influenced API are as follows:

- Handler. insert
- HandlerV1. insert/upsert
- HandlerV2. insert/upsert/search

We do not modify search API in Handler/HandlerV1 because they do not
support fp16/bf16 vectors.

module github.com/milvus-io/milvus/pkg:

Add `Float32ArrayToBFloat16Bytes()`, `Float32ArrayToFloat16Bytes()` and
`Float32ArrayToBytes()`. These method will be used in GoSDK in the
future.

issue: #37448

Signed-off-by: Yinzuo Jiang <yinzuo.jiang@zilliz.com>
Signed-off-by: Yinzuo Jiang <jiangyinzuo@foxmail.com>
2024-11-24 17:46:33 +08:00
ThreadDao
bc131a911c
test: add cases for gosdk v2 load release (#37831)
issue: #33419

Signed-off-by: ThreadDao <yufen.zong@zilliz.com>
2024-11-22 10:54:32 +08:00
sthuang
19572f5b06
enhance: RBAC new grant/revoke privilege (#37785)
issue: https://github.com/milvus-io/milvus/issues/37031
also fix issues: https://github.com/milvus-io/milvus/issues/37843,
https://github.com/milvus-io/milvus/issues/37842,
https://github.com/milvus-io/milvus/issues/37887

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-11-21 22:20:34 +08:00
Buqian Zheng
511edd29fd
enhance: disallow get raw vector data of a BM25 Function output field (#37800)
issue: https://github.com/milvus-io/milvus/issues/35853

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-11-20 14:22:30 +08:00
nish112022
484c6b5c44
feat: Added code for Internal-tls (#36865)
issue : https://github.com/milvus-io/milvus/issues/36864

I have a few questions regarding my approach.I will consolidate them
here for feedback and review.Thanks

---------

Signed-off-by: Nischay Yadav <nischay.yadav@ibm.com>
Signed-off-by: Nischay <Nischay.Yadav@ibm.com>
2024-11-20 06:00:32 +08:00
nico
866d512f6f
test: update test cases and sdk version (#37774)
Signed-off-by: nico <cheng.yuan@zilliz.com>
2024-11-19 14:14:32 +08:00
cai.zhang
dae4160466
enhance: Whether to enable mergeSort mode when performing mixCompaction (#37664)
issue: #37579

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-11-19 11:28:31 +08:00
sthuang
2d72ad33f2
enhance: RBAC built in privilege groups (#37720)
issue: #37031

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-11-18 20:38:39 +08:00
zhuwenxing
3cdb485022
test: update timeout of load in ci cases (#37596)
action for
https://github.com/milvus-io/milvus/issues/37166#issuecomment-2469502955

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-11-16 16:00:31 +08:00
zhuwenxing
3f7352f3cf
test: remove xfail of fts test cases after fix (#37724)
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-11-16 11:14:30 +08:00
wei liu
a1b6be1253
fix: Delegator stuck at unserviceable status (#37694)
issue: #37679

pr #36549 introduce the logic error which update current target when
only parts of channel is ready.

This PR fix the logic error and let dist handler keep pull distribution
on querynode until all delegator becomes serviceable.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-11-15 10:20:31 +08:00
foxspy
5ae347aba0
enhance: update knowhere version (#37688)
issue: #37665 #37631 #37620 #37587 #36906 
knowhere has add default nlist value, so some invalid param test ut with
no nlist param will be valid.

Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
2024-11-15 10:10:31 +08:00
nico
405c12f7d1
test: update test cases (#37660)
Signed-off-by: nico <cheng.yuan@zilliz.com>
2024-11-14 17:46:39 +08:00
yanliang567
af433ffd68
test: Add expression template tests and use error msg only as assertion (#37618)
related issue: https://github.com/milvus-io/milvus/issues/37451

---------

Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
2024-11-13 17:14:30 +08:00
smellthemoon
a654487995
fix: some error in restful (#37487)
#37370

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-11-13 17:12:39 +08:00
nico
2630717836
test: update sdk version and cases (#37507)
Signed-off-by: nico <cheng.yuan@zilliz.com>
2024-11-12 17:48:28 +08:00
jaime
1e8ea4a7e7
feat: add segment/channel/task/slow query render (#37561)
issue: #36621

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-11-12 17:44:29 +08:00
Bingyi Sun
2742e9573f
enhance: Print results if a test fails (#37606)
test_hybrid_search_different_limit_round_decimal fails occasionally. 
Print more details for debug.

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-11-12 17:12:28 +08:00
wei liu
266f8ef1f5
fix: Search may return less result after qn recover (#36549)
issue: #36293 #36242
after qn recover, delegator may be loaded in new node, after all segment
has been loaded, delegator becomes serviceable. but delegator's target
version hasn't been synced, and if search/query comes, delegator will
use wrong target version to filter out a empty segment list, which
caused empty search result.

This pr will block delegator's serviceable status until target version
is synced

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-11-12 16:34:28 +08:00
Zhen Ye
b5b003551e
enhance: use localhost for it and ut (#37529)
issue: #37528

Signed-off-by: chyezh <chyezh@outlook.com>
2024-11-12 11:36:27 +08:00
XuanYang-cn
a45a288a25
fix: Separate L0 and Mix trigger interval (#37190)
See also: #37108

- Add MixCompactionTriggerInterval, default 60s
- Add L0CompactionTriggerInterval, default 10s
- Export Single related compaction configs
- Raise SingleCompactionDeltaLogMaxSize from 2MB to 16MB

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-11-12 10:56:37 +08:00
Chun Han
2d29dcd30c
enhance:refine group_strict_size parameter(#37482) (#37483)
related: #37482

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-11-12 09:56:28 +08:00
binbin
21b68029a0
test: Add nullable test cases for bulk writer (#37572)
issue: #36129

Signed-off-by: binbin lv <binbin.lv@zilliz.com>
2024-11-12 09:46:28 +08:00
congqixia
83d2729f94
enhance: Generate milvusclient mockery with latest proto (#37567)
Related to #37558

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-11-11 14:28:30 +08:00
ThreadDao
f42869c0ce
test: remove case skip label (#37539)
[test] remove case skip label
issue: #33952

Signed-off-by: ThreadDao <yufen.zong@zilliz.com>
2024-11-10 16:14:27 +08:00
aoiasd
12951f0abb
enhance: rename tokenizer to analyzer and check analyzer params (#37478)
relate: https://github.com/milvus-io/milvus/issues/35853

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-11-10 16:12:26 +08:00
sthuang
ff00a12805
enhance: RBAC custom privilege group ut coverage (#37558)
issue: https://github.com/milvus-io/milvus/issues/37031

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-11-09 20:40:25 +08:00
zhuwenxing
ffdde391e3
test: fix unstable testcase caused by server refactor (#37545)
issue: https://github.com/milvus-io/milvus/issues/37533

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-11-09 09:58:26 +08:00
sthuang
70605cf5b3
enhance: Support custom privilege group for RBAC (#37087)
issue: #37031

---------

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-11-09 08:44:28 +08:00
cai.zhang
ae227e3934
enhance: Add integration test for stats task (#37506)
issue: #33744

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-11-08 15:48:26 +08:00
congqixia
11f4fe0177
enhance: [GoSDK] move client pkg go files to sub one (#37492)
Related to #31293

Client source files under client pkg cannot be evaluate correctly by
codecov. This PR moves them to `milvusclient` sub-package to fix this
issue and follow go major version best practice.

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-11-08 07:32:26 +08:00
smellthemoon
86fd3200be
enhance: refactor createIndex in RESTful API (#37235)
Make the parameter input method consistent with miluvs-client.

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-11-07 14:18:30 +08:00
Buqian Zheng
40b770cb7b
fix: add rescorer activation function for BM25 (#37481)
issue: #35853

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-11-07 12:08:25 +08:00
cai.zhang
aed3b94b5d
enhance: Refine error message for contains array (#37383)
issue: #36221

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-11-07 10:44:25 +08:00
wei liu
7cfd609ebc
fix: [skip e2e]unstable integration test TestNodeDownOnSingleReplica (#37480)
issue: #37289
cause pr #37116 introduce retry on get shard leader, which make search
won't fail during query node down.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-11-06 17:08:21 -08:00
nico
cf66fbee8a
test: update test cases (#37215)
Signed-off-by: nico <cheng.yuan@zilliz.com>
2024-11-06 17:50:25 +08:00
aoiasd
d67853fa89
feat: Tokenizer support build with params and clone for concurrency (#37048)
relate: https://github.com/milvus-io/milvus/issues/35853
https://github.com/milvus-io/milvus/issues/36751

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-11-06 17:48:24 +08:00
congqixia
c83b93946e
enhance: [GoSDK] Add alter collection API & expose options (#37365)
Related to #31293

This PR:

- Add `AlterCollection` API for collection property modification
- Expose hidden or missing option methods

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-11-06 14:36:25 +08:00
zhuwenxing
0fc6c634b0
test: fix tokenizer and monkey patch faker function (#37119)
/kind improvement

---------

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
Signed-off-by: zhuwenxing <wxzhuyeah@gmail.com>
2024-11-05 08:42:23 +08:00
cai.zhang
ba9f36ba33
enhance: Increase the concurrency of index tasks in CI (#37335)
#37096

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-11-04 12:04:23 +08:00
foxspy
eaf86f7649
fix: optimize invalid datatype error msg (#37376)
issue: #37151

Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
2024-11-04 11:46:28 +08:00
cai.zhang
50de122dc7
enhance: Rename textmatch to text_match (#37290)
issue: #36672

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-11-03 18:40:27 +08:00
congqixia
be71b98146
enhance: Fix case error msg after knowhere upgrade (#37339)
Previous PR: #37315

Error message updated after knowhere behavior change, this PR update
corresponding test error message.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-11-01 14:00:22 +08:00
wei liu
f190e5d802
fix: [skip e2e] TestNodeDownOnSingleReplica has unstable result (#37288)
issue: #37289
those test case use search to verify replica's status, but if the search
gap is 1s, the node down's effect may be fixed up by balance.

This PR remove the 1 second gap between search operation.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-11-01 10:50:21 +08:00
zhuwenxing
247f75180f
test: add restful cases for full text search and some minor fix (#37148)
/kind improvement

---------

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-10-31 21:18:23 +08:00
zhuwenxing
d24970c090
test: fix potential unstable cases of full-text search (#37328)
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-10-31 21:12:21 +08:00
zhenshan.cao
63843dce33
fix: Fix conan gdal building problem (#37338)
issue:https://github.com/milvus-io/milvus/issues/27576

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-10-31 21:04:16 +08:00
Hao Tan
67c4340565
feat: Geospatial Data Type and GIS Function Support for milvus server (#35990)
issue:https://github.com/milvus-io/milvus/issues/27576

# Main Goals
1. Create and describe collections with geospatial fields, enabling both
client and server to recognize and process geo fields.
2. Insert geospatial data as payload values in the insert binlog, and
print the values for verification.
3. Load segments containing geospatial data into memory.
4. Ensure query outputs can display geospatial data.
5. Support filtering on GIS functions for geospatial columns.

# Solution
1. **Add Type**: Modify the Milvus core by adding a Geospatial type in
both the C++ and Go code layers, defining the Geospatial data structure
and the corresponding interfaces.
2. **Dependency Libraries**: Introduce necessary geospatial data
processing libraries. In the C++ source code, use Conan package
management to include the GDAL library. In the Go source code, add the
go-geom library to the go.mod file.
3. **Protocol Interface**: Revise the Milvus protocol to provide
mechanisms for Geospatial message serialization and deserialization.
4. **Data Pipeline**: Facilitate interaction between the client and
proxy using the WKT format for geospatial data. The proxy will convert
all data into WKB format for downstream processing, providing column
data interfaces, segment encapsulation, segment loading, payload
writing, and cache block management.
5. **Query Operators**: Implement simple display and support for filter
queries. Initially, focus on filtering based on spatial relationships
for a single column of geospatial literal values, providing parsing and
execution for query expressions.
6. **Client Modification**: Enable the client to handle user input for
geospatial data and facilitate end-to-end testing.Check the modification
in pymilvus.

---------

Signed-off-by: tasty-gumi <1021989072@qq.com>
2024-10-31 20:58:20 +08:00
cai.zhang
2ef6cbbf59
feat: The expression supports filling elements through templates (#37033)
issue: #36672

The expression supports filling elements through templates, which helps
to reduce the overhead of parsing the elements.

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-10-31 14:20:22 +08:00
zhuwenxing
6e37372619
test: update checker (#37275)
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-10-31 09:50:20 +08:00