154 Commits

Author SHA1 Message Date
SimFG
302650ae0e
fix: use the default partition for the limit quota when the request partition name is empty (#38005)
- issue: #37685

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-11-27 11:00:36 +08:00
Buqian Zheng
511edd29fd
enhance: disallow get raw vector data of a BM25 Function output field (#37800)
issue: https://github.com/milvus-io/milvus/issues/35853

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-11-20 14:22:30 +08:00
congqixia
b0bd290a6e
enhance: Use internal json(sonic) to replace std json lib (#37708)
Related to #35020

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-11-18 10:46:31 +08:00
foxspy
5ae347aba0
enhance: update knowhere version (#37688)
issue: #37665 #37631 #37620 #37587 #36906 
knowhere has add default nlist value, so some invalid param test ut with
no nlist param will be valid.

Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
2024-11-15 10:10:31 +08:00
aoiasd
12951f0abb
enhance: rename tokenizer to analyzer and check analyzer params (#37478)
relate: https://github.com/milvus-io/milvus/issues/35853

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-11-10 16:12:26 +08:00
wayblink
d119a2541a
fix: fix hasCollection response has no status (#37254)
issue: #37257

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-11-01 15:40:21 +08:00
zhenshan.cao
63843dce33
fix: Fix conan gdal building problem (#37338)
issue:https://github.com/milvus-io/milvus/issues/27576

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-10-31 21:04:16 +08:00
Hao Tan
67c4340565
feat: Geospatial Data Type and GIS Function Support for milvus server (#35990)
issue:https://github.com/milvus-io/milvus/issues/27576

# Main Goals
1. Create and describe collections with geospatial fields, enabling both
client and server to recognize and process geo fields.
2. Insert geospatial data as payload values in the insert binlog, and
print the values for verification.
3. Load segments containing geospatial data into memory.
4. Ensure query outputs can display geospatial data.
5. Support filtering on GIS functions for geospatial columns.

# Solution
1. **Add Type**: Modify the Milvus core by adding a Geospatial type in
both the C++ and Go code layers, defining the Geospatial data structure
and the corresponding interfaces.
2. **Dependency Libraries**: Introduce necessary geospatial data
processing libraries. In the C++ source code, use Conan package
management to include the GDAL library. In the Go source code, add the
go-geom library to the go.mod file.
3. **Protocol Interface**: Revise the Milvus protocol to provide
mechanisms for Geospatial message serialization and deserialization.
4. **Data Pipeline**: Facilitate interaction between the client and
proxy using the WKT format for geospatial data. The proxy will convert
all data into WKB format for downstream processing, providing column
data interfaces, segment encapsulation, segment loading, payload
writing, and cache block management.
5. **Query Operators**: Implement simple display and support for filter
queries. Initially, focus on filtering based on spatial relationships
for a single column of geospatial literal values, providing parsing and
execution for query expressions.
6. **Client Modification**: Enable the client to handle user input for
geospatial data and facilitate end-to-end testing.Check the modification
in pymilvus.

---------

Signed-off-by: tasty-gumi <1021989072@qq.com>
2024-10-31 20:58:20 +08:00
congqixia
d8c1bd24f2
enhance: Utilize proxy metacache for HasCollection (#37185)
Related to #37183

Utilize proxy metacache for `HasCollection` request, if collection
exists in metacache, it could be deducted that collection must exist in
system.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-10-28 18:54:23 +08:00
Buqian Zheng
f7b811450d
feat: add enable_tokenizer params to VarChar field (#36480)
issue: #35922

add an enable_tokenizer param to varchar field: must be set to true so
that a varchar field can enable_match or used as input of BM25 function

---------

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-10-10 20:33:21 +08:00
smellthemoon
2d05b7f219
fix: to add check in partition key field when set nullable==true (#36218)
#36213

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-09-14 10:53:08 +08:00
aoiasd
da227ff9a1
feat: Support create collection with functions (#35973)
relate: https://github.com/milvus-io/milvus/issues/35853
Support create collection with functions. Prepare for support bm25
function.

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-09-12 10:43:06 +08:00
congqixia
985d84d3ee
enhance: Add skip load validation for create collection task (#35737)
Related to #35415

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-29 10:05:08 +08:00
zhagnlu
3107701fe8
enhance: optimize retrieve on dynamic field (#35580)
#35514

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
Co-authored-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-08-22 14:24:56 +08:00
SimFG
731d45abbe
enhance: provide more general configuration to control mmap behavior (#35359)
- issue: #35273

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-08-21 00:22:54 +08:00
congqixia
2fbc628994
feat: Support field partial load collection (#35416)
Related to #35415

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-20 16:49:02 +08:00
congqixia
fe20366b5c
enhance: Remove duplicated schema helper creation in proxy (#35489)
Related to PRs of #35415

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-15 19:18:53 +08:00
congqixia
de8a266d8a
enhance: Enable linux code checker (#35084)
See also #34483

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-30 15:53:51 +08:00
wei liu
c45f38aa61
enhance: Update protobuf-go to protobuf-go v2 (#34394)
issue: #34252

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-29 11:31:51 +08:00
Jiquan Long
a2ac84bd64
feat: record the duration waiting in the proxy queue (#34744)
fix: https://github.com/milvus-io/milvus/issues/34743

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-07-23 14:23:52 +08:00
Patrick Weizhi Xu
104d0966b7
feat: support partition key isolation (#34336)
issue: #34332

---------

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
2024-07-11 19:01:35 +08:00
wayblink
f9a0f7bb25
Add an option to enable/disable vector field clustering key (#34097)
#30633

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-06-25 18:52:04 +08:00
wayblink
a1232fafda
feat: Major compaction (#33620)
#30633

Signed-off-by: wayblink <anyang.wang@zilliz.com>
Co-authored-by: MrPresent-Han <chun.han@zilliz.com>
2024-06-10 21:34:08 +08:00
yihao.dai
9ff023ee35
fix: Fix filtering by partition key fails for importing data (#33274)
Before executing the import, partition IDs should be reordered according
to partition names. Otherwise, the data might be hashed to the wrong
partition during import. This PR corrects this error.

issue: https://github.com/milvus-io/milvus/issues/33237

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-05-23 11:13:40 +08:00
congqixia
f336b2d672
fix: Check schema without vector field in proxy (#33211)
Related to #33199

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-05-21 14:33:39 +08:00
Cai Yudong
4fc7915c70
enhance: unify data generation test APIs (#32955)
Issue: #22837

Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2024-05-14 14:33:33 +08:00
wei liu
c35797c399
enhance: expose DescribeDatabase api in proxy (#32732)
issue: #32707

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-05-09 22:51:30 +08:00
SimFG
7da1ca9efb
enhance: check the partition num when creating collection with partition key (#32670)
issue: #30577

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-05-07 10:43:29 +08:00
wei liu
07720f1a95
enhance: expose alter database api in proxy (#32639)
issue: #30040

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-04-28 18:15:31 +08:00
SimFG
8594b55ad5
enhance: add max insert request size and must use partition key configs (#32433)
issue: https://github.com/milvus-io/milvus/issues/30577
/kind improvement

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-04-19 10:31:20 +08:00
Cai Yudong
00438f408f
enhance: Unify data type check APIs for go (#31887)
Issue: #22837

Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2024-04-07 14:27:22 +08:00
cai.zhang
1f43be4a3c
enhance: Support auto index for scalar index (#31255)
issue: #29309 
reopen pr : #29310

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-03-21 10:37:08 +08:00
Bingyi Sun
eb0cfe3e84
fix: set CollectionID field for AlterCollection request (#30814)
issue: https://github.com/milvus-io/milvus/issues/30800

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-03-07 14:49:02 +08:00
congqixia
9b3005f1be
enhance: Avoid create schema helper for each read task (#30981)
See also #30806

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-03-04 19:39:00 +08:00
Bingyi Sun
dcdf85977c
enhance: [skip e2e] add alter collection task ut (#30447)
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-02-02 15:25:04 +08:00
wayblink
1df3f90696
feat: Implement DescribeAlias and ListAliases interfaces (#29641)
#22882
/kind feature

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-01-11 19:12:51 +08:00
Xu Tong
e429965f32
Add float16 approve for multi-type part (#28427)
issue:https://github.com/milvus-io/milvus/issues/22837

Add bfloat16 vector, add the index part of float16 vector.

Signed-off-by: Writer-X <1256866856@qq.com>
2024-01-11 15:48:51 +08:00
xige-16
9702cef2b5
feat: Support multiple vector search (#29433)
issue #25639 

Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
2024-01-08 15:34:48 +08:00
wayblink
635a7f777c
feat: add clustering key in create/describe collection (#29506)
#28410
/kind feature

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-01-07 19:56:48 +08:00
congqixia
4f8c540c77
enhance: cache collection schema attributes to reduce proxy cpu (#29668)
See also #29113

The collection schema is crucial when performing search/query but some
of the information is calculated for every request.

This PR change schema field of cached collection info into a utility
`schemaInfo` type to store some stable result, say pk field,
partitionKeyEnabled, etc. And provided field name to id map for
search/query services.

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-01-04 17:28:46 +08:00
xige-16
0a70e8b601
enhance: Remove multiple vector field limit (#27827)
issue: https://github.com/milvus-io/milvus/issues/25639

/kind improvement
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
2023-12-28 16:40:46 +08:00
aoiasd
a76e3b2813
Refine delete by expression for forbid proxy dml task scheduler hang (#29340)
relate: https://github.com/milvus-io/milvus/issues/29146

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-12-26 19:52:48 +08:00
yah01
e9ff7ed13d
enhance: remove legacy error related code (#28385)
/kind improvement

Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-11-14 15:02:18 +08:00
yihao.dai
f9c630247d
Construct plan directly when search with vector output (#27928)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-10-26 19:30:10 +08:00
yah01
be980fbc38
Refine state check (#27541)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-10-11 21:01:35 +08:00
jaime
7f7c71ea7d
Decoupling client and server API in types interface (#27186)
Co-authored-by:: aoiasd <zhicheng.yue@zilliz.com>

Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-09-26 09:57:25 +08:00
SimFG
26f06dd732
Format the code (#27275)
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-09-21 09:45:27 +08:00
yah01
b4f86ea55e
Construct all success status with merr (#27226)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-09-20 10:57:23 +08:00
yah01
168e82ee10
Fix panic while handling with the nil status (#27040)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-09-15 10:09:21 +08:00
aoiasd
e107d0794c
support complex delete expression (#25752)
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-09-12 10:19:17 +08:00