41 Commits

Author SHA1 Message Date
sthuang
d8f1af68e9
enhance: [2.4] RBAC built in privilege groups and grant v2 (#37787)
cherry-pick from master: https://github.com/milvus-io/milvus/pull/37720,
https://github.com/milvus-io/milvus/pull/37785
issue: https://github.com/milvus-io/milvus/issues/37031

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-11-25 11:24:54 +08:00
congqixia
cffde80e68
enhance: [2.4] Prevent generate "null" search params (#37811)
pr: #37812
Preventing generating null search params in restful search request

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-11-19 18:20:32 +08:00
smellthemoon
46692d7525
enhance: support upsert autoid==true in Restful API and fix some bugs(#37072)(#37487) (#37766)
pr: #37072
pr: #37487

---------

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-11-18 19:44:31 +08:00
smellthemoon
b3e6482367
enhance: add search params in search request in restful(#36304) (#37673)
pr: #36304 
pr: #36714 
pr: #36448

---------

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-11-15 17:54:30 +08:00
sthuang
9e8b6ace6d
enhance: [2.4] RBAC custom privilege group (#37560)
Cherry-pick from master
pr: https://github.com/milvus-io/milvus/pull/37087,
https://github.com/milvus-io/milvus/pull/37558
issue: #37031

---------

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-11-11 14:20:29 +08:00
smellthemoon
60f963102e
enhance: refactor createIndex in RESTful API(#37235) (#37237)
pr: #37235 
2.5: #37236

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-11-07 14:18:31 +08:00
congqixia
c195f9f76a
enhance: [2.4] Pass rpc stats via gin.Context (#37440)
Cherry pick from master
pr: #37439
Related #37223

RPC stats worked in middleware but faild to get method & collection info

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-11-05 14:24:24 +08:00
yihao.dai
380662153f
fix: [2.4] Revert "enhance: Support db for bulkinsert (#37012) (#37017)" (#37421)
This reverts commit d6adc62765665d1555039c4d256a75d1144d49d0.

issue: https://github.com/milvus-io/milvus/issues/31273

pr: https://github.com/milvus-io/milvus/pull/37420

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-11-05 10:48:24 +08:00
congqixia
49147524be
enhance: [2.4] Use middleware to observe restful v2 in/out rpc stats (#37224)
Cherry pick from master
pr: #37223
Related to #36102

Previous PR #36107 add grpc inteceptor to observe rpc stats. Using same
strategy, this pr add gin middleware to observer restful v2 rpc stats.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-10-29 10:26:24 +08:00
yihao.dai
d6adc62765
enhance: Support db for bulkinsert (#37012) (#37017)
issue: https://github.com/milvus-io/milvus/issues/31273

pr: https://github.com/milvus-io/milvus/pull/37012

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-10-23 16:31:29 +08:00
smellthemoon
d2a1db922f
enhance: refactor createCollection in RESTful API(#36790) (#36885)
pr: #36790 
and pr:#36930

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-10-22 14:21:27 +08:00
Chun Han
a2d4520dc7
enhance: avoid limiting ddl operations repeatedly(#37006) (#37011)
related: #37006

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-10-22 10:41:28 +08:00
yihao.dai
7ee46c582e
enhance: Support new RESTful URL to describe import job (#36753) (#36754)
Support the new RESTful URL for retrieving/describing import progress:
`/v2/vectordb/jobs/import/describe`.
Deprecate the old URL: `/v2/vectordb/jobs/import/get_progress`.

issue: https://github.com/milvus-io/milvus/issues/36752

pr: https://github.com/milvus-io/milvus/pull/36753

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-10-11 16:03:22 +08:00
wei liu
e2542a1bf5
enhance: Update protobuf-go to protobuf-go v2 (#34394) (#35555)
issue: #34252
pr: #34394 #35072 #35084

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-21 18:50:58 +08:00
Chun Han
cf8494ef45
enhance: support httpv1/v2 throttle and add it for httpV2(#35350) (#35504)
related: #35350
pr: https://github.com/milvus-io/milvus/pull/35470

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-08-20 16:32:56 +08:00
Patrick Weizhi Xu
606bdd7d41
feat: [restful v2][2.4] add partition key isolation prop (#34869)
issue: https://github.com/milvus-io/milvus/issues/34332
pr: https://github.com/milvus-io/milvus/pull/34710

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
(cherry picked from commit 703feaf6b4a94d1747141cea073b5dcfa36eac73)
2024-07-22 11:43:41 +08:00
PowderLi
ba2c232331
fix: [cherry-pick] [restful v2] count(*) & hook (#34433)
issue: #31224 #34374
pr: #34369

for query api:

1. param filter is not requried
2. param limit is useless while outputFields = [count(*)]

add hook about grpc call

---------

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-07-05 09:52:10 +08:00
aoiasd
bc8ca593bd
enhance: [Cherry-Pick] Update access log (#34295)
Support restful api,print with consistency level and add new write
cache.
relate: https://github.com/milvus-io/milvus/issues/31823
pr: https://github.com/milvus-io/milvus/pull/33155
https://github.com/milvus-io/milvus/pull/33503
https://github.com/milvus-io/milvus/pull/32213

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-07-02 15:30:15 +08:00
Buqian Zheng
d6a994038c
enhance: [2.4] add sparse float vector support to restful v2 (#33555)
issue: https://github.com/milvus-io/milvus/issues/29419
pr: #33231

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-06-03 19:31:46 +08:00
PowderLi
aaecc92244
fix: [cherry-pick] [restful v2] search result be cut while nq > 1 (#33363)
issue: #33099 #32837 #32419
master pr: #33339

1. len(search result) may be nq * topk, we need return all rather than
topk
2. the  in restful response payload keep the same with milvus error code

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-05-27 00:57:45 +08:00
SimFG
37b2f90c90
enhance: [2.4] the panic when db isn't existed in the rate limit interceptor (#33308)
issue: #33243
pr: #33244

1. fix: the panic when db isn't existed in the rate limit interceptor
#33244
2. enhance: check the auth in some rest v2 api #33256

---------

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-05-23 17:25:41 +08:00
PowderLi
f10ffb9230
fix: [cherry-pick][restful v2]role operations need dbName (#33291)
issue: #33220
master pr: #33283

use dbName as part of privilege entity, so
1. grant / revoke a privilege need dbName
2. we can describe the privileges of the role which belong to one
special database

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-05-23 09:51:45 +08:00
congqixia
c0e62e6629
enhance: Use collection default consistency level for restv2 (#32956)
Set `UseDefaultConsistency` to true so that restv2 read API shall use
collection consistency level setting correctly.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-05-11 00:47:31 +08:00
SimFG
8963615b33
enhance: add the cost response for the rest api (#32620)
issue: https://github.com/milvus-io/milvus/issues/30436

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-05-08 11:53:29 +08:00
PowderLi
29922829da
fix: [restful v2] parameters of create collection (#32485)
issue: #31176

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-04-24 09:43:29 +08:00
PowderLi
a0a23d0666
fix: [restful]use default search parameter nq: 0 (#32355)
issue: #32225 #31978 #32360

1. v1 can only accept one vector, but v2 accept list of vectors #32225
2. cannot get dbName from AliasReq #31978
3. enhance: [restful v2]support config DB-Name in the http header #32360

---------

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-04-18 14:41:20 +08:00
PowderLi
610a65af14
fix: restful v2 (#32144)
issue: #31176

1. cannot get dbName correctly while describe alias #31978
2. return a valid json string even if the user doesn't have the whole
privileges to describe collection #31635
3. rename IndexParam.IndexConfig to IndexParam.Params
4. FieldSchema.ElementTypeParams, IndexParam.Params can not only accept
string

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-04-13 21:55:29 +08:00
Cai Yudong
00438f408f
enhance: Unify data type check APIs for go (#31887)
Issue: #22837

Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2024-04-07 14:27:22 +08:00
yihao.dai
9a13b9822f
enhance: Return more fields in import progress response (#31539)
Return more fields in import progress response, include importedRows and
totalRows. Additionally, ensure compatibility with the old import
progress response by retaining fields of create timestamp and row count.

issue: https://github.com/milvus-io/milvus/issues/31448
https://github.com/milvus-io/milvus/issues/31237
https://github.com/milvus-io/milvus/issues/28521

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-03-24 21:57:06 +08:00
PowderLi
ee56ae7299
fix: [CHERRY-PICK] [restful v2] add hybrid_search (#31395)
search issue: #30688

2.4 pr: #31344

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-03-21 09:59:08 +08:00
yihao.dai
2b7df47aa9
feat: Add RBAC functionality to ImportV2 (#31298)
issue: https://github.com/milvus-io/milvus/issues/31282,
https://github.com/milvus-io/milvus/issues/28521

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-03-19 20:01:07 +08:00
PowderLi
ed0526a1a4
fix: [restful v2] create a collection with vector field name (#31317)
fix: [restful v2] create a collection with vector field name

1. quickly create collection, create an index on the vector field #31149
2. valid consistencyLevel is needed, while create a collection
3. show collection's properties and aliases #31180
4. list aliase for one collection, default is the whole database

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-03-17 20:03:07 +08:00
PowderLi
afa4193fc2
fix: [restful v2] create collection & describe collection (#31269)
1. complete the output of describe collection #31180
2. add a new parameter `params` for create collection #31181 #31149
3. create index according to indexConfig while create collection #31199
4. autoID: false while create a collection quickly, take upsert into
consideration
5. polish error information while search without annsField #31010


fix: [restful v1] insert / upsert with default partitionName JIRA-4952

---------

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-03-14 23:27:03 +08:00
yihao.dai
b5c67948b7
enhance: Enhance and modify the return content of ImportV2 (#31192)
1. The Import APIs now provide detailed progress information for each
imported file, including details such as file name, file size, progress,
and more.
2. The APIs now return the collection name and the completion time.
3. Other modifications include changing jobID to jobId and other similar
adjustments.

issue: https://github.com/milvus-io/milvus/issues/28521

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-03-13 19:51:03 +08:00
PowderLi
58d7b9f902
fix: [restful v2] search & advanced_search API (#31113)
issue: #30688
former pr:  #30946

1. param `vector` is required #31012
2. param `annsField` is optional, for multiply vector fields #31010
3. support BinaryVector, Float16Vector, BFloat16Vector #31013
4. replace vector with data, to align with pymilvus milvus_client #31093
5. create collection quickly, to align with pymilvus milvus_client
#31149

---------

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-03-11 15:57:01 +08:00
yihao.dai
3b66c17279
feat: Revise the RESTful bulk insert API (#29698)
Revise the RESTful bulk insert API from version 1 to version 2.

issue: https://github.com/milvus-io/milvus/issues/28521

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-03-05 15:03:00 +08:00
PowderLi
4869aaeb94
fix: [restful v2] align search api with milvus client (#30946)
issue: #30688

fix: [restful v2] align search api with milvus client #30688
fix: [restful v2] DescribeIndex return Response{ Status: IndexNotFound
}, nil #30722

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-03-01 11:01:01 +08:00
PowderLi
19911a411f
fix: [restful v2] bug list (#30740)
bug list
1. element data type is needed while create a collection with an array
field #30638
2. spelling mistake about metricsType #30643
3. new collection enable dynamic field / auto id as user defined #30665
4. convert rowCount from string to int #30661
5. try it's best to create a new collection #30652
6. int64 percious #20415
7. insert into collection which has multi vector fields #30674
8. cannot rename a collection to other database #30700
9. update the request body of "indexes/create" #30769
10. got [] while list indexes of a collection which has no index #30722
11. restful need encode password before call
CreateCredential/UpdateCredential #30730
12. some parameters missed the required label #30737
13.define the field to be or not to be a partition key while create a
collection #30797

enhance: support opentelemetry
enhance: support dataType: Float16Vector & BFloat16Vector #22837
enhance: describe collection will show the field is partition key or not
#30789

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-02-27 16:48:56 +08:00
PowderLi
1b87bcd60e
feat: restful support new features (#30485)
feat: restful support new features

1. search with groupingField #25324
2. hybrid search #25639

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-02-07 17:16:47 +08:00
PowderLi
5cf9bb236e
enhance: restful support import jobs (#30343)
issue: #28521 #29732

include
1. list collection's import jobs
2. create a new import job
3. get the progress of an import job

fix:
1. mix the order of dbName & collectionName #29728
2. trace log keep the same as v1
3. support traceID
4. azure precheck, blob name cannot end with / #29703

---------

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-01-31 17:57:04 +08:00
PowderLi
6abbab12fa
feat: restful phase two (#29728)
issue: #29732

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-01-28 16:03:01 +08:00