438 Commits

Author SHA1 Message Date
jaime
4746f47282
feat: management WebUI homepage (#36822)
issue: #36784
1. Implement an embedded web server for WebUI access.  
2. Complete the homepage development.

Home page demo:
<img width="2177" alt="iShot_2024-10-10_17 57 34"
src="https://github.com/user-attachments/assets/38539917-ce09-4e54-a5b5-7f4f7eaac353">

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-10-23 11:29:28 +08:00
Chun Han
e2f2fd55a5
enhance: avoid limiting ddl operations repeatedly(#37006) (#37010)
related: #37006

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-10-22 20:11:27 +08:00
smellthemoon
8669153ff2
fix: not add isClusteringKey when describe collection (#36930)
#36928

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-10-18 12:57:26 +08:00
Chun Han
eccc326e8b
enhance: report err when group_size is wrong(#36146) (#36908)
related: #36146

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-10-17 16:05:29 +08:00
smellthemoon
44d80c1355
fix: not return err if consistencyLevel is not set to a valid value (#36714)
https://github.com/milvus-io/milvus/issues/36444

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-10-16 13:23:22 +08:00
smellthemoon
c9752bd2e6
enhance: refactor createCollection in RESTful API (#36790)
1.  support isClusteringKey in restful api;
2. throw err if passed invalid 'enableDynamicField' params
3. parameters in indexparams are not processed properly, related with
#36365

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-10-15 10:29:22 +08:00
Buqian Zheng
16b533cbf0
feat: Restful support for BM25 function (#36713)
issue: https://github.com/milvus-io/milvus/issues/35853

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-10-13 17:41:21 +08:00
smellthemoon
5e1c392243
enhance: support null and default value in restful api (#35825)
#31728

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-10-11 21:23:20 +08:00
yihao.dai
0751c508de
enhance: Support new RESTful URL to describe import job (#36753)
Support the new RESTful URL for retrieving/describing import progress:
`/v2/vectordb/jobs/import/describe`.
Deprecate the old URL: `/v2/vectordb/jobs/import/get_progress`.

issue: https://github.com/milvus-io/milvus/issues/36752

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-10-11 16:07:21 +08:00
Chun Han
a25dc98794
enhance: support group_size and hybridsearch+groupby on httpv2 side(#36386) (#36461)
related: #36386

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-09-30 11:09:16 +08:00
smellthemoon
b1ac3f0df0
enhance: do not log out the full req (#36546)
![img_v3_02f3_61ac166c-df6f-4b5e-b06a-756e5f40a65g](https://github.com/user-attachments/assets/df668339-d0d8-4295-a2b6-52d89ad98766)

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-09-27 16:39:15 +08:00
Zhen Ye
d29e01e284
fix: port listen racing in mix or standalone mode (#36442)
issue: #36441

---------

Signed-off-by: chyezh <chyezh@outlook.com>
2024-09-26 21:23:16 +08:00
smellthemoon
6e880d19a8
enhance: add search params in search request in restful (#36304)
https://github.com/milvus-io/milvus/issues/36321

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-09-23 16:51:12 +08:00
congqixia
aaa8487590
enhance: Remove duplicated log in proxy service (#36110)
Remove duplicate identical log in proxy/service.go

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-09 17:19:05 +08:00
congqixia
51870aef08
enhance: Use stats Handler to record request/response size metrics (#36107)
Related to #36102

This PR use newly added `grpcSizeStatsHandler` to reduce calling
`proto.Size` since the request & response size info is recorded by grpc
framework.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-09 17:13:05 +08:00
congqixia
9e96ed4873
fix: Fix tracing config update logic (#35928)
Related to #35927

There are serveral issue this PR addresses:
- Use `ResetTraceConfig` method instead init one in update event handler
- Implement dynamic stats.Handler to receive tracing config update event
- Update `enable_trace` flag when `ResetTraceConfig` is invoked
- Change `enable_trace` to `std::atomic<bool>` in case of data race

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-05 14:27:04 +08:00
Yinzuo Jiang
407fc933e7
fix: bump bytedance/sonic to v1.12.2 to fix compilation error with go 1.23.0 (#35879)
fixes: #35878

Signed-off-by: Yinzuo Jiang <jiangyinzuo@foxmail.com>
2024-09-01 13:43:01 +08:00
congqixia
c5c9ef9876
fix: Check response size before add to counter (#35778)
Related to #35767

prometheus counter cannot add negative value
when response is not written(say timeout/network broken) panicking may
happen if not check

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-29 00:05:00 +08:00
Chun Han
031ee6f155
enhance: support httpv1/v2 throttle and add it for httpV2(#35350) (#35470)
related: #35350

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-08-20 16:16:55 +08:00
wei liu
1d49358f82
enhance: Add BackupRBAC/RestoreRBAC API to enable rbac backup (#35444)
issue: #35443

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-16 10:10:53 +08:00
shaoyue
1db099973f
enhance: Gin & restfulv1 handler use sonic json (#35020)
/cc @czs007 @congqixia 
/kind enhancement

Signed-off-by: haorenfsa <haorenfsa@gmail.com>
2024-08-05 16:44:17 +08:00
congqixia
de8a266d8a
enhance: Enable linux code checker (#35084)
See also #34483

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-30 15:53:51 +08:00
wei liu
c45f38aa61
enhance: Update protobuf-go to protobuf-go v2 (#34394)
issue: #34252

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-29 11:31:51 +08:00
PowderLi
5584ae2e14
enhance: add metrics for restful apis (#34969)
issue: #34968

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-07-29 00:13:47 +08:00
congqixia
783f9d9c33
fix: Unify hook singleton implementation in proxy (#34887)
Related to #34885

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-26 18:07:53 +08:00
Patrick Weizhi Xu
80c0ae3519
feat: [restful v2] add partition key isolation prop (#34710)
issue: https://github.com/milvus-io/milvus/issues/34332

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
2024-07-22 11:29:59 +08:00
SimFG
203fb554a4
enhance: support to config root user's password (#34752)
- issue: #33058

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-07-17 20:19:42 +08:00
aoiasd
5bb0d21e32
fix: Restful API use deprecate error code cause access log panic. (#34576)
relate: https://github.com/milvus-io/milvus/issues/34578

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-07-12 10:13:35 +08:00
jaime
60be454db0
enhance: add disk quota and max collections into db properties (#34368)
issue: #34385

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-07-05 18:22:17 +08:00
PowderLi
c16a704934
fix: [restful v2] count(*) & hook (#34369)
issue:  #31224 #34374

for query api:
1. param filter is not requried
2. param limit is useless while outputFields = [count(*)]

add hook about grpc call

---------

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-07-04 19:38:09 +08:00
wei liu
0a3d456688
enhance: add restful api to trigger component stop (#32076)
issue: #32698
This PR add two rest api for component stop and status check:
1. `/management/stop?role=querynode` can stop the specified component
2. `/management/check/ready?role=rootcoord` can check whether the target
component is serviceable

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-06-07 10:35:54 +08:00
aoiasd
0cf225fa75
enhance: access log support restful api (#33155)
relate: https://github.com/milvus-io/milvus/issues/31823

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-05-31 11:13:44 +08:00
PowderLi
ee73e62202
fix: [restful v2] search result be cut while nq > 1 (#33339)
issue: #33099 #32837 #32419

1. len(search result) may be nq * topk, we need return all rather than
topk
2. the  in restful response payload keep the same with milvus error code

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-05-27 14:25:41 +08:00
Buqian Zheng
1b67cecd65
enhance: add sparse float vector support to restful v2 (#33231)
issue: #29419
also re-enabled an e2e test using restful api, which is previously
disabled due to https://github.com/milvus-io/milvus/issues/32214.

In restful api, the accepted json formats of sparse float vector are:

* `{"indices": [1, 100, 1000], "values": [0.1, 0.2, 0.3]}`
* {"1": 0.1, "100": 0.2, "1000": 0.3}

for accepted indice and value range, see
https://milvus.io/docs/sparse_vector.md#FAQ

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-05-27 00:47:40 +08:00
PowderLi
b9d7145049
fix: [restful v2]role operations need dbName (#33283)
issue: #33220

use dbName as part of privilege entity, so
1. grant / revoke a privilege need dbName
2. we can describe the privileges of the role which belong to one
special database

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-05-23 09:51:45 +08:00
SimFG
b9b6343c88
enhance: check the auth in some rest v2 api (#33256)
/kind improvement
link master proto:
https://github.com/milvus-io/milvus-proto/blob/master/proto/milvus.proto

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-05-22 16:03:40 +08:00
aoiasd
dc058eaf61
fix: should init access logger before init server (#32976)
relate: https://github.com/milvus-io/milvus/issues/32968

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-05-14 10:47:32 +08:00
congqixia
c0e62e6629
enhance: Use collection default consistency level for restv2 (#32956)
Set `UseDefaultConsistency` to true so that restv2 read API shall use
collection consistency level setting correctly.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-05-11 00:47:31 +08:00
wei liu
c35797c399
enhance: expose DescribeDatabase api in proxy (#32732)
issue: #32707

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-05-09 22:51:30 +08:00
SimFG
8963615b33
enhance: add the cost response for the rest api (#32620)
issue: https://github.com/milvus-io/milvus/issues/30436

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-05-08 11:53:29 +08:00
wei liu
ba02d54a30
enhance: update shard leader cache when leader location changed (#32470)
issue: #32466

this PR enhance that when shard location changed, update proxy's shard
leader cache. in case of query node failover case, proxy can find
replica recover

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-05-08 10:05:29 +08:00
wei liu
07720f1a95
enhance: expose alter database api in proxy (#32639)
issue: #30040

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-04-28 18:15:31 +08:00
SimFG
bed6363feb
enhance: update the go-api version for the list api (#32605)
issue: https://github.com/milvus-io/milvus/issues/32550
/kind improvement

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-04-26 17:49:33 +08:00
PowderLi
29922829da
fix: [restful v2] parameters of create collection (#32485)
issue: #31176

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-04-24 09:43:29 +08:00
PowderLi
a0a23d0666
fix: [restful]use default search parameter nq: 0 (#32355)
issue: #32225 #31978 #32360

1. v1 can only accept one vector, but v2 accept list of vectors #32225
2. cannot get dbName from AliasReq #31978
3. enhance: [restful v2]support config DB-Name in the http header #32360

---------

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-04-18 14:41:20 +08:00
chyezh
48fe977a9d
enhance: declarative resource group api (#31930)
issue: #30647

- Add declarative resource group api

- Add config for resource group management

- Resource group recovery enhancement

---------

Signed-off-by: chyezh <chyezh@outlook.com>
2024-04-15 08:13:19 +08:00
PowderLi
610a65af14
fix: restful v2 (#32144)
issue: #31176

1. cannot get dbName correctly while describe alias #31978
2. return a valid json string even if the user doesn't have the whole
privileges to describe collection #31635
3. rename IndexParam.IndexConfig to IndexParam.Params
4. FieldSchema.ElementTypeParams, IndexParam.Params can not only accept
string

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-04-13 21:55:29 +08:00
Cai Yudong
00438f408f
enhance: Unify data type check APIs for go (#31887)
Issue: #22837

Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2024-04-07 14:27:22 +08:00
SimFG
b1a1cca10b
feat: add more operation detail info for better allocation (#30438)
issue: #30436

---------

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-03-28 06:33:11 +08:00
yihao.dai
9a13b9822f
enhance: Return more fields in import progress response (#31539)
Return more fields in import progress response, include importedRows and
totalRows. Additionally, ensure compatibility with the old import
progress response by retaining fields of create timestamp and row count.

issue: https://github.com/milvus-io/milvus/issues/31448
https://github.com/milvus-io/milvus/issues/31237
https://github.com/milvus-io/milvus/issues/28521

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-03-24 21:57:06 +08:00