407 Commits

Author SHA1 Message Date
tinswzy
27229f7907
enhance: refine exists log print with ctx (#38080)
issue: #35917 
Refines exists log print with ctx

Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
2024-12-14 22:36:44 +08:00
Gao
8977454311
enhance: support recall estimation (#38017)
issue: #37899 
Only `search` api will be supported

---------

Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-12-11 20:40:48 +08:00
Xianhui Lin
db05d4f976
enhance: alterindex & altercollection supports altering properties (#37437)
enhance : 

1. alterindex delete properties
We have introduced a new parameter deleteKeys to the alterindex
functionality, which allows for the deletion of properties within an
index. This enhancement provides users with the flexibility to manage
index properties more effectively by removing specific keys as needed.
2. altercollection delete properties
We have introduced a new parameter deleteKeys to the altercollection
functionality, which allows for the deletion of properties within an
collection. This enhancement provides users with the flexibility to
manage collection properties more effectively by removing specific keys
as needed.

3.support altercollectionfield
We currently support modifying the fieldparams of a field in a
collection using altercollectionfield, which only allows changes to the
max-length attribute.
Key Points:
- New Parameter - deleteKeys: This new parameter enables the deletion of
specified properties from an index. By passing a list of keys to
deleteKeys, users can remove the corresponding properties from the
index.

- Mutual Exclusivity: The deleteKeys parameter cannot be used in
conjunction with the extraParams parameter. Users must choose one
parameter to pass based on their requirement. If deleteKeys is provided,
it indicates an intent to delete properties; if extraParams is provided,
it signifies the addition or update of properties.

issue: https://github.com/milvus-io/milvus/issues/37436

---------

Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
2024-12-11 10:20:42 +08:00
congqixia
fec31fedce
enhance: Move SyncCreatedPartition step to proxy (#38296)
Related to #38275

This PR move sync created partition step to proxy to avoid potential
logic deadlock when create partition happens with target segment change.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-12-10 12:30:43 +08:00
tinswzy
5768dbbb5d
enhance: refine pular related mq interfaces (#38007)
issue: #35917 
Refines the pulsar-related mq APIs to allow the ctx to be passed down

Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
2024-12-04 20:50:39 +08:00
cai.zhang
73aa95f596
fix: Add version to the proxy cache to resolve concurrency issues (#38067)
issue: #36989

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-12-04 18:06:39 +08:00
sthuang
a65d395ecd
fix: grant v2 collection name and privilege error typo (#38195)
issue: https://github.com/milvus-io/milvus/issues/38042

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-12-04 16:14:38 +08:00
wei liu
e09f431891
fix: grant ManualCompact api doesn't work (#38096)
issue: #38086
cause ManualCompact api pass collection id in request, but RBAC requires
to check collection name, so grant ManualCompact api doesn't work.

This PR refine the ManualCompact api to accpet collection name in
request.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-12-03 10:36:38 +08:00
sthuang
a5e0a56a8e
fix: move grant/revoke v2 params check from rootcoord to proxy (#38130)
related issue: https://github.com/milvus-io/milvus/issues/37031

fixed issues #38042: The interface "grant_v2" does not support empty
collectionName while the error says it does

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-12-02 19:48:37 +08:00
sthuang
23dc313c44
fix: fix grant/revoke v2 meta and unclear error messages (#38110)
related issue: https://github.com/milvus-io/milvus/issues/37031

fixed issues:
#37974: better error messages for grant v2 interface
#37903: fix meta built-in privilege group object name
#37843: better error messages for custom privilege group interface 
#38002: fix built-in privilege group meta to pass proxy interceptor
check
#38008: fix revoke v2 to support revoking v1 granted privileges

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-12-02 11:36:39 +08:00
jaime
7bbfe86bcd
enhance: add list index and segment index retrieval API for WebUI (#37861)
issue: #36621

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-11-22 16:58:34 +08:00
sthuang
19572f5b06
enhance: RBAC new grant/revoke privilege (#37785)
issue: https://github.com/milvus-io/milvus/issues/37031
also fix issues: https://github.com/milvus-io/milvus/issues/37843,
https://github.com/milvus-io/milvus/issues/37842,
https://github.com/milvus-io/milvus/issues/37887

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-11-21 22:20:34 +08:00
Xiaofan
33bfb25c73
enhance: refine meta cache log and logic (#37318)
related to #36989
add more logs in proxy meta cache and make it clearer

Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
2024-11-19 10:50:33 +08:00
jaime
12ed40e125
fix: some missing metrics for WebUI restful API (#37747)
issue: #36621

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-11-18 14:00:32 +08:00
jaime
1d06d4324b
fix: Int64 overflow in JSON encoding (#37657)
issue: ##36621

- For simple types in a struct, add "string" to the JSON tag for
automatic string conversion during JSON encoding.
- For complex types in a struct, replace "int64" with "string."

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-11-14 22:52:30 +08:00
jaime
1e8ea4a7e7
feat: add segment/channel/task/slow query render (#37561)
issue: #36621

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-11-12 17:44:29 +08:00
congqixia
f5b06a3c9f
enhance: Invalidate collection cache when release collection (#37577)
Related to #37395

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-11-12 10:16:29 +08:00
sthuang
70605cf5b3
enhance: Support custom privilege group for RBAC (#37087)
issue: #37031

---------

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-11-09 08:44:28 +08:00
yihao.dai
994f52fab8
fix: Revert "enhance: Support db for bulkinsert (#37012)" (#37420)
This reverts commit 6e90f9e8d90440716d596a7fe8fe1db465d529b7.

issue: https://github.com/milvus-io/milvus/issues/31273

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-11-07 17:02:25 +08:00
jaime
f348bd9441
feat: add segment,pipeline, replica and resourcegroup api for WebUI (#37344)
issue: #36621

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-11-07 11:52:25 +08:00
jaime
9d16b972ea
feat: add tasks page into management WebUI (#37002)
issue: #36621

1. Add API to access task runtime metrics, including:
  - build index task
  - compaction task
  - import task
- balance (including load/release of segments/channels and some leader
tasks on querycoord)
  - sync task
2. Add a debug model to the webpage by using debug=true or debug=false
in the URL query parameters to enable or disable debug mode.

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-10-28 10:13:29 +08:00
yihao.dai
b45cf2d49f
enhance: Add max length check for csv import (#37077)
1. Add max length check for csv import.
2. Tidy import options.
3. Tidy common import util functions.

issue: https://github.com/milvus-io/milvus/issues/34150

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-10-25 14:37:29 +08:00
yihao.dai
6e90f9e8d9
enhance: Support db for bulkinsert (#37012)
issue: https://github.com/milvus-io/milvus/issues/31273

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-10-25 14:31:39 +08:00
Gao
1d61b604e1
enhance: support retry search when topk is reduced and result not enough (#35645)
issue: #35576 

This pr is to cover those cases when queryHook optimize search params
and make the result size insufficient, add retry search mechanism and
add related metrics for alarming.

---------

Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-10-23 19:19:30 +08:00
jaime
4746f47282
feat: management WebUI homepage (#36822)
issue: #36784
1. Implement an embedded web server for WebUI access.  
2. Complete the homepage development.

Home page demo:
<img width="2177" alt="iShot_2024-10-10_17 57 34"
src="https://github.com/user-attachments/assets/38539917-ce09-4e54-a5b5-7f4f7eaac353">

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-10-23 11:29:28 +08:00
XuanYang-cn
2e7670908f
enhance: Track complex delete rates (#36957)
See also: #36953

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-10-18 17:47:30 +08:00
congqixia
e4292569c7
enhance: Unify InboundLabel case using metrics.Label (#36613)
Previous label case broken by #36107, this PR make all inbound label
using label constants from metrics package.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-30 14:15:17 +08:00
cai.zhang
ecb2b242e2
enhance: Add sorted for segment info (#36469)
issue: #33744

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-09-30 10:01:16 +08:00
Ted Xu
e7ea1d7a04
enhance: improve log encoding performance on proxy nodes (#36123)
See #36122

This PR is designed to enhance log performance through two improvements:

1. Optimize JSON encoding by switching JSON serializer to
`json-iterator`.
2. Adding support of lazy initialization `WithLazy`.

---------

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-09-11 14:51:07 +08:00
congqixia
51870aef08
enhance: Use stats Handler to record request/response size metrics (#36107)
Related to #36102

This PR use newly added `grpcSizeStatsHandler` to reduce calling
`proto.Size` since the request & response size info is recorded by grpc
framework.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-09 17:13:05 +08:00
jaime
24fb10114b
enhance: remove cooling off in rate limiter for read requests (#35935)
issue: #35934

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-09-04 14:39:10 +08:00
Zhen Ye
99dff06391
enhance: using streaming service in insert/upsert/flush/delete/querynode (#35406)
issue: #33285

- using streaming service in insert/upsert/flush/delete/querynode
- fixup flusher bugs and refactor the flush operation
- enable streaming service for dml and ddl
- pass the e2e when enabling streaming service
- pass the integration tst when enabling streaming service

---------

Signed-off-by: chyezh <chyezh@outlook.com>
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-08-29 10:03:08 +08:00
SimFG
3e1052f889
enhance: use the msg position obj when getting replicate channel position (#35606)
/kind improvement

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-08-27 10:28:59 +08:00
congqixia
2fbc628994
feat: Support field partial load collection (#35416)
Related to #35415

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-20 16:49:02 +08:00
wei liu
14ef88f423
fix: Fix proxy panic after Import related api failed (#35540)
issue: #35539

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-19 15:22:55 +08:00
wei liu
1d49358f82
enhance: Add BackupRBAC/RestoreRBAC API to enable rbac backup (#35444)
issue: #35443

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-16 10:10:53 +08:00
SimFG
b2cc4b0776
feat: add the rbac msg and send them to the replicate channel (#35392)
- issue: #35391

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-08-15 12:06:52 +08:00
congqixia
de8a266d8a
enhance: Enable linux code checker (#35084)
See also #34483

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-30 15:53:51 +08:00
wei liu
c45f38aa61
enhance: Update protobuf-go to protobuf-go v2 (#34394)
issue: #34252

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-29 11:31:51 +08:00
congqixia
783f9d9c33
fix: Unify hook singleton implementation in proxy (#34887)
Related to #34885

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-26 18:07:53 +08:00
congqixia
4ee6c69217
enhance: Add Segment Level in milvus segment info APIs (#34763)
See also #34746

This PR add segment level field in response of
`GetPersistentSegmentInfo` and `GetQuerySegmentInfo`

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-26 10:01:46 +08:00
SimFG
88c38a3756
fix: execute method should return an error when result is a failure (#34813)
- issue: #34812

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-07-20 23:13:40 +08:00
yihao.dai
c8bf6c8a46
enhance: Optimize ConsistencyLevel and GuaranteeTimestamp log (#34738)
/kind improvement

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-18 15:17:41 +08:00
congqixia
b5225df21c
enhance: Use unified requery flag when set metrics (#34632)
See also #34601

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-15 23:03:38 +08:00
Patrick Weizhi Xu
104d0966b7
feat: support partition key isolation (#34336)
issue: #34332

---------

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
2024-07-11 19:01:35 +08:00
aoiasd
186757e622
enhance: support mark error as user error (#33498)
relate: https://github.com/milvus-io/milvus/issues/33492

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-07-01 14:56:12 +08:00
Chun Han
96dcee5dff
fix:load major compaction partial result(#34051) (#34052)
related: #34051

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-06-28 14:04:05 +08:00
yihao.dai
3540eee977
enhance: Support L0 import (#33514)
issue: https://github.com/milvus-io/milvus/issues/33157

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-06-07 14:17:20 +08:00
congqixia
25080bb455
enhance: Print UseDefaultConsistency param in read requests (#33617)
`UseDefaultConsistency` param is crucial for debugging slow query
problems. It could be confusing when guarantee timestamp is 1 while this
param is not logged

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-05 10:27:50 +08:00
SimFG
2964f60edc
enhance: the proxy metric in the query request (#33307)
/kind improvement
issue: #33306

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-05-24 14:19:41 +08:00