wei liu
8646bba134
fix: grant ManualCompact api doesn't work ( #38096 ) ( #38168 )
...
issue: #38086
pr: #38096
cause ManualCompact api pass collection id in request, but RBAC requires
to check collection name, so grant ManualCompact api doesn't work.
This PR refine the ManualCompact api to accpet collection name in
request.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-12-05 16:00:42 +08:00
sthuang
febed0abb7
enhance: [2.4] add list aliases privilege into public role and fix typo ( #38208 )
...
cherry-pick from master: https://github.com/milvus-io/milvus/pull/38176 ,
https://github.com/milvus-io/milvus/pull/38195
related issue: https://github.com/milvus-io/milvus/issues/37031
---------
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-12-04 17:58:40 +08:00
smellthemoon
ab88d23ec0
enhance: support db request in Restful api( #38140 )( #38078 ) ( #38188 )
...
pr: #38078
pr: #38140
issue: #38077
---------
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-12-04 10:40:39 +08:00
sthuang
66f2dac5f5
fix: [2.4] fix grant/revoke v2 meta and unclear error messages ( #38146 )
...
cherry-pick from https://github.com/milvus-io/milvus/pull/38110 ,
https://github.com/milvus-io/milvus/pull/38130
related issue: https://github.com/milvus-io/milvus/issues/37031
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-12-03 22:06:41 +08:00
cai.zhang
580caebb31
fix: [2.4]Check if the dynamic fields contain any static fields ( #38027 )
...
issue: #38024
master pr: #38025
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-12-02 12:22:38 +08:00
cai.zhang
fdf3a8aa0a
fix: [2.4] Use the ID to retrieve the real name when collectionName is empty ( #37859 ) ( #37881 )
...
issue: #36989
master pr: #37859
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-11-29 11:36:37 +08:00
wei liu
fc2886215c
enhance: Optimize param cost in search ( #37738 ) ( #38019 )
...
pr: #37738
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-11-28 14:10:36 +08:00
Gao
165afbba91
enhance: support retry search when topk is reduced and result not enough ( #37093 )
...
issue: #35576
pr: #35645
---------
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-11-28 10:12:37 +08:00
wei liu
b24510164e
enhance: Decouple shard client manager from shard cache ( #37371 ) ( #37753 )
...
issue: #37115
pr: #37371 #37646 #37729
the old implementation update shard cache and shard client manager at
same time, which causes lots of conor case due to concurrent issue
without lock.
This PR decouple shard client manager from shard cache, so only shard
cache will be updated if delegator changes. and make sure shard client
manager will always return the right client, and create a new client if
not exist. in case of client leak, shard client manager will purge
client in async for every 10 minutes.
---------
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Co-authored-by: congqixia <congqi.xia@zilliz.com>
2024-11-25 17:50:34 +08:00
sthuang
d8f1af68e9
enhance: [2.4] RBAC built in privilege groups and grant v2 ( #37787 )
...
cherry-pick from master: https://github.com/milvus-io/milvus/pull/37720 ,
https://github.com/milvus-io/milvus/pull/37785
issue: https://github.com/milvus-io/milvus/issues/37031
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-11-25 11:24:54 +08:00
wei liu
c50cb8d3ef
fix: Make GetShardLeaders only retries on retriable error ( #37687 )
...
issue: #37532
pr: #37684
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-11-14 21:14:40 +08:00
congqixia
8801322371
enhance: [2.4] Invalidate collection cache when release collection ( #37577 ) ( #37628 )
...
Cherry-pick from master
pr: #37577
Related to #37395
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-11-13 14:00:31 +08:00
sthuang
9e8b6ace6d
enhance: [2.4] RBAC custom privilege group ( #37560 )
...
Cherry-pick from master
pr: https://github.com/milvus-io/milvus/pull/37087 ,
https://github.com/milvus-io/milvus/pull/37558
issue: #37031
---------
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-11-11 14:20:29 +08:00
wei liu
6b69170a64
fix: proxy retry to get shard leader on unloaded collection ( #37326 )
...
issue: #37115
pr#37116 let proxy retry to get shard leader if error happens, which
cause if search/query on a unloaded collection, which will keep retrying
until ctx done.
This PR add error type check to skip retry on ErrCollectionLoaded.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-11-05 11:02:25 +08:00
yihao.dai
380662153f
fix: [2.4] Revert "enhance: Support db for bulkinsert ( #37012 ) ( #37017 )" ( #37421 )
...
This reverts commit d6adc62765665d1555039c4d256a75d1144d49d0.
issue: https://github.com/milvus-io/milvus/issues/31273
pr: https://github.com/milvus-io/milvus/pull/37420
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-11-05 10:48:24 +08:00
wei liu
eb712f0db9
fix: dead lock if query node crash during shard client init ( #37354 )
...
issue: #37115
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-11-05 10:46:32 +08:00
wei liu
79e6ef2617
fix: Search/Query may failed during updating delegator cache ( #37174 )
...
issue: #37115
pr: #37116
casue init query node client is too heavy, so we remove
updateShardClient from leader mutex, which cause much more concurrent
cornor cases.
This PR delay query node client's init operation until `getClient` is
called, then use leader mutex to protect updating shard client progress
to avoid concurrent issues.
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-10-28 20:08:25 +08:00
cai.zhang
9c0f59488a
feat: [cherry-pick]The expression supports filling elements through templates ( #37058 )
...
issue: #36672
master pr: #37033
milvus-proto pr: https://github.com/milvus-io/milvus-proto/pull/332
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-10-28 15:17:30 +08:00
yihao.dai
ca2057c57d
enhance: Tidy import options ( #37077 ) ( #37078 )
...
1. Tidy import options.
2. Tidy common import util functions.
issue: https://github.com/milvus-io/milvus/issues/34150
pr: https://github.com/milvus-io/milvus/pull/37077
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-10-25 14:35:45 +08:00
yihao.dai
d6adc62765
enhance: Support db for bulkinsert ( #37012 ) ( #37017 )
...
issue: https://github.com/milvus-io/milvus/issues/31273
pr: https://github.com/milvus-io/milvus/pull/37012
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-10-23 16:31:29 +08:00
SimFG
55caf41169
enhance: [2.4] set the rpc error code to avoid the invalid retry ( #37025 )
...
- pr: #37024
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-10-22 11:17:28 +08:00
XuanYang-cn
6b68ec55e1
enhance: [24]Track complex delete rates ( #36958 )
...
See also: #36953
pr: #36957
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-10-22 10:07:27 +08:00
SimFG
6b9e28bc8f
enhance: [2.4] update the expr version to support automatic conversion of variable types ( #36847 )
...
/kind improvement
- pr: #36832
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-10-15 10:55:23 +08:00
Xiaofan
44564f0403
fix: make sure alias is cached ( #36808 )
...
see also #36806
Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
2024-10-13 17:39:22 +08:00
jaime
43b48a0f9b
enhance: skip alter operation when no change are detected ( #36786 )
...
issue: #36784
pr: #36785
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-10-11 21:43:21 +08:00
congqixia
385bfc2639
enhance: [2.4] Unify InboundLabel case using metrics.Label ( #36613 ) ( #36616 )
...
Cherry pick from master
pr: #36613
Previous label case broken by #36107 , this PR make all inbound label
using label constants from metrics package.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-30 16:19:17 +08:00
wei liu
f8dfd0de53
fix: Skip unnecessary query node health check in proxy ( #36491 ) ( #36553 )
...
issue: #36490
pr: #36491
After the query node changes from a delegator to a worker, proxy should
skip this querynode's health check.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-09-30 14:27:16 +08:00
aoiasd
2231aeab4d
fix:[Cherry-Pick] Split delete task msg to MaxMessageSize ( #36574 )
...
relate: https://github.com/milvus-io/milvus/issues/36089
pr: https://github.com/milvus-io/milvus/pull/36197
split delete task msg to MaxMessageSize to avoid mq message too large
error
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-09-27 18:15:19 +08:00
cai.zhang
e5a6c5b31d
fix: [cherry-pick]Check string array max length after type matching ( #36497 )
...
issue: #36029
master pr: #36449
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-09-27 11:15:14 +08:00
Zhen Ye
e34fa0461b
fix: port listen racing in mix or standalone mode ( #36459 )
...
issue: #36441
pr: #36442
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-09-26 21:01:15 +08:00
wei liu
ad5d24be65
enhance: Optimize workload based replica selection policy ( #36181 ) ( #36384 )
...
issue: #35859
pr: #36181
This PR introduce two new param: toleranceFactor and checkRequestNum,
after every checkRequestNum request has been assigned, try to compute
querynode's workload score.
if the diff is less than the toleranceFactor, replica selection policy
will fallback to round_robin, which reduce the average cost to about
500ns.
if the diff is larger than the toleranceFactor, replica selection policy
will compute querynode's score to select the target node with smallest
score in every assigment.
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-09-26 11:19:14 +08:00
jaime
b92daa1532
fix: iaccurate size estimation for encoded array data ( #36379 )
...
issue: #36029
pr: #36373
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-09-23 21:17:13 +08:00
congqixia
e66beb7ace
enhance: [2.4] Ignore index check for non-loaded vector field ( #36170 ) ( #36280 )
...
Cherry-pick from master
pr: #36170
Related to #35996
For `Field Partial Load` feature, Milvus shall ignore index check for
non-loaded vector field.
Also, this PR unifies the logic of index check for load collection and
load partitions tasks.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-19 10:07:12 +08:00
zhagnlu
d1730b8853
fix: fix rewrite output field because of merge error ( #36338 )
...
issue: #36340
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-09-18 20:01:12 +08:00
cai.zhang
eb47150f66
enhance: [cherry-pick]Disallow the keywords as a field name or dynamic field name ( #36108 )
...
issue: #35873
master pr: #36101
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-09-15 15:19:14 +08:00
zhenshan.cao
2bc454bf40
fix: keep inner topK to avoid exceeding efSearch ( #36287 )
...
issue :https://github.com/milvus-io/milvus/issues/36243
pr : https://github.com/milvus-io/milvus/pull/36284
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-09-15 15:12:38 +08:00
zhenshan.cao
34e5f99bd6
fix: Fix improper use of offset in HybridSearch ( #36253 )
...
pr : https://github.com/milvus-io/milvus/pull/36244
issue : https://github.com/milvus-io/milvus/issues/36243
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-09-13 22:05:14 +08:00
wei liu
5cd860652d
enhance: Enable dynamic update replica selection policy ( #35824 )
...
issue: #35859
pr: #35860
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-09-13 17:05:15 +08:00
wei liu
16daf9177f
fix: fix ReadWrite privilege group deny all global API ( #36145 )
...
issue: #35471
pr: #36144
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-09-13 10:31:07 +08:00
congqixia
1cd8d1bd80
enhance: [2.4] Use stats Handler to record request/response size metrics ( #36107 ) ( #36118 )
...
Cherry-pick from master
pr: #36107
Related to #36102
This PR use newly added `grpcSizeStatsHandler` to reduce calling
`proto.Size` since the request & response size info is recorded by grpc
framework.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-10 17:13:08 +08:00
jaime
256d4e209f
fix: memory leak in proxy meta cache ( #36076 )
...
issue: #36074
pr: #36075
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-09-08 17:49:06 +08:00
wei liu
ceca666e2a
fix: Fix privilege group hasn't been register for validate ( #35938 )
...
issue: #35471
pr: #35937
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-09-05 18:09:11 +08:00
yihao.dai
b578064869
fix: Fix DB limiter nodes are mistakenly cleaned up ( #35991 ) ( #35992 )
...
This issue only occurs for a short time right after a table is created.
To avoid this, we simply reduce the frequency of cleaning up invalid
limiter nodes.
issue: https://github.com/milvus-io/milvus/issues/35933
pr: https://github.com/milvus-io/milvus/pull/35991
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-09-05 17:31:05 +08:00
congqixia
abf3f68ae9
enhance: [2.4] Fix typo of clustering key not loaded msg ( #35948 ) ( #36000 )
...
Cherry-pick from master
pr: #35948
Related to #35415
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-05 16:59:07 +08:00
jaime
2c1fa50412
enhance: remove cooling off in rate limiter for read requests ( #35936 )
...
issue: #35934
pr: #35935
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-09-04 14:39:10 +08:00
congqixia
268c60c301
fix: [2.4] Check clustering key skip load behavior ( #35865 ) ( #35899 )
...
Cherry-pick from master
pr: #35865
feature issue: #35415
See also #35861
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-02 16:03:04 +08:00
SimFG
8b706122a8
enhance: [2.4] support to drop the role which is related the privilege list ( #35863 )
...
- issue: #35545
- pr: #35727
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-08-31 21:57:02 +08:00
wei liu
da026b1e28
enhance: Add depguard rules to ban deprecated proto lib ( #35140 ) ( #35818 )
...
See also #34394 #34252
pr: #35140
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Co-authored-by: congqixia <congqi.xia@zilliz.com>
2024-08-30 14:13:01 +08:00
congqixia
8d3685fadf
enhance: [2.4] Print log only when rate limit updates ( #35806 ) ( #35816 )
...
Cherry-pick from master
pr: #35806
The debug log for "RateLimiter register for rateType" is too frequent
and in e2e cases, the may print 18M times in one run.
This PR make the log be printed only when rate limit is updated.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-29 19:09:01 +08:00
congqixia
12575885d3
enhance: [2.4] Add skip load validation for create collection task ( #35761 )
...
Cherry-pick from master
pr: #35737
Related to #35415
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-28 18:25:00 +08:00
