yihao.dai
df4d5e1096
enhance: [10kcp] Read metadata concurrently to accelerate recovery ( #38404 )
...
Read metadata such as segments, binlogs, and partitions concurrently at
the collection level.
issue: https://github.com/milvus-io/milvus/issues/37630
pr: https://github.com/milvus-io/milvus/pull/38403
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-12-12 16:39:06 +08:00
wei liu
c50cb8d3ef
fix: Make GetShardLeaders only retries on retriable error ( #37687 )
...
issue: #37532
pr: #37684
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-11-14 21:14:40 +08:00
congqixia
8801322371
enhance: [2.4] Invalidate collection cache when release collection ( #37577 ) ( #37628 )
...
Cherry-pick from master
pr: #37577
Related to #37395
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-11-13 14:00:31 +08:00
sthuang
9e8b6ace6d
enhance: [2.4] RBAC custom privilege group ( #37560 )
...
Cherry-pick from master
pr: https://github.com/milvus-io/milvus/pull/37087 ,
https://github.com/milvus-io/milvus/pull/37558
issue: #37031
---------
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-11-11 14:20:29 +08:00
wei liu
6b69170a64
fix: proxy retry to get shard leader on unloaded collection ( #37326 )
...
issue: #37115
pr#37116 let proxy retry to get shard leader if error happens, which
cause if search/query on a unloaded collection, which will keep retrying
until ctx done.
This PR add error type check to skip retry on ErrCollectionLoaded.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-11-05 11:02:25 +08:00
yihao.dai
380662153f
fix: [2.4] Revert "enhance: Support db for bulkinsert ( #37012 ) ( #37017 )" ( #37421 )
...
This reverts commit d6adc62765665d1555039c4d256a75d1144d49d0.
issue: https://github.com/milvus-io/milvus/issues/31273
pr: https://github.com/milvus-io/milvus/pull/37420
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-11-05 10:48:24 +08:00
wei liu
eb712f0db9
fix: dead lock if query node crash during shard client init ( #37354 )
...
issue: #37115
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-11-05 10:46:32 +08:00
wei liu
79e6ef2617
fix: Search/Query may failed during updating delegator cache ( #37174 )
...
issue: #37115
pr: #37116
casue init query node client is too heavy, so we remove
updateShardClient from leader mutex, which cause much more concurrent
cornor cases.
This PR delay query node client's init operation until `getClient` is
called, then use leader mutex to protect updating shard client progress
to avoid concurrent issues.
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-10-28 20:08:25 +08:00
cai.zhang
9c0f59488a
feat: [cherry-pick]The expression supports filling elements through templates ( #37058 )
...
issue: #36672
master pr: #37033
milvus-proto pr: https://github.com/milvus-io/milvus-proto/pull/332
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-10-28 15:17:30 +08:00
yihao.dai
ca2057c57d
enhance: Tidy import options ( #37077 ) ( #37078 )
...
1. Tidy import options.
2. Tidy common import util functions.
issue: https://github.com/milvus-io/milvus/issues/34150
pr: https://github.com/milvus-io/milvus/pull/37077
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-10-25 14:35:45 +08:00
yihao.dai
d6adc62765
enhance: Support db for bulkinsert ( #37012 ) ( #37017 )
...
issue: https://github.com/milvus-io/milvus/issues/31273
pr: https://github.com/milvus-io/milvus/pull/37012
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-10-23 16:31:29 +08:00
SimFG
55caf41169
enhance: [2.4] set the rpc error code to avoid the invalid retry ( #37025 )
...
- pr: #37024
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-10-22 11:17:28 +08:00
XuanYang-cn
6b68ec55e1
enhance: [24]Track complex delete rates ( #36958 )
...
See also: #36953
pr: #36957
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-10-22 10:07:27 +08:00
SimFG
6b9e28bc8f
enhance: [2.4] update the expr version to support automatic conversion of variable types ( #36847 )
...
/kind improvement
- pr: #36832
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-10-15 10:55:23 +08:00
Xiaofan
44564f0403
fix: make sure alias is cached ( #36808 )
...
see also #36806
Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
2024-10-13 17:39:22 +08:00
jaime
43b48a0f9b
enhance: skip alter operation when no change are detected ( #36786 )
...
issue: #36784
pr: #36785
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-10-11 21:43:21 +08:00
congqixia
385bfc2639
enhance: [2.4] Unify InboundLabel case using metrics.Label ( #36613 ) ( #36616 )
...
Cherry pick from master
pr: #36613
Previous label case broken by #36107 , this PR make all inbound label
using label constants from metrics package.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-30 16:19:17 +08:00
wei liu
f8dfd0de53
fix: Skip unnecessary query node health check in proxy ( #36491 ) ( #36553 )
...
issue: #36490
pr: #36491
After the query node changes from a delegator to a worker, proxy should
skip this querynode's health check.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-09-30 14:27:16 +08:00
aoiasd
2231aeab4d
fix:[Cherry-Pick] Split delete task msg to MaxMessageSize ( #36574 )
...
relate: https://github.com/milvus-io/milvus/issues/36089
pr: https://github.com/milvus-io/milvus/pull/36197
split delete task msg to MaxMessageSize to avoid mq message too large
error
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-09-27 18:15:19 +08:00
cai.zhang
e5a6c5b31d
fix: [cherry-pick]Check string array max length after type matching ( #36497 )
...
issue: #36029
master pr: #36449
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-09-27 11:15:14 +08:00
Zhen Ye
e34fa0461b
fix: port listen racing in mix or standalone mode ( #36459 )
...
issue: #36441
pr: #36442
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-09-26 21:01:15 +08:00
wei liu
ad5d24be65
enhance: Optimize workload based replica selection policy ( #36181 ) ( #36384 )
...
issue: #35859
pr: #36181
This PR introduce two new param: toleranceFactor and checkRequestNum,
after every checkRequestNum request has been assigned, try to compute
querynode's workload score.
if the diff is less than the toleranceFactor, replica selection policy
will fallback to round_robin, which reduce the average cost to about
500ns.
if the diff is larger than the toleranceFactor, replica selection policy
will compute querynode's score to select the target node with smallest
score in every assigment.
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-09-26 11:19:14 +08:00
jaime
b92daa1532
fix: iaccurate size estimation for encoded array data ( #36379 )
...
issue: #36029
pr: #36373
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-09-23 21:17:13 +08:00
congqixia
e66beb7ace
enhance: [2.4] Ignore index check for non-loaded vector field ( #36170 ) ( #36280 )
...
Cherry-pick from master
pr: #36170
Related to #35996
For `Field Partial Load` feature, Milvus shall ignore index check for
non-loaded vector field.
Also, this PR unifies the logic of index check for load collection and
load partitions tasks.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-19 10:07:12 +08:00
zhagnlu
d1730b8853
fix: fix rewrite output field because of merge error ( #36338 )
...
issue: #36340
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-09-18 20:01:12 +08:00
cai.zhang
eb47150f66
enhance: [cherry-pick]Disallow the keywords as a field name or dynamic field name ( #36108 )
...
issue: #35873
master pr: #36101
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-09-15 15:19:14 +08:00
zhenshan.cao
2bc454bf40
fix: keep inner topK to avoid exceeding efSearch ( #36287 )
...
issue :https://github.com/milvus-io/milvus/issues/36243
pr : https://github.com/milvus-io/milvus/pull/36284
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-09-15 15:12:38 +08:00
zhenshan.cao
34e5f99bd6
fix: Fix improper use of offset in HybridSearch ( #36253 )
...
pr : https://github.com/milvus-io/milvus/pull/36244
issue : https://github.com/milvus-io/milvus/issues/36243
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-09-13 22:05:14 +08:00
wei liu
5cd860652d
enhance: Enable dynamic update replica selection policy ( #35824 )
...
issue: #35859
pr: #35860
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-09-13 17:05:15 +08:00
wei liu
16daf9177f
fix: fix ReadWrite privilege group deny all global API ( #36145 )
...
issue: #35471
pr: #36144
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-09-13 10:31:07 +08:00
congqixia
1cd8d1bd80
enhance: [2.4] Use stats Handler to record request/response size metrics ( #36107 ) ( #36118 )
...
Cherry-pick from master
pr: #36107
Related to #36102
This PR use newly added `grpcSizeStatsHandler` to reduce calling
`proto.Size` since the request & response size info is recorded by grpc
framework.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-10 17:13:08 +08:00
jaime
256d4e209f
fix: memory leak in proxy meta cache ( #36076 )
...
issue: #36074
pr: #36075
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-09-08 17:49:06 +08:00
wei liu
ceca666e2a
fix: Fix privilege group hasn't been register for validate ( #35938 )
...
issue: #35471
pr: #35937
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-09-05 18:09:11 +08:00
yihao.dai
b578064869
fix: Fix DB limiter nodes are mistakenly cleaned up ( #35991 ) ( #35992 )
...
This issue only occurs for a short time right after a table is created.
To avoid this, we simply reduce the frequency of cleaning up invalid
limiter nodes.
issue: https://github.com/milvus-io/milvus/issues/35933
pr: https://github.com/milvus-io/milvus/pull/35991
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-09-05 17:31:05 +08:00
congqixia
abf3f68ae9
enhance: [2.4] Fix typo of clustering key not loaded msg ( #35948 ) ( #36000 )
...
Cherry-pick from master
pr: #35948
Related to #35415
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-05 16:59:07 +08:00
jaime
2c1fa50412
enhance: remove cooling off in rate limiter for read requests ( #35936 )
...
issue: #35934
pr: #35935
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-09-04 14:39:10 +08:00
congqixia
268c60c301
fix: [2.4] Check clustering key skip load behavior ( #35865 ) ( #35899 )
...
Cherry-pick from master
pr: #35865
feature issue: #35415
See also #35861
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-02 16:03:04 +08:00
SimFG
8b706122a8
enhance: [2.4] support to drop the role which is related the privilege list ( #35863 )
...
- issue: #35545
- pr: #35727
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-08-31 21:57:02 +08:00
wei liu
da026b1e28
enhance: Add depguard rules to ban deprecated proto lib ( #35140 ) ( #35818 )
...
See also #34394 #34252
pr: #35140
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Co-authored-by: congqixia <congqi.xia@zilliz.com>
2024-08-30 14:13:01 +08:00
congqixia
8d3685fadf
enhance: [2.4] Print log only when rate limit updates ( #35806 ) ( #35816 )
...
Cherry-pick from master
pr: #35806
The debug log for "RateLimiter register for rateType" is too frequent
and in e2e cases, the may print 18M times in one run.
This PR make the log be printed only when rate limit is updated.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-29 19:09:01 +08:00
congqixia
12575885d3
enhance: [2.4] Add skip load validation for create collection task ( #35761 )
...
Cherry-pick from master
pr: #35737
Related to #35415
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-28 18:25:00 +08:00
yihao.dai
8e6ec58652
fix: Fix rate wasn't limited to the expected value ( #35699 ) ( #35700 )
...
Each time the rate is reset, the token bucket is fully refilled, causing
the rate wasn't limited to the expected value. This PR addresses the
issue by preventing the token reset.
issue: https://github.com/milvus-io/milvus/issues/35675 ,
https://github.com/milvus-io/milvus/issues/35702
pr: https://github.com/milvus-io/milvus/pull/35699
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-08-28 10:29:00 +08:00
Zhen Ye
a4533f1b8a
enhance: optimize milvus core building ( #35660 )
...
issue: #35549,#35611,#35633
pr: #35610
- remove milvus_segcore milvus_indexbuilder..., add libmilvus_core
- core building only link once
- move opendal compilation into cmake
- fix odr
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-08-27 18:55:00 +08:00
SimFG
fc324b4254
feat: [2.4] add the rbac msg and send them to the replicate channel ( #35562 )
...
- issue: #35391
- pr: #35392
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-08-27 14:45:00 +08:00
congqixia
ab261d0f8b
feat: [2.4] Support field partial load collection ( #35416 ) ( #35696 )
...
Cherry-pick from master
pr: #35416
Related to #35415
---------
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-27 14:07:00 +08:00
SimFG
5b5119a51f
feat: [2.4] provide more general configuration to control mmap behavior ( #35609 )
...
- issue: #35273
- pr: #35359
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-08-23 12:35:02 +08:00
congqixia
9c67965d6c
fix: [2.4] Re-read value after once initialization ( #35643 )
...
Cherry-pick from master
pr: #35642
Related to #35641
See also #35271
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-22 16:30:57 +08:00
zhagnlu
946881e123
enhance: optimize retrieve on dynamic field ( #35602 )
...
pr: #35580
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
Co-authored-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-08-22 14:18:57 +08:00
wei liu
e2542a1bf5
enhance: Update protobuf-go to protobuf-go v2 ( #34394 ) ( #35555 )
...
issue: #34252
pr: #34394 #35072 #35084
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-21 18:50:58 +08:00
wei liu
4bf4cbad85
enhance: Mark query node as read only after suspend ( #35492 ) ( #35586 )
...
issue: #34985 #35493
pr: #35492
after querynode has been suspended, it's not allow to load
segment/channel on it, which means the node is read only. to be
compatible with resource group design, after query node has been
suspend, we remove it from it's original resource group, make it a read
only query node in replica. then two things will happens:
1. it's original resource group will be lacking of query nodes, query
coord will assign new node to it.
2. querycoord will try to move out all segments/channels after querynode
has been suspended
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-20 19:00:56 +08:00
