admin/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2026-01-07 19:31:51 +08:00
Code Issues Packages Projects Releases Wiki Activity
20,474 Commits 137 Branches 188 Tags
Commit Graph

1526 Commits

Author SHA1 Message Date
jaime
43b48a0f9b
enhance: skip alter operation when no change are detected (#36786) 
issue: #36784 
pr: #36785

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-10-11 21:43:21 +08:00
congqixia
385bfc2639
enhance: [2.4] Unify InboundLabel case using metrics.Label (#36613) (#36616) 
Cherry pick from master
pr: #36613 
Previous label case broken by #36107, this PR make all inbound label
using label constants from metrics package.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-09-30 16:19:17 +08:00
wei liu
f8dfd0de53
fix: Skip unnecessary query node health check in proxy (#36491) (#36553) 
issue: #36490
pr: #36491
After the query node changes from a delegator to a worker, proxy should
skip this querynode's health check.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-09-30 14:27:16 +08:00
aoiasd
2231aeab4d
fix:[Cherry-Pick] Split delete task msg to MaxMessageSize (#36574) 
relate: https://github.com/milvus-io/milvus/issues/36089
pr: https://github.com/milvus-io/milvus/pull/36197
split delete task msg to MaxMessageSize to avoid mq message too large
error

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2024-09-27 18:15:19 +08:00
cai.zhang
e5a6c5b31d
fix: [cherry-pick]Check string array max length after type matching (#36497) 
issue: #36029 

master pr: #36449

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-09-27 11:15:14 +08:00
Zhen Ye
e34fa0461b
fix: port listen racing in mix or standalone mode (#36459) 
issue: #36441
pr: #36442

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-09-26 21:01:15 +08:00
wei liu
ad5d24be65
enhance: Optimize workload based replica selection policy (#36181) (#36384) 
issue: #35859
pr: #36181

This PR introduce two new param: toleranceFactor and checkRequestNum,
after every checkRequestNum request has been assigned, try to compute
querynode's workload score.

if the diff is less than the toleranceFactor, replica selection policy
will fallback to round_robin, which reduce the average cost to about
500ns.

if the diff is larger than the toleranceFactor, replica selection policy
will compute querynode's score to select the target node with smallest
score in every assigment.

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-09-26 11:19:14 +08:00
jaime
b92daa1532
fix: iaccurate size estimation for encoded array data (#36379) 
issue: #36029
pr: #36373

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-09-23 21:17:13 +08:00
congqixia
e66beb7ace
enhance: [2.4] Ignore index check for non-loaded vector field (#36170) (#36280) 
Cherry-pick from master
pr: #36170
Related to #35996

For `Field Partial Load` feature, Milvus shall ignore index check for
non-loaded vector field.

Also, this PR unifies the logic of index check for load collection and
load partitions tasks.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-09-19 10:07:12 +08:00
zhagnlu
d1730b8853
fix: fix rewrite output field because of merge error (#36338) 
issue: #36340

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-09-18 20:01:12 +08:00
cai.zhang
eb47150f66
enhance: [cherry-pick]Disallow the keywords as a field name or dynamic field name (#36108) 
issue: #35873

master pr: #36101

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-09-15 15:19:14 +08:00
zhenshan.cao
2bc454bf40
fix: keep inner topK to avoid exceeding efSearch (#36287) 
issue :https://github.com/milvus-io/milvus/issues/36243
pr : https://github.com/milvus-io/milvus/pull/36284

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2024-09-15 15:12:38 +08:00
zhenshan.cao
34e5f99bd6
fix: Fix improper use of offset in HybridSearch (#36253) 
pr : https://github.com/milvus-io/milvus/pull/36244
issue : https://github.com/milvus-io/milvus/issues/36243

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2024-09-13 22:05:14 +08:00
wei liu
5cd860652d
enhance: Enable dynamic update replica selection policy (#35824) 
issue: #35859
pr: #35860

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-09-13 17:05:15 +08:00
wei liu
16daf9177f
fix: fix ReadWrite privilege group deny all global API (#36145) 
issue: #35471
pr: #36144

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-09-13 10:31:07 +08:00
congqixia
1cd8d1bd80
enhance: [2.4] Use stats Handler to record request/response size metrics (#36107) (#36118) 
Cherry-pick from master
pr: #36107 
Related to #36102

This PR use newly added `grpcSizeStatsHandler` to reduce calling
`proto.Size` since the request & response size info is recorded by grpc
framework.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-09-10 17:13:08 +08:00
jaime
256d4e209f
fix: memory leak in proxy meta cache (#36076) 
issue: #36074
pr: #36075

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-09-08 17:49:06 +08:00
wei liu
ceca666e2a
fix: Fix privilege group hasn't been register for validate (#35938) 
issue: #35471
pr: #35937

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-09-05 18:09:11 +08:00
yihao.dai
b578064869
fix: Fix DB limiter nodes are mistakenly cleaned up (#35991) (#35992) 
This issue only occurs for a short time right after a table is created.
To avoid this, we simply reduce the frequency of cleaning up invalid
limiter nodes.

issue: https://github.com/milvus-io/milvus/issues/35933

pr: https://github.com/milvus-io/milvus/pull/35991

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-09-05 17:31:05 +08:00
congqixia
abf3f68ae9
enhance: [2.4] Fix typo of clustering key not loaded msg (#35948) (#36000) 
Cherry-pick from master
pr: #35948
Related to #35415

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-09-05 16:59:07 +08:00
jaime
2c1fa50412
enhance: remove cooling off in rate limiter for read requests (#35936) 
issue: #35934
pr: #35935

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-09-04 14:39:10 +08:00
congqixia
268c60c301
fix: [2.4] Check clustering key skip load behavior (#35865) (#35899) 
Cherry-pick from master
pr: #35865
feature issue: #35415
See also #35861

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-09-02 16:03:04 +08:00
SimFG
8b706122a8
enhance: [2.4] support to drop the role which is related the privilege list (#35863) 
- issue: #35545
- pr: #35727

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-08-31 21:57:02 +08:00
wei liu
da026b1e28
enhance: Add depguard rules to ban deprecated proto lib (#35140) (#35818) 
See also #34394 #34252
pr: #35140

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Co-authored-by: congqixia <congqi.xia@zilliz.com>
 2024-08-30 14:13:01 +08:00
congqixia
8d3685fadf
enhance: [2.4] Print log only when rate limit updates (#35806) (#35816) 
Cherry-pick from master
pr: #35806
The debug log for "RateLimiter register for rateType" is too frequent
and in e2e cases, the may print 18M times in one run.

This PR make the log be printed only when rate limit is updated.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-08-29 19:09:01 +08:00
congqixia
12575885d3
enhance: [2.4] Add skip load validation for create collection task (#35761) 
Cherry-pick from master
pr: #35737 
Related to #35415

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-08-28 18:25:00 +08:00
yihao.dai
8e6ec58652
fix: Fix rate wasn't limited to the expected value (#35699) (#35700) 
Each time the rate is reset, the token bucket is fully refilled, causing
the rate wasn't limited to the expected value. This PR addresses the
issue by preventing the token reset.

issue: https://github.com/milvus-io/milvus/issues/35675,
https://github.com/milvus-io/milvus/issues/35702

pr: https://github.com/milvus-io/milvus/pull/35699

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-08-28 10:29:00 +08:00
Zhen Ye
a4533f1b8a
enhance: optimize milvus core building (#35660) 
issue: #35549,#35611,#35633
pr: #35610

- remove milvus_segcore milvus_indexbuilder..., add libmilvus_core
- core building only link once
- move opendal compilation into cmake
- fix odr

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-08-27 18:55:00 +08:00
SimFG
fc324b4254
feat: [2.4] add the rbac msg and send them to the replicate channel (#35562) 
- issue: #35391
- pr: #35392

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-08-27 14:45:00 +08:00
congqixia
ab261d0f8b
feat: [2.4] Support field partial load collection (#35416) (#35696) 
Cherry-pick from master
pr: #35416
Related to #35415

---------

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-08-27 14:07:00 +08:00
SimFG
5b5119a51f
feat: [2.4] provide more general configuration to control mmap behavior (#35609) 
- issue: #35273
- pr: #35359

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-08-23 12:35:02 +08:00
congqixia
9c67965d6c
fix: [2.4] Re-read value after once initialization (#35643) 
Cherry-pick from master
pr: #35642
Related to #35641
See also #35271

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-08-22 16:30:57 +08:00
zhagnlu
946881e123
enhance: optimize retrieve on dynamic field (#35602) 
pr: #35580

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
Co-authored-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2024-08-22 14:18:57 +08:00
wei liu
e2542a1bf5
enhance: Update protobuf-go to protobuf-go v2 (#34394) (#35555) 
issue: #34252
pr: #34394 #35072 #35084

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-08-21 18:50:58 +08:00
wei liu
4bf4cbad85
enhance: Mark query node as read only after suspend (#35492) (#35586) 
issue: #34985 #35493
pr: #35492
after querynode has been suspended, it's not allow to load
segment/channel on it, which means the node is read only. to be
compatible with resource group design, after query node has been
suspend, we remove it from it's original resource group, make it a read
only query node in replica. then two things will happens:
1. it's original resource group will be lacking of query nodes, query
coord will assign new node to it.
2. querycoord will try to move out all segments/channels after querynode
has been suspended

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-08-20 19:00:56 +08:00
Chun Han
cf8494ef45
enhance: support httpv1/v2 throttle and add it for httpV2(#35350) (#35504) 
related: #35350
pr: https://github.com/milvus-io/milvus/pull/35470

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
 2024-08-20 16:32:56 +08:00
wei liu
fc344d1eae
fix: Fix proxy panic after Import related api failed (#35540) (#35559) 
issue: #35539
pr: #35540

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-08-19 19:42:55 +08:00
wei liu
14ec3dc357
enhance: Enable ReadOnly/ReadWrite/Admin Privilege Group to simplify RBAC grant progress (#35472) (#35543) 
issue: #35471
pr: #35472 #35515

---------

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-08-19 16:24:54 +08:00
wei liu
248a6ea401
enhance: Add BackupRBAC/RestoreRBAC API to enable rbac backup (#35444) (#35513) 
issue: #35443
pr: #35444

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-08-18 13:42:55 +08:00
congqixia
555b7a6aa7
enhance: [2.4] Remove duplicated schema helper creation in proxy (#35489) (#35502) 
Cherry-pick from master
pr: #35489
Related to PRs of #35415

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-08-16 14:32:54 +08:00
SimFG
4fae5407dd
enhance: [2.4] remove the invalid log (#35473) 
/kind improvement
pr: #35465

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-08-15 14:14:53 +08:00
aoiasd
a20cb727eb
enhance:[Cherry-pick] Check by proxy rate limiter when delete get data by query. (#30891) (#35262) 
relate: https://github.com/milvus-io/milvus/issues/30927
pr: https://github.com/milvus-io/milvus/pull/30891

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2024-08-13 14:32:21 +08:00
smellthemoon
23052f6ac2
fix: upsert use the previous pk in insert when autoid(#34672) (#35130) 
issue: #34668 
pr: #34672

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2024-08-12 18:58:25 +08:00
wei liu
a4fa9ced16
fix: delete by expr failed at retry progress (#35241) (#35268) 
issue: #35240
pr: #35241
delete by expr shard the same err object between channels, so if one
channel's query failed, it will fail all channel, which will break
channel level retry policy, and make delete operation failed.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-08-09 16:12:18 +08:00
zhagnlu
db9e4b898a
enhance: support bitmap index (#35336) 
pr: #32902
cherry-pick bitmap index from master

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-08-07 20:12:18 +08:00
Jiquan Long
b11db52160
fix: error message when input is not aligned (#35322) (#35323) 
fix: https://github.com/milvus-io/milvus/issues/35321
pr: https://github.com/milvus-io/milvus/pull/35322

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2024-08-07 09:56:28 +08:00
congqixia
2a1fc700a2
enhance: [2.4] Add in-memory cache for casbin enforcer result (#35272) 
Cherry-pick from master
pr: #35271
See also #35270

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-08-06 12:14:17 +08:00
congqixia
f8444b900f
enhance: [2.4] Support proxy/delegator qn client pooling (#35195) 
Cherry pick from master
pr: #35194
See also #35196
Add param item for proxy/delegator query node client pooling and
implement pooling logic

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-08-02 11:24:19 +08:00
Jiquan Long
86edca8c1b
fix: support auto index for array (#35095) 
/kind branch-feature
pr: #34450

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
Co-authored-by: Zhagnlu <lu.zhang@zilliz.com>
 2024-07-30 17:57:50 +08:00
Patrick Weizhi Xu
e2fc8853b6
fix: [2.4] disallow expr when partition key isolation is enabled (#35025) 
issue: #34336 
pr: #35031

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
 2024-07-29 14:23:49 +08:00