300 Commits

Author SHA1 Message Date
yah01
0e71923408
enhance: enable converting segcore error to merr (#29914) (#30178)
this converts the segcore error to merr if possible
pr: #29914

Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-01-22 16:56:55 +08:00
yah01
1cc5a613d5
enhance: adjust the GPU pool size (#29937) (#30177)
according to benchmark, the GPU pool size with 6 performs best
pr: #29937

Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-01-22 16:55:04 +08:00
MrPresent-Han
6aaccdd5f4
feat: support general capacity restrict for cloud-side resoure contro… (#30017)
related: #29844
pr: #https://github.com/milvus-io/milvus/pull/29845

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-01-22 16:18:56 +08:00
SimFG
2465d86138
enhance: [2.3] support related privilege for grant api (#30154)
/kind improvement
pr: #30153

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-01-22 14:42:55 +08:00
foxspy
0700434c58
fix: patching search cache param when index meta does not hold one (#30116)
patch search cache param from index configs when index meta could not
get the search cache size key

issue: #30113 
pr: #30119

Signed-off-by: xianliang <xianliang.li@zilliz.com>
2024-01-19 11:50:56 +08:00
SimFG
be1470a654
enhance: [2.3] Add load/release partitions to replicate msg stream (#30001)
/kind improvement
pr: #28399

---------

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-01-18 22:50:55 +08:00
congqixia
7f32576f36
enhance: [cherry-pick] replace magic number with ParamItem for dist handler (#30020) (#30070)
Cherry-pick from master
pr: #30020
See also #28817

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-01-18 15:58:54 +08:00
wei liu
0447ef5df3
fix: Unexpected rpc msg size limit (#29682) (#29983)
pr: #29682
due to `clientMaxSendSize` and `serverMaxRecvSize` will limit the rpc
request size limit, they should use same config value, and
`serverMaxSendSize` and `clientMaxRecvSize` will limit the rpc response
size limit, they should use same config value too.

This PR fix unexpected rpc msg limit which caused by the wrong usage of
misunderstanding rpc config items

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-01-16 11:18:52 +08:00
zhenshan.cao
a689ea3228
feat: Add RBAC functionality to alias (#29885) (#29947)
issue: https://github.com/milvus-io/milvus/issues/29781
issue: https://github.com/milvus-io/milvus-proto/issues/237
pr : https://github.com/milvus-io/milvus/pull/29885

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-01-12 23:36:52 +08:00
jaime
fb956536b9
fix: remove checking if running inside container (#29941)
issue: https://github.com/milvus-io/milvus/issues/29846
pr: #29940

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-01-12 21:12:52 +08:00
wayblink
e1446da83c
feat: [Cherry-pick] Implement DescribeAlias and ListAliases interfaces (#29896)
#22882
pr: #29641

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-01-12 16:30:51 +08:00
jaime
c0b711e9fb
enhance: Support read hardware metrics for cgroupv2 (#29847)
issue: #29846
pr: #29850

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-01-11 19:20:57 +08:00
congqixia
cd93954214
enhance: [Cherry-pick] pre-allocate result FieldData space to reduce growslice (#29726) (#29866)
Cherry-pick from master
pr: #29726

See also: #29113

Add a new utitliy function in `pkg/util/typetuil` to pre-allocate field
data slice capacity acoording to search limit. This shall avoid copying
the data during `AppendFieldData` when previous slice is out of space.
And shall also save CPU time during high paylog.

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-01-11 17:59:01 +08:00
wei liu
322e9f39a3
fix: Remove Unnecessary lock in config manager (#29855)
issue: #29709 #291712
pr: #29836
to avoid concurrent recursive RLock and Lock cause deadlock, This PR
remove the unnecessary lock in config manager

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-01-11 15:01:01 +08:00
yah01
38c61594c0
enhance: use GPU pool for gpu tasks (#29678) (#29706)
- this much improve the performance for GPU index
- this also reduce 1x copy while parsing index meta
pr: #29678

Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-01-09 14:04:48 +08:00
SimFG
a2365e4b2a
enhance: [2.3] Add concurrency for datacoord segment GC (#29557)
issue: #29553
pr: https://github.com/milvus-io/milvus/pull/29561
/kind improvement

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-01-03 13:16:57 +08:00
MrPresent-Han
757834602a
enhance: add param for bloomfilter(#29388) (#29614)
related: https://github.com/milvus-io/milvus/issues/29388
pr: https://github.com/milvus-io/milvus/pull/29490

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-01-02 18:58:47 +08:00
MrPresent-Han
151a5c3ca8
fix: iterator lose data for duplicted result(#29406) (#29446)
related: #29406
pr: #29451

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2023-12-27 23:22:46 +08:00
wei liu
26b1853c54
fix: Auto balance param can't be updated by dynamic(#29501) (#29502)
pr: #29501
This PR fixed that auto balance param can't be updated by dynamic

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-12-27 14:30:53 +08:00
aoiasd
f5bba79963
enhance: [Cherry-Pick] add default local path for access log (#29461)
relate: https://github.com/milvus-io/milvus/issues/29459
pr: https://github.com/milvus-io/milvus/pull/29460

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-12-26 17:52:47 +08:00
MrPresent-Han
95d1056e32
refine the range of chunk size config value(#29388) (#29472)
related: #https://github.com/milvus-io/milvus/issues/29388
pr: https://github.com/milvus-io/milvus/pull/29389

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2023-12-26 14:34:46 +08:00
PowderLi
0c3de8d886
fix: Partition related privileges should belong to Collection (#29429)
issue: #29427
master pr: #29428

it will try to find privilege_name from the object_type, while grant a
privilege to a role

Signed-off-by: PowderLi <min.li@zilliz.com>
2023-12-25 14:18:43 +08:00
wei liu
5fa63da4e3
fix: Rename invalid parameterutil package path (#29334) (#29391)
pr: #29334
This PR renames the invalid parameterutil package path

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-12-22 10:06:41 +08:00
SimFG
74e72ce27e
enhance: [2.3] Support to get the param value in the runtime (#29298)
pr: #29297
/kind improvement

Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-12-21 20:36:43 +08:00
yihao.dai
d8800f5ca5
fix: Return time tick delay error and refine quota error messages (#29289) (#29363)
This pr:
Handles the time tick delay error when converting old error codes to
milvus errors.
Enhances quota error messages by eliminating "force deny" and
substituting it with "quota exceeded."

pr: https://github.com/milvus-io/milvus/pull/29289
issue: https://github.com/milvus-io/milvus/issues/29288

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-12-21 14:58:49 +08:00
Jiquan Long
8ef0c571b3
enhance: make consistency level used in delete configurable (#29280) (#29284)
pr: #29280 
issue: #29279

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2023-12-20 14:14:34 +08:00
aoiasd
af54ce9e20
fix: [Cherry-pick] msgstream memory leak caused by config event don't deregister (#29268)
pr: https://github.com/milvus-io/milvus/pull/29266
relate: https://github.com/milvus-io/milvus/issues/28620

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-12-19 12:34:39 +08:00
wei liu
2d33c7fe41
enhance: Add config for querycoord auto balance channel (#29231) (#29262)
issue: #23726
pr: #29231
This PR add control config to querycoord's background auto balance
channel operation

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-12-18 14:32:41 +08:00
PowderLi
9af24da65a
fix: rootCoord decide the builtin role cannot be deleted (#29248)
issue: #29243

only rootCoord read the configuration item `builtinRoles`, so proxy
never know whether the role to be deleted is builtin.

Signed-off-by: PowderLi <min.li@zilliz.com>
2023-12-18 14:22:42 +08:00
congqixia
ea688846d8
enhance: [cherry-pick] make conc.Pool resizable (#29225)
See also #29223
cherry pick part of master commit
pr: #29224
Make `conc.Pool` resizable by adding Resize method for it.

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-15 15:28:38 +08:00
congqixia
efef573c67
enhance: [cherry-pick] Add a config item for partition name as regexp feature (#29154) (#29183)
Cherry pick from master
pr: #29154 
See also #29177
Add a config item for partition name as regexp feature and disable it by
default

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-14 14:46:37 +08:00
Jiquan Long
00e6160848
fix: autoindex panic with flat index type (#29071) (#29072)
issue: https://github.com/milvus-io/milvus/issues/29048
pr: https://github.com/milvus-io/milvus/pull/29071

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2023-12-12 22:08:45 +08:00
wei liu
286dce0d3a
enhance: Add config for auto upgrade segment index (#29112) (#29133)
pr: #29112
This PR add config for auto upgrade segment index to index engine's
version.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-12-12 17:26:38 +08:00
Enwei Jiao
1e2a0dd127
enhance: Support otlp with insecure (#29131)
pr: https://github.com/milvus-io/milvus/pull/29115

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-12-12 16:44:37 +08:00
Jiquan Long
c77fe38c39
fix: txn control not applied to snapshot.Remove (#29124) (#29137)
issue: #29123
pr: #29124 

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2023-12-12 16:37:44 +08:00
yah01
4334e4e7ad
enhance: remove merger for load segments (#29062) (#29064)
remove merger as now QueryNode could load segments concurrently
fix https://github.com/milvus-io/milvus/issues/29063
pr: #29062

Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-12-12 16:22:50 +08:00
wayblink
e49860cb80
feat: Introduce channelCheckpointUpdater to reduce goroutine in ttNode (#29107)
pr: #28570

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2023-12-12 13:48:42 +08:00
PowderLi
92db41518d
enhance: upgrade proto to use partition related privileges (#29100)
issue: [milvus-proto
#212](https://github.com/milvus-io/milvus-proto/issues/212)
master pr: #28961

milvus can't use partition related privileges until upgrade
milvus-proto, even if them were added to milvus-proto

Signed-off-by: PowderLi <min.li@zilliz.com>
2023-12-12 13:22:38 +08:00
shaoyue
ac0f000c29
enhance: Add proxy.ginLogSkipPaths (#29008)
pr: #28945

/cc @czs007

Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
2023-12-08 17:55:30 +08:00
shaoyue
ecbb6160aa
enhance: storeageType default value change to remote (#29009)
pr: #28792
/cc @czs007

Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
2023-12-08 17:54:33 +08:00
zhenshan.cao
bba0e159ba
feat: [pick] Add bypassing logic for ttMsg in flowgraph of DataNode(#28756) (#29036)
In order to minimize the CPU usage of the coroutine and avoid frequent
execution of time-consuming operations in the flowgraph when the message
stream consists solely of "ttMsg," it is recommended to implement a
mechanism for quickly bypassing the subsequent flowgraph node processing
logic.

If "ttMsg" is continuously received for a certain period of time
(coldTime), the flowgraph enters skipMode. Once in skipMode, every
skipNum "ttMsg" messages are merged into one for processing. If a
non-"ttMsg" message is received while in skipMode, the flowgraph exits
skipMode.

pr: #28756

Signed-off-by: wayblink <anyang.wang@zilliz.com>
Co-authored-by: wayblink <anyang.wang@zilliz.com>
2023-12-08 12:10:39 +08:00
PowderLi
10d3224638
enhance: add 3 builtin roles #28961 (#29010)
issue: #28960
master pr: #28961

add new configuration: builtinRoles
user can define roles in config file: milvus.yaml

there is an example:

db_ro, only have read privileges, include load
db_rw, read and write privileges, include create/drop/rename collection
db_admin, not only read and write privileges, but also user
administration

Signed-off-by: PowderLi <min.li@zilliz.com>
2023-12-08 10:52:36 +08:00
cai.zhang
7f59826089
enhance: [Pick]Make import-related error message clearer (#28979)
issue: #28976 
master pr: #28978

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2023-12-08 10:16:35 +08:00
aoiasd
b639fd66b4
fix: [Cherry-pick] accesslog can not print search expression (#28931)
relate: https://github.com/milvus-io/milvus/issues/28893
pr: https://github.com/milvus-io/milvus/pull/28899

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-12-08 02:00:36 +08:00
cqy123456
8fd38c8eea
enhance:[cherry-pick] Use binlog index for better search performance (#29012)
this pr is cherry-pick from master:
pr: https://github.com/milvus-io/milvus/pull/28528
pr: https://github.com/milvus-io/milvus/pull/27673
related issue:
issue: https://github.com/milvus-io/milvus/issues/27678

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2023-12-07 09:52:34 +08:00
groot
1f8c1afe8a
fix: Change ranges of HNSW parameters according to Knowhere (#28877)
fix: https://github.com/milvus-io/milvus/issues/28860
pr: https://github.com/milvus-io/milvus/pull/28875

Signed-off-by: yhmo <yihua.mo@zilliz.com>
2023-12-05 16:20:42 +08:00
congqixia
3a33afd1fb
enhance: [cherry-pick] Change const magic number in querycoord to param (#28819) (#28947)
Cherry-pick from master
pr: #28819 
See also #28817

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-04 19:06:40 +08:00
aoiasd
224f2755c5
enhance: [Cherry-pick]Refine accesslog to support format by yaml and support more metric (#28740)
relate: https://github.com/milvus-io/milvus/issues/28086
pr: https://github.com/milvus-io/milvus/issues/28086

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-11-28 15:34:28 +08:00
cai.zhang
ef115dd7df
enhance: Support Array DataType for bulk_insert (#28578)
issue: https://github.com/milvus-io/milvus/issues/28272
master pr: #28341 
Support array DataType for bulk_insert with json, binlog files.

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2023-11-27 16:56:27 +08:00
wei liu
9e82a75c22
enhance: Increase heartbeat timeout (#28648) (#28721)
pr: #28648
it's easy to trigger heartbeat timeout after 100ms when standalone cpu
usage reach 100%.
This PR increase the heartbeat timeout param to 2000ms

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-11-27 10:40:26 +08:00