7785 Commits

Author SHA1 Message Date
PowderLi
04d351e4bb
fix: wait server to listen the http port (#29076)
issue: #29068
master pr: #29075

wait server to listen the http port
then check whether various urls can be accessed normally

Signed-off-by: PowderLi <min.li@zilliz.com>
2023-12-12 16:24:37 +08:00
yah01
4334e4e7ad
enhance: remove merger for load segments (#29062) (#29064)
remove merger as now QueryNode could load segments concurrently
fix https://github.com/milvus-io/milvus/issues/29063
pr: #29062

Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-12-12 16:22:50 +08:00
cai.zhang
bd46dc2d1b
enhance: [Pick]Support implicit type conversion for parquet (#29119)
issue: https://github.com/milvus-io/milvus/issues/29019
master pr: #29046

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2023-12-12 16:14:43 +08:00
wayblink
af6714ef59
fix: [cherry-pick] fix rootcoord_collection_num metric is twice the real value (#29105)
pr: #28666

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2023-12-12 14:06:43 +08:00
wayblink
e49860cb80
feat: Introduce channelCheckpointUpdater to reduce goroutine in ttNode (#29107)
pr: #28570

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2023-12-12 13:48:42 +08:00
PowderLi
92db41518d
enhance: upgrade proto to use partition related privileges (#29100)
issue: [milvus-proto
#212](https://github.com/milvus-io/milvus-proto/issues/212)
master pr: #28961

milvus can't use partition related privileges until upgrade
milvus-proto, even if them were added to milvus-proto

Signed-off-by: PowderLi <min.li@zilliz.com>
2023-12-12 13:22:38 +08:00
congqixia
d36bf21f28
enhance: [cherry-pick] Avoid initializing casbin enforcer for each request (#29118)
Cherry-pick from master
pr: #29117 
See also #29113

This patch:
- Replace plain Enforcer with `casbin.SyncedEnforcer`
- Add implementation of persist.Adapter with `MetaCacheCasbinAdapter`
- Invoke enforcer.LoadPolicy when policy updated

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-12 10:36:43 +08:00
congqixia
0b532b42ea
enhance: [cherry-pick] Add client connect wrapper to keep connection alive (#29061)
Cherry-pick from master
pr: #29058
See also #29057
Add wrapper to maintain client&connection
When reset operation is needed, close method shall wait until all
on-going request return

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-08 21:12:35 +08:00
shaoyue
ac0f000c29
enhance: Add proxy.ginLogSkipPaths (#29008)
pr: #28945

/cc @czs007

Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
2023-12-08 17:55:30 +08:00
zhenshan.cao
bba0e159ba
feat: [pick] Add bypassing logic for ttMsg in flowgraph of DataNode(#28756) (#29036)
In order to minimize the CPU usage of the coroutine and avoid frequent
execution of time-consuming operations in the flowgraph when the message
stream consists solely of "ttMsg," it is recommended to implement a
mechanism for quickly bypassing the subsequent flowgraph node processing
logic.

If "ttMsg" is continuously received for a certain period of time
(coldTime), the flowgraph enters skipMode. Once in skipMode, every
skipNum "ttMsg" messages are merged into one for processing. If a
non-"ttMsg" message is received while in skipMode, the flowgraph exits
skipMode.

pr: #28756

Signed-off-by: wayblink <anyang.wang@zilliz.com>
Co-authored-by: wayblink <anyang.wang@zilliz.com>
2023-12-08 12:10:39 +08:00
MrPresent-Han
c21a1d59e4
fix: createCollection failed occasionally (#28592) (#28989)
related: #28592
pr:#https://github.com/milvus-io/milvus/pull/28712

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2023-12-08 11:34:39 +08:00
PowderLi
10d3224638
enhance: add 3 builtin roles #28961 (#29010)
issue: #28960
master pr: #28961

add new configuration: builtinRoles
user can define roles in config file: milvus.yaml

there is an example:

db_ro, only have read privileges, include load
db_rw, read and write privileges, include create/drop/rename collection
db_admin, not only read and write privileges, but also user
administration

Signed-off-by: PowderLi <min.li@zilliz.com>
2023-12-08 10:52:36 +08:00
foxspy
82ac914bd4
enhance: [cherry-pick] support mmap for indexes loaded from disk file (#29002)
master pr: #28904 
issue: #28903

Signed-off-by: xianliang <xianliang.li@zilliz.com>
2023-12-08 10:32:34 +08:00
cai.zhang
7f59826089
enhance: [Pick]Make import-related error message clearer (#28979)
issue: #28976 
master pr: #28978

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2023-12-08 10:16:35 +08:00
MrPresent-Han
cd34173e55
fix: fix precision for search reduce(#27325) (#29032)
related: #27325 
pr: https://github.com/milvus-io/milvus/pull/29031

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2023-12-08 10:10:35 +08:00
aoiasd
b639fd66b4
fix: [Cherry-pick] accesslog can not print search expression (#28931)
relate: https://github.com/milvus-io/milvus/issues/28893
pr: https://github.com/milvus-io/milvus/pull/28899

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-12-08 02:00:36 +08:00
MrPresent-Han
5f4ac437b2
enhance: [Cherry-pick] Moving etcd client into session (#27069) (#28996)
relate: #26694
pr: https://github.com/milvus-io/milvus/pull/27069

Signed-off-by: Filip Haltmayer <filip.haltmayer@zilliz.com>
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
Co-authored-by: Filip Haltmayer <81822489+filip-halt@users.noreply.github.com>
2023-12-07 16:22:34 +08:00
aoiasd
8502037cff
fix: [Cherry-pick] sync action load segment with lack collection index info list (#28956)
relate: https://github.com/milvus-io/milvus/issues/28779
https://github.com/milvus-io/milvus/issues/28637
pr: https://github.com/milvus-io/milvus/pull/28788

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-12-07 14:14:42 +08:00
liliu-z
f6700551be
enhance: Bump Knowhere's version to 2.2.3 (#29035)
Knowhere's new bug fix version:
https://github.com/zilliztech/knowhere/releases/tag/v2.2.3

related issues: #28821 #28810 #27552 #27516 #28603 #21483

Signed-off-by: Li Liu <li.liu@zilliz.com>
2023-12-07 14:08:33 +08:00
SimFG
de28b73fe0
enhance: [2.3] Use a non-blocking method to trigger compaction when saveBinlogPath is executed (#28977)
/kind improvement
issue: #28924
pr: #28941

Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-12-07 13:14:34 +08:00
aoiasd
cb7b1d1d51
enhance: [Cherry-pick] pack proxy connection code and support accesslog print SDK_Version (#28844)
relate: https://github.com/milvus-io/milvus/issues/28086
pr: https://github.com/milvus-io/milvus/pull/28835

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-12-07 12:06:34 +08:00
cqy123456
8fd38c8eea
enhance:[cherry-pick] Use binlog index for better search performance (#29012)
this pr is cherry-pick from master:
pr: https://github.com/milvus-io/milvus/pull/28528
pr: https://github.com/milvus-io/milvus/pull/27673
related issue:
issue: https://github.com/milvus-io/milvus/issues/27678

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2023-12-07 09:52:34 +08:00
PowderLi
cdd1305f26
enhance: find collection schema from cache (#28916)
issue: #28781 #28329
master pr: #28782

1. There is no need to call `DescribeCollection`, if the collection's
schema is found in the globalMetaCache
2. did `GetProperties` to check the access to Azure Blob Service

Signed-off-by: PowderLi <min.li@zilliz.com>
2023-12-06 22:44:33 +08:00
congqixia
2e82d65124
fix: [cherry-pick] Align minio object storage ut to new minio server behavior (#29015)
cherry-pick from master
pr: #29014
See also #29013

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-06 15:42:43 +08:00
zhenshan.cao
cb19e590fb
fix: clean snapshots meta (#28502) (#28999)
issue: https://github.com/milvus-io/milvus/issues/28496 /kind bug
pr: #28502 
The input parameters collection.partitions and collection.Field are both
nil, so these two metas have not been cleared.

Signed-off-by: xige-16 <xi.ge@zilliz.com>
Co-authored-by: xige-16 <xi.ge@zilliz.com>
2023-12-06 09:51:01 +08:00
yihao.dai
3d7bcb5e36
enhance: Remove partition load infos with prefix (#28969) (#28973)
Removing partition load infos with prefix instead of removing them one
by one. This enhance can save transactions.

issue: https://github.com/milvus-io/milvus/issues/28343

pr: https://github.com/milvus-io/milvus/pull/28969

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-12-05 19:54:33 +08:00
wei liu
251b55844e
enhance: Remove too frequency read ddl log from rootcoord (#28891) (#28932)
pr: #28891
This PR removed too frequency log for such
`DescribeCollection/ShowPartition` operation from root coord

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-12-05 14:10:32 +08:00
PowderLi
43ec2c2bd7
enhance: [CHERRY-PICK] use already installed vcpkg (#28915)
issue: #28683 master pr: #28505
issue: #28686 master pr: #28703

1. update the base image: milvusdb/milvus-env (#28505) to avoid
downloading installation packages in CI workload install vcpkg and
install some packages in advance
2. use the latest image
2. update azure-identity-cpp from beta to release

Signed-off-by: PowderLi <min.li@zilliz.com>
2023-12-05 11:12:32 +08:00
yah01
1ab538be43
enhance: enable assert method to format arguments (#28812) (#28907)
for now the assert method in segcore could accept a string information,
too many codes don't print the value they assert.

make it happy
related #28811
pr: #28812

Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-12-05 10:22:32 +08:00
congqixia
2873be9264
fix: [2.3] Reject compaction task with growing segments (#28927)
See also #28924
The compaction task generated before datanode finish SaveBinlogPath grpc
call contains segments which are still in Growing state DataNode shall
verify each non-levelzero segments before submit compaction task to
executor

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-04 19:14:33 +08:00
congqixia
3a33afd1fb
enhance: [cherry-pick] Change const magic number in querycoord to param (#28819) (#28947)
Cherry-pick from master
pr: #28819 
See also #28817

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-04 19:06:40 +08:00
wei liu
c650240f31
enhance: Change some frequency log to rated level (#28897) (#28934)
pr: #28897
This pr change some frequency log's level to rated.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-12-04 18:52:32 +08:00
congqixia
ed550ebb7c
enhance: [cherry-pick] Resolve libunwind requirement conflict using 1.7.2 (#28935)
Cherry-pick from master
pr: #28929
Try to resolve libunwind dependency requirement conflict between glog &
folly

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-04 14:06:40 +08:00
congqixia
f4468dfd15
fix: [cherry-pick] Check whether compaction plan exist before update(#28826) (#28892)
pr: #28826 
Fix #28825

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-04 10:02:33 +08:00
wei liu
d2c171354f
fix: Balance channel may cause channel not availble error (#28829) (#28902)
pr: #28829
issue: #28831
release old delegator before new delegator update it's distribution may
cause `channel not available` error
This PR will block release old delgator before new delegator finish
`syncDistribution`

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-12-04 09:40:32 +08:00
jaime
275e3ea456
fix: Fix missing target db name for RenameCollection (#28911)
issue: #28908
pr: #28909

Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-12-03 08:28:31 +08:00
PowderLi
96c8e8d587
fix: azure ListObjects (#27931) (#28894)
issue: #27932
master pr: #27931

Signed-off-by: PowderLi <min.li@zilliz.com>
2023-12-01 18:34:31 +08:00
yihao.dai
c936cc8a95
fix: Fix get binary vector from chunk cache (#28866) (#28884)
The way of getting binary vector size is wrong. This PR will fix it.

issue: https://github.com/milvus-io/milvus/issues/28865

pr: https://github.com/milvus-io/milvus/pull/28866

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-12-01 17:50:32 +08:00
aoiasd
9bf33eb576
fix: [Cherry-Pick] accesslog method status not return failed when error in response (#28827)
relate: https://github.com/milvus-io/milvus/issues/28086
pr: https://github.com/milvus-io/milvus/pull/28824

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-12-01 15:30:38 +08:00
Gao
b6cb9e90d2
fix: [2.3] update folly to resolve simd issue (#28879)
related issue: https://github.com/milvus-io/milvus/issues/27552 , after
this, milvus could run successfully on sse4.2 only machine
pr: #28878

Signed-off-by: chasingegg <chao.gao@zilliz.com>
2023-12-01 13:52:32 +08:00
congqixia
63e1ac0846
fix: [cherry-pick] schema->size() check logic with system field (#28802) (#28841)
Cherry pick from master
pr: #28802

Now segcore load system field info as well, the growing segment
assertion shall not pass with "+ 2" value
This will cause all growing segments load failure
Fix #28801
Related to #28478
See also #28524

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-01 13:28:30 +08:00
yah01
ce8219f1a3
fix: bypass growing index if no index meta (#28791) (#28859)
we shouldn't panic if no index meta, just skip building it
fix #28022
pr: #28791

---------

Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-11-30 21:10:32 +08:00
aoiasd
880189e598
enhance: Delete by complex expression use partitionKey optimization (#27952) (#28128)
relate: https://github.com/milvus-io/milvus/issues/27948
pr: https://github.com/milvus-io/milvus/pull/27952

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-11-30 17:56:31 +08:00
yihao.dai
d0a56ecca5
fix: Fix minio latency monitoring for get operation (#28510) (#28643)
see also: https://github.com/milvus-io/milvus/issues/28509

pr: https://github.com/milvus-io/milvus/pull/28510

Currently Minio latency monitoring for get operation only collects the
duration of getting object (which just returns an io.Reader and does not
really read from minio), this pr will correct this behavior.

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-11-30 10:44:28 +08:00
cai.zhang
00461145b3
enhance: [Pick]Support importing data with parquet file (#28820)
issue: https://github.com/milvus-io/milvus/issues/28272
master pr: #28608 

Numpy does not support array type import.
Array type data is imported through parquet.

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2023-11-29 22:16:29 +08:00
cai.zhang
2062e5b368
enhance: [Pick]Update arrow version to v12 (#28425) (#28828)
master pr: #28425 
issue: https://github.com/milvus-io/milvus/issues/28423

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2023-11-29 20:04:28 +08:00
congqixia
e3653af9c9
fix: data race on rootcoord.proxyClientManager.proxyClient (#28787) (#28798)
Cherry-pick from master
pr: #28787
Fix #28786

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-11-29 18:00:27 +08:00
yihao.dai
a78ea4fea0
fix: Check ErrSegmentNotFound in delete node (#28371) (#28638)
We have been check ErrSegmentNotFound in insert_buffer_node in datanode,
we should also check it in delete_node.

issue: https://github.com/milvus-io/milvus/issues/27145

pr: https://github.com/milvus-io/milvus/pull/28371

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-11-29 17:50:27 +08:00
congqixia
0688ec4d83
enhance: [cherry-pick] Add channel checkpoint ts lag metric (#28767) (#28776)
Cherry-pick from master
pr: #28767
See also #28765

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-11-28 19:28:28 +08:00
Gao
ccca932cc6
fix: [2.3] correct autoindex segment num (#28429)
issue: #28386 
pr: #28387

Signed-off-by: chasingegg <chao.gao@zilliz.com>
2023-11-28 19:24:26 +08:00