18454 Commits

Author SHA1 Message Date
XuanYang-cn
d3cb489154
enhance: Change channel to Interface (#27839) (#29161)
This PR changes `*channel` into RWChannel interface

See also: #25309
pr: #27839

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-12-14 11:50:36 +08:00
SimFG
c2ce7bce1a
fix: [2.3] the unstable unit test TestReplicateManager (#29147)
/kind improvement
pr: #28718

Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-12-13 10:46:44 +08:00
congqixia
78865c02e1
enhance: [cherry-pick] Use zap.Stringer for large log field (#29143) (#29153)
Cherry pick from master
pr: #29143
See also #29113
Using zap.Stringer log field will evaluate log field value only when log
level meets the configuration, which could save some CPU time in search
route

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-13 09:48:38 +08:00
yah01
76757e53c4
enhance: Add more logs for target updating (#29090) (#29141)
This pull request enhances the logging functionality in the code related
to target updating. It adds more logs about the condition satisfying
when updating the target. The logs provide additional information about
the collection ID, replica number, channel readiness, segment readiness,
and leader view readiness. These logs will help in troubleshooting and
monitoring the target updating process.

pr: #29090

Signed-off-by: yah01 <yah2er0ne@outlook.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-12-12 22:28:38 +08:00
Jiquan Long
00e6160848
fix: autoindex panic with flat index type (#29071) (#29072)
issue: https://github.com/milvus-io/milvus/issues/29048
pr: https://github.com/milvus-io/milvus/pull/29071

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2023-12-12 22:08:45 +08:00
wei liu
286dce0d3a
enhance: Add config for auto upgrade segment index (#29112) (#29133)
pr: #29112
This PR add config for auto upgrade segment index to index engine's
version.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-12-12 17:26:38 +08:00
Enwei Jiao
1e2a0dd127
enhance: Support otlp with insecure (#29131)
pr: https://github.com/milvus-io/milvus/pull/29115

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-12-12 16:44:37 +08:00
Jiquan Long
c77fe38c39
fix: txn control not applied to snapshot.Remove (#29124) (#29137)
issue: #29123
pr: #29124 

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2023-12-12 16:37:44 +08:00
PowderLi
04d351e4bb
fix: wait server to listen the http port (#29076)
issue: #29068
master pr: #29075

wait server to listen the http port
then check whether various urls can be accessed normally

Signed-off-by: PowderLi <min.li@zilliz.com>
2023-12-12 16:24:37 +08:00
yah01
4334e4e7ad
enhance: remove merger for load segments (#29062) (#29064)
remove merger as now QueryNode could load segments concurrently
fix https://github.com/milvus-io/milvus/issues/29063
pr: #29062

Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-12-12 16:22:50 +08:00
cai.zhang
bd46dc2d1b
enhance: [Pick]Support implicit type conversion for parquet (#29119)
issue: https://github.com/milvus-io/milvus/issues/29019
master pr: #29046

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2023-12-12 16:14:43 +08:00
wayblink
af6714ef59
fix: [cherry-pick] fix rootcoord_collection_num metric is twice the real value (#29105)
pr: #28666

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2023-12-12 14:06:43 +08:00
wayblink
e49860cb80
feat: Introduce channelCheckpointUpdater to reduce goroutine in ttNode (#29107)
pr: #28570

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2023-12-12 13:48:42 +08:00
PowderLi
92db41518d
enhance: upgrade proto to use partition related privileges (#29100)
issue: [milvus-proto
#212](https://github.com/milvus-io/milvus-proto/issues/212)
master pr: #28961

milvus can't use partition related privileges until upgrade
milvus-proto, even if them were added to milvus-proto

Signed-off-by: PowderLi <min.li@zilliz.com>
2023-12-12 13:22:38 +08:00
nico
30ec598a08
test: modify cases for the update of IVF_FLAT (#29069)
pr: #27179

Signed-off-by: nico <cheng.yuan@zilliz.com>
2023-12-12 11:48:36 +08:00
congqixia
d36bf21f28
enhance: [cherry-pick] Avoid initializing casbin enforcer for each request (#29118)
Cherry-pick from master
pr: #29117 
See also #29113

This patch:
- Replace plain Enforcer with `casbin.SyncedEnforcer`
- Add implementation of persist.Adapter with `MetaCacheCasbinAdapter`
- Invoke enforcer.LoadPolicy when policy updated

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-12 10:36:43 +08:00
congqixia
0b532b42ea
enhance: [cherry-pick] Add client connect wrapper to keep connection alive (#29061)
Cherry-pick from master
pr: #29058
See also #29057
Add wrapper to maintain client&connection
When reset operation is needed, close method shall wait until all
on-going request return

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-08 21:12:35 +08:00
edward.zeng
273be6c77a
fix: Fix etcd healthcheck command (#28357)
Fix etcd health check command.
/kind improvement
pr: #27980 
/cc @yanliang567

Signed-off-by: Edward Zeng <jie.zeng@zilliz.com>
2023-12-08 17:57:23 +08:00
shaoyue
ac0f000c29
enhance: Add proxy.ginLogSkipPaths (#29008)
pr: #28945

/cc @czs007

Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
2023-12-08 17:55:30 +08:00
shaoyue
ecbb6160aa
enhance: storeageType default value change to remote (#29009)
pr: #28792
/cc @czs007

Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
2023-12-08 17:54:33 +08:00
zhenshan.cao
bba0e159ba
feat: [pick] Add bypassing logic for ttMsg in flowgraph of DataNode(#28756) (#29036)
In order to minimize the CPU usage of the coroutine and avoid frequent
execution of time-consuming operations in the flowgraph when the message
stream consists solely of "ttMsg," it is recommended to implement a
mechanism for quickly bypassing the subsequent flowgraph node processing
logic.

If "ttMsg" is continuously received for a certain period of time
(coldTime), the flowgraph enters skipMode. Once in skipMode, every
skipNum "ttMsg" messages are merged into one for processing. If a
non-"ttMsg" message is received while in skipMode, the flowgraph exits
skipMode.

pr: #28756

Signed-off-by: wayblink <anyang.wang@zilliz.com>
Co-authored-by: wayblink <anyang.wang@zilliz.com>
2023-12-08 12:10:39 +08:00
MrPresent-Han
c21a1d59e4
fix: createCollection failed occasionally (#28592) (#28989)
related: #28592
pr:#https://github.com/milvus-io/milvus/pull/28712

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2023-12-08 11:34:39 +08:00
PowderLi
10d3224638
enhance: add 3 builtin roles #28961 (#29010)
issue: #28960
master pr: #28961

add new configuration: builtinRoles
user can define roles in config file: milvus.yaml

there is an example:

db_ro, only have read privileges, include load
db_rw, read and write privileges, include create/drop/rename collection
db_admin, not only read and write privileges, but also user
administration

Signed-off-by: PowderLi <min.li@zilliz.com>
2023-12-08 10:52:36 +08:00
foxspy
82ac914bd4
enhance: [cherry-pick] support mmap for indexes loaded from disk file (#29002)
master pr: #28904 
issue: #28903

Signed-off-by: xianliang <xianliang.li@zilliz.com>
2023-12-08 10:32:34 +08:00
cai.zhang
7f59826089
enhance: [Pick]Make import-related error message clearer (#28979)
issue: #28976 
master pr: #28978

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2023-12-08 10:16:35 +08:00
MrPresent-Han
cd34173e55
fix: fix precision for search reduce(#27325) (#29032)
related: #27325 
pr: https://github.com/milvus-io/milvus/pull/29031

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2023-12-08 10:10:35 +08:00
aoiasd
b639fd66b4
fix: [Cherry-pick] accesslog can not print search expression (#28931)
relate: https://github.com/milvus-io/milvus/issues/28893
pr: https://github.com/milvus-io/milvus/pull/28899

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-12-08 02:00:36 +08:00
MrPresent-Han
5f4ac437b2
enhance: [Cherry-pick] Moving etcd client into session (#27069) (#28996)
relate: #26694
pr: https://github.com/milvus-io/milvus/pull/27069

Signed-off-by: Filip Haltmayer <filip.haltmayer@zilliz.com>
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
Co-authored-by: Filip Haltmayer <81822489+filip-halt@users.noreply.github.com>
2023-12-07 16:22:34 +08:00
aoiasd
8502037cff
fix: [Cherry-pick] sync action load segment with lack collection index info list (#28956)
relate: https://github.com/milvus-io/milvus/issues/28779
https://github.com/milvus-io/milvus/issues/28637
pr: https://github.com/milvus-io/milvus/pull/28788

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-12-07 14:14:42 +08:00
liliu-z
f6700551be
enhance: Bump Knowhere's version to 2.2.3 (#29035)
Knowhere's new bug fix version:
https://github.com/zilliztech/knowhere/releases/tag/v2.2.3

related issues: #28821 #28810 #27552 #27516 #28603 #21483

Signed-off-by: Li Liu <li.liu@zilliz.com>
2023-12-07 14:08:33 +08:00
SimFG
de28b73fe0
enhance: [2.3] Use a non-blocking method to trigger compaction when saveBinlogPath is executed (#28977)
/kind improvement
issue: #28924
pr: #28941

Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-12-07 13:14:34 +08:00
aoiasd
cb7b1d1d51
enhance: [Cherry-pick] pack proxy connection code and support accesslog print SDK_Version (#28844)
relate: https://github.com/milvus-io/milvus/issues/28086
pr: https://github.com/milvus-io/milvus/pull/28835

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-12-07 12:06:34 +08:00
cqy123456
8fd38c8eea
enhance:[cherry-pick] Use binlog index for better search performance (#29012)
this pr is cherry-pick from master:
pr: https://github.com/milvus-io/milvus/pull/28528
pr: https://github.com/milvus-io/milvus/pull/27673
related issue:
issue: https://github.com/milvus-io/milvus/issues/27678

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2023-12-07 09:52:34 +08:00
PowderLi
cdd1305f26
enhance: find collection schema from cache (#28916)
issue: #28781 #28329
master pr: #28782

1. There is no need to call `DescribeCollection`, if the collection's
schema is found in the globalMetaCache
2. did `GetProperties` to check the access to Azure Blob Service

Signed-off-by: PowderLi <min.li@zilliz.com>
2023-12-06 22:44:33 +08:00
congqixia
2e82d65124
fix: [cherry-pick] Align minio object storage ut to new minio server behavior (#29015)
cherry-pick from master
pr: #29014
See also #29013

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-06 15:42:43 +08:00
yihao.dai
51614073ec
enhance: Update pymilvus (#28922) (#28991)
issue: https://github.com/milvus-io/milvus/issues/17614

pr: https://github.com/milvus-io/milvus/pull/28922

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-12-06 11:56:34 +08:00
zhenshan.cao
cb19e590fb
fix: clean snapshots meta (#28502) (#28999)
issue: https://github.com/milvus-io/milvus/issues/28496 /kind bug
pr: #28502 
The input parameters collection.partitions and collection.Field are both
nil, so these two metas have not been cleared.

Signed-off-by: xige-16 <xi.ge@zilliz.com>
Co-authored-by: xige-16 <xi.ge@zilliz.com>
2023-12-06 09:51:01 +08:00
yihao.dai
3d7bcb5e36
enhance: Remove partition load infos with prefix (#28969) (#28973)
Removing partition load infos with prefix instead of removing them one
by one. This enhance can save transactions.

issue: https://github.com/milvus-io/milvus/issues/28343

pr: https://github.com/milvus-io/milvus/pull/28969

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-12-05 19:54:33 +08:00
groot
1f8c1afe8a
fix: Change ranges of HNSW parameters according to Knowhere (#28877)
fix: https://github.com/milvus-io/milvus/issues/28860
pr: https://github.com/milvus-io/milvus/pull/28875

Signed-off-by: yhmo <yihua.mo@zilliz.com>
2023-12-05 16:20:42 +08:00
sre-ci-robot
0f0a529299
[automated] Update Builder image changes (#28975)
Update Builder image changes
See changes:
43ec2c2bd7
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2023-12-05 14:42:33 +08:00
sammy.huang
13d42ec33e
enhance: [skip e2e] upgrade minio version (#28906)
fix: https://github.com/milvus-io/milvus/issues/28880
pr: [#28971](https://github.com/milvus-io/milvus/pull/28971)

Signed-off-by: Sammy Huang <sammy.huang@zilliz.com>
2023-12-05 14:18:33 +08:00
wei liu
251b55844e
enhance: Remove too frequency read ddl log from rootcoord (#28891) (#28932)
pr: #28891
This PR removed too frequency log for such
`DescribeCollection/ShowPartition` operation from root coord

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-12-05 14:10:32 +08:00
PowderLi
43ec2c2bd7
enhance: [CHERRY-PICK] use already installed vcpkg (#28915)
issue: #28683 master pr: #28505
issue: #28686 master pr: #28703

1. update the base image: milvusdb/milvus-env (#28505) to avoid
downloading installation packages in CI workload install vcpkg and
install some packages in advance
2. use the latest image
2. update azure-identity-cpp from beta to release

Signed-off-by: PowderLi <min.li@zilliz.com>
2023-12-05 11:12:32 +08:00
yah01
1ab538be43
enhance: enable assert method to format arguments (#28812) (#28907)
for now the assert method in segcore could accept a string information,
too many codes don't print the value they assert.

make it happy
related #28811
pr: #28812

Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-12-05 10:22:32 +08:00
congqixia
2873be9264
fix: [2.3] Reject compaction task with growing segments (#28927)
See also #28924
The compaction task generated before datanode finish SaveBinlogPath grpc
call contains segments which are still in Growing state DataNode shall
verify each non-levelzero segments before submit compaction task to
executor

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-04 19:14:33 +08:00
congqixia
3a33afd1fb
enhance: [cherry-pick] Change const magic number in querycoord to param (#28819) (#28947)
Cherry-pick from master
pr: #28819 
See also #28817

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-04 19:06:40 +08:00
wei liu
c650240f31
enhance: Change some frequency log to rated level (#28897) (#28934)
pr: #28897
This pr change some frequency log's level to rated.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-12-04 18:52:32 +08:00
congqixia
ed550ebb7c
enhance: [cherry-pick] Resolve libunwind requirement conflict using 1.7.2 (#28935)
Cherry-pick from master
pr: #28929
Try to resolve libunwind dependency requirement conflict between glog &
folly

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-04 14:06:40 +08:00
nico
609800a5b7
test: modify test cases for error msg update(2.3) (#28849)
pr: #28846
issue: #23075

Signed-off-by: nico <cheng.yuan@zilliz.com>
2023-12-04 11:20:47 +08:00
congqixia
f4468dfd15
fix: [cherry-pick] Check whether compaction plan exist before update(#28826) (#28892)
pr: #28826 
Fix #28825

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-04 10:02:33 +08:00