admin/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2026-02-02 01:06:41 +08:00
Code Issues Packages Projects Releases Wiki Activity
18,461 Commits 146 Branches 196 Tags
Commit Graph

1263 Commits

Author SHA1 Message Date
Enwei Jiao
1e2a0dd127
enhance: Support otlp with insecure (#29131) 
pr: https://github.com/milvus-io/milvus/pull/29115

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-12-12 16:44:37 +08:00
PowderLi
92db41518d
enhance: upgrade proto to use partition related privileges (#29100) 
issue: [milvus-proto
#212](https://github.com/milvus-io/milvus-proto/issues/212)
master pr: #28961

milvus can't use partition related privileges until upgrade
milvus-proto, even if them were added to milvus-proto

Signed-off-by: PowderLi <min.li@zilliz.com>
 2023-12-12 13:22:38 +08:00
foxspy
82ac914bd4
enhance: [cherry-pick] support mmap for indexes loaded from disk file (#29002) 
master pr: #28904 
issue: #28903

Signed-off-by: xianliang <xianliang.li@zilliz.com>
 2023-12-08 10:32:34 +08:00
MrPresent-Han
cd34173e55
fix: fix precision for search reduce(#27325) (#29032) 
related: #27325 
pr: https://github.com/milvus-io/milvus/pull/29031

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-12-08 10:10:35 +08:00
liliu-z
f6700551be
enhance: Bump Knowhere's version to 2.2.3 (#29035) 
Knowhere's new bug fix version:
https://github.com/zilliztech/knowhere/releases/tag/v2.2.3

related issues: #28821 #28810 #27552 #27516 #28603 #21483

Signed-off-by: Li Liu <li.liu@zilliz.com>
 2023-12-07 14:08:33 +08:00
cqy123456
8fd38c8eea
enhance:[cherry-pick] Use binlog index for better search performance (#29012) 
this pr is cherry-pick from master:
pr: https://github.com/milvus-io/milvus/pull/28528
pr: https://github.com/milvus-io/milvus/pull/27673
related issue:
issue: https://github.com/milvus-io/milvus/issues/27678

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2023-12-07 09:52:34 +08:00
PowderLi
cdd1305f26
enhance: find collection schema from cache (#28916) 
issue: #28781 #28329
master pr: #28782

1. There is no need to call `DescribeCollection`, if the collection's
schema is found in the globalMetaCache
2. did `GetProperties` to check the access to Azure Blob Service

Signed-off-by: PowderLi <min.li@zilliz.com>
 2023-12-06 22:44:33 +08:00
PowderLi
43ec2c2bd7
enhance: [CHERRY-PICK] use already installed vcpkg (#28915) 
issue: #28683 master pr: #28505
issue: #28686 master pr: #28703

1. update the base image: milvusdb/milvus-env (#28505) to avoid
downloading installation packages in CI workload install vcpkg and
install some packages in advance
2. use the latest image
2. update azure-identity-cpp from beta to release

Signed-off-by: PowderLi <min.li@zilliz.com>
 2023-12-05 11:12:32 +08:00
yah01
1ab538be43
enhance: enable assert method to format arguments (#28812) (#28907) 
for now the assert method in segcore could accept a string information,
too many codes don't print the value they assert.

make it happy
related #28811
pr: #28812

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-12-05 10:22:32 +08:00
congqixia
ed550ebb7c
enhance: [cherry-pick] Resolve libunwind requirement conflict using 1.7.2 (#28935) 
Cherry-pick from master
pr: #28929
Try to resolve libunwind dependency requirement conflict between glog &
folly

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-12-04 14:06:40 +08:00
yihao.dai
c936cc8a95
fix: Fix get binary vector from chunk cache (#28866) (#28884) 
The way of getting binary vector size is wrong. This PR will fix it.

issue: https://github.com/milvus-io/milvus/issues/28865

pr: https://github.com/milvus-io/milvus/pull/28866

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-12-01 17:50:32 +08:00
Gao
b6cb9e90d2
fix: [2.3] update folly to resolve simd issue (#28879) 
related issue: https://github.com/milvus-io/milvus/issues/27552 , after
this, milvus could run successfully on sse4.2 only machine
pr: #28878

Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2023-12-01 13:52:32 +08:00
congqixia
63e1ac0846
fix: [cherry-pick] schema->size() check logic with system field (#28802) (#28841) 
Cherry pick from master
pr: #28802

Now segcore load system field info as well, the growing segment
assertion shall not pass with "+ 2" value
This will cause all growing segments load failure
Fix #28801
Related to #28478
See also #28524

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-12-01 13:28:30 +08:00
yah01
ce8219f1a3
fix: bypass growing index if no index meta (#28791) (#28859) 
we shouldn't panic if no index meta, just skip building it
fix #28022
pr: #28791

---------

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-11-30 21:10:32 +08:00
yah01
8ea1ef03ad
enhance: store system fields in segcore (#28524) (#28727) 
we need the system fields info for some usacase
fix: #28523

pr: #28524

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-28 15:32:31 +08:00
cai.zhang
3b06db1d04
enhance: [Pick] Handle knowhere error for creare diskann index (#28691) 
master pr: #28690

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2023-11-26 18:58:25 +08:00
yah01
c6ba4fa102
enhance: enhance the accuracy of memory usage (#28554) (#28559) 
before this, Milvus use container/system's memory info to get the memory
usage, which could be inaccurate.

we allocates the memory by private anon mmap,
then rss - shared would be the accurate memory usage

resolve https://github.com/milvus-io/milvus/issues/28553
pr: #28554

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-24 14:12:25 +08:00
zhagnlu
0bda17e97b
enhance: Add precheck when chunk manager init (#28593) 
#28329 
pr: https://github.com/milvus-io/milvus/pull/28330

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2023-11-23 20:34:25 +08:00
zhagnlu
41667f3227
enhance: Disable stdout buffer (#28597) 
#28390 
pr: https://github.com/milvus-io/milvus/pull/28391

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2023-11-23 20:20:24 +08:00
PowderLi
1383680b3a
add internal storage metrics (#28279) 
/kind improvement
issue: #28277

pr: #28278

Signed-off-by: PowderLi <min.li@zilliz.com>
 2023-11-19 17:22:25 +08:00
yah01
2073a0a76a
enhance: reduce 1x copy for variable length field while retrieving (#28345) (#28531) 
- Reduce 1x copy for varchar/string/JSON/array types while retrieving
- Reduce 1x copy for int8/int16 while retrieving
fix #28530
pr: #28345

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-17 20:48:21 +08:00
yah01
1743f581e6
enhance: reduce 1x copy while retrieving data from growing segment (#28323) (#28475) 
We first fill the data into a vector and then copy it into the proto,
for some types (exclude variable-length types and int8, int16),
data could be directly copied into the proto.

Sealed segment has been optimized in
https://github.com/milvus-io/milvus/pull/28106.

pr: #28323

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-17 14:40:26 +08:00
Enwei Jiao
9997eeb4bb
fix: Fix rocksdb compile failed (#28486) 
pr: #28484
/kind improvement

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-11-16 16:38:20 +08:00
yah01
c92e6b654f
Reduce 1x memory copy for retrieving data (#28106) (#28217) 
/kind improvement
Before this, while retrieving data (query/search), we first copy the
data into a fixed vector, and then copy data from this into the proto
field.
Now we can directly copy the data into the proto field.

This optimization can't be done with int8, int16 due to the proto
doesn't provide the two types, we store them in int32s

Also, this can't be done with variable length field like string, JSON,
see https://github.com/protocolbuffers/protobuf/issues/10866. I tried
but it seems proto doesn't guarantee the memory layout as we expected,
it crashed
pr: #28106

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-15 10:50:20 +08:00
cai.zhang
c8cab4dfc3
No hit when the index exceeds the array length (#28302) 
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2023-11-09 22:10:32 +08:00
congqixia
83b7f05fe1
Sync proto generated files to avoid --dirty tag (#28316) 
The proto generated files is out of sync for image build env
This will cause --dirty="-dev" tag in Milvus build version
Sync changed files to avoid this case

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-09 21:22:21 +08:00
yah01
0497bbf05d
Handle exception while loading (#28306) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-09 19:28:35 +08:00
PowderLi
39c24fe07b
print azure sdk log (#28241) 
Signed-off-by: PowderLi <min.li@zilliz.com>
 2023-11-09 10:08:20 +08:00
zhagnlu
99350b1be4
Force set aliyun use_virtual_host to true for all (#28237) 
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2023-11-08 11:10:21 +08:00
foxspy
1944e955cb
publish cardinal for milvus 2.3.3 (#28254) 
Signed-off-by: xianliang <xianliang.li@zilliz.com>
 2023-11-08 10:56:20 +08:00
Gao
cc3dd6eda4
Set Knowhere version to v2.2.2 (#28252) 
Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2023-11-08 08:46:19 +08:00
cai.zhang
19230db7f5
Fix bug for constructing ArrayView with fixed-length type (#28186) 
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2023-11-07 23:36:21 +08:00
yihao.dai
d0bea0eb0a
Get vector concurrently (#28119) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-11-07 19:38:21 +08:00
PowderLi
300c1f5529
reduce useless ObjectExists (#28157) 
replace ListBlobs() with GetProperties()
unified style std::string& / char*
config azure requestTimeoutMs

Signed-off-by: PowderLi <min.li@zilliz.com>
 2023-11-06 22:14:21 +08:00
PowderLi
43db873770
add a custom http header: Accept-Type-Allow-Int64 (#28125) 
Signed-off-by: PowderLi <min.li@zilliz.com>
 2023-11-06 16:02:18 +08:00
yah01
5c444218a2
Limit max thread num for pool (#28018) (#28115) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-06 10:50:17 +08:00
cai.zhang
5c06f293f5
Refine code for fixed-length types array (#28109) 
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2023-11-05 18:00:17 +08:00
yah01
218a988941
Increase the ChunkManager request timeout (#28015) (#28154) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-04 10:26:17 +08:00
cai.zhang
355fcc350d
Check length before comparing strings (#28112) 
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2023-11-04 10:04:29 +08:00
yah01
0ab13c935a
Fix QueryNode panic while upgrading (#28034) (#28114) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-03 17:24:18 +08:00
yah01
e22e985390
Fix delete records timestamp may be reordered (#27941) (#28113) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-03 15:44:16 +08:00
yah01
a36a57bfec
Enable mmap for IVF index (#27904) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-10-26 01:48:12 +08:00
Enwei Jiao
4faba61647
Upgrade conan to 1.61.0 (#27870) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-10-24 17:50:10 +08:00
sre-ci-robot
8c605ca858
Update knowhere commit (#27865) 
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 2023-10-24 02:34:08 +08:00
zhagnlu
6060dd7ea8
Add chunk manager request timeout (#27692) 
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2023-10-23 20:08:08 +08:00
SimFG
9b0ecbdca7
Support to replicate the mq message (#27240) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-10-20 14:26:09 +08:00
sre-ci-robot
b6e07d6fe3
Update knowhere commit (#27812) 
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 2023-10-20 03:54:09 +08:00
Gao
9dd369dd99
Update knowhere version to v2.2.2 (#27810) 
Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2023-10-19 21:34:07 +08:00
sre-ci-robot
6b79d2b7d6
Update knowhere commit (#27752) 
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 2023-10-18 07:14:09 +08:00
Alexander Guzhva
fbaec4c1a2
Fix non-removed 2 GB temporary file in test_utils.cpp (#27651) 
Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>
 2023-10-18 02:24:09 +08:00