1162 Commits

Author SHA1 Message Date
yah01
02c5a649cf
enhance: store system fields in segcore (#28524)
we need the system fields info for some usacase
fix: #28523

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-11-21 09:28:22 +08:00
PowderLi
a1c505dbd5
add internal storage metrics (#28278)
/kind improvement
issue: #28277

Signed-off-by: PowderLi <min.li@zilliz.com>
2023-11-19 17:22:25 +08:00
yah01
f7d2ab6677
enhance: reduce 1x copy for variable length field while retrieving (#28345)
- Reduce 1x copy for varchar/string/JSON/array types while retrieving
- Reduce 1x copy for int8/int16 while retrieving

Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-11-15 18:08:20 +08:00
MrPresent-Han
836f300536
support skip-index based on chunk-metrics to accelerate expr filter(#27925) (#28297)
related: #27925

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2023-11-15 11:20:19 +08:00
Xu Tong
8ec85f5f4c
Add template for VectorMemIndex (#28324)
Signed-off-by: Writer-X <1256866856@qq.com>
2023-11-11 13:20:22 +08:00
yah01
267c67dfee
enhance: reduce 1x copy while retrieving data from growing segment (#28323)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-11-10 15:44:22 +08:00
zhagnlu
3920bbc55f
Force set aliyun use_virtual_host to true for all (#28158)
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2023-11-10 10:32:20 +08:00
cai.zhang
01c5484817
No hit when the index exceeds the array length (#28301)
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2023-11-10 10:04:19 +08:00
yah01
30847cad3e
Handle exception while loading (#28304)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-11-09 17:59:12 +08:00
PowderLi
9f9726f8b7
print azure sdk log (#28240)
Signed-off-by: PowderLi <min.li@zilliz.com>
2023-11-08 17:50:18 +08:00
cai.zhang
2b5f632fa4
Fix bug for constructing ArrayView with fixed-length type (#28185)
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2023-11-07 23:38:21 +08:00
PowderLi
7bb0fa9c70
reduce useless ObjectExists (#28156)
replace ListBlobs() with GetProperties()
unified style std::string& / char*
config azure requestTimeoutMs

Signed-off-by: PowderLi <min.li@zilliz.com>
2023-11-07 16:32:20 +08:00
yah01
863e26969a
Reduce 1x memory copy for retrieving data (#28106)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-11-06 17:48:19 +08:00
cai.zhang
8011054a2a
Check length before comparing strings (#28110)
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2023-11-04 10:04:29 +08:00
congqixia
501cba90ce
Use IVFFLAT_CC instead of IVFFLAT for sealed segment tmp index (#28164)
See also #28152

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-11-03 23:50:15 +08:00
cai.zhang
fc2df9514f
Refine code for fixed-length types array (#28108)
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2023-11-03 00:40:14 +08:00
yah01
dc89730a50
Support collection-level mmap control (#26901)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-11-02 23:52:16 +08:00
Enwei Jiao
f8dd589755
Refactor collection's cgo call (#28055)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-11-02 13:02:13 +08:00
yihao.dai
873b29e226
Fix unstable cpp ut (#28083)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-11-02 00:58:16 +08:00
MrPresent-Han
854c0e8a16
fix precision for segcore reduce(#27325) (#28062)
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2023-11-01 21:52:13 +08:00
PowderLi
0c0f012e03
add a custom http header: Accept-Type-Allow-Int64 (#27901)
Signed-off-by: PowderLi <min.li@zilliz.com>
2023-11-01 11:42:16 +08:00
Enwei Jiao
8ae9c947ae
Use OpenDAL to access object store (#25642)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-11-01 09:00:14 +08:00
cqy123456
4fbe3c9142
replace loaded binlog with binlog index for search performance (#27673)
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2023-11-01 02:20:15 +08:00
yah01
1b9b8a31a8
Fix QueryNode panic while upgrading (#28034)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-10-31 15:50:13 +08:00
yah01
ab6dbf7659
Limit max thread num for pool (#28018)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-10-31 13:00:13 +08:00
yah01
2af46d7333
Increase the ChunkManager request timeout (#28015)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-10-31 09:06:13 +08:00
yihao.dai
ab6b0103a3
Get vector concurrently (#27838)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-10-30 15:44:12 +08:00
yah01
f212158d61
Fix delete records timestamp may be reordered (#27941)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-10-27 10:24:10 +08:00
Enwei Jiao
4a33391b8f
rename createindex (#27903)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-10-27 10:12:14 +08:00
yah01
a36a57bfec
Enable mmap for IVF index (#27904)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-10-26 01:48:12 +08:00
zhagnlu
6060dd7ea8
Add chunk manager request timeout (#27692)
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2023-10-23 20:08:08 +08:00
SimFG
9b0ecbdca7
Support to replicate the mq message (#27240)
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-10-20 14:26:09 +08:00
Enwei Jiao
e98e56f75d
Fix SIGSEGV if run within gdb (#27736)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-10-18 02:16:11 +08:00
smellthemoon
5785756d31
Add some log to print size of search result (#27601)
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2023-10-16 10:48:08 +08:00
zhagnlu
c5ea31316d
Fix glog thread-safe bug for standalone mode (#27703)
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2023-10-14 12:10:05 +08:00
yihao.dai
c882e9c87b
Fix compatibility of Flush (#27543)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-10-13 18:01:39 +08:00
Xiaofan
d83869aaeb
Refine minio chunks manager (#27510)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-10-13 14:15:35 +08:00
PowderLi
8d3069b1db
update openssl to 3.1.2 (#27399)
deal with root path's normalization

Signed-off-by: PowderLi <min.li@zilliz.com>
2023-10-08 19:17:31 +08:00
yihao.dai
106c17f304
Make read ahead policy in ChunkCache configurable (#27291)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-09-28 15:47:27 +08:00
Enwei Jiao
b80a3e19d3
Add code for PanicInfo (#27364)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-09-27 12:01:28 +08:00
foxspy
5db4a0489e
dynamic index version control (#27335)
Co-authored-by: longjiquan <jiquan.long@zilliz.com>
2023-09-25 21:39:27 +08:00
foxspy
fa033e586a
disable growing index for flat (#27309)
Signed-off-by: xianliang <xianliang.li@zilliz.com>
2023-09-22 14:19:24 +08:00
foxspy
370b6fde58
milvus support multi index engine (#27178)
Co-authored-by: longjiquan <jiquan.long@zilliz.com>
2023-09-22 09:59:26 +08:00
yah01
93e2eb78c9
Delete only if primary keys exist (#25292)
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-09-20 19:03:25 +08:00
cai.zhang
a362bb1457
Support array datatype (#26369)
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-09-19 14:23:23 +08:00
PowderLi
4feb3fa7c6
support azure (#26398)
Signed-off-by: PowderLi <min.li@zilliz.com>
2023-09-19 10:01:23 +08:00
yah01
0459a662e4
use MADV_WILLNEED for scalar column data (#27170)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-09-18 18:05:22 +08:00
yihao.dai
060d3563ba
Fix compile error at core/storage (#27121)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-09-15 14:41:21 +08:00
yihao.dai
bb6711f28c
Add ChunkCache: support get vector from storage (#26142)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-09-15 10:21:20 +08:00
foxspy
18cbe26876
fix growing hasRawData (#27105)
Signed-off-by: xianliang <xianliang.li@zilliz.com>
2023-09-14 21:11:18 +08:00