74 Commits

Author SHA1 Message Date
PowderLi
0c0f012e03
add a custom http header: Accept-Type-Allow-Int64 (#27901)
Signed-off-by: PowderLi <min.li@zilliz.com>
2023-11-01 11:42:16 +08:00
Enwei Jiao
8ae9c947ae
Use OpenDAL to access object store (#25642)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-11-01 09:00:14 +08:00
yah01
ab6dbf7659
Limit max thread num for pool (#28018)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-10-31 13:00:13 +08:00
yah01
2af46d7333
Increase the ChunkManager request timeout (#28015)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-10-31 09:06:13 +08:00
yihao.dai
ab6b0103a3
Get vector concurrently (#27838)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-10-30 15:44:12 +08:00
zhagnlu
6060dd7ea8
Add chunk manager request timeout (#27692)
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2023-10-23 20:08:08 +08:00
Xiaofan
d83869aaeb
Refine minio chunks manager (#27510)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-10-13 14:15:35 +08:00
PowderLi
8d3069b1db
update openssl to 3.1.2 (#27399)
deal with root path's normalization

Signed-off-by: PowderLi <min.li@zilliz.com>
2023-10-08 19:17:31 +08:00
yihao.dai
106c17f304
Make read ahead policy in ChunkCache configurable (#27291)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-09-28 15:47:27 +08:00
Enwei Jiao
b80a3e19d3
Add code for PanicInfo (#27364)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-09-27 12:01:28 +08:00
foxspy
370b6fde58
milvus support multi index engine (#27178)
Co-authored-by: longjiquan <jiquan.long@zilliz.com>
2023-09-22 09:59:26 +08:00
cai.zhang
a362bb1457
Support array datatype (#26369)
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-09-19 14:23:23 +08:00
PowderLi
4feb3fa7c6
support azure (#26398)
Signed-off-by: PowderLi <min.li@zilliz.com>
2023-09-19 10:01:23 +08:00
yihao.dai
060d3563ba
Fix compile error at core/storage (#27121)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-09-15 14:41:21 +08:00
yihao.dai
bb6711f28c
Add ChunkCache: support get vector from storage (#26142)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-09-15 10:21:20 +08:00
Enwei Jiao
0afdfdb9af
Remove other Exceptions, keeps SegcoreError only (#27017)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-09-14 14:05:20 +08:00
Enwei Jiao
c3f15c6b95
Refactor duplicate error class into one place (#26985)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-09-11 20:43:17 +08:00
Xu Tong
9166011c4a
Add float16 vector (#25852)
Signed-off-by: Writer-X <1256866856@qq.com>
2023-09-08 10:03:16 +08:00
yah01
09218bfd3d
Optimize loading by reduce 1x copy while reading data (#26746)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-09-01 10:15:01 +08:00
zhagnlu
bdc8c507ea
Refactor thread pool to dynamic threads mode (#26407)
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2023-08-18 11:58:19 +08:00
zhagnlu
08547fe1a6
Add errcode to minio chunkmanager exception(#25153) (#25154)
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2023-08-17 20:22:27 +08:00
zhagnlu
411f9ac823
Upgrade minio-go and add region and virtual host config for segcore chunk manager (#26194)
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2023-08-11 10:37:36 +08:00
xige-16
1055c90456
Add default retrieve limit (#24782)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2023-08-10 14:11:15 +08:00
yah01
07f08daf1a
Fix failed to load index due to lost binary (#26135)
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-08-07 14:53:07 +08:00
MrPresent-Han
5634ba777d
add new threadpool with various priority to avoid deadlock(#25781) (#26028)
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2023-08-03 09:31:07 +08:00
zhagnlu
833674c1cb
add glog configurable function and redirect aws log to segcore log (#25664)
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2023-07-27 19:49:02 +08:00
yah01
9618bd9b42
Set channel capacity before consuming it (#25895)
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-07-26 17:35:01 +08:00
shaoyue
7bf39aed08
chunk manager supports using gcs & oss with accessKey (#25233)
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
2023-07-20 15:38:58 +08:00
yah01
227d2c8b3a
Reduce loading index memory usage (#25698)
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-07-19 14:02:57 +08:00
xige-16
94d6cbb238
Fix querynode panic when binlog ts wrong (#25635)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2023-07-18 10:41:20 +08:00
yah01
dd5f896dc8
Load batch by batch (#25212)
This will significantly reduce the memory usage while loading
- 1x memory usage and MBs overhead for buffer (memory mode)
- only MBs overhead for buffer (mmap mode)

Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-07-06 13:58:27 +08:00
cai.zhang
7f3f18e71f
Fix program crash caused by incorrect use of noexcept modifier (#25194) (#25214)
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-06-30 15:08:22 +08:00
xige-16
04082b3de2
Migrate the ability to upload and download binlog to cpp (#22984)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2023-06-25 14:38:44 +08:00
zhagnlu
a79fd1c5a2
Fix GetObject return null value bug in MacOs (#24959) (#25002)
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2023-06-21 13:40:42 +08:00
yah01
247f117096
Refine segcore code style (#24999)
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-06-20 14:16:46 +08:00
Bingyi Sun
cc56efb134
Fix build index performance downgrade (#24651) (#24708)
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
2023-06-19 10:50:47 +08:00
congqixia
cbfbc4e63a
Memset sigaction before using it (#24933)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-06-16 14:08:40 +08:00
congqixia
561021ec5a
Refine sigpipe handling (#24872)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-06-14 10:10:38 +08:00
congqixia
e729b504ab
Use SA_ONSTACK flag for SIGPIPE handler (#24688)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-06-08 15:50:35 +08:00
zhagnlu
0fe8342648
Support google cloud for C++ chunk manager (#21100) (#24418)
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2023-05-29 01:05:28 +08:00
cai.zhang
c80d8f82ce
Reduce memory usage for diskann index (#23600)
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-05-06 10:32:39 +08:00
Enwei Jiao
66a5efeb3d
Fix load failed with DiskANN (#23760)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-04-27 16:56:35 +08:00
Enwei Jiao
967a97b9bd
Support json & array types (#23408)
Signed-off-by: yah01 <yang.cen@zilliz.com>
Co-authored-by: yah01 <yang.cen@zilliz.com>
2023-04-20 11:32:31 +08:00
Enwei Jiao
ef57ba3bac
Remove easylogging and replace by GLog (#22979)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-03-27 18:22:02 +08:00
zhagnlu
7c633e9b9d
Support aliyun OSS for chunk manager (#22663) (#22842)
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2023-03-20 14:45:57 +08:00
xige-16
9aa99aedbb
[Cherry-Pick] Remove arrow uasge in FieldData (#22726)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2023-03-20 10:41:56 +08:00
yah01
a4031da634
Refine string parameters, avoid coping or deref (#22708)
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-03-13 17:53:53 +08:00
yah01
7bc3309918
Replace NULL with nullptr (#22701)
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-03-13 10:35:52 +08:00
yah01
bdd6bc7695
Re-format cpp code (#22513)
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-03-02 15:55:49 +08:00
yah01
4d4bf99098
Fix DiskANN can't recovery after querynode reboots (#22488)
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-03-02 10:17:48 +08:00