admin/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2026-01-07 19:31:51 +08:00
Code Issues Packages Projects Releases Wiki Activity
19,457 Commits 137 Branches 188 Tags
Commit Graph

32 Commits

Author SHA1 Message Date
Cai Yudong
246586be27
enhance: Unify data type check APIs under internal/core (#31800) 
Issue: #22837 

Move and rename following C++ APIs:
datatype_sizeof() ==> GetDataTypeSize()
datatype_name() ==> GetDataTypeName()
datatype_is_vector() / IsVectorType() ==> IsVectorDataType()
datatype_is_variable() ==> IsVariableDataType()
datatype_is_sparse_vector() ==> IsSparseFloatVectorDataType()
datatype_is_string() / IsString() ==> IsDataTypeString()
datatype_is_floating() / IsFloat() ==> IsDataTypeFloat()
datatype_is_binary() ==> IsDataTypeBinary()
datatype_is_json() ==> IsDataTypeJson()
datatype_is_array() ==> IsDataTypeArray()
datatype_is_variable() == IsDataTypeVariable()
datatype_is_integer() / IsIntegral() ==> IsDataTypeInteger()

Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
 2024-04-02 19:15:14 +08:00
Buqian Zheng
070dfc77bf
feat: [Sparse Float Vector] segcore basics and index building (#30357) 
This commit adds sparse float vector support to segcore with the
following:

1. data type enum declarations
2. Adds corresponding data structures for handling sparse float vectors
in various scenarios, including:
* FieldData as a bridge between the binlog and the in memory data
structures
* mmap::Column as the in memory representation of a sparse float vector
column of a sealed segment;
* ConcurrentVector as the in memory representation of a sparse float
vector of a growing segment which supports inserts.
3. Adds logic in payload reader/writer to serialize/deserialize from/to
binlog
4. Adds the ability to allow the index node to build sparse float vector
index
5. Adds the ability to allow the query node to build growing index for
growing segment and temp index for sealed segment without index built

This commit also includes some code cleanness, comment improvement, and
some unit tests for sparse vector.

https://github.com/milvus-io/milvus/issues/29419

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
 2024-03-11 14:45:02 +08:00
zhagnlu
976b6fc0e4
enhance: change opendal as compile configurable (#30384) 
#30373

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-02-20 19:16:52 +08:00
cai.zhang
6cf2f09b60
feat: Support tencent cloud object storage for milvus (#30163) 
issue: #30162

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-01-23 11:28:56 +08:00
yah01
a77693aa19
enhance: convert the GetObject util to async (#30166) 
This makes it much easier to use

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2024-01-22 19:20:57 +08:00
Bingyi Sun
e1258b8cad
feat: integrate storagev2 into loading segment (#29336) 
issue: #29335

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2024-01-12 18:10:51 +08:00
Xu Tong
e429965f32
Add float16 approve for multi-type part (#28427) 
issue：https://github.com/milvus-io/milvus/issues/22837

Add bfloat16 vector, add the index part of float16 vector.

Signed-off-by: Writer-X <1256866856@qq.com>
 2024-01-11 15:48:51 +08:00
yah01
0ae90443ba
enhance: fill missed info for segcore error (#29610) 
- fill missed error info
- format the error message directly

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2024-01-04 17:54:46 +08:00
Bingyi Sun
89b208d27a
enhance: Fix format message (#29159) 
Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2023-12-20 09:30:44 +08:00
zhagnlu
a602171d06
enhance: Refactor runtime and expr framework (#28166) 
#28165

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2023-12-18 12:04:42 +08:00
Bingyi Sun
36f69ea031
feat: integrate storagev2 in building index of segcore (#28768) 
issue: https://github.com/milvus-io/milvus/issues/28655

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2023-12-05 16:48:54 +08:00
Enwei Jiao
8ae9c947ae
Use OpenDAL to access object store (#25642) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-11-01 09:00:14 +08:00
yihao.dai
106c17f304
Make read ahead policy in ChunkCache configurable (#27291) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-09-28 15:47:27 +08:00
Enwei Jiao
b80a3e19d3
Add code for PanicInfo (#27364) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-09-27 12:01:28 +08:00
foxspy
370b6fde58
milvus support multi index engine (#27178) 
Co-authored-by: longjiquan <jiquan.long@zilliz.com>
 2023-09-22 09:59:26 +08:00
cai.zhang
a362bb1457
Support array datatype (#26369) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-09-19 14:23:23 +08:00
PowderLi
4feb3fa7c6
support azure (#26398) 
Signed-off-by: PowderLi <min.li@zilliz.com>
 2023-09-19 10:01:23 +08:00
yihao.dai
bb6711f28c
Add ChunkCache: support get vector from storage (#26142) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-09-15 10:21:20 +08:00
Enwei Jiao
0afdfdb9af
Remove other Exceptions, keeps SegcoreError only (#27017) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-09-14 14:05:20 +08:00
Enwei Jiao
c3f15c6b95
Refactor duplicate error class into one place (#26985) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-09-11 20:43:17 +08:00
Xu Tong
9166011c4a
Add float16 vector (#25852) 
Signed-off-by: Writer-X <1256866856@qq.com>
 2023-09-08 10:03:16 +08:00
xige-16
1055c90456
Add default retrieve limit (#24782) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2023-08-10 14:11:15 +08:00
MrPresent-Han
5634ba777d
add new threadpool with various priority to avoid deadlock(#25781) (#26028) 
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-08-03 09:31:07 +08:00
yah01
227d2c8b3a
Reduce loading index memory usage (#25698) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-07-19 14:02:57 +08:00
yah01
dd5f896dc8
Load batch by batch (#25212) 
This will significantly reduce the memory usage while loading
- 1x memory usage and MBs overhead for buffer (memory mode)
- only MBs overhead for buffer (mmap mode)

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-07-06 13:58:27 +08:00
xige-16
04082b3de2
Migrate the ability to upload and download binlog to cpp (#22984) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2023-06-25 14:38:44 +08:00
Enwei Jiao
967a97b9bd
Support json & array types (#23408) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
Co-authored-by: yah01 <yang.cen@zilliz.com>
 2023-04-20 11:32:31 +08:00
xige-16
9aa99aedbb
[Cherry-Pick] Remove arrow uasge in FieldData (#22726) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2023-03-20 10:41:56 +08:00
yah01
bdd6bc7695
Re-format cpp code (#22513) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-03-02 15:55:49 +08:00
xige-16
8c9c1672ae
Assign different storage config for indexes (#19517) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-10-14 14:45:23 +08:00
xige-16
428840178c
Support diskann index for vector field (#19093) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-09-21 20:16:51 +08:00
xige-16
4de1bfe5bc
Add cpp data codec (#18538) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
Co-authored-by: zhagnlu lu.zhang@zilliz.com

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-09-09 22:12:34 +08:00