admin/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2025-12-07 17:48:29 +08:00
Code Issues Packages Projects Releases Wiki Activity
21,658 Commits 133 Branches 183 Tags
Commit Graph

40 Commits

Author SHA1 Message Date
Gao
75d7978a18
enhance: pass partition key scalar info if enable for vector mem index (#39123) 
issue: #34332

---------

Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2025-01-16 14:33:03 +08:00
Zhen Ye
3e788f0fbd
enhance: record memory size (uncompressed) item for index (#38770) 
issue: #38715

- Current milvus use a serialized index size(compressed) for estimate
resource for loading.
- Add a new field `MemSize` (before compressing) for index to estimate
resource.

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 2025-01-14 10:33:06 +08:00
jaime
2ff3765058
enhance: catch std::stoi exception and improve error msg (#36267) 
issue: #36255

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-09-14 16:17:08 +08:00
Jiquan Long
89bf226f0b
feat: support keyword text match (#35923) 
fix: #35922

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2024-09-10 15:11:08 +08:00
foxspy
9da86529a7
enhance: Add disk filemananger parallel load control to reduce the memory consumption (#35281) 
issue: #35280 
add parallel control to limit the memory consumption during index file
loading

Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
 2024-09-03 18:01:03 +08:00
smellthemoon
80dbe87759
enhance: support null value in index (#35238) 
#31728

---------

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2024-08-16 15:30:54 +08:00
Jiquan Long
91df03afe8
feat: put inverted index into ram (#35222) 
fix: https://github.com/milvus-io/milvus/issues/35224

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2024-08-06 11:54:16 +08:00
zhenshan.cao
aa247f192d
enhance: remove unused code for StorageV2 (#35132) 
issue: https://github.com/milvus-io/milvus/issues/34168

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2024-08-01 12:08:13 +08:00
smellthemoon
5616b7e8d2
enhance: support null in c data_datacodec and load null value (#32183) 
1. support read and write null in segcore
    will store valid_data(use uint8_t type to save memory) in fieldData.
2. support load null
binlog reader read and write data into column(sealed segment),
insertRecord(growing segment). In sealed segment, store valid_data
directly. In growing segment, considering prior implementation and easy
code reading, it covert uint8_t to fbvector<bool>, which may optimize in
future.
3.  retrieve valid_data.
    parse valid_data in search/query.
#31728

---------

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2024-07-23 16:07:51 +08:00
Patrick Weizhi Xu
b961767005
enhance: support integral type for MV and skip MV if there is only one category (#33161) 
issue: #29892

---------

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
 2024-06-24 10:20:01 +08:00
Gao
0d20303e54
fix: fix binary vector data size (#33750) 
issue: https://github.com/milvus-io/milvus/issues/22837

- fix byte size wrong for binary vectors
- fix the expect/actual error msg

Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2024-06-18 21:39:59 +08:00
cqy123456
32f685ff12
enhance: growing segment support mmap (#32633) 
issue: https://github.com/milvus-io/milvus/issues/32984

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2024-06-18 14:42:00 +08:00
Buqian Zheng
47b04ea167
enhance: support sparse cardinal hnsw index (#33656) 
issue: #29419

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
 2024-06-12 16:57:55 +08:00
cai.zhang
27cc9f2630
enhance: Support analyze data (#33651) 
issue: #30633

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Co-authored-by: chasingegg <chao.gao@zilliz.com>
 2024-06-06 17:37:51 +08:00
cqy123456
703fc73f71
enhance: disk index support binary vector (#33631) 
issue:https://github.com/milvus-io/milvus/issues/22837
related https://github.com/milvus-io/milvus/pull/33575

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2024-06-05 19:37:57 +08:00
cqy123456
aba4993c6c
fix: fix some fp16/bf16 code miss in segcore. (#31771) 
issue：https://github.com/milvus-io/milvus/issues/22837

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2024-04-07 14:13:16 +08:00
zhagnlu
976b6fc0e4
enhance: change opendal as compile configurable (#30384) 
#30373

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-02-20 19:16:52 +08:00
yah01
a27c0e86fd
enhance: reduce many I/O operations while loading disk index (#30189) 
before this, every time writting the index chunk data into the disk,
there are 4 I/O operations:
- open the file
- seek to the offset
- write the data
- close the file

this optimized this to open only once and continiously write all data.

This also makes it concurrent to load the files from object storage

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2024-01-25 15:23:02 +08:00
Patrick Weizhi Xu
0907d76253
enhance: pass partition key scalar info if enabled when build vector index (#29931) 
issue: #29892 

Pass optional scalar IVF offsets to Cardinal

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
 2024-01-24 00:04:55 +08:00
yah01
a77693aa19
enhance: convert the GetObject util to async (#30166) 
This makes it much easier to use

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2024-01-22 19:20:57 +08:00
Bingyi Sun
e1258b8cad
feat: integrate storagev2 into loading segment (#29336) 
issue: #29335

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2024-01-12 18:10:51 +08:00
yah01
aef483806d
enhance: improve the segcore logs (#29372) 
- remove the streaming logging
- refine existing logs

fix #29366

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-12-23 21:52:43 +08:00
Bingyi Sun
36f69ea031
feat: integrate storagev2 in building index of segcore (#28768) 
issue: https://github.com/milvus-io/milvus/issues/28655

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2023-12-05 16:48:54 +08:00
foxspy
370b6fde58
milvus support multi index engine (#27178) 
Co-authored-by: longjiquan <jiquan.long@zilliz.com>
 2023-09-22 09:59:26 +08:00
Enwei Jiao
0afdfdb9af
Remove other Exceptions, keeps SegcoreError only (#27017) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-09-14 14:05:20 +08:00
MrPresent-Han
5634ba777d
add new threadpool with various priority to avoid deadlock(#25781) (#26028) 
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-08-03 09:31:07 +08:00
xige-16
04082b3de2
Migrate the ability to upload and download binlog to cpp (#22984) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2023-06-25 14:38:44 +08:00
yah01
247f117096
Refine segcore code style (#24999) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-06-20 14:16:46 +08:00
cai.zhang
c80d8f82ce
Reduce memory usage for diskann index (#23600) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-05-06 10:32:39 +08:00
Enwei Jiao
ef57ba3bac
Remove easylogging and replace by GLog (#22979) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-03-27 18:22:02 +08:00
xige-16
9aa99aedbb
[Cherry-Pick] Remove arrow uasge in FieldData (#22726) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2023-03-20 10:41:56 +08:00
yah01
bdd6bc7695
Re-format cpp code (#22513) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-03-02 15:55:49 +08:00
cai.zhang
5cae49e28d
Store index files in slices to reduce memory peaks (#22369) (#22455) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-03-01 19:43:47 +08:00
xige-16
15badd0263
Fix cache disk index error (#20419) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-11-21 17:45:23 +08:00
zhagnlu
cb2591d1fe
Parallel operations of disk-ann file manager (#20120) (#20122) 
Signed-off-by: zhagnlu <lu.zhang@zilliz.com>

Signed-off-by: zhagnlu <lu.zhang@zilliz.com>
Co-authored-by: zhagnlu <lu.zhang@zilliz.com>
 2022-11-03 14:39:40 +08:00
zhagnlu
02a8a07322
Fix BucketExists bug in minio chunk manager (#19548) (#19854) 
Signed-off-by: zhagnlu <lu.zhang@zilliz.com>

Signed-off-by: zhagnlu <lu.zhang@zilliz.com>
Co-authored-by: zhagnlu <lu.zhang@zilliz.com>
 2022-10-20 10:23:27 +08:00
xige-16
158787811e
Move assemble/disassemble func to core (#19420) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-10-16 21:17:25 +08:00
xige-16
8c9c1672ae
Assign different storage config for indexes (#19517) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-10-14 14:45:23 +08:00
xige-16
49daee2e17
Fix lost disk index data (#19636) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-10-09 14:36:58 +08:00
xige-16
428840178c
Support diskann index for vector field (#19093) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-09-21 20:16:51 +08:00