11 Commits

Author SHA1 Message Date
yihao.dai
f2b83d316b
enhance: Support memory mode chunk cache (#35347)
Chunk cache supports loading raw vectors into memory.

issue: https://github.com/milvus-io/milvus/issues/35273

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-08-25 15:42:58 +08:00
congqixia
14e827dc6c
fix: Implement singleflight for segcore ChunkCache (#34250)
See also #34249

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-01 11:46:06 +08:00
cqy123456
dc4437ff82
enhance: use segment id and type to register in MmapChunkManager and opt malloc in variableChunk (#33993)
issue: https://github.com/milvus-io/milvus/issues/32984

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-06-20 17:42:02 +08:00
cqy123456
32f685ff12
enhance: growing segment support mmap (#32633)
issue: https://github.com/milvus-io/milvus/issues/32984

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-06-18 14:42:00 +08:00
congqixia
18c351efa6
fix: Prevent ChunkCache use absolute path in All-in-one mode (#30666)
See also #30651

Append operator of `std::filesystem::path` will replace whole path when
the param of "/" operation is an absolute path.

In "All-in-one" mode, this shall cause ChunkCache removing the original
vector data file when building chunk cache during/after load procedure.

This PR changes the ChunkCache path generation logic to a separate
function in which will check whether the file path is absolute or not.
If the file path is absolute, it removes the root path prefix and return
concatenated file path.

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-02-19 20:58:51 +08:00
yihao.dai
c02fb64ad6
enhance: Allows proactive warming up of chunk cache (#30182)
Allows proactive warming up of chunk cache. Original vector data will be
asynchronously loaded into the chunk cache during the load process. It
has the potential to significantly reduce query/search latency for a
certain duration after the load, albeit with a concurrent increase in
disk usage.

issue: https://github.com/milvus-io/milvus/issues/30181

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-01-25 19:55:39 +08:00
yah01
aef483806d
enhance: improve the segcore logs (#29372)
- remove the streaming logging
- refine existing logs

fix #29366

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-12-23 21:52:43 +08:00
yihao.dai
873b29e226
Fix unstable cpp ut (#28083)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-11-02 00:58:16 +08:00
yihao.dai
ab6b0103a3
Get vector concurrently (#27838)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-10-30 15:44:12 +08:00
yihao.dai
106c17f304
Make read ahead policy in ChunkCache configurable (#27291)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-09-28 15:47:27 +08:00
yihao.dai
bb6711f28c
Add ChunkCache: support get vector from storage (#26142)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-09-15 10:21:20 +08:00