yah01
1b7f1d7067
enhance: mmap data corrupted after seal the column ( #29422 )
...
this bug was introduced in recent changes
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-12-23 15:20:43 +08:00
yah01
7a2374e698
enhance: reduce the memory usage of variable length data ( #29387 )
...
add all loading data into a buffer and then copy them into the a
fit-in-size memory
---------
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-12-21 18:02:42 +08:00
yah01
04b2518ae7
enhance: fix the incorrect init parameter ( #29357 )
...
as the `driver_` field is not used so this doesn't matter for now
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-12-20 20:50:43 +08:00
yah01
8f89e9cf75
enhance: remove all unnecessary string formatting ( #29323 )
...
done by two regex expressions:
- `PanicInfo\((.+),[. \n]+fmt::format\(([.\s\S]+?)\)\)`
- `AssertInfo\((.+),[. \n]+fmt::format\(([.\s\S]+?)\)\)`
related: #28811
---------
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-12-20 10:04:43 +08:00
zhagnlu
a602171d06
enhance: Refactor runtime and expr framework ( #28166 )
...
#28165
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2023-12-18 12:04:42 +08:00
yah01
342635ed61
enhance: enable assert method to format arguments ( #28812 )
...
for now the assert method in segcore could accept a string information,
too many codes don't print the value they assert.
make it happy
related #28811
---------
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-12-01 18:04:33 +08:00
yah01
c96d07682e
enhance: enhance the accuracy of memory usage ( #28554 )
...
before this, Milvus use container/system's memory info to get the memory
usage, which could be inaccurate.
we allocates the memory by private anon mmap,
then `rss - shared` would be the accurate memory usage
resolve #28553
---------
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-11-23 15:12:23 +08:00
Enwei Jiao
b80a3e19d3
Add code for PanicInfo ( #27364 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-09-27 12:01:28 +08:00
cai.zhang
a362bb1457
Support array datatype ( #26369 )
...
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-09-19 14:23:23 +08:00
yah01
0459a662e4
use MADV_WILLNEED for scalar column data ( #27170 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-09-18 18:05:22 +08:00
yihao.dai
bb6711f28c
Add ChunkCache: support get vector from storage ( #26142 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-09-15 10:21:20 +08:00
Enwei Jiao
c3f15c6b95
Refactor duplicate error class into one place ( #26985 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-09-11 20:43:17 +08:00
yah01
9605c03c3c
Fix the number of rows of column not correct ( #26347 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-08-16 13:35:33 +08:00
yah01
127c23d999
Check data consistency after loading ( #26312 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-08-14 09:01:32 +08:00
yah01
a173486d2e
Fix calculation of memory usage prediction for mmap mode ( #26264 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-08-12 17:19:31 +08:00
yah01
300fef446b
Enable mmap for vector index ( #25877 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-08-10 13:59:15 +08:00
yah01
dd5f896dc8
Load batch by batch ( #25212 )
...
This will significantly reduce the memory usage while loading
- 1x memory usage and MBs overhead for buffer (memory mode)
- only MBs overhead for buffer (mmap mode)
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-07-06 13:58:27 +08:00
yah01
cb4b88d5cf
Refactor the column type ( #25147 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-06-27 19:50:45 +08:00
xige-16
04082b3de2
Migrate the ability to upload and download binlog to cpp ( #22984 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2023-06-25 14:38:44 +08:00
