Related to #43660
This patch reduces the unwanted offset&ts entries having same timestamp
of delete record. Under large amount of upsert, this false hit could
increase large amount of memory usage while applying delete.
The next step could be passing a callback to `search_pk_func_` to handle
hit entry streamingly.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Related to #43655
This patch add a padding when writing mmap file for ScalarSortedIndex in
case of mmap falure due to 0 mmap length.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Related to #43626
Similar to previous pr #43321, null bitmap could be dislocated if the
bitset ptr does not count the offset of array
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Related to #43592
When delete records are large, search pk one by one will result into
many `Pincells` call which creates lots of futures.
This patch make search pk execute in batch to reduce this cost.
Also add `GetAllChunks` API to utilize `PinAllCells` to reduce pins.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Related to #43584
There might be concurrent calls on `translator.get_cells`. The channel
cannot be shared among these calls, otherwise the logic will break.
This patch create new channel for each `get_cells` invocation in case of
data race.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Related to #43584
Directly accessing `fields_` in `get_raw_data` may have race if load vec
index happens concurrently during getting raw data.
This PR make `bulk_subscript` hold shared_ptr of field column prevent
field column being release during reading it.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
relate: https://github.com/milvus-io/milvus/issues/42792
Add char group tokenizer which support use costum char group or use some
build-in char group as delimiters.
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
Related to #43584
When `LoadWithStrategy` throw exception, the ex was wrapped in the
returned future. If the future is not handled, this exception would be
ignored.
This patch add `future.get()` to get exception if any.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #43261
`promise->setValue(folly::Unit());` may run callbacks inline and some of
them may attempt to grab `mtx_`. So we should not call
`promise->setValue(folly::Unit());` while holding the lock.
---------
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
Related to #43261
Read error with catched in `LoadWithStrategy`. Caller could not detect
read failure when some error occurred. This patch make
`LoadWithStrategy` throw ex instead of swallowing it.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #43088
issue: #43038
The current loading process:
* When loading an index, we first download the index files into a list
of buffers, say A
* then constructing(copying) them into a vector of FieldDatas(each file
is a FieldData), say B
* assembles them together as a huge BinarySet, say C
* lastly, copy into the actual index data structure, say D
The problem:
* We can see that, after each step, we don't need the data in previous
step.
* But currently, we release the memory of A, B, C only after we have
finished constructing D
* This leads to a up to 4x peak memory usage comparing with the raw
index size, during the loading process
* This PR allows timely releasing of B after we assembled C. So after
this PR, the peak memory usage during loading will be up to 3x of the
raw index size.
I will create another PR to release A after we created B, that seems
more complicated and need more work.
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
currently we multiplied the requesting size when adding to loading, but
did not do so when estimating projected usage.
issue: #43088
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
Related to #43113
This PR:
- Change member of FieldIndex from `FieldMeta &` to needed `DataType`
and dim member resolving dangling reference after schema change
- Add double check after acquiring lock to reduce multiple assignment
- Change `auto schema` to `auto& schema` to reduce schema copy
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
The root cause of the issue lies in the fact that when a sealed segment
contains multiple row groups, the get_cells function may receive
unordered cids. This can result in row groups being written into
incorrect cells during data retrieval.
Previously, this issue was hard to reproduce because the old Storage V2
writer had a bug that caused it to write row groups larger than 1MB.
These large row groups could lead to uncontrolled memory usage and
eventually an OOM (Out of Memory) error. Additionally, compaction
typically produced a single large row group, which avoided the incorrect
cell-filling issue during query execution.
related: https://github.com/milvus-io/milvus/issues/43388,
https://github.com/milvus-io/milvus/issues/43372,
https://github.com/milvus-io/milvus/issues/43464, #43446, #43453
---------
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
