fix#32979
remove l0 cache and build delete pk and ts everytime. this reduce the
memory and also increase the code readability
Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
issue: #32530
when try to match segment bloom filter with pk, we can reuse the hash
locations. This PR maintain the max hash Func, and compute hash location
once for all segment, reuse hash location can speed up bf access
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
`applyDelete` used to be serial for delete entries on each segments.
This PR make it work in parallel with errgroup to improve performance
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #31479#31797
milvus will add released segment to excluded info, and filter out it's
stream data in filter_node. but for data buffered in insert_node's
channel, if it belongs to growing segment which already be released,
then it will all the growing segment back again.
This PR maintain `excluded segments` in delegator, and check excluded
segment before new growing segment.
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
"-1" as `InvalidPartitionID` previously used as All partition place
holder in delete cases. It's confusing and hard to maintain when a const
var has more than one meaning.
This PR add `AllPartitionsID` to replace these usages in delete
scenarios.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
use retry.handle when request is not able to service but don't throw
unrecoverable erros
fix#31323
Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
issue: #30931
- move resource estimate function outside from segment loader.
- add load info and collection to base segment.
- add resource usage method for sealed segment.
Signed-off-by: chyezh <chyezh@outlook.com>
See also #31125
Delegator shall build level zero delete cache from l0 segments belongs
to it. Previously it build cache from all existing level zero segments
in the querynode which may lead to high memory usage and even panicking
when pk types are not matched
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
the growing segments contribute to this metric while inserting and
putting into the manager, but the current impl inserts data before
putting the segments into manager, which leads to double contributions
fix: #29766
Signed-off-by: yah01 <yah2er0ne@outlook.com>
See also #27349
The segment level label in querynode used `Legacy` before segment level
was correctly passed in Load request. Now this attribute is still using
legacy so the metrics does not look right.
This PR add paramter for `NewSegment` and passes corrent values for each
invocation.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
See also #29625
This PR:
- Add a new implemention of `DeleteBuffer`: listDeleteBuffer
- holds cacheBlock slice
- `Put` method append new delete data into last block
- when a block is full, append a new block into the list
- Add `TryDiscard` method for `DeleteBuffer` interface
- For doubleCacheBuffer, do nothing
- For listDeleteBuffer, try to evict "old" blocks, which are blocks
before the first block whose start ts is behind provided ts
- Add checkpoint field for `UpdateVersion` sync action, which shall be
used to discard old cache delete block
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
See also #27515
When Delegator processes delete data, it forwards delete data with only
segment id specified. When two segments has same segment id but one is
growing and the other is sealed, the delete will be applied to both
segments which causes delete data out of order when concurrent load
segment occurs.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
See also #29332
The segment may be released before or during the request when delegator
tries to forward delete request to yet. Currently, these two situation
returns different error code.
In this particular case, `ErrSegmentNotLoaded` and `ErrSegmentNotFound`
shall both be ignored preventing return search service unavailable by
mistake.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Fix logic error from #26926
function `waitSegmentLoadDone` shall return error when context is done
Make delegator control concurrency for each same segment
Related to #26908
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
