See also #40558
Related to #35303 & #38066 as well
This PR:
- Add `BufferedForward` to limit memory usage forwarding stream delete
- Add `UseLoad` flag to determine `Delete` shall use `segment.Delete` or
`segment.LoadDelta`
- Fix delegator accidentally use always true candidate while load
streaming delta
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #39551
This PR remove querycoord's scheduling of l0 segments:
- only load l0 segment when watch channel
- only release l0 segment when release channel or sync data distribution
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
The level zero mutex could be remove since all operations are guarded by
segment manager mutex
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Related to #35303
Delta data is not needed when using `RemoteLoad` l0 forward policy. By
skipping load delta data, memory pressure could be eased if l0 segment
size/number is large.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Relatedt #36887
DirectFoward streaming delete will cause memory usage explode if the
segments number was large. This PR add batching delete API and using it
for direct forward implementation.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Related to #36887
Forward delete to L0 segment will return error and mark l0 segment
offline causing delegator unserviceable
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
