Cherry pick from master
pr: #37076
Related #36887
DirectFoward streaming delete will cause memory usage explode if the
segments number was large. This PR add batching delete API and using it
for direct forward implementation.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Cherry-pick from master
pr: #37082
Related to #35303
Delta data is not needed when using `RemoteLoad` l0 forward policy. By
skipping load delta data, memory pressure could be eased if l0 segment
size/number is large.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Cherry pick from master
pr: #37081
Related to #35303
This PR add metrics for querynode delegator delete buffer information,
which is related to dml quota logic.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Cherry-pick from master
pr: #36899
Related to #36887
Forward delete to L0 segment will return error and mark l0 segment
offline causing delegator unserviceable
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Cherry-pick from master
pr: #36888
Start position & level info is missing for growing segment loaded in
watch dml channel operation.
Level is important for metrics and start position is crucial for growing
exclude logic.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #36464
pr: #36466
This PR enable balance on querynode with different mem capacity, for
query node which has more mem capactity will be assigned more records,
and query node with the largest difference between assignedScore and
currentScore will have a higher priority to carry the new segment.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
issue: #36257
pr: #36258
during syncTargetVersion, sealed segment should be excluded, to avoid
it's growing segment be conusmed from stream again.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Cherry-pick from master
pr: #36189
Related to #35303
This PR add a param item to support change l0 forward behavior from bf
filtering and forward to remote load.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Cherry-pick from master
pr: #35507
L0 segments now do not contain insert data and may cause confusion for
query hook optimizer if counted as sealed segment number.
This PR add segment level flag in segment entry and exclude L0 segments
while get readable segment snapshot
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #35087
pr: #35197
after qc restarts, and target is not ready yet, if dist_handler try to
update segment dist, it will set legacy level to l0 segment, which may
cause l0 segment be moved to other node, cause search/query failed.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Cherry-pick from master
pr: #35118
Grow slice & map.growWork may cause a lot when segment number is large
for big K query. This PR pre-allocate space for reduce methods to avoid
this cost.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Cherry pick from master
pr: #34767
Related to: #33235
THe querynode pipeline will make map & call ProcessInsert when there is
no write messages. So querynodes will have high CPU usage even when
there is no workload.
This PR check msg length before composing data struct and calling method
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Cherry pick from master
pr: #34594
github.com/gogo/protobuf is deprecated and could be error prune after
upgrade protobuf message to v2.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #34500
pr: #34501
cause the sort in `GetLevel0Deletions` will broken the corresponed order
between pks and tss, then the pks and tss will be sorted in
segment.Delete() interface.
This PR remove this uncessary and incorrect sort progress to avoid query
may return deleted records.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
