wei liu 990a25e51a

fix: Prevent delete records loss during slow segment loading [QueryNodeV2] (#43527) ...

issue: #42884
Fixes an issue where delete records for a segment are lost from the
delete buffer if `load segment` execution on the delegator is too slow,
causing `syncTargetVersion` or other cleanup operations to clear them
prematurely.

Changes include:
- Introduced `Pin` and `Unpin` methods in `DeleteBuffer` interface and
its implementations (`doubleCacheBuffer`, `listDeleteBuffer`).
- Added a `pinnedTimestamps` map to track timestamps protected from
cleanup by specific segments.
- Modified `LoadSegments` in `shardDelegator` to `Pin` relevant segment
delete records before loading and `Unpin` them afterwards.
- Added `isPinned` check in `UnRegister` and `TryDiscard` methods of
`listDeleteBuffer` to skip cleanup if corresponding timestamps are
pinned.
- Added comprehensive unit tests for `Pin`, `Unpin`, and `isPinned`
functionality, covering basic, multiple pins, concurrent, and edge
cases.

This ensures the integrity of delete records by preventing their
premature removal from the delete buffer during segment loading.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>

2025-07-24 01:00:54 +08:00

..

deletebuffer

fix: Prevent delete records loss during slow segment loading [QueryNodeV2] (#43527)

2025-07-24 01:00:54 +08:00

buffered_forwarder_test.go

enhance: Add buffer forwarder for stream delta loading (#40559)

2025-03-17 15:24:10 +08:00

buffered_forwarder.go

enhance: Add buffer forwarder for stream delta loading (#40559)

2025-03-17 15:24:10 +08:00

delegator_data_test.go

fix: [AddField] Prevent concurrent load during UpdateSchema (#43043)

2025-07-02 17:38:44 +08:00

delegator_data.go

fix: Prevent delete records loss during slow segment loading [QueryNodeV2] (#43527)

2025-07-24 01:00:54 +08:00

delegator_test.go

fix: [AddField] Prevent concurrent load during UpdateSchema (#43043)

2025-07-02 17:38:44 +08:00

delegator.go

fix: balance channel before balance segment when upgrading (#43346)

2025-07-17 20:16:52 +08:00

delta_forward_test.go

feat: Implement partial result support on node down (#42009)

2025-05-28 00:12:28 +08:00

delta_forward.go

enhance: Add buffer forwarder for stream delta loading (#40559)

2025-03-17 15:24:10 +08:00

distribution_test.go

fix: Prevent duplicate segment results in count queries (#43173)

2025-07-14 11:10:49 +08:00

distribution.go

fix: Prevent duplicate segment results in count queries (#43173)

2025-07-14 11:10:49 +08:00

exclude_info_test.go

fix: Check exclude segment before add new growing segment (#31803)

2024-04-10 15:29:17 +08:00

exclude_info.go

fix: Prevent delegator unserviceable due to shard leader change (#42689)

2025-06-19 12:10:38 +08:00

idf_oracle_test.go

enhance: add warn log if some segment's bm25 stats lacks (#43111)

2025-07-09 23:22:47 +08:00

idf_oracle.go

enhance: add warn log if some segment's bm25 stats lacks (#43111)

2025-07-09 23:22:47 +08:00

mock_delegator.go

enhance: Add proxy task queue metrics (#42156)

2025-06-04 11:26:32 +08:00

OWNERS

Fix offline set always nil (#23340)

2023-04-11 14:42:30 +08:00

scalar_pruner.go

enhance: Use v2 package name for pkg module (#39990)

2025-02-22 23:15:58 +08:00

segment_pruner_test.go

enhance: Use v2 package name for pkg module (#39990)

2025-02-22 23:15:58 +08:00

segment_pruner.go

enhance: Use v2 package name for pkg module (#39990)

2025-02-22 23:15:58 +08:00

snapshot_test.go

load segment with target version, avoid read redundant segment (#24929)

2023-06-27 11:48:45 +08:00

snapshot.go

enhance: Use v2 package name for pkg module (#39990)

2025-02-22 23:15:58 +08:00

types.go

enhance: Use v2 package name for pkg module (#39990)

2025-02-22 23:15:58 +08:00

util.go

enhance: Use v2 package name for pkg module (#39990)

2025-02-22 23:15:58 +08:00