See also: #27675
The bloom filter set initialized new BF with fixed configured `n`. This
value is always larger than the actual batch size and causes generated
BF using more memory.
This PR make write buffer to initialize BF with estimated batch size
from schema & configuration value.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
See also #27675
Fix logic problem introduced by #29413, which is serializer tries to
merge statslog list while level segments do not have statslog. This
shall result returning error. `writeBufferBase` ignores this error but
it shall only ignore `ErrSegmentNotFound`.
This PR add logic checking segment level before execution of merging
statslog list. And add error type check for getSyncTask failure.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
See also #27675
For now, Level zero segments shall always be synced as `Flushed` ones.
This PR fixes when level zero segments selected by policies other than
flush ts policy will be synced as growing state.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
See also #27675
Since serialization segment buffer does not related to sync manager can
shall be done before submit into sync manager. So that the pk statistic
file could be more accurate and reduce complex logic inside sync
manager.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Related to #27675
The timestamp from, to field is not field for new implementation of
writebuffer & sync manager
This pr fills these field for better log information
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Related to #28736#28748
See also #27675
Previous PR: #28646
This PR fixes `SegmentNotFound` issue when compaction happens multiple
times and the buffer of first generation segment is sync due to stale
policy
Now the `CompactSegments` API of metacache shall update the compactTo
field of segmentInfo if the compactTo segment is also compacted to keep
the bloodline clean
Also, add the `CompactedSegment` SyncPolicy to sync the compacted
segment asap to keep metacache clean
Now the `SyncPolicy` is an interface instead of a function type so that
when it selects some segments to sync, we colud log the reason and
target segment
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
See also #27675
Compacted segment info shall be removed after all buffer belongs to it
is sync-ed.
This PR add the cleanup function after triggerSyncTask logic:
- The buffer is stable and protected by mutex
- Cleanup fetches compacted & non-sync segment
- Remove segment info only there is no buffered maintained in manager
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
See also #27675
- Fix LevelZero segment cannot be flushed
- Add level option for syncTask
- Invoke `AddSegment` when new LevelZero segment is allocated
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
This PR removes all the commented code and files from PR #28320
For naming issue:
- Renaming `MinCheckpoint` to `EarliestPosition`, see #28320 comment
- Renaming `writebuffer.Mananger` to `BufferMananger`, see #27874
comment
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
See also #27675
This PR make previously merged refactory of datanode go online
- Use write node to replace insert/delete node
- Use write buffer manager to control all buffers
- Use sync manager to control sync tasks instead of flush manager
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
