sparknack 4b14ab14e3

enhance: mmap once for each group chunk (#45487) ...

issue: #45486

This commit refactors the chunk writing system by introducing a
two-phase
approach: size calculation followed by writing to a target. This enables
efficient group chunk creation where multiple fields share a single mmap
region, significantly reducing the number of mmap system calls and VMAs.

- Optimize `mmap` usage: single `mmap` per group chunk instead of per
field
- Split ChunkWriter into two phases:
  - `calculate_size()`: Pre-compute required memory without allocation
  - `write_to_target()`: Write data to a provided ChunkTarget
- Implement `ChunkMmapGuard` for unified mmap region lifecycle
management
  - Handles `munmap` and file cleanup via RAII
  - Shared via `std::shared_ptr` across multiple chunks in a group

Signed-off-by: Shawn Wang <shawn.wang@zilliz.com>

---------

Signed-off-by: Shawn Wang <shawn.wang@zilliz.com>

2025-11-26 10:37:08 +08:00

..

reduce

fix: ensure deterministic search result ordering when scores are equal (#44870)

2025-10-16 10:04:00 +08:00

storagev1translator

enhance: mmap once for each group chunk (#45487)

2025-11-26 10:37:08 +08:00

storagev2translator

enhance: mmap once for each group chunk (#45487)

2025-11-26 10:37:08 +08:00

AckResponder.h

enhance: add new LRU cache impl (#30360)

2024-02-27 20:58:40 +08:00

arrow_fs_c.cpp

enhance: integrate StorageV2 FFI interface for manifest-based segment loading (#45798)

2025-11-25 17:27:07 +08:00

arrow_fs_c.h

enhance: avoid shutdown sdk api in minio cm destructor (#42459)

2025-06-04 09:58:39 +08:00

ArrowFsCTest.cpp

enhance: moved more segcore unit test files (#44210)

2025-09-08 10:21:55 +08:00

check_vec_index_c.cpp

feat: impl StructArray -- support create index for vector array (embedding list) and search on it (#43726)

2025-08-20 10:27:46 +08:00

check_vec_index_c.h

feat: impl StructArray -- support create index for vector array (embedding list) and search on it (#43726)

2025-08-20 10:27:46 +08:00

ChunkedSegmentSealedBinlogIndexTest.cpp

enhance: add cancellation checking in each operator and expr (#45354)

2025-11-26 10:15:07 +08:00

ChunkedSegmentSealedImpl.cpp

enhance: mmap once for each group chunk (#45487)

2025-11-26 10:37:08 +08:00

ChunkedSegmentSealedImpl.h

enhance: integrate StorageV2 FFI interface for manifest-based segment loading (#45798)

2025-11-25 17:27:07 +08:00

ChunkedSegmentSealedStorageV2Test.cpp

enhance: add storage resource usage for scalar search (#44414)

2025-09-22 14:28:06 +08:00

ChunkedSegmentSealedTest.cpp

enhance: mmap once for each group chunk (#45487)

2025-11-26 10:37:08 +08:00

CMakeLists.txt

enhance: Refine code for get_deleted_bitmap (#36819)

2024-10-28 15:19:30 +08:00

collection_c_test.cpp

enhance: Move c API unittest aside to src files (#44458)

2025-09-19 10:30:01 +08:00

collection_c.cpp

enhance: collect all cgo calling into metric and log slow cgo call (#43035)

2025-07-03 15:00:44 +08:00

collection_c.h

feat: [Tiered] Make load list work as warmup hint (#42490)

2025-06-04 10:28:32 +08:00

Collection.cpp

fix: segcore collection schema update not concurrent safe. (#45337)

2025-11-14 17:51:37 +08:00

Collection.h

fix: segcore collection schema update not concurrent safe. (#45337)

2025-11-14 17:51:37 +08:00

column_groups_c.cpp

enhance: collect all cgo calling into metric and log slow cgo call (#43035)

2025-07-03 15:00:44 +08:00

column_groups_c.h

enhance: make json path index support term filter (#40140)

2025-03-04 11:56:02 +08:00

ColumnGroupsCTest.cpp

enhance: moved more segcore unit test files (#44210)

2025-09-08 10:21:55 +08:00

ConcurrentVector.cpp

feat: Geospatial Data Type and GIS Function support for milvus (#44547)

2025-09-28 19:43:05 +08:00

ConcurrentVector.h

fix: Use correct regex for cppcheck (#44077)

2025-08-27 20:57:50 +08:00

ConcurrentVectorTest.cpp

enhance: moved more segcore unit test files (#44210)

2025-09-08 10:21:55 +08:00

DeletedRecord.h

fix:fix undefined behavior for dump snapshot (#45611)

2025-11-25 14:17:07 +08:00

DeletedRecordTest.cpp

enhance: moved more segcore unit test files (#44210)

2025-09-08 10:21:55 +08:00

FieldIndexing.cpp

fix: Skip create tmp dir for growing R-Tree index (#45256)

2025-11-04 13:01:32 +08:00

FieldIndexing.h

feat: Geospatial Data Type and GIS Function support for milvus (#44547)

2025-09-28 19:43:05 +08:00

IndexConfigGenerator.cpp

enhance: Make build ratio of interim index configurable (#43939)

2025-08-25 14:43:51 +08:00

IndexConfigGenerator.h

enhance: Make build ratio of interim index configurable (#43939)

2025-08-25 14:43:51 +08:00

IndexCTest.cpp

enhance: moved more segcore unit test files (#44210)

2025-09-08 10:21:55 +08:00

InsertRecord.h

enhance: prevent panic by adding null pointer check when clearing InsertRecord _pk2offset_ (#45281)

2025-11-10 11:37:35 +08:00

InsertRecordOffsetOrderedArrayTest.cpp

enhance: moved more segcore unit test files (#44210)

2025-09-08 10:21:55 +08:00

InsertRecordOffsetOrderedMapTest.cpp

enhance: moved more segcore unit test files (#44210)

2025-09-08 10:21:55 +08:00

load_field_data_c_test.cpp

enhance: Move c API unittest aside to src files (#44458)

2025-09-19 10:30:01 +08:00

load_field_data_c.cpp

enhance: [StorageV2] Make load info contains child info (#44384)

2025-09-16 16:14:00 +08:00

load_field_data_c.h

enhance: [StorageV2] Make load info contains child info (#44384)

2025-09-16 16:14:00 +08:00

load_index_c_test.cpp

enhance: add storage resource usage for vector search (#44308)

2025-09-19 20:20:02 +08:00

load_index_c.cpp

enhance: Move segment loading logic from Go layer to segcore for self-managed loading (#45488)

2025-11-14 11:21:37 +08:00

load_index_c.h

enhance: cachinglayer: resource management for segment loading (#43846)

2025-08-29 11:37:50 +08:00

memory_planner.cpp

enhance: Clean unused conan dependency (#45366)

2025-11-07 17:07:34 +08:00

memory_planner.h

enhance: [StorageV2] Make datanode use non-singleton fs (#44418)

2025-09-18 20:06:00 +08:00

MemoryPlannerTest.cpp

enhance: moved more segcore unit test files (#44210)

2025-09-08 10:21:55 +08:00

metrics_c.cpp

fix: Avoided dereferencing NULL pointer (#34836)

2024-07-27 17:27:52 +08:00

metrics_c.h

Support show knowhere prometheus metrics (#23102)

2023-04-07 15:34:29 +08:00

packed_reader_c.cpp

enhance: support max_connections config for remote storage (#45225)

2025-11-13 15:37:38 +08:00

packed_reader_c.h

feat: Add CMEK cipher plugin (#43722)

2025-08-27 11:15:52 +08:00

packed_writer_c.cpp

enhance: support max_connections config for remote storage (#45225)

2025-11-13 15:37:38 +08:00

packed_writer_c.h

enhance: [StorageV2] Use compressed size as log file size (#44402)

2025-09-16 21:20:02 +08:00

PackedReaderWriterTest.cpp

enhance: moved more segcore unit test files (#44210)

2025-09-08 10:21:55 +08:00

pkVisitor.h

enhance: rename PanicInfo to ThrowInfo (#43384)

2025-07-19 20:22:52 +08:00

plan_c_test.cpp

enhance: Move c API unittest aside to src files (#44458)

2025-09-19 10:30:01 +08:00

plan_c.cpp

fix: [AddField] Use shared_ptr of schema in plan fixing dangling ref (#42693)

2025-06-12 20:46:36 +08:00

plan_c.h

enhance: retrieve output fields after local reduce (#32346)

2024-04-25 09:49:26 +08:00

Record.h

feat: add cachinglayer to sealed segment (#41436)

2025-04-28 10:52:40 +08:00

reduce_c_test.cpp

enhance: add cancellation checking in each operator and expr (#45354)

2025-11-26 10:15:07 +08:00

reduce_c.cpp

enhance: add storage resource usage for vector search (#44308)

2025-09-19 20:20:02 +08:00

reduce_c.h

enhance: add storage resource usage for vector search (#44308)

2025-09-19 20:20:02 +08:00

ReduceCTest.cpp

enhance: add storage resource usage for vector search (#44308)

2025-09-19 20:20:02 +08:00

ReduceStructure.h

fix: ensure deterministic search result ordering when scores are equal (#44870)

2025-10-16 10:04:00 +08:00

ReduceStructureTest.cpp

enhance: move c++ unit test file to aside of the production code (#43932)

2025-09-03 23:45:53 +08:00

ReduceUtils.cpp

feat: Geospatial Data Type and GIS Function support for milvus (#44547)

2025-09-28 19:43:05 +08:00

ReduceUtils.h

feat: LRU cache implementation (#32567)

2024-05-06 20:29:30 +08:00

SealedIndexingRecord.h

feat: add cachinglayer to index (#41653)

2025-05-08 10:12:54 +08:00

segcore_init_c.cpp

enhance: Make GeometryCache an optional configuration (#45192)

2025-11-03 19:59:32 +08:00

segcore_init_c.h

enhance: Make GeometryCache an optional configuration (#45192)

2025-11-03 19:59:32 +08:00

SegcoreConfig.cpp

enhance: rename PanicInfo to ThrowInfo (#43384)

2025-07-19 20:22:52 +08:00

SegcoreConfig.h

enhance: Make GeometryCache an optional configuration (#45192)

2025-11-03 19:59:32 +08:00

SegcoreInitCTest.cpp

enhance: moved more segcore unit test files (#44210)

2025-09-08 10:21:55 +08:00

segment_c_test.cpp

enhance: add storage resource usage for vector search (#44308)

2025-09-19 20:20:02 +08:00

segment_c.cpp

enhance: add cancellation checking in each operator and expr (#45354)

2025-11-26 10:15:07 +08:00

segment_c.h

enhance: Move segment loading logic from Go layer to segcore for self-managed loading (#45488)

2025-11-14 11:21:37 +08:00

SegmentChunkReader.cpp

enhance: add storage resource usage for scalar search (#44414)

2025-09-22 14:28:06 +08:00

SegmentChunkReader.h

enhance: add storage resource usage for scalar search (#44414)

2025-09-22 14:28:06 +08:00

SegmentGrowing.h

fix: [AddField] Add protection logic inserting old data into new schema (#41978)

2025-05-22 11:30:24 +08:00

SegmentGrowingImpl.cpp

enhance: add cancellation checking in each operator and expr (#45354)

2025-11-26 10:15:07 +08:00

SegmentGrowingImpl.h

enhance: add cancellation checking in each operator and expr (#45354)

2025-11-26 10:15:07 +08:00

SegmentGrowingIndexTest.cpp

enhance: add storage resource usage for scalar search (#44414)

2025-09-22 14:28:06 +08:00

SegmentGrowingStorageV2Test.cpp

enhance: [StorageV2] Make datanode use non-singleton fs (#44418)

2025-09-18 20:06:00 +08:00

SegmentGrowingTest.cpp

feat: impl StructArray -- support more types of vector in STRUCT (#44736)

2025-10-15 10:25:59 +08:00

SegmentInterface.cpp

enhance: add cancellation checking in each operator and expr (#45354)

2025-11-26 10:15:07 +08:00

SegmentInterface.h

enhance: add cancellation checking in each operator and expr (#45354)

2025-11-26 10:15:07 +08:00

SegmentSealed.h

enhance: add cachinglayer management for TextMatchIndex (#44741)

2025-10-13 14:37:58 +08:00

SegmentSealedRetrieveTest.cpp

enhance: moved more segcore unit test files (#44210)

2025-09-08 10:21:55 +08:00

TimestampIndex.cpp

enhance: add vector reserve to improve memory allocation in segcore (#45757)

2025-11-25 14:19:07 +08:00

TimestampIndex.h

enhance: cachinglayer: resource management for segment loading (#43846)

2025-08-29 11:37:50 +08:00

TimestampIndexTest.cpp

enhance: moved more segcore unit test files (#44210)

2025-09-08 10:21:55 +08:00

token_stream_c.cpp

fix: use latest timetick to expire cache (#45717)

2025-11-20 21:39:04 +08:00

token_stream_c.h

enhance: removed unused map_c (#44183)

2025-09-09 16:46:04 +08:00

tokenizer_c.cpp

fix: use latest timetick to expire cache (#45717)

2025-11-20 21:39:04 +08:00

tokenizer_c.h

enhance: add global analyzer options (#44684)

2025-10-28 14:52:10 +08:00

TokenizerCTest.cpp

enhance: removed unused map_c (#44183)

2025-09-09 16:46:04 +08:00

Types.h

feat: impl StructArray -- support more types of vector in STRUCT (#44736)

2025-10-15 10:25:59 +08:00

Utils.cpp

enhance: Move segment loading logic from Go layer to segcore for self-managed loading (#45488)

2025-11-14 11:21:37 +08:00

Utils.h

enhance: Move segment loading logic from Go layer to segcore for self-managed loading (#45488)

2025-11-14 11:21:37 +08:00

UtilsTest.cpp

enhance: move c++ unit test file to aside of the production code (#43932)

2025-09-03 23:45:53 +08:00

vector_index_c_test.cpp

enhance: Move c API unittest aside to src files (#44458)

2025-09-19 10:30:01 +08:00

vector_index_c.cpp

fix: Use correct regex for cppcheck (#44077)

2025-08-27 20:57:50 +08:00

vector_index_c.h

feat: impl StructArray -- support create index for vector array (embedding list) and search on it (#43726)

2025-08-20 10:27:46 +08:00