When getting disk usage, files or directories may be removed
concurrently due to segment release. This PR ignores “file or directory
does not exist” errors in such cases.
issue: https://github.com/milvus-io/milvus/issues/45239
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Related to #44956
Integrate the StorageV2 FFI interface as the unified storage layer for
reading packed columnar data, replacing the custom iterative reader with
a manifest-based approach using the milvus-storage library.
Changes:
- Add C++ FFI reader implementation (ffi_reader_c.cpp/h) with Arrow C
Stream interface
- Implement utility functions to convert CStorageConfig to
milvus-storage Properties
- Create ManifestReader in Go that generates manifests from binlogs
- Add FFI packed reader CGO bindings (packed_reader_ffi.go)
- Refactor NewBinlogRecordReader to use ManifestReader for V2 storage
- Support both manifest file paths and direct manifest content
- Enable configurable buffer sizes and column projection
Technical improvements:
- Zero-copy data exchange using Arrow C Data Interface
- Optimized I/O operations through milvus-storage library
- Simplified code path with manifest-based reading
- Better performance with batched streaming reads
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Related to #45157
Fix a bug where DataNode panics when building json stats index throws an
exception before the writer is initialized. The destructor would call
Close() on an uninitialized packed_writer_ pointer, causing a null
pointer dereference.
Changes:
- Add null check for packed_writer_ before calling Flush() and Close()
- Prevents null pointer dereference in edge cases
- Ignore close status as this is a cleanup operation
This ensures safe cleanup even when initialization fails due to
exceptions.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
relate: https://github.com/milvus-io/milvus/issues/43687
Add global analyzer options, avoid having to merge some milvus params
into user's analyzer params.
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
Related to #45081
StringIndexSort now properly handles collections with all-null string
fields by:
- Removing the error thrown when unique_count is 0 in ParseBinaryData
- Adding alignment and padding support in mmap serialization (similar to
ScalarIndexSort)
- Separating data_size_ from mmap_size_ to correctly parse data without
reading padding
This fixes load collection timeout failures when all string field data
is null, particularly affecting STL_SORT and TRIE index types.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
This commit introduces the foundation for enabling segments to manage
their own loading process by passing load information during segment
creation.
Changes:
C++ Layer:
- Add NewSegmentWithLoadInfo() C API to create segments with serialized
load info
- Add SetLoadInfo() method to SegmentInterface for storing load
information
- Refactor segment creation logic into shared CreateSegment() helper
function
- Add comprehensive documentation for the new API
Go Layer:
- Extend CreateCSegmentRequest to support optional LoadInfo field
- Update segment creation in querynode to pass SegmentLoadInfo when
available
- Add ConvertToSegcoreSegmentLoadInfo() and helper converters for proto
translation
Proto Definitions:
- Add segcorepb.SegmentLoadInfo message with essential loading metadata
- Add supporting messages: Binlog, FieldBinlog, FieldIndexInfo,
TextIndexStats, JsonKeyStats
- Remove dependency on data_coord.proto by creating segcore-specific
definitions
Testing:
- Add comprehensive unit tests for proto conversion functions
- Test edge cases including nil inputs, empty data, and nil array/map
elements
This is the first step toward issue #45060 - enabling segments to
autonomously manage their loading process, which will:
- Clarify responsibilities between Go and C++ layers
- Reduce cross-language call overhead
- Enable precise resource management at the C++ level
- Support better integration with caching layer
- Enable proactive schema evolution handling
Related to #45060
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Optimize bitmap initialization in ScalarIndexSort range queries by using
adaptive strategy based on result density. When more than 50% of
elements match the range condition, initialize bitmap with all true
values and clear non-matching elements. Otherwise, use the original
approach of initializing with false and setting matching elements. Also
defer bitmap allocation until after early return checks to avoid
unnecessary memory allocation.
This optimization reduces bit operations for high-selectivity queries
while maintaining the same performance for low-selectivity queries.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Update milvus-storage version from aa189ad to e5f5b4c to include the fix
for duplicate AWS SDK initialization that was causing init conflicts.
This update removes the redundant arrow::fs::InitializeS3() call that
was resulting in duplicate Aws::InitAPI() initialization. The duplicate
initialization was causing AWS SDK to ignore custom configurations,
particularly affecting GCP Workload Identity authentication.
Changes in milvus-storage e5f5b4c:
- Remove redundant arrow::fs::InitializeS3() call
- Keep only the extended S3 initialization with custom AWS SDK options
- Ensure GCP IAM authentication via custom HTTP client factory works
correctly
Relates to #44745
Reference: milvus-io/milvus-storage#285
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: https://github.com/milvus-io/milvus/issues/44399
This PR implements STL_SORT for VARCHAR data type for both RAM and MMAP
mode.
The general idea is that we deduplicate field values and maintains a
posting list for each unique value.
The serialization format of the index is:
```
[unique_count][string_offsets][string_data][post_list_offsets][post_list_data][magic_code]
string_offsets: array of offsets into string_data section
string_data: str_len1, str1, str_len2, str2, ...
post_list_offsets: array of offsets into post_list_data section
post_list_data: post_list_len1, row_id1, row_id2, ..., post_list_len2, row_id1, row_id2, ...
```
---------
Signed-off-by: SpadeA <tangchenjie1210@gmail.com>
issue: #44961
This PR fixes 3 geometry related bugs:
1. Implement `ToString` interface for GisFunctionFilter.
2. Ignore GisFunctionFilter `MoveCursor` for growing segment.
3. Don't skip null geometry for building R-Tree index, should be record
in null_offsets.
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Related to #44897
Add missing JSON data type handling in CreateArrowScalarFromDefaultValue
to fix query failures when dynamic fields are enabled. JSON default
values are now properly converted to arrow::BinaryScalar using
bytes_data().
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #44802
After a Geometry object is serialized into WKB, the resulting binary may
contain '\0' bytes.
When growing mmap is enabled, the append data logic uses strcpy, which
stops copying at the first '\0' bytes.
This causes only part of the WKB---typically the portion up to the
geometry type field to be copied, leading to corrupted data.
As a result, during parsing, all POINT geometries are incorrectly
interperted as POINT(0 0).
To fix this issue, memcpy will be used instead of strcpy.
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Related to #44819
This fix addresses an issue(#44819) where the offset parameter did not
work correctly during searches when multiple results had identical
scores. The problem occurred because results with equal scores were not
consistently ordered, leading to unpredictable pagination behavior.
The solution adds a new sorting step (SortEqualScoresByPks) in the
reduce phase that sorts results with identical scores by their primary
keys in ascending order. This ensures deterministic ordering and enables
proper offset functionality.
Changes:
- Add SortEqualScoresByPks() to sort results with equal scores by PK
- Add SortEqualScoresOneNQ() to handle per-query sorting logic
- Invoke sorting step after FillPrimaryKey() in Reduce() workflow
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
