XuanYang-cn 623a9e5156

fix: Accurate size estimation for sliced arrow arrays in compaction (#45294) ...

Sliced arrow arrays "incorrectly" returned the original array's size via
SizeInBytes(), causing inaccurate memory estimates during compaction.

This resulted in segments closing prematurely in mergeSplit mode -
expected 500MB compactions produced 4x100+MB segments instead.

Fixed by calculating actual byte size of sliced arrays, ensuring proper
segment sizing and more accurate memory usage tracking.

See also: #45293

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

2025-11-06 14:57:34 +08:00

..

compress

fix: pulsar cannot work properly if backlog exceed (#42653)

2025-06-13 14:28:37 +08:00

arrow_util_test.go

fix: Support JSON default value in compaction (#45330)

2025-11-06 10:15:34 +08:00

arrow_util.go

fix: Support JSON default value in compaction (#45330)

2025-11-06 10:15:34 +08:00

azure_object_storage_test.go

enhance: move object storage client creation to pkg package (#40440)

2025-03-12 20:38:07 +08:00

azure_object_storage.go

enhance: Add Size interface to FileReader to eliminate the StatObject call during Read (#42908)

2025-06-25 14:36:41 +08:00

binlog_reader.go

feat: Add CMEK cipher plugin (#43722)

2025-08-27 11:15:52 +08:00

binlog_test.go

feat: impl StructArray -- optimize vector array serialization (#44035)

2025-09-03 16:39:53 +08:00

binlog_util_test.go

Format the code (#27275)

2023-09-21 09:45:27 +08:00

binlog_util.go

enhance: Use v2 package name for pkg module (#39990)

2025-02-22 23:15:58 +08:00

binlog_writer_test.go

feat: Add CMEK cipher plugin (#43722)

2025-08-27 11:15:52 +08:00

binlog_writer.go

feat: Add CMEK cipher plugin (#43722)

2025-08-27 11:15:52 +08:00

data_codec_test.go

feat: impl StructArray -- create schema, insert, and retrieve data (#42855)

2025-07-27 01:30:55 +08:00

data_codec.go

feat: Geospatial Data Type and GIS Function support for milvus (#44547)

2025-09-28 19:43:05 +08:00

data_sorter_test.go

feat: impl StructArray -- create schema, insert, and retrieve data (#42855)

2025-07-27 01:30:55 +08:00

data_sorter.go

feat: Geospatial Data Type and GIS Function support for milvus (#44547)

2025-09-28 19:43:05 +08:00

delta_data_test.go

enhance: Return deltadata for DeleteCodec.Deserialize (#37214)

2024-10-29 12:04:24 +08:00

delta_data.go

enhance: Add buffer forwarder for stream delta loading (#40559)

2025-03-17 15:24:10 +08:00

event_data.go

feat: Add CMEK cipher plugin (#43722)

2025-08-27 11:15:52 +08:00

event_header.go

enhance: Use v2 package name for pkg module (#39990)

2025-02-22 23:15:58 +08:00

event_reader.go

fix: Update logging context and upgrade dependencies (#41318)

2025-04-23 10:52:38 +08:00

event_test.go

feat: impl StructArray -- optimize vector array serialization (#44035)

2025-09-03 16:39:53 +08:00

event_writer_test.go

enhance: Use v2 package name for pkg module (#39990)

2025-02-22 23:15:58 +08:00

event_writer.go

enhance: Use v2 package name for pkg module (#39990)

2025-02-22 23:15:58 +08:00

factory.go

enhance: move object storage client creation to pkg package (#40440)

2025-03-12 20:38:07 +08:00

field_stats_test.go

enhance: Use v2 package name for pkg module (#39990)

2025-02-22 23:15:58 +08:00

field_stats.go

feat: add timesatmptz data type (#44005)

2025-08-26 15:59:53 +08:00

field_value_test.go

enhance: Use internal json(sonic) to replace std json lib (#37708)

2024-11-18 10:46:31 +08:00

field_value.go

feat: add timesatmptz data type (#44005)

2025-08-26 15:59:53 +08:00

gcp_native_object_storage_test.go

enhance: move object storage client creation to pkg package (#40440)

2025-03-12 20:38:07 +08:00

gcp_native_object_storage.go

enhance: Add Size interface to FileReader to eliminate the StatObject call during Read (#42908)

2025-06-25 14:36:41 +08:00

index_data_codec_test.go

enhance: Use v2 package name for pkg module (#39990)

2025-02-22 23:15:58 +08:00

index_data_codec.go

feat: Add CMEK cipher plugin (#43722)

2025-08-27 11:15:52 +08:00

insert_data_test.go

feat: impl StructArray -- create schema, insert, and retrieve data (#42855)

2025-07-27 01:30:55 +08:00

insert_data.go

fix: Fix import null geometry data (#45161)

2025-10-30 14:18:07 +08:00

local_chunk_manager_test.go

enhance: move object storage client creation to pkg package (#40440)

2025-03-12 20:38:07 +08:00

local_chunk_manager.go

enhance: Add Size interface to FileReader to eliminate the StatObject call during Read (#42908)

2025-06-25 14:36:41 +08:00

minio_object_storage_test.go

enhance: move object storage client creation to pkg package (#40440)

2025-03-12 20:38:07 +08:00

minio_object_storage.go

enhance: Add Size interface to FileReader to eliminate the StatObject call during Read (#42908)

2025-06-25 14:36:41 +08:00

OWNERS

[skip ci]Update OWNERS files (#11898)

2021-11-16 15:41:11 +08:00

partition_stats_test.go

feat: Define FieldValue, FieldStats and PartitionStats (#30286)

2024-03-06 20:42:37 -08:00

partition_stats.go

enhance: Use internal json(sonic) to replace std json lib (#37708)

2024-11-18 10:46:31 +08:00

payload_reader_test.go

enhance: upgrade go arrow version from 12.0.1 to 17.0.0 (#39916)

2025-02-25 10:30:02 +08:00

payload_reader.go

feat: impl StructArray -- support more types of vector in STRUCT (#44736)

2025-10-15 10:25:59 +08:00

payload_test.go

feat: impl StructArray -- support more types of vector in STRUCT (#44736)

2025-10-15 10:25:59 +08:00

payload_writer_test.go

feat: impl StructArray -- support more types of vector in STRUCT (#44736)

2025-10-15 10:25:59 +08:00

payload_writer.go

feat: impl StructArray -- ban non-float-vector for now (#44875)

2025-10-17 10:26:09 +08:00

payload.go

feat: Geospatial Data Type and GIS Function support for milvus (#44547)

2025-09-28 19:43:05 +08:00

pk_statistics.go

enhance: Use v2 package name for pkg module (#39990)

2025-02-22 23:15:58 +08:00

primary_key_test.go

enhance: Use internal json(sonic) to replace std json lib (#37708)

2024-11-18 10:46:31 +08:00

primary_key.go

fix: Update logging context and upgrade dependencies (#41318)

2025-04-23 10:52:38 +08:00

primary_keys_test.go

enhance: Add PrimaryKeys interface to reduce memory usage (#30405)

2024-02-01 09:57:11 +08:00

primary_keys.go

enhance: Add buffer forwarder for stream delta loading (#40559)

2025-03-17 15:24:10 +08:00

print_binlog_test.go

feat: Geospatial Data Type and GIS Function support for milvus (#44547)

2025-09-28 19:43:05 +08:00

print_binlog.go

feat: Geospatial Data Type and GIS Function support for milvus (#44547)

2025-09-28 19:43:05 +08:00

remote_chunk_manager_test.go

enhance: move object storage client creation to pkg package (#40440)

2025-03-12 20:38:07 +08:00

remote_chunk_manager.go

enhance: Add Size interface to FileReader to eliminate the StatObject call during Read (#42908)

2025-06-25 14:36:41 +08:00

rw_test.go

enhance: [StorageV2] Integrate FFI interface for packed reader (#45132)

2025-11-05 19:57:34 +08:00

rw.go

enhance: [StorageV2] Integrate FFI interface for packed reader (#45132)

2025-11-05 19:57:34 +08:00

schema_test.go

feat: impl StructArray -- create schema, insert, and retrieve data (#42855)

2025-07-27 01:30:55 +08:00

schema.go

feat: impl StructArray -- optimize vector array serialization (#44035)

2025-09-03 16:39:53 +08:00

serde_delta_test.go

enhance: update delta log serialization APIs to integrate storage V2 (#44998)

2025-10-22 15:58:12 +08:00

serde_delta.go

enhance: update delta log serialization APIs to integrate storage V2 (#44998)

2025-10-22 15:58:12 +08:00

serde_events_test.go

feat: impl StructArray -- create schema, insert, and retrieve data (#42855)

2025-07-27 01:30:55 +08:00

serde_events_v2_test.go

enhance: update delta log serialization APIs to integrate storage V2 (#44998)

2025-10-22 15:58:12 +08:00

serde_events_v2.go

fix: Accurate size estimation for sliced arrow arrays in compaction (#45294)

2025-11-06 14:57:34 +08:00

serde_events.go

enhance: [StorageV2] Integrate FFI interface for packed reader (#45132)

2025-11-05 19:57:34 +08:00

serde_test.go

fix: Accurate size estimation for sliced arrow arrays in compaction (#45294)

2025-11-06 14:57:34 +08:00

serde.go

fix: Accurate size estimation for sliced arrow arrays in compaction (#45294)

2025-11-06 14:57:34 +08:00

sort_test.go

enhance: update delta log serialization APIs to integrate storage V2 (#44998)

2025-10-22 15:58:12 +08:00

sort.go

fix: Fix merge sort out of range (#44230)

2025-09-08 18:01:55 +08:00

stats_collector_test.go

enhance: update delta log serialization APIs to integrate storage V2 (#44998)

2025-10-22 15:58:12 +08:00

stats_collector.go

enhance: update delta log serialization APIs to integrate storage V2 (#44998)

2025-10-22 15:58:12 +08:00

stats_test.go

enhance: Use v2 package name for pkg module (#39990)

2025-02-22 23:15:58 +08:00

stats.go

enhance: support preload sealed segment bm25 stats and optimize bm25 stats serialize (#44279)

2025-09-29 16:35:05 +08:00

storage_test.go

enhance: Use v2 package name for pkg module (#39990)

2025-02-22 23:15:58 +08:00

types.go

enhance: Add Size interface to FileReader to eliminate the StatObject call during Read (#42908)

2025-06-25 14:36:41 +08:00

unsafe_test.go

[skip e2e]Update license for storage unsafe (#14452)

2021-12-28 20:03:56 +08:00

unsafe.go

enhance: support null in go payload (#32296)

2024-06-19 17:08:00 +08:00

utils_test.go

enhance: forbid build analyzer at proxy (#44067)

2025-10-23 10:58:12 +08:00

utils.go

fix: Fix bugs related to TimestampTz (#45111)

2025-11-04 16:51:33 +08:00