wayblink c62bf8a0b0

fix: [Cherry-pick]Pick major compaction fixs and optimizations (#34360) ...

This PR cherry-picks the following commits:

- fix: sync partitiion stats blocking balance task #33742
- fix: Fix meta prefix overlap bug #33830
- fix: Small fixs of major compaction #33929 
- fix: Fix memory buffer error & some renaming #33850
- fix: sync part stats task cannot be finished #34027 
- Add an option to enable/disable vector field clustering key #34097
- fix: fix error ignore in compactor #34169
- fix:load major compaction partial result #34052
- Use new stream segment reader in clustering compaction #34232

issue: #30633
pr: #33742 #33830 #33929 #33850 #34027 #34097 #34169 #34052 #34232

---------

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
Signed-off-by: wayblink <anyang.wang@zilliz.com>
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: Chun Han <116052805+MrPresent-Han@users.noreply.github.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>

2024-07-03 09:53:37 +08:00

..

aliyun

Identify service providers based on addresses (#27907)

2023-10-25 17:28:10 +08:00

gcp

Format the code (#27275)

2023-09-21 09:45:27 +08:00

tencent

feat: Support tencent cloud object storage for milvus (#30163)

2024-01-23 11:28:56 +08:00

azure_object_storage_test.go

enhance: use WalkWithPrefix api for oss, enable piplined file gc (#31740)

2024-04-25 20:41:27 +08:00

azure_object_storage.go

enhance: use WalkWithPrefix api for oss, enable piplined file gc (#31740)

2024-04-25 20:41:27 +08:00

binlog_iterator_test.go

enhance: add helpers to parse sparse float vector in JSON (#32543)

2024-04-25 14:47:24 +08:00

binlog_iterator.go

enhance: Enable binlog deserialize reader in datanode compaction (#31036)

2024-03-08 18:25:02 +08:00

binlog_reader.go

Move some modules from internal to public package (#22572)

2023-04-06 19:14:32 +08:00

binlog_test.go

Format the code (#27275)

2023-09-21 09:45:27 +08:00

binlog_util_test.go

Format the code (#27275)

2023-09-21 09:45:27 +08:00

binlog_util.go

Move some modules from internal to public package (#22572)

2023-04-06 19:14:32 +08:00

binlog_writer_test.go

Format the code (#27275)

2023-09-21 09:45:27 +08:00

binlog_writer.go

enhance: Unify data type check APIs for go (#31887)

2024-04-07 14:27:22 +08:00

data_codec_test.go

fix: [2.4]DeleteData merge wrong data casuing data loss (#33821)

2024-06-13 16:07:56 +08:00

data_codec.go

enhance: [2.4] Avoid merging insert data when buffering insert msgs (#34205)

2024-06-27 10:14:05 +08:00

data_sorter_test.go

enhance: add helpers to parse sparse float vector in JSON (#32543)

2024-04-25 14:47:24 +08:00

data_sorter.go

feat: [Sparse Float Vector] add sparse vector support to milvus components (#30630)

2024-03-13 14:32:54 -07:00

delta_data_test.go

enhance: [2.4] Unify DeleteLog parsing code (#34188)

2024-06-27 10:12:13 +08:00

delta_data.go

enhance: [2.4] Unify DeleteLog parsing code (#34188)

2024-06-27 10:12:13 +08:00

event_data.go

Format the code (#27275)

2023-09-21 09:45:27 +08:00

event_header.go

Move some modules from internal to public package (#22572)

2023-04-06 19:14:32 +08:00

event_reader.go

Use go-api/v2 for milvus-proto (#24770)

2023-06-09 01:28:37 +08:00

event_test.go

Format the code (#27275)

2023-09-21 09:45:27 +08:00

event_writer_test.go

Format the code (#27275)

2023-09-21 09:45:27 +08:00

event_writer.go

enhance: Unify data type check APIs for go (#31887)

2024-04-07 14:27:22 +08:00

factory.go

enhance: use WalkWithPrefix api for oss, enable piplined file gc (#31740)

2024-04-25 20:41:27 +08:00

field_stats_test.go

feat: [cherry-pick] Major compaction (#34326)

2024-07-02 18:29:01 +08:00

field_stats.go

feat: [cherry-pick] Major compaction (#34326)

2024-07-02 18:29:01 +08:00

field_value_test.go

feat: Define FieldValue, FieldStats and PartitionStats (#30286)

2024-03-06 20:42:37 -08:00

field_value.go

feat: [cherry-pick] Major compaction (#34326)

2024-07-02 18:29:01 +08:00

index_data_codec_test.go

enhance: Add memory size for binlog (#33025)

2024-05-15 12:59:34 +08:00

index_data_codec.go

Format the code (#27275)

2023-09-21 09:45:27 +08:00

insert_data_test.go

feat: Support sparse float vector bulk insert for binlog/json/parquet (#32649)

2024-05-07 18:43:30 +08:00

insert_data.go

enhance: [2.4] Avoid merging insert data when buffering insert msgs (#34205)

2024-06-27 10:14:05 +08:00

local_chunk_manager_test.go

enhance: use WalkWithPrefix api for oss, enable piplined file gc (#31740)

2024-04-25 20:41:27 +08:00

local_chunk_manager.go

enhance: use WalkWithPrefix api for oss, enable piplined file gc (#31740)

2024-04-25 20:41:27 +08:00

minio_object_storage_test.go

fix: Removed minio bucket after use in test (#32624)

2024-04-28 13:51:26 +08:00

minio_object_storage.go

enhance: use WalkWithPrefix api for oss, enable piplined file gc (#31740)

2024-04-25 20:41:27 +08:00

options.go

enhance: Support MinIO TLS connection (#31311)

2024-03-21 11:15:20 +08:00

OWNERS

…

partition_stats_test.go

feat: Define FieldValue, FieldStats and PartitionStats (#30286)

2024-03-06 20:42:37 -08:00

partition_stats.go

fix: [Cherry-pick]Pick major compaction fixs and optimizations (#34360)

2024-07-03 09:53:37 +08:00

payload_reader_test.go

Update arrow version to v12 (#28425)

2023-11-15 10:36:19 +08:00

payload_reader.go

enhance: Unify data type check APIs for go (#31887)

2024-04-07 14:27:22 +08:00

payload_test.go

enhance: add helpers to parse sparse float vector in JSON (#32543)

2024-04-25 14:47:24 +08:00

payload_writer_test.go

enhance: add type info for payload writer error message and add log when querynode find new collection (#32522)

2024-05-07 14:45:29 +08:00

payload_writer.go

enhance: [2.4] Avoid merging insert data when buffering insert msgs (#34205)

2024-06-27 10:14:05 +08:00

payload.go

enhance: [2.4] Avoid merging insert data when buffering insert msgs (#34205)

2024-06-27 10:14:05 +08:00

pk_statistics.go

enhance: Use BatchPkExist to reduce bloom filter func call cost (#33752)

2024-06-12 17:45:58 +08:00

primary_key_test.go

Use go-api/v2 for milvus-proto (#24770)

2023-06-09 01:28:37 +08:00

primary_key.go

enhance: Remove StringPrimaryKey to reduce unnecessary copy and function call cost (#33486) (#33649)

2024-06-06 10:40:01 +08:00

primary_keys_test.go

enhance: Add PrimaryKeys interface to reduce memory usage (#30405)

2024-02-01 09:57:11 +08:00

primary_keys.go

enhance: Add PrimaryKeys interface to reduce memory usage (#30405)

2024-02-01 09:57:11 +08:00

print_binlog_test.go

enhance: [2.4] Avoid merging insert data when buffering insert msgs (#34205)

2024-06-27 10:14:05 +08:00

print_binlog.go

fix: fix fp16/bf16 some code missing and add more fp16/bf16 test (#31612)

2024-03-28 14:11:10 +08:00

remote_chunk_manager_test.go

enhance: use WalkWithPrefix api for oss, enable piplined file gc (#31740)

2024-04-25 20:41:27 +08:00

remote_chunk_manager.go

enhance: use WalkWithPrefix api for oss, enable piplined file gc (#31740)

2024-04-25 20:41:27 +08:00

serde_test.go

enhance: [cherry-pick] Move compactor into sub package (#34098)

2024-06-25 11:16:02 +08:00

serde.go

fix: [Cherry-pick]Pick major compaction fixs and optimizations (#34360)

2024-07-03 09:53:37 +08:00

stats_test.go

Add retry time when lazy load BF (#25096)

2023-06-25 11:32:43 +08:00

stats.go

feat: [Sparse Float Vector] add sparse vector support to milvus components (#30630)

2024-03-13 14:32:54 -07:00

storage_test.go

enhance: Remove vector chunk manager (#28569)

2023-11-30 18:00:33 +08:00

types.go

enhance: use WalkWithPrefix api for oss, enable piplined file gc (#31740)

2024-04-25 20:41:27 +08:00

unsafe_test.go

…

unsafe.go

…

utils_test.go

enhance: unify data generation test APIs (#32955)

2024-05-14 14:33:33 +08:00

utils.go

fix: fix fp16/bf16 some code missing and add more fp16/bf16 test (#31612)

2024-03-28 14:11:10 +08:00