yihao.dai a29b3272b0

fix: Improve import memory management to prevent OOM (#43568) ...

1. Use blocking memory allocation to wait until memory becomes available
2. Perform memory allocation at the file level instead of per task
3. Limit Parquet file reader batch size to prevent excessive memory
consumption
4. Limit import buffer size from 20% to 10% of total memory

issue: https://github.com/milvus-io/milvus/issues/43387,
https://github.com/milvus-io/milvus/issues/43131

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

2025-07-28 21:25:35 +08:00

..

hash_test.go

fix: Fix no candidate segments error for small import (#41771)

2025-05-14 15:30:21 +08:00

hash.go

enhance: bulkinsert handles nullable/default (#42127)

2025-05-28 18:02:28 +08:00

memory_allocator_test.go

fix: Improve import memory management to prevent OOM (#43568)

2025-07-28 21:25:35 +08:00

memory_allocator.go

fix: Improve import memory management to prevent OOM (#43568)

2025-07-28 21:25:35 +08:00

mock_task_manager.go

fix: Prevent import from generating small binlogs (#43132)

2025-07-07 21:32:47 +08:00

mock_task.go

fix: Prevent import from generating small binlogs (#43132)

2025-07-07 21:32:47 +08:00

pool_test.go

fix: Prevent import from generating small binlogs (#43132)

2025-07-07 21:32:47 +08:00

pool.go

fix: Prevent import from generating small binlogs (#43132)

2025-07-07 21:32:47 +08:00

scheduler_test.go

fix: Improve import memory management to prevent OOM (#43568)

2025-07-28 21:25:35 +08:00

scheduler.go

fix: Improve import memory management to prevent OOM (#43568)

2025-07-28 21:25:35 +08:00

task_import.go

fix: Improve import memory management to prevent OOM (#43568)

2025-07-28 21:25:35 +08:00

task_l0_import_test.go

enhance: ChunkManager is no longer created during datanode initialization (#42791)

2025-06-17 17:06:38 +08:00

task_l0_import.go

fix: Improve import memory management to prevent OOM (#43568)

2025-07-28 21:25:35 +08:00

task_l0_preimport.go

fix: Improve import memory management to prevent OOM (#43568)

2025-07-28 21:25:35 +08:00

task_manager_test.go

enhance: Use v2 package name for pkg module (#39990)

2025-02-22 23:15:58 +08:00

task_manager.go

enhance: Use v2 package name for pkg module (#39990)

2025-02-22 23:15:58 +08:00

task_preimport.go

fix: Improve import memory management to prevent OOM (#43568)

2025-07-28 21:25:35 +08:00

task.go

fix: Prevent import from generating small binlogs (#43132)

2025-07-07 21:32:47 +08:00

util_test.go

enhance: bulkinsert handle nullable/defaultValue/functionOutput fields (#42956)

2025-07-04 14:20:44 +08:00

util.go

fix: Improve import memory management to prevent OOM (#43568)

2025-07-28 21:25:35 +08:00