mirror of
https://gitee.com/milvus-io/milvus.git
synced 2026-02-02 01:06:41 +08:00
88e761adc9
Add a thread pool to load BM25 stats and save them to local disk during loading, reducing peak memory usage. relate: https://github.com/milvus-io/milvus/issues/41424 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> - Core invariant: sealed-segment BM25 statistics are immutable and may be persisted to disk independently from growing-segment stats; IDF state is reconstructed by combining persisted sealed stats (preloaded from disk) with in-memory growing stats at runtime (preloadSealed + RegisterGrowing). - Capability added: asynchronous BM25 loading via a dedicated BM25LoadPool (config key common.threadCoreCoefficient.bm25Load and param BM25LoadThreadCoreCoefficient) — delegator.loadBM25Stats is executed on the pool to load sealed BM25 stats and call idfOracle.RegisterSealed, reducing peak memory during segment load. - Logic removed/simplified and why: the previous single Register(segID, stats, state) + background local-writer/spool loop was split into RegisterGrowing (in-memory) and RegisterSealed (sealed + on-disk persistence) and the localloop removed; RegisterSealed writes sealed stats directly (ToLocal) and uses singleflight to deduplicate, eliminating redundant spooling and lifecycle complexity while clarifying sealed vs growing flows. - Why this does NOT introduce data loss or behavior regression: sealed stats are still written and reloaded (RegisterSealed persists to dirPath; preloadSealed merges disk and memory on first load), growing segments continue to register in-memory via RegisterGrowing, loadSegments now defers sealed BM25 handling to loadBM25Stats but still registers sealed candidates after load, and tests were updated to reflect RegisterSealed/RegisterGrowing usage—so serialization/deserialization, preload semantics, and test coverage preserve existing persisted state and runtime behavior. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>