Increase insert buffer size from 16MB to 64MB, while keeping delete
buffer size at 16MB.
issue: https://github.com/milvus-io/milvus/issues/42518
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Each cell of a field should be written to its own mmap file, rather than
writing all cells of the field into a single mmap file.
related: #39173
---------
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
the manager's logging lambda should not capture the pipeline object
this creates a circular reference between the manager and the pipeline
object, making it impossible for both to be GC-ed.
issue: https://github.com/milvus-io/milvus/issues/42581
Signed-off-by: Buqian Zheng <buqianzheng@Buqians-MacBook-Air.local>
Co-authored-by: Buqian Zheng <buqianzheng@Buqians-MacBook-Air.local>
Previous PR: #42017 did not work due to following updated points by this
PR:
- Initialize the `name_map`, which not touched at all before
- Trim the thread name under 15 characters to fit syscall limit
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #42523
Resolve deadlock issue in GroupedAllocator where resource release
notifications fail to propagate across hierarchy levels, causing child
allocators to wait indefinitely.
Changes include:
- Add recursive notify() method to GroupedAllocator
- Ensure all child allocators receive resource release notifications
- Fix TOCTOU race condition in hierarchical resource management
This resolves the intermittent test timeout in TestGroupedAllocator test
that exhibited deadlock due to missed condition notifications.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Related to #42530
The cluster id is missing when drop worker drop causing redoing task on
report duplicated task error.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #42498
- fix: sealed segment cannot be flushed after upgrading
- fix: get mvcc panic when upgrading
- ignore the L0 segment when graceful stop of querynode.
---------
Signed-off-by: chyezh <chyezh@outlook.com>
#41918#42482#42049#42513
cherrypick: sn release memory after pop from heap
wp: Encapsulate buffer and chan into a single item for one-to-one
management and cleanup
Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
Co-authored-by: chyezh <chyezh@outlook.com>
Remove the unlimited logID mechanism and switch to redundantly
allocating a large number of IDs.
issue: https://github.com/milvus-io/milvus/issues/42518
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
issue: #42492
- consider the old RO query node (not streaming node) when balancing
channel.
- querynode graceful stop can be done if there's only L0 segment exists.
Signed-off-by: chyezh <chyezh@outlook.com>
Related to #42489
See also #41435
This PR's main target is to make partial load field list work as caching
layer warmup policy hint. If user specify load field list, the fields
not included in the list shall use `disabled` warmup policy and be able
to lazily loaded if any read op uses them.
The major changes are listed here:
- Pass load list to segcore and creating collection&schema
- Add util functions to check field shall be proactively loaded
- Adapt storage v2 column group, which may lead to hint fail if columns
share same group
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Remove the hardcoded batchSize of 100,000 and instead trigger a write
every 64MB based on actual data size. This prevents sort stats from
generating excessively large binlog files.
issue: https://github.com/milvus-io/milvus/issues/42400
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Remove the 'goccy/go-json' library, which was inadvertently introduced,
and revert to using the standard internal JSON handling.
Changes include:
- Removed dependency on 'github.com/goccy/go-json' in go.mod and go.sum.
- Replaced import of 'goccy/go-json' with 'internal/json' in
'internal/querycoordv2/task/scheduler.go'.
This correction ensures the project continues to use the intended JSON
processing libraries and avoids unnecessary external dependencies.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
# PR Summary
This PR resolves the deprecation warnings of the `logger` library:
```python
DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead
```
Signed-off-by: Emmanuel Ferdman <emmanuelferdman@gmail.com>
issue: #42457
Replace unsafe ExpectedCalls modification with atomic.Int32 state
tracking to avoid race conditions in concurrent test execution. Changes
include:
- Use atomic counters instead of direct mock ExpectedCalls manipulation
- Add RunAndReturn with atomic state transitions for thread safety
- Remove github.com/samber/lo dependency
This prevents data race when mock framework and test goroutines access
ExpectedCalls concurrently.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Related to #39173#39718
In storage v2, the `lack_bin_rows` cannot be used since field id is not
column group id, which will not be matched forever.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #42176
Remove the mutual exclusion constraints between channel and segment
balance tasks to allow them to run concurrently.
Changes include:
- Remove permitBalanceChannel() and permitBalanceSegment() methods from
RoundRobinBalancer
- Update ChannelLevelScoreBalancer, MultiTargetBalancer,
RowCountBasedBalancer, and ScoreBasedBalancer to remove constraint
checks
- Allow segment balance tasks to proceed even when channel balance tasks
are running
- Update test cases to reflect new behavior where balance tasks no
longer block each other
This change improves the efficiency of load balancing by removing
unnecessary coordination overhead between different types of balance
operations.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Related to #39173
This PR
- Handle storage v2 log path in local storage mode on querynode
- Ignore field info check when append index for loaded sealed segment
when using storage v2
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #41976
- make drop partition message as a broadcast message.
- add gc when drop partition message is acked.
- add a call back to handle the broadcast message when ack.
- the ack operation of broadcast message will retry until success.
Signed-off-by: chyezh <chyezh@outlook.com>
