Cherry-pick from master
pr: #42920
Related to #42918
Previously the `CollectionTtlTimestamp` could be overflowed when the
guarantee_ts==1, which means using `Eventually` consistency level.
This patch use task timestamp, allocated by scheduler, to generate ttl
timestamp ignore the potential very small timestamp being used.
Also add overflow check for ttl timestamp calculated.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
the manager's logging lambda should not capture the pipeline object
this creates a circular reference between the manager and the pipeline
object, making it impossible for both to be GC-ed.
issue: https://github.com/milvus-io/milvus/issues/42581
pr: https://github.com/milvus-io/milvus/pull/42582
Signed-off-by: Buqian Zheng <buqianzheng@Buqians-MacBook-Air.local>
Co-authored-by: Buqian Zheng <buqianzheng@Buqians-MacBook-Air.local>
issue: #42568
Fix incorrect delegator selection during segment release process which
introduced by pr #42410
- Add serviceable filter to prioritize available shard leaders
- Fix fallback logic with channel-specific lookup
- Add early return when no leader found
- Add comprehensive unit tests for all scenarios
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
issue: #42176
pr: #42177
Remove the mutual exclusion constraints between channel and segment
balance tasks to allow them to run concurrently.
Changes include:
- Remove permitBalanceChannel() and permitBalanceSegment() methods from
RoundRobinBalancer
- Update ChannelLevelScoreBalancer, MultiTargetBalancer,
RowCountBasedBalancer, and ScoreBasedBalancer to remove constraint
checks
- Allow segment balance tasks to proceed even when channel balance tasks
are running
- Update test cases to reflect new behavior where balance tasks no
longer block each other
- Improve error handling in task executor by preferring serviceable
shard leaders for segment release operations
- Add fallback logic to find latest shard leader when serviceable leader
is not available
This change improves the efficiency of load balancing by removing
unnecessary coordination overhead between different types of balance
operations.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
issue: #42132
pr: #42136
When calling knowhere-iterator, the thread pool control of knowhere is
no longer needed.
Signed-off-by: min.tian <min.tian.cn@gmail.com>
issue: #41874
pr: #41875
- Optimize balance_checker to support balancing multiple collections
simultaneously
- Add new parameters for segment and channel balancing batch sizes
- Add enableBalanceOnMultipleCollections parameter
- Update tests for balance checker
This change improves resource utilization by allowing the system to
balance multiple collections in a single trigger with configurable batch
sizes.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Cherry-pick from master
pr: #42109
Related to #40756
Large nq will naturally increase query time, which causing lots of slow
log when user NQ numbers are very large.
This PR make slow search counts span per nq (using avg val) to decide
whether one request is slow or not.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Cherry-pick from master
pr: #42095
The absent nullable field shall be filled at server-side in nullable
design. While the implementation here was buggy causing the feature was
not able to serve.
This PR make proxy fill the field data in correct format so that field
data with absent column(s) will be accepted.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Assign the import task to the worker with the most available slots, even
if availableSlots < requiredSlots. This ensures tasks won’t be blocked
indefinitely.
issue: https://github.com/milvus-io/milvus/issues/41981
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
