issue: #42162
- enhance: add read ahead buffer size issue #42129
- fix: rocksmq consumer's close operation may get stucked
- fix: growing segment from old arch is not flushed after upgrading
---------
Signed-off-by: chyezh <chyezh@outlook.com>
issue: https://github.com/milvus-io/milvus/issues/41690
This commit implements partial search result functionality when query
nodes go down, improving system availability during node failures. The
changes include:
- Enhanced load balancing in proxy (lb_policy.go) to handle node
failures with retry support
- Added partial search result capability in querynode delegator and
distribution logic
- Implemented tests for various partial result scenarios when nodes go
down
- Added metrics to track partial search results in querynode_metrics.go
- Updated parameter configuration to support partial result required
data ratio
- Replaced old partial_search_test.go with more comprehensive
partial_result_on_node_down_test.go
- Updated proto definitions and improved retry logic
These changes improve query resilience by returning partial results to
users when some query nodes are unavailable, ensuring that queries don't
completely fail when a portion of data remains accessible.
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Related to #40756
Large nq will naturally increase query time, which causing lots of slow
log when user NQ numbers are very large.
This PR make slow search counts span per nq (using avg val) to decide
whether one request is slow or not.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Optimize session management by reducing the TTL (Time To Live) value for
service registration from 30 seconds to 10 seconds. This change improves
the system's ability to detect service failures more quickly and
enhances overall cluster responsiveness.
Changes include:
- Update default session TTL from 30s to 10s in milvus.yaml
- Adjust DefaultSessionTTL constant from 30 to 10 seconds
- Update SessionTTL default value from 60 to 10 seconds
- Maintain consistent TTL values across configuration files
This optimization reduces the time required for the system to detect
when services become unavailable, leading to faster failover and
improved cluster stability during node failures or network issues.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
issue: #41874
- Optimize balance_checker to support balancing multiple collections
simultaneously
- Add new parameters for segment and channel balancing batch sizes
- Add enableBalanceOnMultipleCollections parameter
- Update tests for balance checker
This change improves resource utilization by allowing the system to
balance multiple collections in a single trigger with configurable batch
sizes.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
1. Add global scheduler for datacoord.
2. Define and implement new CreateTask, QueryTask, DropTask interfaces.
3. Refine Import, Compaction, Stats, Index task.
issue: https://github.com/milvus-io/milvus/issues/41123
Co-authored-by: Cai Zhang <cai.zhang@zilliz.com>
See #36264
In this PR:
- Enhanced error handling in parse of grouping field.
- Fixed null handling in reduce tasks in proxy nodes.
- Updated tests to reflect changes in error handling and data processing
logic.
---------
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
issue: #41816
pr #37983 introduced an issue, if doesn't specified
`defaultRootPassword` in milvus.yaml, then `"Milvus"` will be used as
default password for root user, instead of `Milvus`.
This PR fix the unexpected password for root, and add comment for case
which use large numeric password requires double quotes.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
issue: #41544
- Implement in-memory shard manager to maintain the shard state at write
ahead.
- Remove all rpc and meta operation at write ahead, make the segment
assignment logic only use wal and memory.
- Refactor global stats management, add node-level flush policy.
- Fix the recovery storage inconsistency bug when graceful close.
Signed-off-by: chyezh <chyezh@outlook.com>
issue: #41544
- simplify the proto message for flush and create segment.
- simplify the msg handler for flowgraph.
---------
Signed-off-by: chyezh <chyezh@outlook.com>
issue: https://github.com/milvus-io/milvus/issues/41435
this PR is based on https://github.com/milvus-io/milvus/pull/41436.
Improvements include:
- Lazy Load support for Storage v1
- Use Low/High watermark to control eviction
- Caching Layer related config changes
- Removed ChunkCache related configs and code in golang
- Add `PinAllCells` helper method to CacheSlot class
- Modified ValueAt, RawAt, PrimitiveRawAt to Bulk version, to reduce
caching layer overhead
- Removed some unclear templated bulk_subscript methods
- CachedSearchIterator to store PinWrapper when searching on
ChunkedColumn, and removed unused contrustor.
---------
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
This PR fixes#41384 .
When using milvus client and compile on windows, the compilation failed
with the undefined RSS error.
On windows, the way to get memory used is the same as on darwin.
Signed-off-by: Julien Salleyron <julien.salleyron@gmail.com>
Merge RootCoord, DataCoord And QueryCoord into MixCoord
Make Session into one
issue : https://github.com/milvus-io/milvus/issues/37764
---------
Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
The restful API default timeout was hard-coded. This PR make this
timeout value configurable via paramtable.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
The traceID is not initialized by client_request_id in context. If the
client sent valid traceID, milvus log will print two different traceID
which is wierd.
This PR add the logic to tray parsing incoming `client_request_id` into
traceID. If it works just use it the request traceID, otherwise set it
to a different field named `client_request_id`.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
