issue: #42661
Add NotStopped() and IsWorking() methods to shardDelegator for better
state management and error handling.
Changes include:
- Add instance state checking methods with proper error messages
- Replace lifetime package calls with delegator instance methods
- Add comprehensive unit tests for state transitions and error cases
- Improve error reporting with channel name for better debugging
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Related to #41858
This PR add log while debugging schema mismatch between pymilvus cache
and proxy schema.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Related to #39173
`null_bitmap_data()` returns raw pointer of null bitmap of Array. While
after slicing, this bitmap is not rewritten due to zero copy
implementation, so the current start pos maybe non-zero while
FillFieldData generating column `valid_data` array.
This PR add `offset` param for `FillFieldData` method, and force all
invocation pass correct offset of `null_bitmap_data` ptr.
Also update milvus-storage commit fixing reader failed to return data
when buffer size smaller than row group size problem.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Related to #42735
Load field list shall work as hint after tiered storage impl, so the
load list compare is meaningless and block load with empty list after
adding a new field.
This PR totally moves the check logic.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #42561
Move the exclude nodes clearing logic from ExecuteWithRetry to
selectNode after shard leader cache refresh to ensure proper retry
behavior:
- Remove premature exclude clearing in ExecuteWithRetry that happened
before shard leader cache update
- Add exclude clearing logic in selectNode after refreshing shard leader
cache when all replicas are excluded
- Ensure multiple retries can properly update shard leader cache and
clear exclude list when needed
- Add comprehensive tests for edge cases including empty shard leaders
and mixed serviceable node scenarios
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Related to #42723
Previous PR #42684 permit insert msg transformation but insertCodec did
not adapt the same skip logic, whic causes panicking.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
1. add assertion for those index not support int8_vector
2. remove dup testcase
3. skip e2e for L2 updates
related issue: #40698
---------
Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
issue: #42649
- the sync operation of different pchannel is concurrent now.
- add a option to notify the backlog clear automatically.
- make pulsar walimpls can be recovered from backlog exceed.
Signed-off-by: chyezh <chyezh@outlook.com>
issue: #42663
Fix panic issue when processing VchannelInfo messages from older
coordinator versions that don't have DeleteCheckpoint field.
Changes:
- Add null safety check for DeleteCheckpoint before accessing methods
- Maintain backward compatibility with legacy message formats
- Improve seek position selection logic for both old and new versions
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Related to #41858#41951#42084
When insert msg consumer (pipeline/flowgraph) have newer schema than
insertMsg, it have to adapter the insert msg used old schema(missing
newly added field)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Related to #42702
This patch add wait logic for `CheckerController` and nil check for
channel checker in case of panicking during server/testcase stop
procedure
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Related to #39718
This PR makes go milvusclient support `AddCollectionField` API and adds
example code for it.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Related to #42640
The search/query plan holded a reference to schema, which could be
destructed after schema change. This PR make plan hold a shared ptr to
it fixing dangling reference problem under concurrent read & schema
change.
This PR also remove field binlog check for loading index for old segment
with old schema may have binlog lack.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
1. Optimize the import process: skip subsequent steps and mark the task
as complete if the number of imported rows is 0.
2. Improve import integration tests:
a. Add a test to verify that autoIDs are not duplicated
b. Add a test for the corner case where all data is deleted
c. Shorten test execution time
3. Enhance import logging:
a. Print imported segment information upon completion
b. Include file name in failure logs
issue: https://github.com/milvus-io/milvus/issues/42488,
https://github.com/milvus-io/milvus/issues/42518
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Ref https://github.com/milvus-io/milvus/issues/42148
This PR mainly enables segcore to support array of vector (read and
write, but not indexing). Now only float vector as the element type is
supported.
---------
Signed-off-by: SpadeA <tangchenjie1210@gmail.com>
Signed-off-by: SpadeA-Tang <tangchenjie1210@gmail.com>
issue: #42028
W20250522 09:52:55.785657 12779 ChunkedSegmentSealedImpl.cpp:1752]
[SERVER][generate_interim_index][CGO_LOAD][]fail to generate binlog
index, because bad optional access
After the cachelayer is added, num_rows_ can not be obtained before
interim index generated , and an external parameter pass is required
Signed-off-by: foxspy <xianliang.li@zilliz.com>
Defer clone and decompress operations until just before removing from
meta, instead of eagerly applying them to all segments in advance.
issue: https://github.com/milvus-io/milvus/issues/42592
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
related issue: #42604
1. update the test expression for all to L3 for now as it takes too many
hours to complete running. Will improve the performance in next pr.
---------
Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
issue: #42165
Implement dynamic task execution capacity calculation based on QueryNode
CPU core count instead of static configuration for better resource
utilization.
Changes include:
- Add CpuCoreNum() method and WithCpuCoreNum() option to NodeInfo
- Implement GetTaskExecutionCap() for dynamic capacity calculation
- Add QueryNodeTaskParallelismFactor parameter for tuning
- Update proto definition to include cpu_core_num field
- Add unit tests for new functionality
This allows QueryCoord to automatically adjust task parallelism based on
actual hardware resources.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Related to #42629
This PR handles converting default value to int8/int18 scalar with int32
default value definition
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
