1. add assertion for those index not support int8_vector
2. remove dup testcase
3. skip e2e for L2 updates
related issue: #40698
---------
Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
issue: #42649
- the sync operation of different pchannel is concurrent now.
- add a option to notify the backlog clear automatically.
- make pulsar walimpls can be recovered from backlog exceed.
Signed-off-by: chyezh <chyezh@outlook.com>
issue: #42663
Fix panic issue when processing VchannelInfo messages from older
coordinator versions that don't have DeleteCheckpoint field.
Changes:
- Add null safety check for DeleteCheckpoint before accessing methods
- Maintain backward compatibility with legacy message formats
- Improve seek position selection logic for both old and new versions
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Related to #41858#41951#42084
When insert msg consumer (pipeline/flowgraph) have newer schema than
insertMsg, it have to adapter the insert msg used old schema(missing
newly added field)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Related to #42702
This patch add wait logic for `CheckerController` and nil check for
channel checker in case of panicking during server/testcase stop
procedure
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Related to #39718
This PR makes go milvusclient support `AddCollectionField` API and adds
example code for it.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Related to #42640
The search/query plan holded a reference to schema, which could be
destructed after schema change. This PR make plan hold a shared ptr to
it fixing dangling reference problem under concurrent read & schema
change.
This PR also remove field binlog check for loading index for old segment
with old schema may have binlog lack.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
1. Optimize the import process: skip subsequent steps and mark the task
as complete if the number of imported rows is 0.
2. Improve import integration tests:
a. Add a test to verify that autoIDs are not duplicated
b. Add a test for the corner case where all data is deleted
c. Shorten test execution time
3. Enhance import logging:
a. Print imported segment information upon completion
b. Include file name in failure logs
issue: https://github.com/milvus-io/milvus/issues/42488,
https://github.com/milvus-io/milvus/issues/42518
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Ref https://github.com/milvus-io/milvus/issues/42148
This PR mainly enables segcore to support array of vector (read and
write, but not indexing). Now only float vector as the element type is
supported.
---------
Signed-off-by: SpadeA <tangchenjie1210@gmail.com>
Signed-off-by: SpadeA-Tang <tangchenjie1210@gmail.com>
issue: #42028
W20250522 09:52:55.785657 12779 ChunkedSegmentSealedImpl.cpp:1752]
[SERVER][generate_interim_index][CGO_LOAD][]fail to generate binlog
index, because bad optional access
After the cachelayer is added, num_rows_ can not be obtained before
interim index generated , and an external parameter pass is required
Signed-off-by: foxspy <xianliang.li@zilliz.com>
Defer clone and decompress operations until just before removing from
meta, instead of eagerly applying them to all segments in advance.
issue: https://github.com/milvus-io/milvus/issues/42592
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
related issue: #42604
1. update the test expression for all to L3 for now as it takes too many
hours to complete running. Will improve the performance in next pr.
---------
Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
issue: #42165
Implement dynamic task execution capacity calculation based on QueryNode
CPU core count instead of static configuration for better resource
utilization.
Changes include:
- Add CpuCoreNum() method and WithCpuCoreNum() option to NodeInfo
- Implement GetTaskExecutionCap() for dynamic capacity calculation
- Add QueryNodeTaskParallelismFactor parameter for tuning
- Update proto definition to include cpu_core_num field
- Add unit tests for new functionality
This allows QueryCoord to automatically adjust task parallelism based on
actual hardware resources.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Related to #42629
This PR handles converting default value to int8/int18 scalar with int32
default value definition
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #42028
- limit the concurrency of zstd compression.
- zstd.go modified from
`github.com/apache/arrow/go/v17/parquet/compress/ztsd.go`
- may be related to #42129
Signed-off-by: chyezh <chyezh@outlook.com>
issue: #39333#41570
Fix delete record missing issue introduced in PR #39552 L0 refactoring:
- Use delete checkpoint as consume start position when deleteCP <
channelCP
- Add logging when delete checkpoint is used instead of seek position
- Prevent delete record loss when deleteCP is earlier than default
channelCP
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
issue #42492
- streamingcoord start before old rootcoord.
- streaming balancer will check the node session synchronously to avoid
redundant operation when cluster startup.
- ddl operation will check if streaming enabled, if the streaming is not
enabled, it will use msgstream.
- msgstream will initialize if streaming is not enabled, and stop when
streaming is enabled.
---------
Signed-off-by: chyezh <chyezh@outlook.com>
Related to #39173
Like logic in #41919, storage v2 fs shall use complete paths with
bucketName prefix to be compatible with its definition. This PR fills
bucket name from config when creating reader for compaction tasks.
NOTE: the bucket name shall be read from task params config for
compaction task pooling.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Related to #39173
`nullable` flag is crucial for serde logic of v2 writer, missing this
flag causes logic bug for v2 nullalbe data.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #42098#42404
related to: ##42009 #41937
Implement new method to handle partition removal from next target
without directly modifying current target.
Changes include:
- Add RemovePartitionFromNextTarget method and deprecate RemovePartition
- Update target_observer to use new method for ReleasePartition
operations
- Add unit tests and mock methods for new functionality
This ensures that all changes to next target will propagates to
delegator's query view.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
