issue: #31222
pr: #31256
grpcclient's `call` func return a unrecoverable error, then the caller's
retry policy also breaks due to this unrecoverable error.
This PR introduce `retry.Handle`, the new func use `func() (bool,
error)` as input parameters, which return `shouldRetry` directly, to
avoid grpcclient return a unrecoverable error
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
issue: #30531
pr: #31277
cause get client from `shardClientMgr`, doesn't means query node is
unavailable. because of the ref counter policy in `shardClientMgr`,
which will clean the client, if no collection use qn as shard leader.
This PR fix that set node unreachable when get shard client failed.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
issue: #31351
pr: #31380
This PR fixed that search doesn't expire shard leader cache when send
request to query node failed, which make every request keep trying to
connect a offline query node
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
issue: #31162
pr: #31379
when give scope CurrentTargetFirst/NextTargetFirst, it's expected to
scan both current and next target.
This PR fixed wrong behavior of CurrentTargetFirst/NextTargetFirst in
target manager, which may cause unexpected task generated, and load
collection may stuck forever due to dirty leader view.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Cherry-pick from master
pr: #31388
See also #30806
`formatKey` may cost lots of CPU on string processing under high QPS
scenario, this PR adds a formattedKeys cache preventing string operation
in each param get value.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Cherry pick from master
pr: #31122
See also #31103
Since querycoord need index meta information from datacoord only, broker
shall use `ListIndexes` to skip segment index building check logic in
datacoord
This PR is also related to #30538, in which DescribeIndex caused lots of
memory usage and lead to OOM eventually
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Cherry-pick from master
pr: #31104
See also #31103
This PR add `listIndexes` API for datacoor server to list all indexes
for provided collection.
Comparing to the existing `DescribeIndex` API, the new one does NOT
check the segment index building progress to ease the burden when
invoking it
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
pr: https://github.com/milvus-io/milvus/pull/31097
* When the collection is created using an SDK and includes array and
JSON datatypes in the schema, data can be inserted using the RESTful
API.
* When the collection is created using the RESTful API and includes JSON
and array datatypes in dynamic fields, data can also be inserted using
the RESTful API.
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
See also #31024#31058
Flush cost boosted from 2 seconds to 5 or more after the change of
channel updater. This PR add a manual trigger method to accelerate flush
procedure.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Cherry pick from master
pr: #31021
- Change load field log from "dy pool" to "load pool"
- Also defer delete when there is no error
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
This PR includes the following adjustments:
1. To prevent channelCP update task backlog, only one task with the same
vchannel is retained in the updater. Additionally, the lastUpdateTime is
refreshed after the flowgraph submits the update task, rather than in
the callBack function.
2. Batch updates of multiple vchannel checkpoints are performed in the
UpdateChannelCheckpoint RPC (default batch size is 128). Additionally,
the lock for channelCPs in DataCoord meta has been switched from key
lock to global lock.
3. The concurrency of UpdateChannelCheckpoint RPCs in the datanode has
been reduced from 1000 to 10.
issue: https://github.com/milvus-io/milvus/issues/30004
pr: https://github.com/milvus-io/milvus/pull/30941
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Cherry-pick from master
pr: #30994
Segment load memory usage is underestimated due to removing the load
memroy factor. This PR adds it back to protect querynode OOM during some
extreme memory cases.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
If DC restarted, those unkonwn compaction tasks
will never get call back in DN, so that the segments in the compaction
task will be locked, unable to sync and compaction again, blocking cp
advance and compaction executing.
See also: #30137
pr: #30850
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
issue: #30723
pr: #30724
This PR skip generate balance task when collection's target isn't ready.
also refine the check stale logic in query coord's scheduler, if channel
exist in current or next target, task won't be canceled.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
pr: #30654
This PR add retry on all interface which belong to indexcoord in milvus
2.2 and. move to data coord in milvus 2.3, to prevent meet unimplemented
error during rolling upgrade from milvus 2.2 to 2.3.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
master pr: #30871 issue: #30870
fix: vector field cannot be empty while insert
did a check whether the vector field is empty in advance
master pr: #30740
fix:
1. spelling mistake about metricsType #30643
2. int64 percious #20415
3. insert into collection which has multi vector fields #30674
enhance: support dataType: Float16Vector & BFloat16Vector #22837
#30980(master pr: #30969)
enhance: describe collection will show the field is partition key or not
#30789
---------
Signed-off-by: PowderLi <min.li@zilliz.com>
issue: #30950
pr: #30951
due to segment version doesn't update as expected.
This PR will update segment version until segment become loaded
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
