19334 Commits

Author SHA1 Message Date
nico
a95e1df7db
enhance: [skip e2e]modify 2.4 branch nihgtly groovy (#31398)
Signed-off-by: nico <cheng.yuan@zilliz.com>
2024-03-19 15:45:06 +08:00
wei liu
f4449d4ef4
fix: Wrong behavior of CurrentTargetFirst/NextTargetFirst in target manager (#31378)
issue: #31162
pr: #31379

when give scope CurrentTargetFirst/NextTargetFirst, it's expected to
scan both current and next target.

This PR fixed wrong behavior of CurrentTargetFirst/NextTargetFirst in
target manager, which may cause unexpected task generated, and load
collection may stuck forever due to dirty leader view.

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-03-19 11:41:05 +08:00
XuanYang-cn
8946aa10d4
enhance: [cherry-pick]Skip submmit empty l0 tasks in DC (#31338)
pr: #31280

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-03-19 10:15:12 +08:00
wei liu
0dbc0f2c11
fix: Hybrid search doesn't expire shard leader cache (#31381)
issue: #31351
pr: #31380
This PR fixed that search/hybrid_search doesn't expire shard leader
cache when send request to query node failed, which make every request
keep trying to connect a offline query node

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-03-19 10:05:05 +08:00
congqixia
3254a14319
fix: [Cherry-pick] Cleanup write buffer when flowgraph released (#31377)
Cherry-pick from master
pr: #31376
See also #30137

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-03-18 23:47:06 +08:00
Xiaofan
b2b107a774
fix: [cherry-pick] get compaction failure when datanode is actually live (#31356)
if get compaction result failed, then skip processing compaction 
pr: #31353
see also #31352

Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
2024-03-18 22:53:05 +08:00
yihao.dai
72acd1d676
feat: Add RBAC functionality to ImportV2 (#31298) (#31337)
issue: https://github.com/milvus-io/milvus/issues/31282,
https://github.com/milvus-io/milvus/issues/28521

pr: https://github.com/milvus-io/milvus/pull/31298

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-03-18 22:29:05 +08:00
Bingyi Sun
e7b053817d
feat: Add global mmap enable configuration (#31267) (#31373)
https://github.com/milvus-io/milvus/issues/31279
related pr: #31267

Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-03-18 21:57:05 +08:00
congqixia
89cff29b6a
enhance: [Cherry-pick] Use different interval for gc scan (#31364)
Cherry-pick from master
pr: #31363
See also #31362

This PR make datacoord garbage collection scan operation using differet
interval than other opeartion.

This interval is a newly added param item, which default value is 7*24
hours.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-03-18 21:43:05 +08:00
wei liu
7ee7d484cc
fix: Transfer l0 segment to new delegator after balance (#31332)
issue: #30186
pr: #31319

during channel balance, after new delegator loaded, instead of syncing
l0 segment's location to new delegator, we should load l0 segment on new
delegator, and release the old l0 segment, then start to release old
delegator.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-03-18 20:49:04 +08:00
yihao.dai
1885d176cf
fix: Return specific error code when encountering incomplete requery results (#31343) (#31359)
During requery, segments may change (e.g., due to compaction), so we
need to return specific error codes when encountering incomplete requery
results. Clients can then retry to avoid this issue.

issue: https://github.com/milvus-io/milvus/issues/29656

pr: https://github.com/milvus-io/milvus/pull/31343

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-03-18 19:43:06 +08:00
congqixia
25d1c0e04b
enhance: Bump milvus & milvus-proto to 2.4.0-rc.1 (#31371)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-03-18 19:07:05 +08:00
sammy.huang
b773581fde
enhance: fetch simdjson directly in the format of targz (#31370)
pr: #31369

Signed-off-by: Liang Huang <sammy.huang@zilliz.com>
2024-03-18 18:55:10 +08:00
congqixia
4e83ca46bc
fix: [cherry-pick] Report offline info in GetDataDistribition (#31349)
Cherry pick from master
pr: #31347
See also #31345

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-03-18 17:05:05 +08:00
PowderLi
f863f58785
fix: [restful v2] add hybrid_search (#31344)
master old pr: #31113
(will updated after 2.4-rc released)

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-03-18 16:51:06 +08:00
yihao.dai
3c3be49cf4
fix: Fix binlog import (#31310) (#31330)
Fix binlog import functionality by removing the existing check and
refining the size retrieval process.

issue: https://github.com/milvus-io/milvus/issues/31221,
https://github.com/milvus-io/milvus/issues/28521

pr: https://github.com/milvus-io/milvus/pull/31310

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-03-18 15:41:05 +08:00
liliu-z
fdb3231151
enhance: Upgrade Knowhere (#31308)
/kind improvement

Signed-off-by: Li Liu <li.liu@zilliz.com>
2024-03-18 14:21:04 +08:00
wei liu
555d96ff05
fix: Unstable ut TestGetClientFailed (#31296) (#31335)
issue: #31295
pr: #31335

This PR fix unstable ut TestGetClientFailed

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-03-18 13:51:04 +08:00
wei liu
3987cd69d7
fix: save current target after target observer stop (#31333)
issue: #28491
pr: #31315

should save target to meta store after target observer stop, incase of
target changed

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-03-18 13:49:11 +08:00
cai.zhang
85a1b6c96e
enhance: When describing an index, fetch the index info in batches (#31305)
issue: #29313
master pr: #31238

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-03-18 12:31:05 +08:00
zhagnlu
7e3ac66617
fix: fix wrong guaranteeTs for hybrid search (#31329)
pr: #31302

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-03-18 12:29:04 +08:00
sammy.huang
506534c29d
enhance: [skip e2e]change Semantics from 2.3.x to 2.X.X (#31348)
pr: #31350

Signed-off-by: Liang Huang <sammy.huang@zilliz.com>
2024-03-18 12:03:04 +08:00
PowderLi
f2814f2eea
fix: [CHERRY-PICK] [restful v2] create a collection with vector field name (#31318)
fix: [restful v2] create a collection with vector field name
master pr: #31317

1. quickly create collection, create an index on the vector field #31149
2. valid consistencyLevel is needed, while create a collection
3. show collection's properties and aliases #31180
4. list aliase for one collection, default is the whole database

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-03-17 20:07:04 +08:00
Buqian Zheng
99e0d28c9e
fix: [2.4 sparse float vector] support the new streaming deserialize reader (#31326)
issue: https://github.com/milvus-io/milvus/issues/31324

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-03-16 23:58:22 -07:00
congqixia
08aba2e05f
fix: Remove QueryNodeEntitiesSize after segment/collection released (#31290)
See also #31289

This PR:
- Set collection level `QueryNodeEntitiesSize` to zero if all segment
released
- Delete `QueryNodeEntitiesSize` metrics value after collection ref is
zero

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-03-15 15:43:04 +08:00
chyezh
465fd474de
enhance: add syncutil type ContextCond and VersionedNotifier (#30648)
issue: #30647

- ContextCond is a broadcast-only condition variable which can be
canceled by context.

- VersionedNotifier is a version-based notifier-listener implementation,
which promise no change can be ignored.

Signed-off-by: chyezh <chyezh@outlook.com>
2024-03-15 15:41:04 +08:00
Gao
038c570ef3
enhance: upgrade folly to run on arm (#31284)
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-03-15 15:39:03 +08:00
Gao
6f1e9cd0f4
fix: fix cal_distance default implementation (#31245)
#31243

Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-03-15 15:37:11 +08:00
elstic
93bb04f884
test: add some mmap cases (#31211)
issue : https://github.com/milvus-io/milvus/issues/30940

Signed-off-by: elstic <hao.wang@zilliz.com>
2024-03-15 15:25:05 +08:00
nico
9e7ce0a8f7
test: update test cases (#31266)
Signed-off-by: nico <cheng.yuan@zilliz.com>
2024-03-15 15:23:10 +08:00
yanliang567
8563c4a5ee
enhance: Update groupby tests (#31297)
related issue: #29883 
skip running for now

Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
2024-03-15 15:21:03 +08:00
yihao.dai
c408a32db6
feat: Add disk quota checks for import V2 (#31131)
Return quota error when the files to be imported exceed the disk quota.

issue: https://github.com/milvus-io/milvus/issues/28521

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-03-15 14:43:03 +08:00
wei liu
d79aa58b37
enhance: Speed up target recovery after query coord restart (#31240)
issue: #28491

after querycoord restart, it will pull a new target, which include
channel and segment list. when segments loaded on querynode has reached
the target, the collection could provide search/query. but if segment
list changes by time, ater querycoord pull a new target, it will takes a
few minutes to catch up the target's segment distribution. and before
that, query/search will fail due to lack of segments.

This PR save the current loaded target to meta storein querycoord's stop
progress, and recover it when query coord starts, to speed up the target
recovery time.

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-03-15 14:19:03 +08:00
aoiasd
b724753137
enhance: Add runtime config to paramtable (#31006)
relate: https://github.com/milvus-io/milvus/issues/30806
Avoid use string convert or format function when get some runtime
parameter

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-03-15 11:07:06 +08:00
yihao.dai
2b035ba2d4
enhance: Allow import tasks to retry for more errors (#31268)
Allow import tasks to retry for a wider range of errors, including all
gRPC errors and unexpected status codes from Milvus.

issue: https://github.com/milvus-io/milvus/issues/31227,
https://github.com/milvus-io/milvus/issues/28521

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-03-15 11:05:04 +08:00
yihao.dai
811316d2ba
fix: Fix binlog import and refine error reporting (#31241)
1. Fix binlog import with partition key.
2. Refine binlog import error reportins.
3. Avoid division by zero when retrieving import progress.

issue: https://github.com/milvus-io/milvus/issues/31221,
https://github.com/milvus-io/milvus/issues/28521

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-03-15 10:55:05 +08:00
chyezh
ff4237bb90
enhance: add hostname into node info (#30673)
issue: https://github.com/milvus-io/milvus/issues/30647

- Address may be reused in k8s environment. Using hostname can be
better.

Signed-off-by: chyezh <chyezh@outlook.com>
2024-03-15 10:45:06 +08:00
jaime
db79be3ae0
fix: ctx cancel should be the last step while stopping server (#31220)
issue: #31219

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-03-15 10:33:05 +08:00
wei liu
ca8eee2c47
fix: Set node unreachable when get shard client failed (#31277)
issue: #30531

cause get client from `shardClientMgr`, doesn't means query node is
unavailable. because of the ref counter policy in `shardClientMgr`,
which will clean the client, if no collection use qn as shard leader.

This PR fix that set node unreachable when get shard client failed.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-03-15 10:23:03 +08:00
XuanYang-cn
a1386bae7f
fix: Skip to submit l0 tasks when scheduler full (#31270)
See also: #31242

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-03-15 10:21:12 +08:00
wei liu
147a3b8bdc
fix: Grpcclient return unrecoverable error (#31256)
issue: #31222

grpcclient's `call` func return a unrecoverable error, then the caller's
retry policy also breaks due to this unrecoverable error.

This PR introduce `retry.Handle`, the new func use `func() (bool,
error)` as input parameters, which return `shouldRetry` directly, to
avoid grpcclient return a unrecoverable error

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-03-15 10:03:05 +08:00
smellthemoon
056e9f0cf5
fix: unmarshal nil when check search params (#31139)
#31020

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-03-15 09:15:04 +08:00
yihao.dai
7d7ef388df
enhance: Remove adding import segments to the datanode (#31244)
With the presence of L0 segments, there's no longer a need to add import
segments to the datanode.

issue: https://github.com/milvus-io/milvus/issues/28521

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-03-15 06:53:03 +08:00
PowderLi
afa4193fc2
fix: [restful v2] create collection & describe collection (#31269)
1. complete the output of describe collection #31180
2. add a new parameter `params` for create collection #31181 #31149
3. create index according to indexConfig while create collection #31199
4. autoID: false while create a collection quickly, take upsert into
consideration
5. polish error information while search without annsField #31010


fix: [restful v1] insert / upsert with default partitionName JIRA-4952

---------

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-03-14 23:27:03 +08:00
XuanYang-cn
a52a52064d
fix: Use lock and map instead of concurrentMap (#31212)
See also: #31209

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-03-14 18:39:04 +08:00
congqixia
773c64ecbb
fix: Set nodeID when remove distribution (#31259)
See also #30930

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-03-14 15:09:03 +08:00
Chun Han
6939ad15f2
fix:possible out-of-bound due to groupby when reduing(#30711) (#31200)
related: #30711

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-03-14 13:07:03 +08:00
congqixia
b1ff8e2749
fix: Fix data race on loadSealedSegmentFields opts (#31230)
See also #31229

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-03-14 10:43:04 +08:00
XuanYang-cn
1fafc72077
fix: Correct the last empty l0 views (#31198)
See also: #31191

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-03-14 10:31:04 +08:00
sre-ci-robot
126bb52f3d
[automated] Update Pytest image changes (#31236)
Update Pytest image changes
See changes:
652b866587
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-03-14 09:59:10 +08:00