admin/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2026-02-02 01:06:41 +08:00
Code Issues Packages Projects Releases Wiki Activity
20,474 Commits 146 Branches 196 Tags
Commit Graph

9242 Commits

Author SHA1 Message Date
congqixia
dde79796da
fix: [hotfix] Load original key if ts is MaxTimestamp (#36935) 
Cherry pick from master
pr: #36934

Related to #36933

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-10-16 21:39:24 +08:00
SimFG
0fe549adf1
enhance: [2.4] gc in the snapshot kv (#36793) 
- issue: #36770
- pr: #36792

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-10-11 22:13:21 +08:00
jaime
43b48a0f9b
enhance: skip alter operation when no change are detected (#36786) 
issue: #36784 
pr: #36785

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-10-11 21:43:21 +08:00
jaime
4110f35f1a
fix: may exceed max tnx in etcd operations (#36773) 
issue: #36772
pr: #36775

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-10-11 19:51:22 +08:00
congqixia
a57626afa0
fix: [2.4] Use metrics values instead of hardcode mock values (#36783) 
Cherry-pick from master
pr: #36782
Resolves #36781

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-10-11 19:49:21 +08:00
cqy123456
58b198c785
enhance: [2.4]support sparse vector mmap in growing segment type (#36565) 
issue: https://github.com/milvus-io/milvus/issues/32984
related pr: https://github.com/milvus-io/milvus/pull/36566

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2024-10-11 17:23:21 +08:00
XuanYang-cn
e976b41f97
fix: Remove enableLevelZeroSegment config (#36507) 
See also: #36504
pr: #36535

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-10-11 16:41:21 +08:00
XuanYang-cn
28de6b86ba
fix: [24]fail to init fg clears flushTs so that slows flush (#36741) 
See also: #36709
pr: #36740

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-10-11 16:39:21 +08:00
yihao.dai
7ee46c582e
enhance: Support new RESTful URL to describe import job (#36753) (#36754) 
Support the new RESTful URL for retrieving/describing import progress:
`/v2/vectordb/jobs/import/describe`.
Deprecate the old URL: `/v2/vectordb/jobs/import/get_progress`.

issue: https://github.com/milvus-io/milvus/issues/36752

pr: https://github.com/milvus-io/milvus/pull/36753

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-10-11 16:03:22 +08:00
yihao.dai
a4ef93457d
enhance: Optimize import scheduling and add time cost metric (#36601) (#36684) 
1. Optimize import scheduling strategic:
a. Revise slot weights, calculating them based on the number of files
and segments for both import and pre-import tasks.
b. Ensure that the DN executes tasks in ascending order of task ID.
2. Add time cost metric and log.

issue: https://github.com/milvus-io/milvus/issues/36600,
https://github.com/milvus-io/milvus/issues/36518

pr: https://github.com/milvus-io/milvus/pull/36601

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-10-11 10:27:22 +08:00
congqixia
b24788b2c7
enhance: [2.4] Add balance report log for qc balancer (#36749) 
Cherry pick from master
pr: #36747 
Related to #36746

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-10-11 10:25:24 +08:00
SimFG
548f8e80c3
enhance: [2.4] the estimate method when loading the collection (#36728) 
- pr: #36307
- issue: #36530

Signed-off-by: SimFG <bang.fu@zilliz.com>
Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
 2024-10-11 10:20:45 +08:00
XuanYang-cn
c69af44472
enhance: [24]Add more info in logs (#36732) 
pr: #36535

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-10-10 17:51:25 +08:00
aoiasd
eaa948752b
enhance: [Cherry-Pick] UpdateSegmentsInfo should update remaining segment info even if some one not exist (#36729) 
pr: https://github.com/milvus-io/milvus/pull/36726

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2024-10-10 15:11:20 +08:00
zhagnlu
02bd916fa4
fix: fix: fix json in [] expr bug (#36722) 
pr: #36721

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-10-09 21:53:20 +08:00
yihao.dai
9cb5396cf6
enhance: Use common gc config (#36668) (#36670) 
Use the GC config from `common` and remove the GC config from
`queryNode`.

issue: https://github.com/milvus-io/milvus/issues/36667

pr: https://github.com/milvus-io/milvus/pull/36668

related pr: https://github.com/milvus-io/milvus/pull/34949

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-10-09 19:49:20 +08:00
congqixia
3a80d1f602
enhance: [2.4] Add streaming forward policy switch for delegator (#36330) (#36712) 
Cherry pick from master
pr: #36330
Related to #35303

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-10-09 17:41:20 +08:00
yihao.dai
93a4574a35
Add buildIndex state for import job (#36705) 
issue: https://github.com/milvus-io/milvus/issues/36698

pr: https://github.com/milvus-io/milvus/pull/35868,
https://github.com/milvus-io/milvus/pull/36699

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-10-09 16:57:20 +08:00
XuanYang-cn
0ff8e13232
fix: [24]Remove neighbors if compactTo is unindexed (#36503) (#36694) 
See also: #36360
pr: #36503

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-10-09 15:43:20 +08:00
Patrick Weizhi Xu
6e93601b2c
enhance: [2.4] remove warnings when there is no partition key set (#36679) 
issue: #36673

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
(cherry picked from commit db60f7cf2a56656b101067a2dbebd4dd32ed6ed2)
 2024-10-09 09:35:20 +08:00
cai.zhang
f7302a78fb
enhance: [cherry-pick]Convert unincode to ascii to improving expression parsing efficiency (#36676) 
issue: #36672 

master pr: #36675

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-10-09 09:23:24 +08:00
congqixia
9073f6281e
fix: [2.4] Add defer Unpin when error happens (#36620) (#36665) 
Cherry-pick from master
pr: #36620
Resolves: #36619

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-10-08 18:29:20 +08:00
Buqian Zheng
03fba95284
fix: [2.4] improve mmap related code in Column.h (#36521) 
pr: #36183
issue: #36182

---------

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
 2024-10-08 18:27:19 +08:00
jaime
da2d3fb430
enhance: enable manual compaction for collections without indexes (#36581) 
issue: #36576
pr: #36577

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-10-08 14:51:20 +08:00
jaime
bf671786d9
enhance: decrease CPU overhead when calculating index file size (#36580) 
issue: #36578
pr: #36579

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-10-08 14:29:25 +08:00
yihao.dai
86004d852e
enhance: Reopen chunk cache cpp ut (#33622) (#33623) 
issue: https://github.com/milvus-io/milvus/issues/33210

pr: https://github.com/milvus-io/milvus/pull/33622

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-10-08 11:21:22 +08:00
yihao.dai
29c1d5042a
fix: Fix import segments leak in segment manager (#36602) (#36631) 
Directly add import segments from the meta, eliminating the dependency
on the segment manager.

issue: https://github.com/milvus-io/milvus/issues/34648

pr: https://github.com/milvus-io/milvus/pull/36602

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-10-08 10:15:22 +08:00
wei liu
2428adea3b
enhance: Enable balance on querynode with different mem capacity (#36466) (#36625) 
issue: #36464
pr: #36466
This PR enable balance on querynode with different mem capacity, for
query node which has more mem capactity will be assigned more records,
and query node with the largest difference between assignedScore and
currentScore will have a higher priority to carry the new segment.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-09-30 18:11:18 +08:00
congqixia
385bfc2639
enhance: [2.4] Unify InboundLabel case using metrics.Label (#36613) (#36616) 
Cherry pick from master
pr: #36613 
Previous label case broken by #36107, this PR make all inbound label
using label constants from metrics package.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-09-30 16:19:17 +08:00
wei liu
f8dfd0de53
fix: Skip unnecessary query node health check in proxy (#36491) (#36553) 
issue: #36490
pr: #36491
After the query node changes from a delegator to a worker, proxy should
skip this querynode's health check.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-09-30 14:27:16 +08:00
wei liu
4120320074
enhance: make TransferChannel/TransferSegment idempotent (#36489) (#36552) 
issue: #36488
pr: #36489
when call TransferChannel/TransferSegment, querycoord will generate and
submit balance task to scheduler, if segment/channel's task already
exist in scheduler, submit task will failed.

to make TransferChannel/TransferSegment idempotent, we should skip to
submit if task already exist in scheduler.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-09-30 14:25:26 +08:00
yihao.dai
c5088b8527
enhance: Add metrics to monitor import throughput and imported rows (#36519) (#36588) 
issue: https://github.com/milvus-io/milvus/issues/36518

pr: https://github.com/milvus-io/milvus/pull/36519

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-09-30 10:03:18 +08:00
zhagnlu
9a07c1bca9
fix:fix term expr overflow bug (#36534) 
pr: #36525

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-09-28 17:43:16 +08:00
cai.zhang
6335c8bf69
fix: [cherry-pick] Record the nodeID before assigning tasks (#36493) 
issue: #33744 

master pr: #36371

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-09-28 17:29:21 +08:00
cai.zhang
132e4c3ba1
fix: [cherry-pick]Fix data race for cluerting compaction (#36499) 
issue: #36438 

master pr: #36440

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-09-28 17:19:21 +08:00
aoiasd
2231aeab4d
fix:[Cherry-Pick] Split delete task msg to MaxMessageSize (#36574) 
relate: https://github.com/milvus-io/milvus/issues/36089
pr: https://github.com/milvus-io/milvus/pull/36197
split delete task msg to MaxMessageSize to avoid mq message too large
error

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2024-09-27 18:15:19 +08:00
cai.zhang
e5a6c5b31d
fix: [cherry-pick]Check string array max length after type matching (#36497) 
issue: #36029 

master pr: #36449

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-09-27 11:15:14 +08:00
Zhen Ye
e34fa0461b
fix: port listen racing in mix or standalone mode (#36459) 
issue: #36441
pr: #36442

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-09-26 21:01:15 +08:00
wei liu
74af00ba8c
fix: Segment unbalance after many times load/release (#36537) (#36543) 
issue: #36536
pr: #36537
query coord use `segmentTaskDeleta/channelTaskDelta` to measure the
executing workload for querynode in scheduler, and we maintains the
`segmentTaskDeleta/channelTaskDelta` by `scheulder.Add(task)` and
`scheduler.remove(task)`, but `scheduler.remove(task)` has been called
in unexpected way, which cause a wrong
`segmentTaskDeleta/channelTaskDelta` value and affect the segment assign
logic, causes segment unbalance.

This PR moves to compute the `segmentTaskDeleta/channelTaskDelta` when
access, to avoid the wrong value affect.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-09-26 20:57:14 +08:00
SimFG
6604bbda8f
enhance: [2.4] update the expr version and format the expr http response (#36467) 
/kind improvement
- pr: #36406

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-09-26 14:27:20 +08:00
wei liu
ad5d24be65
enhance: Optimize workload based replica selection policy (#36181) (#36384) 
issue: #35859
pr: #36181

This PR introduce two new param: toleranceFactor and checkRequestNum,
after every checkRequestNum request has been assigned, try to compute
querynode's workload score.

if the diff is less than the toleranceFactor, replica selection policy
will fallback to round_robin, which reduce the average cost to about
500ns.

if the diff is larger than the toleranceFactor, replica selection policy
will compute querynode's score to select the target node with smallest
score in every assigment.

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-09-26 11:19:14 +08:00
wei liu
975a9797a2
enhance: Enable dynamic update loaded collection's replica (#36417) 
issue: #35821
pr: #35822
After collection loaded, if we need to increase/decrease collection's
replica, we need to release and load it again.

milvus offers 4 solution to update loaded collection's replica, this PR
aims to dynamic change the replica number without release, and after
replica number changed, milvus will execute load replica or release
replica in async, and the replica loaded status can be checked by
getReplicas API.

Notice that if set too much replicas than querynode can afford，the new
replica won't be loaded successfully until enough querynode joins.

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-09-26 10:43:15 +08:00
wei liu
bdc59f3b15
fix: Fix cornor case that segment can't be move out from stopping node (#36431) (#36475) 
issue: #36426
pr: #36431
the old constriant requires only segment on current target can be
balanced, which is wrong, and caused that segment can't be move out from
stopping node, if it's only exist in next target.

by design, stopping balance need to move out all segment on it by
balance task, thus the unfair old constriant should be removed.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-09-25 10:13:18 +08:00
SimFG
a00523f0fd
fix: metric type error when the collection has two vec field (#36473) 
- issue: #36395

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-09-24 20:03:14 +08:00
congqixia
fa6354f6df
enhance: [skip e2e][2.4] Add unittest for reducing duplicated pk from multi segments (#36433) (#36460) 
Cherry-pick from master
pr: #36433
Related to #35505 #36362

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-09-24 18:09:14 +08:00
jaime
b92daa1532
fix: iaccurate size estimation for encoded array data (#36379) 
issue: #36029
pr: #36373

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-09-23 21:17:13 +08:00
congqixia
e083bf39f8
fix: [2.4] Return all compactTo segments after support split (#36361) (#36429) 
Cherry-pick from master
pr: #36361
Related to #36360

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-09-23 14:39:12 +08:00
foxspy
7ac3ef9acb
enhance: Update Knowhere version (#36402) 
/kind branch-feature

Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
 2024-09-20 19:25:11 +08:00
congqixia
eb1602a374
enhance: [2.4] Graceful stop flowgraph manager when stopping datanode (#36229) (#36358) 
Cherry pick from master
pr: #36229

Flowgraph manager is not stopped during datanode stopping procedure
which may lead to unexpect flowgraph behavior during/after datanode stop
progress.

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-09-19 17:39:12 +08:00
Bingyi Sun
e2f1beb543
fix: remove element type check (#35828) (#36324) 
https://github.com/milvus-io/milvus/issues/36275
Array's element type is not same with schema's. It is INT32 for INT16
and INT8

related pr: #35828

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2024-09-19 11:55:13 +08:00