admin/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2026-01-07 19:31:51 +08:00
Code Issues Packages Projects Releases Wiki Activity
20,886 Commits 137 Branches 188 Tags
Commit Graph

9493 Commits

Author SHA1 Message Date
CharlesFeng
7c8b71e26c
fix: BinlogDeserializeReader leak in mix_compactor.go (#36270) 
https://github.com/milvus-io/milvus/issues/36269

Signed-off-by: fengjun2016 <jornfeng@gmail.com>
 2024-10-11 15:41:20 +08:00
cai.zhang
d1060c0e05
enhance: Update antlr version and refine parsing not in (#36745) 
issue: #36672

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-10-11 14:03:21 +08:00
congqixia
3fe0f82923
enhance: Add balance report log for qc balancer (#36747) 
Related to #36746

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-10-11 10:25:24 +08:00
aoiasd
db34572c56
feat: support load and query with bm25 metric (#36071) 
relate: https://github.com/milvus-io/milvus/issues/35853

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2024-10-11 10:23:20 +08:00
zhagnlu
b1e678dcba
fix: fix json in [] expr bug (#36721) 
#36718

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-10-11 01:11:20 +08:00
Buqian Zheng
f7b811450d
feat: add enable_tokenizer params to VarChar field (#36480) 
issue: #35922

add an enable_tokenizer param to varchar field: must be set to true so
that a varchar field can enable_match or used as input of BM25 function

---------

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
 2024-10-10 20:33:21 +08:00
XuanYang-cn
290ceb4e84
enhance: Add more info in logs (#36731) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-10-10 17:51:25 +08:00
aoiasd
a5f98e4cb3
enhance: UpdateSegmentsInfo should update remaining segment info even if some one not exist (#36726) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2024-10-10 15:17:19 +08:00
Ted Xu
62679efa0c
enhance: remove unnecessary clone in meta cache (#36628) 
See #36627

---------

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
 2024-10-10 15:05:20 +08:00
yihao.dai
3685edb264
enhance: Use common gc config (#36668) 
Use the GC config from `common` and remove the GC config from
`queryNode`.

issue: https://github.com/milvus-io/milvus/issues/36667

related pr: https://github.com/milvus-io/milvus/pull/34949

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-10-09 19:47:19 +08:00
Ted Xu
5fc731795b
enhance: Datacoord to support prioritization of compaction tasks (#36547) 
See #36550

This PR made 2 changes:

1. Introducing a prioritization mechanism, if
`dataCoord.compaction.taskPrioritizer` is set to `level`, compaction
tasks are always executed as the priority of L0>Mix>Clustering
2. `dataCoord.compaction.maxParallelTaskNum` now controls the
parallelism of executing tasks, not the task number of queue +
executing.

---------

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
 2024-10-09 19:11:20 +08:00
Zhen Ye
efb10d3b86
fix: streaming metrics do not clear (#36701) 
issue: #33285

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-10-09 17:51:20 +08:00
SimFG
130a923dec
enhance: the estimate method when loading the collection (#36307) 
- issue: #36530

---------

Signed-off-by: SimFG <bang.fu@zilliz.com>
Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
Co-authored-by: xianliang.li <xianliang.li@zilliz.com>
 2024-10-09 17:35:19 +08:00
congqixia
c3d910756b
enhance: Update knowhere commit to fix mac compilation (#36706) 
Related to zilliztech/knowhere#879

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-10-09 16:05:20 +08:00
yihao.dai
0fc2a4aa53
enhance: Optimize import scheduling and add time cost metric (#36601) 
1. Optimize import scheduling strategic:
a. Revise slot weights, calculating them based on the number of files
and segments for both import and pre-import tasks.
b. Ensure that the DN executes tasks in ascending order of task ID.
2. Add time cost metric and log.

issue: https://github.com/milvus-io/milvus/issues/36600,
https://github.com/milvus-io/milvus/issues/36518

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-10-09 14:41:20 +08:00
yihao.dai
4f50b7dbab
enhance: Swap the position of import job states (#36699) 
issue: https://github.com/milvus-io/milvus/issues/36698

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-10-09 11:29:19 +08:00
Patrick Weizhi Xu
90a0e90732
enhance: remove warnings when there is no partition key set (#36674) 
issue: #36673

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
 2024-10-09 10:27:20 +08:00
cai.zhang
fc8b5ab791
enhance: Convert unincode to ascii to improving expression parsing efficiency (#36675) 
issue: #36672

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-10-09 09:23:24 +08:00
Zhen Ye
2ec6e602d6
enhance: add streaming client metrics (#36523) 
issue: #33285

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-10-08 21:25:19 +08:00
jaime
ef1832ff9c
enhance: enable manual compaction for collections without indexes (#36577) 
issue: #36576

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-10-08 19:57:18 +08:00
jaime
1fded42277
enhance: decrease CPU overhead when calculating index file size (#36579) 
issue: #36578

---------

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-10-08 14:29:25 +08:00
XuanYang-cn
4e0ea39235
fix: Remove neighbors if compactTo is unindexed (#36503) 
See also: #36360

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-10-08 14:15:19 +08:00
wayblink
c7a81d5de3
enhance:use proto.clone in compaction tasks to prevent omissions (#36624) 
issue: #36623

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-10-08 11:23:23 +08:00
yihao.dai
1f47d5510b
fix: Fix import segments leak in segment manager (#36602) 
Directly add import segments from the meta, eliminating the dependency
on the segment manager.

issue: https://github.com/milvus-io/milvus/issues/34648

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-10-08 10:11:22 +08:00
sre-ci-robot
3936d12661
[automated] Update Knowhere Commit (#36634) 
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 2024-10-01 01:05:15 +08:00
congqixia
ddc3e76803
fix: Add defer Unpin when error happens (#36620) 
Resolves: #36619

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-09-30 19:49:17 +08:00
wei liu
470bb0cc3f
enhance: Enable balance on querynode with different mem capacity (#36466) 
issue: #36464
This PR enable balance on querynode with different mem capacity, for
query node which has more mem capactity will be assigned more records,
and query node with the largest difference between assignedScore and
currentScore will have a higher priority to carry the new segment.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-09-30 16:15:17 +08:00
wayblink
00a5025949
enhance: support clustering compaction on null value (#36372) 
issue: #36055

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-09-30 14:33:17 +08:00
congqixia
e4292569c7
enhance: Unify InboundLabel case using metrics.Label (#36613) 
Previous label case broken by #36107, this PR make all inbound label
using label constants from metrics package.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-09-30 14:15:17 +08:00
Rijin-N
a05a37a583
enhance: GCS native support (GCS implemented using Google Cloud Storage libraries) (#36214) 
Native support for Google cloud storage using the Google Cloud Storage
libraries. Authentication is performed using GCS service account
credentials JSON.

Currently, Milvus supports Google Cloud Storage using S3-compatible APIs
via the AWS SDK. This approach has the following limitations:

1. Overhead: Translating requests between S3-compatible APIs and GCS can
introduce additional overhead.
2. Compatibility Limitations: Some features of the original S3 API may
not fully translate or work as expected with GCS.

To address these limitations, This enhancement is needed.

Related Issue: #36212
 2024-09-30 13:23:32 +08:00
jaime
5686a9a024
fix: unhandle error in upsert task (#36604) 
issue: #36611

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-09-30 12:59:16 +08:00
Chun Han
a25dc98794
enhance: support group_size and hybridsearch+groupby on httpv2 side(#36386) (#36461) 
related: #36386

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
 2024-09-30 11:09:16 +08:00
Buqian Zheng
94005b7198
fix: Sparse float vector incorrectly ExpandData at mmap mode (#36603) 
issue: #36561

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
 2024-09-30 10:39:16 +08:00
cai.zhang
ecb2b242e2
enhance: Add sorted for segment info (#36469) 
issue: #33744

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-09-30 10:01:16 +08:00
smellthemoon
a78a6b33ab
fix: not check json data when set nullable or default_value (#36575) 
#36354

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2024-09-29 20:39:15 +08:00
Zhen Ye
a47abb2f2b
fix: stats log lost after disable stats log loading on flush (#36592) 
issue: #36555

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-09-29 16:53:15 +08:00
Zhen Ye
a6545b2e29
fix: refactor milvus config and change default txn timeout (#36522) 
issue: #36498

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-09-29 11:01:15 +08:00
yihao.dai
8ed34dce84
enhance: Reopen chunk cache cpp ut (#33622) 
issue: https://github.com/milvus-io/milvus/issues/33210

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-09-28 18:19:15 +08:00
zhagnlu
9e3efa06be
fix:fix empty search result bug (#36582) 
#36450

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-09-28 17:45:16 +08:00
yihao.dai
80f25d497f
enhance: Add metrics to monitor import throughput and imported rows (#36519) 
issue: https://github.com/milvus-io/milvus/issues/36518

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-09-28 17:31:15 +08:00
cai.zhang
acc9b5ada5
fix: Regenerate partition stats after stats task for clustering compaction (#36408) 
issue: #33744

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-09-28 17:29:21 +08:00
cai.zhang
7bf40694fd
fix: Record the nodeID before assigning tasks (#36371) 
issue: #33744

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-09-28 17:21:15 +08:00
cai.zhang
2adca8b754
fix: Fix data race for cluerting compaction (#36440) 
issue: #36438

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-09-28 17:19:21 +08:00
Zhen Ye
bd7910632a
fix: etcd session discoverer cannot recover from ErrCompacted (#36568) 
issue: #36554

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-09-28 10:51:14 +08:00
aoiasd
ffc12fb5c4
fix: split delete task msg to MaxMessageSize to avoid mq message too large error (#36197) 
relate: https://github.com/milvus-io/milvus/issues/36089

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2024-09-27 18:15:19 +08:00
smellthemoon
b1ac3f0df0
enhance: do not log out the full req (#36546) 
![img_v3_02f3_61ac166c-df6f-4b5e-b06a-756e5f40a65g](https://github.com/user-attachments/assets/df668339-d0d8-4295-a2b6-52d89ad98766)

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2024-09-27 16:39:15 +08:00
cai.zhang
50905e0b45
fix: Check string array max length after type matching (#36449) 
issue: #36029

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-09-27 09:45:14 +08:00
Zhen Ye
d29e01e284
fix: port listen racing in mix or standalone mode (#36442) 
issue: #36441

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-09-26 21:23:16 +08:00
wayblink
7ff41697f9
enhance: make slotUsage a field of CompactionTask (#36510) 
#36509

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-09-26 20:19:14 +08:00
wei liu
c056620899
fix: Skip unnecessary query node health check in proxy (#36491) 
issue: #36490
After the query node changes from a delegator to a worker, proxy should
skip this querynode's health check.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-09-26 18:13:15 +08:00