admin/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2026-02-02 01:06:41 +08:00
Code Issues Packages Projects Releases Wiki Activity
20,110 Commits 146 Branches 196 Tags
Commit Graph

2878 Commits

Author SHA1 Message Date
XuanYang-cn
edefc3cbb5
enhance: [skip e2e]Enable compaction it test (#34526) (#34720) 
pr: #34526

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-07-16 18:19:38 +08:00
SimFG
13f129aa9c
enhance: [2.4] format the sdk code (#34699) 
- issue: #31293
- pr: #34033

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-07-16 14:15:38 +08:00
elstic
40f6febf9b
test: [cherry-pick] Fix index creation mismatch for sparse vector columns in multivector case (#34620) 
pr: https://github.com/milvus-io/milvus/pull/34618

Signed-off-by: elstic <hao.wang@zilliz.com>
 2024-07-16 09:39:38 +08:00
smellthemoon
0fdb288de7
enhance: upsert support autoid(#30342) (#34633) 
pr: #30342 
issue: #29258

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
 2024-07-15 20:53:39 +08:00
wei liu
cf701a9bf0
enhance: Preserve fixed-size memory in delegator node for growing segment (#34600) 
issue: #34595
pr: #34596
When consuming insert data on the delegator node, QueryCoord will move
out some sealed segments to manage its memory usage. After the growing
segment gets flushed, some sealed segments from other workers will be
moved back to the delegator node. To avoid the frequent movement of
segments, we estimate the maximum growing row count and preserve a
fixed-size memory in the delegator node.

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-07-15 20:51:46 +08:00
zhuwenxing
f2d0517f96
test: [cherry-pick]add testcase for count query (#34471) 
pr: https://github.com/milvus-io/milvus/pull/34453

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
 2024-07-08 12:54:12 +08:00
Gao
a60e2a65ff
enhance: change autoindex default metric type (#34277) 
issue: #34304 
pr: #34261

Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2024-07-08 10:52:14 +08:00
wei liu
d3e94f9861
enhance: Use Blocked Bloom Filter instead of basic bloom fitler impl (#34377) 
issue: #32995
pr: #33405
To speed up the construction and querying of Bloom filters, we chose a
blocked Bloom filter instead of a basic Bloom filter implementation.

WARN: This PR is compatible with old version bf impl, but if fall back
to old milvus version, it may causes bloom filter deserialize failed.

In single Bloom filter test cases with a capacity of 1,000,000 and a
false positive rate (FPR) of 0.001, the blocked Bloom filter is 5 times
faster than the basic Bloom filter in both querying and construction, at
the cost of a 30% increase in memory usage.

Block BF construct time {"time": "54.128131ms"}
Block BF size {"size": 3021578}
Block BF Test cost {"time": "55.407352ms"}
Basic BF construct time {"time": "210.262183ms"}
Basic BF size {"size": 2396308}
Basic BF Test cost {"time": "192.596229ms"}
In multi Bloom filter test cases with a capacity of 100,000, an FPR of
0.001, and 100 Bloom filters, we reuse the primary key locations for all
Bloom filters to avoid repeated hash computations. As a result, the
blocked Bloom filter is also 5 times faster than the basic Bloom filter
in querying.

Block BF TestLocation cost {"time": "529.97183ms"}
Basic BF TestLocation cost {"time": "3.197430181s"}

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-07-05 17:04:10 +08:00
wayblink
c62bf8a0b0
fix: [Cherry-pick]Pick major compaction fixs and optimizations (#34360) 
This PR cherry-picks the following commits:

- fix: sync partitiion stats blocking balance task #33742
- fix: Fix meta prefix overlap bug #33830
- fix: Small fixs of major compaction #33929 
- fix: Fix memory buffer error & some renaming #33850
- fix: sync part stats task cannot be finished #34027 
- Add an option to enable/disable vector field clustering key #34097
- fix: fix error ignore in compactor #34169
- fix:load major compaction partial result #34052
- Use new stream segment reader in clustering compaction #34232

issue: #30633
pr: #33742 #33830 #33929 #33850 #34027 #34097 #34169 #34052 #34232

---------

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
Signed-off-by: wayblink <anyang.wang@zilliz.com>
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: Chun Han <116052805+MrPresent-Han@users.noreply.github.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
 2024-07-03 09:53:37 +08:00
elstic
fb88267855
test: [cherry-pick] update test case (#34109) 
pr: https://github.com/milvus-io/milvus/pull/34108

Signed-off-by: elstic <hao.wang@zilliz.com>
 2024-07-02 22:04:08 +08:00
wayblink
99586066f5
feat: [cherry-pick] Major compaction (#34326) 
This PR cherry-picks the following commits:
fix: speed up segment lookup via channel name in datacoord (#33530)
needed by the next commit
  feat: Major compaction (#33620)

issue: #30633
pr: #33620

---------

Signed-off-by: yiwangdr <yiwangdr@gmail.com>
Signed-off-by: wayblink <anyang.wang@zilliz.com>
Co-authored-by: yiwangdr <80064917+yiwangdr@users.noreply.github.com>
Co-authored-by: MrPresent-Han <chun.han@zilliz.com>
 2024-07-02 18:29:01 +08:00
zhenshan.cao
14a11e379c
enhance: Refactor Compaction to enable persistence(#33265) (#34268) 
pr : #33265 

issue #33586

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2024-07-01 19:32:07 +08:00
zhuwenxing
9cb3658d74
test: [cherry-pick]remove dup code (#34243) 
pr: https://github.com/milvus-io/milvus/pull/34242

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
 2024-06-28 10:44:05 +08:00
yihao.dai
b1e74dc7cb
enhance: [cherry-pick] Decouple compaction from shard (#34157) 
This PR cherry-picks the following commits:

- Implement task limit control logic in datanode.
https://github.com/milvus-io/milvus/pull/32881
- Load bf from storage instead of memory during L0 compaction.
https://github.com/milvus-io/milvus/pull/32913
- Remove dependencies on shards (e.g. SyncSegments, injection).
https://github.com/milvus-io/milvus/pull/33138
- Rename Compaction interface to CompactionV2.
https://github.com/milvus-io/milvus/pull/33858
- Remove the unused residual compaction logic.
https://github.com/milvus-io/milvus/pull/33932

issue: https://github.com/milvus-io/milvus/issues/32809

pr: https://github.com/milvus-io/milvus/pull/32881,
https://github.com/milvus-io/milvus/pull/32913,
https://github.com/milvus-io/milvus/pull/33138,
https://github.com/milvus-io/milvus/pull/33858,
https://github.com/milvus-io/milvus/pull/33932

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-25 20:22:03 +08:00
sammy.huang
2ecfe1599c
[automated] Update cpu Builder image changes (#34079) 
Signed-off-by: Liang Huang <sammy.huang@zilliz.com>
 2024-06-25 19:02:03 +08:00
yanliang567
59d910320d
test:[cherry-pick]Update tests for range search and add test for query with dup ids (#34069) 
related issue: https://github.com/milvus-io/milvus/issues/33883
pr: #34057

Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
 2024-06-24 11:34:03 +08:00
Jiquan Long
22e6807e9a
feat: support inverted index for array (#33452) (#34053) 
pr: https://github.com/milvus-io/milvus/pull/33184
pr: https://github.com/milvus-io/milvus/pull/33452
pr: https://github.com/milvus-io/milvus/pull/33633
issue: https://github.com/milvus-io/milvus/issues/27704
Co-authored-by: xiaocai2333 <cai.zhang@zilliz.com>

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
Co-authored-by: cai.zhang <cai.zhang@zilliz.com>
 2024-06-24 10:50:03 +08:00
zhuwenxing
630a726f35
test: refine restful testcases trace (#34065) 
pr: https://github.com/milvus-io/milvus/pull/34066

---------

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
 2024-06-24 10:40:03 +08:00
wei liu
061a00c58f
enhance: Enable database level replica num and resource groups for loading collection (#33052) (#33981) 
pr: #33052

issue: #30040

This PR introduce two database level props:
1. database.replica.number
2. database.resource_groups

User can set those two database props by AlterDatabase API, then can
load collection without specified replica_num and resource groups. then
it will use database level load param when try to load collections.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-06-21 16:56:02 +08:00
zhuwenxing
89461db5f3
test: update the lib of bf16 (#34044) 
pr: https://github.com/milvus-io/milvus/pull/34043

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
 2024-06-21 14:20:08 +08:00
elstic
5952c09925
test: [cherry-pick] optimizing variable names (#34036) 
pr: https://github.com/milvus-io/milvus/pull/34035

Signed-off-by: elstic <hao.wang@zilliz.com>
 2024-06-21 10:12:01 +08:00
elstic
e0e2ed073b
test: [cherry-pick] change the number of insertion vectors (#34015) 
pr: https://github.com/milvus-io/milvus/pull/34014

Signed-off-by: elstic <hao.wang@zilliz.com>
 2024-06-20 13:58:05 +08:00
zhuwenxing
c61740d407
test: [cherry-pick]add test case for bulkwriter (#33914) 
pr: https://github.com/milvus-io/milvus/pull/33879

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
 2024-06-20 09:22:18 +08:00
nico
27905de192
test: update test cases (#33971) 
Signed-off-by: nico <cheng.yuan@zilliz.com>
 2024-06-19 16:48:01 +08:00
elstic
caa1b0f06e
test: [cherry-pick] add more sparse test cases (#33918) 
issue: https://github.com/milvus-io/milvus/issues/31483
pr: https://github.com/milvus-io/milvus/pull/33916

Signed-off-by: elstic <hao.wang@zilliz.com>
 2024-06-19 15:24:09 +08:00
congqixia
60695bdb44
enhance: [2.4] Set maxPartitionNum default value to 1024 (#33950) 
Cherry-pick from master
pr: #33949
See also #30059

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-18 18:42:00 +08:00
nico
530756fe42
test: update test cases (#33899) 
pr: #33096

Signed-off-by: nico <cheng.yuan@zilliz.com>
 2024-06-18 11:35:59 +08:00
zhuwenxing
5b847e93fc
test: [cherry-pick]update import test case to support different dim (#33709) (#33866) 
pr: #33709

add test case for https://github.com/milvus-io/milvus/issues/33681

---------

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
(cherry picked from commit ca1f7ab019a310c88f39891becdc1da41290f171)
 2024-06-14 16:23:57 +08:00
zhuwenxing
f06be79a4d
test: [cherry-pick]refine load state check in restful test (#33862) 
pr: https://github.com/milvus-io/milvus/pull/33861

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
 2024-06-14 16:22:10 +08:00
Cai Yudong
ebd0af14f4
enhance: Handle Float16Vector/BFloat16Vector numpy bulk insert as same as BinaryVector (#33760) (#33788) 
pr: #33760
Issue: #22837

Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
 2024-06-13 10:49:57 +08:00
yihao.dai
396f8608dd
fix: Fix multiple vector fields import (#33723) (#33724) 
1. Fix dim mismatch with multi-vector fields and JSON import
2. Enhance: do not display file ID in GetImportResponse.

issue: https://github.com/milvus-io/milvus/issues/33681,
https://github.com/milvus-io/milvus/issues/33682

pr: https://github.com/milvus-io/milvus/pull/33723

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-10 21:55:55 +08:00
yihao.dai
ed1dee9e38
enhance: Support L0 import (#33514) (#33712) 
issue: https://github.com/milvus-io/milvus/issues/33157

pr: https://github.com/milvus-io/milvus/pull/33514

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-08 11:17:52 +08:00
zhuwenxing
29efd69857
test: refine import test (#33691) 
pr: #33600 #33356
 2024-06-07 09:21:53 +08:00
zhuwenxing
b78d7edca6
test: add coo format sparse vector in restful test (#33689) 
pr: https://github.com/milvus-io/milvus/pull/33677

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
 2024-06-07 09:19:58 +08:00
zhuwenxing
c39d7c3f49
test: wait all tasks finished in import test (#33629) 
pr: None

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
 2024-06-05 15:51:52 +08:00
zhuwenxing
6344b788fd
test: [cherry-pick]add sparse vector datatype for import test (#33166) (#33436) 
pr: #33166

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
 2024-06-04 08:59:47 +08:00
Buqian Zheng
d6a994038c
enhance: [2.4] add sparse float vector support to restful v2 (#33555) 
issue: https://github.com/milvus-io/milvus/issues/29419
pr: #33231

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
 2024-06-03 19:31:46 +08:00
yihao.dai
8ff5d2793c
fix: Fill stats log id and check validity (#33477) (#33478) 
1. Fill log ID of stats log from import
2. Add a check to validate the log ID before writing to meta

issue: https://github.com/milvus-io/milvus/issues/33476

pr: https://github.com/milvus-io/milvus/pull/33477

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-05-31 14:13:46 +08:00
zhuwenxing
6de80a0c19
test: add different nq for restful v2 test (#33469) 
pr: https://github.com/milvus-io/milvus/pull/33456

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
 2024-05-31 13:54:07 +08:00
Cai Yudong
68e2d532d8
enhance: Cherry-pick following SparseFloatVector bulk insert PRs to Milvus2.4 (#33391) 
Cherry pick from master
pr: #33064 #33101 #33187 #33259 #33224
#33064 Support readable JSON file import for
Float16/BFloat16/SparseFloat
  #33101 Store SparseFloatVector into parquet as JSON string
  #33187 Fix SparseFloatVector data parse error for parquet
  #33259 Fix SparseFloatVector data parse error for json
  #33224 Optimize bulk insert unittest

Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
 2024-05-30 10:31:45 +08:00
PowderLi
aaecc92244
fix: [cherry-pick] [restful v2] search result be cut while nq > 1 (#33363) 
issue: #33099 #32837 #32419
master pr: #33339

1. len(search result) may be nq * topk, we need return all rather than
topk
2. the  in restful response payload keep the same with milvus error code

Signed-off-by: PowderLi <min.li@zilliz.com>
 2024-05-27 00:57:45 +08:00
nico
ecdaeda6a8
enhance: update nightly groovy (#33362) 
Signed-off-by: nico <cheng.yuan@zilliz.com>
 2024-05-24 18:43:41 +08:00
zhuwenxing
512f57e65a
test: [cherry-pick]add check for partition key for import test (#33323) 
pr:https://github.com/milvus-io/milvus/pull/33253

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
 2024-05-24 10:57:40 +08:00
yanliang567
8c9afd5f07
test: [cherry pick] Add index name tests for multiple vectors (#33315) 
Related issue: https://github.com/milvus-io/milvus/issues/32653
Related pr: #33250

---------

Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
 2024-05-23 17:17:41 +08:00
yihao.dai
ad4c1975bd
fix: Fix filtering by partition key fails for importing data (#33274) (#33277) 
Before executing the import, partition IDs should be reordered according
to partition names. Otherwise, the data might be hashed to the wrong
partition during import. This PR corrects this error.

issue: https://github.com/milvus-io/milvus/issues/33237

pr: https://github.com/milvus-io/milvus/pull/33274

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-05-23 11:25:40 +08:00
sammy.huang
7013d2f906
feat: support arm-based image build and pull request (#33225) 
Signed-off-by: Liang Huang <sammy.huang@zilliz.com>
 2024-05-21 16:54:43 +08:00
sre-ci-robot
780f3137f3
[automated] Update Pytest image changes (#33091) 
Update Pytest image changes
See changes:
9536e79363
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 2024-05-17 15:09:36 +08:00
binbin
9536e79363
test: update pymilvus version (#33077) 
issue: #17614 
pr: #33079

Signed-off-by: binbin lv <binbin.lv@zilliz.com>
 2024-05-16 14:51:34 +08:00
yihao.dai
6fc00e42d7
fix: Fix binlog import (#33027) 
binlog import should only import into one partition.

issue: https://github.com/milvus-io/milvus/issues/32807

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-05-14 21:45:32 +08:00
yanliang567
ba3b2a91a0
test: Remove useless common types and refine error assert in negative cases (#33023) 
Related issue: #32653 

1. Remove some meaningless common types
2. Refine error assertion in negative cases
3. Remove some dup tests

---------

Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
 2024-05-14 15:03:33 +08:00