admin/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2026-01-07 19:31:51 +08:00
Code Issues Packages Projects Releases Wiki Activity
20,866 Commits 137 Branches 188 Tags
Commit Graph

30 Commits

Author SHA1 Message Date
Rijin-N
a05a37a583
enhance: GCS native support (GCS implemented using Google Cloud Storage libraries) (#36214) 
Native support for Google cloud storage using the Google Cloud Storage
libraries. Authentication is performed using GCS service account
credentials JSON.

Currently, Milvus supports Google Cloud Storage using S3-compatible APIs
via the AWS SDK. This approach has the following limitations:

1. Overhead: Translating requests between S3-compatible APIs and GCS can
introduce additional overhead.
2. Compatibility Limitations: Some features of the original S3 API may
not fully translate or work as expected with GCS.

To address these limitations, This enhancement is needed.

Related Issue: #36212
 2024-09-30 13:23:32 +08:00
Zhen Ye
a773836b89
enhance: optimize milvus core building (#35610) 
issue: #35549,#35611,#35633

- remove milvus_segcore milvus_indexbuilder..., add libmilvus_core
- core building only link once
- move opendal compilation into cmake
- fix odr

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-08-23 12:35:02 +08:00
foxspy
8e64bf929c
enhance: add scalar filtering and vector search latency metrics (#34785) 
add scalar filtering and vector search latency metrics to distinguish
the cost of scalar filtering.
To add metrics in query chain, add a monitor module and move the metric
files from original storage module.
issue: #34780

Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
 2024-07-19 14:01:39 +08:00
chyezh
f53ab54c5d
enhance: async cgo utility (#33133) 
issue: #30926, #33132

- implement future-based cgo utility.

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-06-09 22:55:53 +08:00
cai.zhang
27cc9f2630
enhance: Support analyze data (#33651) 
issue: #30633

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Co-authored-by: chasingegg <chao.gao@zilliz.com>
 2024-06-06 17:37:51 +08:00
Alexander Guzhva
c4b37fb285
enhance: Custom bitset and bitsetview prototypes (#30454) 
Issue: #31285 

Basically, I've replaced `FixedVector<bool>` and `boost::dynamic_bitset`
with custom bitset and bitsetview in order to reduce the memory
bandwidth & increase performance for the filtering.

This PR is for internal use only. 

Current progress (numbers are for GCC 9.5.0 on Ubuntu 22.04 LTS;
clang-17 produces better performance numbers):
Baseline:
```
[ RUN      ] CApiTest.AssembeChunkPerfTest
start test
cost: 17903us
[       OK ] CApiTest.AssembeChunkPerfTest (183 ms)

[ RUN      ] Expr.TestMultiLogicalExprsOptimization
cost: 1391us
cost: 5us
cost: 4us
cost: 4us
cost: 6us
cost: 4us
cost: 4us
cost: 4us
cost: 4us
cost: 4us
143
cost: 10us
cost: 8us
cost: 10us
cost: 8us
cost: 8us
cost: 8us
cost: 8us
cost: 8us
cost: 8us
cost: 9us
8
/home/ubuntu/zilliz/milvus4/milvus/internal/core/unittest/test_expr.cpp:1561: Failure
Expected: (cost_op) < (cost_no_op), actual: 143 vs 8
[  FAILED  ] Expr.TestMultiLogicalExprsOptimization (7 ms)
[ RUN      ] Expr.TestExprs
start test
3cost: 889us
start test
10cost: 2us
start test
20cost: 2us
start test
30cost: 2us
start test
50cost: 3us
start test
100cost: 7us
start test
200cost: 16us
[       OK ] Expr.TestExprs (9 ms)

[ RUN      ] Expr.TestUnaryBenchTest
start test type:2
 cost: 124.8us
start test type:3
 cost: 163.1us
start test type:4
 cost: 275.9us
start test type:5
 cost: 590.9us
start test type:10
 cost: 62.7us
start test type:11
 cost: 65.9us
[       OK ] Expr.TestUnaryBenchTest (1153 ms)
[ RUN      ] Expr.TestBinaryRangeBenchTest
start test type:2
 cost: 151.4us
start test type:3
 cost: 198.4us
start test type:4
 cost: 361.9us
start test type:5
 cost: 753.9us
start test type:10
 cost: 64.6us
start test type:11
 cost: 62.2us
[       OK ] Expr.TestBinaryRangeBenchTest (1151 ms)
[ RUN      ] Expr.TestLogicalUnaryBenchTest
start test type:2
 cost: 121.14us
start test type:3
 cost: 156.84us
start test type:4
 cost: 249.76us
start test type:5
 cost: 534.44us
start test type:10
 cost: 82.2us
start test type:11
 cost: 83.52us
[       OK ] Expr.TestLogicalUnaryBenchTest (1202 ms)
[ RUN      ] Expr.TestBinaryLogicalBenchTest
start test type:2
 cost: 80.64us
start test type:3
 cost: 78.22us
start test type:4
 cost: 255.76us
start test type:5
 cost: 532.04us
start test type:10
 cost: 89.26us
start test type:11
 cost: 90us
[       OK ] Expr.TestBinaryLogicalBenchTest (1198 ms)
[ RUN      ] Expr.TestBinaryArithOpEvalRangeBenchExpr
start test type:2
 cost: 401.7us
start test type:3
 cost: 420.96us
start test type:4
 cost: 418.04us
start test type:5
 cost: 470.54us
start test type:10
 cost: 250.32us
start test type:11
 cost: 850.08us
[       OK ] Expr.TestBinaryArithOpEvalRangeBenchExpr (1273 ms)
[ RUN      ] Expr.TestCompareExprBenchTest
start test type:2
 cost: 162us
start test type:3
 cost: 142us
start test type:4
 cost: 374us
start test type:5
 cost: 674us
start test type:10
 cost: 366us
start test type:11
 cost: 645us
[       OK ] Expr.TestCompareExprBenchTest (1214 ms)
[ RUN      ] Expr.TestRefactorExprs
start test
3cost: 1253us
start test
10cost: 1060us
start test
20cost: 681us
start test
30cost: 522us
start test
50cost: 511us
start test
100cost: 506us
start test
200cost: 497us
[       OK ] Expr.TestRefactorExprs (1142 ms)

```

Candidate:
```
[ RUN      ] CApiTest.AssembeChunkPerfTest
start test
cost: 6099us
[       OK ] CApiTest.AssembeChunkPerfTest (153 ms)

[ RUN      ] Expr.TestMultiLogicalExprsOptimization
cost: 42us
cost: 15us
cost: 15us
cost: 14us
cost: 15us
cost: 15us
cost: 15us
cost: 15us
cost: 15us
cost: 15us
17
cost: 41us
cost: 39us
cost: 33us
cost: 33us
cost: 33us
cost: 33us
cost: 34us
cost: 41us
cost: 34us
cost: 34us
35
[       OK ] Expr.TestMultiLogicalExprsOptimization (6 ms)
[ RUN      ] Expr.TestExprs
start test
3cost: 20us
start test
10cost: 2us
start test
20cost: 2us
start test
30cost: 2us
start test
50cost: 4us
start test
100cost: 8us
start test
200cost: 15us
[       OK ] Expr.TestExprs (8 ms)

[ RUN      ] Expr.TestUnaryBenchTest
start test type:2
 cost: 55.7us
start test type:3
 cost: 79.8us
start test type:4
 cost: 177.6us
start test type:5
 cost: 337.2us
start test type:10
 cost: 16.9us
start test type:11
 cost: 15.7us
[       OK ] Expr.TestUnaryBenchTest (1140 ms)
[ RUN      ] Expr.TestBinaryRangeBenchTest
start test type:2
 cost: 57.1us
start test type:3
 cost: 87us
start test type:4
 cost: 177.5us
start test type:5
 cost: 342.7us
start test type:10
 cost: 17.9us
start test type:11
 cost: 16.7us
[       OK ] Expr.TestBinaryRangeBenchTest (1152 ms)
[ RUN      ] Expr.TestLogicalUnaryBenchTest
start test type:2
 cost: 34.58us
start test type:3
 cost: 68.86us
start test type:4
 cost: 151.38us
start test type:5
 cost: 286.8us
start test type:10
 cost: 16.54us
start test type:11
 cost: 16.7us
[       OK ] Expr.TestLogicalUnaryBenchTest (1165 ms)
[ RUN      ] Expr.TestBinaryLogicalBenchTest
start test type:2
 cost: 20us
start test type:3
 cost: 17.1us
start test type:4
 cost: 154.12us
start test type:5
 cost: 286.1us
start test type:10
 cost: 19.6us
start test type:11
 cost: 19.24us
[       OK ] Expr.TestBinaryLogicalBenchTest (1188 ms)
[ RUN      ] Expr.TestBinaryArithOpEvalRangeBenchExpr
start test type:2
 cost: 125.7us
start test type:3
 cost: 111.34us
start test type:4
 cost: 148.02us
start test type:5
 cost: 306.7us
start test type:10
 cost: 149.3us
start test type:11
 cost: 282.94us
[       OK ] Expr.TestBinaryArithOpEvalRangeBenchExpr (1221 ms)
[ RUN      ] Expr.TestCompareExprBenchTest
start test type:2
 cost: 89us
start test type:3
 cost: 79us
start test type:4
 cost: 323us
start test type:5
 cost: 629us
start test type:10
 cost: 313us
start test type:11
 cost: 591us
[       OK ] Expr.TestCompareExprBenchTest (1228 ms)
[ RUN      ] Expr.TestRefactorExprs
start test
3cost: 874us
start test
10cost: 611us
start test
20cost: 290us
start test
30cost: 294us
start test
50cost: 272us
start test
100cost: 278us
start test
200cost: 279us
[       OK ] Expr.TestRefactorExprs (1149 ms)

```

Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>
 2024-03-24 21:49:07 +08:00
zhagnlu
a602171d06
enhance: Refactor runtime and expr framework (#28166) 
#28165

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2023-12-18 12:04:42 +08:00
Enwei Jiao
c3f15c6b95
Refactor duplicate error class into one place (#26985) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-09-11 20:43:17 +08:00
zhagnlu
65cb52d06b
Support dynamic simd framework and using term expr as example (#25260) 
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2023-07-13 16:22:30 +08:00
presburger
5841a3707b
suppport for raft gpu (#22790) 
Signed-off-by: Yusheng.Ma <Yusheng.Ma@zilliz.com>
 2023-03-17 11:59:54 +08:00
xige-16
d4bc00423c
Fix start milvus failed on macos (#19394) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-09-23 16:54:50 +08:00
Enwei Jiao
16c3aedc15
refine complie configuration (#17502) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2022-06-24 21:12:15 +08:00
Jiquan Long
48706f416f
Migrate scalar index from knowhere (#16174) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2022-03-24 14:57:26 +08:00
jaime
307a8ce535
Support compile and run on Mac (#15491) 
Co-authored-by: jaime <yun.zhang@zilliz.com>
Co-authored-by: Cai Yudong <yudong.cai@zilliz.com>
Co-authored-by: Jenny Li <jing.li@zilliz.com>
Co-authored-by: Nemo <yuchen.gao@zilliz.com>
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Co-authored-by: Cai Yudong <yudong.cai@zilliz.com>
Co-authored-by: Jenny Li <jing.li@zilliz.com>
Co-authored-by: Nemo <yuchen.gao@zilliz.com>
 2022-02-09 14:27:46 +08:00
zhenshan.cao
a5e762bb4a
[skip ci]Update license (#13535) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-12-16 20:21:29 +08:00
groot
73c9ab43e0
Simplify cpp (#10413) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2021-10-22 14:33:11 +08:00
groot
525a85ef06
Remove unused code (#10285) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2021-10-20 19:21:18 +08:00
dragondriver
78fe530d87
Initialize the simd type when initialize an indexnode (#8276) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-09-22 16:05:59 +08:00
FluorineDog
b1a9aea6a6
support get entity by ids in segcore (#5456) 
Signed-off-by: fluorinedog <fluorinedog@gmail.com>
 2021-05-28 10:39:30 +08:00
FluorineDog
f39dcdb8f3 Support error code in segcore 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2021-03-26 16:18:30 +08:00
dragondriver
da9ea7d2b9 Add cgo wrapper to index builder 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2020-12-18 15:44:27 +08:00
cai.zhang
85a544b79b Add dockerfile for each component 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2020-11-28 19:06:48 +08:00
dragondriver
568cef0730 Fix InsertTask and SearchTask 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2020-11-28 10:48:29 +08:00
FluorineDog
c9fb34142c Enable primary_key switch 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2020-11-28 09:16:00 +08:00
GuoRentong
15b6963be0 Replace pdf figs with png figs 
Signed-off-by: GuoRentong <rentong.guo@zilliz.com>
 2020-10-31 15:11:47 +08:00
zhenshan.cao
6eee7d1342 Fix bug and update proto 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2020-10-29 19:55:57 +08:00
quicksilver
d66d48c6b6 Enable UnitTest 
Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>
 2020-10-27 15:51:16 +08:00
shengjh
77b2fcf015 Refactor manipulationreq and add tso(pre-alloc from master) 
Signed-off-by: shengjh <1572099106@qq.com>
 2020-10-27 12:01:27 +08:00
quicksilver
eb64839aef Update build environment 
Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>
 2020-10-26 15:45:18 +08:00
zhenshan.cao
64295db471 Refact master and proxy and add etcdutil 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2020-10-15 21:31:50 +08:00