admin/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2026-01-07 19:31:51 +08:00
Code Issues Packages Projects Releases Wiki Activity
19,535 Commits 137 Branches 188 Tags
Commit Graph

72 Commits

Author SHA1 Message Date
Alexander Guzhva
c4b37fb285
enhance: Custom bitset and bitsetview prototypes (#30454) 
Issue: #31285 

Basically, I've replaced `FixedVector<bool>` and `boost::dynamic_bitset`
with custom bitset and bitsetview in order to reduce the memory
bandwidth & increase performance for the filtering.

This PR is for internal use only. 

Current progress (numbers are for GCC 9.5.0 on Ubuntu 22.04 LTS;
clang-17 produces better performance numbers):
Baseline:
```
[ RUN      ] CApiTest.AssembeChunkPerfTest
start test
cost: 17903us
[       OK ] CApiTest.AssembeChunkPerfTest (183 ms)

[ RUN      ] Expr.TestMultiLogicalExprsOptimization
cost: 1391us
cost: 5us
cost: 4us
cost: 4us
cost: 6us
cost: 4us
cost: 4us
cost: 4us
cost: 4us
cost: 4us
143
cost: 10us
cost: 8us
cost: 10us
cost: 8us
cost: 8us
cost: 8us
cost: 8us
cost: 8us
cost: 8us
cost: 9us
8
/home/ubuntu/zilliz/milvus4/milvus/internal/core/unittest/test_expr.cpp:1561: Failure
Expected: (cost_op) < (cost_no_op), actual: 143 vs 8
[  FAILED  ] Expr.TestMultiLogicalExprsOptimization (7 ms)
[ RUN      ] Expr.TestExprs
start test
3cost: 889us
start test
10cost: 2us
start test
20cost: 2us
start test
30cost: 2us
start test
50cost: 3us
start test
100cost: 7us
start test
200cost: 16us
[       OK ] Expr.TestExprs (9 ms)

[ RUN      ] Expr.TestUnaryBenchTest
start test type:2
 cost: 124.8us
start test type:3
 cost: 163.1us
start test type:4
 cost: 275.9us
start test type:5
 cost: 590.9us
start test type:10
 cost: 62.7us
start test type:11
 cost: 65.9us
[       OK ] Expr.TestUnaryBenchTest (1153 ms)
[ RUN      ] Expr.TestBinaryRangeBenchTest
start test type:2
 cost: 151.4us
start test type:3
 cost: 198.4us
start test type:4
 cost: 361.9us
start test type:5
 cost: 753.9us
start test type:10
 cost: 64.6us
start test type:11
 cost: 62.2us
[       OK ] Expr.TestBinaryRangeBenchTest (1151 ms)
[ RUN      ] Expr.TestLogicalUnaryBenchTest
start test type:2
 cost: 121.14us
start test type:3
 cost: 156.84us
start test type:4
 cost: 249.76us
start test type:5
 cost: 534.44us
start test type:10
 cost: 82.2us
start test type:11
 cost: 83.52us
[       OK ] Expr.TestLogicalUnaryBenchTest (1202 ms)
[ RUN      ] Expr.TestBinaryLogicalBenchTest
start test type:2
 cost: 80.64us
start test type:3
 cost: 78.22us
start test type:4
 cost: 255.76us
start test type:5
 cost: 532.04us
start test type:10
 cost: 89.26us
start test type:11
 cost: 90us
[       OK ] Expr.TestBinaryLogicalBenchTest (1198 ms)
[ RUN      ] Expr.TestBinaryArithOpEvalRangeBenchExpr
start test type:2
 cost: 401.7us
start test type:3
 cost: 420.96us
start test type:4
 cost: 418.04us
start test type:5
 cost: 470.54us
start test type:10
 cost: 250.32us
start test type:11
 cost: 850.08us
[       OK ] Expr.TestBinaryArithOpEvalRangeBenchExpr (1273 ms)
[ RUN      ] Expr.TestCompareExprBenchTest
start test type:2
 cost: 162us
start test type:3
 cost: 142us
start test type:4
 cost: 374us
start test type:5
 cost: 674us
start test type:10
 cost: 366us
start test type:11
 cost: 645us
[       OK ] Expr.TestCompareExprBenchTest (1214 ms)
[ RUN      ] Expr.TestRefactorExprs
start test
3cost: 1253us
start test
10cost: 1060us
start test
20cost: 681us
start test
30cost: 522us
start test
50cost: 511us
start test
100cost: 506us
start test
200cost: 497us
[       OK ] Expr.TestRefactorExprs (1142 ms)

```

Candidate:
```
[ RUN      ] CApiTest.AssembeChunkPerfTest
start test
cost: 6099us
[       OK ] CApiTest.AssembeChunkPerfTest (153 ms)

[ RUN      ] Expr.TestMultiLogicalExprsOptimization
cost: 42us
cost: 15us
cost: 15us
cost: 14us
cost: 15us
cost: 15us
cost: 15us
cost: 15us
cost: 15us
cost: 15us
17
cost: 41us
cost: 39us
cost: 33us
cost: 33us
cost: 33us
cost: 33us
cost: 34us
cost: 41us
cost: 34us
cost: 34us
35
[       OK ] Expr.TestMultiLogicalExprsOptimization (6 ms)
[ RUN      ] Expr.TestExprs
start test
3cost: 20us
start test
10cost: 2us
start test
20cost: 2us
start test
30cost: 2us
start test
50cost: 4us
start test
100cost: 8us
start test
200cost: 15us
[       OK ] Expr.TestExprs (8 ms)

[ RUN      ] Expr.TestUnaryBenchTest
start test type:2
 cost: 55.7us
start test type:3
 cost: 79.8us
start test type:4
 cost: 177.6us
start test type:5
 cost: 337.2us
start test type:10
 cost: 16.9us
start test type:11
 cost: 15.7us
[       OK ] Expr.TestUnaryBenchTest (1140 ms)
[ RUN      ] Expr.TestBinaryRangeBenchTest
start test type:2
 cost: 57.1us
start test type:3
 cost: 87us
start test type:4
 cost: 177.5us
start test type:5
 cost: 342.7us
start test type:10
 cost: 17.9us
start test type:11
 cost: 16.7us
[       OK ] Expr.TestBinaryRangeBenchTest (1152 ms)
[ RUN      ] Expr.TestLogicalUnaryBenchTest
start test type:2
 cost: 34.58us
start test type:3
 cost: 68.86us
start test type:4
 cost: 151.38us
start test type:5
 cost: 286.8us
start test type:10
 cost: 16.54us
start test type:11
 cost: 16.7us
[       OK ] Expr.TestLogicalUnaryBenchTest (1165 ms)
[ RUN      ] Expr.TestBinaryLogicalBenchTest
start test type:2
 cost: 20us
start test type:3
 cost: 17.1us
start test type:4
 cost: 154.12us
start test type:5
 cost: 286.1us
start test type:10
 cost: 19.6us
start test type:11
 cost: 19.24us
[       OK ] Expr.TestBinaryLogicalBenchTest (1188 ms)
[ RUN      ] Expr.TestBinaryArithOpEvalRangeBenchExpr
start test type:2
 cost: 125.7us
start test type:3
 cost: 111.34us
start test type:4
 cost: 148.02us
start test type:5
 cost: 306.7us
start test type:10
 cost: 149.3us
start test type:11
 cost: 282.94us
[       OK ] Expr.TestBinaryArithOpEvalRangeBenchExpr (1221 ms)
[ RUN      ] Expr.TestCompareExprBenchTest
start test type:2
 cost: 89us
start test type:3
 cost: 79us
start test type:4
 cost: 323us
start test type:5
 cost: 629us
start test type:10
 cost: 313us
start test type:11
 cost: 591us
[       OK ] Expr.TestCompareExprBenchTest (1228 ms)
[ RUN      ] Expr.TestRefactorExprs
start test
3cost: 874us
start test
10cost: 611us
start test
20cost: 290us
start test
30cost: 294us
start test
50cost: 272us
start test
100cost: 278us
start test
200cost: 279us
[       OK ] Expr.TestRefactorExprs (1149 ms)

```

Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>
 2024-03-24 21:49:07 +08:00
Buqian Zheng
96cfae55a5
feat: [Sparse Float Vector] segcore to support sparse vector search and get raw vector by id (#30629) 
This PR adds the ability to search/get sparse float vectors in segcore,
and added unit tests by modifying lots of existing tests into
parameterized ones.

https://github.com/milvus-io/milvus/issues/29419

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
 2024-03-12 09:16:30 -07:00
Xu Tong
e429965f32
Add float16 approve for multi-type part (#28427) 
issue：https://github.com/milvus-io/milvus/issues/22837

Add bfloat16 vector, add the index part of float16 vector.

Signed-off-by: Writer-X <1256866856@qq.com>
 2024-01-11 15:48:51 +08:00
zhagnlu
79c417b14e
fix: pass active count to query context instead of timestamp (#29541) 
#29319

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2023-12-31 16:08:48 +08:00
zhagnlu
a602171d06
enhance: Refactor runtime and expr framework (#28166) 
#28165

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2023-12-18 12:04:42 +08:00
MrPresent-Han
836f300536
support skip-index based on chunk-metrics to accelerate expr filter(#27925) (#28297) 
related: #27925

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-11-15 11:20:19 +08:00
cai.zhang
a362bb1457
Support array datatype (#26369) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-09-19 14:23:23 +08:00
Enwei Jiao
c3f15c6b95
Refactor duplicate error class into one place (#26985) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-09-11 20:43:17 +08:00
Xu Tong
9166011c4a
Add float16 vector (#25852) 
Signed-off-by: Writer-X <1256866856@qq.com>
 2023-09-08 10:03:16 +08:00
cai.zhang
a0198ce8ae
Support json contains feature (#25384) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-08-11 17:09:30 +08:00
Jiquan Long
6578a27fe7
Support empty expression (#26153) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-08-10 08:37:15 +08:00
zhagnlu
f3f3f8a849
Segcore retrieve by pk optimazation (#24659) (#24660) 
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2023-06-15 15:04:39 +08:00
Jiquan Long
7faf934f91
Fix UnaryRange expression on integer overflow case (#24893) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-06-15 14:06:39 +08:00
zhagnlu
f60b839127
Support element in json array in segcore part(#24677) (#24829) 
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2023-06-14 14:38:37 +08:00
zhagnlu
113f9a0ebc
Support SIMD of several Expr (#23715) (#23717) 
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2023-05-12 14:11:20 +08:00
cai.zhang
9715a850fa
Support expr with json field (#23804) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-05-10 10:19:19 +08:00
yah01
62eea5286f
Support to filter with json expr (#23739) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-04-30 20:36:39 +08:00
yah01
bdd6bc7695
Re-format cpp code (#22513) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-03-02 15:55:49 +08:00
Jiquan Long
8d78d9943a
Fix term filtering failed on indexed scalar field (#20840) (#20882) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>

Signed-off-by: longjiquan <jiquan.long@zilliz.com>

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-11-30 10:05:15 +08:00
bigsheeper
f38637c227
Pass PlaceholderGroup pointer to prevent memory copy in SegCore (#17389) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-06-06 21:34:05 +08:00
xige-16
56778787be
Reverse data from scalar index (#17145) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-05-26 14:58:01 +08:00
xige-16
515d0369de
Support string type in segcore (#16546) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
Co-authored-by: dragondriver <jiquan.long@zilliz.com>

Co-authored-by: dragondriver <jiquan.long@zilliz.com>
 2022-04-29 13:35:49 +08:00
Aivin V. Solatorio
04fffb08ef
Support arithmetic operations on numerical fields for scalar filtering (#16520) 
Signed-off-by: Aivin V. Solatorio <avsolatorio@gmail.com>
 2022-04-24 16:43:45 +08:00
Cai Yudong
54b8b24151
Rename variable names for better readibility (#15700) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-02-22 22:15:52 +08:00
jaime
307a8ce535
Support compile and run on Mac (#15491) 
Co-authored-by: jaime <yun.zhang@zilliz.com>
Co-authored-by: Cai Yudong <yudong.cai@zilliz.com>
Co-authored-by: Jenny Li <jing.li@zilliz.com>
Co-authored-by: Nemo <yuchen.gao@zilliz.com>
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Co-authored-by: Cai Yudong <yudong.cai@zilliz.com>
Co-authored-by: Jenny Li <jing.li@zilliz.com>
Co-authored-by: Nemo <yuchen.gao@zilliz.com>
 2022-02-09 14:27:46 +08:00
zhenshan.cao
5cd9ea5f04
Fixbug: init optional retrieve_ret_ when query result is empty (#8068) 
Add unittest

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-09-16 20:11:49 +08:00
yukun
94272bba87
Support query by expression (#7386) 
Signed-off-by: fishpenguin <kun.yu@zilliz.com>
 2021-09-03 17:12:55 +08:00
xaxys
e43b43e7d8
Refactor RangeExpr & CompareExpr (#6786) 
* Changed `RangeExpr` proto to `UnaryRangeExpr` & `BinaryRangeExpr`
Several unit test unpassed.

Signed-off-by: xaxys <tpnnghd@163.com>

* Fix bugs to pass unit test. Fix format.

Signed-off-by: xaxys <tpnnghd@163.com>

* Remove debug information.

Signed-off-by: xaxys <tpnnghd@163.com>

* Fix format.
Remove debug information.
Unify variable name.
Add error information.
Remove `CompareExpr` test in `test_c_api.cpp`.

Signed-off-by: xaxys <tpnnghd@163.com>

* Fix code format.

Signed-off-by: xaxys <tpnnghd@163.com>

* Update `Plan.cpp`.

Signed-off-by: xaxys <tpnnghd@163.com>
 2021-07-25 10:23:19 +08:00
FluorineDog
aba21baf82
refactor expr executor to use single bitset (#6667) 
Signed-off-by: fluorinedog <fluorinedog@gmail.com>
 2021-07-23 15:30:12 +08:00
Cai Yudong
724f10b9a0
Unify the usage of query and search (#6467) 
Unify the usage of query and search

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-07-13 22:20:33 +08:00
xaxys
e5f8a77a2d
Extend grammar of boolean expression (#6460) 
* Extend grammar of boolean expression

Signed-off-by: xaxys <tpnnghd@163.com>
 2021-07-13 14:37:03 +08:00
FluorineDog
b90b4f2058
renaming (#6192) 
Signed-off-by: fluorinedog <fluorinedog@gmail.com>
 2021-06-30 10:18:26 +08:00
FluorineDog
255e3959af
support time travel (#5894) 
* support time travel

Signed-off-by: fluorinedog <fluorinedog@gmail.com>

* lint

Signed-off-by: fluorinedog <fluorinedog@gmail.com>
 2021-06-19 17:38:11 +08:00
FluorineDog
f39dcdb8f3 Support error code in segcore 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2021-03-26 16:18:30 +08:00
cai.zhang
f940cc455a Add dockerfile for index 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2021-01-26 09:38:40 +08:00
neza2017
7f3aa92d10 Add docker file for masterservice 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-01-25 15:58:04 +08:00
FluorineDog
a8fd6e7686 Enable drop index/field_data and extract info visitor 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2021-01-25 18:59:44 +08:00
FluorineDog
7f044fff82 Enable search for sealed 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2021-01-21 15:29:52 +08:00
FluorineDog
9ff1bc0f99 Adapt Expr Executor to SegmentInternalInterface, fix row_count consistency 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2021-01-16 18:12:58 +08:00
bigsheeper
f02bd8c8f5 Rename query node package, implement watchDmChannel 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-01-16 10:12:14 +08:00
yukun
44c75bdd2a Add consume function in rocksmq 
Signed-off-by: yukun <kun.yu@zilliz.com>
 2021-01-15 20:12:26 +08:00
FluorineDog
c0a3a509f7 Remove todos, implement chunk_data and chunk_scalar_index 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2021-01-15 18:23:50 +08:00
FluorineDog
63ca5f8031 Add span and segment interface, rename segment_growing and use interface as base 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2021-01-13 18:46:25 +08:00
neza2017
4015d7245d Merge operation 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-01-06 14:45:50 +08:00
cai.zhang
1cfc2ff0a5 Add make clean in Makefile 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2021-01-06 13:50:48 +08:00
FluorineDog
5a26f6ef21 Enable sub_query_result 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2021-01-06 12:01:13 +08:00
godchen
1c87faf8b3 Add insert request opentracing 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-01-06 11:17:35 +08:00
FluorineDog
e16a7478da Add verifier 
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
 2021-01-05 20:13:54 +08:00
sunby
ed54d3e26c Fix binlog reader bug 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2020-12-09 09:55:56 +08:00
quicksilver
d09ad77fb2 Add archiving file workflow for cache thirdparty 
Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>
 2020-12-08 19:53:22 +08:00