zhagnlu
6c55db44f1
enhance: reorder sub expr for conjunct expr ( #39872 )
...
two point:
(1) reoder conjucts expr's subexpr, postpone heavy operations
sequence: int(column) -> index(column) -> string(column) -> light
conjuct
...... -> json(column) -> heavy conjuct -> two_column_compare
(2) support pre filter for expr execute, skip scan raw data that had
been skipped
because of preceding expr result.
#39869
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2025-03-19 14:50:14 +08:00
Gao
994fc544e7
enhance: support iterative filter execution ( #37363 )
...
issue: #37360
---------
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-12-11 11:32:44 +08:00
Bingyi Sun
b81f162f6a
fix: fix several bugs and refactor some codes related with chunked segment ( #37168 )
...
issue: https://github.com/milvus-io/milvus/issues/37147
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-10-28 14:17:30 +08:00
Yinzuo Jiang
3628593d20
feat: Implement custom function module in milvus expr ( #36560 )
...
OSPP 2024 project:
https://summer-ospp.ac.cn/org/prodetail/247410235?list=org&navpage=org
Solutions:
- parser (planparserv2)
- add CallExpr in planparserv2/Plan.g4
- update parser_visitor and show_visitor
- grpc protobuf
- add CallExpr in plan.proto
- execution (`core/src/exec`)
- add `CallExpr` `ValueExpr` and `ColumnExpr` (both logical and
physical) for function call and function parameters
- function factory (`core/src/exec/expression/function`)
- create a global hashmap when starting milvus (see server.go)
- the global hashmap stores function signatures and their function
pointers, the CallExpr in execution engine can get the function pointer
by function signature.
- custom functions
- empty(string)
- starts_with(string, string)
- add cpp/go unittests and E2E tests
closes : #36559
Signed-off-by: Yinzuo Jiang <jiangyinzuo@foxmail.com>
2024-10-25 15:25:30 +08:00
smellthemoon
eb3e4583ec
enhance: all op(Null) is false in expr ( #35527 )
...
#31728
---------
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-10-17 21:14:30 +08:00
Bingyi Sun
a75bb85f3a
feat: support chunked column for sealed segment ( #35764 )
...
This PR splits sealed segment to chunked data to avoid unnecessary
memory copy and save memory usage when loading segments so that loading
can be accelerated.
To support rollback to previous version, we add an option
`multipleChunkedEnable` which is false by default.
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-10-12 15:04:52 +08:00
smellthemoon
475c333fa2
enhance: add valid_data in span ( #35030 )
...
#31728
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-08-02 15:40:14 +08:00
zhagnlu
3030e4625e
enhance: refactor variable column to reduce memory cost ( #33875 )
...
#33874
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-06-30 20:16:06 +08:00
cqy123456
32f685ff12
enhance: growing segment support mmap ( #32633 )
...
issue: https://github.com/milvus-io/milvus/issues/32984
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-06-18 14:42:00 +08:00
Alexander Guzhva
c4b37fb285
enhance: Custom bitset and bitsetview prototypes ( #30454 )
...
Issue: #31285
Basically, I've replaced `FixedVector<bool>` and `boost::dynamic_bitset`
with custom bitset and bitsetview in order to reduce the memory
bandwidth & increase performance for the filtering.
This PR is for internal use only.
Current progress (numbers are for GCC 9.5.0 on Ubuntu 22.04 LTS;
clang-17 produces better performance numbers):
Baseline:
```
[ RUN ] CApiTest.AssembeChunkPerfTest
start test
cost: 17903us
[ OK ] CApiTest.AssembeChunkPerfTest (183 ms)
[ RUN ] Expr.TestMultiLogicalExprsOptimization
cost: 1391us
cost: 5us
cost: 4us
cost: 4us
cost: 6us
cost: 4us
cost: 4us
cost: 4us
cost: 4us
cost: 4us
143
cost: 10us
cost: 8us
cost: 10us
cost: 8us
cost: 8us
cost: 8us
cost: 8us
cost: 8us
cost: 8us
cost: 9us
8
/home/ubuntu/zilliz/milvus4/milvus/internal/core/unittest/test_expr.cpp:1561: Failure
Expected: (cost_op) < (cost_no_op), actual: 143 vs 8
[ FAILED ] Expr.TestMultiLogicalExprsOptimization (7 ms)
[ RUN ] Expr.TestExprs
start test
3cost: 889us
start test
10cost: 2us
start test
20cost: 2us
start test
30cost: 2us
start test
50cost: 3us
start test
100cost: 7us
start test
200cost: 16us
[ OK ] Expr.TestExprs (9 ms)
[ RUN ] Expr.TestUnaryBenchTest
start test type:2
cost: 124.8us
start test type:3
cost: 163.1us
start test type:4
cost: 275.9us
start test type:5
cost: 590.9us
start test type:10
cost: 62.7us
start test type:11
cost: 65.9us
[ OK ] Expr.TestUnaryBenchTest (1153 ms)
[ RUN ] Expr.TestBinaryRangeBenchTest
start test type:2
cost: 151.4us
start test type:3
cost: 198.4us
start test type:4
cost: 361.9us
start test type:5
cost: 753.9us
start test type:10
cost: 64.6us
start test type:11
cost: 62.2us
[ OK ] Expr.TestBinaryRangeBenchTest (1151 ms)
[ RUN ] Expr.TestLogicalUnaryBenchTest
start test type:2
cost: 121.14us
start test type:3
cost: 156.84us
start test type:4
cost: 249.76us
start test type:5
cost: 534.44us
start test type:10
cost: 82.2us
start test type:11
cost: 83.52us
[ OK ] Expr.TestLogicalUnaryBenchTest (1202 ms)
[ RUN ] Expr.TestBinaryLogicalBenchTest
start test type:2
cost: 80.64us
start test type:3
cost: 78.22us
start test type:4
cost: 255.76us
start test type:5
cost: 532.04us
start test type:10
cost: 89.26us
start test type:11
cost: 90us
[ OK ] Expr.TestBinaryLogicalBenchTest (1198 ms)
[ RUN ] Expr.TestBinaryArithOpEvalRangeBenchExpr
start test type:2
cost: 401.7us
start test type:3
cost: 420.96us
start test type:4
cost: 418.04us
start test type:5
cost: 470.54us
start test type:10
cost: 250.32us
start test type:11
cost: 850.08us
[ OK ] Expr.TestBinaryArithOpEvalRangeBenchExpr (1273 ms)
[ RUN ] Expr.TestCompareExprBenchTest
start test type:2
cost: 162us
start test type:3
cost: 142us
start test type:4
cost: 374us
start test type:5
cost: 674us
start test type:10
cost: 366us
start test type:11
cost: 645us
[ OK ] Expr.TestCompareExprBenchTest (1214 ms)
[ RUN ] Expr.TestRefactorExprs
start test
3cost: 1253us
start test
10cost: 1060us
start test
20cost: 681us
start test
30cost: 522us
start test
50cost: 511us
start test
100cost: 506us
start test
200cost: 497us
[ OK ] Expr.TestRefactorExprs (1142 ms)
```
Candidate:
```
[ RUN ] CApiTest.AssembeChunkPerfTest
start test
cost: 6099us
[ OK ] CApiTest.AssembeChunkPerfTest (153 ms)
[ RUN ] Expr.TestMultiLogicalExprsOptimization
cost: 42us
cost: 15us
cost: 15us
cost: 14us
cost: 15us
cost: 15us
cost: 15us
cost: 15us
cost: 15us
cost: 15us
17
cost: 41us
cost: 39us
cost: 33us
cost: 33us
cost: 33us
cost: 33us
cost: 34us
cost: 41us
cost: 34us
cost: 34us
35
[ OK ] Expr.TestMultiLogicalExprsOptimization (6 ms)
[ RUN ] Expr.TestExprs
start test
3cost: 20us
start test
10cost: 2us
start test
20cost: 2us
start test
30cost: 2us
start test
50cost: 4us
start test
100cost: 8us
start test
200cost: 15us
[ OK ] Expr.TestExprs (8 ms)
[ RUN ] Expr.TestUnaryBenchTest
start test type:2
cost: 55.7us
start test type:3
cost: 79.8us
start test type:4
cost: 177.6us
start test type:5
cost: 337.2us
start test type:10
cost: 16.9us
start test type:11
cost: 15.7us
[ OK ] Expr.TestUnaryBenchTest (1140 ms)
[ RUN ] Expr.TestBinaryRangeBenchTest
start test type:2
cost: 57.1us
start test type:3
cost: 87us
start test type:4
cost: 177.5us
start test type:5
cost: 342.7us
start test type:10
cost: 17.9us
start test type:11
cost: 16.7us
[ OK ] Expr.TestBinaryRangeBenchTest (1152 ms)
[ RUN ] Expr.TestLogicalUnaryBenchTest
start test type:2
cost: 34.58us
start test type:3
cost: 68.86us
start test type:4
cost: 151.38us
start test type:5
cost: 286.8us
start test type:10
cost: 16.54us
start test type:11
cost: 16.7us
[ OK ] Expr.TestLogicalUnaryBenchTest (1165 ms)
[ RUN ] Expr.TestBinaryLogicalBenchTest
start test type:2
cost: 20us
start test type:3
cost: 17.1us
start test type:4
cost: 154.12us
start test type:5
cost: 286.1us
start test type:10
cost: 19.6us
start test type:11
cost: 19.24us
[ OK ] Expr.TestBinaryLogicalBenchTest (1188 ms)
[ RUN ] Expr.TestBinaryArithOpEvalRangeBenchExpr
start test type:2
cost: 125.7us
start test type:3
cost: 111.34us
start test type:4
cost: 148.02us
start test type:5
cost: 306.7us
start test type:10
cost: 149.3us
start test type:11
cost: 282.94us
[ OK ] Expr.TestBinaryArithOpEvalRangeBenchExpr (1221 ms)
[ RUN ] Expr.TestCompareExprBenchTest
start test type:2
cost: 89us
start test type:3
cost: 79us
start test type:4
cost: 323us
start test type:5
cost: 629us
start test type:10
cost: 313us
start test type:11
cost: 591us
[ OK ] Expr.TestCompareExprBenchTest (1228 ms)
[ RUN ] Expr.TestRefactorExprs
start test
3cost: 874us
start test
10cost: 611us
start test
20cost: 290us
start test
30cost: 294us
start test
50cost: 272us
start test
100cost: 278us
start test
200cost: 279us
[ OK ] Expr.TestRefactorExprs (1149 ms)
```
Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>
2024-03-24 21:49:07 +08:00
zhagnlu
79c417b14e
fix: pass active count to query context instead of timestamp ( #29541 )
...
#29319
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2023-12-31 16:08:48 +08:00
yah01
8f89e9cf75
enhance: remove all unnecessary string formatting ( #29323 )
...
done by two regex expressions:
- `PanicInfo\((.+),[. \n]+fmt::format\(([.\s\S]+?)\)\)`
- `AssertInfo\((.+),[. \n]+fmt::format\(([.\s\S]+?)\)\)`
related: #28811
---------
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-12-20 10:04:43 +08:00
zhagnlu
a602171d06
enhance: Refactor runtime and expr framework ( #28166 )
...
#28165
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2023-12-18 12:04:42 +08:00
