zhagnlu
3030e4625e
enhance: refactor variable column to reduce memory cost ( #33875 )
...
#33874
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-06-30 20:16:06 +08:00
zhagnlu
03a3f50892
enhance: add skip using array index when some situation ( #33947 )
...
#32900
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-06-23 21:26:02 +08:00
zhagnlu
0d7ea8ec42
enhance: Enhance and correct exception module ( #33705 )
...
#33704
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-06-23 21:22:01 +08:00
cqy123456
32f685ff12
enhance: growing segment support mmap ( #32633 )
...
issue: https://github.com/milvus-io/milvus/issues/32984
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-06-18 14:42:00 +08:00
Jiquan Long
ecf2bcee42
enhance: speed up array-equal operator via inverted index ( #33633 )
...
fix : #33632
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-06-11 14:13:54 +08:00
Jiquan Long
0c5d8660aa
feat: support inverted index for array ( #33452 )
...
issue: https://github.com/milvus-io/milvus/issues/27704
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-05-31 09:47:47 +08:00
Jiquan Long
ccce1e928a
fix: regex query can't handle text with newline ( #32569 )
...
issue: https://github.com/milvus-io/milvus/issues/32482
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-04-26 12:01:26 +08:00
Cai Yudong
246586be27
enhance: Unify data type check APIs under internal/core ( #31800 )
...
Issue: #22837
Move and rename following C++ APIs:
datatype_sizeof() ==> GetDataTypeSize()
datatype_name() ==> GetDataTypeName()
datatype_is_vector() / IsVectorType() ==> IsVectorDataType()
datatype_is_variable() ==> IsVariableDataType()
datatype_is_sparse_vector() ==> IsSparseFloatVectorDataType()
datatype_is_string() / IsString() ==> IsDataTypeString()
datatype_is_floating() / IsFloat() ==> IsDataTypeFloat()
datatype_is_binary() ==> IsDataTypeBinary()
datatype_is_json() ==> IsDataTypeJson()
datatype_is_array() ==> IsDataTypeArray()
datatype_is_variable() == IsDataTypeVariable()
datatype_is_integer() / IsIntegral() ==> IsDataTypeInteger()
Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2024-04-02 19:15:14 +08:00
Alexander Guzhva
c4b37fb285
enhance: Custom bitset and bitsetview prototypes ( #30454 )
...
Issue: #31285
Basically, I've replaced `FixedVector<bool>` and `boost::dynamic_bitset`
with custom bitset and bitsetview in order to reduce the memory
bandwidth & increase performance for the filtering.
This PR is for internal use only.
Current progress (numbers are for GCC 9.5.0 on Ubuntu 22.04 LTS;
clang-17 produces better performance numbers):
Baseline:
```
[ RUN ] CApiTest.AssembeChunkPerfTest
start test
cost: 17903us
[ OK ] CApiTest.AssembeChunkPerfTest (183 ms)
[ RUN ] Expr.TestMultiLogicalExprsOptimization
cost: 1391us
cost: 5us
cost: 4us
cost: 4us
cost: 6us
cost: 4us
cost: 4us
cost: 4us
cost: 4us
cost: 4us
143
cost: 10us
cost: 8us
cost: 10us
cost: 8us
cost: 8us
cost: 8us
cost: 8us
cost: 8us
cost: 8us
cost: 9us
8
/home/ubuntu/zilliz/milvus4/milvus/internal/core/unittest/test_expr.cpp:1561: Failure
Expected: (cost_op) < (cost_no_op), actual: 143 vs 8
[ FAILED ] Expr.TestMultiLogicalExprsOptimization (7 ms)
[ RUN ] Expr.TestExprs
start test
3cost: 889us
start test
10cost: 2us
start test
20cost: 2us
start test
30cost: 2us
start test
50cost: 3us
start test
100cost: 7us
start test
200cost: 16us
[ OK ] Expr.TestExprs (9 ms)
[ RUN ] Expr.TestUnaryBenchTest
start test type:2
cost: 124.8us
start test type:3
cost: 163.1us
start test type:4
cost: 275.9us
start test type:5
cost: 590.9us
start test type:10
cost: 62.7us
start test type:11
cost: 65.9us
[ OK ] Expr.TestUnaryBenchTest (1153 ms)
[ RUN ] Expr.TestBinaryRangeBenchTest
start test type:2
cost: 151.4us
start test type:3
cost: 198.4us
start test type:4
cost: 361.9us
start test type:5
cost: 753.9us
start test type:10
cost: 64.6us
start test type:11
cost: 62.2us
[ OK ] Expr.TestBinaryRangeBenchTest (1151 ms)
[ RUN ] Expr.TestLogicalUnaryBenchTest
start test type:2
cost: 121.14us
start test type:3
cost: 156.84us
start test type:4
cost: 249.76us
start test type:5
cost: 534.44us
start test type:10
cost: 82.2us
start test type:11
cost: 83.52us
[ OK ] Expr.TestLogicalUnaryBenchTest (1202 ms)
[ RUN ] Expr.TestBinaryLogicalBenchTest
start test type:2
cost: 80.64us
start test type:3
cost: 78.22us
start test type:4
cost: 255.76us
start test type:5
cost: 532.04us
start test type:10
cost: 89.26us
start test type:11
cost: 90us
[ OK ] Expr.TestBinaryLogicalBenchTest (1198 ms)
[ RUN ] Expr.TestBinaryArithOpEvalRangeBenchExpr
start test type:2
cost: 401.7us
start test type:3
cost: 420.96us
start test type:4
cost: 418.04us
start test type:5
cost: 470.54us
start test type:10
cost: 250.32us
start test type:11
cost: 850.08us
[ OK ] Expr.TestBinaryArithOpEvalRangeBenchExpr (1273 ms)
[ RUN ] Expr.TestCompareExprBenchTest
start test type:2
cost: 162us
start test type:3
cost: 142us
start test type:4
cost: 374us
start test type:5
cost: 674us
start test type:10
cost: 366us
start test type:11
cost: 645us
[ OK ] Expr.TestCompareExprBenchTest (1214 ms)
[ RUN ] Expr.TestRefactorExprs
start test
3cost: 1253us
start test
10cost: 1060us
start test
20cost: 681us
start test
30cost: 522us
start test
50cost: 511us
start test
100cost: 506us
start test
200cost: 497us
[ OK ] Expr.TestRefactorExprs (1142 ms)
```
Candidate:
```
[ RUN ] CApiTest.AssembeChunkPerfTest
start test
cost: 6099us
[ OK ] CApiTest.AssembeChunkPerfTest (153 ms)
[ RUN ] Expr.TestMultiLogicalExprsOptimization
cost: 42us
cost: 15us
cost: 15us
cost: 14us
cost: 15us
cost: 15us
cost: 15us
cost: 15us
cost: 15us
cost: 15us
17
cost: 41us
cost: 39us
cost: 33us
cost: 33us
cost: 33us
cost: 33us
cost: 34us
cost: 41us
cost: 34us
cost: 34us
35
[ OK ] Expr.TestMultiLogicalExprsOptimization (6 ms)
[ RUN ] Expr.TestExprs
start test
3cost: 20us
start test
10cost: 2us
start test
20cost: 2us
start test
30cost: 2us
start test
50cost: 4us
start test
100cost: 8us
start test
200cost: 15us
[ OK ] Expr.TestExprs (8 ms)
[ RUN ] Expr.TestUnaryBenchTest
start test type:2
cost: 55.7us
start test type:3
cost: 79.8us
start test type:4
cost: 177.6us
start test type:5
cost: 337.2us
start test type:10
cost: 16.9us
start test type:11
cost: 15.7us
[ OK ] Expr.TestUnaryBenchTest (1140 ms)
[ RUN ] Expr.TestBinaryRangeBenchTest
start test type:2
cost: 57.1us
start test type:3
cost: 87us
start test type:4
cost: 177.5us
start test type:5
cost: 342.7us
start test type:10
cost: 17.9us
start test type:11
cost: 16.7us
[ OK ] Expr.TestBinaryRangeBenchTest (1152 ms)
[ RUN ] Expr.TestLogicalUnaryBenchTest
start test type:2
cost: 34.58us
start test type:3
cost: 68.86us
start test type:4
cost: 151.38us
start test type:5
cost: 286.8us
start test type:10
cost: 16.54us
start test type:11
cost: 16.7us
[ OK ] Expr.TestLogicalUnaryBenchTest (1165 ms)
[ RUN ] Expr.TestBinaryLogicalBenchTest
start test type:2
cost: 20us
start test type:3
cost: 17.1us
start test type:4
cost: 154.12us
start test type:5
cost: 286.1us
start test type:10
cost: 19.6us
start test type:11
cost: 19.24us
[ OK ] Expr.TestBinaryLogicalBenchTest (1188 ms)
[ RUN ] Expr.TestBinaryArithOpEvalRangeBenchExpr
start test type:2
cost: 125.7us
start test type:3
cost: 111.34us
start test type:4
cost: 148.02us
start test type:5
cost: 306.7us
start test type:10
cost: 149.3us
start test type:11
cost: 282.94us
[ OK ] Expr.TestBinaryArithOpEvalRangeBenchExpr (1221 ms)
[ RUN ] Expr.TestCompareExprBenchTest
start test type:2
cost: 89us
start test type:3
cost: 79us
start test type:4
cost: 323us
start test type:5
cost: 629us
start test type:10
cost: 313us
start test type:11
cost: 591us
[ OK ] Expr.TestCompareExprBenchTest (1228 ms)
[ RUN ] Expr.TestRefactorExprs
start test
3cost: 874us
start test
10cost: 611us
start test
20cost: 290us
start test
30cost: 294us
start test
50cost: 272us
start test
100cost: 278us
start test
200cost: 279us
[ OK ] Expr.TestRefactorExprs (1149 ms)
```
Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>
2024-03-24 21:49:07 +08:00
zhagnlu
c8b54f321a
fix:restrict pk in [...] optimization situations ( #31184 )
...
#31154
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-03-12 14:49:03 +08:00
cai.zhang
1aa97a5c21
enhance: Support more relational operators for binary expressions ( #30902 )
...
issue: #30677
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-03-01 16:57:00 +08:00
Jiquan Long
e2f35954d4
enhance: support pattern matching on json field ( #30779 )
...
issue: https://github.com/milvus-io/milvus/issues/30714
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-02-28 18:31:00 +08:00
Jiquan Long
e2330f02f8
fix: pattern match use incorrect raw data ( #30764 )
...
issue: https://github.com/milvus-io/milvus/issues/30687
We store all the varchar datas in an continuous address and use
string_view to quickly find them. In this case, using string_view.data()
directly will point to all rest varchar datas.
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-02-22 19:56:52 +08:00
Alexander Guzhva
a297baae9d
enhance: remove unused code ( #30601 )
...
Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>
2024-02-13 10:26:47 +08:00
Jiquan Long
e549148a19
enhance: full-support for wildcard pattern matching ( #30288 )
...
issue: #29988
This pr adds full-support for wildcard pattern matching from end to end.
Before this pr, the users can only use prefix match in their expression,
for example, "like 'prefix%'". With this pr, more flexible syntax can be
combined.
To do so, this pr makes these changes:
- 1. support regex query both on index and raw data;
- 2. translate the pattern matching to regex query, so that it can be
handled by the regex query logic;
- 3. loose the limit of the expression parsing, which allows general
pattern matching syntax;
With the support of regex query in segcore backend, we can also add
mysql-like `REGEXP` syntax later easily.
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-02-01 12:37:04 +08:00
zhagnlu
601a8b801b
fix: add move cursor function to physical expr ( #29603 )
...
#29570
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-01-09 17:08:48 +08:00
zhagnlu
d07197ab1a
enhance: add compare simd function ( #29432 )
...
#26137
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-01-07 20:20:57 +08:00
zhagnlu
79c417b14e
fix: pass active count to query context instead of timestamp ( #29541 )
...
#29319
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2023-12-31 16:08:48 +08:00
yah01
aef483806d
enhance: improve the segcore logs ( #29372 )
...
- remove the streaming logging
- refine existing logs
fix #29366
---------
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-12-23 21:52:43 +08:00
zhagnlu
1cbe3cd5fc
fix: fix memory leak when cancel segcore task ( #29431 )
...
#29430
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2023-12-22 20:28:43 +08:00
zhagnlu
a6eb7e5f9a
enhance: skip segment when using pk in (..) expr ( #29394 )
...
#29293
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2023-12-21 20:06:42 +08:00
yah01
04b2518ae7
enhance: fix the incorrect init parameter ( #29357 )
...
as the `driver_` field is not used so this doesn't matter for now
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-12-20 20:50:43 +08:00
yah01
8f89e9cf75
enhance: remove all unnecessary string formatting ( #29323 )
...
done by two regex expressions:
- `PanicInfo\((.+),[. \n]+fmt::format\(([.\s\S]+?)\)\)`
- `AssertInfo\((.+),[. \n]+fmt::format\(([.\s\S]+?)\)\)`
related: #28811
---------
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-12-20 10:04:43 +08:00
zhagnlu
a602171d06
enhance: Refactor runtime and expr framework ( #28166 )
...
#28165
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2023-12-18 12:04:42 +08:00
