36 Commits

Author SHA1 Message Date
cai.zhang
be77ceba84
enhance: Use proto for passing info in cgo (#33184)
issue: #33183

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-05-23 10:31:40 +08:00
congqixia
8cf2cf5c94
enhance: Add go-deadlock as unittest only dependency (#33063)
See also #33062

This PR:

- Add `lock.RWMutex` & `lock.Mutex` alias to switch implementation based
  on build flags
- When build flags has `test` in it, use `go-deadlock` to detect
  possible deadlocks
- Replace all `sync.RWMutex` & `sync.Mutex` in datacoord pkg

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-05-15 16:33:34 +08:00
Patrick Weizhi Xu
52ae47c850
enhance: gather materialized view search info once per request (#31996)
issue: #29892 

This PR:
1. Move the process of gathering materialized search info to when the
search plan is created, before it goes to each segment, to avoid
repeated work and access the plan node under multi-threaded
circumstances.
2. Enforce the supported MV type to `VARCHAR`
3. Add integration test

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
2024-04-11 15:21:19 +08:00
SimFG
b1a1cca10b
feat: add more operation detail info for better allocation (#30438)
issue: #30436

---------

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-03-28 06:33:11 +08:00
Patrick Weizhi Xu
982dd2834b
enhance: add materialized view search info (#30888)
issue: #29892 

This PR
1. Pass Materialized View (MV) search information obtained from the
expression parsing planning procedure to Knowhere. It only performs when
MV is enabled and the partition key is involved in the expression. The
search information includes:
1. Touched field_id and the count of related categories in the
expression. E.g., `color == red && color == blue` yields `field_id ->
2`.
2. Whether the expression only includes AND (&&) logical operator,
default `true`.
    3. Whether the expression has NOT (!) operator, default `false`.
4. Store if turning on MV on the proxy to eliminate reading from
paramtable for every search request.
5. Renames to MV.

## Rebuttals
1. Did not write in `ExtractInfoPlanNodeVisitor` since the new scalar
framework was introduced and this part might be removed in the future.
2. Currently only interested in `==` and `in` expression, `string` data
type, anything else is a bonus.
3. Leave handling expressions like `F == A || F == A` for future works
of the optimizer.

## Detailed MV Info

![image](https://github.com/milvus-io/milvus/assets/6563846/b27c08a0-9fd3-4474-8897-30a3d6d6b36f)

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
2024-03-21 11:19:07 +08:00
groot
c81909bfab
enhance: Support MinIO TLS connection (#31311)
issue: https://github.com/milvus-io/milvus/issues/30709
pr: #31292

Signed-off-by: yhmo <yihua.mo@zilliz.com>
Co-authored-by: Chen Rao <chenrao317328@163.com>
2024-03-21 11:15:20 +08:00
Bingyi Sun
bdc70dfc6a
feat: Add global mmap enable configuration (#31267)
https://github.com/milvus-io/milvus/issues/31279

Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-03-18 15:17:10 +08:00
jaime
4b0c3dd377
enhance: index meta use independent rather than global meta lock (#30869)
issue: https://github.com/milvus-io/milvus/issues/30837

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-03-04 16:56:59 +08:00
foxspy
e1e87d572b
fix: compatibility for diskann cache param (#30119)
patch search cache param from index configs when index meta could not
get the search cache size key
#issue: #30113

Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
2024-02-26 16:54:55 +08:00
Patrick Weizhi Xu
0907d76253
enhance: pass partition key scalar info if enabled when build vector index (#29931)
issue: #29892 

Pass optional scalar IVF offsets to Cardinal

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
2024-01-24 00:04:55 +08:00
smellthemoon
e52ce370b6
enhance:don't store logPath in meta to reduce memory (#28873)
don't store logPath in meta to reduce memory, when service get
segmentinfo, generate logpath from logid.
#28885

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-01-18 22:06:31 +08:00
Bingyi Sun
e1258b8cad
feat: integrate storagev2 into loading segment (#29336)
issue: #29335

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-01-12 18:10:51 +08:00
Bingyi Sun
ad866d2889
feat: integrate storagev2 into index build process (#28995)
issue: https://github.com/milvus-io/milvus/issues/28994

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
2023-12-13 17:24:38 +08:00
cai.zhang
31f442915b
fix: Fix bug for parsing expression that include quotes (#28416)
issue: #28365 
Fix bug for parsing error when a string enclosed in single quotes in an
expression contains multiple double quotes.
such as:
```
expr = "tag == '\"blue\"'"
```

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2023-11-23 17:18:32 +08:00
Enwei Jiao
7445d3711c
feat: trigger compaction to handle index version (#28442)
issue: https://github.com/milvus-io/milvus/issues/28441

---------

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-11-21 09:26:22 +08:00
cai.zhang
fd68837f84
Fix bug for setting index state when indexnode connecting failed (#28220)
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2023-11-07 23:34:26 +08:00
zhagnlu
6060dd7ea8
Add chunk manager request timeout (#27692)
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2023-10-23 20:08:08 +08:00
yah01
8394b3a1ec
Block creating new error from status reason (#27426)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-10-07 11:29:32 +08:00
Jiquan Long
370fdaf50d
Record engine version for segment index (#27384)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2023-09-28 18:03:28 +08:00
jaime
7f7c71ea7d
Decoupling client and server API in types interface (#27186)
Co-authored-by:: aoiasd <zhicheng.yue@zilliz.com>

Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-09-26 09:57:25 +08:00
foxspy
5db4a0489e
dynamic index version control (#27335)
Co-authored-by: longjiquan <jiquan.long@zilliz.com>
2023-09-25 21:39:27 +08:00
foxspy
370b6fde58
milvus support multi index engine (#27178)
Co-authored-by: longjiquan <jiquan.long@zilliz.com>
2023-09-22 09:59:26 +08:00
SimFG
26f06dd732
Format the code (#27275)
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-09-21 09:45:27 +08:00
cai.zhang
22d7fa4e1c
Use GetXXX instead of directly using dots to get value (#27045)
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-09-15 10:07:19 +08:00
Jiquan Long
6f2c1214ed
Fix cloudProvider not passed (#27065)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2023-09-14 19:21:20 +08:00
yah01
00c65fa0d7
Refine QueryNode errors (#27013)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-09-12 16:07:18 +08:00
zhagnlu
411f9ac823
Upgrade minio-go and add region and virtual host config for segcore chunk manager (#26194)
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2023-08-11 10:37:36 +08:00
aoiasd
4cb6351f87
update isDelete of index task meta when delete task from index builder (#25717)
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-07-21 17:15:00 +08:00
Enwei Jiao
66fdc71479
Refactor logs in DataCoord & DataNode (#25574)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-07-14 15:56:31 +08:00
congqixia
41af0a98fa
Use go-api/v2 for milvus-proto (#24770)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-06-09 01:28:37 +08:00
jaime
c9d0c157ec
Move some modules from internal to public package (#22572)
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-04-06 19:14:32 +08:00
congqixia
127867b873
Add ratedgroup for some info/warning log (#23095)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-03-31 15:22:23 +08:00
yah01
319f1773af
Fix segment not found if forward delete to empty segment (#22528)
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-03-03 14:13:49 +08:00
Enwei Jiao
697dedac7e
Use cockroachdb/errors to replace other error pkg (#22390)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-02-26 11:31:49 +08:00
cai.zhang
ae305a54bd
Set scheduler duration of index task can be configurable (#21944)
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-02-05 17:21:53 +08:00
cai.zhang
e5f408dceb
Merge IndexCoord and DataCoord (#21267)
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-01-04 19:37:36 +08:00