admin/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2026-01-07 19:31:51 +08:00
Code Issues Packages Projects Releases Wiki Activity
18,778 Commits 137 Branches 188 Tags
Commit Graph

491 Commits

Author SHA1 Message Date
MrPresent-Han
9e2e7157e9
feat: support search_group_by for milvus(#25324) (#28983) 
related: #25324

Search GroupBy function, used to aggregate result entities based on a
specific scalar column.
several points to mention:

1. Temporarliy, the whole groupby is implemented separated from
iterative expr framework **for the first period**
2. In the long term, the groupBy operation will be incorporated into the
iterative expr framework:https://github.com/milvus-io/milvus/pull/28166
3. This pr includes some unrelated mocked interface regarding alterIndex
due to some unworth-to-mention reasons. All these un-associated content
will be removed before the final pr is merged. This version of pr is
only for review
4. All other related details were commented in the files comparison

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2024-01-05 15:50:47 +08:00
Jiquan Long
6f4791da0b
fix: panic in concurrent insert/query scenario (#29408) 
issue: https://github.com/milvus-io/milvus/issues/29405

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-12-26 15:10:48 +08:00
yah01
b8318fcd7d
enhance: improve the handling for segcore error (#29471) 
- fix lost exception details in segcore
- improve the logs of handling errors from segcore

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-12-26 14:06:46 +08:00
cqy123456
4c979538a4
enhance: update cagra index params in config and add params check (#29045) 
issue:https://github.com/milvus-io/milvus/issues/29230
this pr do two things about cagra index:
 a.milvus yaml config support gpu memory settings

 b.add cagra-params check

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
Co-authored-by: yusheng.ma <yusheng.ma@zilliz.com>
 2023-12-26 11:04:47 +08:00
yah01
aef483806d
enhance: improve the segcore logs (#29372) 
- remove the streaming logging
- refine existing logs

fix #29366

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-12-23 21:52:43 +08:00
chyezh
be87c18b44
fix: fixup data race at generate binlog index (#29370) 
issue: #29339

Signed-off-by: chyezh <ye.zhen@zilliz.com>
 2023-12-21 14:58:49 +08:00
Gao
9b52cb6417
enhance: improve reducing results when many segments are filtered (#29073) 
Do not fill the invalid ids for the empty results, it will incur useless
memory overhead and reduce overhead when nq and topk is large.

---------

Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2023-12-20 12:56:42 +08:00
MrPresent-Han
bfca0a7926
fix: refine skipIndex to resolve cyclic dependcy(#29132) (#29189) 
related: #29132

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-12-19 10:26:40 +08:00
zhagnlu
a602171d06
enhance: Refactor runtime and expr framework (#28166) 
#28165

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2023-12-18 12:04:42 +08:00
MrPresent-Han
464bc9e8f4
fix: fix reduce precision for search(#27325) (#29031) 
related: #27325

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-12-08 10:04:37 +08:00
congqixia
dcb662d9ed
enhance: Refine C.NewSegment response and handle exception (#28952) 
See also #28795

Orignal `C.NewSegment` may panic if some condition is not met, this pr
changes response struct to `CNewSegmentResult`, which contains
`C.CStatus` and may return catched exception

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-12-07 13:34:35 +08:00
cai.zhang
fb089cda8b
enhance: Load raw data while scalar index doesn't have raw data (#28888) 
issue: #28886

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2023-12-06 20:36:36 +08:00
Bingyi Sun
36f69ea031
feat: integrate storagev2 in building index of segcore (#28768) 
issue: https://github.com/milvus-io/milvus/issues/28655

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2023-12-05 16:48:54 +08:00
yihao.dai
f5856812a2
fix: Fix get binary vector from chunk cache (#28866) 
The way of getting binary vector size is wrong. This PR will fix it.

issue: https://github.com/milvus-io/milvus/issues/28865

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-12-01 14:40:32 +08:00
yah01
d69440524b
fix: bypass growing index if no index meta (#28791) 
we shouldn't panic if no index meta, just skip building it
fix #28022

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-11-30 14:10:27 +08:00
congqixia
1dc086496f
fix: schema->size() check logic with system field (#28802) 
Now segcore load system field info as well, the growing segment
assertion shall not pass with "+ 2" value
This will cause all growing segments load failure
Fix #28801
Related to #28478
See also #28524

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-29 22:40:28 +08:00
cqy123456
3b1b14dd78
fix: update binlog index memory uasge before loading segments (#28528) 
issue: #27678 
when interimIndex = true, memory predict should be update with the
memory usage of binlog index build process.

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2023-11-29 16:42:27 +08:00
yah01
02c5a649cf
enhance: store system fields in segcore (#28524) 
we need the system fields info for some usacase
fix: #28523

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-21 09:28:22 +08:00
yah01
f7d2ab6677
enhance: reduce 1x copy for variable length field while retrieving (#28345) 
- Reduce 1x copy for varchar/string/JSON/array types while retrieving
- Reduce 1x copy for int8/int16 while retrieving

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-15 18:08:20 +08:00
MrPresent-Han
836f300536
support skip-index based on chunk-metrics to accelerate expr filter(#27925) (#28297) 
related: #27925

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-11-15 11:20:19 +08:00
Xu Tong
8ec85f5f4c
Add template for VectorMemIndex (#28324) 
Signed-off-by: Writer-X <1256866856@qq.com>
 2023-11-11 13:20:22 +08:00
yah01
267c67dfee
enhance: reduce 1x copy while retrieving data from growing segment (#28323) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-10 15:44:22 +08:00
yah01
30847cad3e
Handle exception while loading (#28304) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-09 17:59:12 +08:00
yah01
863e26969a
Reduce 1x memory copy for retrieving data (#28106) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-06 17:48:19 +08:00
congqixia
501cba90ce
Use IVFFLAT_CC instead of IVFFLAT for sealed segment tmp index (#28164) 
See also #28152

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-03 23:50:15 +08:00
yah01
dc89730a50
Support collection-level mmap control (#26901) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-02 23:52:16 +08:00
Enwei Jiao
f8dd589755
Refactor collection's cgo call (#28055) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-11-02 13:02:13 +08:00
MrPresent-Han
854c0e8a16
fix precision for segcore reduce(#27325) (#28062) 
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-11-01 21:52:13 +08:00
Enwei Jiao
8ae9c947ae
Use OpenDAL to access object store (#25642) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-11-01 09:00:14 +08:00
cqy123456
4fbe3c9142
replace loaded binlog with binlog index for search performance (#27673) 
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2023-11-01 02:20:15 +08:00
yah01
1b9b8a31a8
Fix QueryNode panic while upgrading (#28034) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-10-31 15:50:13 +08:00
yihao.dai
ab6b0103a3
Get vector concurrently (#27838) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-10-30 15:44:12 +08:00
yah01
f212158d61
Fix delete records timestamp may be reordered (#27941) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-10-27 10:24:10 +08:00
Enwei Jiao
e98e56f75d
Fix SIGSEGV if run within gdb (#27736) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-10-18 02:16:11 +08:00
smellthemoon
5785756d31
Add some log to print size of search result (#27601) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-10-16 10:48:08 +08:00
zhagnlu
c5ea31316d
Fix glog thread-safe bug for standalone mode (#27703) 
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2023-10-14 12:10:05 +08:00
Enwei Jiao
b80a3e19d3
Add code for PanicInfo (#27364) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-09-27 12:01:28 +08:00
foxspy
5db4a0489e
dynamic index version control (#27335) 
Co-authored-by: longjiquan <jiquan.long@zilliz.com>
 2023-09-25 21:39:27 +08:00
foxspy
fa033e586a
disable growing index for flat (#27309) 
Signed-off-by: xianliang <xianliang.li@zilliz.com>
 2023-09-22 14:19:24 +08:00
foxspy
370b6fde58
milvus support multi index engine (#27178) 
Co-authored-by: longjiquan <jiquan.long@zilliz.com>
 2023-09-22 09:59:26 +08:00
yah01
93e2eb78c9
Delete only if primary keys exist (#25292) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-09-20 19:03:25 +08:00
cai.zhang
a362bb1457
Support array datatype (#26369) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-09-19 14:23:23 +08:00
yihao.dai
bb6711f28c
Add ChunkCache: support get vector from storage (#26142) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-09-15 10:21:20 +08:00
foxspy
18cbe26876
fix growing hasRawData (#27105) 
Signed-off-by: xianliang <xianliang.li@zilliz.com>
 2023-09-14 21:11:18 +08:00
Enwei Jiao
0afdfdb9af
Remove other Exceptions, keeps SegcoreError only (#27017) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-09-14 14:05:20 +08:00
cqy123456
0ff4ddc76c
remove VectorMemNMIndex (#27000) 
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2023-09-12 17:13:18 +08:00
yah01
3203ce1654
Reduce copy while retrieving primary keys (#26616) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-09-11 21:31:18 +08:00
Enwei Jiao
c3f15c6b95
Refactor duplicate error class into one place (#26985) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-09-11 20:43:17 +08:00
Xu Tong
9166011c4a
Add float16 vector (#25852) 
Signed-off-by: Writer-X <1256866856@qq.com>
 2023-09-08 10:03:16 +08:00
yah01
4a2377e669
Fix retrieving may return more result than limit (#26670) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-06 17:47:15 +08:00