admin/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2026-01-07 19:31:51 +08:00
Code Issues Packages Projects Releases Wiki Activity
5,482 Commits 137 Branches 188 Tags
Commit Graph

290 Commits

Author SHA1 Message Date
shengjun.li
5f830293f9
rm find faiss (#3269) 
Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
 2020-08-15 19:18:03 +08:00
shengjun.li
43c272fd13
fix annoy l2 (#3232) 
Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
 2020-08-12 19:11:13 +08:00
Xiaohai Xu
058cdf03bb
Add ivf index nprobe logging (#3070) 
* add nprobe logging

Signed-off-by: sahuang <xiaohai.xu@zilliz.com>

* add all cases

Signed-off-by: sahuang <xiaohai.xu@zilliz.com>
 2020-08-01 09:46:53 +08:00
shengjun.li
b1b1165c56
Opt faiss builder (#3067) 
Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
 2020-07-31 09:35:31 +08:00
shengjun.li
1c36e6d83d
optimizer parallel policy (#3014) 
* optimizer parallel policy

Signed-off-by: shengjun.li <shengjun.li@zilliz.com>

* modify parallel policy

Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
 2020-07-26 01:37:23 +08:00
cqy123456
91d3fe5cbc
Improve IVF search performance when NQ and nProbe are both large (#2984) 
* fix indexflat  search

Signed-off-by: cqy <yaya645@126.com>

* add parallel_policy_threshold

Signed-off-by: shengjun.li <shengjun.li@zilliz.com>

Co-authored-by: shengjun.li <shengjun.li@zilliz.com>
 2020-07-24 15:56:47 +08:00
shengjun.li
a88f5b6cbb
Fix index size (#2941) 
* add virtual IF UpdateIndexSize

Signed-off-by: shengjun.li <shengjun.li@zilliz.com>

* update ivf index size

Signed-off-by: sahuang <xiaohai.xu@zilliz.com>

* fix PQ logic

Signed-off-by: sahuang <xiaohai.xu@zilliz.com>

* fix index size of index hnsw, annoy and nsg

Signed-off-by: cmli <chengming.li@zilliz.com>

* add GetSize() interface 4 SPTAG

Signed-off-by: cmli <chengming.li@zilliz.com>

* fix binary ivf

Signed-off-by: shengjun.li <shengjun.li@zilliz.com>

Co-authored-by: sahuang <xiaohai.xu@zilliz.com>
Co-authored-by: cmli <chengming.li@zilliz.com>
 2020-07-21 14:26:08 +08:00
XuanYang-cn
ea58edc128
Add python functional tests about timeout (#2793) 
* Add python tests about param timeout

Signed-off-by: yangxuan <jumpthepig@gmail.com>

* add faiss configure args msg print

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* change assertion conditions

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

Co-authored-by: yangxuan <xuan.yang@zilliz.com>
 2020-07-15 14:32:40 +08:00
cqy123456
585c8ea018
fix rnsg ip (#2827) 
Signed-off-by: cqy <yaya645@126.com>
 2020-07-13 14:05:28 +08:00
op-hunter
4922c9532f
fix a bug of getting wrong nprobe limitation in knowhere on GPU version (#2788) 
* fix a bug of getting wrong nprobe limitation in knowhere on GPU version

Signed-off-by: cmli <chengming.li@zilliz.com>

* update changelog and fix build error

Signed-off-by: cmli <chengming.li@zilliz.com>

Co-authored-by: cmli <chengming.li@zilliz.com>
 2020-07-09 23:15:07 +08:00
shengjun.li
4038b0653e
fix too many data copies (#2784) 
Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
 2020-07-09 09:23:49 +08:00
groot
714b777d05
#2642 Create index failed caused by server crashed. (#2701) 
* #2642 Create index failed caused by server crashed.

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* changelog

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* changelog

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* typo

Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2020-07-02 09:37:15 +08:00
shengjun.li
0fb88f733a
fix too many copies (#2660) 
Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
 2020-06-24 09:46:48 +08:00
shengjun.li
cd04f3fc9f
Suit the range of HNSW parameters (#2638) 
Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
 2020-06-22 16:42:15 +08:00
shengjun.li
48e8cd36c2
Fix HNSW and RNSG index files size (#2618) 
Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
 2020-06-20 14:06:05 +08:00
Cai Yudong
6ac3d2bd08 fix Milvus docker image report illegal instruction (#2598) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2020-06-19 10:44:23 +08:00
shengjun.li
290ff2e439 Support IVF_PQ on GPU with using metric_type IP (#2593) 
* Support IVF_PQ on GPU with using metric_type IP

Signed-off-by: shengjun.li <shengjun.li@zilliz.com>

* remove useless code

Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
 2020-06-19 10:42:23 +08:00
Cai Yudong
8254e9ed67
fix performance (#2499) 
* optimize sq_get_distance_computer

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add sq_select_inverted_list_scanner_ref

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add sq_select_inverted_list_scanner_avx

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add sq_select_inverted_list_scanner_avx512

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* optimize Codec

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* optimize ScalarQuantizerCodec_avx.h

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* code format

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* optimize ScalarQuantizerCodec_avx512.h

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update changelog

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* clean code

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2020-06-06 07:41:57 +00:00
op-hunter
9de7f710e6
optimize the implemention of k-selection on faiss gpu version (#2467) 
* optimize the implemention of k-selection on faiss gpu version

Signed-off-by: cmli <chengming.li@zilliz.com>

* update change log, re-classified the pr from feature to improvement

Signed-off-by: cmli <chengming.li@zilliz.com>

* update change log

Signed-off-by: cmli <chengming.li@zilliz.com>

Co-authored-by: cmli <chengming.li@zilliz.com>
 2020-05-30 04:27:59 +00:00
op-hunter
cab9e5c2a6 fix bug of openblas library install fail (#2457) 
* fix bug of openblas library install fail

Signed-off-by: cmli <chengming.li@zilliz.com>

* fix openblas install cmakelist error

Signed-off-by: cmli <chengming.li@zilliz.com>

Co-authored-by: cmli <chengming.li@zilliz.com>
 2020-05-29 20:00:24 +08:00
shengjun.li
7c72dcf4d3
fix search by GPU (#2453) 
Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
 2020-05-28 15:39:16 +00:00
Cai Yudong
f4aeca2988
improve knowhere coverage (#2444) 
* increase nb for NSG code coverage

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* test more APIs in test_annoy

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* disable get_vector_by_id and search_by_id

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* improve code coverage

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* improve code coverage

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update unittest

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* install test_instructionset

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update changelog

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2020-05-28 10:22:14 +08:00
Cai Yudong
ce4dcf69ae
add debug APIs and update GPU temp memory size (#2420) 
* add debug APIs and update GPU temp memory size

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update thread name

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update DESIGN.md

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* retry ci

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2020-05-26 14:39:44 +08:00
shengjun.li
2982004e2b
midify Binary IVF in faiss 1.6.3 (#2416) 
Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
 2020-05-23 11:01:54 +00:00
Xiaohai Xu
6809150b1d
#2395 Fix large nq cudaMalloc (#2409) 
* fix large nq cuda malloc issue

Signed-off-by: sahuang <xiaohai.xu@zilliz.com>

* fix clang format

Signed-off-by: sahuang <xiaohai.xu@zilliz.com>

* Change log

Signed-off-by: sahuang <xiaohai.xu@zilliz.com>
 2020-05-23 08:23:05 +00:00
shengjun.li
34fdc00d19
fix MatchNlist (#2401) 
Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
 2020-05-22 06:20:26 +00:00
Cai Yudong
386e58ce0d
upgrade faiss 1.6.3 (#2400) 
* roll back to original faiss 1.6.0

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update to faiss_1.6.3

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* patch all change to faiss 1.6.3

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* faiss CPU version build pass

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* faiss GPU version build pass

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2020-05-22 01:27:16 +00:00
Cai Yudong
b9e8acb5b2
#2370 clean compile warning (#2371) 
* #2370 clean compile warning

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update changelog

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2020-05-18 02:35:14 +00:00
shengjun.li
bc74d16376
#2283 Suspend the building tasks when any query command arrives (#2285) 
* add builder suspend

Signed-off-by: shengjun.li <shengjun.li@zilliz.com>

* IndexIVF builder check wait

Signed-off-by: sahuang <xiaohai.xu@zilliz.com>

* Build suspend for all IVF

Signed-off-by: sahuang <xiaohai.xu@zilliz.com>

* HNSW suspend

Signed-off-by: sahuang <xiaohai.xu@zilliz.com>

* HNSW suspend

Signed-off-by: sahuang <xiaohai.xu@zilliz.com>

* HNSW suspend

Signed-off-by: sahuang <xiaohai.xu@zilliz.com>

* HNSW suspend

Signed-off-by: sahuang <xiaohai.xu@zilliz.com>

* HNSW suspend

Signed-off-by: sahuang <xiaohai.xu@zilliz.com>

* cpubuilder

Signed-off-by: wxyu <xy.wang@zilliz.com>

* add suspend check during annoy build index

Signed-off-by: cmli <chengming.li@zilliz.com>

* fix hnsw

Signed-off-by: shengjun.li <shengjun.li@zilliz.com>

* fix changelog

Signed-off-by: shengjun.li <shengjun.li@zilliz.com>

* fix clang format

Signed-off-by: shengjun.li <shengjun.li@zilliz.com>

* suspend nsg

Signed-off-by: shengjun.li <shengjun.li@zilliz.com>

Co-authored-by: sahuang <xiaohai.xu@zilliz.com>
Co-authored-by: wxyu <xy.wang@zilliz.com>
Co-authored-by: cmli <chengming.li@zilliz.com>
Co-authored-by: JinHai-CN <hai.jin@zilliz.com>
 2020-05-12 09:27:45 +08:00
op-hunter
a37f9b0764
Openblas macro (#2269) 
* save local work 4 add openblas macro to chose compile openblas lib or not

Signed-off-by: cmli <chengming.li@zilliz.com>

* compile openblas by auto detect

Signed-off-by: cmli <chengming.li@zilliz.com>

* Update build environment docker images

Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>

* update changelog

Signed-off-by: cmli <chengming.li@zilliz.com>

* Update CMakeLists.txt

Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>

* Update CMakeLists.txt

Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>

* Update CMakeLists.txt

Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>

* Update CMakeLists.txt

Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>

* Update CMakeLists.txt

Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>

* Update CMakeLists.txt

Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>

* Update CMakeLists.txt

Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>

* Update CMakeLists.txt

Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>

* Update CMakeLists.txt

Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>

* Update CMakeLists.txt

Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>

* Update CMakeLists.txt

Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>

* set OpenBLAS_SOURCE auto in build.sh

Signed-off-by: cmli <chengming.li@zilliz.com>

Co-authored-by: cmli <chengming.li@zilliz.com>
Co-authored-by: quicksilver <zhifeng.zhang@zilliz.com>
 2020-05-09 22:47:03 +08:00
Cai Yudong
cf6be092ab
code clean (#2265) 
* use macro STATUS_CHECK

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* comment S3 related config APIs

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix clang-format

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2020-05-09 01:51:04 +00:00
shengjun.li
4cea320943
k-means L2 (#2258) 
* k-means L2

Signed-off-by: shengjun.li <shengjun.li@zilliz.com>

* fix change log

Signed-off-by: shengjun.li <shengjun.li@zilliz.com>

Co-authored-by: Jin Hai <hai.jin@zilliz.com>
 2020-05-08 14:03:50 +00:00
Cai Yudong
f218217064
optimize similarity template (#2227) 
* optimize similarity template

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* code opt

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2020-05-06 08:53:16 +00:00
Cai Yudong
ed2ee8873b
fix SingleIndexTest.IVFSQHybrid unittest (#2215) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2020-05-02 10:24:35 +00:00
zxf2017
14b3e95960
add head file to fix two build warnings (#2195) 
* add head file to fix two build warnings

Signed-off-by: zxf <zhuxiaofeng07936@hellobike.com>

* fstream.h => fstream

Signed-off-by: zxf <zhuxiaofeng07936@hellobike.com>

Co-authored-by: zxf <zhuxiaofeng07936@hellobike.com>
 2020-04-30 06:10:44 +00:00
feisiyicl
e77fd12286
Feisiyicl 2190 memory usage (#2191) 
* rename MILVUS_ENABLE_PROFILING to ENABLE_CPU_PROFILING, and add ENABLE_MEM_PROFILING

Signed-off-by: feisiyicl <7764126@qq.com>

* optimize ReadOnlyArrayInvertedLists

Signed-off-by: feisiyicl <7764126@qq.com>

* update changelog

Signed-off-by: feisiyicl <7764126@qq.com>

* retry ci

Signed-off-by: feisiyicl <7764126@qq.com>

* fix build issue

Signed-off-by: feisiyicl <7764126@qq.com>
 2020-04-30 02:03:54 +00:00
feisiyicl
9b2368016c
#1946 fix load index file CPU2GPU fail during searching (#2182) 
Signed-off-by: feisiyicl <7764126@qq.com>
 2020-04-29 20:01:52 +08:00
shengjun.li
a76e705a92
Using elkan K-Means to improve IVF (#2180) 
Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
 2020-04-29 19:58:50 +08:00
feisiyicl
b3aa06fce1
add config simd_type (#2179) 
* add config simd_type

Signed-off-by: feisiyicl <7764126@qq.com>

* update changelog

Signed-off-by: feisiyicl <7764126@qq.com>
 2020-04-29 17:46:36 +08:00
op-hunter
1b25bacee2
build openblas by source (#2154) 
* build openblas by source

Signed-off-by: cmli <chengming.li@zilliz.com>

* remove openblas md5

Signed-off-by: cmli <chengming.li@zilliz.com>

Co-authored-by: cmli <chengming.li@zilliz.com>
Co-authored-by: Jin Hai <hai.jin@zilliz.com>
 2020-04-27 22:48:14 +08:00
Cai Yudong
1384484b7d
Caiyd 2039 avx sse (#2136) 
* #2039 clean redundant compile options

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* #2039 add distances_avx.h and distances_avx.cpp

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* #2039 add ScalarQuantizerDC_avx.cpp, ScalarQuantizerDC_avx.h and ScalarQuantizerCodec_avx.h

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* #2039 remove -mavx2 from default compile option

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update changelog

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* #2039 clean code

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* #2039 clean code

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2020-04-27 15:01:47 +08:00
Gracieeea
509b87f103
replace int with size_t (#2140) 
Signed-off-by: Gracieeea <sly_freesoul@icloud.com>
 2020-04-27 04:52:11 +00:00
Cai Yudong
150cca6bbf
Caiyd 1946 cpu2gpu getres (#2124) 
* #1946 use new log format in knowhere

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* #1946 add more debug info for GPU device initialize

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* retry CI

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* retry CI

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2020-04-26 18:29:21 +08:00
shana0325
53133aafac
train func add catch excetpion (#2096) 
* train func add catch excetpion

Signed-off-by: shana0325 <645842705@qq.com>

* Fix lint error

Signed-off-by: JinHai-CN <hai.jin@zilliz.com>

Co-authored-by: JinHai-CN <hai.jin@zilliz.com>
 2020-04-26 11:29:20 +08:00
chen lu
d22f3c2c5f
[skip ci] remove useless log (#2119) 
Signed-off-by: yiuluchen 1093633856@qq.com
 2020-04-26 10:19:31 +08:00
GuoRentong
fed865c7c9
Values of type 'const size_t' may not fit into the receiver type 'int' (#2109) 
Signed-off-by: GuoRentong <rt.guo@outlook.com>

Co-authored-by: guo rentong <grt@guodeMacBook-Pro.local>
 2020-04-25 18:35:04 +08:00
grtoverflow
6fe2cf1580
Update ConfAdapter.cpp (#2106) 
Signed-off-by: grtoverflow <grtoverflow@gmail.com>
 2020-04-25 16:12:58 +08:00
xge-zilliz
42b1983625
Delete useless boolean (#2107) 
Signed-off-by: xge-zilliz <xi.ge@zilliz.com>
 2020-04-25 15:49:43 +08:00
dd-He
c33df2ee20
Use emplace_back instead of push_back (#2103) 
Signed-off-by: dd-He <1843757857@qq.com>
 2020-04-25 15:37:23 +08:00
格雷大大
bf4cea1d1f
add log 4 annoy if index has been built when BuildAll interface invoked (#2071) 
* add log 4 annoy if index has been built when BuildAll interface invoked

Signed-off-by: grace <yan.gu@zilliz.com>

* fix lint error

Signed-off-by: grace <yan.gu@zilliz.com>
 2020-04-24 19:38:21 +08:00