admin/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2026-01-07 19:31:51 +08:00
Code Issues Packages Projects Releases Wiki Activity
5,693 Commits 137 Branches 188 Tags
Commit Graph

1287 Commits

Author SHA1 Message Date
Ji Bin
b10cc1c473
Adaptation for cuda11 and legacy gpu/cpu (#17143) 
- compile gpu with cuda11
  - add support for sm35, sm50 for legacy GPUs
  - add sm86 for ampere GPU, like RTX 30xx, Tesla Ax
  - support cpu only support SSE42

Signed-off-by: Ji Bin <matrixji@live.com>
 2022-06-07 10:48:06 +08:00
Ji Bin
de8050d597
Support aliyun oss storage (#16002) 
- Support aliyun oss with oss-cpp-sdk
- Add parameter for using s3 with https scheme

Signed-off-by: Ji Bin <matrixji@live.com>
 2022-03-11 14:34:00 +08:00
groot
9193addd8c
Fix #12498 crashed when do specific actions (#14698) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2022-01-04 10:35:19 +08:00
sunwsh
ed01b3874b
Update MemTable.cpp (#14224) 
* Update MemTable.cpp

1. std::vector<segment::doc_id_t> ids_to_check  改成std::set
2. deleted_docs 内容写入 faiss::ConcurrentBitset中，用于查询

Signed-off-by: shu01.wang <shu01.wang@vipshop.com>

* Create MemTable.cpp

改 std::set 为 std::unordered_set

Signed-off-by: shu01.wang <shu01.wang@vipshop.com>

* Update MemTable.cpp

Modify coding specify

Signed-off-by: shu01.wang <shu01.wang@vipshop.com>
 2021-12-28 18:15:52 +08:00
Ji Bin
bb34765faf
Introducing s3 region parameter for aws s3 service. (#7419) 
- bugfix for #7325.

Signed-off-by: Ji Bin <matrixji@live.com>
 2021-09-10 18:15:22 +08:00
groot
181781344f
avoid potential random crash (#6710) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2021-07-24 11:33:20 +08:00
Eitan Zcharia
12f1d3c23c
Flat binary search over APU (#6676) 
Signed-off-by: ezeharia <ezeharia@gsitechnology.com>

solve issue Support search over APU board #5202
support Tanimoto/Hamming Flat binary search over APU device .
Search over APU can be done for FAISS_BIN_IDMAP engaine type only.
Dependencies : build with FPGA (- a ) and APU (-w ) . build must contain FPGA flag .
example : ./build.sh -a -w
*** search is not possible without APU device conneted to a server .**
Resolves: #5202
 2021-07-23 11:06:17 +08:00
groot
91af6ced97
mysql ssl (#6670) 
* mysql ssl

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* changelog

Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2021-07-21 15:24:02 +08:00
shengjun.li
52a8d24d1e
Fix FPGA compile (#6371) 
Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
 2021-07-08 16:47:13 +08:00
groot
a34cb9602d
improve getEntityById performance (#6310) 
* improve getEntityById performance

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* improve getEntityById performance

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* improve getEntityById performance

Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2021-07-06 17:57:02 +08:00
shengjun.li
e8eb476fea
Fix GetEntityById in the default partition (#6285) 
Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
 2021-07-03 17:18:30 +08:00
shengjun.li
d27f4d47dd
fix insert to or delete from the default partition (#6269) 
Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
 2021-07-03 14:10:32 +08:00
groot
e6f3f6212c
search result of the restful api contains -1 (#6127) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2021-06-28 11:45:25 +08:00
shengjun.li
f1847cc489
allow compile on cuda 11 (#6139) 
Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
 2021-06-28 10:50:14 +08:00
groot
3fc8123645
fix unittest random failure (#5877) 
* fix unittest random failure

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* fix ut

Signed-off-by: shengjun.li <shengjun.li@zilliz.com>

Co-authored-by: shengjun.li <shengjun.li@zilliz.com>
 2021-06-19 11:41:13 +08:00
groot
330cc61bed
refine log (#5776) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2021-06-15 19:21:04 +08:00
groot
96054dfafa
Fix the crash in BaseRequest (#5762) 
Fix #5398

Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2021-06-15 12:14:14 +08:00
groot
a2ff18c40d
Search with big nq and big topk (#5748) 
* fix #5747 #5115

Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2021-06-15 09:53:12 +08:00
shengjun.li
73190282bd
Fix apply deletes (#5752) 
(1) Even if the bloom filter is destroyed, it can delete entity by ID
(2) No duplicate deletion

Resolves: #5537

Signed-off-by: shengjun.li shengjun.li@zilliz.com
 2021-06-11 20:20:12 +08:00
shengjun.li
f6f57c203a
Even if the bloom filter is destroyed, it can get entity by ID (#5724) 
Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
 2021-06-10 18:55:32 +08:00
groot
84318f251b
clean code (#5707) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2021-06-10 13:54:03 +08:00
shengjun.li
2214397646
update the bloom filter by renaming a temporary file (#5683) 
MemTable::ApplyDeletes() may update the bloom filter file.
If it is written in place, a damaged file will be left in case of
power failure. To update the bloom filter by renaming a temporary
file can avoid it.

Resolves: #5537

Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
 2021-06-10 10:46:19 +08:00
groot
98fd6fa69a
modify error/warning message (#5695) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2021-06-09 19:10:36 +08:00
groot
5386d7ddb7
loose limitation of index_file_size (#5669) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2021-06-09 14:26:46 +08:00
shengjun.li
71c6eaeddb
index created by std::make_shared (#5628) 
Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
 2021-06-05 16:34:51 +08:00
shengjun.li
9e7e1dcba4
modify the version to 1.1.1 (#5615) 
* modify the version to 1.1.1

Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
 2021-06-05 10:06:55 +08:00
shengjun.li
2b82120c01
IVF_SQ8 and IVF_PQ cannot be built on multiple GPUs (#5584) 
Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
 2021-06-04 11:24:34 +08:00
groot
4193e18bb5
loose limitation of index_file_size (#5568) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2021-06-03 18:50:33 +08:00
Ji Bin
c753c7b0c3
storage: enableing compile with aws sdk by default. (#5485) 
* enabling compile with AWS by default
* add s3 related configure item's description in servier_config

Signed-off-by: Ji Bin <matrixji@live.com>
 2021-05-31 16:07:34 +08:00
Ji Bin
930600371d
storage: s3_address support domain/hostname. (#5413) 
In most cases, it is more appropriate to use the hostname or domain name
as the address of the S3 server.

Signed-off-by: Ji Bin <matrixji@live.com>
 2021-05-27 16:32:32 +08:00
Ji Bin
e213e5b321
Bugfix: increase offset after read in s3 IO reader. (#5393) 
Signed-off-by: Ji Bin <matrixji@live.com>
 2021-05-25 15:07:46 +08:00
shengjun.li
d1413e47c6
Partition existence check (#5326) 
(1) Exception should be raised if insert or delete entity on the none-existed partition
(2) Creating duplicate collections may result in insert or delete operation loss

Resolves: #5164

Signed-off-by: shengjun1985 shengjun.li@zilliz.com
 2021-05-20 19:18:03 +08:00
Ji Bin
b9e9bb22d0
Bugfix: fix local directory creating while using s3 storage (#5306) 
- remove unneeded CachePut for bloom filter in s3 storage
- using create_directories instead of create_directory
Resolves: #5287
Signed-off-by: Ji Bin <matrixji@live.com>
 2021-05-20 10:23:51 +08:00
shengjun.li
28cabb2a4e
Add a timestamp to the blacklist and cache it (#5297) 
(1) Add a timestamp to the blacklist and cache it.
(2) If timestamp mismatches, it will load DeleteDocs to regenerate it.
(3) Only SearchTask will load blacklist.
(4) Fix BloomFilter cache cleaning.

Resolves: #4897, #4625

Signed-off-by: shengjun.li shengjun.li@zilliz.com
 2021-05-19 19:53:11 +08:00
shengjun.li
53cacf60b2
Dirty blacklist is left in the cache (#5272) 
Multi threads accessing will leave dirty blacklist in the cache.
To fix it, we let knowhere::index not hold the blacklist.
And each query will regenerate it by DeleteDoc.
Later, we will add blacklist to the cache to improve performance.

Resolves: #4897

Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
 2021-05-18 17:49:54 +08:00
cqy123456
dce238e939
Dirty BloomFilter is left in the cache (#5252) 
Multi threads accessing will leave dirty BloomFilter in the cache.
Now, reading threads will not overwrite the cache. And it should be solved.
Issue 4897 involves BloomFilter and Blacklist.
This commit try to fix BloomFilter firstly.

Resolves: #4897

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2021-05-18 11:02:10 +08:00
shengjun.li
623ae55ff5
Improve IVF query on GPU when no entity deleted (#5204) 
Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
 2021-05-14 11:29:57 +08:00
shengjun.li
2e4aaff531
Remove daemonize code (#5181) 
Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
 2021-05-13 09:19:06 +08:00
ophunter233
0ec2a6d33a
Enable gpu cache (#5161) 
* enable gpu cache

Signed-off-by: cmli <chengming.li@zilliz.com>
 2021-05-12 09:51:28 +08:00
yukun
fe55f5353f
[skip ci]Fix webserver readme (#5099) 
Signed-off-by: fishpenguin <kun.yu@zilliz.com>
 2021-04-30 11:17:13 +08:00
shengjun.li
c19290b1ba
fix crash when creating a GPU IVF index (#5097) 
Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
 2021-04-30 11:01:18 +08:00
groot
6a9990168a
Fix #5063 (#5071) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2021-04-28 10:00:48 +08:00
groot
5667d00344
Index type returned by get_collection_stats() is incorrect (#5052) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2021-04-26 14:17:57 +08:00
groot
3d3799967c
Failed to set timezone like "UTC + 5:30" (#5042) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2021-04-26 09:24:21 +08:00
ophunter233
5e5067f1ae
add validation of parameter nbits for IndexIVFPQ on Gpu version (#5011) 
* add validate of parameter nbits for IndexIVFPQ on Gpu version

Signed-off-by: cmli <chengming.li@zilliz.com>
 2021-04-23 17:31:49 +08:00
groot
6fb9a82974
Fix #5008 (#5009) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2021-04-23 17:19:50 +08:00
shengjun.li
3451fa642a
fix nlist adapter (#5000) 
Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
 2021-04-23 11:33:01 +08:00
ophunter233
0ede2fcf1c
fix the wrong size of new operate (#4999) 
* fix the wrong size of new operate

Signed-off-by: cmli <chengming.li@zilliz.com>
 2021-04-22 20:21:19 +08:00
shengjun.li
26d84621aa
improve parallel (#4993) 
Signed-off-by: shengjun.li <shengjun.li@zilliz.com>
 2021-04-22 17:41:31 +08:00
groot
afb53bbb22
set version (#4967) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2021-04-20 17:32:45 +08:00