243 Commits

Author SHA1 Message Date
BossZou
167743c993
read_only node detect delete vector operation using mishards #2368 (#2473)
* Add new request ReLoadSegments

Signed-off-by: yhz <413554850@qq.com>

* Finish load segments functionality

Signed-off-by: yhz <413554850@qq.com>

* Add api in grpc

Signed-off-by: yhz <413554850@qq.com>

* update Reloadsegments

Signed-off-by: yhz <413554850@qq.com>

* .

Signed-off-by: yhz <413554850@qq.com>

* create new blacklist if not exists

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* update api names

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* Finish mishard for support api reloadsegments

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* update changlog

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* Add more details when failed in search task

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* Fix compile issue

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* update mishards requirements

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* Code format

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* modify docker images in mysql all_in_one

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* update shards code

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* Move updatedeldocs function to dbimpl

Signed-off-by: yhz <413554850@qq.com>

* Move reload segment del docs function to dbimpl

Signed-off-by: yhz <413554850@qq.com>

* [skip ci] correct shards requirements

Signed-off-by: yhz <413554850@qq.com>
2020-06-02 14:34:19 +08:00
groot
dcb60f3021
performance issues (#2438)
* performance issues

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* fix bugs

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* preload collection issue

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* uncomment boring log

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* reduce unittest time

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* reduce metric test time cost

Signed-off-by: yhmo <yihua.mo@zilliz.com>
2020-05-28 00:19:08 +08:00
groot
2d9b358e16
optimize merge process (#2419)
* optimize merge process

Signed-off-by: groot <yihua.mo@zilliz.com>

* typo

Signed-off-by: groot <yihua.mo@zilliz.com>

* refine code

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* drop collecion issue

Signed-off-by: yhmo <yihua.mo@zilliz.com>
2020-05-25 14:19:25 +08:00
Cai Yudong
344ae823b2
#2353 remove log_config (#2406)
* remove log_config from code and script

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update changelog

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update script

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2020-05-22 10:57:16 +00:00
Cai Yudong
b532a69c8b
clean compile warning (#2380)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2020-05-19 17:58:23 +00:00
Cai Yudong
b9e8acb5b2
#2370 clean compile warning (#2371)
* #2370 clean compile warning

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update changelog

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2020-05-18 02:35:14 +00:00
Cai Yudong
eba33cc46d
Caiyd 090 rename config (#2340)
* rename config items

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update config unittest

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix clang-format

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update config unittest

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update log rotate default value

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update parameter usage in LogUtil.cpp (#3)

Signed-off-by: wxyu <xy.wang@zilliz.com>

* retry ci

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix ut

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

Co-authored-by: JinHai-CN <hai.jin@zilliz.com>
Co-authored-by: Wang XiangYu <scsven@qq.com>
2020-05-14 17:57:18 +00:00
BossZou
860cc97a06
[skip ci] Update http document for v0.9.0 (#2347) (#2348)
Signed-off-by: yhz <413554850@qq.com>
2020-05-14 23:42:01 +08:00
BossZou
3c7c9ad645
Change url for behavior 'get_entities_by_id' (#2330) (#2336)
Signed-off-by: yhz <413554850@qq.com>
2020-05-14 21:01:20 +08:00
BossZou
1b7aeaffdf
Fix server start failed when mode is 'read_only' (#2294)
* fix deploy bug

Signed-off-by: yhz <413554850@qq.com>

* fix unittest fail

Signed-off-by: yhz <413554850@qq.com>

* remove unused yml

Signed-off-by: yhz <413554850@qq.com>

* change to readonly

Signed-off-by: yhz <413554850@qq.com>
2020-05-12 09:28:10 +08:00
groot
fb49e580c5
rewrite HasPartition (#2289)
* rewrite HasPartition

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* miss file

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* use layered merge

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* changelog

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* sdk HasPartition

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* rename sdk api

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* fix ut

Signed-off-by: groot <yihua.mo@zilliz.com>

Co-authored-by: JinHai-CN <hai.jin@zilliz.com>
2020-05-12 09:27:58 +08:00
groot
f32cba6111
Merge strategy (#2277)
* merge manager

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* fix build error

Signed-off-by: groot <yihua.mo@zilliz.com>

* layered merge strategy

Signed-off-by: groot <yihua.mo@zilliz.com>

* fix ut

Signed-off-by: groot <yihua.mo@zilliz.com>

* use simple

Signed-off-by: groot <yihua.mo@zilliz.com>

* fix parallel multi collections query crashissue

Signed-off-by: groot <yihua.mo@zilliz.com>
2020-05-11 01:38:50 +00:00
BossZou
8d732656a7
Fix show partitions failed in http module (fix #2228) (#2234)
* Fix show partitions failed in http module (fix #2228)

Signed-off-by: yhz <413554850@qq.com>

* catch json string convert error

Signed-off-by: yhz <413554850@qq.com>
2020-05-10 12:49:26 +00:00
groot
aeb9edf5c6
#2261 (#2270)
* has collection bug

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* #2261

Signed-off-by: groot <yihua.mo@zilliz.com>

* fix typo

Signed-off-by: groot <yihua.mo@zilliz.com>
2020-05-09 18:32:20 +08:00
Cai Yudong
cf6be092ab
code clean (#2265)
* use macro STATUS_CHECK

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* comment S3 related config APIs

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix clang-format

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2020-05-09 01:51:04 +00:00
groot
cf68c9918e
#1925 (#2257)
* write error

Signed-off-by: groot <yihua.mo@zilliz.com>

* out of storage

Signed-off-by: groot <yihua.mo@zilliz.com>

* clang format

Signed-off-by: groot <yihua.mo@zilliz.com>

* fix ut

Signed-off-by: groot <yihua.mo@zilliz.com>

* fix #1955

Signed-off-by: groot <yihua.mo@zilliz.com>

* refine code

Signed-off-by: groot <yihua.mo@zilliz.com>

* compact threashold

Signed-off-by: groot <yihua.mo@zilliz.com>

* changelog

Signed-off-by: groot <yihua.mo@zilliz.com>

* search by id for hnsw/pq/annoy

Signed-off-by: groot <yihua.mo@zilliz.com>

* fix python test

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* file cleanup timeout config

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* typo

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* changelog

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* #1925\

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* typo

Signed-off-by: yhmo <yihua.mo@zilliz.com>
2020-05-08 11:40:41 +00:00
BossZou
1811254de6
Obtain running rpc requests information (#2240) (#2241)
* add cmd to get requests

Signed-off-by: yhz <413554850@qq.com>

* forward class declaration

Signed-off-by: yhz <413554850@qq.com>

* Obtain running rpc requests information (fix #2240)

Signed-off-by: yhz <413554850@qq.com>

* Fix unittest compile failed issue

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* Log error with msg

Signed-off-by: Yhz <yinghao.zou@zilliz.com>
2020-05-08 11:17:48 +00:00
Wang XiangYu
104cda1a76
print warning on stdout (#2243)
* print warning on stdout

Signed-off-by: wxyu <xy.wang@zilliz.com>

* update

Signed-off-by: wxyu <xy.wang@zilliz.com>

* fix compile failed

Signed-off-by: wxyu <xy.wang@zilliz.com>
2020-05-08 08:57:08 +00:00
groot
da55622042
file cleanup timeout config (#2250)
* write error

Signed-off-by: groot <yihua.mo@zilliz.com>

* out of storage

Signed-off-by: groot <yihua.mo@zilliz.com>

* clang format

Signed-off-by: groot <yihua.mo@zilliz.com>

* fix ut

Signed-off-by: groot <yihua.mo@zilliz.com>

* fix #1955

Signed-off-by: groot <yihua.mo@zilliz.com>

* refine code

Signed-off-by: groot <yihua.mo@zilliz.com>

* compact threashold

Signed-off-by: groot <yihua.mo@zilliz.com>

* changelog

Signed-off-by: groot <yihua.mo@zilliz.com>

* search by id for hnsw/pq/annoy

Signed-off-by: groot <yihua.mo@zilliz.com>

* fix python test

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* file cleanup timeout config

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* typo

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* changelog

Signed-off-by: yhmo <yihua.mo@zilliz.com>
2020-05-08 06:24:29 +00:00
groot
ac8e9ff020
write error (#2184)
* write error

Signed-off-by: groot <yihua.mo@zilliz.com>

* out of storage

Signed-off-by: groot <yihua.mo@zilliz.com>

* clang format

Signed-off-by: groot <yihua.mo@zilliz.com>

* fix ut

Signed-off-by: groot <yihua.mo@zilliz.com>

* fix #1955

Signed-off-by: groot <yihua.mo@zilliz.com>

* refine code

Signed-off-by: groot <yihua.mo@zilliz.com>

* compact threashold

Signed-off-by: groot <yihua.mo@zilliz.com>

* changelog

Signed-off-by: groot <yihua.mo@zilliz.com>

* search by id for hnsw/pq/annoy

Signed-off-by: groot <yihua.mo@zilliz.com>

* fix python test

Signed-off-by: yhmo <yihua.mo@zilliz.com>
2020-05-06 16:52:37 +08:00
BossZou
82ab21aec3
Http update endpoints (#2187)
* replace table with collection in http module

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* add search_by_ids in http module

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* add API

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* add new API and unittest

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* Update endpoint in http module(#2186); change vector id to string format(2185)

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* allow filter when show partition

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* move constant variables defination to .cpp

Signed-off-by: Yhz <yinghao.zou@zilliz.com>
2020-05-06 14:07:35 +08:00
Wang XiangYu
ff48427911
Log file rotating (#2207)
* log file rotating

Signed-off-by: wxyu <xy.wang@zilliz.com>

* update changelog

Signed-off-by: wxyu <xy.wang@zilliz.com>

* fix compile failed

Signed-off-by: wxyu <xy.wang@zilliz.com>
2020-04-30 23:31:14 +08:00
BossZou
b458034ad5
Fix Fail to access WAL storage path (#2196) (#2198)
Signed-off-by: Yhz <yinghao.zou@zilliz.com>
2020-04-30 09:47:12 +00:00
Wang XiangYu
be2feb1531
Log with request id (#2192)
* add function to get request id

Signed-off-by: wxyu <xy.wang@zilliz.com>

* update log in GrpcRequestHandler

Signed-off-by: wxyu <xy.wang@zilliz.com>

* update request id format

Signed-off-by: wxyu <xy.wang@zilliz.com>
2020-04-30 13:39:55 +08:00
feisiyicl
e77fd12286
Feisiyicl 2190 memory usage (#2191)
* rename MILVUS_ENABLE_PROFILING to ENABLE_CPU_PROFILING, and add ENABLE_MEM_PROFILING

Signed-off-by: feisiyicl <7764126@qq.com>

* optimize ReadOnlyArrayInvertedLists

Signed-off-by: feisiyicl <7764126@qq.com>

* update changelog

Signed-off-by: feisiyicl <7764126@qq.com>

* retry ci

Signed-off-by: feisiyicl <7764126@qq.com>

* fix build issue

Signed-off-by: feisiyicl <7764126@qq.com>
2020-04-30 02:03:54 +00:00
BossZou
2c889636ab
Check Storage permission (#2174)
* add class storage checker

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* code format

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* code format

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* add cpu instruction sets check (#2055)

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* add more details for storage

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* Check storage path permission (fix #2173)

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* Fix logs path access fail issue

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

Co-authored-by: Jin Hai <hai.jin@zilliz.com>
2020-04-29 17:23:32 +00:00
feisiyicl
b3aa06fce1
add config simd_type (#2179)
* add config simd_type

Signed-off-by: feisiyicl <7764126@qq.com>

* update changelog

Signed-off-by: feisiyicl <7764126@qq.com>
2020-04-29 17:46:36 +08:00
BossZou
0a2b77c363
Upgrade http thirdparty oatpp to v1.0.0 (fix #2153) (#2157)
* Upgrade thirdparty oatpp to v1.0.0 (fix #2153)

Signed-off-by: yhz <413554850@qq.com>

* adjust header order

Signed-off-by: Yhz <yinghao.zou@zilliz.com>
2020-04-28 15:57:51 +00:00
groot
ebb01aa9b2
Fix failed to open file (#2138)
* file reference

Signed-off-by: groot <yihua.mo@zilliz.com>

* print info

Signed-off-by: groot <yihua.mo@zilliz.com>

* avoid metric crash

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* refine code

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* apply delete bug

Signed-off-by: groot <yihua.mo@zilliz.com>

* has partition check

Signed-off-by: groot <yihua.mo@zilliz.com>

* duplicate id search

Signed-off-by: groot <yihua.mo@zilliz.com>

* changelog

Signed-off-by: groot <yihua.mo@zilliz.com>
2020-04-28 21:10:01 +08:00
Wang XiangYu
3be8aad1af
Merge config file (#2168)
* Using el::Configurations Class init easylog

Signed-off-by: wxyu <xy.wang@zilliz.com>

* add logs config constant

Signed-off-by: wxyu <xy.wang@zilliz.com>

* add config check function

Signed-off-by: wxyu <xy.wang@zilliz.com>

* add config get function

Signed-off-by: wxyu <xy.wang@zilliz.com>

* logs config set function

Signed-off-by: wxyu <xy.wang@zilliz.com>

* update

Signed-off-by: wxyu <xy.wang@zilliz.com>

* update InitLog function

Signed-off-by: wxyu <xy.wang@zilliz.com>

* fix clang-format

Signed-off-by: wxyu <xy.wang@zilliz.com>

* update server_config.template

Signed-off-by: wxyu <xy.wang@zilliz.com>

* update changelog

Signed-off-by: wxyu <xy.wang@zilliz.com>
2020-04-28 20:21:21 +08:00
BossZou
4ab92f3032
Add config to switch off web server (#2137)
* Add a config parameter to switch off http server (fix #2057)

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* modify web enable config name

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* fix cmake bug & output exanption msg

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* rename web enbale config as web_enable

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* optimize code

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* log out errno

Signed-off-by: yhz <413554850@qq.com>
2020-04-28 17:04:04 +08:00
Wang XiangYu
b65f5072f3
fix server start failed if wal directory exist (#2144)
* fix server start failed if wal directory exist

Signed-off-by: wxyu <xy.wang@zilliz.com>

* add wal_enable check

Signed-off-by: wxyu <xy.wang@zilliz.com>

* fix compile error

Signed-off-by: wxyu <xy.wang@zilliz.com>
2020-04-27 20:05:04 +08:00
BossZou
bf42dbd5b6
Gpu check (#2129)
* add GpuCheck class

Signed-off-by: yhz <413554850@qq.com>

* Check gpu environment

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* add server checker unittest

Signed-off-by: Yhz <yinghao.zou@zilliz.com>
2020-04-27 16:56:12 +08:00
JinHai-CN
7b1b75a90d Fix lint error
Signed-off-by: JinHai-CN <hai.jin@zilliz.com>
2020-04-27 10:26:05 +08:00
JinHai-CN
adc6dc38bf Fix compiling error
Signed-off-by: JinHai-CN <hai.jin@zilliz.com>
2020-04-27 10:25:07 +08:00
dongwei
2c6fcacdbd delete unused file
Signed-off-by: dongwei <d279617552@gmail.com>
2020-04-26 23:21:06 +08:00
Wang XiangYu
12b3bf0693
Add instance lock (#2060)
* Add instance lock

Signed-off-by: wxyu <xy.wang@zilliz.com>

* update message

Signed-off-by: wxyu <xy.wang@zilliz.com>

* update unittest CMakeLists.txt

Signed-off-by: wxyu <xy.wang@zilliz.com>

* update

Signed-off-by: wxyu <xy.wang@zilliz.com>

* update

Signed-off-by: wxyu <xy.wang@zilliz.com>

* update

Signed-off-by: wxyu <xy.wang@zilliz.com>

* update

Signed-off-by: wxyu <xy.wang@zilliz.com>

* fix clang-format

Signed-off-by: wxyu <xy.wang@zilliz.com>

* update

Signed-off-by: wxyu <xy.wang@zilliz.com>

Co-authored-by: Jin Hai <hai.jin@zilliz.com>
2020-04-26 08:31:18 +00:00
BossZou
8edb7b6a09
add cpu instruction sets check (#2055)
Signed-off-by: Yhz <yinghao.zou@zilliz.com>
2020-04-26 00:59:25 +08:00
groot
777c36a0ac
#1997 index file missed after compact (#2108)
* #1997 index file missed after compact

Signed-off-by: groot <yihua.mo@zilliz.com>

* changelog

Signed-off-by: groot <yihua.mo@zilliz.com>

* typo

Signed-off-by: groot <yihua.mo@zilliz.com>
2020-04-25 22:06:26 +08:00
JinHai-CN
9db19a9a20 Merge remote-tracking branch 'main/master' 2020-04-25 18:58:59 +08:00
chengpu
6d2c50ac21
Use std::copy to replace forloop assignment (#2087)
Signed-off-by: chengpu <chengpu@live.com>
2020-04-25 15:28:31 +08:00
Cross
7deef8029f
Update GrpcRequestHandler.cpp (#2085)
Signed-off-by: Cross <xinyuanm9@gmail.com>
2020-04-25 14:53:49 +08:00
groot
82f4c289eb
New api HasPartition, SearchByID, GetVectorsByID (#2056)
* modify proto

Signed-off-by: groot <yihua.mo@zilliz.com>

* modify c sdk

Signed-off-by: groot <yihua.mo@zilliz.com>

* rewrite search by id

Signed-off-by: groot <yihua.mo@zilliz.com>

* change cases

Signed-off-by: zw <zw@zilliz.com>

* fix cases

Signed-off-by: zw <zw@zilliz.com>

* fix wal case

Signed-off-by: zw <zw@zilliz.com>

* fix search by id

Signed-off-by: groot <yihua.mo@zilliz.com>

* modify sdk

Signed-off-by: groot <yihua.mo@zilliz.com>

* refine code

Signed-off-by: groot <yihua.mo@zilliz.com>

* refine code

Signed-off-by: groot <yihua.mo@zilliz.com>

* changelog

Signed-off-by: groot <yihua.mo@zilliz.com>

* add search_by_ids case

Signed-off-by: zw <zw@zilliz.com>

* disable test_search)by_ids

Signed-off-by: zw <zw@zilliz.com>

Co-authored-by: zw <zw@zilliz.com>
Co-authored-by: del-zhenwu <56623710+del-zhenwu@users.noreply.github.com>
2020-04-25 14:49:32 +08:00
yukun
67d8a9b54c
Add http interface for hybrid search (#2079)
* Add http interface for hybrid search

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Add unittest for http hybrid interface

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* clang format

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix codacy quality

Signed-off-by: fishpenguin <kun.yu@zilliz.com>
2020-04-25 13:19:17 +08:00
JinHai-CN
fb85d8b3f0 Fix lint error
Signed-off-by: JinHai-CN <hai.jin@zilliz.com>
2020-04-25 11:21:58 +08:00
PJZero
fedbb697e3
Update GrpcRequestHandler.cpp (#2069)
Signed-off-by: pengjian05 <pengjian05@baidu.com>

Co-authored-by: pengjian05 <pengjian05@baidu.com>
2020-04-25 11:19:30 +08:00
Yhz
87778052f4 move http parser to http utils
Signed-off-by: Yhz <yinghao.zou@zilliz.com>
2020-04-23 23:16:04 +08:00
JinHai-CN
a41949f075 Merge remote-tracking branch 'main/master' into AllenYu1987-patch-1 2020-04-23 14:08:28 +08:00
Jin Hai
3267ed406f
Merge pull request #2032 from ggaaooppeenngg/code-clean
Delete CommandDto class and remove file CmdDto.hpp
2020-04-23 14:01:48 +08:00
Allen Zhang
f741be0a79 Move http query parser
Signed-off-by: Allen Zhang <allen.zhangyilun@gmail.com>
2020-04-23 12:14:11 +08:00