admin/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2026-01-07 19:31:51 +08:00
Code Issues Packages Projects Releases Wiki Activity
20,010 Commits 137 Branches 188 Tags
Commit Graph

1236 Commits

Author SHA1 Message Date
chyezh
f53ab54c5d
enhance: async cgo utility (#33133) 
issue: #30926, #33132

- implement future-based cgo utility.

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-06-09 22:55:53 +08:00
zhagnlu
6ce9df913f
fix: clean vector memory (#33692) 
#33533

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-06-07 10:37:54 +08:00
cai.zhang
27cc9f2630
enhance: Support analyze data (#33651) 
issue: #30633

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Co-authored-by: chasingegg <chao.gao@zilliz.com>
 2024-06-06 17:37:51 +08:00
wei liu
b69740c8f3
enhance: Remove unnecessary log info during load segment (#33663) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-06-06 14:13:50 +08:00
cqy123456
703fc73f71
enhance: disk index support binary vector (#33631) 
issue:https://github.com/milvus-io/milvus/issues/22837
related https://github.com/milvus-io/milvus/pull/33575

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2024-06-05 19:37:57 +08:00
zhagnlu
8ad26093ba
fix: fix load failure (#33599) 
issue: #33533

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-06-05 19:19:51 +08:00
Gao
545d4725fb
fix: correct get vector data size for bf16/fp16/binary vector (#33377) 
related #22837

Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2024-06-05 14:31:57 +08:00
congqixia
597f4c5e03
enhance: Make hasMoreResult accurate when hit number larger than limit (#33609) 
See also milvus-io/milvus-sdk-go#756

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-05 11:51:51 +08:00
zhagnlu
c6f8a73bb2
enhance: optimize some cache to reduce memory usage (#33534) 
#33533

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-06-04 14:09:47 +08:00
congqixia
2b285e5573
fix: Wrap init segcore tracing with golang timeout (#33494) 
See also #33483

Wrap `C.InitTrace` & `C.SetTrace` with timeout preventing otlp
initializtion hangs forever when endpoint is not set correctly

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-03 19:25:51 +08:00
Buqian Zheng
4171414222
enhance: update knowhere version (#33490) 
issue: https://github.com/milvus-io/milvus/issues/33489
update knowhere version to latest. remove usage of `seed_ef` as it be
replaced by existing `ef`.

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
 2024-05-31 10:17:50 +08:00
Jiquan Long
0c5d8660aa
feat: support inverted index for array (#33452) 
issue: https://github.com/milvus-io/milvus/issues/27704

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2024-05-31 09:47:47 +08:00
Chun Han
416a2cf507
fix: query iterator lack results(#33137) (#33422) 
related: #33137 
adding has_more_result_tag for various level's reduce to rectify
reduce_stop_for_best

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2024-05-30 17:51:44 +08:00
zhagnlu
589d4dfd82
enhance: optimize bitmap index (#33358) 
#32900

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-05-30 13:09:43 +08:00
Alexander Guzhva
9f9dff33f1
fix: make milvus compilable using gcc-13 (#30149) 
add a missing header

Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>
 2024-05-27 14:52:53 +08:00
yihao.dai
5cf4161394
fix: Fix exception info is missing (#33393) 
Replace based std::exception to prevent "object slicing"

issue: https://github.com/milvus-io/milvus/issues/33392

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-05-27 14:33:41 +08:00
Buqian Zheng
c5918ffbdb
enhance: mark sparse inverted index as mmap-able (#33281) 
issue: #29419

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
 2024-05-23 14:11:42 +08:00
cai.zhang
32d3e22d7d
fix: Throw an exception after all the threads in thread pool finished (#32810) 
issue: #32487

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-05-23 11:47:40 +08:00
cai.zhang
be77ceba84
enhance: Use proto for passing info in cgo (#33184) 
issue: #33183

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-05-23 10:31:40 +08:00
congqixia
3c4df81261
enhance: Assert insert data length not overflow int (#33248) 
When InsertData is too large for cpp proto unmarshalling, the error
message is confusing since the length is overflowed

This PR adds assertion for insert data length.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-05-22 15:11:39 +08:00
Alexander Guzhva
648d5661ca
enhance: Upgrade bitset for ARM SVE (#32718) 
issue: #32826 
improve ARM SVE performance for `internal/core/src/bitset`

Baseline timings for gcc 11.4 + Graviton 3 + manually enabled SVE:
https://gist.github.com/alexanderguzhva/a974b50134c8bb9255fb15f144e5ac83

Candidate timings for gcc 11.4 + Graviton 3 + manually enabled SVE:
https://gist.github.com/alexanderguzhva/19fc88f4ad3757e05e0f7feaf563b3d3

Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>
 2024-05-22 11:37:40 +08:00
zhagnlu
d669fbcf46
enhance: support bitmap index for scalar type (#32902) 
#32900

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-05-19 21:49:38 +08:00
Buqian Zheng
7c60d725cc
fix: validate sparse vector in search request (#32856) 
issue: #32368

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
 2024-05-15 15:39:33 +08:00
foxspy
f6777267e3
enhance: add score compute consistency config for knowhere (#32997) 
issue: https://github.com/milvus-io/milvus/issues/32583
related: #32584

Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
 2024-05-13 14:21:31 +08:00
aoiasd
54a51b1236
enhance: Support dynamic config for opentelemetry trace (#32169) 
relate: https://github.com/milvus-io/milvus/issues/31940

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2024-05-09 17:43:30 +08:00
Jiquan Long
9837ad6a8d
enhance: remove deprecated api (#32808) 
issue: #32728

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2024-05-07 20:03:30 +08:00
Jiquan Long
1f58cda957
enhance: add more trace for search & query (#32734) 
issue: https://github.com/milvus-io/milvus/issues/32728

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2024-05-07 13:03:29 +08:00
Bingyi Sun
fecd9c21ba
feat: LRU cache implementation (#32567) 
issue: https://github.com/milvus-io/milvus/issues/32783
This pr is the implementation of lru cache on branch lru-dev.

Signed-off-by: sunby <sunbingyi1992@gmail.com>
Co-authored-by: chyezh <chyezh@outlook.com>
Co-authored-by: MrPresent-Han <chun.han@zilliz.com>
Co-authored-by: Ted Xu <ted.xu@zilliz.com>
Co-authored-by: jaime <yun.zhang@zilliz.com>
Co-authored-by: wayblink <anyang.wang@zilliz.com>
 2024-05-06 20:29:30 +08:00
PowderLi
6289f3a9eb
fix: build milvus in rockylinux8 (#32619) 
issue: #32299

1. xz utils recovers
2. forget to install ninja

Signed-off-by: PowderLi <min.li@zilliz.com>
 2024-04-29 14:53:26 +08:00
Buqian Zheng
858599d831
enhance: sparse float vector to support brute force iterator and range search (#32635) 
issue: #29419

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
 2024-04-29 14:35:26 +08:00
smellthemoon
4fb8044a27
enhance: delete some no lint code (#32182) 
#31728

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2024-04-26 14:15:26 +08:00
Jiquan Long
ccce1e928a
fix: regex query can't handle text with newline (#32569) 
issue: https://github.com/milvus-io/milvus/issues/32482

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2024-04-26 12:01:26 +08:00
Jiquan Long
c002745902
enhance: retrieve output fields after local reduce (#32346) 
issue: #31822

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2024-04-25 09:49:26 +08:00
Gao
0fab265eed
enhance: update knowhere and some header changes (#32468) 
Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2024-04-22 15:47:26 +08:00
zhagnlu
93f0c262c1
fix: add bitset sve support macro protect (#32463) 
#32137

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-04-19 18:05:21 +08:00
zhagnlu
9248a6a149
fix: remove sve flags (#32270) 
#32129

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-04-16 15:21:19 +08:00
chyezh
e19d17076f
fix: delete may lost when enable lru cache, some field should be reset when ReleaseData (#32012) 
issue: #30361

- Delete may be lost when segment is not data-loaded status in lru
cache. skip filtering to fix it.

- `stats_` and `variable_fields_avg_size_` should be reset when
`ReleaseData`

- Remove repeat load delta log operation in lru.

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-04-16 11:17:20 +08:00
Chun Han
337cc0756d
fix: lack good results for insufficient ef(#29883) (#32080) 
related: #29883

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2024-04-13 22:13:23 +08:00
Jiquan Long
4fb85be525
fix: put inverted index into local storage (#32209) 
issue: https://github.com/milvus-io/milvus/issues/32154

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2024-04-13 21:57:19 +08:00
Alexander Guzhva
b5455d176e
fix: dynamically resolve whether SVE is available for bitset (#32137) 
Issue: #32129 
This PR adds a dynamic SVE detection for ARM CPU families for the bitset
code.
Also, allows the code to be compiled if the compiler does not support
NEON (arm-v7).

Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>
 2024-04-11 19:27:18 +08:00
Patrick Weizhi Xu
52ae47c850
enhance: gather materialized view search info once per request (#31996) 
issue: #29892 

This PR:
1. Move the process of gathering materialized search info to when the
search plan is created, before it goes to each segment, to avoid
repeated work and access the plan node under multi-threaded
circumstances.
2. Enforce the supported MV type to `VARCHAR`
3. Add integration test

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
 2024-04-11 15:21:19 +08:00
Chun Han
f3f2a5a7e9
fix: evicted segments in the serverlss mode(#31959) (#31961) 
related: #31959
1. reset segment index status after evicting to lazyload=true
2. reset num_rows to null_opt

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2024-04-10 15:15:19 +08:00
Cai Yudong
a0a4ec8b67
enhance: make range search param check message more meaningful (#32006) 
Issue: #31970

Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
 2024-04-09 16:17:26 +08:00
cai.zhang
1b767669a4
enhance: Throw error instead of crash when index cannot be built (#31844) 
issue: #27589

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-04-09 11:51:18 +08:00
chyezh
7b400252ff
fix: add configuration disk capacity config for lru and fix some bug (#31977) 
issue: #30361

- Add configurable disk capacity limit

- fix bitset reset logic

- make insert record reinsert after clear

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-04-08 15:55:16 +08:00
cqy123456
aba4993c6c
fix: fix some fp16/bf16 code miss in segcore. (#31771) 
issue：https://github.com/milvus-io/milvus/issues/22837

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2024-04-07 14:13:16 +08:00
Alexander Guzhva
cae5722229
enhance: performance improvements for the bitset (#31753) 
Issue: #31752

This PR improves the performance for bitset utilities (introduced in PR
#30454), including varchar filtering

Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>
 2024-04-06 05:19:22 +08:00
Cai Yudong
246586be27
enhance: Unify data type check APIs under internal/core (#31800) 
Issue: #22837 

Move and rename following C++ APIs:
datatype_sizeof() ==> GetDataTypeSize()
datatype_name() ==> GetDataTypeName()
datatype_is_vector() / IsVectorType() ==> IsVectorDataType()
datatype_is_variable() ==> IsVariableDataType()
datatype_is_sparse_vector() ==> IsSparseFloatVectorDataType()
datatype_is_string() / IsString() ==> IsDataTypeString()
datatype_is_floating() / IsFloat() ==> IsDataTypeFloat()
datatype_is_binary() ==> IsDataTypeBinary()
datatype_is_json() ==> IsDataTypeJson()
datatype_is_array() ==> IsDataTypeArray()
datatype_is_variable() == IsDataTypeVariable()
datatype_is_integer() / IsIntegral() ==> IsDataTypeInteger()

Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
 2024-04-02 19:15:14 +08:00
PowderLi
d299fa502e
fix: use milvus-io/vcpkg (#31770) 
issue: #31769
GitHub Disables The XZ Repository because of CVE-2024-3094

Signed-off-by: PowderLi <min.li@zilliz.com>
 2024-04-01 15:01:13 +08:00
chyezh
5655ec4fc0
enhance: add mmap usage metrics (#31708) 
issue: #31707

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-04-01 11:35:12 +08:00