admin/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2026-02-02 01:06:41 +08:00
Code Issues Packages Projects Releases Wiki Activity
20,091 Commits 146 Branches 196 Tags
Commit Graph

1505 Commits

Author SHA1 Message Date
zhagnlu
4e02e57044
enhance: mark duplicated pk as deleted (#34619) 
pr: #34586

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-07-12 10:27:37 +08:00
congqixia
0d8defb1f9
fix: [2.4] Write padding at end of mmap file not chunk (#34530) 
Cherry-pick from master
pr: #34529
Related to #34508

The padding bytes shall be written only at the end of the mmap file not
the chunk of each field data file.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-07-09 20:32:14 +08:00
congqixia
014820e9d2
fix: [2.4] Write padding into mmap file in case of SIGBUS (#34443) (#34455) 
Cherry-pick from master
pr: #34443
See also #34442

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-07-08 10:06:19 +08:00
zhagnlu
173c02902e
enhance: refactor variable column to reduce memory cost (#33875) (#34367) 
cherry-pick commit from master:
pr: #33875

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-07-05 15:52:09 +08:00
Gao
261b61e875
fix: centroids file not removed when data skew in major compaction (#34359) 
issue: https://github.com/milvus-io/milvus/issues/30633
pr: #34050

Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2024-07-05 10:42:10 +08:00
zhagnlu
74da97796b
enhance: Enhance and correct exception module (#34366) 
cherry-pick commit from master:
pr: #33705

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-07-05 10:06:10 +08:00
zhagnlu
190a2ca7b8
enhance: reduce cpp ut test cost time (#34414) 
#34413
cherry-pick part from master commit: pr: #33358

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-07-04 20:40:10 +08:00
foxspy
b33243f236
enhance: Update Knowhere version (#34405) 
1. cherry-pick pr #34223 (update the parameter dataset from reference to
share_ptr)
2. update knowhere version from v2.3.5 to v2.3.6
(https://github.com/zilliztech/knowhere/releases/tag/v2.3.6)

/kind branch-feature

---------

Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
Co-authored-by: Cai Yudong <yudong.cai@zilliz.com>
 2024-07-04 14:46:11 +08:00
chyezh
a1a0a56f86
enhance: async search and retrieve in cgo (#34200) 
issue: #33132
pr: #33133
other pr: #33228, #34084, #33946

- implement future-based cgo utility
- async search and retrieve in cgo
- modify gc configuration document

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-07-04 13:02:09 +08:00
congqixia
58c4d061d3
fix: [2.4] Implement singleflight for segcore ChunkCache (#34250) (#34283) 
Cherry-pick from master
pr: #34250
See also #34249

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-07-02 10:24:08 +08:00
cai.zhang
c924b0b502
enhance: [cherry-pick] Refine index code and support analyze data (#34311) 
This PR primary picks up the support analyzing functionality, including
the following commits:
- main functionality: https://github.com/milvus-io/milvus/pull/33651
- refine indexnode code: https://github.com/milvus-io/milvus/pull/33458
- related fixes:
  - https://github.com/milvus-io/milvus/pull/33832
  - https://github.com/milvus-io/milvus/pull/33161

issue: #30633 
master prs: #33651 , #33458 , #33832 , #33161

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
Co-authored-by: chasingegg <chao.gao@zilliz.com>
Co-authored-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
 2024-07-02 09:50:39 +08:00
yihao.dai
3c5ad4999e
fix: Reduce duplicate PKs in segcore (#34267) (#34302) 
issue: issue: https://github.com/milvus-io/milvus/issues/34247

pr: https://github.com/milvus-io/milvus/pull/34267

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-07-01 21:08:07 +08:00
zhagnlu
fc6bd387b8
fix: expr add skip using index while index exists (#34202) 
This PR cherry-pick part from commit:

-  enhance: add skip using array index when some situation #33947 
-  fix: [ut] regex query under unsupported index  #34087

pr: #33947, #34087

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-06-30 20:03:13 +08:00
congqixia
07a05db51b
fix: [2.4] Wrap init segcore tracing with golang timeout (#33494) (#34191) 
Cherry-pick from master
pr: #33494
See also #33483

Wrap `C.InitTrace` & `C.SetTrace` with timeout preventing otlp
initializtion hangs forever when endpoint is not set correctly

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-26 19:44:05 +08:00
cqy123456
f5344abdaf
enhance: [cherry-pick]growing segment support mmap (#34110) 
issue: issue: https://github.com/milvus-io/milvus/issues/32984
related pr: https://github.com/milvus-io/milvus/pull/32633,
https://github.com/milvus-io/milvus/pull/33951,
https://github.com/milvus-io/milvus/pull/33993

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2024-06-25 14:52:07 +08:00
Patrick Weizhi Xu
1168e8c107
enhance: [2.4] support integral type for MV and skip MV if there is only one category (#34005) 
pr: #33161
issue: #29892

---------

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
 2024-06-24 19:12:02 +08:00
Jiquan Long
22e6807e9a
feat: support inverted index for array (#33452) (#34053) 
pr: https://github.com/milvus-io/milvus/pull/33184
pr: https://github.com/milvus-io/milvus/pull/33452
pr: https://github.com/milvus-io/milvus/pull/33633
issue: https://github.com/milvus-io/milvus/issues/27704
Co-authored-by: xiaocai2333 <cai.zhang@zilliz.com>

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
Co-authored-by: cai.zhang <cai.zhang@zilliz.com>
 2024-06-24 10:50:03 +08:00
congqixia
e02a95e3c2
fix: [2.4] Return record with largest timestamp for entires with same PK (#33936) (#34024) 
Cherry-pick from master
pr: #33936
See also #33883

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-21 14:14:01 +08:00
Jiquan Long
032685239f
enhance: fix copying hits of inverted index twice (#33968) (#33975) 
issue: https://github.com/milvus-io/milvus/issues/29793
pr: https://github.com/milvus-io/milvus/pull/33968
The custom `VecCollector` have already transformed the results into
vector of offsets, no need to copy them twice.

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2024-06-20 10:06:06 +08:00
presburger
30b70b05d0
fix: fix milvus gpu compile error (#33990) 
NVIDIA RAFT uses customized versions of spdlog and fmt, which conflict
with the versions provided by Conan. Therefore, it is necessary to
isolate the build environment for knowhere to avoid these conflicts.
issue: #33991

Signed-off-by: yusheng.ma <yusheng.ma@zilliz.com>
 2024-06-20 09:20:00 +08:00
presburger
79546a6c25
fix: fix not find arrow::parquet lib (#33941) 
Upgrade the Arrow version and remove unnecessary library link
dependencies.
issue: #33942
pr: https://github.com/milvus-io/milvus/pull/33928

Signed-off-by: yusheng.ma <yusheng.ma@zilliz.com>
 2024-06-18 13:09:59 +08:00
cqy123456
562369627d
enhance: [cherry-pick]check index with data type in knowhere api (#33878) 
issue: https://github.com/milvus-io/milvus/issues/22837
related: https://github.com/milvus-io/milvus/pull/33880

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2024-06-14 19:45:58 +08:00
zhagnlu
b071ebe49b
fix: readd timestamp index because segment timestamp not ordered (#33857) 
pr: #33856

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-06-14 14:34:01 +08:00
Gao
7ef28927e6
fix: [2.4] fix binary vector data size (#33751) 
issue #22837 
pr: #33750

Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2024-06-12 20:11:55 +08:00
zhagnlu
729001c3a8
fix: clean vector memory (#33692) (#33708) 
pr: #33692

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-06-07 18:17:53 +08:00
wei liu
a425a041a8
enhance: Remove unnecessary log info during load segment (#33663) (#33670) 
pr: #33663

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-06-06 17:11:50 +08:00
foxspy
beaa94f170
enhance: [2.4] support sparse cardinal hnsw index (#33662) 
issue: https://github.com/milvus-io/milvus/issues/29419
pr: https://github.com/milvus-io/milvus/pull/33656

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
Co-authored-by: Buqian Zheng <zhengbuqian@gmail.com>
 2024-06-06 14:47:54 +08:00
Buqian Zheng
0ecdd5a59c
enhance: [2.4] Improve GetVectorById of Sparse Float Vector (#33652) 
issue: https://github.com/milvus-io/milvus/issues/29419
pr: #33209 

codecov will fail due to newly added ut in test_sealed.cpp skipped due
to https://github.com/milvus-io/milvus/issues/33210

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
 2024-06-06 13:55:51 +08:00
congqixia
084140b093
enhance: [2.4] Make hasMoreResult accurate when hit number larger than limit(#33609) (#33642) 
Cherry-pick from master
pr: #33609

See also milvus-io/milvus-sdk-go#756

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-06 11:45:51 +08:00
foxspy
58a7111599
enhance: [cherry-pick] add autoindex mapping for binary/sparse datatype (#33625) 
issue: #22837 
pr: #33624

Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
 2024-06-06 10:33:52 +08:00
cqy123456
443197bdfc
enhance: [cherry-pick]disk index support binary_vector (#33575) 
issue:https://github.com/milvus-io/milvus/issues/22837
related https://github.com/milvus-io/milvus/pull/33631

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2024-06-05 19:37:57 +08:00
zhagnlu
f94a72252c
enhance: optimize some cache to reduce memory usage (#33560) 
pr: https://github.com/milvus-io/milvus/pull/33534
pr: #33599

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-06-05 19:21:52 +08:00
Chun Han
627b787aed
fix: query iterator lack results(#33137) (#33422) (#33506) 
related: #33137 
pr: https://github.com/milvus-io/milvus/pull/33422

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2024-06-05 18:51:52 +08:00
Gao
b32c14f96b
fix: [2.4] correct get vector data size for bf16/fp16/binary vector (#33488) 
issue: #22837 
pr: #33377

Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2024-06-05 14:31:57 +08:00
yihao.dai
3a037884ac
fix: Fix exception info is missing (#33393) (#33396) 
Replace based std::exception to prevent "object slicing"

issue: https://github.com/milvus-io/milvus/issues/33392

pr: https://github.com/milvus-io/milvus/pull/33393

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-04 15:15:48 +08:00
Buqian Zheng
38e5e33196
enhance: [2.4] update knowhere version (#33491) 
issue: https://github.com/milvus-io/milvus/issues/33489
pr: #33490
update knowhere version to latest. remove usage of seed_ef as it be
replaced by existing ef.

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
 2024-05-31 10:17:50 +08:00
Alexander Guzhva
713c9ce4c7
fix: [Cherry-pick][2.4] make milvus compilable using gcc-13 (#33441) 
issue: https://github.com/milvus-io/milvus/issues/30190
pr: https://github.com/milvus-io/milvus/pull/30149

add a missing header

Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>
 2024-05-29 16:19:45 +08:00
Alexander Guzhva
5a668a17a9
enhance: [Cherry-pick][2.4] Upgrade bitset for ARM SVE (#33440) 
issue: https://github.com/milvus-io/milvus/issues/32826
pr: https://github.com/milvus-io/milvus/pull/32718
improve ARM SVE performance for internal/core/src/bitset

Baseline timings for gcc 11.4 + Graviton 3 + manually enabled SVE:
https://gist.github.com/alexanderguzhva/a974b50134c8bb9255fb15f144e5ac83

Candidate timings for gcc 11.4 + Graviton 3 + manually enabled SVE:
https://gist.github.com/alexanderguzhva/19fc88f4ad3757e05e0f7feaf563b3d3

Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>
 2024-05-29 16:17:51 +08:00
cai.zhang
ef10d15658
fix: [2.4]Throw an exception after all the threads in thread pool finished (#32810) (#33314) 
issue: #32487
master pr: #32810

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-05-23 17:55:40 +08:00
Buqian Zheng
ec33024637
fix: [2.4] in segcore validate search sparse vector to make sure it is legal (#33312) 
pr: #32856

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
 2024-05-23 17:35:40 +08:00
foxspy
2b62324d7b
enhance: Update Knowhere version (#33310) 
/kind branch-feature

Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
 2024-05-23 16:53:40 +08:00
congqixia
3c31499bbc
enhance: [2.4]Assert insert data length not overflow int (#33248) (#33271) 
Cherry-pick from master
pr: #33248 
When InsertData is too large for cpp proto unmarshalling, the error
message is confusing since the length is overflowed

This PR adds assertion for insert data length.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-05-22 19:07:40 +08:00
Gao
3777eb2f68
enhance: update knowhere new version (#33043) 
for milvus new release, knowhere update to latest release version

Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2024-05-14 20:45:32 +08:00
sre-ci-robot
09cca1fb51
[automated] Update Knowhere Commit (#33030) 
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 2024-05-14 01:47:32 +08:00
foxspy
f6777267e3
enhance: add score compute consistency config for knowhere (#32997) 
issue: https://github.com/milvus-io/milvus/issues/32583
related: #32584

Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
 2024-05-13 14:21:31 +08:00
Gao
f7d29118a5
enhance: update knowhere version (#32931) 
Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2024-05-10 16:13:31 +08:00
Buqian Zheng
bb7765cbd6
fix: fix Indexing.Iterator ut: build index with all data at once (#32844) 
issue: #32843

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
 2024-05-10 11:31:30 +08:00
aoiasd
54a51b1236
enhance: Support dynamic config for opentelemetry trace (#32169) 
relate: https://github.com/milvus-io/milvus/issues/31940

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2024-05-09 17:43:30 +08:00
sre-ci-robot
3d78b90fe7
[automated] Update Knowhere Commit (#32872) 
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 2024-05-09 01:57:30 +08:00
Jiquan Long
035a508722
fix: make sure inverted index has only one segment (#32858) 
issue: #32717

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2024-05-08 21:25:30 +08:00