admin/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2026-01-04 18:02:08 +08:00
Code Issues Packages Projects Releases Wiki Activity
20,477 Commits 136 Branches 188 Tags
Commit Graph

1293 Commits

Author SHA1 Message Date
zhagnlu
626b1b2f5e
fix:redefine hybrid internal index type (#35314) 
#32900

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-08-08 10:32:16 +08:00
zhagnlu
c19fe95154
fix: support string match for hybrid and bitmap index (#35294) 
#34841

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-08-07 09:54:22 +08:00
Cai Yudong
3c9a47c8db
feat: Encode traceID and spanID as hex string (#34807) 
Issue: https://github.com/zilliztech/knowhere/pull/714

Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
 2024-08-06 15:20:16 +08:00
Jiquan Long
91df03afe8
feat: put inverted index into ram (#35222) 
fix: https://github.com/milvus-io/milvus/issues/35224

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2024-08-06 11:54:16 +08:00
zhagnlu
4b553b0333
enhance: revert remove duplicated pk function (#35103) 
issue: #34778
 Revert "fix: fix query count(*) concurrently"
 Revert "enhance: mark duplicated pk as deleted "

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-08-05 10:48:17 +08:00
zhagnlu
16dd53e7cf
enhance: remove timestamp_filter after retrieve (#35207) 
#35226

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-08-02 19:32:46 +08:00
smellthemoon
475c333fa2
enhance: add valid_data in span (#35030) 
#31728

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2024-08-02 15:40:14 +08:00
congqixia
a642a26ed4
enhance: Resolve ChunkFileWriter lint issue (#35166) 
See also #34483

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-08-01 16:52:13 +08:00
zhagnlu
f8c1b138a8
fix:fix get array error for int type (#35154) 
#35055

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-08-01 14:30:12 +08:00
zhenshan.cao
aa247f192d
enhance: remove unused code for StorageV2 (#35132) 
issue: https://github.com/milvus-io/milvus/issues/34168

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2024-08-01 12:08:13 +08:00
Bingyi Sun
f229f244d2
enhance: add chunk basic impl (#34634) 
https://github.com/milvus-io/milvus/issues/35112
This pr would not affect milvus functionality by now.
It implments a Chunk memory layout that looks like 
```
VariableColumn
|offset|offset|offset|
|data|data|data|

```
We maybe move offsets to the beginning and add null bitmaps later but
not in this PR.
And mmap test will also be added in another PR.

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2024-08-01 10:29:51 +08:00
foxspy
48b14c56f6
fix: add prometheus dependency for milvus_monitor (#35078) 
add prometheus dependency for monitor module. Or else Some compilers may
report a compilation failure.
issue: #35077

Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
 2024-07-31 10:15:48 +08:00
zhagnlu
dd0c26cf58
enhance: redefine variable column block size (#35040) 
#35013

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-07-30 19:23:50 +08:00
congqixia
de8a266d8a
enhance: Enable linux code checker (#35084) 
See also #34483

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-07-30 15:53:51 +08:00
zhagnlu
a8a4779749
fix: fix parse plan proto failed for search type (#34944) 
#25848

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-07-29 21:19:49 +08:00
zhagnlu
86322e0468
fix: fix query count(*) concurrently (#35007) 
#34778
#34849
fix two problems:
1. count(*) incorrect, if growing insert duplicated (pk, timestamp)
pairs that pk and timestamp all same, need to keep just one pair.
2. count(*) may core dump, if get_real_count interface get snapshot and
do mvcc at not consistency status, mainly happens under concurrency.

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-07-29 19:53:50 +08:00
congqixia
972752258a
enhance: Support otlp http exporter (#35053) 
See also #35052

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-07-29 17:43:49 +08:00
zhagnlu
f77f5364b2
fix: disable use_index when some array expr (#34894) 
#34797

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-07-29 00:17:46 +08:00
Aldrin
9463eeef2b
fix: Avoided dereferencing NULL pointer (#34836) 
issue : https://github.com/milvus-io/milvus/issues/34835

Signed-off-by: Ald392 <imagesai32@gmail.com>
 2024-07-27 17:27:52 +08:00
zhagnlu
804ec24c02
fix:fix retrieve raw data from bitmap array index (#34848) 
#34795

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-07-27 01:53:47 +08:00
smellthemoon
5616b7e8d2
enhance: support null in c data_datacodec and load null value (#32183) 
1. support read and write null in segcore
    will store valid_data(use uint8_t type to save memory) in fieldData.
2. support load null
binlog reader read and write data into column(sealed segment),
insertRecord(growing segment). In sealed segment, store valid_data
directly. In growing segment, considering prior implementation and easy
code reading, it covert uint8_t to fbvector<bool>, which may optimize in
future.
3.  retrieve valid_data.
    parse valid_data in search/query.
#31728

---------

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2024-07-23 16:07:51 +08:00
Min Tian
a4aed9b0b5
enhance: new knowhere param for range_search (#34686) 
issue: #34685 
knowhere needs a new json param `range_search_k` for RangeSearch to
early terminate the iterator.

Signed-off-by: min.tian <min.tian.cn@gmail.com>
 2024-07-23 11:45:43 +08:00
Chun Han
6c19f9baf8
enhance: optimize search reduce perf(#32507) (#34607) 
related: #32507

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
 2024-07-22 20:57:43 +08:00
Chun Han
ed057e6fce
fix: non-init seg_offset for growing raw-data when doing groupby (#34748) 
related:  #34713

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
 2024-07-19 17:01:40 +08:00
foxspy
8e64bf929c
enhance: add scalar filtering and vector search latency metrics (#34785) 
add scalar filtering and vector search latency metrics to distinguish
the cost of scalar filtering.
To add metrics in query chain, add a monitor module and move the metric
files from original storage module.
issue: #34780

Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
 2024-07-19 14:01:39 +08:00
zhagnlu
f1b2f7b640
enhance: refactor bitmap index and internal hybrid index (#34450) 
#32900

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-07-18 10:39:42 +08:00
Buqian Zheng
aa5418a5a9
fix: correctly set search params when using knowhere iterator (#34731) 
issue: #34730

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
 2024-07-17 15:13:41 +08:00
zhagnlu
804dd5409a
enhance: mark duplicated pk as deleted (#34586) 
fix #34247

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-07-16 14:25:39 +08:00
zhagnlu
bd9727a1f7
fix: fix bug that set incorrect info to columnbase (#34428) 
#34427

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-07-14 22:27:46 +08:00
zhagnlu
18c83c6466
fix: fix auto merge error (#34661) 
#33704

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-07-13 13:07:37 +08:00
Chun Han
f00c529aea
feat: support group_size for search_group_by(#33544) (#33720) 
related: #33544

mainly changes in three aspects:

1. enable setting group_size for group by function
2. separate normal reduce and group by reduce
3. eleminate uncessary padding in search result for reducing

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
 2024-07-12 10:17:36 +08:00
Patrick Weizhi Xu
104d0966b7
feat: support partition key isolation (#34336) 
issue: #34332

---------

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
 2024-07-11 19:01:35 +08:00
congqixia
4850336ca3
fix: Write padding at end of mmap file not chunk (#34529) 
Related to #34508

The padding bytes shall be written only at the end of the mmap file not
the chunk of each field data file.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-07-10 11:12:14 +08:00
congqixia
6b4d977a10
fix: Write padding into mmap file in case of SIGBUS (#34443) 
See also #34442

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-07-05 17:44:09 +08:00
yihao.dai
734415b8a2
fix: Reduce duplicate PKs in segcore (#34267) 
issue: https://github.com/milvus-io/milvus/issues/34247

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-07-01 17:42:06 +08:00
smellthemoon
ef3ced8138
fix: descriptor event in previous version not has nullable to parse error (#34235) 
#34176

---------

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2024-07-01 16:38:06 +08:00
congqixia
14e827dc6c
fix: Implement singleflight for segcore ChunkCache (#34250) 
See also #34249

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-07-01 11:46:06 +08:00
zhagnlu
3030e4625e
enhance: refactor variable column to reduce memory cost (#33875) 
#33874

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-06-30 20:16:06 +08:00
zhagnlu
cc1bc07bfd
enhance: add log to bitmap index (#34197) 
#32900

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-06-30 20:02:06 +08:00
Cai Yudong
ad90360162
enhance: Update knowhere commit (#34223) 
Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
 2024-06-27 18:20:06 +08:00
Gao
a933f6731b
fix: centroids file not removed when data skew in major compaction (#34050) 
issue: https://github.com/milvus-io/milvus/issues/30633

Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2024-06-26 10:48:04 +08:00
Patrick Weizhi Xu
b961767005
enhance: support integral type for MV and skip MV if there is only one category (#33161) 
issue: #29892

---------

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
 2024-06-24 10:20:01 +08:00
zhagnlu
03a3f50892
enhance: add skip using array index when some situation (#33947) 
#32900

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-06-23 21:26:02 +08:00
zhagnlu
0d7ea8ec42
enhance: Enhance and correct exception module (#33705) 
#33704

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-06-23 21:22:01 +08:00
chyezh
259a682673
enhance: async search and retrieve in cgo (#33228) 
issue: #30926, #33132
related pr: #33133

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-06-22 09:38:02 +08:00
cqy123456
dc4437ff82
enhance: use segment id and type to register in MmapChunkManager and opt malloc in variableChunk (#33993) 
issue: https://github.com/milvus-io/milvus/issues/32984

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2024-06-20 17:42:02 +08:00
smellthemoon
2a1356985d
enhance: support null in go payload (#32296) 
#31728

---------

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2024-06-19 17:08:00 +08:00
cqy123456
298e50b834
enhance: check index with data type (#33880) 
issue: https://github.com/milvus-io/milvus/issues/22837
related: https://github.com/milvus-io/milvus/pull/33878

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2024-06-19 10:15:59 +08:00
Gao
0d20303e54
fix: fix binary vector data size (#33750) 
issue: https://github.com/milvus-io/milvus/issues/22837

- fix byte size wrong for binary vectors
- fix the expect/actual error msg

Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2024-06-18 21:39:59 +08:00
cqy123456
b460862537
fix: can't find Chunk struct after growing support mmap (#33951) 
issue: https://github.com/milvus-io/milvus/issues/32984

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2024-06-18 18:37:58 +08:00