admin/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2025-12-07 09:38:39 +08:00
Code Issues Packages Projects Releases Wiki Activity
20,424 Commits 133 Branches 183 Tags
Commit Graph

79 Commits

Author SHA1 Message Date
smellthemoon
475c333fa2
enhance: add valid_data in span (#35030) 
#31728

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2024-08-02 15:40:14 +08:00
zhenshan.cao
aa247f192d
enhance: remove unused code for StorageV2 (#35132) 
issue: https://github.com/milvus-io/milvus/issues/34168

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2024-08-01 12:08:13 +08:00
zhagnlu
804dd5409a
enhance: mark duplicated pk as deleted (#34586) 
fix #34247

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-07-16 14:25:39 +08:00
zhagnlu
3030e4625e
enhance: refactor variable column to reduce memory cost (#33875) 
#33874

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-06-30 20:16:06 +08:00
zhagnlu
c6f8a73bb2
enhance: optimize some cache to reduce memory usage (#33534) 
#33533

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-06-04 14:09:47 +08:00
Chun Han
416a2cf507
fix: query iterator lack results(#33137) (#33422) 
related: #33137 
adding has_more_result_tag for various level's reduce to rectify
reduce_stop_for_best

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2024-05-30 17:51:44 +08:00
Jiquan Long
9837ad6a8d
enhance: remove deprecated api (#32808) 
issue: #32728

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2024-05-07 20:03:30 +08:00
Jiquan Long
1f58cda957
enhance: add more trace for search & query (#32734) 
issue: https://github.com/milvus-io/milvus/issues/32728

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2024-05-07 13:03:29 +08:00
Jiquan Long
c002745902
enhance: retrieve output fields after local reduce (#32346) 
issue: #31822

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2024-04-25 09:49:26 +08:00
chyezh
e19d17076f
fix: delete may lost when enable lru cache, some field should be reset when ReleaseData (#32012) 
issue: #30361

- Delete may be lost when segment is not data-loaded status in lru
cache. skip filtering to fix it.

- `stats_` and `variable_fields_avg_size_` should be reset when
`ReleaseData`

- Remove repeat load delta log operation in lru.

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-04-16 11:17:20 +08:00
xige-16
e9fdd2475d
fix: fix searchPlan metricType modified concurrently (#30227) 
issue: #30225
/kind bug
Signed-off-by: xige-16 <xi.ge@zilliz.com>

---------

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2024-01-26 14:03:09 +08:00
yah01
f542bdbf3c
enhance: calc the accurate mem size of segment (#30093) 
this stats the real memory size of segment, also reduces the memory
usage in mmap mode
resolve #30095

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2024-01-19 12:32:53 +08:00
xige-16
fa7cf587b0
enhance: Opt metric type does not match error message (#29927) 
issue: #29791 
/kind improvement
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2024-01-17 20:25:03 +08:00
zhenshan.cao
60e88fb833
fix: Restore the MVCC functionality. (#29749) 
When the TimeTravel functionality was previously removed, it
inadvertently affected the MVCC functionality within the system. This PR
aims to reintroduce the internal MVCC functionality as follows:

1. Add MvccTimestamp to the requests of Search/Query and the results of
Search internally.
2. When the delegator receives a Query/Search request and there is no
MVCC timestamp set in the request, set the delegator's current tsafe as
the MVCC timestamp of the request. If the request already has an MVCC
timestamp, do not modify it.
3. When the Proxy handles Search and triggers the second phase ReQuery,
divide the ReQuery into different shards and pass the MVCC timestamp to
the corresponding Query requests.

issue: #29656

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2024-01-09 11:38:48 +08:00
MrPresent-Han
9e2e7157e9
feat: support search_group_by for milvus(#25324) (#28983) 
related: #25324

Search GroupBy function, used to aggregate result entities based on a
specific scalar column.
several points to mention:

1. Temporarliy, the whole groupby is implemented separated from
iterative expr framework **for the first period**
2. In the long term, the groupBy operation will be incorporated into the
iterative expr framework:https://github.com/milvus-io/milvus/pull/28166
3. This pr includes some unrelated mocked interface regarding alterIndex
due to some unworth-to-mention reasons. All these un-associated content
will be removed before the final pr is merged. This version of pr is
only for review
4. All other related details were commented in the files comparison

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2024-01-05 15:50:47 +08:00
MrPresent-Han
bfca0a7926
fix: refine skipIndex to resolve cyclic dependcy(#29132) (#29189) 
related: #29132

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-12-19 10:26:40 +08:00
Bingyi Sun
36f69ea031
feat: integrate storagev2 in building index of segcore (#28768) 
issue: https://github.com/milvus-io/milvus/issues/28655

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2023-12-05 16:48:54 +08:00
MrPresent-Han
836f300536
support skip-index based on chunk-metrics to accelerate expr filter(#27925) (#28297) 
related: #27925

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-11-15 11:20:19 +08:00
yah01
93e2eb78c9
Delete only if primary keys exist (#25292) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-09-20 19:03:25 +08:00
Enwei Jiao
c3f15c6b95
Refactor duplicate error class into one place (#26985) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-09-11 20:43:17 +08:00
Enwei Jiao
ca1349708b
Remove time travel ralted testcase (#26119) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-08-10 18:53:17 +08:00
xige-16
1055c90456
Add default retrieve limit (#24782) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2023-08-10 14:11:15 +08:00
Jiquan Long
bafb183a2b
Optimize bitset usage (#26096) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-08-03 15:25:09 +08:00
Jiquan Long
5c1f79dc54
Push down the limit operator to segcore (#25959) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-08-01 20:29:05 +08:00
xige-16
04082b3de2
Migrate the ability to upload and download binlog to cpp (#22984) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2023-06-25 14:38:44 +08:00
yah01
a413842e38
Fix deleted data is still visible (#24849) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-06-16 17:16:41 +08:00
zhagnlu
f3f3f8a849
Segcore retrieve by pk optimazation (#24659) (#24660) 
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2023-06-15 15:04:39 +08:00
yihao.dai
092d743917
Add support for getting vectors by ids (#23450) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-04-23 09:00:32 +08:00
yah01
bdd6bc7695
Re-format cpp code (#22513) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-03-02 15:55:49 +08:00
yah01
7478e44911
Support using mmap to load data (#22052) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-03-01 18:07:49 +08:00
smellthemoon
f5ab719f21
timestamp decided if the pks were the same (#20166) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2022-11-21 10:55:10 +08:00
xige-16
428840178c
Support diskann index for vector field (#19093) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-09-21 20:16:51 +08:00
Cai Yudong
dcf45df029
Optimize API vector_search parameter in segcore (#18827) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-08-25 16:16:54 +08:00
Jeng.Gwan
638f6c36e9
Support to get real row count of segment (#18115) 
Signed-off-by: xaxys <zheng.guan@zilliz.com>
 2022-07-18 09:58:28 +08:00
xige-16
54d17bc5da
Fix query too slow when insert multi repeated pk data (#18231) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-07-13 10:22:26 +08:00
Cai Yudong
7385770014
Upgrade to knowhere-v1.1.12 (#17692) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-06-24 10:34:18 +08:00
Cai Yudong
e78269f450
Optimize search related interface in segcore (#17568) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-06-23 18:16:13 +08:00
bigsheeper
f38637c227
Pass PlaceholderGroup pointer to prevent memory copy in SegCore (#17389) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-06-06 21:34:05 +08:00
xige-16
56778787be
Reverse data from scalar index (#17145) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-05-26 14:58:01 +08:00
Letian Jiang
b3eb2b1d0d
Support deltaLog loading on growing segment (#16903) 
Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>
 2022-05-12 11:59:53 +08:00
xige-16
515d0369de
Support string type in segcore (#16546) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
Co-authored-by: dragondriver <jiquan.long@zilliz.com>

Co-authored-by: dragondriver <jiquan.long@zilliz.com>
 2022-04-29 13:35:49 +08:00
zhenshan.cao
58ea38142f
Use boost dynamic_bitset in segcore (#16476) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-04-14 22:37:34 +08:00
xige-16
27b4cbc098
Cherry pick remove translateHits commit to mater (#16436) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Co-authored-by: bigsheeper <yihao.dai@zilliz.com>
 2022-04-08 20:27:31 +08:00
Jiquan Long
fd589baca7
Integrates marisa trie index (#16192) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2022-04-01 15:31:29 +08:00
zhenshan.cao
b03da87df5
Accelerate query speed in sealed segment (#16287) 
Fix delete duplicate key

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-03-31 17:33:28 +08:00
Cai Yudong
54b8b24151
Rename variable names for better readibility (#15700) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-02-22 22:15:52 +08:00
Cai Yudong
cbb01051f0
Update Search return type (#12578) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-12-02 11:45:32 +08:00
Cai Yudong
d24cc8f846
Reorder header files for segcore/SegmentInterface.h (#12434) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-11-30 18:39:49 +08:00
Cai Yudong
db2a0a3bd3
Fix reduce panic (#11325) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-11-05 18:17:00 +08:00
yukun
7acb48919b
Add Delete func in Segcore SegmentSealed (#10501) 
Signed-off-by: fishpenguin <kun.yu@zilliz.com>
 2021-10-24 14:19:10 +08:00