admin/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2025-12-07 17:48:29 +08:00
Code Issues Packages Projects Releases Wiki Activity
20,299 Commits 133 Branches 183 Tags
Commit Graph

70 Commits

Author SHA1 Message Date
congqixia
67324eb809
enhance: Add l0 segment entry num quota (#34733) 
See also #34670

This PR add quota configuration for l0 segment entry number per
collection. If l0 compaction cannot keep up the insertion/upsertion
rate, this feature could back press the related rate.

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-07-17 17:35:41 +08:00
congqixia
1a248f2668
enhance: Add param item for segmentFlushInterval (#34629) 
See also #28817

Add paramitem for segment flush interval

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-07-12 18:59:35 +08:00
wayblink
a1232fafda
feat: Major compaction (#33620) 
#30633

Signed-off-by: wayblink <anyang.wang@zilliz.com>
Co-authored-by: MrPresent-Han <chun.han@zilliz.com>
 2024-06-10 21:34:08 +08:00
smellthemoon
c61fb1eff5
enhance: do check when add not empty logpath (#33640) 
meta only store logid

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2024-06-07 10:19:51 +08:00
yiwangdr
180d754158
fix: speed up segment lookup via channel name in datacoord (#33530) 
issue: #33342

Signed-off-by: yiwangdr <yiwangdr@gmail.com>
 2024-06-03 14:47:47 +08:00
jaime
3d29907b6e
enhance: decrease cpu overhead during filter segments on datacoord (#33130) 
issue: #33129

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-05-28 19:17:43 +08:00
yiwangdr
e895cfed84
fix: reduce redundant map operations in datacoord (#33343) 
More refactories will be added.
issue: #33342

Signed-off-by: yiwangdr <yiwangdr@gmail.com>
 2024-05-24 12:47:40 +08:00
cai.zhang
6ea7633bd5
enhance: Add memory size for binlog (#33025) 
issue: #33005
1. add `MemorySize` field for insert binlog.
2. `LogSize` means the file size in the storage object.
3. `MemorySize` means the size of the data in the memory.

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2024-05-15 12:59:34 +08:00
congqixia
cedb33ceec
enhance: Improve datacoord segment filtering with collection (#32831) 
See also #32165

This PR modify the `SelectSegments` interface to utilizing collection id
information when selecting segment with provided collection

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-05-08 21:37:29 +08:00
chyezh
2586c2f1b3
enhance: use WalkWithPrefix api for oss, enable piplined file gc (#31740) 
issue: #19095,#29655,#31718

- Change `ListWithPrefix` to `WalkWithPrefix` of OOS into a pipeline
mode.

- File garbage collection is performed in other goroutine.

- Segment Index Recycle clean index file too.

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-04-25 20:41:27 +08:00
yiwangdr
037de8e4d3
enhance: speed up minor functions calls in datacoord (#32389) 
Related to https://github.com/milvus-io/milvus/issues/32165

1. nodeid based channel store access should use map access instead of
iteration.

2. The join-ish functions calls are slow when # collections/segments
increases (e.g. 10k).
e.g.
getNumRowsOfCollectionUnsafe is O(num_segments); GetAllCollectionNumRows
is of O(num_collections*num_segments).

Signed-off-by: yiwangdr <yiwangdr@gmail.com>
 2024-04-20 07:55:21 +08:00
yihao.dai
0fe5e90e8b
enhance: Remove import v1 (#31403) 
Remove all code and logic related to import v1.

issue: https://github.com/milvus-io/milvus/issues/28521

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-03-22 15:29:09 +08:00
chyezh
8f7019468f
fix: starve lock caused by slow GetCompactionTo method when too much segments (#30963) 
issue: #30823

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-03-05 10:04:59 +08:00
jaime
4b0c3dd377
enhance: index meta use independent rather than global meta lock (#30869) 
issue: https://github.com/milvus-io/milvus/issues/30837

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-03-04 16:56:59 +08:00
smellthemoon
e52ce370b6
enhance:don't store logPath in meta to reduce memory (#28873) 
don't store logPath in meta to reduce memory, when service get
segmentinfo, generate logpath from logid.
#28885

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2024-01-18 22:06:31 +08:00
SimFG
26f06dd732
Format the code (#27275) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-09-21 09:45:27 +08:00
congqixia
41af0a98fa
Use go-api/v2 for milvus-proto (#24770) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-06-09 01:28:37 +08:00
congqixia
6375236533
Make SegmentInfo.size atomic (#24457) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-05-29 11:31:28 +08:00
jaime
c9d0c157ec
Move some modules from internal to public package (#22572) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-04-06 19:14:32 +08:00
jaime
d126f06946
Decouple mq module from internal proto definition (#22536) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-03-04 23:21:50 +08:00
yah01
78fd1987eb
Fix data race caused by no clone while updating index (#21772) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-01-17 18:11:42 +08:00
congqixia
204db183f3
Prevent panicking when index segment info not exists (#21664) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-01-12 21:39:40 +08:00
cai.zhang
e5f408dceb
Merge IndexCoord and DataCoord (#21267) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-01-04 19:37:36 +08:00
SimFG
a55f739608
Separate public proto files (#19782) 
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-10-16 20:49:27 +08:00
Ten Thousand Leaves
b30c9d4f60
Support Bulk Load in Milvus 2.2 (#18982) 
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-09-26 18:06:54 +08:00
Jiquan Long
5141e05c47
Add policy to seal long time idle segment (#19222) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-09-20 20:54:50 +08:00
SimFG
d7f38a803d
Separate some proto files (#19218) 
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-09-16 16:56:49 +08:00
Ten Thousand Leaves
fc42ee5bb3
Fix # of rows of recovering segment (#18736) 
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-08-19 19:50:50 +08:00
congqixia
e5fe46124b
Fix isCompacting flag lost after segment clone (#18248) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-07-13 10:08:28 +08:00
Xiaofan
54aeb077e9
Refine compaction selection poliy (#17486) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-06-15 23:14:10 +08:00
godchen
8fe4af3584
[skip e2e]Sort imports (#14809) 
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
 2022-01-05 14:06:48 +08:00
godchen
ef2a62387b
[skip e2e]Sort imports (#14329) 
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
 2021-12-27 18:54:37 +08:00
godchen
c1cb69ed7e
[skip ci]Fix segment info typo (#13643) 
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
 2021-12-21 09:19:53 +08:00
congqixia
1dcd06cc78
Add log file info for all kind of segment binlogs (#13606) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-12-19 20:00:42 +08:00
edward.zeng
6b74aed32a
[skip e2e] Refine datacoord segment info comment (#13602) 
Signed-off-by: Edward Zeng <jie.zeng@zilliz.com>
 2021-12-17 18:54:26 +08:00
sunby
c26cd79532
Add compaction (#9906) 
issue: #9904
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-11-05 22:25:00 +08:00
XuanYang-cn
548634ff05
[skip ci]Update license for dc segmentinfo (#10588) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-10-25 19:44:37 +08:00
congqixia
7ce7cb7a5e
Add a timed flush trigger mechanism (#10197) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-10-20 15:02:36 +08:00
zhenshan.cao
32a029fbed
[skip ci]Fix golint error in datacoord (#10135) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-10-18 21:32:49 +08:00
sunby
044b06421a
Use etcd to watch channel on datanode (#7115) 
issue: #6578
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-10-14 15:44:34 +08:00
sunby
aba9aa450d
[skip ci]Add comment for SetFlushTime (#9668) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-10-11 21:06:37 +08:00
sunby
6fc64b836a
[skip ci]Add comment for SetBinlogs (#9667) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-10-11 21:04:42 +08:00
sunby
1e0c3cbcb9
[skip ci]Add comment for SetCurrentRows (#9526) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-10-09 00:16:56 +08:00
sunby
7834649dbc
[skip ci]Add comment for AddAllocation (#9525) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-10-09 00:15:13 +08:00
sunby
292aeeac63
[skip ci]Add comment for SetAllocations (#9524) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-10-09 00:13:21 +08:00
congqixia
d828a63c94
[skip ci] Add comment for SegmentsInfo AddBinglogPaths (#9386) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-10-08 17:31:50 +08:00
sunby
21655d54be
[skip ci]Add comment for AddSegmentBinlogs (#9389) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-10-07 18:32:50 +08:00
sunby
ad15e786cc
[skip ci]Add comment for SetStartPosition (#9399) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-10-07 17:59:31 +08:00
sunby
d5081783c3
[skip ci]Add comment for SetDmlPosition (#9398) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-10-07 17:57:47 +08:00
sunby
d41cdbb909
[skip ci]Add comment for SetState (#9397) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-10-07 17:55:53 +08:00