admin/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2026-01-07 19:31:51 +08:00
Code Issues Packages Projects Releases Wiki Activity
20,515 Commits 137 Branches 188 Tags
Commit Graph

70 Commits

Author SHA1 Message Date
wei liu
c45f38aa61
enhance: Update protobuf-go to protobuf-go v2 (#34394) 
issue: #34252

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-07-29 11:31:51 +08:00
XuanYang-cn
d7a3697fb5
enhance: Add back compactionTaskNum metrics (#34583) 
Fix L0 compaction task recover unable to set segment not isCompacting

See also: #34460

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-07-11 17:45:37 +08:00
jaime
9630974fbb
enhance: move rocksmq from internal to pkg module (#33881) 
issue: #33956

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-06-25 21:18:15 +08:00
yihao.dai
fb870d2426
fix: Do compressBinlog to fix logID 0 (#34060) 
issue: https://github.com/milvus-io/milvus/issues/34059

Do compressBinlog to ensure that reloadFromKV will fill binlogs' logID
after datacoord restarts.

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-22 16:14:01 +08:00
wayblink
08fcf3f62b
fix: Fix meta prefix overlap bug (#33830) 
#30633

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-06-13 19:27:57 +08:00
wayblink
a1232fafda
feat: Major compaction (#33620) 
#30633

Signed-off-by: wayblink <anyang.wang@zilliz.com>
Co-authored-by: MrPresent-Han <chun.han@zilliz.com>
 2024-06-10 21:34:08 +08:00
smellthemoon
c61fb1eff5
enhance: do check when add not empty logpath (#33640) 
meta only store logid

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2024-06-07 10:19:51 +08:00
cai.zhang
27cc9f2630
enhance: Support analyze data (#33651) 
issue: #30633

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Co-authored-by: chasingegg <chao.gao@zilliz.com>
 2024-06-06 17:37:51 +08:00
yihao.dai
35532a3e7d
fix: Fill stats log id and check validity (#33477) 
1. Fill log ID of stats log from import
2. Add a check to validate the log ID before writing to meta

issue: https://github.com/milvus-io/milvus/issues/33476

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-05 11:17:56 +08:00
zhenshan.cao
ac4f3997ce
enhance: Reconstructing Compaction to possess persistence capability (#33265) 
issue #33586

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2024-06-05 10:17:50 +08:00
smellthemoon
225f4a6134
enhance: use the only MaxEtcdTxnNum (#33070) 
#33071

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2024-05-17 14:27:42 +08:00
cai.zhang
6ea7633bd5
enhance: Add memory size for binlog (#33025) 
issue: #33005
1. add `MemorySize` field for insert binlog.
2. `LogSize` means the file size in the storage object.
3. `MemorySize` means the size of the data in the memory.

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2024-05-15 12:59:34 +08:00
jaime
f48a7ff8ff
enhance: use Delete instead of DeletePartialMatch to remove metrics (#33029) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-05-14 18:49:33 +08:00
yihao.dai
c411cb4a49
enhance: Prevent the backlog of channelCP update tasks, perform batch updates of channelCPs (#30941) 
This PR includes the following adjustments:
1. To prevent channelCP update task backlog, only one task with the same
vchannel is retained in the updater. Additionally, the lastUpdateTime is
refreshed after the flowgraph submits the update task, rather than in
the callBack function.
2. Batch updates of multiple vchannel checkpoints are performed in the
UpdateChannelCheckpoint RPC (default batch size is 128). Additionally,
the lock for channelCPs in DataCoord meta has been switched from key
lock to global lock.
3. The concurrency of UpdateChannelCheckpoint RPCs in the datanode has
been reduced from 1000 to 10.

issue: https://github.com/milvus-io/milvus/issues/30004

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Co-authored-by: jaime <yun.zhang@zilliz.com>
Co-authored-by: congqixia <congqi.xia@zilliz.com>
 2024-03-07 20:39:02 +08:00
yihao.dai
a434d33e75
feat: Add import scheduler and manager (#29367) 
This PR introduces novel managerial roles for importv2:
1. ImportMeta: To manage all the import tasks;
2. ImportScheduler: To process tasks and modify their states;
3. ImportChecker: To ascertain the completion of all tasks and instigate
relevant operations.

issue: https://github.com/milvus-io/milvus/issues/28521

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-03-01 18:31:02 +08:00
smellthemoon
e52ce370b6
enhance:don't store logPath in meta to reduce memory (#28873) 
don't store logPath in meta to reduce memory, when service get
segmentinfo, generate logpath from logid.
#28885

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2024-01-18 22:06:31 +08:00
smellthemoon
073de855b3
enhance:remove some unused code (#29827) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2024-01-14 10:38:51 +08:00
wei liu
9fc5f1176c
fix: Drop segment meta info with prefix (#29856) 
If segment has more than 128 log fils, drop segment will exceed etcd txn
ops limit, which will failed the drop segment request
This PR drop segment meta info with prefix, to avoid drop segment meta
failed

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-01-11 12:22:51 +08:00
aoiasd
1d4be0d257
Adjust datacoord for L0 Delta (#28021) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-11-06 15:26:16 +08:00
Xiaofan
2ea7579dbb
Reduce rpc size for GetRecoveryInfoV2 (#27483) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-10-23 21:44:09 +08:00
SimFG
26f06dd732
Format the code (#27275) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-09-21 09:45:27 +08:00
congqixia
cc9974979f
Add staticcheck linter and fix existing problems (#27174) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-19 10:05:22 +08:00
Enwei Jiao
fb0705df1b
Decouple basetable and componentparam (#26725) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-09-05 10:31:48 +08:00
Xiaofan
f9c47bbc1e
Fix drop index with large txn exceed etcd limit (#25622) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-08-17 21:04:21 +08:00
Filip Haltmayer
7bd9eceb15
Remove unused KV functions (#25972) 
Remove RevertAlterSegmentsAndAddNewSegment



Remove AlterIndex



Remove AlterSegmentsAndAddNewSegment



Remove AlterSegmentIndex



Remove IndexCoordCatalog

Signed-off-by: Filip Haltmayer <filip.haltmayer@zilliz.com>
 2023-07-28 10:13:03 +08:00
Jiquan Long
bccdef1ad7
Fix segment meta only record last flushed binlog (#25707) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-07-19 15:28:57 +08:00
Enwei Jiao
66fdc71479
Refactor logs in DataCoord & DataNode (#25574) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-07-14 15:56:31 +08:00
Bingyi Sun
b22d6d2a5d
Avoid update all binlogs every a segment is updated (#25526) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2023-07-13 14:08:29 +08:00
Bingyi Sun
70c4ddc6c5
Replace LoadWithPrefix with HasPrefix (#25471) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2023-07-11 19:38:28 +08:00
Bingyi Sun
3112dad356
Remove kv mocks (#25426) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2023-07-08 13:20:26 +08:00
Bingyi Sun
268368031c
Remove redundant kv mocks (#25420) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2023-07-07 19:38:26 +08:00
Bingyi Sun
54acb5b725
Remove alter segment interface (#25382) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2023-07-06 20:46:25 +08:00
Bingyi Sun
fe22720ff3
Avoid rewriting all binlogs every time a segment is updated (#25339) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2023-07-06 18:20:26 +08:00
congqixia
5659a3b728
Ignore log number check for compaction (#24835) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-06-13 20:04:38 +08:00
congqixia
41af0a98fa
Use go-api/v2 for milvus-proto (#24770) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-06-09 01:28:37 +08:00
yah01
ebd0279d3f
Check error by Error() and NoError() for better report message (#24736) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-06-08 15:36:36 +08:00
aoiasd
c84bdcea49
merge stats log when segment flushing or compacting (#23570) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-05-29 10:21:28 +08:00
jaime
c9d0c157ec
Move some modules from internal to public package (#22572) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-04-06 19:14:32 +08:00
wayblink
aa6212d5b0
[Cherry-pick] Fix channel checkpoint issues and fix GC (#22559) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
Co-authored-by: Ten Thousand Leaves <69466447+soothing-rain@users.noreply.github.com>
 2023-03-09 14:13:52 +08:00
jaime
d126f06946
Decouple mq module from internal proto definition (#22536) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-03-04 23:21:50 +08:00
yah01
319f1773af
Fix segment not found if forward delete to empty segment (#22528) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-03-03 14:13:49 +08:00
Enwei Jiao
697dedac7e
Use cockroachdb/errors to replace other error pkg (#22390) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-02-26 11:31:49 +08:00
congqixia
f2575e5fa8
Add unconvert & durationcheck linters and fix issues (#22161) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-02-15 17:22:34 +08:00
jaime
a2435cfc4f
[Cherry-Pick]Add a segment metrics for count number of binlog files (#22103) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-02-13 10:24:33 +08:00
Jiquan Long
9fd9bed2b9
Remove collection meta after GC finished (#21595) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-01-12 09:55:42 +08:00
cai.zhang
e127cf7b99
Reset indexpb for upgrade (#21620) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-01-11 14:35:40 +08:00
jaime
73d8509600
Improve ListSegments performance for datacoord catalog (#21477) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-01-06 14:33:36 +08:00
cai.zhang
e5f408dceb
Merge IndexCoord and DataCoord (#21267) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-01-04 19:37:36 +08:00
Xiaofan
5d044c4f29
Add check for stats log and bin log number (#21264) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-12-26 16:35:30 +08:00
Ten Thousand Leaves
9d8f422cb8
Reduce maxFieldNum from 256 to 64 (#21104) 
issue: #21077

/kind improvement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-12-14 10:39:22 +08:00