zhagnlu
fc876639cf
enhance: support json stats with shredding design ( #42534 )
...
#42533
Co-authored-by: luzhang <luzhang@zilliz.com>
2025-09-01 10:49:52 +08:00
Zhen Ye
e97e44d56e
enhance: limit the gc concurrency when cpu is high ( #43059 )
...
issue: #42833
Signed-off-by: chyezh <chyezh@outlook.com>
2025-07-04 09:22:43 +08:00
Zhen Ye
2fd8f910b0
fix: data duplicated when msgdispatcher make splitting ( #42827 )
...
issue: #41570
Signed-off-by: chyezh <chyezh@outlook.com>
2025-06-19 16:32:39 +08:00
yihao.dai
9acba25fad
enhance: Replace pointer-based map key with id in garbage collector ( #42647 )
...
issue: https://github.com/milvus-io/milvus/issues/42592
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-06-13 20:50:36 +08:00
Bingyi Sun
1bf960b1a8
enhance: Check loaded segments before gc ( #42639 )
...
issue: https://github.com/milvus-io/milvus/issues/42412
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2025-06-13 17:44:38 +08:00
Xianhui Lin
98067f5fc6
fix: datacoord stop get stuck After upgrading from 2.5 to 2.6 ( #42674 )
...
datacoord stop get stuck After upgrading from 2.5 to 2.6
issue:https://github.com/milvus-io/milvus/issues/42656
Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
2025-06-12 16:56:36 +08:00
yihao.dai
a72463c619
enhance: Optimize memory usage during garbage collection ( #42593 )
...
Defer clone and decompress operations until just before removing from
meta, instead of eagerly applying them to all segments in advance.
issue: https://github.com/milvus-io/milvus/issues/42592
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-06-11 20:40:39 +08:00
cai.zhang
63246c040f
fix: Use locking to ensure the atomicity of dropping segment indexes ( #42075 )
...
issue: #41288
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2025-05-28 10:00:28 +08:00
yihao.dai
65dd3982d8
fix: Fix ants.Pool goroutine leak ( #41892 )
...
1. Release the pool after it is no longer in use.
2. Upgrade ants.Pool to fix the goroutine leak issue (see [PR
#287 ](https://github.com/panjf2000/ants/pull/287 )).
issue: https://github.com/milvus-io/milvus/issues/41838
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-05-19 17:56:22 +08:00
Xianhui Lin
3bc24c264f
enhance: Add json key inverted index in stats for optimization ( #38039 )
...
Add json key inverted index in stats for optimization
https://github.com/milvus-io/milvus/issues/36995
---------
Signed-off-by: Xianhui.Lin <xianhui.lin@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2025-04-10 15:20:28 +08:00
cai.zhang
a74580c1ca
fix: Set task version for stats task ( #40035 )
...
issue: #40034
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2025-02-27 17:49:59 +08:00
congqixia
cb7f2fa6fd
enhance: Use v2 package name for pkg module ( #39990 )
...
Related to #39095
https://go.dev/doc/modules/version-numbers
Update pkg version according to golang dep version convention
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-02-22 23:15:58 +08:00
Zhen Ye
bb8d1ab3bf
enhance: make new go package to manage proto ( #39114 )
...
issue: #39095
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2025-01-10 10:49:01 +08:00
tinswzy
27229f7907
enhance: refine exists log print with ctx ( #38080 )
...
issue: #35917
Refines exists log print with ctx
Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
2024-12-14 22:36:44 +08:00
tinswzy
1dbb6cd7cb
enhance: refine the datacoord meta related interfaces ( #37957 )
...
issue: #35917
This PR refines the meta-related APIs in datacoord to allow the ctx to
be passed down to the catalog operation interfaces
Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
2024-11-26 19:46:34 +08:00
aoiasd
4682bba586
enhance: Support gc with bm25 stats logs ( #37408 )
...
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-11-12 11:30:28 +08:00
jaime
ef1832ff9c
enhance: enable manual compaction for collections without indexes ( #36577 )
...
issue: #36576
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-10-08 19:57:18 +08:00
cai.zhang
2c9bb4dfa3
feat: Support stats task to sort segment by PK ( #35054 )
...
issue: #33744
This PR includes the following changes:
1. Added a new task type to the task scheduler in datacoord: stats task,
which sorts segments by primary key.
2. Implemented segment sorting in indexnode.
3. Added a new field `FieldStatsLog` to SegmentInfo to store token index
information.
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-09-02 14:19:03 +08:00
jaime
b7ea1defd3
fix: mistaken deletions may occur during GC channel checkpoints ( #35707 )
...
issue: #35706
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-08-28 10:11:05 +08:00
XuanYang-cn
82743c5c50
fix: Clear channelcp meta and metrics ASAP ( #35658 )
...
See also: #35588
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-08-26 12:22:57 +08:00
chyezh
a495754034
enhance: the datacoord gc should fast quitable ( #35050 )
...
issue: #35049
Signed-off-by: chyezh <chyezh@outlook.com>
2024-07-29 17:03:48 +08:00
jaime
a08a0c831f
fix: encountering orphan channel-cp meta after DataCoord GC ( #34612 )
...
issue: #34545
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-07-11 23:01:35 +08:00
wayblink
a1232fafda
feat: Major compaction ( #33620 )
...
#30633
Signed-off-by: wayblink <anyang.wang@zilliz.com>
Co-authored-by: MrPresent-Han <chun.han@zilliz.com>
2024-06-10 21:34:08 +08:00
cai.zhang
27cc9f2630
enhance: Support analyze data ( #33651 )
...
issue: #30633
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Co-authored-by: chasingegg <chao.gao@zilliz.com>
2024-06-06 17:37:51 +08:00
congqixia
cedb33ceec
enhance: Improve datacoord segment filtering with collection ( #32831 )
...
See also #32165
This PR modify the `SelectSegments` interface to utilizing collection id
information when selecting segment with provided collection
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-05-08 21:37:29 +08:00
Bingyi Sun
fecd9c21ba
feat: LRU cache implementation ( #32567 )
...
issue: https://github.com/milvus-io/milvus/issues/32783
This pr is the implementation of lru cache on branch lru-dev.
Signed-off-by: sunby <sunbingyi1992@gmail.com>
Co-authored-by: chyezh <chyezh@outlook.com>
Co-authored-by: MrPresent-Han <chun.han@zilliz.com>
Co-authored-by: Ted Xu <ted.xu@zilliz.com>
Co-authored-by: jaime <yun.zhang@zilliz.com>
Co-authored-by: wayblink <anyang.wang@zilliz.com>
2024-05-06 20:29:30 +08:00
chyezh
77f4f0c5a8
fix: gc only do once ( #32722 )
...
issue: #29655
produced by pr: #31740
Signed-off-by: chyezh <chyezh@outlook.com>
2024-04-30 14:19:27 +08:00
chyezh
2586c2f1b3
enhance: use WalkWithPrefix api for oss, enable piplined file gc ( #31740 )
...
issue: #19095,#29655,#31718
- Change `ListWithPrefix` to `WalkWithPrefix` of OOS into a pipeline
mode.
- File garbage collection is performed in other goroutine.
- Segment Index Recycle clean index file too.
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-04-25 20:41:27 +08:00
congqixia
3f7c77481d
fix: Use correct ticker for gc work defer ( #31456 )
...
See also #31362
Fix the second defer using same ticker var.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-03-20 22:17:06 +08:00
congqixia
16c661c722
enhance: Use different interval for gc scan ( #31363 )
...
See also #31362
This PR make datacoord garbage collection scan operation using differet
interval than other opeartion.
This interval is a newly added param item, which default value is 7*24
hours.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-03-19 11:27:06 +08:00
jaime
4b0c3dd377
enhance: index meta use independent rather than global meta lock ( #30869 )
...
issue: https://github.com/milvus-io/milvus/issues/30837
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-03-04 16:56:59 +08:00
wayblink
1635211c3f
enhance: Add log when garbage collection resumed ( #30535 )
...
/kind enhancement
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-02-05 17:09:15 +08:00
smellthemoon
e52ce370b6
enhance:don't store logPath in meta to reduce memory ( #28873 )
...
don't store logPath in meta to reduce memory, when service get
segmentinfo, generate logpath from logid.
#28885
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-01-18 22:06:31 +08:00
SimFG
d23f87a393
enhance: Add concurrency for datacoord segment GC ( #29561 )
...
issue: https://github.com/milvus-io/milvus/issues/29553
/kind improvement
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-01-03 13:16:57 +08:00
congqixia
8a63e53421
enhance: Add http method to control datacoord garbage collection ( #29052 )
...
See also #29051
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-14 19:26:39 +08:00
congqixia
1d76565894
Add metrics for garbage collection ( #27303 )
...
Also fix second metrics usage in compaction
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-22 18:47:25 +08:00
SimFG
26f06dd732
Format the code ( #27275 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-09-21 09:45:27 +08:00
congqixia
19e4deb792
Make garbage collector fetch meta after list from storage ( #27203 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-19 17:05:34 +08:00
PowderLi
4feb3fa7c6
support azure ( #26398 )
...
Signed-off-by: PowderLi <min.li@zilliz.com>
2023-09-19 10:01:23 +08:00
yah01
168e82ee10
Fix panic while handling with the nil status ( #27040 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-09-15 10:09:21 +08:00
congqixia
f8bcf60e55
Adjust some confusing Warning log to INFO ( #26356 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-08-15 17:49:33 +08:00
zhenshan.cao
7619db1edc
Fast GC compacted dropped segment ( #26207 )
...
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2023-08-09 19:39:15 +08:00
Zhihong Yu
150947a5f4
Conditionally log warning when DropSegment fails ( #25774 )
...
Signed-off-by: Ted Yu <yuzhihong@gmail.com>
2023-07-25 10:39:01 +08:00
cai.zhang
b15e34db21
Add contraint for compaction based indexed segments ( #25709 )
...
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-07-23 21:31:00 +08:00
Enwei Jiao
66fdc71479
Refactor logs in DataCoord & DataNode ( #25574 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-07-14 15:56:31 +08:00
Xiaofan
dbf0130803
Faster garbage collect on compacted data ( #25088 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-07-12 10:14:29 +08:00
congqixia
41af0a98fa
Use go-api/v2 for milvus-proto ( #24770 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-06-09 01:28:37 +08:00
zhenshan.cao
dc5abe086c
Revert "Restrain gc files by collection ( #22504 )" ( #23380 )
...
This reverts commit fa86de530d31840dd3dd154f4e9b3cd7c3a6662f.
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2023-04-12 19:38:28 +08:00
cai.zhang
9288020da3
Fix bug for batch delete files on gcp of minio ( #23052 ) ( #23083 ) ( #23090 )
...
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-04-09 16:22:30 +08:00
jaime
c9d0c157ec
Move some modules from internal to public package ( #22572 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-04-06 19:14:32 +08:00
