SimFG
18eb627533
fix: [2.5] Update logging context and upgrade dependencies ( #41319 )
...
- issue: #41291
- pr: #41318
---------
Signed-off-by: SimFG <bang.fu@zilliz.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2025-04-24 23:50:40 +08:00
yihao.dai
27ea5d14dc
fix: [2.5] Fix delete data loss due to duplicate binlogID ( #40976 )
...
With concurrenct L0 compaction
(https://github.com/milvus-io/milvus/pull/36816 ), delta logs might be
written to the same L1 segment, causing logID duplication when using the
incremental beginLogID. This PR removes the beginLogID mechanism and
instead passes a log ID range, where the number of IDs in the range
equals the number of compaction segment binlogs multiplied by an
expansion factor.
issue: https://github.com/milvus-io/milvus/issues/40207
pr: https://github.com/milvus-io/milvus/pull/40960
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-03-28 14:34:21 +08:00
congqixia
709594f158
enhance: [2.5] Use v2 package name for pkg module ( #40117 )
...
Cherry-pick from master
pr: #39990
Related to #39095
https://go.dev/doc/modules/version-numbers
Update pkg version according to golang dep version convention
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-02-23 00:46:01 +08:00
XuanYang-cn
afef5fed60
fix: Clustering compaction ignoring deltalogs ( #39133 )
...
See also: #39131
pr: #39132
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2025-01-10 14:07:05 +08:00
Zhen Ye
95809ca767
enhance: make new go package to manage proto ( #39128 )
...
issue: #39095
pr: #39114
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2025-01-10 10:53:01 +08:00
cai.zhang
41b19c6b1d
enhance: Determine the number of buffers based on the resource limits of the DataNode ( #38209 )
...
issue: #28410
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-12-08 18:02:40 +08:00
yihao.dai
81879425e1
enhance: Optimize the performance of stats task ( #37374 )
...
1. Increase the writer's `batchSize` to avoid multiple serialization
operations.
2. Perform asynchronous upload of binlog files to prevent blocking the
data processing flow.
3. Reduce multiple calls to `writer.Flush()`.
issue: https://github.com/milvus-io/milvus/issues/37373
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-11-08 10:08:27 +08:00
Ted Xu
262a994d6d
enhance: generally improve the performance of mix compactions ( #37163 )
...
See #37234
---------
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-10-29 18:12:20 +08:00
aoiasd
5ec4163d0f
feat: support bm25 logs mixcompaction ( #36072 )
...
relate: https://github.com/milvus-io/milvus/issues/35853
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-10-14 16:57:22 +08:00
wayblink
00a5025949
enhance: support clustering compaction on null value ( #36372 )
...
issue: #36055
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-09-30 14:33:17 +08:00
smellthemoon
3f75bf1f20
fix: clustering compact not support null ( #36152 )
...
#36055
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-09-11 14:49:06 +08:00
cai.zhang
2c9bb4dfa3
feat: Support stats task to sort segment by PK ( #35054 )
...
issue: #33744
This PR includes the following changes:
1. Added a new task type to the task scheduler in datacoord: stats task,
which sorts segments by primary key.
2. Implemented segment sorting in indexnode.
3. Added a new field `FieldStatsLog` to SegmentInfo to store token index
information.
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-09-02 14:19:03 +08:00
yihao.dai
1413ffe9b1
enhance: Rename preAllocatedSegments ( #35871 )
...
Rename `preAllocatedSegments` to `preAllocatedSegmentIDs` to avoid
confusion.
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-09-01 17:09:01 +08:00
cai.zhang
aaab827a16
fix: Fix the issue of missing stats log after clustering compaction ( #35266 )
...
issue: #35265
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-08-08 14:24:17 +08:00
yihao.dai
a4439cc911
enhance: Implement flusher in streamingNode ( #34942 )
...
- Implement flusher to:
- Manage the pipelines (creation, deletion, etc.)
- Manage the segment write buffer
- Manage sync operation (including receive flushMsg and execute flush)
- Add a new `GetChannelRecoveryInfo` RPC in DataCoord.
- Reorganize packages: `flushcommon` and `datanode`.
issue: https://github.com/milvus-io/milvus/issues/33285
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-08-02 18:30:23 +08:00
wayblink
95462668ca
enhance: unify time in clustering compaction task to unix ( #35167 )
...
#34495
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-08-02 10:30:19 +08:00
zhenshan.cao
aa247f192d
enhance: remove unused code for StorageV2 ( #35132 )
...
issue: https://github.com/milvus-io/milvus/issues/34168
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-08-01 12:08:13 +08:00
wayblink
d294fdd060
enhance: add ut for clustering_compactor ( #34852 )
...
issue: #34792
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-21 19:29:42 +08:00
yihao.dai
ca758c36cc
enhance: Pre-allocate ids for compaction ( #34187 )
...
This PR removes the dependency of compaction on the ID allocator by
pre-allocating the logID and segmentID.
issue: https://github.com/milvus-io/milvus/issues/33957
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-17 13:23:42 +08:00
wayblink
a1232fafda
feat: Major compaction ( #33620 )
...
#30633
Signed-off-by: wayblink <anyang.wang@zilliz.com>
Co-authored-by: MrPresent-Han <chun.han@zilliz.com>
2024-06-10 21:34:08 +08:00
