cai.zhang
0943713481
fix: Skip Finished tasks when recovery with compatibility ( #46515 )
...
### **User description**
issue: #46466
___
### **PR Type**
Bug fix
___
### **Description**
- Extract finished task state check into reusable helper function
- Skip finished tasks during compaction recovery to prevent reprocessing
- Add backward compatibility check for pre-allocated segment IDs
___
### Diagram Walkthrough
```mermaid
flowchart LR
A["Compaction Task States"] -->|"Check with helper"| B["isCompactionTaskFinished()"]
B -->|"Used in"| C["compactionInspector.loadMeta()"]
B -->|"Used in"| D["compactionTaskMeta.reloadFromKV()"]
C -->|"Skip finished tasks"| E["Recovery Process"]
D -->|"Backward compatibility"| E
```
<details><summary><h3>File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>compaction_util.go</strong><dd><code>Add
isCompactionTaskFinished helper function</code>
</dd></summary>
<hr>
internal/datacoord/compaction_util.go
<ul><li>Added new helper function
<code>isCompactionTaskFinished()</code> to check if a <br>compaction
task is in a terminal state<br> <li> Function checks for failed,
timeout, completed, cleaned, or unknown <br>states<br> <li> Centralizes
task state validation logic for reuse across multiple
<br>components</ul>
</details>
</td>
<td><a
href="https://github.com/milvus-io/milvus/pull/46515/files#diff-8f2cb8d0fef37617202c5a2290ad2bdbf2df5b5983604b5b505bc73a65c7eb43 ">+8/-0</a>
</td>
</tr>
</table></td></tr><tr><td><strong>Bug fix</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>compaction_inspector.go</strong><dd><code>Refactor to
use finished task helper function</code>
</dd></summary>
<hr>
internal/datacoord/compaction_inspector.go
<ul><li>Replaced inline state checks with call to
<code>isCompactionTaskFinished()</code> <br>helper<br> <li> Simplifies
code by removing repetitive state comparison logic<br> <li> Maintains
same behavior of skipping finished tasks during recovery</ul>
</details>
</td>
<td><a
href="https://github.com/milvus-io/milvus/pull/46515/files#diff-1c884001f2e84de177fea22b584f3de70a6e73695dbffa34031be9890d17da6d ">+1/-5</a>
</td>
</tr>
<tr>
<td>
<details>
<summary><strong>compaction_task_meta.go</strong><dd><code>Add finished
task check for backward compatibility</code>
</dd></summary>
<hr>
internal/datacoord/compaction_task_meta.go
<ul><li>Added check to skip finished tasks before processing
pre-allocated <br>segment IDs<br> <li> Ensures backward compatibility
for tasks without pre-allocated segment <br>IDs<br> <li> Prevents
marking already-finished tasks as failed during reload</ul>
</details>
</td>
<td><a
href="https://github.com/milvus-io/milvus/pull/46515/files#diff-0dae7214c4c79ddf5106bd51d375b5fb2f41239d5d433798afa90708e443eca8 ">+1/-1</a>
</td>
</tr>
</table></td></tr></tbody></table>
</details>
___
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Improved detection of finished compaction tasks to reduce false
failures.
* Prevented finished tasks with missing pre-allocations from being
incorrectly marked as failed.
* Simplified abandonment logic for completed/timeout/cleaned tasks to
reduce erroneous retries and noisy logs.
<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2025-12-23 18:09:18 +08:00
XuanYang-cn
0bbb134e39
feat: Enable to backup and reload ez ( #46332 )
...
see also: #40013
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2025-12-16 17:19:16 +08:00
congqixia
ba88cfa7a9
enhance: Add unified GRPC latency metrics in inteceptor ( #44089 )
...
Related to #43966
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-08-28 09:53:51 +08:00
XuanYang-cn
37a447d166
feat: Add CMEK cipher plugin ( #43722 )
...
1. Enable Milvus to read cipher configs
2. Enable cipher plugin in binlog reader and writer
3. Add a testCipher for unittests
4. Support pooling for datanode
5. Add encryption in storagev2
See also: #40321
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2025-08-27 11:15:52 +08:00
yihao.dai
5b78ef0a49
fix: Fix delete data loss due to duplicate binlogID ( #40960 )
...
With concurrenct L0 compaction
(https://github.com/milvus-io/milvus/pull/36816 ), delta logs might be
written to the same L1 segment, causing logID duplication when using the
incremental beginLogID. This PR removes the beginLogID mechanism and
instead passes a log ID range, where the number of IDs in the range
equals the number of compaction segment binlogs multiplied by an
expansion factor.
issue: https://github.com/milvus-io/milvus/issues/40207
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-04-01 10:36:22 +08:00
