cai.zhang
0943713481
fix: Skip Finished tasks when recovery with compatibility ( #46515 )
...
### **User description**
issue: #46466
___
### **PR Type**
Bug fix
___
### **Description**
- Extract finished task state check into reusable helper function
- Skip finished tasks during compaction recovery to prevent reprocessing
- Add backward compatibility check for pre-allocated segment IDs
___
### Diagram Walkthrough
```mermaid
flowchart LR
A["Compaction Task States"] -->|"Check with helper"| B["isCompactionTaskFinished()"]
B -->|"Used in"| C["compactionInspector.loadMeta()"]
B -->|"Used in"| D["compactionTaskMeta.reloadFromKV()"]
C -->|"Skip finished tasks"| E["Recovery Process"]
D -->|"Backward compatibility"| E
```
<details><summary><h3>File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>compaction_util.go</strong><dd><code>Add
isCompactionTaskFinished helper function</code>
</dd></summary>
<hr>
internal/datacoord/compaction_util.go
<ul><li>Added new helper function
<code>isCompactionTaskFinished()</code> to check if a <br>compaction
task is in a terminal state<br> <li> Function checks for failed,
timeout, completed, cleaned, or unknown <br>states<br> <li> Centralizes
task state validation logic for reuse across multiple
<br>components</ul>
</details>
</td>
<td><a
href="https://github.com/milvus-io/milvus/pull/46515/files#diff-8f2cb8d0fef37617202c5a2290ad2bdbf2df5b5983604b5b505bc73a65c7eb43 ">+8/-0</a>
</td>
</tr>
</table></td></tr><tr><td><strong>Bug fix</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>compaction_inspector.go</strong><dd><code>Refactor to
use finished task helper function</code>
</dd></summary>
<hr>
internal/datacoord/compaction_inspector.go
<ul><li>Replaced inline state checks with call to
<code>isCompactionTaskFinished()</code> <br>helper<br> <li> Simplifies
code by removing repetitive state comparison logic<br> <li> Maintains
same behavior of skipping finished tasks during recovery</ul>
</details>
</td>
<td><a
href="https://github.com/milvus-io/milvus/pull/46515/files#diff-1c884001f2e84de177fea22b584f3de70a6e73695dbffa34031be9890d17da6d ">+1/-5</a>
</td>
</tr>
<tr>
<td>
<details>
<summary><strong>compaction_task_meta.go</strong><dd><code>Add finished
task check for backward compatibility</code>
</dd></summary>
<hr>
internal/datacoord/compaction_task_meta.go
<ul><li>Added check to skip finished tasks before processing
pre-allocated <br>segment IDs<br> <li> Ensures backward compatibility
for tasks without pre-allocated segment <br>IDs<br> <li> Prevents
marking already-finished tasks as failed during reload</ul>
</details>
</td>
<td><a
href="https://github.com/milvus-io/milvus/pull/46515/files#diff-0dae7214c4c79ddf5106bd51d375b5fb2f41239d5d433798afa90708e443eca8 ">+1/-1</a>
</td>
</tr>
</table></td></tr></tbody></table>
</details>
___
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Improved detection of finished compaction tasks to reduce false
failures.
* Prevented finished tasks with missing pre-allocations from being
incorrectly marked as failed.
* Simplified abandonment logic for completed/timeout/cleaned tasks to
reduce erroneous retries and noisy logs.
<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2025-12-23 18:09:18 +08:00
yihao.dai
c15290c125
fix: Fix invalid pre-allocated segment IDs ( #44350 )
...
Mark the task as failed when PreAllocatedSegmentIDs is nil.
issue: https://github.com/milvus-io/milvus/issues/44349
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-09-15 10:27:58 +08:00
congqixia
cb7f2fa6fd
enhance: Use v2 package name for pkg module ( #39990 )
...
Related to #39095
https://go.dev/doc/modules/version-numbers
Update pkg version according to golang dep version convention
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-02-22 23:15:58 +08:00
jaime
8a4ac8cccd
enhance: expose more metrics data ( #39456 )
...
issue: #36621 #39417
1. Adjust the server-side cache size.
2. Add source information for configurations.
3. Add node ID for compaction and indexing tasks.
4. Resolve localhost access issues to fix health check failures for
etcd.
Signed-off-by: jaime <yun.zhang@zilliz.com>
2025-02-07 11:50:50 +08:00
Zhen Ye
bb8d1ab3bf
enhance: make new go package to manage proto ( #39114 )
...
issue: #39095
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2025-01-10 10:49:01 +08:00
jaime
f03a85725a
enhance: add db name in replica ( #38672 )
...
issue: #36621
Signed-off-by: jaime <yun.zhang@zilliz.com>
2025-01-09 19:40:59 +08:00
tinswzy
1dbb6cd7cb
enhance: refine the datacoord meta related interfaces ( #37957 )
...
issue: #35917
This PR refines the meta-related APIs in datacoord to allow the ctx to
be passed down to the catalog operation interfaces
Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
2024-11-26 19:46:34 +08:00
congqixia
b0bd290a6e
enhance: Use internal json(sonic) to replace std json lib ( #37708 )
...
Related to #35020
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-11-18 10:46:31 +08:00
jaime
1d06d4324b
fix: Int64 overflow in JSON encoding ( #37657 )
...
issue: ##36621
- For simple types in a struct, add "string" to the JSON tag for
automatic string conversion during JSON encoding.
- For complex types in a struct, replace "int64" with "string."
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-11-14 22:52:30 +08:00
jaime
1e8ea4a7e7
feat: add segment/channel/task/slow query render ( #37561 )
...
issue: #36621
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-11-12 17:44:29 +08:00
jaime
9d16b972ea
feat: add tasks page into management WebUI ( #37002 )
...
issue: #36621
1. Add API to access task runtime metrics, including:
- build index task
- compaction task
- import task
- balance (including load/release of segments/channels and some leader
tasks on querycoord)
- sync task
2. Add a debug model to the webpage by using debug=true or debug=false
in the URL query parameters to enable or disable debug mode.
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-10-28 10:13:29 +08:00
yihao.dai
d230b91bd1
enhance: Add PreallocatedSegmentIDs for the compaction task ( #36734 )
...
Add `PreallocatedSegmentIDs` field to the compaction task, allowing the
`ResultSegments` in the compaction task to represent the final segments
produced by the compaction.
issue: https://github.com/milvus-io/milvus/issues/36733
also related: https://github.com/milvus-io/milvus/issues/36686
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-10-13 17:59:21 +08:00
wei liu
c45f38aa61
enhance: Update protobuf-go to protobuf-go v2 ( #34394 )
...
issue: #34252
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-29 11:31:51 +08:00
zhenshan.cao
ac4f3997ce
enhance: Reconstructing Compaction to possess persistence capability ( #33265 )
...
issue #33586
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-06-05 10:17:50 +08:00
