admin/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2026-01-07 19:31:51 +08:00
Code Issues Packages Projects Releases Wiki Activity
23,822 Commits 137 Branches 188 Tags
Commit Graph

7 Commits

Author SHA1 Message Date
XuanYang-cn
146f173eea
enhance: tidy code in import plugin context (#46374) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2026-01-04 16:51:23 +08:00
cai.zhang
8d12bfb436
fix: Restore the compaction task correctly to ensure it can be properly cleaned up (#46577) 
issue: #46576 

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
- Core invariant: During meta load, only tasks that are truly
terminal-cleaned (states cleaned or unknown) should be dropped; all
other non-terminal tasks (including timeout and completed) must be
restored so the inspector can reattach them to executing/cleaning queues
and finish their cleanup lifecycle.
- Removed/simplified logic: loadMeta no longer uses the broad
isCompactionTaskFinished predicate (which treated timeout, completed,
cleaned, unknown as terminal). It now uses the new
isCompactionTaskCleaned predicate that only treats cleaned/unknown as
terminal. This removes the redundant exclusion of timeout/completed
tasks and simplifies the guard to drop only cleaned/unknown tasks.
- Bug fix (root cause & exact change): Fixes issue #46576 — the previous
isCompactionTaskFinished caused timeout/completed tasks to be skipped
during meta load and thus not passed into restoreTask(). The PR adds
isCompactionTaskCleaned and replaces the finished check so timeout and
completed tasks are included in restoreTask() and re-attached to the
inspector’s existing executing/cleaning queues.
- No data loss or regression: Tasks in cleaned/unknown remain dropped
(isCompactionTaskCleaned still returns true for cleaned/unknown).
Non-terminal timeout/completed tasks now follow the same restoreTask()
control path used previously for restored tasks — they are enqueued into
the inspector’s queue/executing/cleaning flows rather than being
discarded. No exported signatures changed and all restored tasks flow
into existing handlers, avoiding behavior regression or data loss.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2025-12-26 18:57:19 +08:00
cai.zhang
0943713481
fix: Skip Finished tasks when recovery with compatibility (#46515) 
### **User description**
issue: #46466


___

### **PR Type**
Bug fix


___

### **Description**
- Extract finished task state check into reusable helper function

- Skip finished tasks during compaction recovery to prevent reprocessing

- Add backward compatibility check for pre-allocated segment IDs


___

### Diagram Walkthrough


```mermaid
flowchart LR
  A["Compaction Task States"] -->|"Check with helper"| B["isCompactionTaskFinished()"]
  B -->|"Used in"| C["compactionInspector.loadMeta()"]
  B -->|"Used in"| D["compactionTaskMeta.reloadFromKV()"]
  C -->|"Skip finished tasks"| E["Recovery Process"]
  D -->|"Backward compatibility"| E
```



<details><summary><h3>File Walkthrough</h3></summary>

<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><table>
<tr>
  <td>
    <details>
<summary><strong>compaction_util.go</strong><dd><code>Add
isCompactionTaskFinished helper function</code>&nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
</dd></summary>
<hr>

internal/datacoord/compaction_util.go

<ul><li>Added new helper function
<code>isCompactionTaskFinished()</code> to check if a <br>compaction
task is in a terminal state<br> <li> Function checks for failed,
timeout, completed, cleaned, or unknown <br>states<br> <li> Centralizes
task state validation logic for reuse across multiple
<br>components</ul>


</details>


  </td>
<td><a
href="https://github.com/milvus-io/milvus/pull/46515/files#diff-8f2cb8d0fef37617202c5a2290ad2bdbf2df5b5983604b5b505bc73a65c7eb43">+8/-0</a>&nbsp;
&nbsp; &nbsp; </td>

</tr>
</table></td></tr><tr><td><strong>Bug fix</strong></td><td><table>
<tr>
  <td>
    <details>
<summary><strong>compaction_inspector.go</strong><dd><code>Refactor to
use finished task helper function</code>&nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></summary>
<hr>

internal/datacoord/compaction_inspector.go

<ul><li>Replaced inline state checks with call to
<code>isCompactionTaskFinished()</code> <br>helper<br> <li> Simplifies
code by removing repetitive state comparison logic<br> <li> Maintains
same behavior of skipping finished tasks during recovery</ul>


</details>


  </td>
<td><a
href="https://github.com/milvus-io/milvus/pull/46515/files#diff-1c884001f2e84de177fea22b584f3de70a6e73695dbffa34031be9890d17da6d">+1/-5</a>&nbsp;
&nbsp; &nbsp; </td>

</tr>

<tr>
  <td>
    <details>
<summary><strong>compaction_task_meta.go</strong><dd><code>Add finished
task check for backward compatibility</code>&nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; </dd></summary>
<hr>

internal/datacoord/compaction_task_meta.go

<ul><li>Added check to skip finished tasks before processing
pre-allocated <br>segment IDs<br> <li> Ensures backward compatibility
for tasks without pre-allocated segment <br>IDs<br> <li> Prevents
marking already-finished tasks as failed during reload</ul>


</details>


  </td>
<td><a
href="https://github.com/milvus-io/milvus/pull/46515/files#diff-0dae7214c4c79ddf5106bd51d375b5fb2f41239d5d433798afa90708e443eca8">+1/-1</a>&nbsp;
&nbsp; &nbsp; </td>

</tr>
</table></td></tr></tbody></table>

</details>

___



<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Improved detection of finished compaction tasks to reduce false
failures.
* Prevented finished tasks with missing pre-allocations from being
incorrectly marked as failed.
* Simplified abandonment logic for completed/timeout/cleaned tasks to
reduce erroneous retries and noisy logs.

<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2025-12-23 18:09:18 +08:00
XuanYang-cn
0bbb134e39
feat: Enable to backup and reload ez (#46332) 
see also: #40013

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2025-12-16 17:19:16 +08:00
congqixia
ba88cfa7a9
enhance: Add unified GRPC latency metrics in inteceptor (#44089) 
Related to #43966

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2025-08-28 09:53:51 +08:00
XuanYang-cn
37a447d166
feat: Add CMEK cipher plugin (#43722) 
1. Enable Milvus to read cipher configs
2. Enable cipher plugin in binlog reader and writer
3. Add a testCipher for unittests
4. Support pooling for datanode
5. Add encryption in storagev2

See also: #40321 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2025-08-27 11:15:52 +08:00
yihao.dai
5b78ef0a49
fix: Fix delete data loss due to duplicate binlogID (#40960) 
With concurrenct L0 compaction
(https://github.com/milvus-io/milvus/pull/36816), delta logs might be
written to the same L1 segment, causing logID duplication when using the
incremental beginLogID. This PR removes the beginLogID mechanism and
instead passes a log ID range, where the number of IDs in the range
equals the number of compaction segment binlogs multiplied by an
expansion factor.

issue: https://github.com/milvus-io/milvus/issues/40207

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2025-04-01 10:36:22 +08:00