5 Commits

Author SHA1 Message Date
Zhen Ye
7d6d279e9c
fix: set enable.auto.commit false to prevent from creating kafka consumer group (#46508)
### **User description**
issue: #46507

we use the assign/unassign api to manage the consumer manually, the
commit operation will generate a new consumer group which is not what we
want. so we disable the auto commit to avoid it, also see:
https://github.com/confluentinc/confluent-kafka-python/issues/250#issuecomment-331377925


___

### **PR Type**
Bug fix


___

### **Description**
- Disable auto-commit in Kafka consumer configuration

- Prevents unwanted consumer group creation from manual offset
management

- Clarifies offset reset behavior with explanatory comments


___

### Diagram Walkthrough


```mermaid
flowchart LR
  A["Kafka Consumer Config"] --> B["Set enable.auto.commit to false"]
  B --> C["Prevent auto consumer group creation"]
  A --> D["Set auto.offset.reset to earliest"]
  D --> E["Handle deleted offsets gracefully"]
```



<details><summary><h3>File Walkthrough</h3></summary>

<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Bug
fix</strong></td><td><table>
<tr>
  <td>
    <details>
<summary><strong>builder.go</strong><dd><code>Disable auto-commit and
add configuration comments</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; </dd></summary>
<hr>

pkg/streaming/walimpls/impls/kafka/builder.go

<ul><li>Added <code>enable.auto.commit</code> configuration set to
<code>false</code> to prevent <br>automatic consumer group creation<br>
<li> Added explanatory comments for both <code>auto.offset.reset</code>
and <br><code>enable.auto.commit</code> settings<br> <li> Clarifies that
manual assign/unassign API is used for consumer <br>management</ul>


</details>


  </td>
<td><a
href="https://github.com/milvus-io/milvus/pull/46508/files#diff-4b5635821fdc8b585d16c02d8a3b59079d8e667b2be43a073265112d72701add">+7/-0</a>&nbsp;
&nbsp; &nbsp; </td>

</tr>
</table></td></tr></tbody></table>

</details>

___



<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

## Bug Fixes
* Kafka consumer now reads from the earliest available messages and
auto-commit has been disabled to support manual offset management.

<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Signed-off-by: chyezh <chyezh@outlook.com>
2025-12-22 21:07:18 +08:00
Zhen Ye
31a609c21d
fix: kafka should auto reset the offset from earliest to read (#45237)
issue: #44172, #45210, #44851

kafka will auto reset the offset to "latest" if the offset is
Out-of-range. the recovery of milvus wal cannot read any message from
that. So once the offset is out-of-range, kafka should read from eariest
to read the latest uncleared data.


https://kafka.apache.org/documentation/#consumerconfigs_auto.offset.reset

Signed-off-by: chyezh <chyezh@outlook.com>
2025-11-03 21:07:33 +08:00
yihao.dai
51f69f32d0
feat: Add CDC support (#44124)
This PR implements a new CDC service for Milvus 2.6, providing log-based
cross-cluster replication.

issue: https://github.com/milvus-io/milvus/issues/44123

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Signed-off-by: chyezh <chyezh@outlook.com>
Co-authored-by: chyezh <chyezh@outlook.com>
2025-09-16 16:32:01 +08:00
congqixia
cb7f2fa6fd
enhance: Use v2 package name for pkg module (#39990)
Related to #39095

https://go.dev/doc/modules/version-numbers

Update pkg version according to golang dep version convention

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-02-22 23:15:58 +08:00
Zhen Ye
285289d5b0
enhance: implement kafka for wal (#38598)
issue: #38399

---------

Signed-off-by: chyezh <chyezh@outlook.com>
2024-12-25 10:22:50 +08:00