mirror of
https://gitee.com/milvus-io/milvus.git
synced 2025-12-28 22:45:26 +08:00
21b0e5ca9d
### **PR Type**
Enhancement
___
### **Description**
- Only flush and fence segments for schema-changing alter collection
messages
- Skip segment sealing for collection property-only alterations
- Add conditional check using messageutil.IsSchemaChange utility
function
___
### Diagram Walkthrough
```mermaid
flowchart LR
A["Alter Collection Message"] --> B{"Is Schema Change?"}
B -->|Yes| C["Flush and Fence Segments"]
B -->|No| D["Skip Segment Operations"]
C --> E["Set Flushed Segment IDs"]
D --> E
E --> F["Append Operation"]
```
<details><summary><h3>File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>shard_interceptor.go</strong><dd><code>Conditional
segment sealing based on schema changes</code>
</dd></summary>
<hr>
internal/streamingnode/server/wal/interceptors/shard/shard_interceptor.go
<ul><li>Added import for <code>messageutil</code> package to access
schema change detection <br>utility<br> <li> Modified
<code>handleAlterCollection</code> to conditionally flush and fence
<br>segments only for schema-changing messages<br> <li> Wrapped segment
flushing logic in <code>if
</code><br><code>messageutil.IsSchemaChange(header)</code> check<br>
<li> Skips unnecessary segment sealing when only collection properties
are <br>altered</ul>
</details>
</td>
<td><a
href="https://github.com/milvus-io/milvus/pull/46488/files#diff-c1acf785e5b530e59137b21584cf567ccd9aeeb613fb3684294b439289e80beb">+9/-3</a>
</td>
</tr>
</table></td></tr></tbody></table>
</details>
___
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Optimized collection schema alteration to conditionally perform
segment allocation operations only when schema changes are detected,
reducing unnecessary overhead in unmodified collection scenarios.
<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
WAL
wal package is the basic defination of wal interface of milvus streamingnode.
wal use github.com/milvus-io/milvus/pkg/streaming/walimpls to implement the final wal service.
Project arrangement
wal/: only define exposed interfaces./adaptor/: adaptors to implementwalinterface fromwalimplsinterface/utility/: A utility code for common logic or data structure.
github.com/milvus-io/milvus/pkg/streaming/walimpls/: define the underlying message system interfaces need to be implemented./registry/: A static lifetime registry to regsiter new implementation for inverting dependency./helper/: A utility used to help developer to implementwalimplsconveniently./impls/: A official implemented walimpls sets.
Lifetime Of Interfaces
OpenerBuilderhas a static lifetime in a programs:Openerkeep same lifetime with underlying resources (such as mq client).WALkeep same lifetime with underlying writer of wal, and it's lifetime is always included in relatedOpener.Scannerkeep same lifetime with underlying reader of wal, and it's lifetime is always included in relatedWAL.
Add New Implemetation Of WAL
developper who want to add a new implementation of wal should implements the github.com/milvus-io/milvus/pkg/streaming/walimpls package interfaces. following interfaces is required:
walimpls.OpenerBuilderImplswalimpls.OpenerImplswalimpls.ScannerImplswalimpls.WALImpls
OpenerBuilderImpls create OpenerImpls; OpenerImpls creates WALImpls; WALImpls create ScannerImpls.
Then register the implmentation of walimpls.OpenerBuilderImpls into github.com/milvus-io/milvus/pkg/streaming/walimpls/registry package.
import "github.com/milvus-io/milvus/pkg/streaming/walimpls/registry"
var _ OpenerBuilderImpls = b{};
registry.RegisterBuilder(b{})
All things have been done.
Use WAL
import "github.com/milvus-io/milvus/internal/streamingnode/server/wal/registry"
name := "your builder name"
var yourCh *options.PChannelInfo
opener, err := registry.MustGetBuilder(name).Build()
if err != nil {
panic(err)
}
ctx := context.Background()
logger, err := opener.Open(ctx, wal.OpenOption{
Channel: yourCh
})
if err != nil {
panic(err)
}
Adaptor
package adaptor is used to adapt walimpls and wal together.
common wal function should be implement by it. Such as:
- lifetime management
- interceptor implementation
- scanner wrapped up
- write ahead cache implementation