mirror of
https://gitee.com/milvus-io/milvus.git
synced 2026-01-07 19:31:51 +08:00
9d29e6ee64
issue: #45077 We need to promise the state of wal consistent with the memory state of streamingnode. So we don't allow the append operation can be cancelled by the append caller to avoid leave a inconsistent state of alive wal. The wal append operation can only be cancelled when the wal is shutting down. Signed-off-by: chyezh <chyezh@outlook.com>
WAL
wal package is the basic defination of wal interface of milvus streamingnode.
wal use github.com/milvus-io/milvus/pkg/streaming/walimpls to implement the final wal service.
Project arrangement
wal/: only define exposed interfaces./adaptor/: adaptors to implementwalinterface fromwalimplsinterface/utility/: A utility code for common logic or data structure.
github.com/milvus-io/milvus/pkg/streaming/walimpls/: define the underlying message system interfaces need to be implemented./registry/: A static lifetime registry to regsiter new implementation for inverting dependency./helper/: A utility used to help developer to implementwalimplsconveniently./impls/: A official implemented walimpls sets.
Lifetime Of Interfaces
OpenerBuilderhas a static lifetime in a programs:Openerkeep same lifetime with underlying resources (such as mq client).WALkeep same lifetime with underlying writer of wal, and it's lifetime is always included in relatedOpener.Scannerkeep same lifetime with underlying reader of wal, and it's lifetime is always included in relatedWAL.
Add New Implemetation Of WAL
developper who want to add a new implementation of wal should implements the github.com/milvus-io/milvus/pkg/streaming/walimpls package interfaces. following interfaces is required:
walimpls.OpenerBuilderImplswalimpls.OpenerImplswalimpls.ScannerImplswalimpls.WALImpls
OpenerBuilderImpls create OpenerImpls; OpenerImpls creates WALImpls; WALImpls create ScannerImpls.
Then register the implmentation of walimpls.OpenerBuilderImpls into github.com/milvus-io/milvus/pkg/streaming/walimpls/registry package.
import "github.com/milvus-io/milvus/pkg/streaming/walimpls/registry"
var _ OpenerBuilderImpls = b{};
registry.RegisterBuilder(b{})
All things have been done.
Use WAL
import "github.com/milvus-io/milvus/internal/streamingnode/server/wal/registry"
name := "your builder name"
var yourCh *options.PChannelInfo
opener, err := registry.MustGetBuilder(name).Build()
if err != nil {
panic(err)
}
ctx := context.Background()
logger, err := opener.Open(ctx, wal.OpenOption{
Channel: yourCh
})
if err != nil {
panic(err)
}
Adaptor
package adaptor is used to adapt walimpls and wal together.
common wal function should be implement by it. Such as:
- lifetime management
- interceptor implementation
- scanner wrapped up
- write ahead cache implementation