mirror of
https://gitee.com/milvus-io/milvus.git
synced 2026-01-07 19:31:51 +08:00
ca8740c7c0
issue: #45841 - CPP log make the multi log line in one debug, remove the "\n\t". - remove some log that make no sense. - slow down some log like ChannelDistManager. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> - Core invariant: logging is purely observational — this PR only reduces, consolidates, or reformats diagnostic output (removing per-item/noise logs, consolidating batched logs, and converting multi-line log strings) while preserving all control flow, return values, and state mutations across affected code paths. - Removed / simplified logic: deleted low-value per-operation debug/info logs (e.g., ListIndexes, GetRecoveryInfo, GcConfirm, push-to-reorder-buffer, several streaming/wal/debug traces), replaced per-item inline logs with single batched deferred logs in querynodev2/delegator (logExcludeInfo) and CleanInvalid, changed C++ PlanNode ToString() multi-line output to compact single-line bracketed format (removed "\n\t"), and added thresholded interceptor logging (InterceptorMetrics.ShouldBeLogged) and message-type-driven log levels to avoid verbose entries. - Why this does NOT cause data loss or behavioral regression: no function signatures, branching, state updates, persistence calls, or return values were changed — examples: ListIndexes still returns the same Status/IndexInfos; GcConfirm still constructs and returns resp.GetGcFinished(); Insert and CleanInvalid still perform the same insert/removal operations (only their per-item logging was aggregated); PlanNode ToString changes only affect emitted debug strings. All error handling and control flow paths remain intact. - Enhancement intent: reduce log volume and improve signal-to-noise for debugging by removing redundant, noisy logs and emitting concise, rate-/threshold-limited summaries while preserving necessary diagnostics and original program behavior. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: chyezh <chyezh@outlook.com>
WAL
wal package is the basic defination of wal interface of milvus streamingnode.
wal use github.com/milvus-io/milvus/pkg/streaming/walimpls to implement the final wal service.
Project arrangement
wal/: only define exposed interfaces./adaptor/: adaptors to implementwalinterface fromwalimplsinterface/utility/: A utility code for common logic or data structure.
github.com/milvus-io/milvus/pkg/streaming/walimpls/: define the underlying message system interfaces need to be implemented./registry/: A static lifetime registry to regsiter new implementation for inverting dependency./helper/: A utility used to help developer to implementwalimplsconveniently./impls/: A official implemented walimpls sets.
Lifetime Of Interfaces
OpenerBuilderhas a static lifetime in a programs:Openerkeep same lifetime with underlying resources (such as mq client).WALkeep same lifetime with underlying writer of wal, and it's lifetime is always included in relatedOpener.Scannerkeep same lifetime with underlying reader of wal, and it's lifetime is always included in relatedWAL.
Add New Implemetation Of WAL
developper who want to add a new implementation of wal should implements the github.com/milvus-io/milvus/pkg/streaming/walimpls package interfaces. following interfaces is required:
walimpls.OpenerBuilderImplswalimpls.OpenerImplswalimpls.ScannerImplswalimpls.WALImpls
OpenerBuilderImpls create OpenerImpls; OpenerImpls creates WALImpls; WALImpls create ScannerImpls.
Then register the implmentation of walimpls.OpenerBuilderImpls into github.com/milvus-io/milvus/pkg/streaming/walimpls/registry package.
import "github.com/milvus-io/milvus/pkg/streaming/walimpls/registry"
var _ OpenerBuilderImpls = b{};
registry.RegisterBuilder(b{})
All things have been done.
Use WAL
import "github.com/milvus-io/milvus/internal/streamingnode/server/wal/registry"
name := "your builder name"
var yourCh *options.PChannelInfo
opener, err := registry.MustGetBuilder(name).Build()
if err != nil {
panic(err)
}
ctx := context.Background()
logger, err := opener.Open(ctx, wal.OpenOption{
Channel: yourCh
})
if err != nil {
panic(err)
}
Adaptor
package adaptor is used to adapt walimpls and wal together.
common wal function should be implement by it. Such as:
- lifetime management
- interceptor implementation
- scanner wrapped up
- write ahead cache implementation