issue: #43897
- UpdateReplicateConfig operation will broadcast AlterReplicateConfig
message into all pchannels with cluster-exclusive-lock.
- Begin txn message will use commit message timetick now (to avoid
timetick rollback when CDC with txn message).
- If current cluster is secondary, the UpdateReplicateConfig will wait
until the replicate configuration is consistent with the config
replicated from primary.
---------
Signed-off-by: chyezh <chyezh@outlook.com>
issue: #44800
This commit enhances the upsert and validation logic to properly handle
nullable Geometry (WKT/WKB) and Timestamptz data types:
- Add ToCompressedFormatNullable support for TimestamptzData,
GeometryWktData, and GeometryData to filter out null values during data
compression
- Implement GenNullableFieldData for Timestamptz and Geometry types to
generate nullable field data structures
- Update FillWithNullValue to handle both GeometryData and
GeometryWktData with null value filling logic
- Add UpdateFieldData support for Timestamptz, GeometryData, and
GeometryWktData field updates
- Comprehensive unit tests covering all new data type handling scenarios
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Make CDC watch etcd replicate pchannel meta instead of listing them
periodically.
issue: https://github.com/milvus-io/milvus/issues/44123
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
This commit addresses issue #44788 where the
`datacoord_stored_binlog_size` metric could become inaccurate when
multiple concurrent `GetMetrics` calls arrived at DataCoord.
### Problem
The original implementation called `Reset()` followed by `Add()`
operations on Prometheus metrics within the `GetQuotaInfo()` method.
When multiple goroutines invoked this method concurrently, race
conditions occurred:
- Thread 1: Reset() → Add(value1)
- Thread 2: Reset() → Add(value2)
- Result: Metrics could be reset multiple times and values added in an
interleaved fashion, leading to inaccurate and inflated metric values
### Solution
Changed the approach from `Reset() + Add()` to aggregating metric values
in local maps first, then using `Set()` to update metrics atomically:
1. Collect segment size data into local maps:
- `storedBinlogSize`: tracks size per collection per segment state
- `binlogFileSize`: tracks total file count per collection
- `coll2DbName`: maps collection IDs to database names
2. After aggregation is complete, use `Set()` (instead of `Add()`) to
update metrics in a single operation per label combination
This ensures that concurrent `GetMetrics` calls don't interfere with
each other, as each invocation works with its own local state and only
updates the final metric value atomically.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #44014
- On standalone, the query node inside need to load segment and watch
channel, so the querynode is not a embeded querynode in streamingnode
without `LabelStreamingNodeEmbeddedQueryNode`. The channel dist manager
can not confirm a standalone node is a embededStreamingNode.
Bug is introduced by #44099
Signed-off-by: chyezh <chyezh@outlook.com>
Related to #44761
This commit refactors the privilege management system in the proxy
component by:
1. **Separation of Concerns**: Extracts privilege-related functionality
from MetaCache into a dedicated `internal/proxy/privilege` package,
improving code organization and maintainability.
2. **New Package Structure**: Creates `internal/proxy/privilege/` with:
- `cache.go`: Core privilege cache implementation (PrivilegeCache)
- `result_cache.go`: Privilege enforcement result caching
- `model.go`: Casbin model and policy enforcement functions
- `meta_cache_adapter.go`: Casbin adapter for MetaCache integration
- Corresponding test files and mock implementations
3. **MetaCache Simplification**: Removes privilege and credential
management methods from MetaCache interface and implementation:
- Removed: GetCredentialInfo, RemoveCredential, UpdateCredential
- Removed: GetPrivilegeInfo, GetUserRole, RefreshPolicyInfo,
InitPolicyInfo
- Deleted: meta_cache_adapter.go, privilege_cache.go and their tests
4. **Updated References**: Updates all callsites to use the new
privilegeCache global:
- Authentication interceptor now uses privilegeCache for password
verification
- Credential cache operations (InvalidateCredentialCache,
UpdateCredentialCache, UpdateCredential) now use privilegeCache
- Policy refresh operations (RefreshPolicyInfoCache) now use
privilegeCache
- Privilege interceptor uses new privilege.GetEnforcer() and privilege
result cache
5. **Improved API**: Renames cache functions for clarity:
- GetPrivilegeCache → GetResultCache
- SetPrivilegeCache → SetResultCache
- CleanPrivilegeCache → CleanResultCache
This refactoring makes the codebase more modular, separates privilege
management concerns from general metadata caching, and provides a
clearer API for privilege enforcement operations.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #44558, #44123
- Update config(A->C) to A and C, config(B) to B on replicate topology
(A->B,A->C) can remove the B from replicate topology
- Fix some metric error of CDC
Signed-off-by: chyezh <chyezh@outlook.com>
Related to #36672
Add accesslog field displaying value length for search/query request may
help developers debug related issues
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Remove stopTask.
Replace multiple task tracking maps with single unified taskState map.
Fix slot tracking, improve state transitions, and add comprehensive test
See also: #44714
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
relate: https://github.com/milvus-io/milvus/issues/44758
Wrong code which should be `(result.seg_offsets_[i] >= 0 &&
result.seg_offsets_[j] < 0)`, but was `(result.seg_offsets_[j] >= 0 &&
result.seg_offsets_[j] < 0) ` now.
But because all placeholder which was offset -1, will fill with worst
distance value.
For IP, L2 or COSIN, it will be +inf or -inf. So sort distance was
enough.
But when use BM25, it will be NAN. Will case sort out of ordered.
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
issue: #44730
Fix the issue where logs were not outputting as expected due to
incorrect log package imports across multiple components.
Changes include:
- Add golangci-lint rule to forbid github.com/pingcap/log usage
- Replace github.com/pingcap/log with
github.com/milvus-io/milvus/pkg/v2/log
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Related to #44620
This PR:
- Use eventually instead of `time.Sleep` in accesslog writer unit test
- Make sure compaction task results have only one state from executor
API
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Related to #44703
This PR:
- Add `SetActualConsistencyLevel` to `info.AccessInfo` interface and
related util method processing it
- Make `$consistency_level` returning actual value if set
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Related to #44647
This patch make milvus-storage using singleton credential provider in
case of data race when concurrent index build task recieved.
See also milvus-io/milvus-storage#44647
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #41611
- After enabling streaming arch, channel manager of data coord is a
redundant component.
---------
Signed-off-by: chyezh <chyezh@outlook.com>
Related to #44535
This PR:
- Fix the unittest creating `DiskFileManagerImpl` without `filesystem`
- Add comments for methods need `fs_`
- Remove fallback creation and add assertion for nullptr fs
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
The collection not found err could contains db id in err message, which
is not meaningful to users.
This patch make error message wrapping dbname instead of db id.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Related to #44620
Make unittest stable by waiting eventch instead of async read
len(eventCh), which is unstable
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
#44212
Also, record metrics only when storageUsageTracking is enabled.
Use MB for scanned_remote counter and scanned_total counter metrics to
avoid overflow.
---------
Signed-off-by: chasingegg <chao.gao@zilliz.com>
