Related to #44620
The test was flaky because pendingMessages.Len() assertion happened
before async message processing completed. Changed to assert.Eventually
to wait up to 1 second for the pending queue to drain, fixing the race
condition where actual was 1 but expected was 0.
Fixes TestReplicateStreamClient_Reconnect flakiness.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
This pull request updates dependencies for Milvus deployments, focusing
on the images used for etcd and Kafka in both Helm and Docker Compose
configurations. The main goal is to align the deployment with newer or
organization-specific images for improved compatibility and maintenance.
**Helm deployment image updates:**
* Switched the etcd image repository from `bitnami/etcd` to
`milvusdb/etcd` and updated the etcd image tag from
`3.5.0-debian-10-r24` to `3.5.5-r2` in both standalone and cluster Helm
installation commands in `DeployTest.groovy`.
**Docker Compose image updates:**
* Changed the Kafka image from `bitnami/kafka:3.1.0` to
`bitnamilegacy/kafka:3.1.0` in `docker-compose.yml`.
issue: #44978
Signed-off-by: AlintaLu <yiqing.lu@zilliz.com>
Issue: #44518
On branch feature/timestamps
Changes to be committed:
modified: common/common_func.py
new file: milvus_client/test_milvus_client_timestamptz.py
---------
Signed-off-by: Eric Hou <eric.hou@zilliz.com>
Co-authored-by: Eric Hou <eric.hou@zilliz.com>
1. Import in replicating cluster is not supported yet, so disable it for
now.
2. Remove GetReplicateConfiguration wal API
issue: https://github.com/milvus-io/milvus/issues/44123
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
issue: #43897
- Alias related DDL is implemented by WAL-based DDL framework now.
- Support following message type in wal AlterAlias, DropAlias.
- Alias DDL can be synced by new CDC now.
- Refactor some UT for Alias DDL.
Signed-off-by: chyezh <chyezh@outlook.com>
Related to #44897
Add missing JSON data type handling in CreateArrowScalarFromDefaultValue
to fix query failures when dynamic fields are enabled. JSON default
values are now properly converted to arrow::BinaryScalar using
bytes_data().
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #44648
If the value is `null` during insertion, it will be omitted instead of
being filled with nil. Therefore, when performing checks, there’s no
need to retrieve data based on the valid offset.
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
issue: #44802
After a Geometry object is serialized into WKB, the resulting binary may
contain '\0' bytes.
When growing mmap is enabled, the append data logic uses strcpy, which
stops copying at the first '\0' bytes.
This causes only part of the WKB---typically the portion up to the
geometry type field to be copied, leading to corrupted data.
As a result, during parsing, all POINT geometries are incorrectly
interperted as POINT(0 0).
To fix this issue, memcpy will be used instead of strcpy.
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
issue: #43897
- Database related DDL is implemented by WAL-based DDL framework now.
- Support following message type in wal CreateDatabase, AlterDatabase,
DropDatabase.
- Database DDL can be synced by new CDC now.
- Refactor some UT for Database DDL.
Signed-off-by: chyezh <chyezh@outlook.com>
issue: #43897
- RBAC(Roles/Users/Privileges/Privilege Groups) is implemented by
WAL-based DDL framework now.
- Support following message type in wal `AlterUser`, `DropUser`,
`AlterRole`, `DropRole`, `AlterUserRole`, `DropUserRole`,
`AlterPrivilege`, `DropPrivilege`, `AlterPrivilegeGroup`,
`DropPrivilegeGroup`, `RestoreRBAC`.
- RBAC can be synced by new CDC now.
- Refactor some UT for RBAC.
---------
Signed-off-by: chyezh <chyezh@outlook.com>
Related to #44819
This fix addresses an issue(#44819) where the offset parameter did not
work correctly during searches when multiple results had identical
scores. The problem occurred because results with equal scores were not
consistently ordered, leading to unpredictable pagination behavior.
The solution adds a new sorting step (SortEqualScoresByPks) in the
reduce phase that sorts results with identical scores by their primary
keys in ascending order. This ensures deterministic ordering and enables
proper offset functionality.
Changes:
- Add SortEqualScoresByPks() to sort results with equal scores by PK
- Add SortEqualScoresOneNQ() to handle per-query sorting logic
- Invoke sorting step after FillPrimaryKey() in Reduce() workflow
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #43858
Fix the issue introduced in PR #43992 where deactivating the balance
checker incorrectly stops stopping balance operations.
Changes:
- Move IsActive() check after stopping balance logic
- Only skip normal balance when checker is inactive
- Allow stopping balance to proceed regardless of checker state
This ensures stopping balance can execute even when the balance checker
is deactivated.
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
issue: #43897
- UpdateReplicateConfig operation will broadcast AlterReplicateConfig
message into all pchannels with cluster-exclusive-lock.
- Begin txn message will use commit message timetick now (to avoid
timetick rollback when CDC with txn message).
- If current cluster is secondary, the UpdateReplicateConfig will wait
until the replicate configuration is consistent with the config
replicated from primary.
---------
Signed-off-by: chyezh <chyezh@outlook.com>
issue: #44800
This commit enhances the upsert and validation logic to properly handle
nullable Geometry (WKT/WKB) and Timestamptz data types:
- Add ToCompressedFormatNullable support for TimestamptzData,
GeometryWktData, and GeometryData to filter out null values during data
compression
- Implement GenNullableFieldData for Timestamptz and Geometry types to
generate nullable field data structures
- Update FillWithNullValue to handle both GeometryData and
GeometryWktData with null value filling logic
- Add UpdateFieldData support for Timestamptz, GeometryData, and
GeometryWktData field updates
- Comprehensive unit tests covering all new data type handling scenarios
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Make CDC watch etcd replicate pchannel meta instead of listing them
periodically.
issue: https://github.com/milvus-io/milvus/issues/44123
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
