issue: #44909
When requery optimization is enabled, search results contain IDs
but empty FieldsData. During reduce/rerank operations, if the
first shard has empty FieldsData while others have data,
PrepareResultFieldData initializes an empty array, causing
AppendFieldData to panic when accessing array indices.
Changes:
- Find first non-empty FieldsData as template in 3 functions:
reduceAdvanceGroupBy, reduceSearchResultDataWithGroupBy,
reduceSearchResultDataNoGroupBy
- Add length check before 2 AppendFieldData calls in reduce
functions to prevent panic
- Improve newRerankOutputs to find first non-empty fieldData
using len(FieldsData) check instead of GetSizeOfIDs
- Add length check in appendResult before AppendFieldData
- Add comprehensive unit tests for empty and partial empty
FieldsData scenarios in both reduce and rerank functions
This fix handles both pure requery (all empty) and mixed
scenarios (some empty, some with data) without breaking normal
search flow. The key improvement is checking FieldsData length
directly rather than IDs, as requery may have IDs but empty
FieldsData.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Related to #44995
Added missing case for JSON data type in GetDefaultValue function to
properly retrieve default values for JSON fields. This prevents crashes
when enabling dynamic fields with default values during concurrent
insert operations.
Changes:
- Added JSON data type case in GetDefaultValue to return BytesData
- Added comprehensive tests for fillMissingFields covering JSON and
other data types with default values
- Added tests for nullable fields, required fields validation, and edge
cases
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #43897
- Resource group related DDL is implemented by WAL-based DDL framework
now.
- Support following message type in wal AlterResourceGroup,
DropResourceGroup.
- Resource group DDL can be synced by new CDC now.
- Refactor some UT for resource group DDL.
---------
Signed-off-by: chyezh <chyezh@outlook.com>
issue: #44961
This PR fixes 3 geometry related bugs:
1. Implement `ToString` interface for GisFunctionFilter.
2. Ignore GisFunctionFilter `MoveCursor` for growing segment.
3. Don't skip null geometry for building R-Tree index, should be record
in null_offsets.
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Related to #44620
The test was flaky because pendingMessages.Len() assertion happened
before async message processing completed. Changed to assert.Eventually
to wait up to 1 second for the pending queue to drain, fixing the race
condition where actual was 1 but expected was 0.
Fixes TestReplicateStreamClient_Reconnect flakiness.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
This pull request updates dependencies for Milvus deployments, focusing
on the images used for etcd and Kafka in both Helm and Docker Compose
configurations. The main goal is to align the deployment with newer or
organization-specific images for improved compatibility and maintenance.
**Helm deployment image updates:**
* Switched the etcd image repository from `bitnami/etcd` to
`milvusdb/etcd` and updated the etcd image tag from
`3.5.0-debian-10-r24` to `3.5.5-r2` in both standalone and cluster Helm
installation commands in `DeployTest.groovy`.
**Docker Compose image updates:**
* Changed the Kafka image from `bitnami/kafka:3.1.0` to
`bitnamilegacy/kafka:3.1.0` in `docker-compose.yml`.
issue: #44978
Signed-off-by: AlintaLu <yiqing.lu@zilliz.com>
Issue: #44518
On branch feature/timestamps
Changes to be committed:
modified: common/common_func.py
new file: milvus_client/test_milvus_client_timestamptz.py
---------
Signed-off-by: Eric Hou <eric.hou@zilliz.com>
Co-authored-by: Eric Hou <eric.hou@zilliz.com>
1. Import in replicating cluster is not supported yet, so disable it for
now.
2. Remove GetReplicateConfiguration wal API
issue: https://github.com/milvus-io/milvus/issues/44123
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
issue: #43897
- Alias related DDL is implemented by WAL-based DDL framework now.
- Support following message type in wal AlterAlias, DropAlias.
- Alias DDL can be synced by new CDC now.
- Refactor some UT for Alias DDL.
Signed-off-by: chyezh <chyezh@outlook.com>
Related to #44897
Add missing JSON data type handling in CreateArrowScalarFromDefaultValue
to fix query failures when dynamic fields are enabled. JSON default
values are now properly converted to arrow::BinaryScalar using
bytes_data().
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #44648
If the value is `null` during insertion, it will be omitted instead of
being filled with nil. Therefore, when performing checks, there’s no
need to retrieve data based on the valid offset.
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
issue: #44802
After a Geometry object is serialized into WKB, the resulting binary may
contain '\0' bytes.
When growing mmap is enabled, the append data logic uses strcpy, which
stops copying at the first '\0' bytes.
This causes only part of the WKB---typically the portion up to the
geometry type field to be copied, leading to corrupted data.
As a result, during parsing, all POINT geometries are incorrectly
interperted as POINT(0 0).
To fix this issue, memcpy will be used instead of strcpy.
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
issue: #43897
- Database related DDL is implemented by WAL-based DDL framework now.
- Support following message type in wal CreateDatabase, AlterDatabase,
DropDatabase.
- Database DDL can be synced by new CDC now.
- Refactor some UT for Database DDL.
Signed-off-by: chyezh <chyezh@outlook.com>
issue: #43897
- RBAC(Roles/Users/Privileges/Privilege Groups) is implemented by
WAL-based DDL framework now.
- Support following message type in wal `AlterUser`, `DropUser`,
`AlterRole`, `DropRole`, `AlterUserRole`, `DropUserRole`,
`AlterPrivilege`, `DropPrivilege`, `AlterPrivilegeGroup`,
`DropPrivilegeGroup`, `RestoreRBAC`.
- RBAC can be synced by new CDC now.
- Refactor some UT for RBAC.
---------
Signed-off-by: chyezh <chyezh@outlook.com>
Related to #44819
This fix addresses an issue(#44819) where the offset parameter did not
work correctly during searches when multiple results had identical
scores. The problem occurred because results with equal scores were not
consistently ordered, leading to unpredictable pagination behavior.
The solution adds a new sorting step (SortEqualScoresByPks) in the
reduce phase that sorts results with identical scores by their primary
keys in ascending order. This ensures deterministic ordering and enables
proper offset functionality.
Changes:
- Add SortEqualScoresByPks() to sort results with equal scores by PK
- Add SortEqualScoresOneNQ() to handle per-query sorting logic
- Invoke sorting step after FillPrimaryKey() in Reduce() workflow
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #43858
Fix the issue introduced in PR #43992 where deactivating the balance
checker incorrectly stops stopping balance operations.
Changes:
- Move IsActive() check after stopping balance logic
- Only skip normal balance when checker is inactive
- Allow stopping balance to proceed regardless of checker state
This ensures stopping balance can execute even when the balance checker
is deactivated.
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
