admin/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2025-12-28 14:35:27 +08:00
Code Issues Packages Projects Releases Wiki Activity
milvus/tests
History
marcelo-cjl 3b599441fd
feat: Add nullable vector support for proxy and querynode (#46305) 
related: #45993 

This commit extends nullable vector support to the proxy layer,
querynode,
and adds comprehensive validation, search reduce, and field data
handling
    for nullable vectors with sparse storage.
    
    Proxy layer changes:
- Update validate_util.go checkAligned() with getExpectedVectorRows()
helper
      to validate nullable vector field alignment using valid data count
- Update checkFloatVectorFieldData/checkSparseFloatVectorFieldData for
      nullable vector validation with proper row count expectations
- Add FieldDataIdxComputer in typeutil/schema.go for logical-to-physical
      index translation during search reduce operations
- Update search_reduce_util.go reduceSearchResultData to use
idxComputers
      for correct field data indexing with nullable vectors
- Update task.go, task_query.go, task_upsert.go for nullable vector
handling
    - Update msg_pack.go with nullable vector field data processing
    
    QueryNode layer changes:
    - Update segments/result.go for nullable vector result handling
- Update segments/search_reduce.go with nullable vector offset
translation
    
    Storage and index changes:
- Update data_codec.go and utils.go for nullable vector serialization
- Update indexcgowrapper/dataset.go and index.go for nullable vector
indexing
    
    Utility changes:
- Add FieldDataIdxComputer struct with Compute() method for efficient
      logical-to-physical index mapping across multiple field data
- Update EstimateEntitySize() and AppendFieldData() with fieldIdxs
parameter
    - Update funcutil.go with nullable vector support functions

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Full support for nullable vector fields (float, binary, float16,
bfloat16, int8, sparse) across ingest, storage, indexing, search and
retrieval; logical↔physical offset mapping preserves row semantics.
  * Client: compaction control and compaction-state APIs.

* **Bug Fixes**
* Improved validation for adding vector fields (nullable + dimension
checks) and corrected search/query behavior for nullable vectors.

* **Chores**
  * Persisted validity maps with indexes and on-disk formats.

* **Tests**
  * Extensive new and updated end-to-end nullable-vector tests.

<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: marcelo-cjl <marcelo.chen@zilliz.com>
2025-12-24 10:13:19 +08:00
..
_helm
test: optimize ci test about compaction and flush (#46097)
2025-12-20 12:37:21 +08:00
benchmark
test: fix dependabot alerts (#44037)
2025-08-26 09:05:51 +08:00
docker
[automated] Update Pytest image changes (#37067)
2024-10-24 16:53:28 +08:00
go_client
feat: Add nullable vector support for proxy and querynode (#46305)
2025-12-24 10:13:19 +08:00
integration
test: Temporarily disable partialResultCounter assertion in flaky tests (#46364)
2025-12-16 17:29:16 +08:00
java_client
Replace sdk source and merge tests and tests20 (#7182)
2021-08-20 11:00:56 +08:00
python_client
feat: Add nullable vector support for proxy and querynode (#46305)
2025-12-24 10:13:19 +08:00
restful_client
test: add e2e-v2 helm for amd (#45621)
2025-11-20 13:45:11 +08:00
restful_client_v2
feat: Add nullable vector support for proxy and querynode (#46305)
2025-12-24 10:13:19 +08:00
scripts
test: upgrade minio sdk from 7.1.5 to 7.2.0 (#46186)
2025-12-09 11:39:12 +08:00
OWNERS
[skip ci]Update OWNERS files (#11898)
2021-11-16 15:41:11 +08:00
README_CN.md
enhance: use docker compose instead of docker-compose (#35208)
2024-08-02 19:32:32 +08:00
README.md
enhance: use docker compose instead of docker-compose (#35208)
2024-08-02 19:32:32 +08:00

README.md

Tests

E2E Test

Configuration Requirements

Operating System
Operating System Version
Amazon Linux 2023 or above
Ubuntu 20.04 or above
Mac 10.14 or above
Hardware
Hardware Type Recommended Configuration
CPU x86_64 architecture
Intel CPU Sandy Bridge or above
CPU Instruction Set
- SSE4_2
- AVX
- AVX2
- AVX512 or arm64 Linux/MacOS
Memory 16 GB or more
Software
Software Name Version
Docker 19.05 or above
Docker Compose 1.25.5 or above
jq 1.3 or above
kubectl 1.14 or above
helm 3.0 or above
kind 0.10.0 or above

Installing Dependencies

Troubleshooting Docker and Docker Compose
  1. Confirm that Docker Daemon is running：
$ docker info

  • Ensure that Docker is installed. Refer to the official installation instructions for Docker CE/EE.

  • Start the Docker Daemon if it is not already started.

  • To run Docker without root privileges, create a user group labeled docker, then add a user to the group with sudo usermod -aG docker $USER. Log out and log back into the terminal for the changes to take effect. For more information, see the official Docker documentation for Managing Docker as a Non-Root User.

  1. Check the version of Docker-Compose
$ docker compose version

docker compose version 1.25.5, build 8a1c60f6
docker-py version: 4.1.0
CPython version: 3.7.5
OpenSSL version: OpenSSL 1.1.1f  31 Mar 2020
Install jq
Install kubectl
Install helm
Install kind

Run E2E Tests

$ cd tests/scripts
$ ./e2e-k8s.sh

Getting help

You can get help with the following command:

$ ./e2e-k8s.sh --help