admin/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2026-01-07 19:31:51 +08:00
Code Issues Packages Projects Releases Wiki Activity
milvus/tests
History
Zhen Ye c7b5c23ff6
enhance: filter the empty timetick from consuming side (#46541) 
issue: #46540

Empty timetick is just used to sync up the time clock between different
component in milvus. So empty timetick can be ignored if we achieve the
lsn/mvcc semantic for timetick. Currently, some components need the
empty timetick to trigger some operation, such as flush/tsafe. So we
only slow down the empty time tick for 5 seconds.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
- Core invariant: with LSN/MVCC semantics consumers only need (a) the
first timetick that advances the latest-required-MVCC to unblock
MVCC-dependent waits and (b) occasional periodic timeticks (~≤5s) for
clock synchronization—therefore frequent non-persisted empty timeticks
can be suppressed without breaking MVCC correctness.
- Logic removed/simplified: per-message dispatch/consumption of frequent
non-persisted empty timeticks is suppressed — an MVCC-aware filter
emptyTimeTickSlowdowner (internal/util/pipeline/consuming_slowdown.go)
short-circuits frequent empty timeticks in the stream pipeline
(internal/util/pipeline/stream_pipeline.go), and the WAL flusher
rate-limits non-persisted timetick dispatch to one emission per ~5s
(internal/streamingnode/server/flusher/flusherimpl/wal_flusher.go); the
delegator exposes GetLatestRequiredMVCCTimeTick to drive the filter
(internal/querynodev2/delegator/delegator.go).
- Why this does NOT introduce data loss or regressions: the slowdowner
always refreshes latestRequiredMVCCTimeTick via
GetLatestRequiredMVCCTimeTick and (1) never filters timeticks <
latestRequiredMVCCTimeTick (so existing tsafe/flush waits stay
unblocked) and (2) always lets the first timetick ≥
latestRequiredMVCCTimeTick pass to notify pending MVCC waits;
separately, WAL flusher suppression applies only to non-persisted
timeticks and still emits when the 5s threshold elapses, preserving
periodic clock-sync messages used by flush/tsafe.
- Enhancement summary (where it takes effect): adds
GetLatestRequiredMVCCTimeTick on ShardDelegator and
LastestMVCCTimeTickGetter, wires emptyTimeTickSlowdowner into
NewPipelineWithStream (internal/util/pipeline), and adds WAL flusher
rate-limiting + metrics
(internal/streamingnode/server/flusher/flusherimpl/wal_flusher.go,
pkg/metrics) to reduce CPU/dispatch overhead while keeping MVCC
correctness and periodic synchronization.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: chyezh <chyezh@outlook.com>
2026-01-06 20:53:24 +08:00
..
_helm
test: optimize ci test about compaction and flush (#46097)
2025-12-20 12:37:21 +08:00
benchmark
test: fix dependabot alerts (#44037)
2025-08-26 09:05:51 +08:00
docker
[automated] Update Pytest image changes (#37067)
2024-10-24 16:53:28 +08:00
go_client
enhance: filter the empty timetick from consuming side (#46541)
2026-01-06 20:53:24 +08:00
integration
enhance: Forbid writing V1 format and always use StorageV2 (#46791)
2026-01-06 11:55:23 +08:00
java_client
Replace sdk source and merge tests and tests20 (#7182)
2021-08-20 11:00:56 +08:00
python_client
feat: support query aggregtion(#36380) (#44394)
2026-01-06 16:29:25 +08:00
restful_client
test: add e2e-v2 helm for amd (#45621)
2025-11-20 13:45:11 +08:00
restful_client_v2
feat: Add nullable vector support for proxy and querynode (#46305)
2025-12-24 10:13:19 +08:00
scripts
test: upgrade minio sdk from 7.1.5 to 7.2.0 (#46186)
2025-12-09 11:39:12 +08:00
OWNERS
[skip ci]Update OWNERS files (#11898)
2021-11-16 15:41:11 +08:00
README_CN.md
enhance: use docker compose instead of docker-compose (#35208)
2024-08-02 19:32:32 +08:00
README.md
enhance: use docker compose instead of docker-compose (#35208)
2024-08-02 19:32:32 +08:00

README.md

Tests

E2E Test

Configuration Requirements

Operating System
Operating System Version
Amazon Linux 2023 or above
Ubuntu 20.04 or above
Mac 10.14 or above
Hardware
Hardware Type Recommended Configuration
CPU x86_64 architecture
Intel CPU Sandy Bridge or above
CPU Instruction Set
- SSE4_2
- AVX
- AVX2
- AVX512 or arm64 Linux/MacOS
Memory 16 GB or more
Software
Software Name Version
Docker 19.05 or above
Docker Compose 1.25.5 or above
jq 1.3 or above
kubectl 1.14 or above
helm 3.0 or above
kind 0.10.0 or above

Installing Dependencies

Troubleshooting Docker and Docker Compose
  1. Confirm that Docker Daemon is running：
$ docker info

  • Ensure that Docker is installed. Refer to the official installation instructions for Docker CE/EE.

  • Start the Docker Daemon if it is not already started.

  • To run Docker without root privileges, create a user group labeled docker, then add a user to the group with sudo usermod -aG docker $USER. Log out and log back into the terminal for the changes to take effect. For more information, see the official Docker documentation for Managing Docker as a Non-Root User.

  1. Check the version of Docker-Compose
$ docker compose version

docker compose version 1.25.5, build 8a1c60f6
docker-py version: 4.1.0
CPython version: 3.7.5
OpenSSL version: OpenSSL 1.1.1f  31 Mar 2020
Install jq
Install kubectl
Install helm
Install kind

Run E2E Tests

$ cd tests/scripts
$ ./e2e-k8s.sh

Getting help

You can get help with the following command:

$ ./e2e-k8s.sh --help