admin/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2026-01-07 19:31:51 +08:00
Code Issues Packages Projects Releases Wiki Activity
23,818 Commits 137 Branches 188 Tags
Commit Graph

70 Commits

Author SHA1 Message Date
zhuwenxing
e3a85be435
test: replace parquet with jsonl for EventRecords and RequestRecords in checker (#46671) 
/kind improvement

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
- Core invariant: tests' persistence of EventRecords and RequestRecords
must be append-safe under concurrent writers; this PR replaces Parquet
with JSONL and uses per-file locks and explicit buffer flushes to
guarantee atomic, append-safe writes (EventRecords uses event_lock +
append per line; RequestRecords buffers under request_lock and flushes
to file when threshold or on sink()).

- Logic removed/simplified and rationale: DataFrame-based parquet
append/read logic (pyarrow/fastparquet) and implicit parquet buffering
were removed in favor of simple line-oriented JSON writes and explicit
buffer management. The complex Parquet append/merge paths were redundant
because parquet append under concurrent test-writer patterns caused
corruption; JSONL removes the append-mode complexity and the
parquet-specific buffering/serialization code.

- Why no data loss or behavior regression (concrete code paths):
EventRecords.insert writes a complete JSON object per event under
event_lock to /tmp/ci_logs/event_records_*.jsonl and get_records_df
reads every JSON line under the same lock (or returns an empty DataFrame
with the same schema on FileNotFound/Error), preserving all fields
event_name/event_status/event_ts. RequestRecords.insert appends to an
in-memory buffer under request_lock and triggers _flush_buffer() when
len(buffer) >= 100; _flush_buffer() writes each buffered JSON line to
/tmp/ci_logs/request_records_*.jsonl and clears the buffer; sink() calls
_flush_buffer() under request_lock before get_records_df() reads the
file — ensuring all buffered records are persisted before reads. Both
read paths handle FileNotFoundError and exceptions by returning empty
DataFrames with identical column schemas, so external callers see the
same API and no silent record loss.

- Enhancement summary (concrete): Replaces flaky Parquet append/read
with JSONL + explicit locking and deterministic flush semantics,
removing the root cause of parquet append corruption in tests while
keeping the original DataFrame-based analysis consumers unchanged
(get_records_df returns equivalent schemas).
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
 2025-12-30 14:13:21 +08:00
zhuwenxing
c937da5234
test: add multi analyzer test (#41578) 
/kind improvement

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
 2025-05-22 17:46:24 +08:00
yanliang567
f20e085b22
test: Update ivf_rabitq error msg and groupby support nullable field (#41997) 
related issue: #41898

Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
 2025-05-21 22:20:25 +08:00
yanliang567
d475d93a3d
test: Add ivf_rabitq index tests (#41914) 
related issue: #41760

---------

Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
 2025-05-20 19:28:24 +08:00
yanliang567
7d35d023a3
test: Remove the old get index params method (#40319) 
related issue: https://github.com/milvus-io/milvus/issues/40156
remove the old get index params method and update the new one

---------

Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
 2025-03-04 17:12:06 +08:00
yanliang567
5fdc7578bb
test: Add sparse invert index algo check tests (#39691) 
related issue: #39332

---------

Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
 2025-02-12 12:14:46 +08:00
nico
5cdd906d4b
test: update test cases and sdk version (#39443) 
Signed-off-by: nico <cheng.yuan@zilliz.com>
 2025-02-11 18:42:45 +08:00
ThreadDao
ea339c13c6
test: add cases for async milvus client (#38699) 
issue: #38697

Signed-off-by: ThreadDao <yufen.zong@zilliz.com>
 2024-12-25 16:54:50 +08:00
foxspy
d7b2ffe5aa
enhance: add an unify vector index config checker (#36844) 
issue: #34298

Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
 2024-10-28 10:11:37 +08:00
zhuwenxing
3336b91ce6
test: add channel exclusive balance test and resource group test (#33093) 
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
 2024-05-31 13:55:52 +08:00
yanliang567
5bb672d70d
test: Add a new range search test for all indexes and align some index params (#32724) 
related issue: https://github.com/milvus-io/milvus/issues/32653

1. align some default index params
2. add a new range search tests for all indexes and float vectors

---------

Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
 2024-04-30 17:37:27 +08:00
zhuwenxing
0a2655dba0
test: fix chaos apply time (#31076) 
* increase waiting time 
* still apply chaos even timeout

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
 2024-03-08 10:25:10 +08:00
zhuwenxing
1bad19a121
[test]Fix apply chaos condition (#27625) 
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
 2023-10-11 14:11:33 +08:00
zhuwenxing
567fb23126
[test]Update health_checkers assertion for standby test (#26986) 
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
 2023-09-11 17:55:23 +08:00
zhuwenxing
68a2940b66
[test]Refine chaos apply (#26823) 
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
 2023-09-05 10:17:48 +08:00
zhuwenxing
b3de99e336
[test]Add method to analyze chaos test result (#26724) 
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
 2023-09-01 10:31:01 +08:00
Cai Yudong
8dc16b599b
Add binary metric types SUBSTRUCTURE/SUPERSTRUCTURE back (#26766) 
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
 2023-08-31 20:07:00 +08:00
jaime
c603f1c244
Remove mysql metastore (#26633) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-08-29 14:36:26 +08:00
Cai Yudong
9a4761dcc7
Remove binary metrics TANIMOTO/SUPERSTRUCTURE/SUBSTRUCTURE (#25708) 
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
 2023-07-19 16:16:58 +08:00
zhuwenxing
b70da0859a
[test]Add RPO and RTO metric for rolling update test (#25612) 
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
 2023-07-14 19:22:36 +08:00
zhuwenxing
2bcd1bb0d8
[test]Add standby test and adapt to different schemas (#24781) 
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
 2023-06-09 15:20:36 +08:00
zhuwenxing
41cb09473a
[test]Add etcd leader or followers chaos test (#24091) 
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
 2023-05-15 09:39:22 +08:00
zhuwenxing
717f712062
[test]Fix get collection for chaos test (#23484) 
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
 2023-04-18 16:06:34 +08:00
Cai Yudong
ef63e64ded
Remove ANNOY index type (#23189) 
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
 2023-04-04 16:30:27 +08:00
zhuwenxing
58a9f260ad
[test]fix k8s client init (#19590) 
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
 2022-09-30 17:34:57 +08:00
Cai Yudong
686b0ce796
Upgrade to knowhere-v1.3.0, remove following index support: (#18935) 
- IVF_SQ8H
- RHNSW_FLAT/RHNSW_PQ/RHNSW_SQ
- NGT
- NSG
- SPTAG

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-09-05 10:41:11 +08:00
zhuwenxing
030d8fb206
[test]Add testcase for verification (#18715) 
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
 2022-08-19 12:54:54 +08:00
zhuwenxing
66f197a6f5
[test]Update chaos test for milvus deployed by operator (#18682) 
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
 2022-08-17 14:32:49 +08:00
ThreadDao
a7b5dc6e04
[test] Update scale test image pull (#18437) 
Signed-off-by: ThreadDao <yufen.zong@zilliz.com>
 2022-07-28 16:00:31 +08:00
ThreadDao
19f2971e11
[test] Upgrade milvus scale test and Change k8s client config (#18404) 
Signed-off-by: ThreadDao <yufen.zong@zilliz.com>
 2022-07-26 09:00:33 +08:00
zhuwenxing
dbbe6557ed
[test]Refine deploy test (#18212) 
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
 2022-07-11 17:08:25 +08:00
elstic
e1839e3534
Code optimisation of api_request (#17918) 
Signed-off-by: elstic <hao.wang@zilliz.com>
 2022-06-30 16:00:19 +08:00
Jenny Li
4ac5ebc9f1
[skip e2e]Move milvus-dev to milvus (#17941) 
Signed-off-by: Jenny Li <jing.li@zilliz.com>
 2022-06-30 15:20:22 +08:00
ThreadDao
eb737d4d59
[test] [skip-e2e] Add L3 case to test sq load balance cross replicas (#17279) 
Signed-off-by: ThreadDao <yufen.zong@zilliz.com>
 2022-05-31 08:56:02 +08:00
zhuwenxing
4eaacf49f5
[skip e2e]Add bulk load checker for chaos test (#17101) 
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
 2022-05-20 09:31:57 +08:00
zhuwenxing
95bf91795d
[skip e2e]Add util to get milvus instance name (#17081) 
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
 2022-05-18 17:56:02 +08:00
zhuwenxing
cd6f539948
[skip e2e]Add utils to get node id and corresponding pod name (#16952) 
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
 2022-05-12 18:49:53 +08:00
ThreadDao
8945b80a64
[test][skip-e2e] Add counter decorater and scale query multi replicas test (#16845) 
Signed-off-by: ThreadDao <yufen.zong@zilliz.com>
 2022-05-09 14:51:52 +08:00
zhuwenxing
b971e2b3d0
[skip e2e]Add to util to map pod ip to pod name (#16471) 
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
 2022-04-13 10:39:34 +08:00
zhuwenxing
974371c06c
[test]Add performance monitor for ApiCollectionWrapper (#16221) 
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
 2022-03-28 16:25:27 +08:00
zhuwenxing
b745b6f707
[skip e2e]Add all pods kill chaos test (#15761) 
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
 2022-02-25 18:41:53 +08:00
zhuwenxing
bebbc841e1
[test]Fix testcase assertion (#15595) 
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
 2022-02-16 19:35:49 +08:00
binbin
89c3386907
[skip e2e] Refine PQM for RHNSW index (#15381) 
Signed-off-by: Binbin Lv <binbin.lv@zilliz.com>
 2022-01-25 17:29:40 +08:00
ThreadDao
9612557fe4
[skip e2e] Update scale test server logs export (#15302) 
Signed-off-by: ThreadDao <yufen.zong@zilliz.com>
 2022-01-20 09:11:41 +08:00
ThreadDao
2abc3b3260
[skii e2e] Add scale test pipeline server logs archive (#15234) 
Signed-off-by: ThreadDao <yufen.zong@zilliz.com>
 2022-01-18 09:21:36 +08:00
ThreadDao
892763b823
[skip e2e] Update scale test image tag to dynamic (#15040) 
Signed-off-by: ThreadDao <yufen.zong@zilliz.com>
 2022-01-13 14:57:33 +08:00
ThreadDao
2fdd74affa
[skip e2e] Remove duplicate and unused var (#14947) 
Signed-off-by: ThreadDao <yufen.zong@zilliz.com>
 2022-01-06 18:51:21 +08:00
zhuwenxing
639137a45d
[skip e2e]Refine wait_pods_ready function (#14544) 
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
 2021-12-29 20:58:37 +08:00
ThreadDao
925cf4cb5b
Unified python test log usage (#14301) 
Signed-off-by: ThreadDao <yufen.zong@zilliz.com>
 2021-12-28 13:16:10 +08:00
ThreadDao
f7787d9f57
Rename old pymilvus utils to util_pymilvus (#13957) 
Signed-off-by: ThreadDao <yufen.zong@zilliz.com>
 2021-12-22 17:07:48 +08:00