milvus/tests/milvus_benchmark

milvus_benchmark is a non-functional testing tool which allows users to run tests on k8s cluster or at local, the primary use case is performance/load/stability testing, the objective is to expose problems in milvus project.

Quick start

Description：

• Test cases in milvus_benchmark can be organized with yaml
• Test can run with local mode or helm mode
  • local: install and start your local server, and pass the host/port param when start the tests
  • helm: install the server by helm, which will manage the milvus in k8s cluster, and you can interagte the test stage into argo workflow or jenkins pipeline

Usage：

1. Using jenkins: Use ci/main_jenkinsfile as the jenkins pipeline file

2. Using argo： example argo workflow yaml configuration: ci/argo.yaml

3. Local test：

   1). set PYTHONPATH:

   export PYTHONPATH=/your/project/path/milvus_benchmark

   2). prepare data:

   if we need to use the sift/deep dataset as the raw data input, we need to mount NAS and update RAW_DATA_DIR in config.py, the example mount command: sudo mount -t cifs -o username=test,vers=1.0 //172.16.70.249/test /test

   3). install requirements:

   pip install -r requirements.txt

   4). write test yaml and run with the yaml param:

   cd milvus-benchmark/ && python main.py --local --host=* --port=19530 --suite=suites/2_insert_data.yaml

Definitions of test suite：

Test suite yaml defines the test process, users need to write test suite yaml if adding a customized test into the current test framework.

Take the test file 2_insert_data.yaml as an example, the top level is the test type: insert_performance, there are lots of test types including: search_performance/build_performance/insert_performance/accuracy/locust_insert/..., each test type corresponds to this different runner defined in directory runnners, the other parts in the test yaml is the params pass to the runner, such as the field collection_name means which kind of collection will be created in milvus.

Test result：

Test result will be uploaded if run with the helm mode, which will be used to judge if the test run pass or failed.