mirror of
https://gitee.com/milvus-io/milvus.git
synced 2025-12-30 15:35:33 +08:00
Merge remote-tracking branch 'source/0.6.0' into 0.6.0
This commit is contained in:
groot
commit
f06e0e3fe1
266
shards/README.md
Normal file
266
shards/README.md
Normal file
@ -0,0 +1,266 @@
|
||||
# Mishards - An Experimental Sharding Middleware
|
||||
|
||||
[中文版](README_CN.md)
|
||||
|
||||
Milvus aims to achieve efficient similarity search and analytics for massive-scale vectors. A standalone Milvus instance can easily handle vector search among billion-scale vectors. However, for 10 billion, 100 billion or even larger datasets, a Milvus cluster is needed.
|
||||
|
||||
Ideally, this cluster can be accessed and used just as the standalone instance, meanwhile it satisfies the business requirements such as low latency and high concurrency.
|
||||
|
||||
This page meant to demonstrates how to use Mishards, an experimental sharding middleware for Milvus, to establish an orchestrated cluster.
|
||||
|
||||
## What is Mishards
|
||||
|
||||
Mishards is a middleware that is developed using Python. It provides unlimited extension of memory and computation capacity through request forwarding, read/write splitting, horizontal scalability and dynamic extension. It works as the proxy of the Milvus system.
|
||||
|
||||
Using Mishards in Milvus cluster deployment is an experimental feature available for user test and feedback.
|
||||
|
||||
## How Mishards works
|
||||
|
||||
Mishards splits the upstream requests to sub-requests and forwards them to Milvus servers. When the search computation is completed, all results are collected by Mishards and sent back to the client.
|
||||
|
||||
Below graph is a demonstration of the process:
|
||||
|
||||
|
||||
|
||||
## Mishards example codes
|
||||
|
||||
Below examples codes demonstrate how to build from source code a Milvus server with Mishards on a standalone machine, as well as how to use Kubernetes to establish Milvus cluster with Mishards.
|
||||
|
||||
Before executing these examples, make sure you meet the prerequisites of [Milvus installation](https://www.milvus.io/docs/en/userguide/install_milvus/).
|
||||
|
||||
### Build from source code
|
||||
|
||||
#### Prequisites
|
||||
|
||||
Make sure Python 3.6 or higher is installed.
|
||||
|
||||
#### Start Milvus and Mishards from source code
|
||||
|
||||
Follow below steps to start a standalone Milvus instance with Mishards from source code:
|
||||
|
||||
1. Clone milvus repository.
|
||||
|
||||
```shell
|
||||
git clone <milvus repo http/ssh url>
|
||||
```
|
||||
|
||||
2. Install Mishards dependencies.
|
||||
|
||||
```shell
|
||||
$ cd milvus/shards
|
||||
$ pip install -r requirements.txt
|
||||
```
|
||||
|
||||
3. Start Milvus server.
|
||||
|
||||
```shell
|
||||
$ sudo nvidia-docker run --rm -d -p 19530:19530 -v /tmp/milvus/db:/opt/milvus/db milvusdb/milvus:0.5.0-d102119-ede20b
|
||||
```
|
||||
|
||||
4. Update path permissions.
|
||||
|
||||
```shell
|
||||
$ sudo chown -R $USER:$USER /tmp/milvus
|
||||
```
|
||||
|
||||
5. Configure Mishards environmental variables.
|
||||
|
||||
```shell
|
||||
$ cp mishards/.env.example mishards/.env
|
||||
```
|
||||
|
||||
6. Start Mishards server.
|
||||
|
||||
```shell
|
||||
$ python mishards/main.py
|
||||
```
|
||||
|
||||
### Docker example
|
||||
|
||||
The `all_in_one` example shows how to use Docker container to start 2 Milvus instances, 1 Mishards instance and 1 Jaeger instance.
|
||||
|
||||
1. Install [Docker Compose](https://docs.docker.com/compose/install/).
|
||||
|
||||
2. Build docker images for these instances.
|
||||
|
||||
```shell
|
||||
$ make build
|
||||
```
|
||||
|
||||
3. Start all instances.
|
||||
|
||||
```shell
|
||||
$ make deploy
|
||||
```
|
||||
|
||||
4. Confirm instance status.
|
||||
|
||||
```shell
|
||||
$ make probe_deploy
|
||||
Pass ==> Pass: Connected
|
||||
Fail ==> Error: Fail connecting to server on 127.0.0.1:19530. Timeout
|
||||
```
|
||||
|
||||
To check the service tracing, open the [Jaeger page](http://127.0.0.1:16686/) on your browser.
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
To stop all instances, use the following command:
|
||||
|
||||
```shell
|
||||
$ make clean_deploy
|
||||
```
|
||||
|
||||
### Kubernetes example
|
||||
|
||||
Using Kubernetes to deploy Milvus cluster requires that the developers have a basic understanding of [general concepts](https://kubernetes.io/docs/concepts/) of Kubernetes.
|
||||
|
||||
This example mainly demonstrates how to use Kubernetes to establish a Milvus cluster containing 2 Milvus instances（1 read instance and 1 write instance), 1 MySQL instance and 1 Mishards instance.
|
||||
|
||||
This example does not include tasks such as setting up Kubernetes cluster, [installing shared storage](https://kubernetes.io/docs/concepts/storage/volumes/) and using command tools such as [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/).
|
||||
|
||||
Below is the architecture of Milvus cluster built upon Kubernetes:
|
||||
|
||||
|
||||
|
||||
#### Prerequisites
|
||||
|
||||
- A Kubernetes cluster is already established.
|
||||
- [nvidia-docker 2.0](https://github.com/nvidia/nvidia-docker/wiki/Installation-(version-2.0)) is already installed.
|
||||
- Shared storage is already installed.
|
||||
- kubectl is installed and can access the Kubernetes cluster.
|
||||
|
||||
#### Use Kubernetes to build a Milvus cluster
|
||||
|
||||
1. Start Milvus cluster
|
||||
|
||||
```shell
|
||||
$ make cluster
|
||||
```
|
||||
|
||||
2. Confirm that Mishards is connected to Milvus.
|
||||
|
||||
```shell
|
||||
$ make probe_cluster
|
||||
Pass ==> Pass: Connected
|
||||
```
|
||||
|
||||
To check cluster status:
|
||||
|
||||
```shell
|
||||
$ make cluster_status
|
||||
```
|
||||
|
||||
To delete the cluster:
|
||||
|
||||
```shell
|
||||
$ make clean_cluster
|
||||
```
|
||||
|
||||
To add a read instance:
|
||||
|
||||
```shell
|
||||
$ cd kubernetes_demo
|
||||
$ ./start.sh scale-ro-server 2
|
||||
```
|
||||
|
||||
To add a proxy instance:
|
||||
|
||||
```shell
|
||||
$ cd kubernetes_demo
|
||||
$ ./start.sh scale-proxy 2
|
||||
```
|
||||
|
||||
To check cluster logs:
|
||||
|
||||
```shell
|
||||
$ kubectl logs -f --tail=1000 -n milvus milvus-ro-servers-0
|
||||
```
|
||||
|
||||
## Mishards Unit test
|
||||
|
||||
**Unit test**
|
||||
|
||||
```shell
|
||||
$ cd milvus/shards
|
||||
$ make test
|
||||
```
|
||||
|
||||
**Code coverage test**
|
||||
|
||||
```shell
|
||||
$ cd milvus/shards
|
||||
$ make coverage
|
||||
```
|
||||
|
||||
**Code format check**
|
||||
|
||||
```shell
|
||||
$ cd milvus/shards
|
||||
$ make style
|
||||
```
|
||||
|
||||
## Mishards configuration
|
||||
|
||||
### Overall configuration
|
||||
|
||||
| Name | Required | Type | Default | Description |
|
||||
| ------------- | -------- | ------- | ------- | ------------------------------------------------------------ |
|
||||
| `Debug` | No | boolean | `True` | Choose if to enable `Debug` work mode. |
|
||||
| `TIMEZONE` | No | string | `UTC` | Timezone |
|
||||
| `MAX_RETRY` | No | integer | `3` | The maximum retry times allowed to connect to Milvus. |
|
||||
| `SERVER_PORT` | No | integer | `19530` | Define the server port of Mishards. |
|
||||
| `WOSERVER` | **Yes** | string | ` ` | Define the address of Milvus write instance. Currently, only static settings are supported. Format for reference: `tcp://127.0.0.1:19530`. |
|
||||
|
||||
### Metadata
|
||||
|
||||
| Name | Required | Type | Default | Description |
|
||||
| ------------------------------ | -------- | ------- | ------- | ------------------------------------------------------------ |
|
||||
| `SQLALCHEMY_DATABASE_URI` | **Yes** | string | ` ` | Define the database address for metadata storage. Format standard: RFC-738-style. For example: `mysql+pymysql://root:root@127.0.0.1:3306/milvus?charset=utf8mb4`. |
|
||||
| `SQL_ECHO` | No | boolean | `False` | Choose if to print SQL statements. |
|
||||
| `SQLALCHEMY_DATABASE_TEST_URI` | No | string | ` ` | Define the database address of metadata storage in test environment. |
|
||||
| `SQL_TEST_ECHO` | No | boolean | `False` | Choose if to print SQL statements in test environment. |
|
||||
|
||||
### Service discovery
|
||||
|
||||
| Name | Required | Type | Default | Description |
|
||||
| ------------------------------------- | -------- | ------- | ------------- | ------------------------------------------------------------ |
|
||||
| `DISCOVERY_PLUGIN_PATH` | No | string | ` ` | Define the search path to locate the plug-in. The default path is used if the value is not set. |
|
||||
| `DISCOVERY_CLASS_NAME` | No | string | `static` | Under the plug-in search path, search the class based on the class name, and instantiate it. Currently, the system provides 2 classes: `static` and `kubernetes`. |
|
||||
| `DISCOVERY_STATIC_HOSTS` | No | list | `[]` | When `DISCOVERY_CLASS_NAME` is `static` , define a comma-separated service address list, for example`192.168.1.188,192.168.1.190`. |
|
||||
| `DISCOVERY_STATIC_PORT` | No | integer | `19530` | When `DISCOVERY_CLASS_NAME` is `static`, define the server port. |
|
||||
| `DISCOVERY_KUBERNETES_NAMESPACE` | No | string | ` ` | When `DISCOVERY_CLASS_NAME` is `kubernetes`, define the namespace of Milvus cluster. |
|
||||
| `DISCOVERY_KUBERNETES_IN_CLUSTER` | No | boolean | `False` | When `DISCOVERY_CLASS_NAME` is `kubernetes` , choose if to run the server in Kubernetes. |
|
||||
| `DISCOVERY_KUBERNETES_POLL_INTERVAL` | No | integer | `5` (Seconds) | When `DISCOVERY_CLASS_NAME` is `kubernetes` , define the listening cycle of the server. |
|
||||
| `DISCOVERY_KUBERNETES_POD_PATT` | No | string | ` ` | When `DISCOVERY_CLASS_NAME` is `kubernetes` , map the regular expression of Milvus Pod. |
|
||||
| `DISCOVERY_KUBERNETES_LABEL_SELECTOR` | No | string | ` ` | When `SD_PROVIDER` is `kubernetes`, map the label of Milvus Pod. For example: `tier=ro-servers`. |
|
||||
|
||||
### Tracing
|
||||
|
||||
| Name | Required | Type | Default | Description |
|
||||
| ----------------------- | -------- | ------- | ---------- | ------------------------------------------------------------ |
|
||||
| `TRACER_PLUGIN_PATH` | No | string | ` ` | Define the search path to locate the tracing plug-in. The default path is used if the value is not set. |
|
||||
| `TRACER_CLASS_NAME` | No | string | ` ` | Under the plug-in search path, search the class based on the class name, and instantiate it. Currently, only `Jaeger` is supported. |
|
||||
| `TRACING_SERVICE_NAME` | No | string | `mishards` | When `TRACING_CLASS_NAME` is [`Jaeger`](https://www.jaegertracing.io/docs/1.14/), the name of the tracing service. |
|
||||
| `TRACING_SAMPLER_TYPE` | No | string | `const` | When `TRACING_CLASS_NAME` is [`Jaeger`](https://www.jaegertracing.io/docs/1.14/), the [sampling type](https://www.jaegertracing.io/docs/1.14/sampling/) of the tracing service. |
|
||||
| `TRACING_SAMPLER_PARAM` | No | integer | `1` | When `TRACING_CLASS_NAME` is [`Jaeger`](https://www.jaegertracing.io/docs/1.14/), the [sampling frequency](https://www.jaegertracing.io/docs/1.14/sampling/) of the tracing service. |
|
||||
| `TRACING_LOG_PAYLOAD` | No | boolean | `False` | When `TRACING_CLASS_NAME` is [`Jaeger`](https://www.jaegertracing.io/docs/1.14/), choose if to sample Payload. |
|
||||
|
||||
### Logging
|
||||
|
||||
| Name | Required | Type | Default | Description |
|
||||
| ----------- | -------- | ------ | --------------- | ------------------------------------------------------------ |
|
||||
| `LOG_LEVEL` | No | string | `DEBUG` | Log recording levels. Currently supports `DEBUG` ,`INFO` ,`WARNING` and `ERROR`. |
|
||||
| `LOG_PATH` | No | string | `/tmp/mishards` | Log recording path. |
|
||||
| `LOG_NAME` | No | string | `logfile` | Log recording name. |
|
||||
|
||||
### Routing
|
||||
|
||||
| Name | Required | Type | Default | Description |
|
||||
| ------------------------ | -------- | ------ | ------------------------- | ------------------------------------------------------------ |
|
||||
| `ROUTER_PLUGIN_PATH` | No | string | ` ` | Define the search path to locate the routing plug-in. The default path is used if the value is not set. |
|
||||
| `ROUTER_CLASS_NAME` | No | string | `FileBasedHashRingRouter` | Under the plug-in search path, search the class based on the class name, and instantiate it. Currently, only `FileBasedHashRingRouter` is supported. |
|
||||
| `ROUTER_CLASS_TEST_NAME` | No | string | `FileBasedHashRingRouter` | Under the plug-in search path, search the class based on the class name, and instantiate it. Currently, `FileBasedHashRingRouter` is supported for test environment only. |
|
||||
|
||||
261
shards/README_CN.md
Normal file
261
shards/README_CN.md
Normal file
@ -0,0 +1,261 @@
|
||||
# Mishards - Milvus 集群分片中间件
|
||||
|
||||
Milvus 旨在帮助用户实现海量非结构化数据的近似检索和分析。单个 Milvus 实例可处理十亿级数据规模，而对于百亿或者千亿级数据，则需要一个 Milvus 集群实例。该实例对于上层应用可以像单机实例一样使用，同时满足海量数据低延迟、高并发业务需求。
|
||||
|
||||
本文主要展示如何使用 Mishards 分片中间件来搭建 Milvus 集群。
|
||||
|
||||
## Mishards 是什么
|
||||
|
||||
Mishards 是一个用 Python 开发的 Milvus 集群分片中间件，其内部处理请求转发、读写分离、水平扩展、动态扩容，为用户提供内存和算力可以无限扩容的 Milvus 实例。
|
||||
|
||||
Mishards 的设计尚未完成，属于试用功能，希望大家多多测试、提供反馈。
|
||||
|
||||
## Mishards 如何工作
|
||||
|
||||
Mishards 负责将上游请求拆分，并路由到内部各细分子服务，最后将子服务结果汇总，返回给上游。
|
||||
|
||||
|
||||
|
||||
## Mishards 相关示例
|
||||
|
||||
以下分别向您展示如何使用源代码在单机上启动 Mishards 和 Milvus 服务，以及如何使用 Kubernetes 启动 Milvus 集群和 Mishards。
|
||||
|
||||
Milvus 启动的前提条件请参考 [Milvus 安装](https://www.milvus.io/docs/zh-CN/userguide/install_milvus/)。
|
||||
|
||||
### 源代码启动示例
|
||||
|
||||
#### 前提条件
|
||||
|
||||
Python 版本为3.6及以上。
|
||||
|
||||
#### 源代码启动 Milvus 和 Mishards 实例
|
||||
|
||||
请按照以下步骤在单机上启动单个 Milvus 实例和 Mishards 服务：
|
||||
|
||||
1. 将 milvus repository 复制到本地。
|
||||
|
||||
```shell
|
||||
git clone <ssh url>
|
||||
```
|
||||
|
||||
2. 安装 Mishards 的依赖库。
|
||||
|
||||
```shell
|
||||
$ cd milvus/shards
|
||||
$ pip install -r requirements.txt
|
||||
```
|
||||
|
||||
3. 启动 Milvus 服务。
|
||||
|
||||
```shell
|
||||
$ sudo nvidia-docker run --rm -d -p 19530:19530 -v /tmp/milvus/db:/opt/milvus/db milvusdb/milvus:0.5.0-d102119-ede20b
|
||||
```
|
||||
|
||||
4. 更改目录权限。
|
||||
|
||||
```shell
|
||||
$ sudo chown -R $USER:$USER /tmp/milvus
|
||||
```
|
||||
|
||||
5. 配置 Mishards 环境变量
|
||||
|
||||
```shell
|
||||
$ cp mishards/.env.example mishards/.env
|
||||
```
|
||||
|
||||
6. 启动 Mishards 服务
|
||||
|
||||
```shell
|
||||
$ python mishards/main.py
|
||||
```
|
||||
|
||||
### Docker 示例
|
||||
|
||||
`all_in_one` 使用 Docker 容器启动2个 Milvus 实例，1个 Mishards 中间件实例，和1个 Jaeger 链路追踪实例。
|
||||
|
||||
1. 安装 [Docker Compose](https://docs.docker.com/compose/install/)。
|
||||
|
||||
2. 制作实例镜像。
|
||||
|
||||
```shell
|
||||
$ make build
|
||||
```
|
||||
|
||||
3. 启动所有服务。
|
||||
|
||||
```shell
|
||||
$ make deploy
|
||||
```
|
||||
|
||||
4. 检查确认服务状态。
|
||||
|
||||
```shell
|
||||
$ make probe_deploy
|
||||
Pass ==> Pass: Connected
|
||||
Fail ==> Error: Fail connecting to server on 127.0.0.1:19530. Timeout
|
||||
```
|
||||
|
||||
若要查看服务踪迹，使用浏览器打开 [Jaeger 页面](http://127.0.0.1:16686/)。
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
若要清理所有服务，请使用如下命令：
|
||||
|
||||
```shell
|
||||
$ make clean_deploy
|
||||
```
|
||||
|
||||
### Kubernetes 示例
|
||||
|
||||
使用 Kubernetes 部署 Milvus 分布式集群要求开发人员对 Kubernetes 的[基本概念](https://kubernetes.io/docs/concepts/)和操作有基本了解。
|
||||
|
||||
本示例主要展示如何使用 Kubernetes 搭建 Milvus 集群，包含2个 Milvus 实例（1个可读实例，1个可写实例）、1个 MySQL 实例和1个 Mishards 实例。
|
||||
|
||||
本示例不包括如何搭建 Kubernetes 集群，如何安装[共享存储](https://kubernetes.io/docs/concepts/storage/volumes/)和如何安装 [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/) 命令行工具等。
|
||||
|
||||
以下是 Kubernetes 示例架构图：
|
||||
|
||||
|
||||
|
||||
#### 前提条件
|
||||
|
||||
使用 Kubernetes 启动多个 Milvus 实例之前，请确保您已满足以下条件：
|
||||
|
||||
- 已创建 Kubernetes 集群
|
||||
- 已安装 [nvidia-docker 2.0](https://github.com/nvidia/nvidia-docker/wiki/Installation-(version-2.0))
|
||||
- 已安装共享存储
|
||||
- 已安装 kubectl，且能访问集群
|
||||
|
||||
#### Kubernetes 启动集群
|
||||
|
||||
1. 启动 Milvus 集群。
|
||||
|
||||
```shell
|
||||
$ make cluster
|
||||
```
|
||||
|
||||
2. 确认 Mishards 是否可用。
|
||||
|
||||
```shell
|
||||
$ make probe_cluster
|
||||
Pass ==> Pass: Connected
|
||||
```
|
||||
|
||||
查看集群状态：
|
||||
|
||||
```shell
|
||||
$ make cluster_status
|
||||
```
|
||||
|
||||
删除 Milvus 集群：
|
||||
|
||||
```shell
|
||||
$ make clean_cluster
|
||||
```
|
||||
|
||||
扩容 Milvus 可读实例到2个：
|
||||
|
||||
```shell
|
||||
$ cd kubernetes_demo
|
||||
$ ./start.sh scale-ro-server 2
|
||||
```
|
||||
|
||||
扩容 Mishards（代理）实例到2个：
|
||||
|
||||
```shell
|
||||
$ cd kubernetes_demo
|
||||
$ ./start.sh scale-proxy 2
|
||||
```
|
||||
|
||||
查看计算节点 `milvus-ro-servers-0` 日志：
|
||||
|
||||
```shell
|
||||
$ kubectl logs -f --tail=1000 -n milvus milvus-ro-servers-0
|
||||
```
|
||||
|
||||
## 单元测试
|
||||
|
||||
**单元测试**
|
||||
|
||||
```shell
|
||||
$ cd milvus/shards
|
||||
$ make test
|
||||
```
|
||||
|
||||
**代码覆盖率测试**
|
||||
|
||||
```shell
|
||||
$ cd milvus/shards
|
||||
$ make coverage
|
||||
```
|
||||
|
||||
**代码格式检查**
|
||||
|
||||
```shell
|
||||
$ cd milvus/shards
|
||||
$ make style
|
||||
```
|
||||
|
||||
## Mishards 配置
|
||||
|
||||
### 全局配置
|
||||
|
||||
| 参数 | 是否必填 | 类型 | 默认值 | 说明 |
|
||||
| ------------- | -------- | ------- | ------- | ------------------------------------------------------------ |
|
||||
| `Debug` | No | boolean | `True` | 选择是否启用 `Debug` 工作模式。 |
|
||||
| `TIMEZONE` | No | string | `UTC` | 时区 |
|
||||
| `MAX_RETRY` | No | integer | `3` | Mishards 连接 Milvus 的最大重试次数。 |
|
||||
| `SERVER_PORT` | No | integer | `19530` | 定义 Mishards 的服务端口。 |
|
||||
| `WOSERVER` | **Yes** | string | ` ` | 定义 Milvus 可写实例的地址，目前只支持静态设置。参考格式： `tcp://127.0.0.1:19530`。 |
|
||||
|
||||
### 元数据
|
||||
|
||||
| 参数 | 是否必填 | 类型 | 默认值 | 说明 |
|
||||
| ------------------------------ | -------- | ------- | ------- | ------------------------------------------------------------ |
|
||||
| `SQLALCHEMY_DATABASE_URI` | **Yes** | string | ` ` | 定义元数据存储的数据库地址，格式标准为 RFC-738-style。例如：`mysql+pymysql://root:root@127.0.0.1:3306/milvus?charset=utf8mb4`。 |
|
||||
| `SQL_ECHO` | No | boolean | `False` | 选择是否打印 SQL 详细语句。 |
|
||||
| `SQLALCHEMY_DATABASE_TEST_URI` | No | string | ` ` | 定义测试环境下元数据存储的数据库地址。 |
|
||||
| `SQL_TEST_ECHO` | No | boolean | `False` | 选择测试环境下是否打印 SQL 详细语句。 |
|
||||
|
||||
### 服务发现
|
||||
|
||||
| 参数 | 是否必填 | 类型 | 默认值 | 说明 |
|
||||
| ------------------------------------- | -------- | ------- | -------- | ------------------------------------------------------------ |
|
||||
| `DISCOVERY_PLUGIN_PATH` | No | string | ` ` | 用户自定义服务发现插件的搜索路径，默认使用系统搜索路径。 |
|
||||
| `DISCOVERY_CLASS_NAME` | No | string | `static` | 在插件搜索路径下，根据类名搜索类，并将其实例化。目前系统提供 `static` 和 `kubernetes` 两种类，默认使用 `static`。 |
|
||||
| `DISCOVERY_STATIC_HOSTS` | No | list | `[]` | `DISCOVERY_CLASS_NAME`为 `static` 时，定义服务地址列表，地址之间以逗号隔开，例如 `192.168.1.188,192.168.1.190`。 |
|
||||
| `DISCOVERY_STATIC_PORT` | No | integer | `19530` | `DISCOVERY_CLASS_NAME` 为 `static` 时，定义服务地址监听端口。 |
|
||||
| `DISCOVERY_KUBERNETES_NAMESPACE` | No | string | ` ` | `DISCOVERY_CLASS_NAME` 为 `kubernetes`时，定义 Milvus 集群的namespace。 |
|
||||
| `DISCOVERY_KUBERNETES_IN_CLUSTER` | No | boolean | `False` | `DISCOVERY_CLASS_NAME` 为 `kubernetes` 时，选择服务发现是否在集群中运行。 |
|
||||
| `DISCOVERY_KUBERNETES_POLL_INTERVAL` | No | integer | `5` | `DISCOVERY_CLASS_NAME` 为 `kubernetes` 时，定义服务发现监听周期，单位：second。 |
|
||||
| `DISCOVERY_KUBERNETES_POD_PATT` | No | string | ` ` | `DISCOVERY_CLASS_NAME` 为 `kubernetes` 时，匹配 Milvus Pod 名字的正则表达式。 |
|
||||
| `DISCOVERY_KUBERNETES_LABEL_SELECTOR` | No | string | ` ` | `SD_PROVIDER`为`kubernetes`时，匹配 Milvus Pod 的标签。例如：`tier=ro-servers`。 |
|
||||
|
||||
### 链路追踪
|
||||
|
||||
| 参数 | 是否必填 | 类型 | 默认值 | 说明 |
|
||||
| ----------------------- | -------- | ------- | ---------- | ------------------------------------------------------------ |
|
||||
| `TRACER_PLUGIN_PATH` | No | string | ` ` | 用户自定义链路追踪插件的搜索路径，默认使用系统搜索路径。 |
|
||||
| `TRACER_CLASS_NAME` | No | string | ` ` | 在插件搜索路径下，根据类名搜索类，并将其实例化。目前只支持 `Jaeger`, 默认不使用。 |
|
||||
| `TRACING_SERVICE_NAME` | No | string | `mishards` | `TRACING_CLASS_NAME` 为 [`Jaeger`](https://www.jaegertracing.io/docs/1.14/)时，链路追踪的 service。 |
|
||||
| `TRACING_SAMPLER_TYPE` | No | string | `const` | `TRACING_CLASS_NAME`为 `Jaeger` 时，链路追踪的[采样类型](https://www.jaegertracing.io/docs/1.14/sampling/)。 |
|
||||
| `TRACING_SAMPLER_PARAM` | No | integer | `1` | `TRACING_CLASS_NAME` 为 `Jaeger`时，链路追踪的[采样频率](https://www.jaegertracing.io/docs/1.14/sampling/)。 |
|
||||
| `TRACING_LOG_PAYLOAD` | No | boolean | `False` | `TRACING_CLASS_NAME`为 `Jaeger`时，链路追踪是否采集 Payload。 |
|
||||
|
||||
### 日志
|
||||
|
||||
| 参数 | 是否必填 | 类型 | 默认值 | 说明 |
|
||||
| ----------- | -------- | ------ | --------------- | ------------------------------------------------------------ |
|
||||
| `LOG_LEVEL` | No | string | `DEBUG` | 日志记录级别，目前支持 `DEBUG` 、`INFO` 、`WARNING` 和`ERROR`。 |
|
||||
| `LOG_PATH` | No | string | `/tmp/mishards` | 日志记录路径。 |
|
||||
| `LOG_NAME` | No | string | `logfile` | 日志记录名。 |
|
||||
|
||||
### 路由
|
||||
|
||||
| 参数 | 是否必填 | 类型 | 默认值 | 说明 |
|
||||
| ------------------------ | -------- | ------ | ------------------------- | ------------------------------------------------------------ |
|
||||
| `ROUTER_PLUGIN_PATH` | No | string | ` ` | 用户自定义路由插件的搜索路径，默认使用系统搜索路径。 |
|
||||
| `ROUTER_CLASS_NAME` | No | string | `FileBasedHashRingRouter` | 在插件搜索路径下，根据类名搜索路由的类，并将其实例化。目前系统只提供了 `FileBasedHashRingRouter`。 |
|
||||
| `ROUTER_CLASS_TEST_NAME` | No | string | `FileBasedHashRingRouter` | 在插件搜索路径下，根据类名搜索路由的类，并将其实例化。目前系统只提供了 `FileBasedHashRingRouter`，仅限测试环境下使用。 |
|
||||
@ -1,147 +0,0 @@
|
||||
# Mishards使用文档
|
||||
---
|
||||
Milvus 旨在帮助用户实现海量非结构化数据的近似检索和分析。单个 Milvus 实例可处理十亿级数据规模，而对于百亿或者千亿规模数据的需求，则需要一个 Milvus 集群实例，该实例对于上层应用可以像单机实例一样使用，同时满足海量数据低延迟，高并发业务需求。mishards就是一个集群中间件，其内部处理请求转发，读写分离，水平扩展，动态扩容，为用户提供内存和算力可以无限扩容的 Milvus 实例。
|
||||
|
||||
## 运行环境
|
||||
---
|
||||
|
||||
### 单机快速启动实例
|
||||
**`python >= 3.4`环境**
|
||||
|
||||
```
|
||||
1. cd milvus/shards
|
||||
2. pip install -r requirements.txt
|
||||
3. nvidia-docker run --rm -d -p 19530:19530 -v /tmp/milvus/db:/opt/milvus/db milvusdb/milvus:0.5.0-d102119-ede20b
|
||||
4. sudo chown -R $USER:$USER /tmp/milvus
|
||||
5. cp mishards/.env.example mishards/.env
|
||||
6. 在python mishards/main.py #.env配置mishards监听19532端口
|
||||
7. make probe port=19532 #健康检查
|
||||
```
|
||||
|
||||
### 容器启动实例
|
||||
`all_in_one`会在服务器上开启两个milvus实例，一个mishards实例，一个jaeger链路追踪实例
|
||||
|
||||
**启动**
|
||||
```
|
||||
cd milvus/shards
|
||||
1. 安装docker-compose
|
||||
2. make build
|
||||
3. make deploy #监听19531端口
|
||||
4. make clean_deploy #清理服务
|
||||
5. make probe_deplopy #健康检查
|
||||
```
|
||||
|
||||
**打开Jaeger UI**
|
||||
```
|
||||
浏览器打开 "http://127.0.0.1:16686/"
|
||||
```
|
||||
|
||||
### kubernetes中快速启动
|
||||
**准备**
|
||||
```
|
||||
- kubernetes集群
|
||||
- 安装nvidia-docker
|
||||
- 共享存储
|
||||
- 安装kubectl并能访问集群
|
||||
```
|
||||
|
||||
**步骤**
|
||||
```
|
||||
cd milvus/shards
|
||||
1. make deploy_cluster #启动集群
|
||||
2. make probe_cluster #健康检查
|
||||
3. make clean_cluster #关闭集群
|
||||
```
|
||||
|
||||
**扩容计算实例**
|
||||
```
|
||||
cd milvus/shards/kubernetes_demo/
|
||||
./start.sh scale-ro-server 2 扩容计算实例到2
|
||||
```
|
||||
|
||||
**扩容代理器实例**
|
||||
```
|
||||
cd milvus/shards/kubernetes_demo/
|
||||
./start.sh scale-proxy 2 扩容代理服务器实例到2
|
||||
```
|
||||
|
||||
**查看日志**
|
||||
```
|
||||
kubectl logs -f --tail=1000 -n milvus milvus-ro-servers-0 查看计算节点milvus-ro-servers-0日志
|
||||
```
|
||||
|
||||
## 测试
|
||||
|
||||
**启动单元测试**
|
||||
```
|
||||
1. cd milvus/shards
|
||||
2. make test
|
||||
```
|
||||
|
||||
**单元测试覆盖率**
|
||||
```
|
||||
1. cd milvus/shards
|
||||
2. make coverage
|
||||
```
|
||||
|
||||
**代码风格检查**
|
||||
```
|
||||
1. cd milvus/shards
|
||||
2. make style
|
||||
```
|
||||
|
||||
## mishards配置详解
|
||||
|
||||
### 全局
|
||||
| Name | Required | Type | Default Value | Explanation |
|
||||
| --------------------------- | -------- | -------- | ------------- | ------------- |
|
||||
| Debug | No | bool | True | 是否Debug工作模式 |
|
||||
| TIMEZONE | No | string | "UTC" | 时区 |
|
||||
| MAX_RETRY | No | int | 3 | 最大连接重试次数 |
|
||||
| SERVER_PORT | No | int | 19530 | 配置服务端口 |
|
||||
| WOSERVER | **Yes** | str | - | 配置后台可写Milvus实例地址。目前只支持静态设置,例"tcp://127.0.0.1:19530" |
|
||||
|
||||
### 元数据
|
||||
| Name | Required | Type | Default Value | Explanation |
|
||||
| --------------------------- | -------- | -------- | ------------- | ------------- |
|
||||
| SQLALCHEMY_DATABASE_URI | **Yes** | string | - | 配置元数据存储数据库地址 |
|
||||
| SQL_ECHO | No | bool | False | 是否打印Sql详细语句 |
|
||||
| SQLALCHEMY_DATABASE_TEST_URI | No | string | - | 配置测试环境下元数据存储数据库地址 |
|
||||
| SQL_TEST_ECHO | No | bool | False | 配置测试环境下是否打印Sql详细语句 |
|
||||
|
||||
### 服务发现
|
||||
| Name | Required | Type | Default Value | Explanation |
|
||||
| --------------------------- | -------- | -------- | ------------- | ------------- |
|
||||
| DISCOVERY_PLUGIN_PATH | No | string | - | 用户自定义服务发现插件搜索路径，默认使用系统搜索路径|
|
||||
| DISCOVERY_CLASS_NAME | No | string | static | 在服务发现插件搜索路径下搜索类并实例化。目前系统提供 **static** 和 **kubernetes** 两种类，默认使用 **static** |
|
||||
| DISCOVERY_STATIC_HOSTS | No | list | [] | **DISCOVERY_CLASS_NAME** 为 **static** 时，配置服务地址列表，例"192.168.1.188,192.168.1.190"|
|
||||
| DISCOVERY_STATIC_PORT | No | int | 19530 | **DISCOVERY_CLASS_NAME** 为 **static** 时，配置 Hosts 监听端口 |
|
||||
| DISCOVERY_KUBERNETES_NAMESPACE | No | string | - | **DISCOVERY_CLASS_NAME** 为 **kubernetes** 时，配置集群 namespace |
|
||||
| DISCOVERY_KUBERNETES_IN_CLUSTER | No | bool | False | **DISCOVERY_CLASS_NAME** 为 **kubernetes** 时，标明服务发现是否在集群中运行 |
|
||||
| DISCOVERY_KUBERNETES_POLL_INTERVAL | No | int | 5 | **DISCOVERY_CLASS_NAME** 为 **kubernetes** 时，标明服务发现监听服务列表频率,单位 Second |
|
||||
| DISCOVERY_KUBERNETES_POD_PATT | No | string | - | **DISCOVERY_CLASS_NAME** 为 **kubernetes** 时，匹配可读 Milvus 实例的正则表达式 |
|
||||
| DISCOVERY_KUBERNETES_LABEL_SELECTOR | No | string | - | **SD_PROVIDER** 为**Kubernetes**时，匹配可读Milvus实例的标签选择 |
|
||||
|
||||
### 链路追踪
|
||||
| Name | Required | Type | Default Value | Explanation |
|
||||
| --------------------------- | -------- | -------- | ------------- | ------------- |
|
||||
| TRACER_PLUGIN_PATH | No | string | - | 用户自定义链路追踪插件搜索路径，默认使用系统搜索路径|
|
||||
| TRACER_CLASS_NAME | No | string | "" | 链路追踪方案选择，目前只实现 **Jaeger**, 默认不使用|
|
||||
| TRACING_SERVICE_NAME | No | string | "mishards" | **TRACING_TYPE** 为 **Jaeger** 时，链路追踪服务名 |
|
||||
| TRACING_SAMPLER_TYPE | No | string | "const" | **TRACING_TYPE** 为 **Jaeger** 时，链路追踪采样类型 |
|
||||
| TRACING_SAMPLER_PARAM | No | int | 1 | **TRACING_TYPE** 为 **Jaeger** 时，链路追踪采样频率 |
|
||||
| TRACING_LOG_PAYLOAD | No | bool | False | **TRACING_TYPE** 为 **Jaeger** 时，链路追踪是否采集 Payload |
|
||||
|
||||
### 日志
|
||||
| Name | Required | Type | Default Value | Explanation |
|
||||
| --------------------------- | -------- | -------- | ------------- | ------------- |
|
||||
| LOG_LEVEL | No | string | "DEBUG" if Debug is ON else "INFO" | 日志记录级别 |
|
||||
| LOG_PATH | No | string | "/tmp/mishards" | 日志记录路径 |
|
||||
| LOG_NAME | No | string | "logfile" | 日志记录名 |
|
||||
|
||||
### 路由
|
||||
| Name | Required | Type | Default Value | Explanation |
|
||||
| --------------------------- | -------- | -------- | ------------- | ------------- |
|
||||
| ROUTER_PLUGIN_PATH | No | string | - | 用户自定义路由插件搜索路径，默认使用系统搜索路径|
|
||||
| ROUTER_CLASS_NAME | No | string | FileBasedHashRingRouter | 处理请求路由类名, 可注册自定义类。目前系统只提供了类 **FileBasedHashRingRouter** |
|
||||
| ROUTER_CLASS_TEST_NAME | No | string | FileBasedHashRingRouter | 测试环境下处理请求路由类名, 可注册自定义类 |
|
||||
Loading…
x
Reference in New Issue
Block a user