diff --git a/docs/developer_guides/figs/guarantee-ts-consistency-relationship.png b/docs/developer_guides/figs/guarantee-ts-consistency-relationship.png new file mode 100644 index 0000000000..1625a3bf03 Binary files /dev/null and b/docs/developer_guides/figs/guarantee-ts-consistency-relationship.png differ diff --git a/docs/developer_guides/figs/guarantee-ts-do-search-right-now.png b/docs/developer_guides/figs/guarantee-ts-do-search-right-now.png new file mode 100644 index 0000000000..269d46f0b1 Binary files /dev/null and b/docs/developer_guides/figs/guarantee-ts-do-search-right-now.png differ diff --git a/docs/developer_guides/figs/guarantee-ts-ts-mask.png b/docs/developer_guides/figs/guarantee-ts-ts-mask.png new file mode 100644 index 0000000000..6703d9cd50 Binary files /dev/null and b/docs/developer_guides/figs/guarantee-ts-ts-mask.png differ diff --git a/docs/developer_guides/figs/guarantee-ts-wait-for-service-time.png b/docs/developer_guides/figs/guarantee-ts-wait-for-service-time.png new file mode 100644 index 0000000000..27edffcbfa Binary files /dev/null and b/docs/developer_guides/figs/guarantee-ts-wait-for-service-time.png differ diff --git a/docs/developer_guides/how-guarantee-ts-works-cn.md b/docs/developer_guides/how-guarantee-ts-works-cn.md new file mode 100644 index 0000000000..3b6a8a6660 --- /dev/null +++ b/docs/developer_guides/how-guarantee-ts-works-cn.md @@ -0,0 +1,62 @@ +# Guarantee Timestamp in Search Requests + +[English Version of This Doc](./how-guarantee-ts-works.md) + +很多同学接触 Milvus 时都会对 Search 请求里面茫茫多的参数感到迷惑不解,尤其是为 Milvus 开 +发 sdk 客户端的同学。这个文档就会介绍 Search 请求里面一个比较特殊的参数——“Guarantee +Timestamp”,以下简称 “GuaranteeTs”。 + +## Milvus 时钟机制 + +像大多数分布式系统一样,Milvus 会为每一条进入系统的记录分配一个时间戳。与此同时,Milvus +是一个存储计算分离的系统,数据持久化负载由 DataNodes 承担,最终会落盘到 MinIO/S3 之类的 +分布式对象存储中。Search 之类的计算任务由 QueryNodes 承担,计算链路会同时处理两类数据—— +批式数据以及流式数据。其中,批式数据不会再被更改,Search 请求会看到批式数据里面的所有数据。 +QueryNodes 和 DataNodes 通过同一个订阅机制消费用户的插入请求,这些构成了流式数据。由于 +存在网络延迟,QueryNodes 往往不会持有最新的流式数据。如果没有其他保障,在流式数据上直接做 +Search 会损失许多未消费的数据,降低查询精度,这个时候 GuaranteeTs 就派上用场了。 + +Milvus 通过时间戳水印来保障读链路的一致性,如下图所示,在往消息队列插入数据时, +Milvus 不光会为这些插入记录打上时间戳,还会不间断地插入同步时间戳,以图中同步时间戳 +syncTs1 为例,当下游消费者(比如QueryNodes)看到 syncTs1,那么意味着 syncTs1 以前的 +数据已经全部被消费了,换句话说,比 syncTs1 时间戳还小的插入记录不会再出现在消息队列中。 +当然了,有的话,那肯定是系统有 bug,如果你发现了,还希望尽快告诉我们。 + +![ts-watermask](./figs/guarantee-ts-ts-mask.png) + +## Guarantee Timestamp + +上面说了,QueryNodes 会不断从消息队列里面拿到插入记录以及同步时间戳,每消费到一个同步时间 +戳,QueryNodes 会把这个时间戳称为可服务时间——“ServiceTime”,有了上图,ServiceTime +实际上就很好理解了,意味着 QueryNodes 能够看到 ServiceTime 以前所有的数据了。 + +有了这个 ServiceTime,Milvus 根据不同用户对一致性以及可用性的需求,提供了 GuaranteeTs, +用户可以指定 GuaranteeTs 告知 QueryNodes 我这次 Search 请求必须看到 GuaranteeTs 以前 +的所有数据。 + +如下图所示,如果 GuaranteeTs 小于 ServiceTime,QueryNodes 可以立刻执行 Search 查询。 + +![do-search-right-now](./figs/guarantee-ts-do-search-right-now.png) + +如果 GuaranteeTs 大于 ServiceTime,QueryNodes 必须从消息队列里持续消费同步时间戳, +直到 ServiceTime 大于 GuaranteeTs 才能执行 Search 查询。 + +![wait-for-service-time](./figs/guarantee-ts-wait-for-service-time.png) + +如果用户希望得到足够高的查询精度,对一致性有较高的要求,对查询时延不敏感,那么 GuaranteeTs +应该尽可能大;反之,如果用户希望尽快得到搜索结果,对可用性有较高的要求,对查询精度有较高的 +容忍程度,那么 GuaranteeTs 可以不必特别大。 + +如下图所示,不同的 GuaranteeTs 分别对应四种不同的一致性: + +![relationship-between-consistency-and-guaranteeTs](./figs/guarantee-ts-consistency-relationship.png) + +- 强一致性:GuaranteeTs 设为系统最新时间戳,QueryNodes 需要等待 ServiceTime 推进到 +当前最新时间戳才能执行该 Search 请求; +- 最终一致性:GuaranteeTs 设为 0,跳过一致性检查,立刻在当前已有数据上执行 Search 查询; +- 有界一致性:GuaranteeTs 是一个比系统最新时间稍旧的时间,在可容忍范围内可以立刻执行查询; +- 客户端一致性:客户端使用上一次写入的时间戳作为 GuaranteeTs,那么每个客户端至少能看到 +自己插入的全部数据。 + +Milvus 默认提供强一致性,如果用户不传入 GuaranteeTs,那么会将 GuaranteeTs 设为系统 +当前的最新时间戳。 diff --git a/docs/developer_guides/how-guarantee-ts-works.md b/docs/developer_guides/how-guarantee-ts-works.md new file mode 100644 index 0000000000..38a36fcc27 --- /dev/null +++ b/docs/developer_guides/how-guarantee-ts-works.md @@ -0,0 +1,80 @@ +# Guarantee Timestamp in Search Requests + +[中文文档](./how-guarantee-ts-works-cn.md) + +Users and developers may feel confused by the so many parameters in the search requests, especially for those developers +who develop sdk client for Milvus. + +This document will explain a special parameter in the search request-"Guarantee Timestamp", hereinafter referred to as +"GuaranteeTs". + +## Milvus Clock Mechanism + +Like most distributed systems, Milvus will assign a timestamp to each record that enters the system. + +At the same time, Milvus is a system that separates storage and computing. + +The DataNodes handle the data persistence load, and the data will eventually be flushed to MinIO/S3 or other distributed +object storage. + +QueryNodes handle the computing(read) requests. The reading link will process two types of data at the same time--batch +data and streaming data. + +Among them, the batch data will no longer be changed, and the search request will see all the data in the batch data. + +QueryNodes and DataNodes consume user insert requests through the same subscription mechanism, which constitute the +streaming data. + +Due to the network delays, QueryNodes often do not hold all the latest streaming data. Without other control mechanism, +some unconsumed data will be invisible and thus reduce the query accuracy. At this time, let's introduce the +"GuaranteeTs". + +Milvus uses timestamp and timetick watermark to ensure the consistency of the read link. + +As shown in the figure below, when inserting data into the message queue, Milvus will not only time stamp these inserted +records, but also insert timetick continuously. Taking "syncTs1" in the figure as an example, when downstream consumers +(such as QueryNodes) see syncTs1, it means that all the data which is previous before syncTs1 has been consumed. In +other words, the inserted record with a timestamp smaller than syncTs1 will no longer appear in the message queue. +Of course, if there is, there must be a bug in the system. If you find it, we are pleased to solve it as soon as +possible. + +![ts-watermask](./figs/guarantee-ts-ts-mask.png) + +## Guarantee Timestamp + +As mentioned above, QueryNodes will continue to get insert records and timetick from the message queue. Every time a +timetick is consumed, QueryNodes will update this timetick as the serviceable time-"ServiceTime". With the above figure, +it is easy to understand ServiceTime that QueryNodes can see all the data before ServiceTime. + +With this ServiceTime, Milvus provides GuaranteeTs according to the needs of different users for consistency and +availability. Users can specify GuaranteeTs to inform QueryNodes that search request must see all the data before +GuaranteeTs. + +As shown in the figure below, if GuaranteeTs is less than ServiceTime, QueryNodes can execute search request +immediately. + +![do-search-right-now](./figs/guarantee-ts-do-search-right-now.png) + +If GuaranteeTs is greater than ServiceTime, QueryNodes must continue to consume timetick from the message queue. Search +request cannot be executed until ServiceTime is greater than GuaranteeTs. + +![wait-for-service-time](./figs/guarantee-ts-wait-for-service-time.png) + +If users want to obtain high query accuracy, have higher requirements for consistency, and are not sensitive to query +delay, then GuaranteeTs should be set to the latest wall clock; on the contrary, if users have higher requirements for +availability, in other words, if they want to get the search results as soon as possible, then GuaranteeTs may not be +set very large. + +As shown in the figure below, different GuaranteeTs correspond to four different consistency levels: + +![relationship-between-consistency-and-guaranteeTs](./figs/guarantee-ts-consistency-relationship.png) + +- Strong consistency: GuaranteeTs is set to the newest timestamp of the system, QueryNodes will wait for the ServiceTime +to be greater than or equal to the GuaranteeTs; +- Eventual consistency: Set GuaranteeTs to 0 and skip the check, QueryNodes will execute search requests immediately; +- Bounded Staleness: Set GuaranteeTs to an older timestamp, such as 1 minutes ago, the query can be executed immediately +within a tolerable range; +- Read your own write (Session): Set GuaranteeTs to the client last write, in this way, every client will see all their +own data. + +Milvus provides the strong consistency by default.