issue: https://github.com/milvus-io/milvus/issues/27467
>My plan is as follows.
>- [x] M1 Create collection with timestamptz field
>- [x] M2 Insert timestamptz field data
>- [x] M3 Retrieve timestamptz field data
>- [x] M4 Implement handoff
>- [x] M5 Implement compare operator
>- [x] M6 Implement extract operator
>- [x] M8 Support database/collection level default timezone
>- [x] M7 Support STL-SORT index for datatype timestamptz
---
The third PR of issue: https://github.com/milvus-io/milvus/issues/27467,
which completes M5, M6, M7, M8 described above.
## M8 Default Timezone
We will be able to use alter_collection() and alter_database() in a
future Python SDK release to modify the default timezone at the
collection or database level.
For insert requests, the timezone will be resolved using the following
order of precedence: String Literal-> Collection Default -> Database
Default.
For retrieval requests, the timezone will be resolved in this order:
Query Parameters -> Collection Default -> Database Default.
In both cases, the final fallback timezone is UTC.
## M5: Comparison Operators
We can now use the following expression format to filter on the
timestamptz field:
- `timestamptz_field [+/- INTERVAL 'interval_string'] {comparison_op}
ISO 'iso_string' `
- The interval_string follows the ISO 8601 duration format, for example:
P1Y2M3DT1H2M3S.
- The iso_string follows the ISO 8601 timestamp format, for example:
2025-01-03T00:00:00+08:00.
- Example expressions: "tsz + INTERVAL 'P0D' != ISO
'2025-01-03T00:00:00+08:00'" or "tsz != ISO
'2025-01-03T00:00:00+08:00'".
## M6: Extract
We will be able to extract sepecific time filed by kwargs in a future
Python SDK release.
The key is `time_fields`, and value should be one or more of "year,
month, day, hour, minute, second, microsecond", seperated by comma or
space. Then the result of each record would be an array of int64.
## M7: Indexing Support
Expressions without interval arithmetic can be accelerated using an
STL-SORT index. However, expressions that include interval arithmetic
cannot be indexed. This is because the result of an interval calculation
depends on the specific timestamp value. For example, adding one month
to a date in February results in a different number of added days than
adding one month to a date in March.
---
After this PR, the input / output type of timestamptz would be iso
string. Timestampz would be stored as timestamptz data, which is int64_t
finally.
> for more information, see https://en.wikipedia.org/wiki/ISO_8601
---------
Signed-off-by: xtx <xtianx@smail.nju.edu.cn>
issue: #42032#44212
- special case for warmup param and cell storage size for tiered index
- add a config to enable/disable storage usage tracking
---------
Signed-off-by: chasingegg <chao.gao@zilliz.com>
Remove redundant `VectorPlanNode` subclasses and simplify the visitor
pattern by consolidating to a single `VectorPlanNode`.
The previous design used distinct `VectorPlanNode` subclasses and a
templated `VectorVisitorImpl` for type-directed dispatch. However, the
template parameter was not functionally used to implement different
logic for each vector type, making the subclasses redundant for their
intended purpose.
This PR is created by Cursor Agent and manually moved from
https://github.com/zhengbuqian/milvus/pull/14.
Signed-off-by: zhengbuqian <zhengbuqian@gmail.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: buqian.zheng <buqian.zheng@zilliz.com>
issue: #44212
Implement search/query storage usage statistics in go side(result
reduce), only record storage usage in vector search C++ path. Need to be
implemented in query c++ path in next prs.
---------
Signed-off-by: chasingegg <chao.gao@zilliz.com>
Signed-off-by: marcelo.chen <marcelo.chen@zilliz.com>
Co-authored-by: marcelo.chen <marcelo.chen@zilliz.com>
relate: https://github.com/milvus-io/milvus/issues/41035
This PR adds support for a gRPC-based tokenizer.
- The protobuf definition was added in
[milvus-proto#445](https://github.com/milvus-io/milvus-proto/pull/445).
- Based on this, the corresponding Rust client code was generated and
added under `tantivi-binding`.
- The generated file is `milvus.proto.tokenizer.rs`.
I'm not very experienced with Rust, so there might be parts of the code
that could be improved.
I’d appreciate any suggestions or improvements.
---------
Signed-off-by: park.sanghee <park.sanghee@navercorp.com>
Related to #39173
According to the current design, datanode shall create fs from storage
config in request instead of using singleton fs. This PR upgrade
milvus-storage and make packed reader/writer compose new fs from storage
config.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Related to #39173
backlog issue that memory size and log size shared same value. This
patch add `GetFileSize` api to get remote compressed binlog size as meta
log file size to calculate usage more accurate.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Related to #39173
- Put bucket name concatenation logic back for azure support
This reverts commit 8f97eb355fde6b86cf37f166d2191750b4210ba3.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Related to #44257
This PR:
- Utilize configurable split policy for storage v2, enabling system
field policy
- Store split result in field binlog struct
- Adapt legacy binlog without child fields
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Related to #39173
This PR:
- Bump milvus-storage commit to handle bucket name concatenation logic
in multipart s3 fs
- Remove all user-side bucket name concatenation code
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: https://github.com/milvus-io/milvus/issues/42032
- Use bytes to estimate load resource in the whole estimation procedure
- Add num_rows and dim info for vector index to better estimate
- Disable eviction for tiered index's meta
---------
Signed-off-by: chasingegg <chao.gao@zilliz.com>
issue: https://github.com/milvus-io/milvus/issues/42148
Optimized from
Go VectorArray → VectorArray Proto → Binary → C++ VectorArray Proto →
C++ VectorArray local impl → Memory
to
Go VectorArray → Arrow ListArray → Memory
---------
Signed-off-by: SpadeA <tangchenjie1210@gmail.com>
issue: #42942
This pr includes the following changes:
1. Added checks for index checker in querycoord to generate drop index
tasks
2. Added drop index interface to querynode
3. To avoid search failure after dropping the index, the querynode
allows the use of lazy mode (warmup=disable) to load raw data even when
indexes contain raw data.
4. In segcore, loading the index no longer deletes raw data; instead, it
evicts it.
5. In expr, the index is pinned to prevent concurrent errors.
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
Related to #44150
This PR make enabling `dynamic schema` feature for an existing
collection possible.
This related API is to reuse `AlterCollection` and underhood its
redirected to `adding nullable json field`
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
