Cherry-pick from master
pr: #40892
Related to #40866
This PR:
- update go-api/v2 and support partition id/channel/segment level manual
compaction
- refines the compaction trigger implementation
- unify the compaction signal usage
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Introduce a batch subscription mechanism in msgdispatcher: the
msgdispatcher now includes a vchannel watch task queue, where all
vchannels in the queue will subscribe to the MQ only once and pull
messages from the oldest vchannel checkpoint to the latest.
issue: https://github.com/milvus-io/milvus/issues/39862
pr: https://github.com/milvus-io/milvus/pull/39863
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
issue: #35528
pr: #36750
this pr includes json path index pr and some related prs:
1. update tantivy version #39253
2. json path index #36750
3. fall back to brute force #40076
4. term filter #40140
5. bug fix#40336
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
Cherry-pick from master
pr: #40134
Related to #39095
Previous PR #39990 update pkg module path using "/v2" package name, this
PR update milvusclient go sdk dependency for this update
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Cherry-pick from master
pr: #39533
The error message was malformated or missing some meta info, say field
name. This PR recitfies some message format and add field name in error
message when type param check fails.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Related to #31293
- Rename `UsingDatabase` to `UseDatabase`
- Uncomment default value methods
- Add missing RBAC APIs
- Add some resource group APIs
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
When delete by partition_key, Milvus will generates L0 segments
globally. During L0 Compaction, those L0 segments will touch all
partitions collection wise. Due to the false-positive rate of segment
bloomfilters, L0 compactions will append false deltalogs to completed
irrelevant partitions, which causes *partition deletion amplification.
This PR uses partition_key to set targeted partitionID when producing
deleteMsgs into MsgStreams. This'll narrow down L0 segments scope to
partition level, and remove the false-positive influence
collection-wise.
However, due to DeleteMsg structure, we can only label one partition to
one deleteMsg, so this enhancement fails if user wants to delete over 2
partition_keys in one deletion.
See also: #34665
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
Related to #31293
Client source files under client pkg cannot be evaluate correctly by
codecov. This PR moves them to `milvusclient` sub-package to fix this
issue and follow go major version best practice.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Related to #31293
This PR:
- Add `AlterCollection` API for collection property modification
- Expose hidden or missing option methods
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #36672
The expression supports filling elements through templates, which helps
to reduce the overhead of parsing the elements.
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
OSPP 2024 project:
https://summer-ospp.ac.cn/org/prodetail/247410235?list=org&navpage=org
Solutions:
- parser (planparserv2)
- add CallExpr in planparserv2/Plan.g4
- update parser_visitor and show_visitor
- grpc protobuf
- add CallExpr in plan.proto
- execution (`core/src/exec`)
- add `CallExpr` `ValueExpr` and `ColumnExpr` (both logical and
physical) for function call and function parameters
- function factory (`core/src/exec/expression/function`)
- create a global hashmap when starting milvus (see server.go)
- the global hashmap stores function signatures and their function
pointers, the CallExpr in execution engine can get the function pointer
by function signature.
- custom functions
- empty(string)
- starts_with(string, string)
- add cpp/go unittests and E2E tests
closes: #36559
Signed-off-by: Yinzuo Jiang <jiangyinzuo@foxmail.com>
