Iterators are long deprecated, but sort are still using it. This PR
unifies stats task with the latest compaction common functions and
remove the usage of iterators.
1. Rename `datanode/compaction` to `datanode/compactor`
2. Add `internal/compaction` and move some compaction commons into it.
3. Replace `DeltalogIterators` with `ComposeDeleteFromDeltalogs`
4. Remove `datanode/iterators`
See also: #39242
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
issue: #38399
related PR: #39522
- Just implement exclusive broadcaster between broadcast message with
same resource key to keep same order in different wal.
- After simplify the broadcast model, original watch-based broadcast is
too complicated and redundant, remove it.
---------
Signed-off-by: chyezh <chyezh@outlook.com>
issue: #40052
task delta cache rely on the taskID is unique, so it incDeltaCache at
AddTask, and decDeltaCache at RemoveTask, but the taskID allocator is
not atomic, which cause two task with same taskID, in such case, it will
call incDeltaCache twice, but call decDeltaCacheOnce, which cause delta
cache leak.
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
This PR introduces two restrictions:
1. Before dropping a collection, all aliases associated with that
collection must be dropped.
2. When creating a collection, if the collection name duplicates any
alias, the collection creation will fail.
issue: https://github.com/milvus-io/milvus/issues/40142
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
issue: #39551
This PR remove querycoord's scheduling of l0 segments:
- only load l0 segment when watch channel
- only release l0 segment when release channel or sync data distribution
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
issue: #40198
Fix random sample does not consider empty input, that is no data is hit
by filter expression.
---------
Signed-off-by: SpadeA <tangchenjie1210@gmail.com>
issue: #38399
- add metrics for broadcaster component.
- add metrics for wal flusher component.
- add metrics for wal interceptors.
- add slow log for wal.
- add more label for some wal metrics. (local or remote/catcup or
tailing...)
Signed-off-by: chyezh <chyezh@outlook.com>
issue: https://github.com/milvus-io/milvus/issues/35528
If the query data type does not match the index type, fall back to a
brute-force search
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
The partition num shall be determined by core logic if user did not
specifiy the partition num in request.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Related to #40078
Add a subTaskPool to execute sub task in case of logic deadlock
described in issue.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Using unsafe.Slice to convert string to []byte by directly using
underlying data could avoid lots of copy and cpu time
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
* use the new packed reader and writer api to be compatible with current
etcd meta
* For the new packed writer API: column groups and paths are explicitly
defined by users and won't split column groups by memory in storage v2.
Packed writer follows the user-defined column groups to split arrow
record and write into the corresponding file path.
* For the new packed reader API: read paths are explicitly defined by
users.
related: #39173
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
