yihao.dai
a4439cc911
enhance: Implement flusher in streamingNode ( #34942 )
...
- Implement flusher to:
- Manage the pipelines (creation, deletion, etc.)
- Manage the segment write buffer
- Manage sync operation (including receive flushMsg and execute flush)
- Add a new `GetChannelRecoveryInfo` RPC in DataCoord.
- Reorganize packages: `flushcommon` and `datanode`.
issue: https://github.com/milvus-io/milvus/issues/33285
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-08-02 18:30:23 +08:00
yihao.dai
8aab6cbfac
enhance: Organize the common modules of streamingNode and dataNode ( #34773 )
...
1. Move the common modules of streamingNode and dataNode to flushcommon
2. Add new GetVChannels interface for rootcoord
issue: https://github.com/milvus-io/milvus/issues/33285
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-22 11:33:51 +08:00
wayblink
c79d1af390
enhance: Add compaction task slot usage logic ( #34581 )
...
#34544
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-18 10:27:41 +08:00
yihao.dai
ca758c36cc
enhance: Pre-allocate ids for compaction ( #34187 )
...
This PR removes the dependency of compaction on the ID allocator by
pre-allocating the logID and segmentID.
issue: https://github.com/milvus-io/milvus/issues/33957
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-17 13:23:42 +08:00
XuanYang-cn
8a2be8a457
fix: DataNode might OOM by estimating based on MemorySize ( #34201 )
...
See also: #34136
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-07-04 10:16:17 +08:00
jaime
d6afb31b94
enhance: make subfunctions of datanode component modular ( #33992 )
...
issue: #33994
also remove deprecated channel manager based on the etcd implementation
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-07-01 14:46:07 +08:00
XuanYang-cn
04edb07d82
enhance: Add deltaRowCount in l0 compaction ( #33997 )
...
See also: #33998
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-06-20 17:46:01 +08:00
wei liu
31ef0a1fe8
enhance: Add trace for bf cost in l0 compactor ( #33860 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-06-20 10:10:05 +08:00
wei liu
4987067375
enhance: Execute bloom filter apply in parallel to speed up segment predict ( #33792 )
...
issue: #33610
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-06-14 11:37:56 +08:00
wei liu
ab93d9c23d
enhance: Use BatchPkExist to reduce bloom filter func call cost ( #33611 )
...
issue:#33610
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-06-13 17:57:56 +08:00
wayblink
a1232fafda
feat: Major compaction ( #33620 )
...
#30633
Signed-off-by: wayblink <anyang.wang@zilliz.com>
Co-authored-by: MrPresent-Han <chun.han@zilliz.com>
2024-06-10 21:34:08 +08:00
XuanYang-cn
4dd0c54ca0
fix: Fix l0 compactor may cause DN from OOM ( #33554 )
...
See also: #33547
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-06-06 14:33:52 +08:00
