congqixia
fc968ff1c2
enhance: [StorageV2] Pass args for avg size split policy ( #44301 )
...
Related to #44257
This PR
- Pass column stats for avg size split policy
- Add param items for policy configuration
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-09-11 10:43:57 +08:00
congqixia
f5618d5153
enhance: [StorageV2] Utilized advance split policy and persist in meta ( #44282 )
...
Related to #44257
This PR:
- Utilize configurable split policy for storage v2, enabling system
field policy
- Store split result in field binlog struct
- Adapt legacy binlog without child fields
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-09-10 14:47:57 +08:00
sthuang
a2c7ed2780
fix: [StorageV2] sort field binlogs paths for packed reader and writer ( #43585 )
...
key changes:
* fix unstable storage v2 compaction unit test by guaranteeing the order
of paths during sync.
* bump milvus-storage version, include
https://github.com/milvus-io/milvus-storage/pull/222
https://github.com/milvus-io/milvus-storage/pull/223
https://github.com/milvus-io/milvus-storage/pull/224
https://github.com/milvus-io/milvus-storage/pull/225
https://github.com/milvus-io/milvus-storage/pull/226
* Also fix the below related oom issue.
related: https://github.com/milvus-io/milvus/issues/43310
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2025-07-30 08:09:36 +08:00
Zhen Ye
25b76e1fde
fix: cannot auto balance the channel from old arch to streamingnode ( #43424 )
...
issue: #43416 , #43413
- also fix the panic on streamingnode when concurrent sync
Signed-off-by: chyezh <chyezh@outlook.com>
2025-07-20 23:00:52 +08:00
cai.zhang
3ffd44f302
fix: Fix remaining issues with Datanode pooling and StorageV2 ( #43147 )
...
issue: #43146
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2025-07-10 14:26:48 +08:00
yihao.dai
ed55b14484
fix: Release data memory after sync task completes ( #42627 )
...
Release data memory after sync task completes to prevent datanode oom
during import.
issue: https://github.com/milvus-io/milvus/issues/42608
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-06-10 16:28:34 +08:00
congqixia
b8d7045539
enhance: [Add Field] Use consistent schema for single buffer ( #41891 )
...
Related to #41873
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-05-17 19:46:22 +08:00
XuanYang-cn
e6c46a25ea
enhance: Use correct counter metrics for overall wa calculation ( #40394 )
...
- Use CounterVec to calculate sum of increase during a time period.
- Use entries number instead of binlog size
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2025-03-10 16:34:06 +08:00
sthuang
63a7c4570e
feat: storage v2 sync ( #39663 )
...
related: #39173
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2025-03-05 11:22:15 +08:00
congqixia
cb7f2fa6fd
enhance: Use v2 package name for pkg module ( #39990 )
...
Related to #39095
https://go.dev/doc/modules/version-numbers
Update pkg version according to golang dep version convention
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-02-22 23:15:58 +08:00
XuanYang-cn
1f14053c70
enhance: Enable to observe write amplification ( #39661 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2025-02-08 18:38:43 +08:00
Ted Xu
56659bacbb
enhance: make serialization be part of sync task to support file format change ( #38946 )
...
See #38945
---------
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2025-01-23 15:49:05 +08:00
aoiasd
9cb4c4e8ac
fix: bm25 import segment without bm25 stats meta ( #38855 )
...
relate: https://github.com/milvus-io/milvus/issues/38854
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2025-01-21 11:09:04 +08:00
Zhen Ye
bb8d1ab3bf
enhance: make new go package to manage proto ( #39114 )
...
issue: #39095
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2025-01-10 10:49:01 +08:00
congqixia
b0bd290a6e
enhance: Use internal json(sonic) to replace std json lib ( #37708 )
...
Related to #35020
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-11-18 10:46:31 +08:00
XuanYang-cn
5a23c80f20
fix: Change memoryCheck write lock to read lock ( #37525 )
...
See also: milvus-io#37493
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-11-15 10:44:31 +08:00
jaime
1d06d4324b
fix: Int64 overflow in JSON encoding ( #37657 )
...
issue: ##36621
- For simple types in a struct, add "string" to the JSON tag for
automatic string conversion during JSON encoding.
- For complex types in a struct, replace "int64" with "string."
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-11-14 22:52:30 +08:00
jaime
f348bd9441
feat: add segment,pipeline, replica and resourcegroup api for WebUI ( #37344 )
...
issue: #36621
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-11-07 11:52:25 +08:00
jaime
9d16b972ea
feat: add tasks page into management WebUI ( #37002 )
...
issue: #36621
1. Add API to access task runtime metrics, including:
- build index task
- compaction task
- import task
- balance (including load/release of segments/channels and some leader
tasks on querycoord)
- sync task
2. Add a debug model to the webpage by using debug=true or debug=false
in the URL query parameters to enable or disable debug mode.
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-10-28 10:13:29 +08:00
aoiasd
20750c061b
fix: loss small bm25 binlogs ( #36942 )
...
Sync task don't flush small bm25 logs, cause growing segment bm25 stats
loss.
relate: https://github.com/milvus-io/milvus/issues/36805
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-10-17 14:49:23 +08:00
yihao.dai
80f25d497f
enhance: Add metrics to monitor import throughput and imported rows ( #36519 )
...
issue: https://github.com/milvus-io/milvus/issues/36518
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-09-28 17:31:15 +08:00
aoiasd
139787371e
feat: support embedding bm25 sparse vector and flush bm25 stats log ( #36036 )
...
relate: https://github.com/milvus-io/milvus/issues/35853
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-09-19 10:57:12 +08:00
yihao.dai
6130a85444
enhance: Remove bf from streaming node ( #35902 )
...
Remove bf from streaming node:
1. When watching vchannels, skip loading bloom filters for segments.
2. Bypass bloom filter checks for delete messages, directly writing to
L0 segments.
3. Remove flushed segments proactively after flush.
issue: https://github.com/milvus-io/milvus/issues/33285 ,
https://github.com/milvus-io/milvus/issues/34585
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-09-03 14:17:02 +08:00
yihao.dai
8aab6cbfac
enhance: Organize the common modules of streamingNode and dataNode ( #34773 )
...
1. Move the common modules of streamingNode and dataNode to flushcommon
2. Add new GetVChannels interface for rootcoord
issue: https://github.com/milvus-io/milvus/issues/33285
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-22 11:33:51 +08:00
