sthuang
a2c7ed2780
fix: [StorageV2] sort field binlogs paths for packed reader and writer ( #43585 )
...
key changes:
* fix unstable storage v2 compaction unit test by guaranteeing the order
of paths during sync.
* bump milvus-storage version, include
https://github.com/milvus-io/milvus-storage/pull/222
https://github.com/milvus-io/milvus-storage/pull/223
https://github.com/milvus-io/milvus-storage/pull/224
https://github.com/milvus-io/milvus-storage/pull/225
https://github.com/milvus-io/milvus-storage/pull/226
* Also fix the below related oom issue.
related: https://github.com/milvus-io/milvus/issues/43310
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2025-07-30 08:09:36 +08:00
Zhen Ye
e9ab73e93d
enhance: add schema version at recovery storage ( #43500 )
...
issue: #43072 , #43289
- manage the schema version at recovery storage.
- update the schema when creating collection or alter schema.
- get schema at write buffer based on version.
- recover the schema when upgrading from 2.5.
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2025-07-23 21:38:54 +08:00
Zhen Ye
25b76e1fde
fix: cannot auto balance the channel from old arch to streamingnode ( #43424 )
...
issue: #43416 , #43413
- also fix the panic on streamingnode when concurrent sync
Signed-off-by: chyezh <chyezh@outlook.com>
2025-07-20 23:00:52 +08:00
yihao.dai
ed55b14484
fix: Release data memory after sync task completes ( #42627 )
...
Release data memory after sync task completes to prevent datanode oom
during import.
issue: https://github.com/milvus-io/milvus/issues/42608
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2025-06-10 16:28:34 +08:00
congqixia
b8d7045539
enhance: [Add Field] Use consistent schema for single buffer ( #41891 )
...
Related to #41873
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-05-17 19:46:22 +08:00
XuanYang-cn
015a8f7631
fix: L0 brings its own start pos when syncing ( #40663 )
...
See also: #40388 , #40207
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2025-03-18 15:56:16 +08:00
sthuang
63a7c4570e
feat: storage v2 sync ( #39663 )
...
related: #39173
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2025-03-05 11:22:15 +08:00
congqixia
cb7f2fa6fd
enhance: Use v2 package name for pkg module ( #39990 )
...
Related to #39095
https://go.dev/doc/modules/version-numbers
Update pkg version according to golang dep version convention
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2025-02-22 23:15:58 +08:00
Ted Xu
56659bacbb
enhance: make serialization be part of sync task to support file format change ( #38946 )
...
See #38945
---------
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2025-01-23 15:49:05 +08:00
Zhen Ye
bb8d1ab3bf
enhance: make new go package to manage proto ( #39114 )
...
issue: #39095
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2025-01-10 10:49:01 +08:00
jaime
1d06d4324b
fix: Int64 overflow in JSON encoding ( #37657 )
...
issue: ##36621
- For simple types in a struct, add "string" to the JSON tag for
automatic string conversion during JSON encoding.
- For complex types in a struct, replace "int64" with "string."
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-11-14 22:52:30 +08:00
jaime
f348bd9441
feat: add segment,pipeline, replica and resourcegroup api for WebUI ( #37344 )
...
issue: #36621
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-11-07 11:52:25 +08:00
jaime
9d16b972ea
feat: add tasks page into management WebUI ( #37002 )
...
issue: #36621
1. Add API to access task runtime metrics, including:
- build index task
- compaction task
- import task
- balance (including load/release of segments/channels and some leader
tasks on querycoord)
- sync task
2. Add a debug model to the webpage by using debug=true or debug=false
in the URL query parameters to enable or disable debug mode.
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-10-28 10:13:29 +08:00
aoiasd
139787371e
feat: support embedding bm25 sparse vector and flush bm25 stats log ( #36036 )
...
relate: https://github.com/milvus-io/milvus/issues/35853
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-09-19 10:57:12 +08:00
congqixia
ab532ae199
enhance: Add back BF lazy load logic for datanode watch channel ( #35646 )
...
Add back lazy loading statslog when watch dml channel on datanode.
Related to #22994 #27675
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-22 19:42:57 +08:00
yihao.dai
a4439cc911
enhance: Implement flusher in streamingNode ( #34942 )
...
- Implement flusher to:
- Manage the pipelines (creation, deletion, etc.)
- Manage the segment write buffer
- Manage sync operation (including receive flushMsg and execute flush)
- Add a new `GetChannelRecoveryInfo` RPC in DataCoord.
- Reorganize packages: `flushcommon` and `datanode`.
issue: https://github.com/milvus-io/milvus/issues/33285
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-08-02 18:30:23 +08:00
yihao.dai
8aab6cbfac
enhance: Organize the common modules of streamingNode and dataNode ( #34773 )
...
1. Move the common modules of streamingNode and dataNode to flushcommon
2. Add new GetVChannels interface for rootcoord
issue: https://github.com/milvus-io/milvus/issues/33285
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-22 11:33:51 +08:00
