admin/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2025-12-07 01:28:27 +08:00
Code Issues Packages Projects Releases Wiki Activity
23,470 Commits 133 Branches 183 Tags
Commit Graph

26 Commits

Author SHA1 Message Date
congqixia
569a5b40d2
enhance: [StorageV2] add manifest path support for FFI integration (#44991) 
Related to #44956

Add manifest_path field throughout the data path to support LOON Storage
V2 manifest tracking. The manifest stores metadata for segment data
files and enables the unified Storage V2 FFI interface.

Changes include:
- Add manifest_path field to SegmentInfo and SaveBinlogPathsRequest
proto messages
- Add UpdateManifest operator to datacoord meta operations
- Update metacache, sync manager, and meta writer to propagate manifest
paths
- Include manifest_path in segment load info for query coordinator

This is part of the Storage V2 FFI interface integration.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2025-10-27 19:24:10 +08:00
congqixia
1e3ec42e54
fix: Check child fields len instead of nil (#44405) 
Related to #44398

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2025-09-17 11:36:00 +08:00
congqixia
fc968ff1c2
enhance: [StorageV2] Pass args for avg size split policy (#44301) 
Related to #44257

This PR
- Pass column stats for avg size split policy
- Add param items for policy configuration

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2025-09-11 10:43:57 +08:00
congqixia
f5618d5153
enhance: [StorageV2] Utilized advance split policy and persist in meta (#44282) 
Related to #44257

This PR:
- Utilize configurable split policy for storage v2, enabling system
field policy
- Store split result in field binlog struct
- Adapt legacy binlog without child fields

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2025-09-10 14:47:57 +08:00
sthuang
a2c7ed2780
fix: [StorageV2] sort field binlogs paths for packed reader and writer (#43585) 
key changes:
* fix unstable storage v2 compaction unit test by guaranteeing the order
of paths during sync.
* bump milvus-storage version, include
https://github.com/milvus-io/milvus-storage/pull/222
https://github.com/milvus-io/milvus-storage/pull/223
https://github.com/milvus-io/milvus-storage/pull/224
https://github.com/milvus-io/milvus-storage/pull/225
https://github.com/milvus-io/milvus-storage/pull/226
* Also fix the below related oom issue.
related: https://github.com/milvus-io/milvus/issues/43310

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
 2025-07-30 08:09:36 +08:00
Zhen Ye
25b76e1fde
fix: cannot auto balance the channel from old arch to streamingnode (#43424) 
issue: #43416, #43413

- also fix the panic on streamingnode when concurrent sync

Signed-off-by: chyezh <chyezh@outlook.com>
 2025-07-20 23:00:52 +08:00
cai.zhang
3ffd44f302
fix: Fix remaining issues with Datanode pooling and StorageV2 (#43147) 
issue: #43146

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2025-07-10 14:26:48 +08:00
yihao.dai
ed55b14484
fix: Release data memory after sync task completes (#42627) 
Release data memory after sync task completes to prevent datanode oom
during import.

issue: https://github.com/milvus-io/milvus/issues/42608

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2025-06-10 16:28:34 +08:00
congqixia
b8d7045539
enhance: [Add Field] Use consistent schema for single buffer (#41891) 
Related to #41873

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2025-05-17 19:46:22 +08:00
XuanYang-cn
e6c46a25ea
enhance: Use correct counter metrics for overall wa calculation (#40394) 
- Use CounterVec to calculate sum of increase during a time period.
- Use entries number instead of binlog size

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2025-03-10 16:34:06 +08:00
sthuang
63a7c4570e
feat: storage v2 sync (#39663) 
related: #39173

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
 2025-03-05 11:22:15 +08:00
congqixia
cb7f2fa6fd
enhance: Use v2 package name for pkg module (#39990) 
Related to #39095

https://go.dev/doc/modules/version-numbers

Update pkg version according to golang dep version convention

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2025-02-22 23:15:58 +08:00
XuanYang-cn
1f14053c70
enhance: Enable to observe write amplification (#39661) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2025-02-08 18:38:43 +08:00
Ted Xu
56659bacbb
enhance: make serialization be part of sync task to support file format change (#38946) 
See #38945

---------

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
 2025-01-23 15:49:05 +08:00
aoiasd
9cb4c4e8ac
fix: bm25 import segment without bm25 stats meta (#38855) 
relate: https://github.com/milvus-io/milvus/issues/38854

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2025-01-21 11:09:04 +08:00
Zhen Ye
bb8d1ab3bf
enhance: make new go package to manage proto (#39114) 
issue: #39095

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 2025-01-10 10:49:01 +08:00
congqixia
b0bd290a6e
enhance: Use internal json(sonic) to replace std json lib (#37708) 
Related to #35020

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-11-18 10:46:31 +08:00
XuanYang-cn
5a23c80f20
fix: Change memoryCheck write lock to read lock (#37525) 
See also: milvus-io#37493

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-11-15 10:44:31 +08:00
jaime
1d06d4324b
fix: Int64 overflow in JSON encoding (#37657) 
issue: ##36621

- For simple types in a struct, add "string" to the JSON tag for
automatic string conversion during JSON encoding.
- For complex types in a struct, replace "int64" with "string."

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-11-14 22:52:30 +08:00
jaime
f348bd9441
feat: add segment,pipeline, replica and resourcegroup api for WebUI (#37344) 
issue: #36621

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-11-07 11:52:25 +08:00
jaime
9d16b972ea
feat: add tasks page into management WebUI (#37002) 
issue: #36621

1. Add API to access task runtime metrics, including:
  - build index task
  - compaction task
  - import task
- balance (including load/release of segments/channels and some leader
tasks on querycoord)
  - sync task
2. Add a debug model to the webpage by using debug=true or debug=false
in the URL query parameters to enable or disable debug mode.

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-10-28 10:13:29 +08:00
aoiasd
20750c061b
fix: loss small bm25 binlogs (#36942) 
Sync task don't flush small bm25 logs, cause growing segment bm25 stats
loss.
relate: https://github.com/milvus-io/milvus/issues/36805

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2024-10-17 14:49:23 +08:00
yihao.dai
80f25d497f
enhance: Add metrics to monitor import throughput and imported rows (#36519) 
issue: https://github.com/milvus-io/milvus/issues/36518

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-09-28 17:31:15 +08:00
aoiasd
139787371e
feat: support embedding bm25 sparse vector and flush bm25 stats log (#36036) 
relate: https://github.com/milvus-io/milvus/issues/35853

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2024-09-19 10:57:12 +08:00
yihao.dai
6130a85444
enhance: Remove bf from streaming node (#35902) 
Remove bf from streaming node:
1. When watching vchannels, skip loading bloom filters for segments.
2. Bypass bloom filter checks for delete messages, directly writing to
L0 segments.
3. Remove flushed segments proactively after flush.

issue: https://github.com/milvus-io/milvus/issues/33285,
https://github.com/milvus-io/milvus/issues/34585

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-09-03 14:17:02 +08:00
yihao.dai
8aab6cbfac
enhance: Organize the common modules of streamingNode and dataNode (#34773) 
1. Move the common modules of streamingNode and dataNode to flushcommon
2. Add new GetVChannels interface for rootcoord

issue: https://github.com/milvus-io/milvus/issues/33285

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-07-22 11:33:51 +08:00