admin/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2026-01-01 16:35:29 +08:00
Code Issues Packages Projects Releases Wiki Activity
21,563 Commits 138 Branches 188 Tags
Commit Graph

21 Commits

Author SHA1 Message Date
congqixia
636e107e44
fix: Remove sync task after finished (#38681) 
Related to #38680

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-12-24 13:34:48 +08:00
jaime
29e620fa6d
fix: sync task still running after DataNode has stopped (#38377) 
issue: #38319

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-12-17 18:06:44 +08:00
congqixia
b0bd290a6e
enhance: Use internal json(sonic) to replace std json lib (#37708) 
Related to #35020

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-11-18 10:46:31 +08:00
XuanYang-cn
5a23c80f20
fix: Change memoryCheck write lock to read lock (#37525) 
See also: milvus-io#37493

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-11-15 10:44:31 +08:00
jaime
1d06d4324b
fix: Int64 overflow in JSON encoding (#37657) 
issue: ##36621

- For simple types in a struct, add "string" to the JSON tag for
automatic string conversion during JSON encoding.
- For complex types in a struct, replace "int64" with "string."

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-11-14 22:52:30 +08:00
jaime
1e8ea4a7e7
feat: add segment/channel/task/slow query render (#37561) 
issue: #36621

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-11-12 17:44:29 +08:00
Zhen Ye
49657c4690
enhance: add create segment message, enable empty segment flush (#37407) 
issue: #37172

- add redo interceptor to implement append context refresh. (make new
timetick)
- add create segment handler for flusher.
- make empty segment flushable and directly change it into dropped.
- add create segment message into wal when creating new growing segment.
- make the insert operation into following seq: createSegment -> insert
-> insert -> flushSegment.
- make manual flush into following seq: flushTs -> flushsegment ->
flushsegment -> manualflush.

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-11-08 10:16:34 +08:00
yihao.dai
81879425e1
enhance: Optimize the performance of stats task (#37374) 
1. Increase the writer's `batchSize` to avoid multiple serialization
operations.
2. Perform asynchronous upload of binlog files to prevent blocking the
data processing flow.
3. Reduce multiple calls to `writer.Flush()`.

issue: https://github.com/milvus-io/milvus/issues/37373

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-11-08 10:08:27 +08:00
jaime
f348bd9441
feat: add segment,pipeline, replica and resourcegroup api for WebUI (#37344) 
issue: #36621

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-11-07 11:52:25 +08:00
jaime
9d16b972ea
feat: add tasks page into management WebUI (#37002) 
issue: #36621

1. Add API to access task runtime metrics, including:
  - build index task
  - compaction task
  - import task
- balance (including load/release of segments/channels and some leader
tasks on querycoord)
  - sync task
2. Add a debug model to the webpage by using debug=true or debug=false
in the URL query parameters to enable or disable debug mode.

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-10-28 10:13:29 +08:00
aoiasd
fd72151037
fix: merge datanode bm25 error after reload growing segment with no data (#37154) 
Segment with numrow 0 don't init bm25 stats, cause flush with bm25 stats
failed.
relate: https://github.com/milvus-io/milvus/issues/37150

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2024-10-26 07:43:28 +08:00
aoiasd
20750c061b
fix: loss small bm25 binlogs (#36942) 
Sync task don't flush small bm25 logs, cause growing segment bm25 stats
loss.
relate: https://github.com/milvus-io/milvus/issues/36805

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2024-10-17 14:49:23 +08:00
yihao.dai
80f25d497f
enhance: Add metrics to monitor import throughput and imported rows (#36519) 
issue: https://github.com/milvus-io/milvus/issues/36518

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-09-28 17:31:15 +08:00
aoiasd
139787371e
feat: support embedding bm25 sparse vector and flush bm25 stats log (#36036) 
relate: https://github.com/milvus-io/milvus/issues/35853

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2024-09-19 10:57:12 +08:00
congqixia
c6eb6c7cb2
enhance: Add error handler for write buffer (#36216) 
Related to #36215

This PR add error handler setting option providing the possibility to
change error handling behavior other than panicking.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-09-13 10:11:09 +08:00
yihao.dai
6130a85444
enhance: Remove bf from streaming node (#35902) 
Remove bf from streaming node:
1. When watching vchannels, skip loading bloom filters for segments.
2. Bypass bloom filter checks for delete messages, directly writing to
L0 segments.
3. Remove flushed segments proactively after flush.

issue: https://github.com/milvus-io/milvus/issues/33285,
https://github.com/milvus-io/milvus/issues/34585

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-09-03 14:17:02 +08:00
congqixia
ab532ae199
enhance: Add back BF lazy load logic for datanode watch channel (#35646) 
Add back lazy loading statslog when watch dml channel on datanode.

Related to #22994 #27675

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-08-22 19:42:57 +08:00
Ted Xu
41646c8439
feat: integrate new deltalog format (#35522) 
See #34123

---------

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
 2024-08-20 19:06:56 +08:00
yihao.dai
a4439cc911
enhance: Implement flusher in streamingNode (#34942) 
- Implement flusher to:
  - Manage the pipelines (creation, deletion, etc.)
  - Manage the segment write buffer
  - Manage sync operation (including receive flushMsg and execute flush)
- Add a new `GetChannelRecoveryInfo` RPC in DataCoord.
- Reorganize packages: `flushcommon` and `datanode`.

issue: https://github.com/milvus-io/milvus/issues/33285

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-08-02 18:30:23 +08:00
zhenshan.cao
aa247f192d
enhance: remove unused code for StorageV2 (#35132) 
issue: https://github.com/milvus-io/milvus/issues/34168

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2024-08-01 12:08:13 +08:00
yihao.dai
8aab6cbfac
enhance: Organize the common modules of streamingNode and dataNode (#34773) 
1. Move the common modules of streamingNode and dataNode to flushcommon
2. Add new GetVChannels interface for rootcoord

issue: https://github.com/milvus-io/milvus/issues/33285

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-07-22 11:33:51 +08:00