admin/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2026-01-06 19:02:18 +08:00
Code Issues Packages Projects Releases Wiki Activity
21,899 Commits 137 Branches 188 Tags
Commit Graph

1333 Commits

Author SHA1 Message Date
cai.zhang
7bf40694fd
fix: Record the nodeID before assigning tasks (#36371) 
issue: #33744

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-09-28 17:21:15 +08:00
wayblink
7ff41697f9
enhance: make slotUsage a field of CompactionTask (#36510) 
#36509

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-09-26 20:19:14 +08:00
yihao.dai
9e8cafcbe2
enhance: Skip loading bf in datanode (#36367) 
Skip loading bf in datanode:
1. When watching vchannels, skip loading bloom filters for segments.
2. Bypass bloom filter checks for delete messages, directly writing to
L0 segments.
3. Remove flushed segments proactively after flush.

issue: https://github.com/milvus-io/milvus/issues/34585

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-09-26 10:11:15 +08:00
SimFG
c50fe71163
fix: long buffering causes mq to be unable to receive messages. (#36420) 
- issue: #36397

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-09-23 16:33:18 +08:00
congqixia
d2c774fb6d
fix: Return all compactTo segments after support split (#36361) 
Related to #36360

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-09-20 14:11:11 +08:00
aoiasd
139787371e
feat: support embedding bm25 sparse vector and flush bm25 stats log (#36036) 
relate: https://github.com/milvus-io/milvus/issues/35853

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2024-09-19 10:57:12 +08:00
yihao.dai
a61668c77e
feat: Introduce stats task for import (#35868) 
This PR introduce stats task for import:
1. Define new `Stats` and `IndexBuilding` states for importJob
2. Add new stats step to the import process: trigger the stats task and
wait for its completion
3. Abort stats task if import job failed

issue: https://github.com/milvus-io/milvus/issues/33744

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-09-15 15:17:08 +08:00
zhenshan.cao
9d8d332c88
fix: Fix improper use of offset in HybridSearch (#36244) 
issue :https://github.com/milvus-io/milvus/issues/36243

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2024-09-13 22:05:15 +08:00
jaime
22cce44afc
fix: metrics stored_index_files_size is never cleared (#36160) 
issue: #36159

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-09-13 20:09:15 +08:00
cai.zhang
8395c8a8db
enhance: Update stats task to optional (#35947) 
issue: #33744

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-09-12 20:37:08 +08:00
XuanYang-cn
e8840a1b41
enhance: Add metrics for Delete entries num of L0seg (#36175) 
- Add metrics *DataCoordL0DeleteEntriesNum*
- Remove metrics *DataCoordRateStoredL0Segment*

See also: #36147

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-09-12 18:07:08 +08:00
zhagnlu
208c8a2328
fix:support config index offsetcache and fix create same index again (#35985) 
#35971

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-09-08 18:23:05 +08:00
yihao.dai
6b4ae0c65e
enhance: Log warn on delayed compaction tasks (#36049) 
/kind enhancement

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-09-08 17:23:05 +08:00
Chun Han
e480b103bd
feat: supporing hybrid search group_by (#35982) 
related: #35096

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
 2024-09-08 17:09:04 +08:00
zhagnlu
325f1987d9
enhance: rewrite index params for compatibility (#35788) 
#32900

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-09-02 19:41:03 +08:00
cai.zhang
2c9bb4dfa3
feat: Support stats task to sort segment by PK (#35054) 
issue: #33744 

This PR includes the following changes:
1. Added a new task type to the task scheduler in datacoord: stats task,
which sorts segments by primary key.
2. Implemented segment sorting in indexnode.
3. Added a new field `FieldStatsLog` to SegmentInfo to store token index
information.

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-09-02 14:19:03 +08:00
yihao.dai
1413ffe9b1
enhance: Rename preAllocatedSegments (#35871) 
Rename `preAllocatedSegments` to `preAllocatedSegmentIDs` to avoid
confusion.

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-09-01 17:09:01 +08:00
XuanYang-cn
323400c190
enhance: Enable to write multiple segments in mix compactor (#35705) 
Prevent segments to be written larger than maxSize * expansionRate

See also: #35584

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-08-30 11:29:01 +08:00
Zhen Ye
99dff06391
enhance: using streaming service in insert/upsert/flush/delete/querynode (#35406) 
issue: #33285

- using streaming service in insert/upsert/flush/delete/querynode
- fixup flusher bugs and refactor the flush operation
- enable streaming service for dml and ddl
- pass the e2e when enabling streaming service
- pass the integration tst when enabling streaming service

---------

Signed-off-by: chyezh <chyezh@outlook.com>
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-08-29 10:03:08 +08:00
jaime
b7ea1defd3
fix: mistaken deletions may occur during GC channel checkpoints (#35707) 
issue: #35706

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-08-28 10:11:05 +08:00
XuanYang-cn
f12e368a76
fix: Fill nil schema so that Milvus can watch channel for those upgraded from 2.2 to 2.4 #35695 (#35694) 
See also: [#35701 ](https://github.com/milvus-io/milvus/issues/35701)

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-08-27 10:36:59 +08:00
XuanYang-cn
82743c5c50
fix: Clear channelcp meta and metrics ASAP (#35658) 
See also: #35588

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-08-26 12:22:57 +08:00
Patrick Weizhi Xu
4236c3bafb
enhance: enable MV on binary vectors (#35668) 
Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
 2024-08-25 16:27:09 +08:00
zhagnlu
42f7800b5b
enhance: add bitmap offset cache to speed up retrieve raw data (#35498) 
#35458

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-08-24 01:40:58 +08:00
congqixia
582d2eec79
enhance: Move datanode/indexnode manager to session pkg (#35634) 
Related to #28861

Move session manager, worker manager to session package. Also renaming
each manager to corresponding node name(datanode, indexnode).

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-08-22 16:02:56 +08:00
congqixia
c992a61a23
enhance: Separate allocator pkg in datacoord (#35622) 
Related to #28861

Move allocator interface and implementation into separate package. Also
update some unittest logic.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-08-22 10:06:56 +08:00
SimFG
731d45abbe
enhance: provide more general configuration to control mmap behavior (#35359) 
- issue: #35273

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-08-21 00:22:54 +08:00
yihao.dai
227ecd3901
enhance: Remove the check for channel cp lag when generating compaction plan (#35383) 
issue: https://github.com/milvus-io/milvus/issues/35382

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-08-19 19:40:55 +08:00
wei liu
1d49358f82
enhance: Add BackupRBAC/RestoreRBAC API to enable rbac backup (#35444) 
issue: #35443

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-08-16 10:10:53 +08:00
XuanYang-cn
c42976ee6f
enhance: Init ChannelCP when creating a channel (#35387) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-08-14 10:16:58 +08:00
CharlesFeng
f5b4a972a0
fix: checkResultTicker leak (#35377) 
https://github.com/milvus-io/milvus/issues/35376

Signed-off-by: fengjun2016 <jornfeng@gmail.com>
 2024-08-13 21:36:52 +08:00
jaime
1b0ea49d25
fix: incorrect memory size and delta count in SegmentInfo (#35369) 
issue: #35368

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-08-13 17:20:19 +08:00
XuanYang-cn
06f9ba294e
enhance: Add more ut for l0CompactionTask (#35100) 
See also: #34796

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-08-09 14:24:18 +08:00
XuanYang-cn
61a426bc6f
enhance: Init channel cp metric when DataCoord recover (#35353) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-08-09 10:08:18 +08:00
cai.zhang
aaab827a16
fix: Fix the issue of missing stats log after clustering compaction (#35266) 
issue: #35265

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-08-08 14:24:17 +08:00
wei liu
838f06323f
fix: datacoord binlog size metrics include dropped segment (#35338) 
issue: #35337

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-08-07 16:24:16 +08:00
wayblink
c6253f9c9b
enhance: improve ut cov of clustering compaction task (#35242) 
#34792

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-08-07 14:44:21 +08:00
wayblink
bb15ecdc13
enhance: improve ut for compaction_policy_clustering (#35205) 
#34792

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-08-07 14:42:17 +08:00
yihao.dai
678018d9ca
enhance: Avoid unnecessary compaction (#35148) 
Estimate the import segment size based on DiskSegmentMaxSize(2G) to
avoid unnecessary compaction after import completed.

issue: https://github.com/milvus-io/milvus/issues/35147

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-08-06 10:30:21 +08:00
congqixia
a2b517523d
enhance: Add in-memory cache for casbin enforcer result (#35271) 
See also #35270

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-08-05 18:48:15 +08:00
yihao.dai
b71e058bc5
enhance: Add import option to skip disk quota check (#35274) 
Add an option to skip the disk quota check for backup-restore import.

issue: https://github.com/milvus-io/milvus/issues/33775

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-08-05 16:40:16 +08:00
cai.zhang
6542c1ab0e
enhance: Add monitoring metrics for task execution time in datacoord (#35139) 
issue: #35138

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-08-05 16:26:17 +08:00
chyezh
9871966415
enhance: segment alloc interceptor (#34996) 
#33285

- add segment alloc interceptor for streamingnode.
- add add manual alloc segment rpc for datacoord.

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-08-04 07:40:15 +08:00
yihao.dai
a4439cc911
enhance: Implement flusher in streamingNode (#34942) 
- Implement flusher to:
  - Manage the pipelines (creation, deletion, etc.)
  - Manage the segment write buffer
  - Manage sync operation (including receive flushMsg and execute flush)
- Add a new `GetChannelRecoveryInfo` RPC in DataCoord.
- Reorganize packages: `flushcommon` and `datanode`.

issue: https://github.com/milvus-io/milvus/issues/33285

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-08-02 18:30:23 +08:00
wayblink
81773bfadf
enhance: add commit time in partitionStats proto (#35125) 
fix: #35110

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-08-02 16:16:14 +08:00
wei liu
f466129924
enhance: add segment state on data coord bin log size metrics (#35153) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-08-02 14:48:14 +08:00
wayblink
95462668ca
enhance: unify time in clustering compaction task to unix (#35167) 
#34495

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-08-02 10:30:19 +08:00
cai.zhang
196a7986b3
enhance: Change the fixed value to a ratio for clustering segment size (#35076) 
issue: #34495

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-08-01 22:04:14 +08:00
wayblink
5bbb1c201c
enhance:support l2 single compaction (#34935) 
#34928

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-08-01 14:36:13 +08:00
zhenshan.cao
aa247f192d
enhance: remove unused code for StorageV2 (#35132) 
issue: https://github.com/milvus-io/milvus/issues/34168

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2024-08-01 12:08:13 +08:00