zhagnlu
208c8a2328
fix:support config index offsetcache and fix create same index again ( #35985 )
...
#35971
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-09-08 18:23:05 +08:00
yihao.dai
6b4ae0c65e
enhance: Log warn on delayed compaction tasks ( #36049 )
...
/kind enhancement
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-09-08 17:23:05 +08:00
Chun Han
e480b103bd
feat: supporing hybrid search group_by ( #35982 )
...
related: #35096
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-09-08 17:09:04 +08:00
zhagnlu
325f1987d9
enhance: rewrite index params for compatibility ( #35788 )
...
#32900
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-09-02 19:41:03 +08:00
cai.zhang
2c9bb4dfa3
feat: Support stats task to sort segment by PK ( #35054 )
...
issue: #33744
This PR includes the following changes:
1. Added a new task type to the task scheduler in datacoord: stats task,
which sorts segments by primary key.
2. Implemented segment sorting in indexnode.
3. Added a new field `FieldStatsLog` to SegmentInfo to store token index
information.
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-09-02 14:19:03 +08:00
yihao.dai
1413ffe9b1
enhance: Rename preAllocatedSegments ( #35871 )
...
Rename `preAllocatedSegments` to `preAllocatedSegmentIDs` to avoid
confusion.
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-09-01 17:09:01 +08:00
XuanYang-cn
323400c190
enhance: Enable to write multiple segments in mix compactor ( #35705 )
...
Prevent segments to be written larger than maxSize * expansionRate
See also: #35584
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-08-30 11:29:01 +08:00
Zhen Ye
99dff06391
enhance: using streaming service in insert/upsert/flush/delete/querynode ( #35406 )
...
issue: #33285
- using streaming service in insert/upsert/flush/delete/querynode
- fixup flusher bugs and refactor the flush operation
- enable streaming service for dml and ddl
- pass the e2e when enabling streaming service
- pass the integration tst when enabling streaming service
---------
Signed-off-by: chyezh <chyezh@outlook.com>
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-08-29 10:03:08 +08:00
jaime
b7ea1defd3
fix: mistaken deletions may occur during GC channel checkpoints ( #35707 )
...
issue: #35706
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-08-28 10:11:05 +08:00
XuanYang-cn
f12e368a76
fix: Fill nil schema so that Milvus can watch channel for those upgraded from 2.2 to 2.4 #35695 ( #35694 )
...
See also: [#35701 ](https://github.com/milvus-io/milvus/issues/35701 )
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-08-27 10:36:59 +08:00
XuanYang-cn
82743c5c50
fix: Clear channelcp meta and metrics ASAP ( #35658 )
...
See also: #35588
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-08-26 12:22:57 +08:00
Patrick Weizhi Xu
4236c3bafb
enhance: enable MV on binary vectors ( #35668 )
...
Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
2024-08-25 16:27:09 +08:00
zhagnlu
42f7800b5b
enhance: add bitmap offset cache to speed up retrieve raw data ( #35498 )
...
#35458
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-08-24 01:40:58 +08:00
congqixia
582d2eec79
enhance: Move datanode/indexnode manager to session pkg ( #35634 )
...
Related to #28861
Move session manager, worker manager to session package. Also renaming
each manager to corresponding node name(datanode, indexnode).
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-22 16:02:56 +08:00
congqixia
c992a61a23
enhance: Separate allocator pkg in datacoord ( #35622 )
...
Related to #28861
Move allocator interface and implementation into separate package. Also
update some unittest logic.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-22 10:06:56 +08:00
SimFG
731d45abbe
enhance: provide more general configuration to control mmap behavior ( #35359 )
...
- issue: #35273
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-08-21 00:22:54 +08:00
yihao.dai
227ecd3901
enhance: Remove the check for channel cp lag when generating compaction plan ( #35383 )
...
issue: https://github.com/milvus-io/milvus/issues/35382
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-08-19 19:40:55 +08:00
wei liu
1d49358f82
enhance: Add BackupRBAC/RestoreRBAC API to enable rbac backup ( #35444 )
...
issue: #35443
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-16 10:10:53 +08:00
XuanYang-cn
c42976ee6f
enhance: Init ChannelCP when creating a channel ( #35387 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-08-14 10:16:58 +08:00
CharlesFeng
f5b4a972a0
fix: checkResultTicker leak ( #35377 )
...
https://github.com/milvus-io/milvus/issues/35376
Signed-off-by: fengjun2016 <jornfeng@gmail.com>
2024-08-13 21:36:52 +08:00
jaime
1b0ea49d25
fix: incorrect memory size and delta count in SegmentInfo ( #35369 )
...
issue: #35368
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-08-13 17:20:19 +08:00
XuanYang-cn
06f9ba294e
enhance: Add more ut for l0CompactionTask ( #35100 )
...
See also: #34796
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-08-09 14:24:18 +08:00
XuanYang-cn
61a426bc6f
enhance: Init channel cp metric when DataCoord recover ( #35353 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-08-09 10:08:18 +08:00
cai.zhang
aaab827a16
fix: Fix the issue of missing stats log after clustering compaction ( #35266 )
...
issue: #35265
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-08-08 14:24:17 +08:00
wei liu
838f06323f
fix: datacoord binlog size metrics include dropped segment ( #35338 )
...
issue: #35337
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-07 16:24:16 +08:00
wayblink
c6253f9c9b
enhance: improve ut cov of clustering compaction task ( #35242 )
...
#34792
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-08-07 14:44:21 +08:00
wayblink
bb15ecdc13
enhance: improve ut for compaction_policy_clustering ( #35205 )
...
#34792
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-08-07 14:42:17 +08:00
yihao.dai
678018d9ca
enhance: Avoid unnecessary compaction ( #35148 )
...
Estimate the import segment size based on DiskSegmentMaxSize(2G) to
avoid unnecessary compaction after import completed.
issue: https://github.com/milvus-io/milvus/issues/35147
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-08-06 10:30:21 +08:00
congqixia
a2b517523d
enhance: Add in-memory cache for casbin enforcer result ( #35271 )
...
See also #35270
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-05 18:48:15 +08:00
yihao.dai
b71e058bc5
enhance: Add import option to skip disk quota check ( #35274 )
...
Add an option to skip the disk quota check for backup-restore import.
issue: https://github.com/milvus-io/milvus/issues/33775
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-08-05 16:40:16 +08:00
cai.zhang
6542c1ab0e
enhance: Add monitoring metrics for task execution time in datacoord ( #35139 )
...
issue: #35138
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-08-05 16:26:17 +08:00
chyezh
9871966415
enhance: segment alloc interceptor ( #34996 )
...
#33285
- add segment alloc interceptor for streamingnode.
- add add manual alloc segment rpc for datacoord.
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-08-04 07:40:15 +08:00
yihao.dai
a4439cc911
enhance: Implement flusher in streamingNode ( #34942 )
...
- Implement flusher to:
- Manage the pipelines (creation, deletion, etc.)
- Manage the segment write buffer
- Manage sync operation (including receive flushMsg and execute flush)
- Add a new `GetChannelRecoveryInfo` RPC in DataCoord.
- Reorganize packages: `flushcommon` and `datanode`.
issue: https://github.com/milvus-io/milvus/issues/33285
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-08-02 18:30:23 +08:00
wayblink
81773bfadf
enhance: add commit time in partitionStats proto ( #35125 )
...
fix : #35110
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-08-02 16:16:14 +08:00
wei liu
f466129924
enhance: add segment state on data coord bin log size metrics ( #35153 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-02 14:48:14 +08:00
wayblink
95462668ca
enhance: unify time in clustering compaction task to unix ( #35167 )
...
#34495
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-08-02 10:30:19 +08:00
cai.zhang
196a7986b3
enhance: Change the fixed value to a ratio for clustering segment size ( #35076 )
...
issue: #34495
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-08-01 22:04:14 +08:00
wayblink
5bbb1c201c
enhance:support l2 single compaction ( #34935 )
...
#34928
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-08-01 14:36:13 +08:00
zhenshan.cao
aa247f192d
enhance: remove unused code for StorageV2 ( #35132 )
...
issue: https://github.com/milvus-io/milvus/issues/34168
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-08-01 12:08:13 +08:00
cai.zhang
089fb2b5fb
enhance: Skip updating the level that are already at the specified level ( #35094 )
...
issue: #34495
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-08-01 11:18:12 +08:00
Patrick Weizhi Xu
acaa78db58
fix: disable sparse and binary in MV ( #35126 )
...
issue: #35120
Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
(cherry picked from commit 0d340ebe7412cd85963825bc8abd3c874e339679)
2024-07-31 15:02:01 +08:00
wayblink
f919b6e1b8
enhance: refine error handle in clustering compaction task ( #34956 )
...
related: #30633 #34792
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-30 20:37:57 +08:00
wayblink
ce3f836876
fix: compaction task not be cleaned correctly ( #34765 )
...
1.fix compaction task not be cleaned correctly
2.add a new parameter to control compaction gc loop interval
3.remove some useless configs of clustering compaction
bug: #34764
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-30 20:21:56 +08:00
congqixia
de8a266d8a
enhance: Enable linux code checker ( #35084 )
...
See also #34483
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-30 15:53:51 +08:00
chyezh
a495754034
enhance: the datacoord gc should fast quitable ( #35050 )
...
issue: #35049
Signed-off-by: chyezh <chyezh@outlook.com>
2024-07-29 17:03:48 +08:00
wei liu
c45f38aa61
enhance: Update protobuf-go to protobuf-go v2 ( #34394 )
...
issue: #34252
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-29 11:31:51 +08:00
XuanYang-cn
9a07790215
enhance: Add ut for l0CompactionTask processExecuting ( #34800 )
...
See also: #34796
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-07-29 10:53:48 +08:00
cai.zhang
497afcb897
enhance: Refine code for GetRecoveryInfo ( #34973 )
...
issue: #34495
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-29 00:01:46 +08:00
wei liu
6e9fbd1630
enhance: Solve channel unbalance on datanode ( #34984 )
...
issue: #33583
the old policy permit datanode has at most 2 more channels than other
datanode. so if milvus has 2 datanode and 2 channels, both 2 channels
will be assign to 1 datanode, left another datanode empty.
This PR refine the balance policy to solve channel unbalance on datanode
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-26 17:35:47 +08:00
chyezh
1cff55381d
enhance: add manual alloc segment rpc for datacoord ( #35002 )
...
issue: #33285
- segment allocation will move to streamingnode, so a manual alloc
segment rpc is required
Signed-off-by: chyezh <chyezh@outlook.com>
2024-07-26 10:15:46 +08:00
