cai.zhang
feeb869ff9
enhance: Remove compaction plans on the datanode ( #33548 )
...
issue: #33546
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-06-05 20:27:51 +08:00
yihao.dai
01ce32caa1
enhance: Print more disk quota info ( #33596 )
...
/kind enhancement
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-06-05 16:51:56 +08:00
yihao.dai
35532a3e7d
fix: Fill stats log id and check validity ( #33477 )
...
1. Fill log ID of stats log from import
2. Add a check to validate the log ID before writing to meta
issue: https://github.com/milvus-io/milvus/issues/33476
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-06-05 11:17:56 +08:00
zhenshan.cao
ac4f3997ce
enhance: Reconstructing Compaction to possess persistence capability ( #33265 )
...
issue #33586
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-06-05 10:17:50 +08:00
yiwangdr
180d754158
fix: speed up segment lookup via channel name in datacoord ( #33530 )
...
issue: #33342
Signed-off-by: yiwangdr <yiwangdr@gmail.com>
2024-06-03 14:47:47 +08:00
yihao.dai
7a2127b09f
enhance: Avoid redundant meta operations of import ( #33518 )
...
issue: https://github.com/milvus-io/milvus/issues/33513
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-06-03 14:15:53 +08:00
cai.zhang
77637180fa
enhance: Periodically synchronize segments to datanode watcher ( #33420 )
...
issue: #32809
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-05-30 13:37:44 +08:00
congqixia
54797b4286
enhance: Refine frequent log in datacoord ( #33449 )
...
This PR changes:
- Frequent `ListIndexes` success log to debug level
- Aggregate collection missing log after collection dropped in
`meta.GetCollectionIndexFilesSize`
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-05-29 19:15:43 +08:00
jaime
3d29907b6e
enhance: decrease cpu overhead during filter segments on datacoord ( #33130 )
...
issue: #33129
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-05-28 19:17:43 +08:00
XuanYang-cn
5e39aa9272
enhance: Make channel meta able to writer 200k plus segments ( #33279 )
...
See also: #33125
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-05-28 12:33:42 +08:00
SimFG
2453181218
fix: not found database name in the datacoord meta object ( #33411 )
...
- issue: #33410
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-05-28 10:09:48 +08:00
yiwangdr
e895cfed84
fix: reduce redundant map operations in datacoord ( #33343 )
...
More refactories will be added.
issue: #33342
Signed-off-by: yiwangdr <yiwangdr@gmail.com>
2024-05-24 12:47:40 +08:00
yihao.dai
7730b910b9
enhance: Decouple compaction from shard ( #33138 )
...
Decouple compaction from shard, remove dependencies on shards (e.g.
SyncSegments, injection).
issue: https://github.com/milvus-io/milvus/issues/32809
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-05-24 09:07:41 +08:00
cai.zhang
be77ceba84
enhance: Use proto for passing info in cgo ( #33184 )
...
issue: #33183
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-05-23 10:31:40 +08:00
XuanYang-cn
22bddde5ff
enhance: Tidy compactor and remove dup codes ( #32198 )
...
See also: #32451
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-05-23 09:53:40 +08:00
XuanYang-cn
819a624753
fix: Return error when startup Delete/AddNode fail ( #33193 )
...
See also: #33151 , #33149
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-05-22 11:17:38 +08:00
cai.zhang
ed39a38953
enhance: Reduce the frequency of logs describing indexing failures ( #33212 )
...
issue: #33001 #33102
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-05-21 19:27:39 +08:00
yihao.dai
017fd7bc25
enhance: Select L2 segments in L0Compaction as well ( #32991 )
...
/kind improvement
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-05-21 16:13:39 +08:00
congqixia
f31a20faad
fix: [Backport] Mark channel checkpoint dropped prevent cp lag metrics leakage ( #32454 ) ( #33198 )
...
Cherry-pick from 2.3
pr: #32454
See also #31506 #31508
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-05-21 11:59:39 +08:00
jaime
0d99db23b8
fix: metrics leak on the coord nodes ( #33075 )
...
issue: #32980
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-05-20 22:03:39 +08:00
congqixia
f76d16780b
enhance: Refine channel mgr v2 implementation ( #33156 )
...
Related to #25309
- Remove ctx from struct
- Add ctx parameters for internal check logic methods
- Add Waitgroup to make sure worker goroutine quit before close returns
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-05-20 14:13:38 +08:00
yihao.dai
32560263fa
enhance: Query slot for compaction task ( #32881 )
...
Query slot of compaction in datanode, and transfer the control logic for
limiting compaction tasks from datacoord to the datanode.
issue: https://github.com/milvus-io/milvus/issues/32809
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-05-17 18:19:38 +08:00
congqixia
8cf2cf5c94
enhance: Add go-deadlock as unittest only dependency ( #33063 )
...
See also #33062
This PR:
- Add `lock.RWMutex` & `lock.Mutex` alias to switch implementation based
on build flags
- When build flags has `test` in it, use `go-deadlock` to detect
possible deadlocks
- Replace all `sync.RWMutex` & `sync.Mutex` in datacoord pkg
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-05-15 16:33:34 +08:00
cai.zhang
4382cf5283
enhance: Reduce the frequency of index ralted logs ( #33050 )
...
issue: #33001
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-05-15 14:37:34 +08:00
wayblink
259bc97a2b
fix: Fix segments lost in flush response ( #33061 )
...
#33055
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-05-15 13:49:34 +08:00
cai.zhang
6ea7633bd5
enhance: Add memory size for binlog ( #33025 )
...
issue: #33005
1. add `MemorySize` field for insert binlog.
2. `LogSize` means the file size in the storage object.
3. `MemorySize` means the size of the data in the memory.
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2024-05-15 12:59:34 +08:00
jaime
f48a7ff8ff
enhance: use Delete instead of DeletePartialMatch to remove metrics ( #33029 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-05-14 18:49:33 +08:00
congqixia
dea7faf394
fix: Remove RLock re-entry in GetNumRowsOfPartition ( #33045 )
...
Resolves #33044
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-05-14 17:51:32 +08:00
XuanYang-cn
40e5c0be70
fix: Enable to balance released standby channel ( #32986 )
...
See also: #32879
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-05-14 11:51:32 +08:00
jaime
ba625835bc
enhance: Add metrics for segment index files size ( #32979 )
...
issue:#32980
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-05-13 17:59:32 +08:00
XuanYang-cn
bc015df962
fix: [skip-e2e] Fix unstable policy_test ( #32883 )
...
See also: #32849
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-05-09 15:39:40 +08:00
congqixia
cedb33ceec
enhance: Improve datacoord segment filtering with collection ( #32831 )
...
See also #32165
This PR modify the `SelectSegments` interface to utilizing collection id
information when selecting segment with provided collection
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-05-08 21:37:29 +08:00
XuanYang-cn
0b0d279515
fix: [skip-e2e] Fix AvgAssignByCount unstable ( #32850 )
...
See also: #32849
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-05-08 19:09:35 +08:00
yihao.dai
a8db16ad4b
enhance: Change ddl log to info ( #32838 )
...
/kind improvement
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-05-08 14:17:50 +08:00
XuanYang-cn
6843d6d376
fix: Compaction trigger choose 2 same segments ( #32800 )
...
DataNode would stuck at compactor try to lock the
same segmentID
See also: #32765
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-05-07 19:01:31 +08:00
yiwangdr
b1eacb2ae8
feat: datacoord/node watch based on rpc ( #32036 )
...
issue: https://github.com/milvus-io/milvus/issues/25309
Signed-off-by: yiwangdr <yiwangdr@gmail.com>
2024-05-07 15:49:30 +08:00
Bingyi Sun
fecd9c21ba
feat: LRU cache implementation ( #32567 )
...
issue: https://github.com/milvus-io/milvus/issues/32783
This pr is the implementation of lru cache on branch lru-dev.
Signed-off-by: sunby <sunbingyi1992@gmail.com>
Co-authored-by: chyezh <chyezh@outlook.com>
Co-authored-by: MrPresent-Han <chun.han@zilliz.com>
Co-authored-by: Ted Xu <ted.xu@zilliz.com>
Co-authored-by: jaime <yun.zhang@zilliz.com>
Co-authored-by: wayblink <anyang.wang@zilliz.com>
2024-05-06 20:29:30 +08:00
congqixia
6af5275066
fix: Refine index param check for Create/Alter Index ( #32712 )
...
See also #32711
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-05-06 15:43:29 +08:00
chyezh
77f4f0c5a8
fix: gc only do once ( #32722 )
...
issue: #29655
produced by pr: #31740
Signed-off-by: chyezh <chyezh@outlook.com>
2024-04-30 14:19:27 +08:00
wayblink
42d0412e93
enhance: Add channelCPs in FlushResponse ( #32044 )
...
#32609
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-04-30 09:45:27 +08:00
jaime
3b5cbed86a
enhance: add db label for deletion and bulkinsert operation ( #32611 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-04-26 15:53:29 +08:00
chyezh
2586c2f1b3
enhance: use WalkWithPrefix api for oss, enable piplined file gc ( #31740 )
...
issue: #19095,#29655,#31718
- Change `ListWithPrefix` to `WalkWithPrefix` of OOS into a pipeline
mode.
- File garbage collection is performed in other goroutine.
- Segment Index Recycle clean index file too.
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-04-25 20:41:27 +08:00
wei liu
04f355a802
enhance: Enable alter database props in rootcoord ( #32458 )
...
issue: #30040
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-04-25 10:53:25 +08:00
Ted Xu
fef7812254
enhance: trigger compaction by size ( #32326 )
...
See #32328
---------
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-04-23 12:11:24 +08:00
yiwangdr
037de8e4d3
enhance: speed up minor functions calls in datacoord ( #32389 )
...
Related to https://github.com/milvus-io/milvus/issues/32165
1. nodeid based channel store access should use map access instead of
iteration.
2. The join-ish functions calls are slow when # collections/segments
increases (e.g. 10k).
e.g.
getNumRowsOfCollectionUnsafe is O(num_segments); GetAllCollectionNumRows
is of O(num_collections*num_segments).
Signed-off-by: yiwangdr <yiwangdr@gmail.com>
2024-04-20 07:55:21 +08:00
XuanYang-cn
7e830f6185
fix: [skip-e2e] TestAvgReassignPolicy ut ( #32450 )
...
See also: #32448
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-04-19 15:11:20 +08:00
zhagnlu
e2c38750c7
fix: modify retry error ( #32351 )
...
#32322
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-04-18 14:25:14 +08:00
zhagnlu
4586bcef9f
fix: correct AssignSegmentID return and add retry for loadCollectionF… ( #32335 )
...
#32322
#31942
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-04-16 10:20:10 -07:00
congqixia
83da08c388
enhance: Use map instead of slice to maintain channel info ( #32273 )
...
See also #32165
`ChannelManager.Match` is a frequent operation for datacoord. When the
collection number is large, iteration over all channels will cost lots
of CPU time and time consuming.
This PR change the data structure storing datanode-channel info to map
avoiding this iteration when checking channel existence.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-04-16 15:57:19 +08:00
wei liu
0d849a6c0a
fix: fix collectionInfo leak in datacoord ( #32175 )
...
issue: #32029
lack of logic to clean collection info in datacoord's meta, This PR
clean collection info after drop channel, to avoid collection info leak
in datacoord
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-04-15 16:33:19 +08:00
