295 Commits

Author SHA1 Message Date
congqixia
f9bb8e9648
enhance: Change const magic number in querycoord to param (#28819)
See also #28817

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-11-30 09:06:28 +08:00
cqy123456
3b1b14dd78
fix: update binlog index memory uasge before loading segments (#28528)
issue: #27678 
when interimIndex = true, memory predict should be update with the
memory usage of binlog index build process.

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2023-11-29 16:42:27 +08:00
XuanYang-cn
606ec77b66
enhance: Unify levelzero segment config in DN (#28720)
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-11-28 18:04:26 +08:00
aoiasd
89d8ce2f73
enhance: refine access log to support format access log by yaml and print name info. (#28319)
relate: https://github.com/milvus-io/milvus/issues/28086

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-11-28 15:32:31 +08:00
Bingyi Sun
2843664a20
fix: channel name can not contains '-partition-' (#28689)
issue: https://github.com/milvus-io/milvus/issues/28675

Signed-off-by: sunby <sunbingyi1992@gmail.com>
2023-11-24 10:14:24 +08:00
wei liu
4b8d3eaa41
enhance: Increase heartbeat timeout (#28648)
it's easy to trigger heartbeat timeout after 100ms when standalone cpu
usage reach 100%.
This PR increase the heartbeat timeout param to 2000ms

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-11-23 19:50:24 +08:00
Bingyi Sun
4fedff6d47
feat: integrate storage v2 into the write path (#28440)
#28378

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
2023-11-23 17:26:24 +08:00
XuanYang-cn
e88bbaac24
feat: Add universal levelzero segment switch (#28483)
See also: #27349

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-11-20 19:04:22 +08:00
SimFG
cfb6edea61
Support to trace the grpc request (#28349)
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-11-13 20:16:18 +08:00
XuanYang-cn
f8aa46419a
Add LevelZeroCompaction configs (#28190)
See also: #27606

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-11-13 11:18:19 +08:00
SimFG
7dda2e8814
Change some log level in the pkg package (#28181)
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-11-08 23:34:22 +08:00
zhenshan.cao
2057fdce43
Change default storage type to minio (#28236)
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2023-11-07 21:15:31 +08:00
wei liu
5b45a138b1
disable auto balance when old node exists (#28191)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-11-07 14:02:20 +08:00
Xiaofan
da19e49daf
Support purge old session for standalone (#28184)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-11-06 21:21:42 +08:00
yah01
90e2c63d9e
Fix getting incorrect CPU num (#28146)
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-11-06 06:02:16 +08:00
congqixia
bf2f62c1e7
Add WriteBuffer to provide abstraction for delta policy (#27874)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-11-04 12:10:17 +08:00
aoiasd
d2727cc0ff
Add write cache for access log (#27792)
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-11-03 10:28:17 +08:00
Enwei Jiao
8ae9c947ae
Use OpenDAL to access object store (#25642)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-11-01 09:00:14 +08:00
cqy123456
4fbe3c9142
replace loaded binlog with binlog index for search performance (#27673)
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2023-11-01 02:20:15 +08:00
MrPresent-Han
ccaec44930
enable parameter for dataCoord broker timeout(#26972) (#27825)
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2023-10-28 06:14:11 +08:00
aoiasd
53246b1b38
Set accesslog default to close and use stdout (#27891)
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-10-25 10:30:10 +08:00
wei liu
40723a292e
reduce compact parallel task num (#27899)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-10-25 09:40:12 +08:00
congqixia
93a877f55e
Make qcv2 target&leader observer execute in parallel (#27844)
- Add `taskDispatcher` to submit and run task async safely
- Change `LeaderObeserver` and `TargetObserver` schedule and manual check action to submitting task into dispatcher
- Fix logic problem in collection observer when manual check return false

See also #27494

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-10-24 10:14:11 +08:00
SimFG
9b0ecbdca7
Support to replicate the mq message (#27240)
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-10-20 14:26:09 +08:00
smellthemoon
4b0ec156b3
Set channel work pool size in datanode (#27728)
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2023-10-19 08:28:08 +08:00
yah01
9467de79c3
Use pool to block exceeded tasks (#27767)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-10-19 03:30:06 +08:00
wei liu
43f2452f4b
speed up shard leader cache update frequency (#27640)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-10-16 10:52:10 +08:00
XuanYang-cn
56c94cdfa7
Add channel manager in DataNode (#27308)
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-10-08 21:37:33 +08:00
yihao.dai
106c17f304
Make read ahead policy in ChunkCache configurable (#27291)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-09-28 15:47:27 +08:00
MrPresent-Han
505e8509b7
refine the default values for segcore threadpool size (#27406)
/kind improvement

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2023-09-27 20:43:28 +08:00
wei liu
4071132f6a
reload loading collection when qc recover (#27300)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-09-27 11:43:28 +08:00
foxspy
5db4a0489e
dynamic index version control (#27335)
Co-authored-by: longjiquan <jiquan.long@zilliz.com>
2023-09-25 21:39:27 +08:00
foxspy
370b6fde58
milvus support multi index engine (#27178)
Co-authored-by: longjiquan <jiquan.long@zilliz.com>
2023-09-22 09:59:26 +08:00
SimFG
26f06dd732
Format the code (#27275)
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-09-21 09:45:27 +08:00
Xiaofan
6635398a6d
Fix Bin log concurrency by adding a pool (#27189)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-09-19 18:05:22 +08:00
Enwei Jiao
fb0705df1b
Decouple basetable and componentparam (#26725)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-09-05 10:31:48 +08:00
wei liu
1097776477
stop heartbeat if reach heartbeat limit (#26728)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-09-04 17:51:48 +08:00
XuanYang-cn
b2e7cbdf4b
Remove TimeTravel in compactor (#26785)
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-09-04 17:41:48 +08:00
yihao.dai
4340cbfba2
Merge syncCPLagTooBehind policy into syncPeriodically policy (#26713)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-09-04 14:37:47 +08:00
XuanYang-cn
86e9f1818f
Change default compaction timeout to 15mins (#26757)
See also: #26566

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-08-31 18:07:00 +08:00
Gao
7e36f819b4
Make autoindex config clean (#26732)
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2023-08-31 10:25:01 +08:00
SimFG
d13ca54414
Change the default gracefulStopTimeout and only warn when fail to refresh policy cache (#26450)
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-08-22 09:04:21 +08:00
wayblink
c5a1b41f95
Update session ttl to 60s (#26346)
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2023-08-18 22:52:20 +08:00
wei liu
79ccf06cf6
refine proxy to querynode heartbeat interval (#26426)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-08-18 14:26:19 +08:00
jaime
4a388cf450
[skip e2e] Align default value with milvus.yaml file (#25619)
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-08-17 20:24:19 +08:00
XuanYang-cn
be30838cf9
Refine some comments (#26401)
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-08-17 10:54:18 +08:00
MrPresent-Han
3421956afa
modify default value for sync max parallel to mitigate oom on dn(#26763) (#26231)
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2023-08-10 10:23:15 +08:00
wei liu
f33a89387f
enable config shard level cost metrics whether contains worker's cost (#26132)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-08-09 11:37:15 +08:00
congqixia
767955ec6b
Reduce MQ buffer length and flowgraph wait queue length to 16 (#26179)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-08-09 10:05:14 +08:00
xige-16
d1d0169fa3
Delete useless config (#26173)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2023-08-08 10:23:08 +08:00